BigQuery Search: Unify Structured & Unstructured Data

BigQuery just dropped a triple threat of search innovations that make unstructured data feel right at home in your warehouse. No more stitching together separate vector databases or managing complex embedding pipelines—Google’s data warehouse is now a one-stop swamp for all your data, structured or not.

Autonomous Embedding Generation (GA)

Forget about building and maintaining your own embedding infrastructure. BigQuery now automatically generates embeddings as new data rolls in, using either Vertex AI models or native Gemma models. Just define a column in your schema, and BigQuery handles the rest—asynchronously and continuously.

This feature also supports generating embeddings over images via ObjectRefs, unlocking true multimodal search. No more wrangling separate pipelines; just let BigQuery do the heavy lifting while you focus on building cool stuff.

AI.SEARCH (GA) – Natural Language at Scale

Once your data is embedded, querying it becomes a breeze with the new AI.SEARCH() function. It lets you find semantically related records using natural language, without needing to generate embeddings in your search path. And thanks to some serious optimization, single-query searches are now up to 133x more slot-efficient—perfect for user-facing agentic searches.

Hybrid Search (Public Preview)

Semantic search is great, but sometimes you need exact keyword matches—like searching for a specific drug code “MK3475.” BigQuery’s new hybrid search combines vector search with lexical matching (BM25, Reciprocal Rank Fusion) to give you the best of both worlds. Just set mode => 'HYBRID' in your AI.SEARCH() call, and you’re golden.

To speed things up, you can extend your vector indexes to include keyword columns, making hybrid queries fly even at scale.

End-to-End Unstructured Data Analytics

These features are part of a bigger vision: making unstructured data a first-class citizen in BigQuery. The full lifecycle includes:

• Access: Query PDFs and documents directly in Cloud Storage using Object Tables (zero ETL).
• Process: Use AI.PARSE_DOCUMENT (coming soon), AI.GENERATE, and AI.CLASSIFY for chunking, entity extraction, and categorization.
• Ground: Build accurate context with Autonomous Embeddings and Hybrid Search.
• Relate: Map extracted entities into BigQuery Graph—no specialized graph DB needed.
• Activate: Chat with your data using Conversational Analytics agents and AI.AGG.

Ready to dive in? Check out the Document Analytics on BigQuery GitHub repo for a full clinical trials demo, or read the docs on Autonomous Embeddings and Hybrid Search.

Original announcement published on Google Cloud.