AlloyDB AI: Solving PostgreSQL Full-Text Search for CJK

If you’ve ever tried to build a full-text search for Chinese, Japanese, or Korean content in PostgreSQL, you know the pain: standard tokenizers choke on continuous text, treating entire sentences as one giant lexeme. AlloyDB AI just fixed that by bringing Gemini’s multilingual smarts directly into your database—no ETL pipelines, no external microservices, just SQL.

The Problem: Logographical Languages and PostgreSQL

PostgreSQL’s to_tsvector splits on whitespace. For languages like Chinese, where words run together, that means a sentence like “你们研究所有十个图书馆” becomes a single, unsearchable blob. Searching for “研究所” (research institute) returns nothing—it’s trapped inside the larger string.

Traditional workarounds—third-party extensions like zhparser, external preprocessing with Python, or rule-based tokenizers—all come with baggage: unsupported in managed DBs, static dictionaries that miss modern jargon, or complex ETL that moves data out of the database.

AlloyDB AI: In-Database Intelligence with Gemini

AlloyDB AI introduces native AI Functions like ai.generate(), letting you call Gemini directly from SQL. No data movement, no extra services. You keep your data and intelligence in one place.

Here’s the gist: you create a table with a generated tsvector column that automatically updates when you populate a content_segmented field. Then you run a stored procedure that batches rows, calls Gemini to insert spaces between words, and commits each batch—avoiding row locks and rollback nightmares.

CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    original_content TEXT NOT NULL,
    content_segmented TEXT,
    search_vector tsvector GENERATED ALWAYS AS (to_tsvector('english', content_segmented)) STORED,
    embedding vector(3072) GENERATED ALWAYS AS (embedding('gemini-embedding-001', content_segmented)) STORED
);

The stored procedure uses array aggregation and GENERATE_SERIES to safely unpack results, processing rows in parallel. After segmentation, your sentence becomes “你们 研究所 有 十个 图书馆”—now searchable.

Hybrid Search: Text + Vectors with RRF

AlloyDB also supports hybrid search, combining RUM-indexed full-text search with ScaNN vector search. A single SQL query with Reciprocal Rank Fusion (RRF) merges results, giving you both exact keyword matches and semantic relevance.

WITH vector_search AS (
    SELECT id, RANK() OVER (ORDER BY embedding <=> ai.embedding('gemini-embedding-001', '研究所 图书馆')::vector) AS rank
    FROM documents
    ORDER BY embedding <=> ai.embedding('gemini-embedding-001', '研究所 图书馆')::vector
    LIMIT 10
),
text_search AS (
    SELECT id, RANK() OVER (ORDER BY search_vector <=> plainto_tsquery('english', '研究所 & 图书馆')) AS rank
    FROM documents
    WHERE search_vector @@ plainto_tsquery('english', '研究所 & 图书馆')
    ORDER BY search_vector <=> plainto_tsquery('english', '研究所 & 图书馆')
    LIMIT 10
)
SELECT COALESCE(v.id, t.id) AS id,
       COALESCE(1.0 / (60 + v.rank), 0.0) + COALESCE(1.0 / (60 + t.rank), 0.0) AS rrf_score
FROM vector_search v
FULL OUTER JOIN text_search t ON v.id = t.id
ORDER BY rrf_score DESC
LIMIT 5;

This approach is a game-changer for developers building multilingual apps. No more wrestling with external search engines or brittle tokenizers. AlloyDB AI keeps it simple, fast, and accurate—right where your data lives.

For more on managed database options, check out our Neon Database Review or Supabase Review. If you’re comparing cloud providers, our Google Cloud Review has you covered.

Original announcement published on Google Cloud.