Bilal Tahseen
All posts

EmbeddingGemma 2: One Embedding Space for Text, Code, Images, Audio & Video

· Bilal Tahseen

Most retrieval stacks I inherit still treat modalities as separate products. Text lives in one embedder. Code in another. Screenshots get shoved through a vision model that does not share a space with the docs. Audio is a speech-to-text afterthought. Then someone asks "can we search the product video with the same query as the FAQ?" and the honest answer is no, not without a pile of adapters.

EmbeddingGemma 2 is the first open model I would actually reach for to collapse that mess. Google DeepMind released it on 6 October 2026. It maps text (including code), images, video, audio, and interleaved combinations into one 768-dimensional space. Apache 2.0. 740M parameters if you load everything, 270M if you only need text. I am writing this from the Hugging Face model card and the Google AI docs — the snippets below are those APIs, not a wrapper I invented.

This is not a "throw it in production tonight" post. I have not GPU-tested the model on this box. What I have done is read the card, the model card 2 docs, and the Sentence Transformers guide, and composed a retriever from those exact calls. If you skip the prefixes, the dtype, or the re-normalize-after-truncate step, you will get plausible scores and a worse index. That is the kind of silent failure I keep seeing in RAG systems.

What it is

EmbeddingGemma 2 is a multimodal embedding model built on the Gemma 4 architecture. One forward pass, one vector. The card's breakdown:

  • 740M total = 270M text model (130M backbone + 140M embedder) + 170M vision + 300M audio encoders
  • Native output: 768-d, with Matryoshka truncation to 512 / 256 / 128 (up to 6x less vector storage)
  • Context: 8,192 tokens shared across modalities
  • Languages: 100+ (pretraining web data in 140+ languages). Data cutoff January 2025
  • Precision: bfloat16 or float32. Never float16 — the card is explicit that float16 produces NaNs or silent degradation

Token budget, also from the card, because this is how you blow the 8k window without noticing:

  • Text: 1 token per subword
  • Image: 280 tokens default (~29 images)
  • Video: 140 tokens per frame (~58 frames at the default 1 fps)
  • Audio: 25 tokens per second (~327 s, mono 16 kHz)

Vision is configurable from 70 to 1,120 soft tokens, which the card says gets you up to ~114 images or frames. Interleaved inputs share the same 8,192-token pool, so mixing modalities reduces how much of each fits.

You do not have to load 740M. The card's config_kwargs menu:

  • Text only: 270M — {"vision_config": None, "audio_config": None}
  • Text + image: 440M — {"audio_config": None}
  • Text + audio: 570M — {"vision_config": None}
  • Full multimodal: 740M — {}

That is the first production decision. Most of my RAG work is still text. Loading vision and audio encoders for a PDF FAQ is wasted RAM.

What changed from v1

EmbeddingGemma 1 was a 300M text embedder. I used it. It was good. It was not multimodal, and the code numbers were the weak spot.

Google's developer overview and the announcement spell out the jump. The numbers I will actually quote are the ones on the v2 card, 768-d, full-precision checkpoint:

  • MTEB multilingual v2: 61.36 (v1: 61.15). Text quality is essentially flat.
  • MTEB code v1: 78.68 (v1: 68.76). That is the ~14% code improvement the card calls out. If you were waiting on EmbeddingGemma for code search, this is the reason to move.
  • New multimodal scores (v1 has no comparable numbers): MIEB lite 64.64, MMEB v2 Image Hit@1 57.28, MMEB v2 VisDoc NDCG@5 67.84, MMEB v2 Video Hit@1 50.67, MSEB Retrieval MRR@10 69.54, MAEB 49.39.

The other changes I care about as someone who ships indexes:

  • Native multimodality in one space, including interleaved text + media
  • Smaller text-only footprint: 270M instead of v1's 300M
  • 8,192-token context
  • Selective encoder loading
  • Weights on Hugging Face and Kaggle, Apache 2.0. The blog notes v1 crossed 20M+ downloads and points at LiteRT Community builds on Hugging Face for on-device

I would not switch a healthy v1 text index "because it is new." I would switch for code retrieval, for a single multimodal index, or because I want the 270M text-only load and Matryoshka truncation in one card.

When I would use it

Use it when you want one embedding model you can run yourself, under Apache 2.0, for:

  • Semantic search / RAG over text, and you are willing to run a 270M local model instead of a hosted API
  • Code search from natural-language queries (prompt_name="CodeRetrieval")
  • Cross-modal retrieval: text query against screenshots, clips, or audio, without a separate CLIP + Whisper pipeline
  • Interleaved product listings — caption plus images plus a demo video as one vector

Do not use it as a magic replacement for structured lookup. If the user is asking for order #1842, that is still a query, not an embedding problem. I wrote about that failure mode in Stop Evaluating RAG With Vibes and When an AI Agent Is the Wrong Tool. An embedder this capable makes it easier to dump the wrong corpus into a vector DB. The model will not save you from that.

Also do not pick 128-d for a multimodal index because the diagram looks tidy. The card is blunt: quality is near-lossless down to 256-d; 128-d hurts multimodal substantially. I will come back to that with the truncation table.

How it sits in a RAG pipeline

How EmbeddingGemma 2 fits a RAG pipeline: text, code, images, audio, and video go into the 740M model, which emits a 768-d vector; Matryoshka truncation offers 512-d, 256-d, or 128-d; a vector DB stores the chosen vectors; retrieval feeds an LLM answer

Read the diagram as a menu, not as an instruction to store four copies of every vector.

The picture shows the database holding 512-d, 256-d, and 128-d at once. That is a useful way to explain Matryoshka: the leading dimensions are a valid shorter embedding. In production I pick one dimension per index. Queries and documents have to share a dim. A 768-d query against a 256-d corpus is not "close enough" — it is an invalid comparison. If you later decide 256-d is too aggressive, you re-embed into a new collection. You do not mix dims in the same HNSW index and hope cosine sorts it out.

The rest of the pipeline is the usual RAG shape I use with customer-support agents: embed at ingest, retrieve, then generate. The embedding model is not the generator. Pair it with whatever LLM you already trust. Record the embedding model id and the truncation dim on the retrieval span so you can replay a run later — that is the index_version I want in replayable traces.

Official sentence-transformers usage

Install what the card says:

pip install -U sentence-transformers transformers

The card's hello-world. This is the API. I am not wrapping it.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("google/embeddinggemma-2")

query = "What causes the northern lights?"
document = "The northern lights are caused by charged particles from the sun."

query_emb = model.encode(query, prompt_name="SearchQuery")
doc_emb = model.encode(document, prompt_name="Document")
print(model.similarity(query_emb, doc_emb))

Three details I would not skip, all from the same card.

1. Task prefixes are not optional if you care about ranking. EmbeddingGemma 2 is trained with short instruction prefixes on text only. Media gets no prefix. Asymmetric retrieval uses a query prefix and a document prefix. Symmetric tasks (classification, clustering, similarity) use the same prefix on both sides.

The prompt_name values I actually use, mapped to the prefixes on the card:

  • SearchQuery — task: search result | query: {q}
  • QuestionAnswering — question answering
  • FactChecking — fact checking
  • CodeRetrieval — task: code retrieval | query:
  • Classification, Clustering, SentenceSimilarity — symmetric
  • Document — applies title: none | text: {content}

If the document has a real title, do not use prompt_name="Document". Format it yourself: title: {title} | text: {content}. The card is explicit that Document forces title: none.

2. Truncate with the library, then re-normalize. Slicing a unit vector yourself leaves you with a vector that is no longer unit length. Cosine then lies. Pass truncate_dim and normalize_embeddings=True:

query_emb = model.encode(query, truncate_dim=128, normalize_embeddings=True)  # or 512, 256

Supported dims are 768, 512, 256, 128. Queries and docs in one index must share one of those.

3. Dtype is a correctness issue, not a perf knob.

import torch
dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32
model = SentenceTransformer("google/embeddinggemma-2", model_kwargs={"torch_dtype": dtype})

bfloat16 on GPUs that support it. float32 everywhere else, including most CPUs. I have seen teams set float16 because it is "what we use for inference." On this model that is how you get NaNs, or worse, numbers that look fine and rank wrong.

Text-only load, 270M:

# text-only, 270M
model = SentenceTransformer("google/embeddinggemma-2",
    config_kwargs={"vision_config": None, "audio_config": None})

A small semantic search / RAG retriever

This is not a framework. It is the official encode / similarity calls plus torch.topk so you can see ranking before you involve a vector DB. Same prefixes, same truncation, same dtype. Swap the in-memory corpus for pgvector or whatever you already run once you have proven the gold chunks come back — that is the eval I described in Stop Evaluating RAG With Vibes.

import torch
from sentence_transformers import SentenceTransformer

dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32

model = SentenceTransformer(
    "google/embeddinggemma-2",
    model_kwargs={"torch_dtype": dtype},
    config_kwargs={"vision_config": None, "audio_config": None},
)

# One dimension per index. 256-d is the card's "near-lossless" floor.
DIM = 256

corpus = [
    {
        "id": "aurora",
        "title": "Aurora physics",
        "text": "The northern lights are caused by charged particles from the sun.",
    },
    {
        "id": "mrl",
        "title": "Matryoshka embeddings",
        "text": "EmbeddingGemma 2 natively outputs 768-d and truncates to 512, 256, or 128.",
    },
    {
        "id": "audio",
        "title": "Audio sampling",
        "text": "Supply audio as mono 16 kHz. The model spends 25 tokens per second.",
    },
]

# Titled docs: format manually. prompt_name="Document" would force title: none.
doc_inputs = [f"title: {item['title']} | text: {item['text']}" for item in corpus]
doc_embs = model.encode(
    doc_inputs,
    truncate_dim=DIM,
    normalize_embeddings=True,
)

query = "What causes the northern lights?"
query_emb = model.encode(
    query,
    prompt_name="SearchQuery",
    truncate_dim=DIM,
    normalize_embeddings=True,
)

scores = model.similarity(query_emb, doc_embs)
top = torch.topk(scores, k=min(3, scores.shape[-1]))

for rank, (score, idx) in enumerate(zip(top.values[0], top.indices[0]), start=1):
    hit = corpus[int(idx)]
    print(f"{rank}. {hit['id']}\tscore={float(score):.4f}\t{hit['title']}")

# Hand the top texts to your generator. Retrieval is done; generation is a different model.
retrieved = [corpus[int(i)]["text"] for i in top.indices[0]]

For a FAQ with no titles, prompt_name="Document" is correct. For code search, switch the query to prompt_name="CodeRetrieval" and format the file as title: {filename} | text: {code}.

What I would not do: embed the query with no prefix, embed docs with no prefix, and call it a baseline. The card says omitting the prefix still works and reduces precision. In retrieval, "still works" is how you ship a demo and fail the golden set.

Wire this into an agent the same way I do any retriever: behind a tool contract, with a golden-set gate before it faces users. The embedder does not change that. See Tool Contracts Beat Prompt Rewrites and I Won't Ship an AI Agent Without a Golden-Set Eval Gate.

Multimodal: the card's interleaved example

One vector for mixed media. Placeholders from the model's vocabulary mark where each item sits in the 8,192-token sequence: <|image|>, <|video|>, <|audio|>. Prefixes do not apply to media.

This is the card's example, verbatim:

emb_interleaved = model.encode({
    "text": "Waterproof running shoes. <|image|> Featuring a breathable mesh upper. <|image|> Grip test on wet rock: <|video|>",
    "image": ["shoe.jpg", "mesh.jpg"],
    "video": "demo.mp4",
})

Each placeholder is filled from the corresponding key, in order. The result is one embedding. You can compare it to a text-only query such as "waterproof shoes for trail running" encoded with prompt_name="SearchQuery". Same 768-d space (or the same truncated dim, re-normalized).

That is the product-catalog case I actually want this model for. Previously I would embed the caption with a text model and the images with a vision model, then invent a fusion score. Now the listing is one document.

Defaults worth pinning in your ingest code, again from the card: video at 1 fps through the vision encoder, audio as mono 16 kHz. If you feed 48 kHz stereo and wonder why ranking is weird, start there.

For this path you need the encoders. Drop the text-only config_kwargs. Full 740M, or at least keep vision if you have no audio.

Choosing a dimension

Matryoshka means the first k dimensions are a trained shorter embedding. You are not PCA-ing at rest. You are slicing and re-normalizing.

Official truncation scores from the card (MTEB multilingual / English / code, then MIEB, MMEB overall, MSEB, MAEB):

  • 768-d (1:1): 61.36 / 68.46 / 78.68 / 64.64 / 59.01 / 69.54 / 49.39
  • 512-d (1:1.5): 61.17 / 68.41 / 77.24 / 64.32 / 58.38 / 69.18 / 49.21
  • 256-d (1:3): 60.41 / 67.78 / 76.18 / 63.13 / 56.24 / 66.76 / 48.91
  • 128-d (1:6): 57.89 / 65.68 / 71.41 / 59.06 / 45.65 / 56.71 / 46.92

How I read that as someone paying for vector storage:

  • 768-d if the index is small or ranking is the product. Default until I measure otherwise.
  • 512-d as the first cut. Text barely moves. Code drops 78.68 → 77.24. Fine for most corpora.
  • 256-d when storage or HNSW RAM is the constraint. Card calls this near-lossless. MMEB overall 59.01 → 56.24 — I would A/B this on a multimodal golden set, not assume.
  • 128-d for text-only, and only after I run my own queries. MMEB overall falls to 45.65. That is not a rounding error. The card says 128-d hurts multimodal substantially. I believe it.

Pick the dim before you build the index. Write it into the collection name (kb_eg2_256). Queries encode with the same truncate_dim. If you need two dims, you have two indexes, not one index with a hope.

Running it locally

This model is meant to run on a machine you own. That is the point of a 270M–740M Apache 2.0 checkpoint.

What I set on a fresh box:

  1. pip install -U sentence-transformers transformers
  2. Load google/embeddinggemma-2 with torch_dtype set to bfloat16 or float32 as above. Never float16.
  3. If the workload is text or code only, pass the text-only config_kwargs and save yourself the vision and audio encoders.
  4. Encode docs offline. Encode queries online with the matching prompt_name and truncate_dim.
  5. Put the vectors in the DB you already have. The model does not care whether that is pgvector, Qdrant, or a numpy array in a test.

I would not serve this from a laptop CPU for a user-facing p95 if I can avoid it. I would happily run the 270M text-only load on a single GPU next to the API, or batch-embed a corpus on a workstation. On-device is in Google's framing (LiteRT Community builds on Hugging Face); I have not profiled those here, so I will not quote RAM numbers I did not measure.

One operational rule I already use for any embedder swap: the index version is (model_id, dim, prompt scheme). Changing any of the three means rebuild. Mixing EmbeddingGemma 1 vectors with EmbeddingGemma 2 vectors in one collection is how you get a retrieval outage that looks like "the LLM got worse."

Close

EmbeddingGemma 2 is a 740M open multimodal embedder that puts text, code, images, audio, and video in one 768-d space, with a 270M text-only load and Matryoshka dims down to 128. The code jump versus v1 is real. The multimodal scores exist because v1 had nothing to compare. The failure modes I care about are the unglamorous ones: wrong prompt_name, float16, forgetting to re-normalize after truncate, and stuffing four dims into one index because a diagram showed all four.

Use the official card. Use SearchQuery / Document (or a manual title prefix). Pick one dim. Measure retrieval on a golden set before you argue about the generator.

I help teams stand up retrieval stacks that survive contact with real queries, including multimodal ones. If you want that built, hire me.

Building something with AI?

I help teams ship production AI agents, retrieval systems, and document intelligence. Let's talk about yours.

Book a call