.png)
Engineering AI for Production
Author: Konstantinos Perifanos, Hybrid Mind Consulting
Getting an AI system to work in a demo is one thing. Building one that can be measured, changed and operated reliably in production is another.
In this five-part engineering series, we look beneath the final model output at some of the layers that determine how production AI systems actually behave.
We start with embeddings and retrieval, before moving through agent evaluation, observability and deployment, long-horizon agents and the runtime engineering needed to keep them working.
The common theme is simple: Don't just look at the final answer. Measure what is happening underneath it.
Part 1 starts at the retrieval layer.
Two models can encode the same data very differently
Two embedding models can process exactly the same documents, produce vectors with exactly the same number of dimensions and still organise that information very differently.
That matters because vector search depends on the geometry underneath it.
Change the embedding model, preprocessing pipeline or underlying data and the application may still appear healthy while search behaviour has shifted underneath it.
So how do we detect those changes?
We compared two compact embedding models:
MiniLM (sentence-transformers/all-MiniLM-L6-v2)
BGE-small (BAAI/bge-small-en-v1.5)
Both represent text using 384-dimensional vectors.
Rather than asking which model was simply "better", we measured four properties of the resulting embedding spaces:
- pairwise cosine similarity;
- near-duplicate rates;
- effective rank; and
- hubness.
The aim was to establish a measurable baseline for how each model organises the same collection of text.
That baseline can later help answer a more useful production question:
Has something changed in our retrieval layer, and where should we investigate?
The comparison setup
We used a corpus of 1,000 artificial intelligence and machine-learning papers, split into 93,079 passages.
Both embedding stores contain exactly the same passages with the same boundaries.
The vectors were explicitly normalised to unit length.
For the evaluation, we selected 5,000 passages in a deterministic order based on a hash of paper ID and passage ordinal. This allowed us to compare corresponding passages across both models.
Effective rank uses all 5,000 vectors.
Pairwise cosine statistics, near-duplicate rates and hubness use the first 1,500 vectors from that sample, excluding self-comparisons.
The measurements therefore describe the combination of model and corpus. A different dataset or sampling strategy could produce different values.
At a high level, BGE-small shows:
- much higher average cosine similarity;
- a narrower similarity distribution;
- slightly higher effective rank; and
- lower hubness.
The interesting part is what those measurements mean when read together.
Pairwise cosine: how aligned are the vectors?
Cosine similarity measures how closely two vectors point in the same direction.
cosine(x, y) = dot(x, y) / (norm(x) × norm(y))
Because our vectors are normalised, this simplifies to the dot product.
A cosine similarity of:
- 1 means the vectors point in the same direction;
- 0 means they are perpendicular;
- -1 means they point in opposite directions.
MiniLM's mean pairwise cosine similarity is 0.1807.
BGE-small's is 0.6515.
That is the largest numerical difference in the comparison.
BGE's vectors are therefore much more aligned with one another on average.
The distribution is also tighter.
BGE's cosine standard deviation is 0.0852, around 53% lower than MiniLM's 0.1821.
In practical terms, BGE's passage similarities are clustered more closely around a higher baseline.
That matters because an absolute cosine score does not mean the same thing in every embedding space.
A value such as 0.65 is not "65% semantic agreement", and a threshold that works for one model should not automatically be carried over to another.
Near-duplicates: are highly similar passages taking over?
Next, we looked at the extreme upper end of the similarity distribution.
The near-duplicate rate measures the percentage of passage pairs with cosine similarity greater than 0.999.
Both models produced approximately the same result:
0.00205%.
Among the 1,124,250 unique passage pairs in the 1,500-passage sample, that corresponds to 23 pairs per model.
This is useful when compared with the previous result.
BGE has a much higher average similarity, but it does not produce more pairs crossing this near-identical threshold.
So:
higher average similarity does not automatically mean more near-duplicate vectors.
The metric does not tell us whether the same 23 pairs are involved in both models, nor whether several pairs belong to the same small cluster.
But it gives us a simple way to monitor whether near-identical content is becoming more common over time.
Effective rank: how evenly is information spread?
Each embedding contains 384 dimensions.
That does not mean every direction contributes equally to the structure of the space.
Effective rank gives us a way to describe how evenly the observed variation is distributed.
For this evaluation, we first subtract the mean vector from the 5,000 embeddings. We then calculate the singular values of the centred matrix and convert them into normalised variance shares.
p_i = singular_value_i² / sum(singular_value_j²)
effective_rank = exp(-sum(p_i × log(p_i)))
If variation were spread evenly across 100 directions and zero elsewhere, the effective rank would be 100.
More concentrated variation produces a lower value.
More evenly distributed variation produces a higher value.
MiniLM's effective rank is 108.37.
BGE-small's is 111.69.
As a proportion of the available 384 dimensions, that gives:
- MiniLM: 28.22%
- BGE-small: 29.09%
BGE therefore has a modestly more even distribution of centred variance in this sample.
This should be treated as a comparative signal, not a compression target.
An effective rank of 108 does not mean the remaining dimensions are useless, nor does it prove that one model captures more useful semantic information than another.
It tells us how concentrated the observed variation is.
Hubness: which passages keep becoming neighbours?
Another property worth monitoring is hubness.
For each of the 1,500 sampled vectors, we selected its single closest neighbour and counted how frequently each passage was chosen.
A passage that repeatedly becomes the nearest neighbour of many other passages becomes a hub.
MiniLM's largest hub was selected 21 times, representing 1.40% of the sample.
BGE-small's largest hub was selected 12 times, or 0.80%.
BGE's maximum hub share was therefore around 43% lower.
We also measured hubness skewness, which describes the shape of the overall neighbour-count distribution.
MiniLM recorded 4.04.
BGE-small recorded 2.51.
Together, those measurements show a less extreme upper tail of neighbour counts for BGE in this sample.
Why does that matter?
In a retrieval system, excessive hubness can become a practical problem if a small number of generic passages repeatedly appear across otherwise different searches.
However, the metric itself cannot tell us why a passage has become a hub.
It may be caused by:
- common subject matter;
- repeated wording;
- characteristics of the embedding model; or
- the structure of the corpus.
That requires inspecting the passages themselves.
Reading the metrics together
No single metric tells the whole story.
BGE's passage vectors have:
- much higher average cosine similarity;
- a narrower cosine distribution;
- slightly higher centred effective rank; and
- lower hubness.
Yet both models have the same near-duplicate rate at the threshold we measured.
So in this dataset, higher global similarity coexists with less extreme local neighbour concentration.
MiniLM's profile is broadly:
lower similarity baseline, wider similarity distribution, slightly more concentrated centred variance and a more pronounced hub tail.
BGE-small's is:
higher shared alignment, narrower similarity distribution, slightly more evenly distributed centred variance and a less pronounced hub tail.
Neither profile tells us which model will produce the best answers.
It tells us that their embedding spaces behave differently.
That distinction is important.
What could these patterns mean for search?
Geometric measurements are useful because they can point us towards retrieval problems worth investigating.
There is no universally "good" value for cosine similarity, effective rank or hubness.
The question is whether those measurements change in ways that accompany a loss of useful search behaviour.
These are signals, not conclusions.
The metrics in this experiment compare passages with passages. They do not directly measure whether a user query retrieves the most relevant content.
That requires query-level evaluation.
Takeaway: an embedding-space health check
For a production RAG system, we would not use these measurements as a one-off model comparison.
We would use them as baselines.
A simple health-check process could look like this:
- Maintain a fixed reference dataset.
Use the same sample to detect changes introduced by model or preprocessing updates. - Sample fresh production content.
Track whether changes in incoming data are altering the shape of the space. - Record model and pipeline versions.
Every measurement should be traceable to a deployment or data change. - Keep the evaluation configuration fixed.
Sample size, centring method, duplicate threshold and neighbour count should remain comparable over time. - Connect geometric alerts to search tests.
A change in geometry should trigger retrieval evaluation, not an automatic conclusion that quality has improved or degraded.
For example:
- a shift in cosine distributions could trigger analysis of relevant and irrelevant query scores;
- increasing hubness could trigger inspection of frequently retrieved passages;
- a change in effective rank could be compared against recall and ranking quality across different topics;
- a rise in near-duplicates could be checked against how many retrieval slots are occupied by repeated content.
The key distinction is:
Geometry tells us where something may have changed. Retrieval evaluation tells us whether that change actually matters.
Conclusion
The same text and the same vector dimensions do not produce the same embedding space.
In this comparison, BGE-small showed higher average cosine similarity and a narrower similarity distribution, while also producing slightly higher centred effective rank and lower hubness.
MiniLM produced a lower similarity baseline, a wider spread and a more pronounced hub tail.
Both produced the same near-duplicate pair rate at the threshold we measured.
None of that makes one model universally better.
What it shows is that embedding spaces have measurable characteristics, and those characteristics can change when the model, data or preprocessing pipeline changes.
For production systems, that gives us another layer of observability.
Rather than waiting until an AI answer starts looking wrong, we can monitor the systems underneath it and identify where to investigate.
That is the principle we will carry through the rest of this series.
Measure the layer underneath, not just the output above it.
Next in the series
In Part 2 of Engineering AI for Production, we move one layer up the stack.
Once an agent works, how do you safely change its prompts, models, tools and behaviour without quietly degrading its quality?
In Agents That Survive Their Second Month, Gregorio Ferreira looks at the engineering loop behind maintainable agents, including prompt versioning, evaluation baselines, production feedback and treating agent changes with the same discipline as changes to any other production service.
Series: Engineering AI for Production
Part 1: What Is Your Embedding Space Actually Doing?
Part 2: Agents That Survive Their Second Month
Part 3: The Platform Under the Agents
Part 4: Beyond the Chatbot
Part 5: Durable by Design
