Searching requirements without sending them anywhere
Thousands of requirements and test cases, and a hard rule that none of it leaves the building. The self-hosted retrieval design I am testing, and the two choices that matter most.
Keyword search fails engineers in a specific way. Someone searches for “brake hold on slope” and misses the requirement that says “maintain clamping force on an incline”. Same meaning, no shared words. Across thousands of requirements and test cases in an ALM tool like Siemens Polarion, that gap costs real time, and it hides duplicates and conflicts.
Semantic search fixes this. The catch is that the usual way to do it sends every requirement to an external embedding API. For engineering data that is a non-starter. So the rule for this design is simple: nothing leaves the trust boundary. No external embedding calls, no external LLM calls.
Figure 1. Everything inside the dashed line runs on infrastructure we control. The search API returns work-item IDs and links, so people always land on the real record in the ALM tool.
Choice one: how to embed
| EmbeddingGemma | LFM2.5-ColBERT | |
|---|---|---|
| Approach | One vector per chunk | One vector per token, matched late (“late interaction”) |
| Model size | 308M parameters | Small, from the same family as my EARS helper |
| Strength | Simple, small index, fast | Better on exact technical terms, part numbers, signal names |
| Cost | Lowest storage | Many vectors per chunk, so a much larger index |
Engineering text is full of identifiers that matter exactly: EPB_SW_REQ_104, a CAN signal, a part number. Late interaction tends to handle those better because it compares token by token. Whether that is worth a much bigger index is what I am measuring.
Choice two: where to store it
| pgvector | Qdrant | |
|---|---|---|
| What it is | An extension to PostgreSQL | A dedicated vector database |
| Operations | One database you probably already run | One more service to run and patch |
| Filtering | Plain SQL joins and WHERE clauses | Payload filters on each point |
| Multi-vector (ColBERT) | Possible, but you build it | Supported natively |
The two choices are linked. If ColBERT wins on quality, Qdrant’s native multi-vector support makes it the easier store. If a single-vector model is good enough, pgvector keeps the system to one database, which matters for something that has to live inside a governed environment for years.
How I am deciding
Not on vibes. Three numbers, on a labelled set of real queries:
| Metric | What it tells me |
|---|---|
| Recall at 10 | Does the right requirement appear in the first ten results? |
| Index size per 1,000 items | What the storage bill and backup window look like at scale |
| Query latency, 95th percentile | Whether engineers will actually use it |
The agent side
Search is half of it. The other half is letting an agent work with the ALM tool once it has found something. For that I wrote a skill package for my agent setup with a Python client for the Polarion REST API: work items, test cases and runs, LiveDocs and workflows, with automatic pagination and JSON:API handling. Every destructive operation needs an explicit confirmation step. An agent that can find a requirement should not be able to delete it on a misunderstanding.
Retrieval and correction are separate jobs. This system finds requirements. The EARS helper fixes how they are written. Together they cover most of what engineers ask for, and none of it needs to leave the building.