Similar retrieval scores, very different costs

Companies using AI quickly discover that a useful answer depends on having the right information. For a question about a customer project, that information might be spread across meeting notes, several tasks and a technical document. The system needs to find the relevant passages and understand their connections before it can produce a well-supported answer.

How it does this affects quality, response time and cost. When questions arrive frequently, the division of work between retrieval and the language model deserves particular attention.

Similar overall performance, substantial cost differences

Adnan El Assadi, Niklas Muennighoff and Jinhyuk Lee investigate this question in The Embedder’s Dilemma: LLMs Are Better, but at What Cost?. They compare ten LLMs with 26 embedding models across 37 tasks, including retrieval, classification and semantic similarity.

The leading models achieve similar aggregate scores: 77.6 for the best LLM and 77.2 for the best embedding model. Their costs differ substantially. At comparable quality, the authors report up to 1,431 times higher costs for an LLM: approximately $154 versus $0.11 per benchmark pass.

The reasoning budget also deserves attention. Reasoning tokens account for 28–81% of LLM inference costs in the study. Reducing that budget preserves or improves retrieval quality for most of the models tested.

Strengths vary by task: LLMs lead on reasoning-intensive retrieval, while embedding models lead on classification. An aggregate score alone is therefore insufficient for choosing a model. Similarly, the largest cost difference reflects particular benchmark comparisons; it is not a general savings estimate for an enterprise application.

Which work should a language model do?

Embedding models turn content into numerical representations that make it possible to find related information efficiently. This provides a useful foundation for selecting relevant passages from a large collection of documents.

A language model can then bring the evidence together, identify contradictions and decide what information is still missing. For difficult questions, reasoning can also help during retrieval, for example by interpreting an ambiguous request or determining the next search step.

For companies, this creates a concrete design challenge: each step needs an approach suited to its task. What matters is the total effort the system requires to reach a useful answer.

Why we prepare context upfront

At Traize, our approach is to structure and connect company knowledge as information enters the system. A document gains context through its relationships with the projects, tasks or people it refers to. That context can be reused for subsequent questions.

This is our architectural response to the importance of efficient information processing. The study itself compares models and does not validate a particular Traize architecture. It reinforces our focus on identifying which work specialised methods can handle and where additional reasoning adds value.

Preprocessing also has a cost. It is particularly useful when prepared context serves multiple requests. As information changes, the affected connections need updating, and retrieval must respect current access permissions. We therefore consider preparation, maintenance and queries together.

An example from everyday work

“Why is the customer rollout delayed?”

Relevant evidence could include a postponed milestone, a pending approval and a decision from the latest project meeting. When this information is already connected to the project, the agent has a concrete starting point for its investigation.

It can examine the evidence, explain the connection and follow up where something remains unclear. The value for the team is an answer with traceable sources and a clear explanation, reached as efficiently as possible.

These are the outcomes we want to measure our systems against: relevant retrieval, supported answers and a considered use of computing resources. In production, answer quality, speed and cost belong in the same evaluation.