Evaluating Agents on Requirements Discovery
Intro
Our mission at Datafruit is to put the most capable AI in the hands of implementation teams. Delivering on this means we have to know, rigorously and continually, how well agents perform on real implementation work.
Software implementation projects can be broken into two kinds of work. First interpretation: where the consultant listens to how the customer describes their business and translates that to requirements, and secondly judgement: where they apply years of platform expertise to turn those requirements into a working system. We’ve written previously about why we think that rushed interpretation is the secret killer for many consulting projects but to recap, interpretation is the foundation that everything downstream rests on.
We are building long horizon agents for complex implementation work, and it's crucial that that work starts from reliable interpretation. Before improving anything downstream we wanted to ask: how good are models at these tasks today?
We built a benchmark to find out. Each task gives an agent a set of discovery meeting transcripts and asks for the complete set of business requirements, which is then graded against a golden set per task. The best model recovers 73.1% of them. Every model misses more than a quarter. This post covers how we built it, how we score it, and what the results tell us.
The Data
Discovery meetings resist clean extraction. Requirements are decided across many meetings, and often times stated ambiguously rather than directly. Here’s how we assembled our data to account for these real world difficulties.
Every task in the eval starts from a set of “golden” requirements that we derive from an internal dataset of anonymized implementation work from real projects. We then generate discovery transcripts around these gold requirements through a 7-stage synthetic data generation pipeline across real scenarios ranging from enterprise software to media publishing and to healthcare services.
Each task has three elements:
- Transcripts: a set of discovery meetings for one engagement, generated to express the “golden” requirements and difficulties around them
- Golden Requirements: the complete set of business requirements for an engagement
- Rubric: per-task grading criteria derived from the requirements and their parts.
An individual requirement describes workflows that can be decomposed into these parts:
- Primary: the core capability or claim
- Secondary: nuance and supporting reasoning
- Software: a class of tool ie. a CPQ
- Name: a branded product ie. Salesforce CPQ
What makes extraction hard?
There are three properties that make real discovery much harder than it looks. We built each of these into our generation pipeline, correlated to specific meetings so we can trace where they occurred.
Traps: plausible ideas that are discussed, considered, and decided against. This could be a client requesting an auto invoicing feature, only for it to be rolled back in another meeting.
Ambiguity: requirements must be inferred rather than explicitly stated, or discussed over a number of meetings to clarify details, or are just open ended. This is the broadest trap; it could be anything from clients not knowing what they want, to clients who complain about a tool they're about to replace.
DWIM: Do what I mean, not what I say. In these scenarios the client asks for a specific build but platform experience or auxiliary context says the right choice is different. Each DWIM case is drawn from documented enterprise implementation failure modes.
Findings
Opus 5 takes the lead with a mean recall of 73.1%, followed by Fable 5 at 65.2%, and Kimi K3 at 59.1%. Even the best frontier models consistently miss over 1/4 of requirements, with every model besides Opus 5 missing at least 1/3 on average. We take a deeper dive into the extractions below.
Precision
Where recall asks what the agent missed, precision asks what it added. De-duplicated precision then discounts repeated claims that do not add new coverage; it measures the percentage of the agents’ claims that are both unique and matched by the gold. De-duplicated precision is our primary precision metric.
Terra leads at 36.7%, followed by Sol at 32.7%, Sonnet 5 at 32.5%, and Luna at 32.4%. Opus 5, which won recall outright, finishes last at 22.4%. Every model lands between 22.4-36.7%: even at best, barely over a third of the extracted claims are both unique and matched by the gold.
Recall vs Precision
Here we compare recall directly against precision for each model:
Opus is the clear winner in recall and the clear loser in precision. Terra leads precision but recovers just 37.3% of the gold. GLM 5.2, Sonnet 5, DeepSeek V4 Pro, and Luna are Pareto dominated: another model beats each one on both measures. Recall spans 33.8-73.1%, while precision spans 22.4-36.7%.
Eval Harness
Our eval harness is a coding-agent style harness given a bash environment. Every datapoint is stored as a directory: transcripts in input/, and the answer key in evaluation/. Every trial gets a fresh directory containing only copies of the transcripts from input/ and INSTRUCTIONS.md, a per datapoint instruction set generated from one template for more specified instructions. Every model is given the same harness so that the only difference between trials is the model. Each agent outputs one requirements.json to be evaluated.
Verifier
Every agent output is scored against a set of gold standard requirements belonging to the same group of meetings. We treat the gold requirements as the set of truth and compare via a number of criteria. The grader itself is split into 5 parts, each with a specific purpose:
J0: Normalization
This is a completely deterministic pass that checks that the agent’s extraction surfaced in the required format. A failure to output a requirements.json results in a score of 0 on any particular extraction. Note that in the final experiments this never occurred. An output containing more than 60 requirements raises a flag that is naturally penalized by precision. J0 finishes by enforcing hygiene such as removing empty statements and fixing duplicate requirements IDs.
J1: Grounding
Every prediction the agent makes records evidence: a list of source-and-quote pairs pointing at the transcript the claim came from. J1 checks each one against the transcript it names, first as a whitespace- and case-normalized substring, then as a fuzzy window match at a 0.75 similarity threshold. The whole pass is deterministic string matching with no LLM calls. J1 doesn't directly contribute to any of the scores. It produces a grounding rate, the share of cited quotes that were found, and it attaches a verified flag to every individual quote.
J2: Recall
Each gold requirement is composed of parts, here J2 checks each gold part against the entire agent extraction as a binary check, answering met or not met. Essentially, the grader does not check the extraction part by part, instead it answers if the output as a whole establishes each gold part. There is deliberately no bipartite matching: one prediction may satisfy several parts, and several predictions may jointly satisfy one. If the judge cannot decide if a part is met or not, it is deemed not met. This same judge conservatism applies to J3 and J4 as well.
J3: Traps
Similar to J2, J3 runs one binary check per trap. However it asks a different question: is the trap present, and is it stated as live requirement? This means that it is okay for the trap to be mentioned in some requirement, as long as it is not treated as a requirement itself. Any output that correctly mentions the trap as rejected, superseded, or undecided may go unpunished.
J4: Precision
J4 has one main job: figure out what the extractor asserted that the answer key didn’t ask for. The first step is decomposing each prediction into parts; this is done by one LLM call and its effectiveness has been human verified.
The second step is to classify each part as either matching a gold part or not. This is essentially the same as how J2 works; for each gold part, compare it to each predicted claim and cite the ones that match.
Verifying the Verifier
How do we know if our verifier works? We have a few methods to both check that our verifier works correctly and to quantify error and uncertainty.
The first and most basic, but also most important, is hand verification. We went through a panel of frontier model judges, checking for same family bias, variance across the same extraction, and directly auditing the judge results, before settling on using GPT 5.6 Sol as our judging model. Another note is that using atomic parts for each requirement significantly reduced same family bias, with the largest decrease in Anthropic models as they had the most same family bias to begin with.
The next check is gold replay: does the answer key score 100% when used as the input? And the results speak for themselves, on every datapoint the gold scores 100% on recall, precision, and trap violations.
We ran a fixed extraction variance test: on 5 datapoints we fixed one extraction and judged it 3 times each. We achieved an average standard deviation of ~0.03 for both recall and precision within datapoints.
More Results and Experiments
Part recall is higher than primary recall, with improvements of 9.3% for Opus, 15.3% for Fable, and 21.4% for Kimi, but over 1/5 of requirement parts are still missing from Opus, and close to or over 1/4 for every other model.
Predictably, explicit requirements are across the board easier to extract, with Opus taking the lead in 91.1% explicit recall. Implicit requirements are much more difficult to track, as we see 24.4-44.6% drops across every model.
Many of the models predict a lot more requirements than the gold, with Opus predicting 1.88x the number of requirements as gold on average, and every non-OpenAI model predicting at least 1.5x as much.
Claims per gold part measures the total number of “parts” of predicted requirements compared to the total number of parts within the gold. Anthropic has a lot of cross model variance under claims per part, ranging from 1.17 to 2.26 versus 1.63 to 1.88 predictions per requirement. OpenAI models also range from 0.89 to 1.21, much more than their 0.90 to 1.06 predictions per requirement. The number of predicted parts is not a reliable predictor of predicted requirements.
We tested different prompts and reasoning levels for Opus. Here are the results:
The strict and lenient harness variants guide extraction volume, with medium being given no specific directions as to how many extractions it should make. The results in terms of recall are very close, whereas there are some drop offs in precision. The strict outcome is much more human readable, making less than 2/3 of the predictions as medium and almost 1/3 of lenient and a much higher de-duplicated precision. It is worth noting that all other trials are done on the strict prompting.
There is a noticeable decline in recall with low reasoning, but high and max are almost indistinguishable. The same follows for predictions per gold requirement. However, high reasoning has a 2.4% lead in deduplicated precision. For almost every purpose, high reasoning seems to be the best.
The approach of directly prompting a model with each meeting transcript and instruction works surprisingly well. The coding agent harness raises Opus recall from 67.9% to 73.1%, while de-duplicated precision is essentially unchanged: 22.6% for the one-shot call versus 22.4% for the coding agent harness. This suggests that simply placing an agent in a coding sandbox may not be the right approach to improving requirements extraction.
We have omitted a graph of the trap scores, as all of the models perform incredibly well. No model scored below 98% in correctly not mis-claiming traps as requirements. Frontier models seem to be good at classifying and excluding rejected ideas.
What’s Next?
The first version of the benchmark tests just requirements extraction. In the next iterations, we want to expand to a dynamic environment containing the whole pre-sales workflow, including more discovery features and contract drafting capabilities. In the future we will show how our internal harness performs on this benchmark and the steps we used to make our internal agent better.