The cost of being helpful

In 1958, the legal philosopher H.L.A. Hart devised an intentionally ambiguous rule to make a point: 'no vehicle may be taken into the park'. Obviously, this includes cars. But what about a child's bicycle or a pair of roller skates? What about an ambulance answering a 999 call or a tank on a plinth as a war memorial?

The words we use ordinarily have a well-understood core and then a fuzzy border, where interpretation becomes much more subjective. Hart called this the open texture of language.

This isn’t unique to legal interpretation. Ask a programmer to ‘remove duplicates from a list’ and the result might be a list in which every unique item appears just once, or one from which every repeated item has been removed.

Nor is it limited to technical disciplines like law and software engineering: tell a colleague that ‘next Wednesday’s meeting has been moved forward two days’ and, as the cognitive scientist Lera Boroditsky has found, about half will interpet the new day as Monday, and the other half Friday.

The challenge of clear communication has taken on a new urgency with the rise of AI, now that many people with domain-specific knowledge have to explain it to an agent. Research has repeatedly shown that LLMs will transparently resolve uncertainties with plausible interpretations. This can sometimes be helpful, but what about when it isn’t?

Some uncertainties turn out to have material implications. In financial applications, you need to be certain of how the currency should round or whether fees come out before penalties.

From the AI’s perspective, there may be many systems which appear similar to the one you are building. The decisions other similar systems make will exert a gravitational pull on the direction your ambiguities get resolved by an AI, whether you agree with them or not.

We wanted to see how the dedicated requirements-capture skills handle this when given little to go on. Do they prompt you to be more specific, or fill the gaps with guesswork?

Running the gauntlet

At JUXT we build software for banks and capital markets, so we built a test gauntlet around three core-banking tasks: applying a loan payment, building a repayment schedule, and posting interest on a savings account. Each task sets a high level objective and intentionally leaves almost every important detail undefined.

For example, the loan task describes some desired functionality in a few lines, then ends with an instruction to “establish whatever details you need to implement it correctly”. The brief is reproduced in full at the end of this article.

Driving each skill under test is a stakeholder agent, briefed with fourteen ‘latent requirements’ that would shape the implementation. It offers them up when asked directly, but it doesn’t volunteer them otherwise. This behaviour isn’t intended as a trap: it simulates what we see on real projects all the time. A stakeholder may simply not know which details are relevant, may struggle to recall them in the moment, or may regard them as ‘so obvious’ that they don’t need to be mentioned at all.

Each of the fourteen latent requirements is an ordinary choice that isn’t the common default. To make sure we weren’t being unfairly obscure, every latent requirement maps directly to a setting that Apache Fineract (a widely used open-source core-banking platform) exposes as a configuration option. The loan, for instance, is denominated in Bahraini dinar, which runs to three decimal places, and a residual of 0.005 dinar or less is written off. After eliciting requirements from our stakeholder agent, each skill produces a specification in Markdown or some other structured form.

In a second phase, that specification is handed to an evaluator agent, along with the same fourteen latent requirements. The evaluator studies the spec and reports how many of the fourteen were captured.

Each skill was installed from its instructions and, aside from being driven by an AI agent, is run the way a user would run it. We captured a full transcript of every exchange, all of which you can read in our repository:

• Allium is our own, an open-source behavioural specification language. We ran its elicit skill, which interviews you and builds a specification you can execute.
• GitHub Spec Kit is a spec-driven development toolkit. We ran its /specify and /clarify commands.
• OpenSpec is an open-source spec-driven workflow. We ran its /opsx:propose command.
• BMAD-METHOD is a multi-agent workflow. We ran its analyst interview and product-requirements skills.
• Kiro is AWS’s spec-driven IDE. We ran the requirement-gathering step of its Spec agent, which writes requirements in EARS notation.
• The AI Unified Process models a feature as use cases and an entity model. We ran its requirements and use-case skills.
• Tessl ships a requirement-gathering skill, which we ran. It asks one question at a time.
• Superpowers is a library of agent skills. We ran its brainstorming skill, which also probes one question at a time.

Lastly we added a short prompt as a control that asks the LLM to act as an engineer and ‘ask the stakeholder clarifying questions’ before drafting a spec. The prompt is short and replicated in full at the end of this article. It sets a baseline which the dedicated skills ought to clear.

What the tools showed

The chart shows each tool’s mean proportion of captured requirements across the three tasks.

The baseline prompt captured just over 60% of the hidden requirements, placing it near the bottom of the table. All of the dedicated skills did better except one: Spec Kit.

The full results are shown below, every task across all three repeats.

What the tools missed

Spec Kit is the only tool that explicitly limits how much it asks during requirements capture, and it shows. Its instructions cap the stakeholder exchange at a ‘maximum of 5 total questions across the whole session’ and allow at most ‘3 [NEEDS CLARIFICATION] markers total’ in the spec. For any unresolved requirements, the model is told to ‘make informed guesses based on context and industry standards’.

That may be a reasonable trade-off in many situations. For software in financial services, we don’t think it is. Our own tool Allium makes the opposite choice. Its elicitation skill is instructed to leave no material detail unasked, down to an explicit instruction not to settle for a vague answer: ‘Do not accept “yes” without specifics. You said yes, [X can happen]. How many times? Is there a limit? What happens after that?’ The full script is published in the Allium repository.

Draw a trend line between the number of questions asked and the share of requirements captured, and a clear pattern emerges: the more a skill asks, the more it tends to capture. Allium asks more questions, which makes it a more demanding partner in the conversation. I think these results demonstrate the value of the approach.

The cost of being helpful

We set out to learn whether the dedicated requirements-capture skills would push a vague user to be more precise.

The answer is that most tools don’t demand too much of the stakeholder. They tend to cluster together as measured by the number of questions asked. They accept the ambiguity, with one tool even intentionally masking it with guesswork. I think this is a risk for software built with AI.

Remember Hart’s vehicle rule: everything we communicate usually has a well-understood core and a fuzzy border. Back when specifications were gathered by hand, an engineer would usually stop and ask when a requirement’s interpretation seemed to sit near the subjective edge.

The designer Charles Eames said it simply: ‘the details are not the details, they make the design’. If you’re using an AI to elicit requirements and write specifications, are you taking the time to ensure you agree with the ‘informed guesses’ being made on your behalf?

We’ve published the gauntlet, the three tasks, and every scored transcript so that others can verify and replicate this analysis.

The loan-allocation brief in full, the text every skill starts from and nothing more:

We’re adding payment allocation to our loan servicing system. When a borrower pays, the money has to be applied across what they currently owe on the loan. Build a function allocate_payment(payment, loan) that returns how the payment is applied and the loan’s updated balances. The loan can have several instalments outstanding, each with penalties, fees, interest and principal components. Establish whatever details you need to implement it correctly.

The baseline prompt in full, which performed better than many of the dedicated skills:

You are an engineer. Write a requirements specification for the feature in the brief, detailed enough that a developer could implement it correctly.

You may ask the stakeholder clarifying questions first, or not, as you judge best. When you are ready, write the specification.