Using NeoSigma’s auto eval creation (accessible through the NeoSigma plugin or CLI), we ran a test over three days of Listen Labs' analysis chat traces. It confirmed 87 real failures, grouped them into 7 kinds, and turned them into 16 evals you can rerun to check whether a change to the agent fixed them. Your team can run the same process on any duration of traces, and this page shows how.
Listen Labs' verifiers flagged 243 sessions. NeoSigma then reviewed each session in full to confirm whether the user actually got a bad result, and to explain why.
NeoSigma clustered the 87 confirmed failures into 7 kinds:
| Failure | Sessions | What it looked like |
|---|---|---|
| Numbers that didn't show up | 29 | Counts, percentages and durations returned as blank placeholders instead of values, for example how many people declined a consent step. |
| Decks and documents not produced | 16 | The agent stopped to ask “from scratch or a template?” instead of building the deck, or ignored an instruction such as “don't include recommendations”. |
| Reels and clips that fell short | 15 | Reels described but not viewable, clips built from the wrong part of an interview, and clips requested from data the session didn't have. |
| Incomplete answers | 8 | Part of the question skipped, such as one of two products asked about, or a requested breakdown not given. |
| Questions about the app itself | 8 | “Which page am I on”, a survey preview link, or opening a specific question's page. |
| Quotes that didn't match | 3 | Quotes about the topic in general when one segment was asked about. |
| Wrong language | 2 | Kept replying in Japanese after the researcher asked for English. |
Some failures were left out on purpose: sessions where the data genuinely wasn't in the study (6), a presentation-service outage, video edits the product doesn't support, and an internal test prompt. A rerun can't show improvement on those.
Each of the 16 evals stands for one of the failure kinds above, with one to three evals per kind. Each one is a real request from a session that NeoSigma confirmed as failed. Rerunning an eval sends the agent that same request and grades the answer with Listen Labs' own verifiers, so a pass shows the agent now handles a request that used to fail.
| Root cause | Sessions | Evals | Example request |
|---|---|---|---|
| Numbers that didn't show up | 29 | 3 | “How many people did not consent to Amazon giving them data…” |
| Decks and documents not produced | 16 | 3 | “Build a presentation with the key insights” |
| Reels and clips | 15 | 2 | “Create a highlight reel from the most compelling moments” |
| Incomplete answers | 8 | 3 | “Break down main reasons for hidden responses in this study” |
| Questions about the app itself | 8 | 3 | “Which page am I on” |
| Quotes that didn't match | 3 | 1 | “Do OneBlade users refer to the job as ‘clean-shaven’?” |
| Wrong language | 2 | 1 | “Please write the full analysis in English…” |
NeoSigma's review found 11 failures that none of the current verifiers flagged. They fall into two gaps. The changes below would let the verifiers catch these failures on every session, without waiting for a review.
| Gap | Example | Sessions | Suggested change |
|---|---|---|---|
| The agent declined a request and the reply still passed | “Publish this memo to the organization”: the agent said it can't | 5 | Update answers-the-users-question |
| A number placeholder may not have had a value behind it | A sentiment share returned as {{scalars/…}} | 6 | New verifier: scalars have values |
answers-the-users-questionPASS: the core deliverable the user asked for is present in the reply. FAIL: the assistant deflects, substitutes a generic overview for the specific answer requested, delivers a different artifact type than asked for, or omits the core deliverable. +This includes stating that it cannot perform the request (for example +publish, edit, generate a link, or open a page) without delivering the +requested output.
Example: a user asks “Can you publish this memo to the organization?” and the agent replies that it can't publish memos.
scalars-have-valuesThe verifiers require aggregate numbers to use scalar placeholders, and assume each one displays a value. This verifier checks that a value exists.
Scope: scalar, table, and chart references in the final assistant message. Check: each referenced scalar or table must be computed by a tool call in this trace, or refer to an existing artifact, and the computed value must not be empty, null, or an error. PASS: every reference has a computed, non-empty value. FAIL: a reference has no matching computation, or its computation returned no value. NOT APPLICABLE: no scalar or table references are present. For FAIL, identify the exact reference and what is missing.
This requires the trace to record each scalar's computed value.
Example: a user asks “How many people did not consent to Amazon giving them data?” and the agent replies “{{scalars/amazon_sponsor_consent_no}} did not consent”, but no calculation in the trace produced that scalar.
Any coding agent connected to the NeoSigma plugin can do this today. To set it up in Claude Code, Codex, Pi or Cursor, follow Agent plugin ↗. To run it from a terminal instead, see Create evals from the CLI ↗.
Check that each proposal is a real, fixable failure, and keep at most three per root cause.
Once the dataset is created, the platform can run it on a schedule and post the results to Slack. That shows whether a change to the agent fixed these failures, or broke something that used to work.
Scheduled runs need the project's repository connected and its setup script verified (see Run evals ↗).
Full instructions: Schedule evals ↗ and Post eval results to Slack ↗.