Benchmark · measured 12 and 14 November 2026
How an agent answers H-1B questions, with and without Filings
We gave the same AI agent 210 typical H-1B and PERM questions, in two sets of 105. Once with Filings, once with web search only. Below: how we set it up, how answers were graded, what came out, and where the test favours us.
Results
Two question sets, three ways to answer
Score: a correct answer counts 1, a partly correct one ½, a wrong one 0. The interval is a 95% bootstrap interval over the questions. The score leaves out the caveat rule (see Grading): the Filings tools always state that caveat, so the rule favours them. The score with that rule is in the last column.
| Set | Agent with | Score | 95% interval | Wrong figure | USD per correct answer (estimate) | Score with the caveat rule |
|---|---|---|---|---|---|---|
| 1 · 105 questions used while building Filings | Filings MCP | 90.5% | 86.2–94.3 | 1.0% | $0.027 | 89.5% |
| 1 | Filings CLI | 92.9% | 88.6–96.7 | 2.9% | $0.019 | 92.4% |
| 1 | Web search only | 64.8% | 56.2–73.3 | 21.9% | $0.070 | 63.3% |
| 2 · 105 new questions, written for this test | Filings MCP | 94.8% | 91.9–97.6 | 0.0% | $0.025 | 94.8% |
| 2 | Filings CLI | 93.3% | 89.0–97.1 | 1.9% | $0.019 | 93.3% |
| 2 | Web search only | 83.3% | 76.2–90.0 | 9.5% | $0.056 | 79.5% |
Set 1 ran on 12 November 2026 (MCP and web search) and the CLI again on 14 November, after we tightened the CLI's instructions on figures; its first run scored 91.0% (90.5% with the caveat rule). Set 2 ran on 14 November 2026. Web search did better on set 2 than on set 1, which shows how much a single question set moves the result: we only state what holds in both sets.
Where web search did as well or better: questions on the job titles an employer files, and on entry-level and wage-level pay in set 1.
Setup
One agent, three sources
Every answer comes from the same model (Claude Sonnet 5.5), run headless with the same effort and a limit of 15 turns and 900 seconds per question. Only the tools differ:
- Filings MCP: the hosted Filings tools, nothing else.
- Filings CLI: the
filingscommand line and its skill text, nothing else. - Web search only: web search and page fetch, no Filings. It never reached Filings.
The questions ran in a shuffled order, two at a time, against a copy of the live service with the same data.
Questions
15 kinds of question, 7 of each
Sponsor checks, salary by employer and role, offer checks, top sponsors, PERM (green card), trends, job titles, entry level and wage level, new hires vs transfers, locations, compliance, sales leads, misspelled company names, the data itself, and questions the data cannot answer.
Set 1 holds questions we used while building Filings: expect Filings to do better there than on questions it has not seen. Set 2 was written for this benchmark by an agent that saw neither our test sets nor any answers, and was checked to be unlike the questions Filings was built on. Filings was not tuned on it.
Grading
Rules first, then a judge
Each question has reference figures computed from the DOL OFLC disclosure data. Rule checks compare every figure in an answer against them and check that the period is there. A further rule checks for the caveat "an LCA is not a USCIS approval". The Filings tools return that caveat with every answer, so this rule favours Filings: the headline scores leave it out, and the table shows the score with it next to them.
A second model (Claude Sonnet 5.5) then judges each answer against the question, the rule checks and the evidence the agent retrieved, without being told which agent wrote it. An answer's score is the lower of the two grades. "Wrong figure" means an answer stated a checked figure that does not match the reference.
Cost
An estimate, not a bill
USD per correct answer is the agent's token use priced at public API rates, divided by the number of correct answers. The runs themselves used a subscription, so this is an estimate.
Limits
Where this test favours us
The reference is our source
Reference figures come from the same DOL data Filings serves. A web answer with a correct figure from another source or period counts as wrong on that figure.
One run per set
Each set ran once, with one judge model. Agents vary between runs; the intervals cover the choice of questions, not that variation.
The judge can guess
The judge is not told which agent answered, but the shape of the evidence (tool output or web pages) can show it.
What the data is
The questions are about DOL LCA and PERM filings. A certified LCA is not a USCIS H-1B approval, and the data has no lottery odds or approval rates.
Try it on your own questions.
Keys are issued by hand during the pilot. No card, no sales call.