Benchmark · measured 12 and 14 November 2026

How an agent answers H-1B questions, with and without Filings

We gave the same AI agent 210 typical H-1B and PERM questions, in two sets of 105. Once with Filings, once with web search only. Below: how we set it up, how answers were graded, what came out, and where the test favours us.

90–95%of answers fully correct with Filings, against 65–83% with web search only (two sets of 105 questions)
≤ 3 in 100answers with a wrong figure with Filings, against 10–22 in 100 with web search only
About halfthe cost per correct answer of web search only (API-price estimate)

Results

Two question sets, three ways to answer

Score: a correct answer counts 1, a partly correct one ½, a wrong one 0. The interval is a 95% bootstrap interval over the questions. The score leaves out the caveat rule (see Grading): the Filings tools always state that caveat, so the rule favours them. The score with that rule is in the last column.

SetAgent withScore95% intervalWrong figureUSD per correct answer (estimate)Score with the caveat rule
1 · 105 questions used while building FilingsFilings MCP90.5%86.2–94.31.0%$0.02789.5%
1Filings CLI92.9%88.6–96.72.9%$0.01992.4%
1Web search only64.8%56.2–73.321.9%$0.07063.3%
2 · 105 new questions, written for this testFilings MCP94.8%91.9–97.60.0%$0.02594.8%
2Filings CLI93.3%89.0–97.11.9%$0.01993.3%
2Web search only83.3%76.2–90.09.5%$0.05679.5%

Set 1 ran on 12 November 2026 (MCP and web search) and the CLI again on 14 November, after we tightened the CLI's instructions on figures; its first run scored 91.0% (90.5% with the caveat rule). Set 2 ran on 14 November 2026. Web search did better on set 2 than on set 1, which shows how much a single question set moves the result: we only state what holds in both sets.

Where web search did as well or better: questions on the job titles an employer files, and on entry-level and wage-level pay in set 1.

Setup

One agent, three sources

Every answer comes from the same model (Claude Sonnet 5.5), run headless with the same effort and a limit of 15 turns and 900 seconds per question. Only the tools differ:

  • Filings MCP: the hosted Filings tools, nothing else.
  • Filings CLI: the filings command line and its skill text, nothing else.
  • Web search only: web search and page fetch, no Filings. It never reached Filings.

The questions ran in a shuffled order, two at a time, against a copy of the live service with the same data.

Questions

15 kinds of question, 7 of each

Sponsor checks, salary by employer and role, offer checks, top sponsors, PERM (green card), trends, job titles, entry level and wage level, new hires vs transfers, locations, compliance, sales leads, misspelled company names, the data itself, and questions the data cannot answer.

Set 1 holds questions we used while building Filings: expect Filings to do better there than on questions it has not seen. Set 2 was written for this benchmark by an agent that saw neither our test sets nor any answers, and was checked to be unlike the questions Filings was built on. Filings was not tuned on it.

Grading

Rules first, then a judge

Each question has reference figures computed from the DOL OFLC disclosure data. Rule checks compare every figure in an answer against them and check that the period is there. A further rule checks for the caveat "an LCA is not a USCIS approval". The Filings tools return that caveat with every answer, so this rule favours Filings: the headline scores leave it out, and the table shows the score with it next to them.

A second model (Claude Sonnet 5.5) then judges each answer against the question, the rule checks and the evidence the agent retrieved, without being told which agent wrote it. An answer's score is the lower of the two grades. "Wrong figure" means an answer stated a checked figure that does not match the reference.

Cost

An estimate, not a bill

USD per correct answer is the agent's token use priced at public API rates, divided by the number of correct answers. The runs themselves used a subscription, so this is an estimate.

Limits

Where this test favours us

The reference is our source

Reference figures come from the same DOL data Filings serves. A web answer with a correct figure from another source or period counts as wrong on that figure.

One run per set

Each set ran once, with one judge model. Agents vary between runs; the intervals cover the choice of questions, not that variation.

The judge can guess

The judge is not told which agent answered, but the shape of the evidence (tool output or web pages) can show it.

What the data is

The questions are about DOL LCA and PERM filings. A certified LCA is not a USCIS H-1B approval, and the data has no lottery odds or approval rates.

Try it on your own questions.

Keys are issued by hand during the pilot. No card, no sales call.