AEC-Bench: Mebbian took the same AI from 82% to 89% on drawing index tests

• Score up from 81.9% to 88.9%
• 12 public sets, 3 tries each
• Same AI model, test's own grader

We ran this test to measure one part of our drawing review agent against a public benchmark that anyone can rerun. Sheet index mismatches are a common source of missed scope in bidding. A page is on the sheet list but missing from the set, or a page carries a different title from the one on the list, and someone who prices the job from the list can miss work. Mebbian reads every sheet and compares it with the index, then writes notes on what it finds. We used AEC-Bench because Nomic AI built it, it uses real public drawing sets, and its own grader scores every answer the same way. That keeps the result independent of our own judgment.

Drawing sets go out to bidders with sheet lists that do not always match the sheets in the set. A page can be listed and missing, or carry a different title. These mismatches are easy to miss when a team reviews hundreds of sheets under a bid deadline, and a missed sheet can mean missed scope in the price.

The same AI model took the test on its own, with no Mebbian notes, and scored 81.9% across the 12 sets.

The score rose from 81.9% to 88.9%, a gain of 7.0 points, with the same model and the same 12 sets. With Mebbian's notes, the model reported the mismatch on two more sets on every try, and those two sets went to a perfect score. On the two sets where Mebbian missed the problem, the score stayed the same, so the gain comes from what Mebbian catches. This is a benchmark result on public drawing sets. It measures one check inside our drawing review agent on public sets. Time and cost savings at a firm come from a measured client job. We publish the per-set scores, the method and the limits so any team can rerun the test and check the result for itself.

Mebbian is AI that understands buildings. Drawing review is one of our AI agents. Full results: mebbian.ai/research/aec-bench.