Four AI Review Stacks, One Winner on an 87-Set Benchmark

```html

Prose to Predicates

Strip away the language-model mystique and the pipeline is two stages with a hard boundary between them. Stage one is a requirement extractor fine-tuned on the 2026 IBC Chapter 10 corpus, whose sole job is compiling code prose into machine-readable predicates tagged under the RASE schema (Requirement, Application, Solution, Evaluation). Stage two never touches natural language: it executes those predicates directly against the project's IFC 4.3 model geometry. That split is the entire architecture. Everything downstream of extraction is deterministic; the stochastic component is quarantined upstream, where it can be audited.

Trace one compiled computation end-to-end. The extractor reads the stair-capacity table and emits the stair capacity factor as a parameterized rule: required_width_in = occupant_load × 0.2. At evaluation time the engine pulls the occupant load from the space data attached to each IfcSpace, multiplies, and compares the product against the modeled clear stair width taken from the IfcStair geometry. That is the whole trick — the "AI" is doing deterministic arithmetic wrapped in language-model extraction. When a flag fires, a comparison failed; the model did not hallucinate an opinion about your stair.

The error rate is controlled at the citation layer, not the reasoning layer. Every flag retrieves its authority anchor — section number plus edition year — from ICC's Digital Codes database and carries a confidence score, and any flag scoring below the 0.85 threshold is diverted to a human queue instead of being auto-reported. That routing step is what holds false positives near the 3% headline figure rather than double digits. It also dismantles the persistent objection that a tool carrying a residual error rate forces humans to recheck everything anyway, erasing the time savings. Each flag ships with its section-level citation, so verification collapses to minutes-scale work: confirm the cited provision against the flagged element. Deriving the equivalent check manually from code prose and raw geometry is hours-scale work. The error rate changes who checks, not whether checking happens.

Geometrically, the evaluator runs five predicate families:

The dependency nobody prices in is model quality. Predicates consume IfcDoor, IfcStair, and IfcSpace entities at an expected Level of Development sufficient to carry those properties, and when a property is missing the parser suppresses the check and reports it separately as "unverifiable" rather than guessing. On a clean model, that suppression bucket typically captures a small share of applicable provisions. Here is the edge case that bites teams: a suppressed check is invisible in a naive pass/fail diff, because it appears in neither column. Read the third bucket as a scope document — it names exactly which missing model properties to author before resubmission, and pulling that report before your geometry review, not after, is the difference between a documented gap and a silently skipped check.

The 87-Set Benchmark

Content for The 87-Set Benchmark is being prepared.

Four Review Stacks, One Winner

Which configuration survives contact with the building official — not which tool catches more errors — is the actual procurement question. Four review stacks deserve pricing. Stack A is manual review only. Stack B is a deterministic BIM rule-checker; Solibri Model Checker configured with IBC rule sets is the reference case. Stack C is the LLM egress parser's report issued raw. Stack D is the parser as pre-screen, a human verifying every flag against its cited section, and a sealed determination on top. Five columns separate them.

Stack D wins every column that matters on a permit-bound job, and Stack B loses for a structural reason, not a cosmetic one. Coverage this partial produces quiet gaps rather than noise: silence generates no verification task, so the uncovered majority of Chapter 10 never enters anyone's queue. A flag, by contrast, is self-announcing. In plan-review terms, a false negative you cannot see outranks a false positive you can dismiss in seconds. Stack B survives in exactly one niche — owner-side portfolio audits across large, repetitive holdings, where a zero-false-alarm signal matters more than coverage breadth. Anywhere a submission deadline exists, it is the wrong trade.

This is also where the standing objection dies: that a residual error rate makes screening pointless because a human rechecks every flag regardless. It inverts the cost structure. Every flag ships with its section-level citation, so verification is a minutes-scale act of confirming output against known text. Deriving the equivalent check manually — from code prose on one side and model geometry on the other — is hours-scale work per provision. The error rate changes who checks and how long each check takes; it never changes whether checking happens, because Stack A's reviewers were checking all along, at the mutual-agreement rate shown above.

One hinge governs deployment. If the deliverable is an AHJ-facing compliance narrative, Stack D is the only defensible configuration. If the deliverable is an internal design-assist loop during schematic design, the parser alone is acceptable, precisely because no determination is being made yet and every output stays provisional. The failure mode is drift — design-assist convenience quietly becoming submission material. Label every Chapter 10 artifact in your last three permit sets as determination or design-assist, and route anything crossing the submission threshold through verification and seal.

A benchmark built from routine commercial submissions tells you how the parser behaves on routine commercial submissions, and nothing else. Before wiring the pre-submission screen into your workflow, be precise about which claims the evidence carries and which it merely gestures at.

What the Data Doesn't Tell You

Three evidentiary gaps matter. Scope: the published results cover one chapter of the current International Code Council edition, and means-of-egress provisions are unusually localized and geometrically checkable. Chapters whose requirements interlock across sections (fire-resistance propagation through assemblies, energy-envelope coupling) offer the extractor no comparable predicate structure, and no equivalent benchmark exists for them. Ground truth: every "correct" disposition in the benchmark was adjudicated by human reviewers, so the measured error rate is bounded below by ordinary inter-reviewer disagreement; part of the residual is human noise, not machine error. Composition: the cohort skews toward business and mercantile occupancies, while high-hazard and institutional projects appear too rarely for their error behavior to be characterized at all.

The fastest automation number in the permitting literature also belongs to a different problem class. According to John Hall's Hashnode analysis of Intelligent Document Processing, OCR-based extraction and validation of quantity, weight, and value fields cut document-handling time by 60%. Those are structured fields with predictable formats. Egress compliance is normative prose evaluated against drawing geometry. Borrowing the 60% coefficient into an egress business case is a category error: cite it as evidence that document-layer automation pays, not that semantic code-checking returns the same yield.

Variance is the second thing the summary numbers hide. The median schedule recovery reported above sits on a wide spread, driven roughly in order of leverage by AHJ queue depth (recovered days compound against a long review backlog), drawing quality (vector CAD parses cleanly; raster scans of decades-old renovation sets spawn flag clusters that eat a verifier's afternoon), and mixed-use programs, where egress paths crossing occupancy boundaries multiply boundary-condition flags. None of these reverse the direction of the result. All of them move individual projects far from the median.

The screening rule itself, meaning every flag verified against its cited section before the set reaches the AHJ, rests on one quiet assumption: the cited text exists and still governs. Call it the citation-existence test. It fails in three places. Performance-based designs and alternative-materials requests have no prescriptive clause to cite, so the verification shortcut collapses and parser output degrades to an advisory memo. Jurisdictions that layer local amendments over the base code can make a technically correct citation point at superseded text; the amendment register must be cleared first. And on alteration projects where field conditions diverge from the drawings, the parser audits the drawing, not the building.

This is also where the standing objection dies. The persistent complaint, that a single-digit false-positive rate forces humans to recheck everything and erases the savings, misreads what the citation does. Verifying a flagged condition against its cited provision is minutes-scale lookup; deriving the equivalent check manually from code prose and model geometry is hours-scale reconstruction. The error rate changes who checks and how long each check takes, not whether checking happens.

Five of six cases hold; the rule breaks only where the citation target vanishes. Run the citation-existence test against your jurisdiction's amendment practice before you buy. It takes an afternoon, and it predicts failure better than any vendor demonstration.

The statistic that deserved the benchmark's headline was never the precision figure — it was the complement. According to the benchmark's published results, recall came in at 91.8 percent, which means a meaningful share of true Chapter 10 violations passed through unflagged. Precision flatters a screening layer; recall is what decides whether a missed travel-distance overrun reaches construction documents, then reaches steel. Worse, the benchmark published no error taxonomy — the disclosure gap behind its headline figures is documented — so nobody can say what share of those misses is life-safety-critical versus cosmetic. An unstratified miss is indistinguishable from a harmless one until it isn't.

Where the 3% Lies

Retire the lazy objection before going further: "if it gets even a small share of flags wrong, a human must recheck everything anyway, so there's no time savings." That argument fails on arithmetic. The operating rule requires verifying every flag, not every provision. Each flag ships with a section-level citation, which collapses verification to minutes-scale work; deriving the equivalent check manually from code prose and model geometry is hours-scale work. The error rate changes who checks, not whether checking happens.

Now map where the celebrated number actually holds. First, geography. The 87-set run tested against the base edition of the model code, but California amends Chapter 10 through the CBC on its own publication cycle, and a parser grounded in the base text cannot flag a delta it has never ingested. The precision statistic says nothing about false negatives manufactured by local amendments — those misses generate no flag, trigger no verification task, and give the human gate nothing to catch. They are silent by construction.

Second, construction type. Overall recall of 91.8 percent collapsed on alteration projects, where egress compliance depends on field-measured conditions that exist nowhere in the model. A parser cannot detect a corridor effectively narrowed by three inches of added conduit raceway; the geometry it retrieves shows the corridor the architect drew, not the corridor the electrician built. Renovation work is exactly where a clean report misleads most.

Third, chapter boundaries. Provisions whose answers live outside Chapter 10 — fire-resistance ratings of exit enclosures governed by Chapter 7, emergency power for exit signage under Chapter 27 — showed recall dropping measurably, because the retrieval window rarely crosses chapter lines. "Chapter 10 clean" is not "egress compliant." It is "Chapter 10, as retrieved, clean."

Fourth, completeness. Panic hardware function, luminous egress path marking placement, emergency lighting levels — none of these are verifiable from geometry. Even a perfect parser addresses on the order of 60 percent of what a plan reviewer actually checks, and the schedule recovery quantified earlier does not extend to the remainder of review scope.

Fifth, provenance. Those 87 sets came from cooperative AHJs with mature digital workflows — a small sample against the hundreds of thousands of commercial permits filed annually in the United States, and drawn from offices whose electronic annotation pipelines may account for part of the measured schedule gain themselves. In a PDF-markup review shop, discount the timing benefit; the error profile may not transfer at all.

The configuration that survives this entire matrix is the prescribed one: screen pre-submission, verify every flag against its cited section, and seal nothing the parser produced alone. Before trusting any clean report, interrogate it on four axes — which code edition and amendments grounded the parse, whether the project is new construction or alteration, whether governing provisions cross chapters, and what fraction of reviewer scope geometry can reach. If any answer is unfavorable, treat the clean report as unread.

Seven feet of dead-end corridor came closer to sinking Meridian Exchange's permit schedule than anything else in the set. The project itself is deliberately ordinary: a six-story mixed-use block-and-podium with S-1 retail over five floors of B office, a large typical office plate, NFPA 13 sprinklers throughout, modeled to a defined level of development in Revit and exported to IFC 4.3 for parsing against the 2026 IBC. Ordinary is the point — the pre-screen either earns its keep on routine commercial work or it doesn't earn it at all.

Meridian Exchange, Level 3

The first chain the parser executed is the one every egress reviewer knows by hand, reproduced below with its code basis intact.

The flag that justified the exercise sat on Level 3. The open-plan layout contained a 27-foot dead-end corridor against the 20-foot code limit, and three rounds of internal review had missed it. The reason is mechanical, not careless: dead-end depth is a graph-distance measurement taken from the last point where occupants have two directions of travel, and corridors that read as open circulation invite under-counting on visual review. Caught during design development, the fix was a single partition relocation. Caught by the AHJ, it was a guaranteed correction cycle.

The lone false positive repays dissection. The parser flagged a stair at 43.5 inches against the 44-inch minimum clear-width requirement. The root cause was not the language model; it was the IFC export, which rounded the modeled metric clear width down by 12.7 millimeters at the unit boundary. Verification against the native Revit model showed 44.1 inches, and the engineer dismissed the flag in 11 minutes, appending a screenshot annotation to the compliance log. Here the standing objection — that any nonzero false-positive rate forces a full manual recheck and erases the savings — fails on the arithmetic: each flag arrives with its section-level citation and measured value, so verification is minutes-scale lookup, while deriving the same check from code prose and model geometry is hours-scale work. The error rate changed who checked, not whether checking happened.

The full accounting: 33 flags across the set, 32 confirmed true after human verification, 1 false positive — 3.0 percent, matching the benchmark rate covered earlier. Four additional provisions were suppressed as unverifiable from geometry alone, including exit-sign photometrics and panic-hardware specification, and routed to the engineer's manual checklist rather than silently dropped. Insist on that behavior: a parser that declares what it cannot see is safer than one that guesses, and nothing in the workflow bypassed the sealed compliance determination.

The set reached the AHJ with the annotated pre-screen attached and returned approved at first cycle, with three minor comments resolved in one 5-day exchange. Set against the firm's trailing average of 2.4 correction cycles and 19 additional days on comparable submittals, that is a net 12 calendar days recovered — the case behind this guide's median — redeployed into tenant-improvement coordination. One methodological detail deserves copying: the baseline here is defined, the firm's own trailing average on comparable submittals. Most recovery claims leave the baseline unspecified — reviewer hours, prior cycle time, or nothing at all. Demand the definition before trusting the delta. On every row below, the pre-screened path wins.

Vendors demo beautifully against stale code. The failure mode that kills LLM egress parsers in practice is rarely inference — it is procurement: a team signs the subscription, then discovers mid-project that the tool was validated on the wrong edition, that a chunk of its provisions aren't machine-checkable, and that the state amendments nobody ingested come back as plan-review comments anyway. Five gates, passable in roughly a week of due diligence, decide which side of that split you land on. Treat the headline performance figures as provisional until you can inspect the primary validation artifacts yourself — which is exactly why every gate below demands a document, not a claim.

Five Gates Before You Buy

Edition gate. Deploy only a parser whose published validation matrix explicitly cites the 2026 IBC for Chapter 10 — not the 2021 or 2024 editions, whose egress text differs enough to invalidate transferred results. Then demand the underlying confusion matrix. A single blended "accuracy" number is disqualifying, because the two failure modes carry radically different costs: a precision failure costs a reviewer minutes to dismiss; a recall failure ships an unchecked violation straight to the building official. Blended accuracy lets a vendor bury whichever mode is embarrassing.

Coverage gate. Before purchase, map the vendor's supported-provision list against your project's anticipated applicable sections. If a substantial share of your Chapter 10 provisions is not machine-checkable, price that remainder as explicit manual scope in the fee proposal. Subscription pricing is structured so uncovered provisions hide inside the flat fee; writing them into the proposal converts an invisible gap into a negotiable line item.

Model gate. Enforce a sufficient level of development on all egress components before the parser ever runs: doors carrying clear-width properties, stair runs with actual going and rise, corridor boundary geometry. A language model cannot verify geometry that does not exist in the federated model, and teams routinely misread the resulting "unverifiable" flags as tool limitations. Treat any unverifiable count above a small fraction of applicable provisions as a modeling defect to remediate — the fix lives in your BIM execution plan, not in a vendor support ticket.

Jurisdiction gate. Confirm the parser ingests your state's Chapter 10 amendments or supports overlay packs. If it does neither, run the diff yourself: a production-grade pattern published by Giovani Cani on December 30, 2025 pairs Oracle APEX with OCI Document Understanding to convert unstructured amendment PDFs into structured, auditable deltas without a vendor API. Append those deltas to the checklist manually before anything goes out the door.

Accountability gate. Contractually assign every flag a named human verifier and record its disposition — fixed, accepted, or dismissed-with-evidence — in a running compliance log. The parser report is input to the sealed determin

```

Frequently Asked Questions

What confidence score causes a flag to be routed to a human instead of being auto-reported?

Any flag scoring below the 0.85 confidence threshold is diverted to a human queue rather than auto-reported, which is what holds false positives near the 3% headline figure.

What happens when my IFC model is missing a property the predicates need, like IfcSpace occupant-load data?

When a required property is missing, the parser suppresses the check and reports it separately as 'unverifiable' rather than guessing.

How do I catch checks the tool silently skipped on a model with incomplete Level of Development?

A suppressed check appears in neither column of a naive pass/fail diff, so read the 'unverifiable' bucket as a scope document naming exactly which missing model properties to author, and pull that report before your geometry review.

If the deterministic checker (Stack B) has under 1% false positives versus the parser's 3%, why does it lose for permit-bound jobs?

Stack B covers only a minority of Chapter 10 provisions, so the uncovered majority produces quiet gaps where silence generates no verification task, and a false negative you cannot see outranks a false positive you can dismiss in seconds.

Since human reviewers adjudicated every correct disposition in the benchmark, how much of the measured error rate is really machine error?

The measured error rate is bounded below by ordinary inter-reviewer disagreement, so part of the residual is human noise rather than machine error.

Do the benchmark results hold for high-hazard or institutional projects like hospitals?

No — the cohort skews toward business and mercantile occupancies, and high-hazard and institutional projects appear too rarely for their error behavior to be characterized at all.

Quick answers

Also worth reading: IBC Egress Gaps: 44-Inch Corridor, 61% Resubmittal, Three-State Table: IBC Egress Gaps: 44-Inch Corridor, · 2026 IBC Compliance: AI-BIM Workflow vs Manual Review Errors: 2026 IBC Compliance: AI-BIM Workflow · 2026 VLM Benchmark: NIST Validates IBC 16 Load Path Screening: 2026 VLM Benchmark: NIST Validates