Run a file
Pick a set and a file. Claude's read of each report is frozen, so what you see is exactly what was scored. Flip a messy file to engine v1 to see what broke.
Try to slip one past the guard
Federal rules on appraiser independence bar any request that steers the opinion of value. Every revision request this engine drafts runs through the same check you can try here.
The eval card
Every rule, scored against planted defects. The numbers decide which rung a rule earns. Policy can cap a rung. Nothing can raise one.
A rung here is a starting position, not a promotion. Thirty files can't prove 98%. The lower bound column shows how far the evidence actually reaches. In production a rule moves up on its lower bound, which takes a few hundred labeled files per rule.
Claude read the mess fine. My matching code broke.
Engine v1 scored 60 of 60 on clean files and 5 of 30 on messy ones. Almost every miss came from the deterministic layer, not the model. That's the layer a PM can actually own.
How I'd run this for real
The staged rollout Loancrate describes for appraisal review, with the eval as the gate at every step.
Baseline
Count revision requests per appraisal today and tag why each one went out. That's the number this has to move.
Shadow
Run every rule silently on live files for two weeks. Underwriters decide as usual. The rules get graded.
Promote on evidence
A rule moves up when its 95% lower bound clears the bar. Never on the point estimate.
Hold out real reports
Keep a sealed set by month and by appraiser. Synthetic mess only proves the mess I thought of.
Close the loop
Every QC finding and investor defect becomes a labeled case. Every engine change reruns the whole set.