Max AbramskyFlorida entity resolution, 2026earlyco.ai

Incident write-up · August 15 to 17, 2026

39 false merges, repaired in about 13 hours

On the night of August 15 a batch merge job ran against production for the first time. It merged distinct businesses that only shared an address: grills inside one arena, stands at one venue. By 10:37 the next morning the 39 confirmed false merges were repaired in production with decision records, merges had become reversible, and a precision gate guarded the merge path in CI. Within 48 hours, four more false-merge classes found in dry runs had become vetoes.

Impact
39 confirmed false merges of distinct businesses in the production entity graph
Run to repair
13 h 4 min
Found by
A read-only audit of the run the next morning
Root cause
A second matcher that hadn’t been through the same checks ran against production

Written from the build record. Collection has been off since early September 2026.

Timeline

Time from the production merge run, August 15 at 21:33 EDTMatcher rebuilt and merged: 12 h 40 min after the run; 39 false merges repaired in production: 13 h 4 min after the run; Every distinct-identifier pair adjudicated: 14 h 46 min after the run; Four more false-merge classes vetoed: 43 h 53 min after the run.0 h12 h24 h36 h48 hMatcher rebuilt and mergedvetoes, reversible merges, CI gateMatcher rebuilt and merged: 12 h 40 min after the run12 h 40 min39 false merges repaired in productionwith decision records39 false merges repaired in production: 13 h 4 min after the run13 h 4 minEvery distinct-identifier pair adjudicatedverdicts stored as decisionsEvery distinct-identifier pair adjudicated: 14 h 46 min after the run14 h 46 minFour more false-merge classes vetoedfound in dry runsFour more false-merge classes vetoed: 43 h 53 min after the run43 h 53 minTime from the production merge run, August 15 at 21:33 EDTMatcher rebuilt and merged: 12 h 40 min after the run; 39 false merges repaired in production: 13 h 4 min after the run; Every distinct-identifier pair adjudicated: 14 h 46 min after the run; Four more false-merge classes vetoed: 43 h 53 min after the run.0 h12 h24 h36 h48 hMatcher rebuilt and mergedMatcher rebuilt and merged: 12 h 40 min after the run12 h 40 min39 false merges repaired in production39 false merges repaired in production: 13 h 4 min after the run13 h 4 minEvery distinct-identifier pair adjudicatedEvery distinct-identifier pair adjudicated: 14 h 46 min after the run14 h 46 minFour more false-merge classes vetoedFour more false-merge classes vetoed: 43 h 53 min after the run43 h 53 min
Time from the production run, August 15 at 21:33 EDT, to each fix. Times are commit times.

  1. A fix lets the batch merge job return candidates for the first time. Its commit message: “first production run is merging split folds tonight.”

  2. Candidate sampling is randomized so that repeated rounds cover every address.

  3. A read-only analysis pass, “triggered by the 2026-08-15 production cluster-merge run”, audits the result. A forensic script ranks every merge by risk class.

  4. The matcher is rebuilt: identifier vetoes, reversible merges, rollback tooling and a CI precision gate (PR #2188).

  5. The repair merges: “39 confirmed false merges from 08-15 repaired in prod with decision-edge provenance” (PR #2189).

  6. Human verdicts become decision records, and every distinct-identifier pair from the run is dispositioned (PR #2200).

  7. Dry runs surface four more false-merge classes. Each becomes a veto and a test fixture.

Root cause

Two matchers existed. Path A was governed: an identifier graph plus a language-model verifier, gated in CI at precision 1.0. Path B, a duplicate-cluster merge, blocked on address and merged on a name heuristic. It had not been through the fixtures, gate, verifier and rollback that Path A had.

Path B had never actually run. A row-shape bug made it return zero candidates on every invocation, silently. Fixing that bug on August 15 let it run for the first time.

Its rule merged any pair at a shared canonical address once name containment, shared tokens over the smaller name, reached 0.7. It used no negative evidence: two different active food licenses at one address were never consulted. A one-word generic name scores 1.0 against any longer name that contains it: “TACOS” fully matches “TACOS” plus any other words. Food halls and arenas file without unit numbers, so whole venues collapse onto one address. The arena grills scored 0.70 to 0.81.

Merges did not yet record pre-merge state, so making them reversible became part of the fix.

The cause, in one line: a second matching step that hadn’t been through the same checks as the first ran against production.

Detection

The next morning a read-only analysis pass audited the run. A forensic script ranked every merge by risk class, worst first: pairs that held distinct identifiers. Distinct current licenses in the same series at one address made the clearest class, because two active licenses in one series are two businesses.

Fix

  1. Rebuild, 10:13

    Identifier vetoes, one shared name-similarity module, and run-scoped reversible merges: every merge edge now records its run and the prior canonical record of both sides, and that migration was applied to production before the code merged. Rollback tooling reverts newest-first. A nightly pipeline keeps the scheduled merge in dry run. The rebuild also fixed three latent defects, including an entity-resolution tier that could not run at all: its tests mocked the database, so its SQL had never executed.

  2. Repair, 10:37

    Only the unambiguous class was repaired automatically: distinct current licenses in the same series. Cross-series pairs stayed merged as dual licensing, which avoided 52 false vetoes. Ambiguous-series and registry or FEIN conflicts went to a person, not to the script. The script ran dry by default and needed an explicit apply flag.

  3. Adjudication, 12:19

    Human verdicts became decision records (same entity, different entity, needs review), so reviewed pairs stop reappearing in forensics.

Verification

  • In production. 39 confirmed false merges repaired with decision-edge provenance, and every distinct-identifier pair from the run dispositioned by 12:19.
  • In CI. A merge precision fixture of 23 labeled pairs, most negatives taken from real production dry runs. One false positive fails the build.
  • Locally. A real-data merge evaluation over 231 pairs scored precision 1.000 and recall 1.000, with three identifier vetoes firing at a food hall. This was a local run, not part of CI.
  • Rollback refuses to guess. Merges made before reversibility existed are reported as unrevertible instead of being undone by inference.

What changed after

  • Dry runs became the fixture source. By August 17 four more false-merge classes were vetoes with fixtures: license succession, where 310 of 469 shared-identifier proposals were confirmed hand-offs; hotels pulled into merge scope, 262 of 1,950 entities in the first dry run; numbered carts, 95 proposals at one venue; and outlets that share a venue name.
  • Decisions are data. Every accept, veto or deferral writes a decision record, and human verdicts are durable.
  • Ingest supersedes instead of dropping. An identifier conflict at ingest is recorded and superseded, never silently dropped.
  • The gate is asymmetric, on purpose. A missed merge leaves a visible duplicate. A wrong merge ships a false identity claim. The merge path is tuned against the second.

The lessonKeep one governed matcher. Name containment is the wrong thing to decide identity, and a second path with write access will eventually run.