Trevor PaughSystems engineer
LinkedIn GitHub
Trevor PaughProject Chronicles
Project Chronicles/chronicles

Chronicles / PermitBuzz

A health check, and a guard against a shrinking feed

Progress

The nightly data pull can now say whether it is well: a health command goes red when a scheduled source has not produced in two days or a parcel load is older than forty-five, and the nightly job runs it last. Research into the next large county found its feed with no stable key, no last-edited stamp, and a row count that fell by seven percent in a week, so full pulls got a shrink guard that refuses to overwrite good data with a short one. A smaller city with a clean, open feed got a connector plan instead.

Milestone 1 made the data pull work. Milestone 2’s data work made it able to tell when it does not.

Health

A new command reads the sources table and reports, for every scheduled source in a county that is in use, when it last produced rows. Two days without is stale; a parcel load older than forty-five days is stale. Stale means the command exits red, the nightly job’s last step runs it, and the run’s own summary line carries the same verdict. A source that has never run is stale by design. The report prints counts and names, nothing from inside the rows.

The shrink guard

The research pass on the state’s largest county found a feed that is awkward in three ways: no unique key per permit, no field saying when a row last changed, and a total that shrank by about seven percent between one week and the next. A naive full pull would have replaced a complete table with a smaller one and called it a night.

So full pulls gained a guard. Before and after a pull, the connector asks the source for its count; a mismatch or short paging fails the run before any write. A connector can opt into a floor, and a pull that falls below the last good share warns in the summary line rather than silently accepting the loss. Repeated rows are folded with a stable order rather than left to collide. Three places where the built thing diverged from the design were written down as accepted divergences rather than papered over.

Asheville

The same research pass found a smaller city whose permit layer carries a point for every permit, a unique record id, and an open data licence. Its connector plan was accepted: incremental pulls on the status date, a full pull every seven days, and the comment fields never requested. Whether to store an applicant’s name at all is a question held for the owner; the default is no.

One more rule came out of the day, for the researchers rather than the code: before sampling rows from a new source, check its field list for anything name-shaped and leave it out of the request. Samples go to a file; only counts and field names get printed.