Two counties were planned on paper this day, with no requests made to either, while the first connectors kept running.
The shrinking layer
The larger county rebuilds its permit layers nightly, and between one check and the next the all-types layer had lost about seven percent of its rows. Nothing in the app had noticed, and that was the problem worth designing for.
- A short pull cannot delete anything. The runner only ever adds or updates; a permit missing from a pull is simply left alone. So the parcel loader’s old fear, a bad reload wiping the table, does not apply here.
- But a short pull looks fine. If paging stops early or the layer is caught mid-rebuild, the run is recorded as healthy, the source shows as fresh, and permits that changed upstream quietly stop changing.
- And repeated keys churn. With up to a hundred rows for one permit number, each row overwrites the last in turn, so the stored version flips from night to night and may raise spurious events.
The guard and the fold
The guard is generic and opt-in. Before a full pull the connector asks the server how many rows it holds; after, it compares that to what it fetched and to the last good pull. Fetching fewer than the server said fails the run. The server holding much less than last time is a warning, which the manager asked to appear in the scheduled run’s summary line so it is seen without reading logs. Two extra requests a night, accepted.
The fold orders rows by permit number so a permit’s rows arrive together, and merges them into one record, taking status and dates from the newest row by a fixed rule so the result is stable. What a repeated row actually means is a question that has gone to the county; if each row is a separate unit, the key changes and the folding goes away.
The second county
A plan for the smaller county’s feed found that the manager’s brief had been slightly wrong: the layer has point geometry after all, so no parcel join is needed. The plan also set a rule for the research notes themselves: sample values are printed only as formats, and tables show counts or field names, never a row’s contents. The manager approved the three aggregate requests the plan needs, to be made once the current work merges.