Trevor PaughSystems engineer
LinkedIn GitHub
Trevor PaughProject Chronicles
Project Chronicles/chronicles

Chronicles / PermitBuzz

Green, yellow, red: how a feed earns its place

Decision

Every data source gets a colour from its written terms, and anything ambiguous waits for the agency's written answer. On day one of the build that put Cary and Forsyth on yellow with connectors built but unscheduled, and Wake County on green but held, because its published file turned out five weeks stale. A freshness guard now refuses any file older than a week.

The owner’s hard rule, in the pull quote, was always going to collide with the real feeds. On the first day of ingest work it did, three times, and the response each time was the same: build the connector, do not schedule it, and ask.

The colour system

Each source is marked from its written terms. Green means the terms plainly allow the use and the connector may run nightly. Yellow means the terms are silent or contradict one another, so the connector exists but stays off until the agency answers in writing. Red means the site forbids automated access, and the source is off the list. Nothing ambiguous is read as permission.

Day one

  • Cary: yellow. The city’s open-data terms and the licence on its map service do not agree. The connector was built and merged, and it waits on an email to the city, which the owner sends personally.
  • Wake County: green, but held. The county’s map host forbids all automated access in its robots file, so only the open-data hub’s download route is used, which both hosts allow. Then the downloaded file turned out to be five weeks old. A stale feed means misleading alerts, so the source is marked “coming soon” rather than scheduled, and a freshness guard in the pipeline now flags any file older than seven days instead of trusting it. A second email, to the county’s mapping team, asks about the refresh schedule and permission for one query a day. The owner has not yet signed off on this one and may still overrule it.
  • Forsyth: yellow. The county publishes no robots file at all, which is neither a yes nor a no. The builder was allowed exactly one call to read the feed’s metadata and nothing more until the county’s written OK arrives.

Also merged

The third pull request of the day carried the Cary connector, the incremental-window fix from the first review, and a guard that stops the nightly run from starting at all unless a contact address for the operator is configured, so any agency that looks at its logs can see who is asking and how to reach them. All three reviews so far have ended in a merge after fixes.