The owner reported a fair number of 403 and 404 errors when adding recipes by URL. A probe of the failing blogs showed they answered with full recipe data to any client that did not claim to be a browser. The importer had been sending a Chrome-like user agent, and that disguise was exactly what tripped the bot challenge in front of those sites.
The decision
The importer now identifies itself honestly as CookbookImporter/1.0. Only if a site answers 401, 403, or 406 does it try once more with a browser-like agent, and then it stops.
Ruled out, and why: impersonating a search crawler (dishonest, and the sites would catch on), proxy or unblocking services (the same thing one step removed), the Wayback Machine (stale pages, not the live recipe), and any AI reading of the page, which the standing no-AI rule for this product forbids.
What the user is told
The reviewer caught a second problem in the same package: a 429 is the site’s rate limit, so retrying at once under another name works against what the site asked and then tells the user the wrong thing. So:
- A 429 is not retried. The page says the site is busy and to try again in a while, instead of suggesting the recipe be typed by hand.
- A 404 says the link is wrong.
- A site that refuses every client, which includes Allrecipes, Serious Eats, and the rest of the Dotdash Meredith group, says plainly that it does not let apps read its pages.
For that last group the only route left is reading the page in the user’s own browser. Whether to build that was held for the owner, who said yes four days later; see the entry on the dev site.
Shipped
Both changes went live the same evening, along with the docs and the What’s new card fix, across all three Lambda functions. The layout-2 database cut-over stayed built, reviewed, and deliberately unmerged: it is the switch itself, it waits on the owner’s freeze window, and nothing that changes the Lambda code may land during that window, so this went first.