87 guides
Guides
Field notes from building the tools: what actually breaks on real sites, what we measured, and the fixes we shipped.
Your scraper's filter did nothing, and the run looked perfectAn input key that is valid for an Actor but out of scope for the mode you picked gets accepted, dropped, and never mentioned. The run exits 0 with a clean-looking row count, and that count is indistinguishable from "everything matched" — found live on our own Apple Podcasts Actor, fixed, then swept across the fleet.
We swept a visual-diff threshold from 0.02 to 0.3 expecting a curve. We measured a cliff.Pixel-diffing two unrelated pages at three diffThreshold settings — the strict end reads as 99.75% different, the default as 6.22%, and the loose end barely moves from there. The useful range is one order of magnitude wide, not the whole 0-1 scale.
A federal-awards watch mode that alerts on changes — unless you filtered on the field that changed"Filtering a watch on minAwardAmount/maxAwardAmount or expiringWithinDays silently hides the exact change you're watching for. Measured live today: award 70T05026F5900N033 ($958,748.70) matches maxAwardAmount=1,000,000 and vanishes under maxAwardAmount=900,000 — so a watch bounded at $900k would never see that same award's amount grow into the gap."
A bare Substack handle can redirect to a profile page instead of the archive — and the response looks nothing like an errorA small slice of <handle>.substack.com URLs 302 to a substack.com/@handle profile page instead of the newsletter's own archive. No 404, no error JSON — just a 200 HTML page where you expected a JSON array. Measured live across 30 real newsletters.
"A government ID that isn't unique: USAspending reuses sub-award numbers across different filings"FSRS re-files a contract modification under the same Sub-Award ID as the original, and rarely reuses one under a different sub-awardee entirely. Keying on the ID alone silently dropped real rows. Live-measured on 500 sub-awards, reproduced, and fixed.
Steam hides review-bomb reviews from its own score by default — includeOffTopic undoes itWhen a game gets review-bombed over something unrelated to the game itself, Valve flags that stretch of reviews as off-topic and excludes them from the score Steam shows you. The exclusion is silent and the gap is large — measured on three games, live.
SAM.gov's wage-determination search doesn't match trades — and a short query silently becomes a different kind of search"q=roofing returns zero against 85,000+ live Davis-Bacon wage determinations — it matches the reference number, not the trade. Measured live: a 2-letter state prefix like q=AK silently stops being an exact match and becomes a 106-result OR across every determination whose code starts with it, with no error either way."
NIH RePORTER codes Quebec as two different states, and asking for the valid one returns 2 institutions out of 21org_states QC and PQ are both live codes in NIH RePORTER today, holding disjoint sets of awards. QC is effectively McGill plus one Montreal institute; PQ holds the other 19 organizations and is still being issued in FY2026. A valid-value check cannot catch this — live-measured counts, per-institution.
A Hacker News comment has no URL of its own — so GitHub links get attributed to the wrong repoHN's search API attaches the parent story's URL to every comment record and gives the comment no url field at all. Measured live on 500 comment hits: where both the story and the comment link a GitHub repo, they are different repos 30 times out of 33.
"Does a watchChanges deadline alert actually fire? We seeded 711 opportunities and forced it to find out""grants-gov-scraper's watchChanges re-delivers a saved opportunity if its deadline, status, forecast/posted type, award range or eligibility text moved. Live baseline of 711 posted opportunities, one stored snapshot deliberately altered, and the exact JSON the next run returned."
Google Play's "developer replied" watch event only fires for replies added after you start watching"Measured live across 8 popular apps: on some (Cash App 91%, Robinhood 85%, Coinbase 65%), the large majority of reviews visible right now already carry a developer reply. A fresh watch baseline swallows all of that silently — developerReplied can only ever fire on the minority left over."
"A federal-contracts scraper's \"AI opportunity score\" is a spreadsheet formula, confirmed live — ours just says so""A rival USAspending Actor markets AI opportunity analysis; re-confirmed live today it's still a plain weighted formula with no model call behind it. Our own opportunityScore field runs the same idea in the open: the full 0-100 formula, every weight, no ANTHROPIC_API_KEY required."
"A court that publishes one sparse opinion field tells you nothing about the others""Re-measured CourtListener's opinion metadata live across Ohio, Illinois, the Ninth Circuit and SCOTUS — found our own prior claim that Illinois matched Ohio on syllabus/neutralCite was wrong, and that SCOTUS fills judge on 20/20 despite leaving syllabus and neutralCite empty on every row."
A zero in federal program funding history doesn't mean zero dollars"Read all 7,396 CFDA assistance listings live: 97.3% of programs with a funding history do carry real dollar amounts, so the usual 'it's just flags' excuse is wrong. The actual trap is that 5,680 of 16,594 year-entries carry a numeric 0 that means 'not funded' or 'no data' — and 1,771 entries carry both an estimate and an actual that disagree."
"Location is not one field either: normalizing ten ATS job-board APIs down to city/region/country"Three of ten applicant-tracking systems hand back structured geography, three hand back one free-text string a human typed, and Workday's list payload gives you neither — it takes a second per-job call to resolve, and until 0.1.39 that call ran too late to help its own filter.
"Workable's job board API: no pagination, and the description flag is the one that's actually worth gating"apply.workable.com/api/v1/widget/accounts/<slug> serves any company's open roles as public JSON with no key. page/limit/offset are all silently ignored, but details=true is a genuine on/off switch — 4.4x-11.3x more bytes for one field, description, that we were fetching unconditionally even when nobody asked for it.
USPTO's own trademark APIs now all require a key — the keyless answer is a different office's databaseTSDR, the Open Data Portal, the old bulk-data API and the search SPA all fail without an API key as of October 2026. A federated EU-run index, TMview, answers the same question with the same official USPTO records and no key at all.
Teamtailor, Personio and Breezy — three more ATS job-board APIs, three more shapesTeamtailor's board is a JSON Feed with an embedded schema.org JobPosting, Personio's is XML with the title field called "name", and Breezy's structured location only shows up once you stop testing on the vendor's own demo tenant.
"SmartRecruiters' job board API: real pagination, a real per-posting cost, and one board that quietly leaks a salary nobody else publishes"api.smartrecruiters.com/v1/companies/<slug>/postings is the only one of seven ATS APIs we scrape that paginates for real — verified live from a 2-posting board up to a 24,867-posting one. The list endpoint carries no description at all, so a detail call per posting costs ~130x more per item than the list page it came from. And a freeform admin-configured field on one real board carries genuine pay-transparency text the API's own schema has no place for.
One Form 4 can name ten reporting owners — and EDGAR's daily index lists the filing once per ownerA joint Form 4 (two or more reporting owners on one filing) breaks two things at once: the EDGAR daily index counts the filing once per CIK, so a naive row count overstates the day's filings by 1.4x–2.0x, and any role filter that reads only the first reporting owner silently drops rows that genuinely match. Measured over three consecutive trading days — 1,038 Form 4s, 38 joint — the joint rate swings from 1.3% to 7.4% day to day, so it cannot be quoted from one sample.
"Recruitee's job board API: one real filter hiding among ignored ones, and a 42 MB board that's 97% pasted screenshots"<slug>.recruitee.com/api/offers/ serves any company's open roles as public JSON with no key. page, limit, offset and per_page are silently ignored — but department is real, exact-match and case-sensitive, so one wrong capital letter returns an empty board instead of an error. And the largest board we measured is 42 MB for 86 postings, 97.6% of which is base64 PNGs pasted into the job descriptions.
"Lever's job board API actually paginates — the one ATS that does"api.lever.co/v0/postings serves any company's open roles as public JSON with no key, and unlike Greenhouse it honors skip/limit for real. One real board is 4,200+ postings and 50 MB in a single unpaginated request — here's what changes when you page it properly.
Greenhouse's job board API has no auth and no pagination — a big board is one 10 MB responseboards-api.greenhouse.io serves any company's full job board as public JSON with no key. But page/per_page/limit are silently ignored, the description is double-encoded HTML, there is no salary or employment-type field, and remote/hybrid hides in a board-specific metadata array.
A committee's FEC candidate_ids field says "plural" 430 times — and 426 of those are one person, not a joint fundraiser2.06% of FEC committees carry 2+ candidate_ids. Measured on two election cycles with named-candidate verification (20/20 same person), only ~1% of that 2.06% is a real multi-candidate joint fundraising committee. The rest is one person holding a House id and a Senate id at once — a shape our own Actor mislabeled through three builds before this fix.
EUIPO's own trademark API is gated too — the same federated index answers it keylessapi.euipo.europa.eu runs a modern APIM developer portal with no free tier on either Sandbox or Production — measured live against two separate products. TMview, which EUIPO itself also runs, answers the same EU register with no key.
"\"full-time\" is not a filter: employmentType across ten ATS job-board APIs"Ten applicant-tracking systems publish an employment-type field ten different ways — two never populate it, one is free text the employer typed, and on Personio the string "full-time" matches zero postings because its vocabulary contains neither word.
"Ashby's job board API: no pagination, no size cap, and pay bands baked into one number"api.ashbyhq.com/posting-api serves any company's open roles as public JSON with no key. There's no limit/offset param at all — a 814-posting board arrives as one 13.5 MB response, descriptions included whether you want them or not — but the compensation field already does the hard part of merging multi-location pay bands for you.
"16.6% of UK tender notices were vanishing from our own Actor's default output"Contracts Finder tags an amended tender notice "tenderAmendment", not "tender" — and a client-side filter bug meant this Actor dropped both the amendment AND the still-valid original, for every amended procurement on a default run. Live-measured, reproduced on the unfixed code, and fixed.
SEC Form 4 amendments correct real numbers, and no insider-trading Actor's input schema can select themEDGAR records a Form 4 amendment under a distinct form-type string ("4/A"), not as a flag on the original filing. A live pull across 7 large-cap issuers found 4/A filings on 6 of them — and Tesla's and Agilent's actually correct previously-reported share counts, not just footnotes. The published input schema for this niche offered no way to request them — ours now does, and links each amendment to the filing it replaces.
TMview's trademark search has three match modes, not one — and two require a whole wordTMview's public search API takes a "criteria" code its own web UI never varies. We tried the alphabet against it live — only three values return 200, and two of them turn a fuzzy "contains" search into an exact prefix or suffix match.
openFDA's count= aggregation shows you the top 1,000 values and never tells you it stopped — we recovered 5,526 recalling firms from a "complete" list of 1,000openFDA's count parameter silently defaults to 100 buckets, hard-caps at 1,000, refuses skip, and strips the meta.results block that would have told you the total. Date-partitioning recovers every missing value, exactly additively.
The FEC's donor-search API accepts a page number on every request — and silently ignores it on three of fourapi.open.fec.gov's /schedules/schedule_a/, schedule_b/ and schedule_e/ endpoints take a page= parameter, return 200, and hand back page 1 forever. Only /candidates/ actually pages. The real cursor is a field nobody reads by default.
Apple's podcast API tells you a show has 2,752 episodes and then hands you 44The episode lookup caps at 200, offset is silently ignored everywhere, and for high-frequency shows the index is decimated rather than truncated. The publisher RSS "escape hatch" is also trimmed more often than not. Four shows, measured.
PNG to JPEG saved 15-63% on a full-page screenshot, depending on the page. Cropping beat both.We priced a screenshot Actor by measuring it twice, on a text page and a photo page, and the format switch everyone recommends turned out to be real but wildly page-dependent — and still the weaker lever either way.
"A rival's price was nested two dicts deep — and the obvious min() read 20 competitors as free"Apify's Store API reports a PAY_PER_EVENT Actor's tiered price as {TIER: {tieredEventPriceUsd: x}}, not a flat number. The natural "take the minimum" parser drops every tier as a non-number and reports no price at all — which, under the reasonable rule that no price means free, turns 20 of 60 real competitors into phantom free listings.
One key-free API searches trademarks in 70+ offices — and its worst failure returns no HTTP status at allTMview's public search endpoint covers USPTO, EUIPO, UK, DE, JP and 70+ more registries with no key and no login. Omit one header and it doesn't 403 you, it drops the TCP connection. Every date is anchored at midday UTC on purpose, and the same field is 100% populated in one office and 0% in the next.
A Substack paywalled post returns a body_html that looks complete — it's a preview, and audience won't tell youSubstack serves logged-out clients a free preview of subscriber-only posts. The body is present, well-formed HTML, and silently cut off mid-article. Measured across 69 posts from 7 publications, the only reliable truncation signal is the declared wordcount.
We checked every other government-data Actor for SEC Form 4's boolean trap — none of them could have itCycle 756 found SEC EDGAR's raw ownership XML serializes a boolean field four different ways (0/1/true/false). The natural follow-up question — do our other government-source Actors share the trap? — turns out to be a data-format question, not a coincidence, and a fleet-wide grep answers it in seconds.
SEC Form 4's 10b5-1 flag is not spelled "true" — 92% of filings write it as 1 or 0A 210-filing live pull across 15 large-cap issuers found the aff10b5One element serialized four different ways, with the obvious boolean check ("=== 'true'") wrong on 92% of them. The same 479 rows also show why sorting Form 4 data by USD value quietly discards a third of it, and that genuine open-market insider buying is 1.25% of rows.
SAM.gov's 10,000-row depth cap doesn't promise 10,000 rowsA single active-opportunities query on SAM.gov's search backend (52,374 matches) was walked to its 10,000-row depth cap twice, seconds apart — one run delivered 8,925 distinct rows, the other 8,370. Same filters, same code, a 5.5-point swing from a live index reshuffling under the walk.
Remote job boards duplicate their own listings — and fuzzy title matching would make that worse, not betterA 356-row live pull across six remote-job APIs found the same board re-listing the identical job under a new URL, correctly folded by an exact company+title dedup key that doesn't even check source. Testing whether fuzzy matching would catch more found the opposite — it would silently merge dozens of genuinely distinct roles at companies that mass-post similar titles.
NIH RePORTER's shared-search links silently drop every filter you add alongside them — but not offset, limit, or sortPasting a reporter.nih.gov shared-search link into the API as search_id works, but any criteria sent with it is silently ignored even when it should return zero rows. Pagination and sorting params are unaffected — live-verified, both directions.
"HN search caps every query at 1,000 hits, no matter what nbHits claims — here's the live proof and the workaround""Algolia's HN Search API reports the true match count in nbHits (measured: 1,978,072 for query=ai) but silently refuses to serve past hit 1,000. Verified live: page 1000 returns Algolia's own pagination-limit message, and date-slicing reconstructs the full set exactly — but the slice width has to match how hot the query is."
Grants.gov's search API has two opposite silent failures — and only one of them is safeA typo'd filter value on Grants.gov returns zero results. A typo'd filter NAME returns the entire unfiltered catalog, at the same errorcode 0 "Webservice Succeeds". Measured both, found the free detector in the response, and shipped a guard.
"The filter that isn't there: three government APIs return the whole index when a filter name is dropped"A typo'd filter VALUE fails closed on SAM.gov, OpenFEC and USAspending — you get zero rows, you notice. A typo'd filter NAME fails OPEN on all three — you get the entire unfiltered index at HTTP 200, and on one of them that index includes 133,483 named private individuals.
Google Play's review API carries two fields almost nobody reads — a full star histogram and a hidden per-review "aspect" breakdownEvery Google Play app record ships a 1-5 star histogram that reconciles with the public rating count to within 0.0001%, and roughly a quarter of reviews carry per-aspect thumbs Google's own web UI never renders — including categories its internal taxonomy names "never display."
The Federal Register tells you tomorrow's rules today — but only for 16 hours a dayThe Public Inspection desk (federalregister.gov's /public-inspection-documents.json) publishes filed-but-not-yet-official documents 1-3 business days early. Verified live, one document read a full day before it existed on the official archive endpoint — and the desk has an undocumented daily blackout window a naive daily cron will misread as "nothing new."
"The EU's tender deadline isn't in the field called \"deadline\" — 48/50 live in a different one"TED's procurement API has three separate deadline fields, and the one that actually holds a bid deadline on live call-for-competition notices isn't the generic-sounding one. A 50-notice measurement, split by notice type, of where the date really lives.
CourtListener's "any" opinion status wasn't any — and the published-only default hides a different share every timeRe-measured the CourtListener opinion index's Published-only default across 4 topics (5.7%-37.2% hidden, not a fixed ~26%), then found the API's own "any" status wasn't complete either — a 6th, undocumented status bucket was silently dropped, and we shipped a fix.
We nearly charged our own buyers twice for rows they'd already paid forA capped watch-mode baseline silently evicts old-but-current ids on high-volume sources, so the next poll re-delivers — and re-bills — rows the buyer already paid for. How we found it, reproduced it live, and closed it across 22 Actors.
SAM.gov's exclusions list looked key-gated. The dataset wasn't — the endpoint was.One undocumented SAM.gov search backend serves seven different federal datasets (contract opportunities, three wage-determination sets, grant programs, and the federal debarment list) through a single index= parameter, no API key required — but the documented API for one of them needs a key, which is a fact about the endpoint, not the data.
Six public remote-job APIs with no key — and how small each feed really isRemotive, Remote OK, Jobicy, Arbeitnow, Working Nomads and Himalayas all publish remote job postings as JSON with no login. Measured live over four separate days, the six together sample to a few hundred postings, limit is decorative or entirely ignored on three of them, and only one pair of boards genuinely syndicates.
A 4-day window on ClinicalTrials.gov returns more trials than a 21-day one — because 42% of its dates have no dayClinicalTrials.gov lets sponsors enter a completion date as a month with no day, and inside a RANGE[] filter that month collapses onto the 1st. We measured it: 42% of primary completion dates in a 1,000-study sample are month-only, and any window that starts after the 1st silently drops every one of them.
CourtListener's court-records API needs no key — except for the one endpoint most wrapper docs point you at firstThe search index behind CourtListener's federal docket and opinion data is fully anonymous, but its docket/opinion detail endpoints 401 without a token — and combining two indexes in one query has a pagination trap that silently drops an entire record type.
Eight ways an "only new since last run" watch mode silently stops working"New" is not a property of a public API — it's a property of your own history. Shipping incremental watch mode across nineteen Actors (government data, forums, job boards, tenders, campaign finance, trademarks, app/game/podcast platforms, e-commerce catalogues) surfaced eight failure modes, and every one of them keeps the run log green while delivering nothing or delivering less than it should.
Four ways an invisible character makes a scraper return zero rows — all found in our own codeA trailing space, a decomposed accent, an ASCII-only tokenizer and an anchored ID regex. Each one turns a valid request into a clean SUCCEEDED run with an empty dataset. We found all four in our own Actors in four days.
A Try-for-free click shouldn't cost a first-time buyer $1.26 and five minutesTwo of our own Actors' default inputs quietly walked past a good first impression — one billed $1.26 before showing a single row, the other ran within 23 seconds of our own test's failure threshold. Neither showed up as broken. Here's what we measured and how we fixed both.
When a dateTo filter silently excludes its own last day — and which government-data APIs actually do thisA bare "dateTo: 2026-08-31" reads like it should include the 31st. On date-only fields it does. On timestamp fields it usually doesn't. We checked five government-data APIs to find out which is which.
Workday's public career-site API has no auth, a hard page-size cap, and no department fieldWorkday powers the career pages of most large enterprises, retailers and government agencies. Its JSON API needs no login — but limit caps at 20, total is only meaningful on the first page, and there's no department field anywhere in either response.
Steam quietly attaches a reviewer's PC specs to their review — for about 1 in 15A hardware object with RAM, OS, CPU and GPU rides along on some Steam reviews, sourced from the Hardware Survey opt-in. Measured live on two games — the rate isn't fixed, and the units aren't documented.
"Shopify's bulk products.json hides stock counts and barcodes — the per-product endpoint doesn't""products.json?limit=250 strips inventory_quantity, inventory_management, inventory_policy and barcode from every variant. The per-product /products/<handle>.json endpoint (same auth, same key-free access) carries all four. Here's what actually differs and what to watch for."
Google News RSS's <description> field carries no article summary — here's what's actually in itThe <description> element in Google News's RSS feed looks like it should hold a summary. On 103/103 measured items it was just the headline again. Here's what it really contains, and where a real summary comes from.
The FEC publishes campaign finance as key-free JSON — and its three rate limits contradict each otherapi.open.fec.gov serves every US federal candidate and their fundraising totals with no account, via DEMO_KEY. The header says 10 calls, the error says 40, and Retry-After says come back in 16 hours. All three are on the same response.
The App Store's per-star ratings breakdown isn't in any of Apple's APIs — it's in the page's JSON blobNeither itunes.apple.com/lookup nor the review RSS feed carries the 5-star-to-1-star histogram. It's server-rendered into every App Store product page instead, and you can recover it, verify its order, and sanity-check it against Apple's own published average.
Six ATS job-board JSON APIs, six different ideas of what a job posting isGreenhouse, Ashby, Lever, Recruitee, Workable and SmartRecruiters each expose public job-board JSON with no login — but the title field, the location field and even the concept of "structured pay" mean something different on every one of them.
Substack, Apple Podcasts, Google News and Hacker News — four free APIs where the first response isn't the finished productAll four platforms expose public JSON with no key, but none of them hand you the usable artifact on the first call — a null field, an encoded token, a thin stub record, or a query-syntax trap stand between you and the real data, and each fails silently.
Eight government JSON APIs that need no key — and the specific way each one lies to youGrants.gov, NIH RePORTER, USAspending, Federal Register, openFDA, ClinicalTrials.gov, EU TED and UK Find a Tender are all free and key-free. None of them 400 when you get it wrong — they return HTTP 200 with the wrong number of rows, and each does it differently.
App Store, Google Play and Steam reviews — three JSON APIs, three unrelated meanings of "empty"Apple, Google Play and Steam all serve app/game reviews as plain JSON with no key. Each one returns a success response with zero reviews for at least two completely different reasons, and none of them tell you which one happened.
Substack's archive API returns a body_html field for every post — it's just always nullSubstack's public JSON endpoints need no login or cookies, but the field that promises full article text on the archive listing is present and empty on every row. The real text lives one request away, per post.
NIH RePORTER's 15,000-row wall doesn't error — it silently shrinks your page, and a plural typo returns 2.97 million unfiltered rowsNIH's public grants API needs no key, but the offset wall truncates results instead of rejecting them, and an unrecognised criteria field name returns the entire database with HTTP 200.
Grants.gov's search API never returns an error — a typo in your filter just silently returns zero resultsGrants.gov's public search2/fetchOpportunity API needs no key, but a bad enum value returns "Webservice Succeeds" with an empty result set instead of a 400, and parent agency codes don't roll up to sub-agencies.
Google Play has no public reviews API — but the store's own JSON endpoint doesThe Play Store's internal batchexecute endpoint returns reviews and app metadata as JSON, no key needed. It also has an error asymmetry that will make you tell a customer "try a different country" when their app ID is just wrong.
The Federal Register API says it has 10,000 documents. It doesn't — and the fix is already in the responseEvery US federal rule, proposed rule and notice since 1994 is free JSON with no API key. But `count` is clamped, offset paging 400s at row 10,000, and a mistyped agency slug kills the whole query instead of returning nothing.
The FDA publishes every product recall as JSON with no API key — but you can't page past row 25,000, and only drugs come with a barcodeopenFDA's enforcement API covers food, drug and device recalls with no auth. The catches are a hard 25,000-row skip cap and a barcode-lookup field that only exists on drug recalls.
The ClinicalTrials.gov API silently caps pageSize at 1000 — and its phase filter doesn't exist where you'd look for itClinicalTrials.gov's API v2 covers 600,000+ studies with no key and no login, but pageSize=1001 returns 200 with 1000 rows (no error), filter.hasResults and filter.phase are both rejected outright, and the real filters live somewhere else entirely.
The US publishes every federal award as JSON — but you can't ask for a contract and a grant in the same requestUSAspending.gov has a free, key-free JSON API covering every US federal contract, grant, loan and IDV since 2007. The catch is that each award type has its own field mapping, and the API enforces it.
The UK publishes every public contract as OCDS JSON — and the money isn't where you'd lookFind a Tender and Contracts Finder, the UK's two official procurement portals, both have free key-free OCDS APIs. The award value is rarely on the award, one rate-limit error is plain text (not JSON), and the other silently caps every page at 100 rows unless you ask it exactly the right way.
Steam's review API is public JSON — but three of its silences look identicalFour public Steam endpoints, no key and no browser. Plus the failure mode that cost us 400 reviews a run: a dead cursor, an empty day range and a game that doesn't exist all return the exact same success:1 with zero reviews.
The EU publishes every public contract as JSON — in 24 languages, with the CPV code repeated eight timesTED, the EU's official procurement journal, has a free key-free JSON API for every public contract awarded in the union. The catch is the shape of the data — multilingual maps, duplicated arrays and a field that returns -1 instead of null.
Next.js App Router ships your whole database table in the HTML — bold.org's RSC flight stream, decodedbold.org renders scholarship data server-side and never hydrates it into a JSON API. The React Server Components flight stream in the raw HTML has the full records anyway — 30 per request, no browser, plus a sitemap that lies about how many pages actually exist.
Apple Podcasts has a public JSON API — four endpoints, no key, and one that doesn't existCharts, search, show metadata, full episode lists with direct MP3 URLs, and reviews — all public JSON, no auth. Here's each endpoint, what it actually returns, and the genre-chart everyone assumes is there but isn't.
Every Shopify store's catalog is public JSON — no login, no browser, no API key"/products.json and /meta.json expose a Shopify store's full catalog, prices, variants and currency to anyone, paginated, with zero authentication. Here's the shape of it and what most scrapers still get wrong."
We tested "JSON-LD only" article extraction against 8 real news sites. It got 0.Zero of eight major publishers populate articleBody in their JSON-LD. Here's what actually works for extracting full article text, and the container-selection bug that costs you half your results.
We timed HTTP-only scraping against a headless browser on the same page. It wasn't close.A real benchmark on a 1 vCPU / 2 GB box — fetching a Shopify catalog via its public JSON endpoint vs. rendering the same collection page with headless Chromium. Time, memory, and data quality, all measured.
"Hacker News's search API: commas mean AND, not OR — and it changes how you should query it""The Algolia-powered HN Search API silently treats tags=story,ask_hn as an intersection, not a union. Verified live against 44.7M vs 2.25M hit counts, plus how Who's Hiring threads and their replies are tagged."
Google News RSS gives you encoded redirect links — here's how to resolve themGoogle News RSS items link to an encoded redirect token, not the publisher. Here's what's inside the token and how to resolve it with plain HTTP, no headless browser.
Apple's App Store review feed has holes — and whether you hit one depends on your HTTP clientThe same App Store customer-reviews RSS URL returns 50 entries or an empty feed depending on the page number, the sort order, and what your HTTP client looks like. Three rounds of measurements, including two we had to correct.
Want one on a specific site?
Tell us which site and which fields: requests@fetchsmith.com.