Tools / Hacker News (HN) API Scraper – Who Is Hiring, Tech News API

Hacker News (HN) API Scraper – Who Is Hiring, Tech News API

Search or browse Hacker News stories, comments, Ask HN, Show HN and Who's Hiring threads via the official Algolia API, with points, date and exclude-keyword filters. Watch mode returns only what's new since your last run, and can re-alert when a story crosses a points or comment-count milestone. Optional GitHub enrichment adds stars, language, last-push date and a computed freshness/maturity classification for any linked repo.

Uses HN's official Algolia Search API, so results are fast and reliable. Filter by tag (story, comment, ask_hn, show_hn, job), minimum points, date range, and excludeKeywords to drop off-topic matches HN's search can't exclude server-side. Leave queries empty and sort by date to pull the newest items for a tag, e.g. the latest monthly Who's Hiring thread. Optional enrichGithubLinks adds star count, language and last-push date for any linked GitHub repo. Watch mode (watchLabel) returns only hits it has never delivered under that label before; add watchChanges to also be re-alerted when an already-seen story crosses a points or comment-count milestone (100, 500, ... — tunable), which a plain id-based watch can never tell you.

Run it

curl -X POST https://fetchsmith.com/api/v1/run/hacker-news-scraper \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"maxItemsPerQuery": 50, "max_results": 50, "queries": ["apify"], "sortBy": "relevance", "tags": ["story"]}'

Output fields

id type title url hnUrl author points numComments engagementScore storyId storyTitle storyUrl parentId commentDepth text createdAt query karma about accountCreatedAt submissionCount authorInfluenceScore githubRepo githubStars githubLanguage githubPushedAt githubOpenIssues githubCreatedAt githubArchived githubFreshness githubMaturity _watchChangeType _watchPrevious _watchMilestone

Sample output

{
  "author": "jancurn",
  "createdAt": "2017-10-26T13:03:29Z",
  "engagementScore": 55.5,
  "id": "15558405",
  "numComments": 21,
  "points": 45,
  "title": "Show HN: Apify \u2013 Turn any website into an API",
  "type": "story",
  "url": "https://www.apify.com/"
}

Guides

A Hacker News comment has no URL of its own — so GitHub links get attributed to the wrong repo — HN's search API attaches the parent story's URL to every comment record and gives the comment no url field at all. Measured live on 500 comment hits: where both the story and the comment link a GitHub repo, they are different repos 30 times out of 33.
"HN search caps every query at 1,000 hits, no matter what nbHits claims — here's the live proof and the workaround" — "Algolia's HN Search API reports the true match count in nbHits (measured: 1,978,072 for query=ai) but silently refuses to serve past hit 1,000. Verified live: page 1000 returns Algolia's own pagination-limit message, and date-slicing reconstructs the full set exactly — but the slice width has to match how hot the query is."
"Hacker News's search API: commas mean AND, not OR — and it changes how you should query it" — "The Algolia-powered HN Search API silently treats tags=story,ask_hn as an intersection, not a union. Verified live against 44.7M vs 2.25M hit counts, plus how Who's Hiring threads and their replies are tagged."

Only publicly accessible pages are read. Respect the source site's terms when using the data.