Tools / Substack Scraper

Substack Scraper

Any Substack publication as structured data: posts with full article text, engagement stats, and comment threads. Custom domains supported.

Give it a Substack handle (astralcodexten), a *.substack.com URL or a custom domain (bigtechnology.com) and get every post as a flat record — including the full article text, which the public archive listing does not contain. Optional comment threads come back flattened with parentCommentId so you can rebuild the tree. Filter by keyword (searchQuery), date range, or free vs. paywalled posts. Reads only what a logged-out visitor sees: no login, no paywall bypass, no headless browser.

Run it

curl -X POST https://fetchsmith.com/api/v1/run/substack-scraper \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"includeBodyText": true, "includeComments": true, "maxCommentsPerPost": 3, "maxPostsPerPublication": 4, "max_results": 20, "publicationUrls": ["astralcodexten", "https://www.bigtechnology.com"], "searchQuery": ""}'

Output fields

type id title subtitle slug url publicationName publicationUrl authors postDate audience isPaid wordCount reactionCount commentCount restackCount tags podcastUrl bodyText bodyTruncated postId parentCommentId author authorHandle body date

Sample output

{
  "authors": [
    "Scott Alexander"
  ],
  "bodyText": "The Story So Far\nMechanistic interpretability is the science of \u201creading an AI\u2019s mind\u201d\u2026",
  "commentCount": 118,
  "id": 207542232,
  "isPaid": false,
  "postDate": "2026-09-08T12:04:21.658Z",
  "publicationName": "Astral Codex Ten",
  "reactionCount": 199,
  "title": "God Help Us, Let\u0027s Try To Learn About Mechanistic Interpretability Techniques",
  "type": "post",
  "url": "https://www.astralcodexten.com/p/god-help-us-lets-try-to-learn-about",
  "wordCount": 4695
}

Only publicly accessible pages are read. Respect the source site's terms when using the data.