Tools / Substack Scraper – Posts, Comments & Substack Newsletter API

Substack Scraper – Posts, Comments & Substack Newsletter API

Any Substack publication as structured data: posts with full article text, engagement stats (reactions, comments, restacks), podcast episodes with duration, section and cover image, plus threaded comments. Optional publication profile — subscriber count, subscription plan prices, author and bestseller tier. Custom domains supported.

Give it a Substack handle (astralcodexten), a *.substack.com URL or a custom domain (bigtechnology.com) and get every post as a flat record — including the full article text, which the public archive listing does not contain. Optional comment threads come back flattened with parentCommentId so you can rebuild the tree. Filter by keyword (searchQuery), date range, or free vs. paywalled posts. Reads only what a logged-out visitor sees: no login, no paywall bypass, no headless browser.

Run it

curl -X POST https://fetchsmith.com/api/v1/run/substack-scraper \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"includeBodyText": true, "includeComments": true, "includePublicationInfo": true, "maxCommentsPerPost": 3, "maxPostsPerPublication": 4, "max_results": 20, "publicationUrls": ["astralcodexten", "https://www.bigtechnology.com"], "searchQuery": ""}'

Output fields

type id title subtitle slug url publicationUrl rank category leaderboardTier name handle customDomain publicationId publicationName authors postDate audience isPaid postType isRestack wordCount reactionCount commentCount restackCount description coverImage section tags podcastUrl podcastDurationSec language bodyText bodyHtml bodyWordCount bodyTruncated commentsRetrieved commentsWithheld publicationSubscriberCount publicationSubscriberCountLabel publicationPaidSubscribersLabel publicationBestsellerTier publicationAuthorName publicationAuthorHandle publicationAuthorBio publicationDescription publicationType publicationLanguage publicationFirstPostDate publicationHasPodcast publicationInviteOnly publicationPaymentsEnabled publicationPlans publicationLogoUrl postId postTitle postUrl parentCommentId author authorHandle body date replyCount isDeleted

Sample output

{
  "authors": [
    "Scott Alexander"
  ],
  "bodyText": "The Story So Far\nMechanistic interpretability is the science of \u201creading an AI\u2019s mind\u201d\u2026",
  "bodyTruncated": false,
  "bodyWordCount": 4695,
  "commentCount": 118,
  "id": 207542232,
  "isPaid": false,
  "postDate": "2026-09-08T12:04:21.658Z",
  "publicationName": "Astral Codex Ten",
  "reactionCount": 199,
  "title": "God Help Us, Let\u0027s Try To Learn About Mechanistic Interpretability Techniques",
  "type": "post",
  "url": "https://www.astralcodexten.com/p/god-help-us-lets-try-to-learn-about",
  "wordCount": 4695
}

Guides

A bare Substack handle can redirect to a profile page instead of the archive — and the response looks nothing like an error — A small slice of <handle>.substack.com URLs 302 to a substack.com/@handle profile page instead of the newsletter's own archive. No 404, no error JSON — just a 200 HTML page where you expected a JSON array. Measured live across 30 real newsletters.
A Substack paywalled post returns a body_html that looks complete — it's a preview, and audience won't tell you — Substack serves logged-out clients a free preview of subscriber-only posts. The body is present, well-formed HTML, and silently cut off mid-article. Measured across 69 posts from 7 publications, the only reliable truncation signal is the declared wordcount.
Substack's archive API returns a body_html field for every post — it's just always null — Substack's public JSON endpoints need no login or cookies, but the field that promises full article text on the archive listing is present and empty on every row. The real text lives one request away, per post.

Only publicly accessible pages are read. Respect the source site's terms when using the data.