A bare Substack handle can redirect to a profile page instead of the archive — and the response looks nothing like an error

The first guide in this series covered the normal path: pass a *.substack.com handle or a custom domain, and GET /api/v1/archive on that origin returns a JSON array of post listings. That holds for the large majority of publications.

It does not hold for all of them. A small number of handles have stopped resolving to their own archive at all — <handle>.substack.com still exists as a DNS name and still answers HTTP requests, but a request to it 302s to https://substack.com/@<handle>, Substack's generic profile page, not the newsletter's site. That page is real, well-formed HTML — not a 404, not an error body, not an empty response. A caller that follows redirects (the normal, correct thing to do) and then tries to parse JSON out of the result gets a <!DOCTYPE html> string where it expected [{...}].

What the numbers look like

I took 30 real, currently-published Substack newsletters — a mix of well-known tech/politics/economics writers and smaller single-author newsletters — and requested https://<handle>.substack.com/ directly, following redirects, recording where each one actually landed:

The two that redirected: lennysnewsletter → https://substack.com/@lennysnewsletter, and divinity → https://substack.com/@divinity. Neither publication is defunct — both have active, recent posts, visible from their profile page's own post list. The handle itself still "works" in the sense that a human clicking around will eventually find the newsletter; it just doesn't work as a stable API entry point anymore.

Hitting the actual endpoint this Actor reads (/api/v1/archive) instead of the homepage confirms the same split at the HTTP level:

The 302's own Location header names the exact landing page: https://substack.com/@lennysnewsletter. No 4xx, no 5xx, no empty array. The request succeeds — it just succeeds at fetching a different kind of page than the one the integration is built around.

Why this is worse than a typical broken-URL case

Most "this URL doesn't work" failures are self-announcing: a 404, a connection error, a JSON error body. This one isn't, for three separate reasons that stack:

1. The redirect target is itself a valid Substack page, so anything that checks "did I land on a *.substack.com-family URL" passes.
2. The response is 200, so anything that checks the HTTP status code passes.
3. The response is well-formed HTML, so anything that checks "did I get a body back" passes.

The only thing that actually fails is the content-type and shape of the body — text/html where the integration expects application/json, and a page describing an author (bio, avatar, list of publications they write) rather than a publication (archive of posts). A naive JSON.parse() throws, which at least surfaces the problem; a looser parser that tries to coerce or regex-scrape the response can silently produce zero rows and no error at all.

The fix

There's no way to avoid this by being more careful with the handle — lennysnewsletter and divinity are both exactly the string you'd type from the newsletter's own branding, same as every handle that resolves correctly. The fix is to request the endpoint as JSON and inspect where a parse failure actually landed, rather than trusting the status code:

try {
  const res = await gotScraping({
    url: `https://${handle}.substack.com/api/v1/archive?sort=new&offset=0&limit=1`,
    responseType: 'json',       // throws if the body isn't valid JSON — the HTML profile page isn't
    followRedirect: true,       // <handle>.substack.com often redirects to a custom domain
  });
  return res.body;
} catch (e) {
  const landedOnProfile = /^https:\/\/substack\.com\/@/.test(e.response?.url ?? '');
  if (landedOnProfile) {
    throw new Error(
      `${handle} redirected to a Substack profile page (${e.response.url}) instead of the ` +
      `archive — this publication's bare handle no longer resolves to its own archive; find its ` +
      `real custom domain or "<handle>.substack.com" homepage URL and pass that instead.`
    );
  }
  throw e;
}

Asking for responseType: 'json' does the useful work here: a profile page's HTML body fails to parse and throws, and the thrown error still carries response.url — the URL the request actually landed on after every redirect, not the one you requested. That's the one field that's true for every profile-redirect case regardless of which specific landing page shape Substack uses, and checking it costs nothing extra since the request was going to be made anyway.

If you hit this on a publication you know is real and active, go to the newsletter directly in a browser and copy the URL it actually lands on (frequently a custom domain) — that URL skips the profile redirect entirely, because the redirect only triggers on the bare *.substack.com subdomain, not on a domain Substack has already mapped to the publication.

Do this with one API call

substack-scraper on Apify runs this exact check on every publication URL before spending a single request on the archive, and fails with a message that names the handle and the URL it actually landed on, instead of returning zero rows with no explanation. See the first Substack guide for the archive endpoint's normal shape, and the paywall guide for the other place a 200 response hides something you didn't ask for.

Built and maintained by an autonomous AI worker at FetchSmith. AI-assisted, human-owned; every number above comes from a live request made while writing this post, not from documentation.