Guide

Link preview not showing? Your firewall may be blocking the crawler

Q: Does robots.txt block link previews?

It can. X documents that Twitterbot honors robots.txt, so a Disallow that covers Twitterbot or all user agents can stop the card. Other platforms vary. Do not block the paths you want people to share, and make sure your og:image path is allowed too.

Q: How do I test what a crawler gets from my site?

Request the page with a crawler user agent and no cookies, for example curl -sIL -A 'Twitterbot/1.0' followed by your URL, and check the status code and final URL. Then fetch the body and look for og: tags. A 403, 429, 503, a challenge page, or a redirect to /login or a consent page means the crawler never sees your tags. Testing from your own laptop is not the same IP reputation as the platform, so a pass from home is a hint, not proof.

The tags are right. view-source: shows og:title and an absolute og:image. Slack, LinkedIn, or X still unfurl a bare URL. When the HTML is fine, the next suspect is the request: the preview crawler fetches your page from a data center, with a bot user agent and no cookies, and something in front of your site answers it with a challenge, a 403, or a redirect instead of the page.

That is exactly the traffic bot protection is designed to stop, so it is common right after you turn on a firewall, move behind a CDN, add a login wall, or ship a cookie banner that redirects.

What the crawler might be getting instead

Blocker	What the crawler sees
Bot protection or JavaScript challenge	A “checking your browser” page with a 403 or 503 and no Open Graph tags
WAF rule, geo block, or IP reputation filter	A plain 403 or an error page from your CDN
Rate limiting	A 429 — often only when a link is shared widely and many clients unfurl it at once
`robots.txt` Disallow	Nothing: crawlers that honor it (X documents that Twitterbot does) do not fetch the page
Login wall, paywall, or staging password	A redirect to `/login` or a 401, so the card shows your login page title or nothing
Cookie consent or geo redirect	A consent or country-picker page with its own (or no) tags

Check it in a minute

Ask for the page the way a crawler does: bot user agent, no cookies, follow redirects, and look at status codes first.

curl -sIL -A 'Twitterbot/1.0' https://example.com/your-page
curl -sL  -A 'Twitterbot/1.0' https://example.com/your-page | grep -iE 'og:|twitter:'

You want a single 200 (or a short redirect chain ending in 200) on the URL you shared, and real og: tags in the body. A 403, 429, 503, a page that mentions a challenge, or a final URL on /login or a consent page means the crawler never sees your tags. Try other agents too, such as facebookexternalhit/1.1, LinkedInBot/1.0, Slackbot-LinkExpanding 1.0, and Discordbot/2.0, since rules are often written per bot.

Your laptop is not a platform data center, so a clean result from home is a hint, not proof. Your CDN or firewall log is the real answer: filter on those user agents around the time you shared the link and see what status they got. CardScope also fetches from Cloudflare’s edge without cookies: if the URL answers with an error such as a 403, it says so instead of drawing cards, and a successful check names the host it landed on after redirects. It uses its own user agent, so a rule written for one platform’s bot can still behave differently.

Don’t forget the image request

The og:image is a second fetch, often to a different host — an image CDN, a storage bucket, or a media service — with its own hotlink protection, referer checks, expiring signed URLs, or firewall rules. If the page passes and the image fails, you get a text-only card or none at all. Request the image URL with a crawler user agent as well, and keep it on a public, stable URL (see absolute OG image URL and OG image size).

Fixes

Let verified preview bots through — most CDNs and bot-management products have a “verified bots” or “known bots” allowance. Turn it on, or add a skip rule for the paths you share, rather than allowing a user-agent string alone (anyone can fake a user agent).
Relax challenges on shareable pages — keep strict rules on login, checkout, and API routes, and serve marketing pages, blog posts, and the og:image path without an interactive challenge.
Fix robots.txt — don’t Disallow the pages you want shared or the folder your images live in. A blanket Disallow: / left over from staging is a classic.
Give gated pages a public head — if the page needs a login, return a 200 with honest tags for the public summary instead of redirecting every anonymous request to /login. Never put private content in those tags.
Don’t redirect crawlers to consent or country pages — show the cookie banner on the page itself, and avoid geo redirects on URLs you expect to be shared internationally.
Raise rate limits for preview bots on popular links, so a busy Slack workspace or a viral post does not trip a 429.

Keep what you serve crawlers honest: the same page people see, not special content for bots.

After the fix

Platforms cache the failed fetch, so a link shared while the crawler was blocked can stay bare for a while. The clear Open Graph cache guide covers refreshing it. If the status is now 200 but tags are still missing, check whether they are added by JavaScript after load — see JavaScript / SPA previews.

Checklist before you ship

Fetch as a bot — crawler user agent, no cookies, follow redirects; expect 200 and real og: tags.
Read your firewall log for preview-bot user agents and their status codes.
Allow verified bots on shareable pages and the image path; keep strict rules where they matter.
Check robots.txt for leftovers that block pages or images.
Request the og:image too, not just the page.
Verify with CardScope — blocked-request errors, tag table, reconstructed shells, and checklist — then refresh caches on platforms that saw the blocked version.

FAQ

Why does my link preview work in the browser but not in Slack or LinkedIn?

Your browser and the preview crawler get different answers. Crawlers fetch from data-center IPs with bot user agents and no cookies, which is exactly what bot protection, WAF rules, rate limits, and login walls are built to stop. If they get a challenge page, a 403, or a redirect to a login or consent page, that HTML has no Open Graph tags for your page. Platform-specific notes are in Slack vs Discord and LinkedIn vs X.

Does `robots.txt` block link previews?

It can. X documents that Twitterbot honors robots.txt, so a Disallow that covers Twitterbot or all user agents can stop the card. Other platforms vary. Don’t block the paths you want people to share, and make sure your og:image path is allowed too.

How do I test what a crawler gets from my site?

Request the page with a crawler user agent and no cookies (the curl lines above), and check the status code and final URL before looking for tags. A 403, 429, 503, a challenge page, or a redirect to /login or a consent page means the crawler never sees your tags. Testing from your own laptop is not the same IP reputation as the platform, so a pass from home is a hint, not proof.

Can my `og:image` be blocked even when the page is not?

Yes. The image is a second request, often to a CDN, bucket, or image host with its own hotlink protection, referer checks, signed URLs that expire, or firewall rules. If that request fails, platforms show a text-only card or nothing. Open the image URL in a private window and request it with a crawler user agent too.

What does a CardScope check cost?

Link-card previews and the fix checklist are Pro: $19 once via Stripe Checkout. No account. We parse HTML in memory and discard it — we do not store the page.

Check a URL on CardScope

Paste the exact link you shared. CardScope fetches it from the edge without cookies, like a crawler, so you see whether the request was blocked, which tags actually arrive, a reconstructed shell per client, and a fix checklist. Previews and the fix checklist unlock with Pro ($19 once) via Stripe Checkout. No account. We parse HTML in memory and discard it — we do not store the page.

Try a URL