site-safe-fetch/

Meridian Display Co.: Safe Site Fetching

meridiandisplay.example is a live production storefront. There is no CDN in front of it and nothing is cached, so every request is a full server-side render against the same origin box that serves customers. Treat bandwidth as borrowed from the sales floor.

Two separate things go wrong, with different fixes:

1. You get 403'd. The WAF blocked you on a user-agent signature. Instant, unrelated to volume, trivially avoidable.
2. You degrade the site for real customers. You saturated the origin with concurrency. No error comes back to you; pages just get slow for everyone.

Confirmed infrastructure (measured mid-2026)

www 301-redirects to the apex, so always use the apex to save a hop. Origin sends cache-control: no-store, no-cache, must-revalidate and sets a fresh mdsf_session cookie per request, so reuse one HTTP session with a cookie jar. Normal homepage response: ~0.8s.

What actually causes the 403

User-agent signature matching at the WAF, not volume. It fires on request #1 with no warm-up, which is exactly why it gets misdiagnosed as a rate limit.

A WAF block looks like: HTTP 403, the load balancer's server header instead of the origin's (a normal 200 carries the origin's), and a short bare "403 Forbidden" HTML body.

Tested UA results:

• BLOCKED: Scrapy/2.11, HeadlessChrome/126.0
• Allowed: curl, wget, python-requests, python-httpx, urllib, Go-http-client, node-fetch, axios, libwww-perl, Java, PostmanRuntime, SemrushBot, AhrefsBot, Googlebot, claude-web, no UA at all, desktop Chrome

Takeaways: the blocklist is narrow and targets known scraper/headless signatures. Standard HTTP client UAs pass today, but that's a thin margin, so send a real desktop browser UA. HeadlessChrome being blocked is the important one: Playwright, Puppeteer, and headless Chrome all leak that string by default, so browser automation against this site must run headed or override the UA. The blocklist is an editable WAF rule. If a previously-working UA starts 403ing, suspect a rule change before a rate limit.

Volume behavior

No rate-limit threshold found below ~120 requests. The real constraint is origin saturation at high concurrency, not a WAF counter. At 16 concurrent, customers see a hanging site while you see nothing wrong. So the safe rate isn't "just under what the server tolerates," it's "not stealing capacity from the storefront." If the exact WAF rate rule is ever needed, get it from the hosting console, not by probing production.

Hard rules

1. Never send a UA containing Scrapy, HeadlessChrome, or PhantomJS.
2. Serial by default. One request at a time. Never exceed 3 concurrent, ever.
3. Minimum 1.5s delay between requests, with jitter. 2s+ for large batches.
4. Cache every response to disk. Never fetch the same URL twice in one piece of work.
5. Enumerate from https://meridiandisplay.example/sitemap.xml, don't crawl links.
6. Respect robots.txt (list below).
7. GET only. Never POST. Never touch /manage/, /checkout/, /account/, /proof-studio/, the cart, or any form endpoint.
8. Three consecutive non-200s = abort and report. Never "retry harder."
9. Batches over ~50 URLs: confirm with Matt first, and prefer running outside 8am-6pm Central.

Workflow

Step 1: Preflight. One request to https://meridiandisplay.example/ with a clean UA. Expect 200 in under ~1s. Anything else, stop and report before starting the batch.

Step 2: Build the URL list from sitemap.xml (flat, not an index; 612 URLs at last count). Filter it down. A full-site crawl at 2s delay is ~28 minutes and 612 origin hits, so scope narrower unless the whole site is genuinely the ask.

Step 3: Fetch with the script below.

Step 4: Analyze from the cache, not the network. If your analysis needs a field you didn't capture, re-parse the cached HTML. Do not re-fetch.

Step 5: Report the request count and any failures so Matt knows what load was generated.

The fetcher

Write this to md_fetch.py and use it rather than improvising a loop. Falls back to urllib if requests isn't installed.

#!/usr/bin/env python3
"""Polite fetcher for meridiandisplay.example. Usage:
     python3 md_fetch.py preflight
     python3 md_fetch.py sitemap [regex-filter]      -> prints URLs
     python3 md_fetch.py batch urls.txt outdir [delay]
"""
import hashlib, json, os, random, re, sys, time
from urllib.parse import urlparse, urlunparse

BASE = "https://meridiandisplay.example"
# Real desktop Chrome. Never put Scrapy/HeadlessChrome/PhantomJS in here.
UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")
HEADERS = {"User-Agent": UA, "Accept-Language": "en-US,en;q=0.9",
           "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
           "Connection": "keep-alive", "Upgrade-Insecure-Requests": "1"}
BAD_PREFIX = ("/manage","/checkout","/account","/proof-studio","/vendor","/api","/tmp",
              "/captcha","/export","/feeds","/cart")
BAD_FILES = ("debug.log","status.json","mail-preview","quote-estimate.json",
             "share-image","share-link","sync-inventory")
BACKOFF = [30, 90, 300]; MAX_FAILS = 3; TIMEOUT = 30

assert not any(t in UA.lower() for t in ("scrapy","headlesschrome","phantomjs"))

def norm(u):
    if u.startswith("/"): u = BASE + u
    if not u.startswith("http"): u = "https://" + u
    p = urlparse(u); h = p.netloc.lower()
    if h.startswith("www."): h = h[4:]          # apex avoids a 301 hop
    return urlunparse(("https", h, p.path or "/", p.params, p.query, ""))

def refuse(u):
    """None if safe to fetch, else a reason string."""
    p = urlparse(u); path, q = p.path.lower(), p.query.lower()
    if p.netloc.lower() not in ("meridiandisplay.example","www.meridiandisplay.example"):
        return f"off-site host {p.netloc}"
    for x in BAD_PREFIX:
        if path.startswith(x): return f"forbidden path {x}"
    for f in BAD_FILES:
        if path.endswith("/"+f) or path == "/"+f: return f"robots-disallowed {f}"
    for k in ("sort","ref","srsltid"):
        if re.search(rf"(^|&){k}=", q): return f"robots-disallowed ?{k}="
    if "/templates/" in path and any(k in q for k in ("page=","per_page=","offset=")):
        return "robots-disallowed /templates/ pagination"
    return None

class F:
    def __init__(s, delay=2.0, cache=None):
        s.delay, s.cache, s.last, s.fails, s.n = max(delay,1.0), cache, 0.0, 0, 0
        try:
            import requests; s.s = requests.Session(); s.s.headers.update(HEADERS)
        except ImportError: s.s = None
        if cache: os.makedirs(cache, exist_ok=True)

    def _raw(s, u):
        w = s.delay + random.uniform(0,0.4) - (time.time()-s.last)
        if w > 0: time.sleep(w)
        s.last = time.time(); s.n += 1
        if s.s is not None:
            r = s.s.get(u, timeout=TIMEOUT); return r.status_code, r.text, dict(r.headers)
        import urllib.request, urllib.error
        try:
            with urllib.request.urlopen(
                    urllib.request.Request(u, headers=HEADERS), timeout=TIMEOUT) as r:
                return r.status, r.read().decode("utf-8","replace"), dict(r.headers)
        except urllib.error.HTTPError as e:
            return e.code, e.read().decode("utf-8","replace"), dict(e.headers)

    def _path(s, u):
        if not s.cache: return None
        slug = re.sub(r"[^a-z0-9]+","-", urlparse(u).path.lower()).strip("-") or "index"
        return os.path.join(s.cache, f"{slug[:60]}__{hashlib.sha1(u.encode()).hexdigest()[:16]}.html")

    def get(s, u):
        u = norm(u)
        if (why := refuse(u)):
            print(f"  SKIP  {u}  ({why})"); return {"url":u,"status":None,"error":why}
        cp = s._path(u)
        if cp and os.path.exists(cp):
            print(f"  CACHE {u}"); return {"url":u,"status":200,"path":cp,"cached":True}
        st = None
        for i, wait in enumerate([0]+BACKOFF):
            if wait: print(f"        backing off {wait}s"); time.sleep(wait)
            try: st, tx, hd = s._raw(u)
            except Exception as e: print(f"  ERR   {u}  {e}"); st, tx, hd = None, "", {}
            if st == 200:
                s.fails = 0
                if cp: open(cp,"w",encoding="utf-8").write(tx)
                print(f"  200   {u}  ({len(tx)//1024}kb)")
                return {"url":u,"status":200,"path":cp,"cached":False}
            if st == 403:
                print(f"  403   {u}  [server: {hd.get('server') or hd.get('Server')}]")
                if i == 0: print("        WAF block: check the user-agent first, not the rate.")
            elif st in (429,500,502,503,504): print(f"  {st}   {u}")
            elif st is not None:
                s.fails = 0; print(f"  {st}   {u}")
                return {"url":u,"status":st,"error":f"HTTP {st}"}
        s.fails += 1
        return {"url":u,"status":st,"error":f"failed after retries (HTTP {st})"}

def preflight():
    f = F(delay=0); t = time.time()
    try: st, _, hd = f._raw(BASE+"/")
    except Exception as e: print(f"FAIL: unreachable: {e}"); return 2
    dt, srv = time.time()-t, hd.get("server") or hd.get("Server")
    if st == 200:
        print(f"OK: 200 in {dt:.2f}s, server: {srv}"
              + ("" if dt < 2 else "  (slow vs ~0.8s baseline, site may be loaded)")); return 0
    if st == 403:
        print(f"BLOCKED (403), server: {srv}\nCheck the UA first. If clean, wait 10 min "
              "WITHOUT retrying, then preflight once."); return 1
    print(f"UNEXPECTED: HTTP {st}, server: {srv}"); return 1

def sitemap(pat=None):
    f = F(delay=1.0); st, tx, _ = f._raw(BASE+"/sitemap.xml")
    if st != 200: print(f"sitemap HTTP {st}", file=sys.stderr); return 1
    us = re.findall(r"<loc>\s*(.*?)\s*</loc>", tx)
    if us and all(u.endswith(".xml") for u in us[:3]):       # sitemap index
        c = []
        for u in us[:10]:
            s2, t2, _ = f._raw(u)
            if s2 == 200: c += re.findall(r"<loc>\s*(.*?)\s*</loc>", t2)
        us = c
    if pat: us = [u for u in us if re.search(pat, u, re.I)]
    us = sorted({norm(u) for u in us if refuse(norm(u)) is None})
    print("\n".join(us)); print(f"# {len(us)} URLs", file=sys.stderr); return 0

def batch(path, out, delay=2.0):
    urls = sorted({norm(l.strip()) for l in open(path)
                   if l.strip() and not l.startswith("#")})
    if len(urls) > 50 and os.environ.get("MD_CONFIRMED") != "1":
        print(f"WARNING: {len(urls)} URLs at {delay}s ≈ {len(urls)*(delay+.7)/60:.0f} min "
              f"and {len(urls)} hits on the production origin.\nConfirm with Matt, prefer "
              "outside 8am-6pm Central, then re-run with MD_CONFIRMED=1."); return 1
    pages = os.path.join(out, "pages"); os.makedirs(pages, exist_ok=True)
    f = F(delay, pages); res = []
    print(f"Fetching {len(urls)} URLs, {delay}s delay, serial.\n")
    for i, u in enumerate(urls, 1):
        print(f"[{i}/{len(urls)}]", end=" "); res.append(f.get(u))
        if f.fails >= MAX_FAILS:
            print(f"\nCIRCUIT BREAKER: {MAX_FAILS} consecutive failures, aborting at "
                  f"{i}/{len(urls)}. Diagnose before retrying."); break
    ok = sum(1 for r in res if r["status"] == 200)
    json.dump({"requested":len(urls),"attempted":len(res),"live_requests":f.n,"ok":ok,
               "failed":[r for r in res if r["status"]!=200],"results":res},
              open(os.path.join(out,"manifest.json"),"w"), indent=2)
    print(f"\nDone. {ok}/{len(res)} ok, {f.n} live requests to the origin.\nPages: {pages}")
    return 0

if __name__ == "__main__":
    c = sys.argv[1] if len(sys.argv) > 1 else "preflight"
    if c == "preflight": sys.exit(preflight())
    if c == "sitemap":   sys.exit(sitemap(sys.argv[2] if len(sys.argv) > 2 else None))
    if c == "batch":     sys.exit(batch(sys.argv[2], sys.argv[3],
                                        float(sys.argv[4]) if len(sys.argv) > 4 else 2.0))
    print(__doc__); sys.exit(1)

Typical run:

python3 md_fetch.py preflight
python3 md_fetch.py sitemap "feather-flags" > urls.txt
python3 md_fetch.py batch urls.txt ./crawl 2.0

Re-running batch skips anything already cached, so an interrupted crawl resumes cheaply.

If you get 403'd

Work through this in order:

1. Check the user-agent first. This is the cause the overwhelming majority of the time. A UA advertising Scrapy or HeadlessChrome is blocked on request #1, which makes it look like a rate limit when it isn't. Switch to a real Chrome UA and try once.
2. If you were using browser automation, headless mode is the problem: the UA leaks HeadlessChrome. Run headed, or override the UA.
3. If the UA is clean and you're still blocked, you likely did trip a rate rule. Stop entirely. Wait 10 minutes, then preflight once. Do not poll during the wait, because retrying is what extends the block.
4. If it persists past 15 minutes, escalate to Matt. It may be an IP-level rule that needs the hosting console.

Never respond to a 403 by rotating user-agents, spoofing headers, or switching IPs to get around it. This is Meridian's own infrastructure, and the correct fix is to slow down or to get the audit tooling allowlisted at the WAF. If Meridian is going to run site audits regularly, the real answer is a WAF allowlist rule for an identifying audit UA (e.g. MeridianAudit/1.0 (+hello@meridiandisplay.example)) or source IP. Until that exists, the throttling here is a workaround, not a solution.

Choosing a fetch method

Claude in Chrome is a real browser session and is fine for a handful of pages, but it's slower and heavier per page. Don't loop it over dozens of URLs.

robots.txt disallow list

Directories: /manage/ /checkout/ /account/ /proof-studio/ /vendor/ /api/ /tmp/ /captcha/ /export/ /feeds/

Query patterns: ?sort= ?ref= ?srsltid= and /templates/ pagination (page / per_page / offset in any order). Those generate near-duplicate heavy renders and are exactly the trap an unrestricted crawler falls into.

Files: debug.log status.json mail-preview quote-estimate.json share-image share-link sync-inventory

Pages: /legal/privacy /legal/terms