SKILL.md

skills/site-safe-fetch/SKILL.md

1. ---
2. name: site-safe-fetch
3. description: >-
4. Fetch pages from meridiandisplay.example without tripping the site's WAF (403 Forbidden) or
5. overloading the production origin. Use for ANY HTTP work against meridiandisplay.example,
6. especially batch or multi-page work: site audits, SEO crawls, category sweeps, spec or
7. production-speed pulls, link checking, price checks, sitemap enumeration, or comparing many
8. product pages. Trigger on "audit the site", "crawl these pages", "check every product in this
9. category", "scrape the storefront", "run an SEO audit on our site", "the site is throwing 403s",
10. "we got blocked", or any time more than one meridiandisplay.example URL is about to be fetched.
11. Also use for a single fetch after a 403.
12. ---
14. # Meridian Display Co.: Safe Site Fetching
16. meridiandisplay.example is a **live production storefront**. There is no CDN in front of it and nothing is cached, so every request is a full server-side render against the same origin box that serves customers. Treat bandwidth as borrowed from the sales floor.
18. Two separate things go wrong, with different fixes:
20. 1. **You get 403'd.** The WAF blocked you on a user-agent signature. Instant, unrelated to volume, trivially avoidable.
21. 2. **You degrade the site for real customers.** You saturated the origin with concurrency. No error comes back to you; pages just get slow for everyone.
23. ## Confirmed infrastructure (measured mid-2026)
25. | Layer | What's there |
26. |---|---|
27. | A record | `meridiandisplay.example` and `www` resolve to a single cloud host (e.g. `203.0.113.40`) |
28. | Edge | **A cloud load balancer** with its own `server` header |
29. | WAF | Rules at the load balancer. This is what returns 403. |
30. | Origin | A web server in front of the storefront app, with a different `server` header |
31. | CDN | **None.** Every request reaches the origin. |
33. `www` 301-redirects to the apex, so always use the apex to save a hop. Origin sends `cache-control: no-store, no-cache, must-revalidate` and sets a fresh `mdsf_session` cookie per request, so reuse one HTTP session with a cookie jar. Normal homepage response: ~0.8s.
35. ## What actually causes the 403
37. **User-agent signature matching at the WAF, not volume.** It fires on request #1 with no warm-up, which is exactly why it gets misdiagnosed as a rate limit.
39. A WAF block looks like: HTTP `403`, the load balancer's `server` header instead of the origin's (a normal 200 carries the origin's), and a short bare "403 Forbidden" HTML body.
41. Tested UA results:
43. - **BLOCKED:** `Scrapy/2.11`, `HeadlessChrome/126.0`
44. - **Allowed:** curl, wget, python-requests, python-httpx, urllib, Go-http-client, node-fetch, axios, libwww-perl, Java, PostmanRuntime, SemrushBot, AhrefsBot, Googlebot, `claude-web`, no UA at all, desktop Chrome
46. Takeaways: the blocklist is narrow and targets known scraper/headless signatures. Standard HTTP client UAs pass today, but that's a thin margin, so send a real desktop browser UA. **`HeadlessChrome` being blocked is the important one:** Playwright, Puppeteer, and headless Chrome all leak that string by default, so browser automation against this site must run headed or override the UA. The blocklist is an editable WAF rule. If a previously-working UA starts 403ing, suspect a rule change before a rate limit.
48. ## Volume behavior
50. | Test | Result |
51. |---|---|
52. | 10 sequential, no delay | all 200, ~0.8s each |
53. | 120 requests @ 8 concurrent | all 200, no degradation |
54. | 300 requests @ 16 concurrent | **origin stalled**, recovered ~30s after stopping |
56. No rate-limit threshold found below ~120 requests. **The real constraint is origin saturation at high concurrency, not a WAF counter.** At 16 concurrent, customers see a hanging site while you see nothing wrong. So the safe rate isn't "just under what the server tolerates," it's "not stealing capacity from the storefront." If the exact WAF rate rule is ever needed, get it from the hosting console, not by probing production.
58. ## Hard rules
60. 1. **Never send a UA containing `Scrapy`, `HeadlessChrome`, or `PhantomJS`.**
61. 2. **Serial by default.** One request at a time. Never exceed 3 concurrent, ever.
62. 3. **Minimum 1.5s delay** between requests, with jitter. 2s+ for large batches.
63. 4. **Cache every response to disk.** Never fetch the same URL twice in one piece of work.
64. 5. **Enumerate from `https://meridiandisplay.example/sitemap.xml`**, don't crawl links.
65. 6. **Respect robots.txt** (list below).
66. 7. **GET only.** Never POST. Never touch `/manage/`, `/checkout/`, `/account/`, `/proof-studio/`, the cart, or any form endpoint.
67. 8. **Three consecutive non-200s = abort and report.** Never "retry harder."
68. 9. **Batches over ~50 URLs: confirm with Matt first**, and prefer running outside 8am-6pm Central.
70. ## Workflow
72. **Step 1: Preflight.** One request to `https://meridiandisplay.example/` with a clean UA. Expect 200 in under ~1s. Anything else, stop and report before starting the batch.
74. **Step 2: Build the URL list** from `sitemap.xml` (flat, not an index; **612 URLs** at last count). Filter it down. A full-site crawl at 2s delay is ~28 minutes and 612 origin hits, so scope narrower unless the whole site is genuinely the ask.
76. **Step 3: Fetch** with the script below.
78. **Step 4: Analyze from the cache, not the network.** If your analysis needs a field you didn't capture, re-parse the cached HTML. Do not re-fetch.
80. **Step 5: Report** the request count and any failures so Matt knows what load was generated.
82. ## The fetcher
84. Write this to `md_fetch.py` and use it rather than improvising a loop. Falls back to urllib if `requests` isn't installed.
86. ```python
87. #!/usr/bin/env python3
88. """Polite fetcher for meridiandisplay.example. Usage:
89. python3 md_fetch.py preflight
90. python3 md_fetch.py sitemap [regex-filter] -> prints URLs
91. python3 md_fetch.py batch urls.txt outdir [delay]
92. """
93. import hashlib, json, os, random, re, sys, time
94. from urllib.parse import urlparse, urlunparse
96. BASE = "https://meridiandisplay.example"
97. # Real desktop Chrome. Never put Scrapy/HeadlessChrome/PhantomJS in here.
98. UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
99. "(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36")
100. HEADERS = {"User-Agent": UA, "Accept-Language": "en-US,en;q=0.9",
101. "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
102. "Connection": "keep-alive", "Upgrade-Insecure-Requests": "1"}
103. BAD_PREFIX = ("/manage","/checkout","/account","/proof-studio","/vendor","/api","/tmp",
104. "/captcha","/export","/feeds","/cart")
105. BAD_FILES = ("debug.log","status.json","mail-preview","quote-estimate.json",
106. "share-image","share-link","sync-inventory")
107. BACKOFF = [30, 90, 300]; MAX_FAILS = 3; TIMEOUT = 30
109. assert not any(t in UA.lower() for t in ("scrapy","headlesschrome","phantomjs"))
111. def norm(u):
112. if u.startswith("/"): u = BASE + u
113. if not u.startswith("http"): u = "https://" + u
114. p = urlparse(u); h = p.netloc.lower()
115. if h.startswith("www."): h = h[4:] # apex avoids a 301 hop
116. return urlunparse(("https", h, p.path or "/", p.params, p.query, ""))
118. def refuse(u):
119. """None if safe to fetch, else a reason string."""
120. p = urlparse(u); path, q = p.path.lower(), p.query.lower()
121. if p.netloc.lower() not in ("meridiandisplay.example","www.meridiandisplay.example"):
122. return f"off-site host {p.netloc}"
123. for x in BAD_PREFIX:
124. if path.startswith(x): return f"forbidden path {x}"
125. for f in BAD_FILES:
126. if path.endswith("/"+f) or path == "/"+f: return f"robots-disallowed {f}"
127. for k in ("sort","ref","srsltid"):
128. if re.search(rf"(^|&){k}=", q): return f"robots-disallowed ?{k}="
129. if "/templates/" in path and any(k in q for k in ("page=","per_page=","offset=")):
130. return "robots-disallowed /templates/ pagination"
131. return None
133. class F:
134. def __init__(s, delay=2.0, cache=None):
135. s.delay, s.cache, s.last, s.fails, s.n = max(delay,1.0), cache, 0.0, 0, 0
136. try:
137. import requests; s.s = requests.Session(); s.s.headers.update(HEADERS)
138. except ImportError: s.s = None
139. if cache: os.makedirs(cache, exist_ok=True)
141. def _raw(s, u):
142. w = s.delay + random.uniform(0,0.4) - (time.time()-s.last)
143. if w > 0: time.sleep(w)
144. s.last = time.time(); s.n += 1
145. if s.s is not None:
146. r = s.s.get(u, timeout=TIMEOUT); return r.status_code, r.text, dict(r.headers)
147. import urllib.request, urllib.error
148. try:
149. with urllib.request.urlopen(
150. urllib.request.Request(u, headers=HEADERS), timeout=TIMEOUT) as r:
151. return r.status, r.read().decode("utf-8","replace"), dict(r.headers)
152. except urllib.error.HTTPError as e:
153. return e.code, e.read().decode("utf-8","replace"), dict(e.headers)
155. def _path(s, u):
156. if not s.cache: return None
157. slug = re.sub(r"[^a-z0-9]+","-", urlparse(u).path.lower()).strip("-") or "index"
158. return os.path.join(s.cache, f"{slug[:60]}__{hashlib.sha1(u.encode()).hexdigest()[:16]}.html")
160. def get(s, u):
161. u = norm(u)
162. if (why := refuse(u)):
163. print(f" SKIP {u} ({why})"); return {"url":u,"status":None,"error":why}
164. cp = s._path(u)
165. if cp and os.path.exists(cp):
166. print(f" CACHE {u}"); return {"url":u,"status":200,"path":cp,"cached":True}
167. st = None
168. for i, wait in enumerate([0]+BACKOFF):
169. if wait: print(f" backing off {wait}s"); time.sleep(wait)
170. try: st, tx, hd = s._raw(u)
171. except Exception as e: print(f" ERR {u} {e}"); st, tx, hd = None, "", {}
172. if st == 200:
173. s.fails = 0
174. if cp: open(cp,"w",encoding="utf-8").write(tx)
175. print(f" 200 {u} ({len(tx)//1024}kb)")
176. return {"url":u,"status":200,"path":cp,"cached":False}
177. if st == 403:
178. print(f" 403 {u} [server: {hd.get('server') or hd.get('Server')}]")
179. if i == 0: print(" WAF block: check the user-agent first, not the rate.")
180. elif st in (429,500,502,503,504): print(f" {st} {u}")
181. elif st is not None:
182. s.fails = 0; print(f" {st} {u}")
183. return {"url":u,"status":st,"error":f"HTTP {st}"}
184. s.fails += 1
185. return {"url":u,"status":st,"error":f"failed after retries (HTTP {st})"}
187. def preflight():
188. f = F(delay=0); t = time.time()
189. try: st, _, hd = f._raw(BASE+"/")
190. except Exception as e: print(f"FAIL: unreachable: {e}"); return 2
191. dt, srv = time.time()-t, hd.get("server") or hd.get("Server")
192. if st == 200:
193. print(f"OK: 200 in {dt:.2f}s, server: {srv}"
194. + ("" if dt < 2 else " (slow vs ~0.8s baseline, site may be loaded)")); return 0
195. if st == 403:
196. print(f"BLOCKED (403), server: {srv}\nCheck the UA first. If clean, wait 10 min "
197. "WITHOUT retrying, then preflight once."); return 1
198. print(f"UNEXPECTED: HTTP {st}, server: {srv}"); return 1
200. def sitemap(pat=None):
201. f = F(delay=1.0); st, tx, _ = f._raw(BASE+"/sitemap.xml")
202. if st != 200: print(f"sitemap HTTP {st}", file=sys.stderr); return 1
203. us = re.findall(r"<loc>\s*(.*?)\s*</loc>", tx)
204. if us and all(u.endswith(".xml") for u in us[:3]): # sitemap index
205. c = []
206. for u in us[:10]:
207. s2, t2, _ = f._raw(u)
208. if s2 == 200: c += re.findall(r"<loc>\s*(.*?)\s*</loc>", t2)
209. us = c
210. if pat: us = [u for u in us if re.search(pat, u, re.I)]
211. us = sorted({norm(u) for u in us if refuse(norm(u)) is None})
212. print("\n".join(us)); print(f"# {len(us)} URLs", file=sys.stderr); return 0
214. def batch(path, out, delay=2.0):
215. urls = sorted({norm(l.strip()) for l in open(path)
216. if l.strip() and not l.startswith("#")})
217. if len(urls) > 50 and os.environ.get("MD_CONFIRMED") != "1":
218. print(f"WARNING: {len(urls)} URLs at {delay}s ≈ {len(urls)*(delay+.7)/60:.0f} min "
219. f"and {len(urls)} hits on the production origin.\nConfirm with Matt, prefer "
220. "outside 8am-6pm Central, then re-run with MD_CONFIRMED=1."); return 1
221. pages = os.path.join(out, "pages"); os.makedirs(pages, exist_ok=True)
222. f = F(delay, pages); res = []
223. print(f"Fetching {len(urls)} URLs, {delay}s delay, serial.\n")
224. for i, u in enumerate(urls, 1):
225. print(f"[{i}/{len(urls)}]", end=" "); res.append(f.get(u))
226. if f.fails >= MAX_FAILS:
227. print(f"\nCIRCUIT BREAKER: {MAX_FAILS} consecutive failures, aborting at "
228. f"{i}/{len(urls)}. Diagnose before retrying."); break
229. ok = sum(1 for r in res if r["status"] == 200)
230. json.dump({"requested":len(urls),"attempted":len(res),"live_requests":f.n,"ok":ok,
231. "failed":[r for r in res if r["status"]!=200],"results":res},
232. open(os.path.join(out,"manifest.json"),"w"), indent=2)
233. print(f"\nDone. {ok}/{len(res)} ok, {f.n} live requests to the origin.\nPages: {pages}")
234. return 0
236. if __name__ == "__main__":
237. c = sys.argv[1] if len(sys.argv) > 1 else "preflight"
238. if c == "preflight": sys.exit(preflight())
239. if c == "sitemap": sys.exit(sitemap(sys.argv[2] if len(sys.argv) > 2 else None))
240. if c == "batch": sys.exit(batch(sys.argv[2], sys.argv[3],
241. float(sys.argv[4]) if len(sys.argv) > 4 else 2.0))
242. print(__doc__); sys.exit(1)
243. ```
245. Typical run:
247. ```bash
248. python3 md_fetch.py preflight
249. python3 md_fetch.py sitemap "feather-flags" > urls.txt
250. python3 md_fetch.py batch urls.txt ./crawl 2.0
251. ```
253. Re-running `batch` skips anything already cached, so an interrupted crawl resumes cheaply.
255. ## If you get 403'd
257. Work through this in order:
259. 1. **Check the user-agent first.** This is the cause the overwhelming majority of the time. A UA advertising Scrapy or HeadlessChrome is blocked on request #1, which makes it look like a rate limit when it isn't. Switch to a real Chrome UA and try once.
260. 2. **If you were using browser automation**, headless mode is the problem: the UA leaks `HeadlessChrome`. Run headed, or override the UA.
261. 3. **If the UA is clean and you're still blocked**, you likely did trip a rate rule. Stop entirely. Wait 10 minutes, then preflight once. Do not poll during the wait, because retrying is what extends the block.
262. 4. **If it persists past 15 minutes**, escalate to Matt. It may be an IP-level rule that needs the hosting console.
264. Never respond to a 403 by rotating user-agents, spoofing headers, or switching IPs to get around it. This is Meridian's own infrastructure, and the correct fix is to slow down or to get the audit tooling allowlisted at the WAF. If Meridian is going to run site audits regularly, the real answer is a WAF allowlist rule for an identifying audit UA (e.g. `MeridianAudit/1.0 (+hello@meridiandisplay.example)`) or source IP. Until that exists, the throttling here is a workaround, not a solution.
266. ## Choosing a fetch method
268. | Situation | Use |
269. |---|---|
270. | Any batch of 2+ pages | `md_fetch.py batch` |
271. | One page, static HTML | `md_fetch.py`'s single fetch, or WebFetch |
272. | Page is JS-rendered and HTML comes back empty | Claude in Chrome (`navigate` + `get_page_text`), **headed, not headless** |
273. | Product pricing / order data | Don't scrape it. Use the Plinth order tools. |
274. | Product specs / turnaround times | Use the `product-lookup` or `production-speed-csv` skills |
276. Claude in Chrome is a real browser session and is fine for a handful of pages, but it's slower and heavier per page. Don't loop it over dozens of URLs.
278. ## robots.txt disallow list
280. **Directories:** `/manage/` `/checkout/` `/account/` `/proof-studio/` `/vendor/` `/api/` `/tmp/` `/captcha/` `/export/` `/feeds/`
282. **Query patterns:** `?sort=` `?ref=` `?srsltid=` and `/templates/` pagination (`page` / `per_page` / `offset` in any order). Those generate near-duplicate heavy renders and are exactly the trap an unrestricted crawler falls into.
284. **Files:** `debug.log` `status.json` `mail-preview` `quote-estimate.json` `share-image` `share-link` `sync-inventory`
286. **Pages:** `/legal/privacy` `/legal/terms`