Retail Provider Fix

Diagnose and fix scraping failures for individual dealers in the StakTrakr retail pipeline. All code lives in StakTrakrApi/devops/fly-poller/. All URL config is in providers.json on the api branch.

Step 1 — Live Firecrawl Diagnostic

Run directly inside the Fly.io container against the self-hosted Firecrawl at localhost:3002:

bash
1# Substitute URL from providers.json for the failing vendor/coin
2fly ssh console -a staktrakr -C "curl -s -X POST http://localhost:3002/v1/scrape \
3  -H 'Content-Type: application/json' \
4  -d '{\"url\":\"PROVIDER_URL_HERE\",\"formats\":[\"markdown\"],\"waitFor\":6000,\"onlyMainContent\":false}' \
5  | python3 -c \"import sys,json; d=json.load(sys.stdin); \
6    md=d.get('data',{}).get('markdown',''); \
7    w=d.get('data',{}).get('warning',''); \
8    print('WARNING:',w); print('LEN:',len(md)); print(md[:5000])\""

For home poller, replace fly ssh console -a staktrakr -C with ssh homepoller:

bash
1ssh homepoller "curl -s -X POST http://localhost:3002/v1/scrape ..."

Step 2 — Interpret the Output

Step 3 — Config Map in price-extract.js

All extraction config is in StakTrakrApi/devops/fly-poller/price-extract.js.

URL changes go in providers.json on the api branch — NOT in price-extract.js. Both pollers auto-sync providers.json before every run via curl ... /api/data/retail/providers.json.

Step 4 — Self-Hosted Firecrawl waitFor Not Supported

This is the current state (confirmed Feb 2026): The self-hosted Firecrawl container does not support the waitFor parameter. It returns the static/pre-render HTML immediately.

Consequence: All JS-heavy SPAs (JMBullion, BullionExchanges, Monument Metals) get unrendered HTML from Firecrawl. Monument Metals and HeroBullion recover via the Playwright fallback. JMBullion does NOT recover because fractional_weight detection fires on the pre-render content and exits without triggering Playwright.

Fix options (in order of preference):

1. Upgrade Firecrawl to a version that supports waitFor — check ghcr.io/firecrawl/firecrawl:latest changelog
2. Force Playwright path for specific providers — add them to a PLAYWRIGHT_ONLY_PROVIDERS set that skips Firecrawl entirely
3. Treat fractional_weight as Playwright trigger — same as price === null in the fallback condition

Step 5 — Bot Detection (BullionExchanges Pattern)

When Firecrawl returns a redirect to the site's homepage with only a banner image, bot detection is active.

Diagnosis: url in response metadata differs from sourceURL. e.g., sourceURL = product page, url = bullionexchanges.com.

Current status: BullionExchanges redirects ALL headless browser requests to homepage.

Fix options:

• Playwright with residential proxy chain (HOME_PROXY_URL_2 → Webshare) — verify tinyproxy ConnectPort allows 443
• Add browser stealth headers: real user-agent, Accept-Language, Sec-Fetch-* headers
• If Tailscale exit node is active, ALL traffic routes via home residential IP — verify tailscale status in container before debugging further

Step 6 — Test Single Vendor (Dry Run)

From inside the container or home LXC:

bash
1# Fly.io — SSH in then run
2fly ssh console -a staktrakr
3cd /app
4DATA_DIR=/data/staktrakr-api-export/data \
5FIRECRAWL_BASE_URL=http://localhost:3002 \
6PLAYWRIGHT_LAUNCH=1 \
7COINS=ase PROVIDERS=jmbullion \
8DRY_RUN=1 node price-extract.js
9
10# Filter to multiple vendors
11COINS=ase,age PROVIDERS=jmbullion,bullionexchanges DRY_RUN=1 node price-extract.js

A successful result shows ✓ jmbullion: $XXX.XX (playwright) or ✓ jmbullion: $XXX.XX (firecrawl).

Step 7 — Deploy the Fix

After verifying the fix in dry-run:

bash
1# Fly.io — redeploy container (required for price-extract.js changes)
2cd /Volumes/DATA/GitHub/StakTrakrApi/devops/fly-poller
3fly deploy -a staktrakr
4
5# Home LXC — no deploy needed; run-home.sh reads price-extract.js directly
6# Just commit + pull on the home server:
7# git pull origin main && (restart cron if needed)
8
9# providers.json URL changes — push to api branch
10git checkout api
11# edit data/retail/providers.json
12git add data/retail/providers.json && git commit -m "fix(providers): update VENDOR urls"
13git push origin api
14# Both pollers pick this up automatically on next run (no redeploy needed)

Common Provider Failure Patterns

JMBullion — fractional_weight

Cause: Firecrawl's waitFor not supported → product table renders as "Loading..." → no price → but header skip exposes fractional nav links or spot prices that trigger false fractional_weight detection. Fix: Force Playwright path; verify MARKDOWN_HEADER_SKIP_PATTERNS.jmbullion still matches the header timestamp format.

BullionExchanges — page loaded, no price

Cause: Bot detection redirects to homepage; Firecrawl gets one-line banner. Playwright fallback also redirected. Fix: Check tinyproxy HTTPS CONNECT config; use Playwright with stealth headers.

Monument Metals — out_of_stock — PRE-ORDER

Cause: Coin legitimately on pre-order (common for new-year coins Jan–Mar). Fix: Update providers.json URL to a different in-stock coin, OR wait for inventory to arrive.

SDB — wrong price from "Add on Items" carousel

Cause: sdbullion was in USES_AS_LOW_AS; carousel prices bled through. Fix (applied 2026-02-20): sdbullion removed from USES_AS_LOW_AS; MARKDOWN_CUTOFF_PATTERNS.sdbullion added.

Firecrawl Engine Capabilities (Self-Hosted)

The self-hosted container uses Firecrawl's basic engine (not playwright). Capabilities:

To check current Firecrawl version in container:

bash
1fly ssh console -a staktrakr -C "node -e \"const p=require('/opt/firecrawl/package.json'); console.log(p.version)\""