debug-stuck-eval — how to use debug-stuck-eval how to use debug-stuck-eval, debug-stuck-eval setup guide, debug-stuck-eval alternative, debug-stuck-eval vs hawk, what is debug-stuck-eval, debug-stuck-eval install, debug-stuck-eval troubleshooting, UK AISI Inspect in the Cloud, AI evaluation debugging, hawk auth and status commands

Quick Checklist

1. Verify auth: hawk auth access-token > /dev/null || echo "Run 'hawk login' first"
2. Get eval-set-id from user
3. Check status: hawk status <eval-set-id> - JSON report with pod state, logs, metrics
4. View logs: hawk logs <eval-set-id> or hawk logs -f for follow mode
5. List samples: hawk list samples <eval-set-id> - see completion status
6. Look for error patterns (see below)
7. Test API directly if logs show retries without clear errors

Error Patterns

Key Techniques

1. Retry messages have sample context - All retry messages include a [sample_uuid task/sample_id/epoch model] prefix. Inspect's own retries also include a compact error summary suffix like [RateLimitError 429 rate_limit_exceeded]. The OpenAI SDK's internal retry messages still don't show the actual error — use curl for full details.
2. FAIL-OK patterns are fine - Alternating failures and successes mean the eval IS progressing. Only worry about consistent FAIL-FAIL-FAIL patterns.
3. Use S3 for buffer access - Download .buffer/ from S3 rather than accessing the runner pod directly.
4. Read .eval files with inspect_ai - Use from inspect_ai.log import read_eval_log instead of manually extracting zips.

Test API Directly

Middleman is the auth proxy. If middleman fails but direct provider calls work, it's a middleman issue.

bash
1TOKEN=$(hawk auth access-token)
2
3# Test through middleman
4curl --max-time 300 -X POST https://middleman.internal.metr.org/anthropic/v1/messages \
5  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
6  -d '{"model": "claude-sonnet-4-20250514", "max_tokens": 100, "messages": [{"role": "user", "content": "Say hello"}]}'
7
8# Test OpenAI-compatible
9curl --max-time 300 -X POST https://middleman.internal.metr.org/openai/v1/chat/completions \
10  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
11  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Say hello"}], "max_tokens": 100}'

Recovery

bash
1# Delete stuck eval and restart
2hawk delete <eval-set-id>
3hawk eval-set <config.yaml>

The sample buffer in S3 allows Inspect to resume from where it left off (unless you use --no-resume).

HTTP Retry Count

Task progress logs include "HTTP retries: X". High retry counts indicate API instability even while tasks complete.

Severity: Retry count × wait time = stuck duration. E.g., 45 retries × 1800s = 22+ hours stuck.

More Details

See docs/debugging-stuck-evals.md for:

• Sample buffer SQL queries
• Detailed API testing examples
• Escalation checklist

References

• Inspect AI Model Providers - Model configuration
• Inspect AI Eval Logs - .eval file format

Filing Issues

• Middleman: https://github.com/metr-middleman/middleman-server/issues
• Hawk: Linear issue on Evals Execution team
• Inspect AI: https://github.com/UKGovernmentBEIS/inspect_ai/issues