eval — for Claude Code thoughtbox, community, for Claude Code, ide skills, evaluation harness, AI decision metrics, Thoughtbox intention ledger, sequential thinking, observability, reasoning, Claude Code

Evaluation harness: $ARGUMENTS

Commands

Parse the first word of $ARGUMENTS to determine the command:

metrics — Show current session metrics

Collect and display metrics for the current session:

1. Count commits: git log --oneline --since="today" | wc -l
2. Count test results: check for recent vitest output or .eval/metrics/ entries
3. Count beads changes: bd list --status=closed recently
4. Token usage: check LangSmith state file if available
5. Pattern usage: check .dgm/fitness.json for patterns used this session
6. Session duration: check session start time from logs

Display as:

## Current Session Metrics

| Metric | Value | Baseline | Delta |
|--------|-------|----------|-------|
| Commits | 5 | 3.2 avg | +56% |
| Tests passing | 42/42 | 40/42 | +2 |
| Beads closed | 3 | 2.1 avg | +43% |
| Files changed | 12 | 8.5 avg | +41% |
| Patterns used | 7 | 5.3 avg | +32% |

baseline — Set or update baselines

1. Read the last N session metric snapshots from .eval/metrics/
2. Calculate averages for each metric
3. Write to .eval/baselines.json
4. Report what changed

compare — Compare sessions

Usage: compare --last N or compare --session <id>

1. Load metric snapshots from .eval/metrics/
2. Compare against baselines
3. Highlight regressions (metric dropped >10% below baseline)
4. Highlight improvements (metric improved >10% above baseline)

report — Generate weekly evaluation report

1. Load all metrics from the past 7 days
2. Calculate trends (improving, stable, declining)
3. Identify top improvements and top regressions
4. Generate recommendations based on trends

capture — Capture current session metrics

Write a metric snapshot to .eval/metrics/session-{timestamp}.json:

json
1{
2  "session_id": "<session id>",
3  "timestamp": "<ISO 8601>",
4  "branch": "<git branch>",
5  "metrics": {
6    "commits": 0,
7    "tests_total": 0,
8    "tests_passing": 0,
9    "beads_closed": 0,
10    "beads_created": 0,
11    "files_changed": 0,
12    "patterns_referenced": 0,
13    "assumptions_verified": 0,
14    "escalations": 0,
15    "spiral_detections": 0
16  },
17  "qualitative": {
18    "session_focus": "<what the session was about>",
19    "memory_usefulness": 0,
20    "knowledge_gaps_found": []
21  }
22}

Notes

• If .eval/baselines.json doesn't exist, skip baseline comparisons and suggest running baseline
• Metric collection should be best-effort — missing data is noted, not an error
• Regressions trigger a structured escalation suggestion (not automatic action)