Analyse Trajectory

Generate trajectory files

bash
1uv run python -c "from varro.playground.trajectory import generate_chat_trajectory; print(generate_chat_trajectory(user_id=USER_ID, chat_id=CHAT_ID))"

Idempotent: turns regenerate only when turn.md is missing or .trajectory_version is outdated.

Trajectory file structure

Output at data/trajectory/{user_id}/{chat_id}/:

chat.md                    # one-line summary per turn: user input, tools, final excerpt
system_instructions.md     # full system prompt given to the agent
tool_instructions.md       # all tools with descriptions and parameter schemas
{turn_idx}/
  turn.md                  # trajectory: User → Steps (Thinking/Actions/Observations) → Final response → Usage
  tool_calls/              # extracted .sql, .py, large .txt results
  images/                  # extracted plots and images

Review process

1. Read chat.md for the overview
2. Read system_instructions.md and tool_instructions.md once to understand what the agent was given
3. For each turn, read turn.md and inspect extracted artifacts in tool_calls/
4. Evaluate each turn against the framework below
5. Write findings to data/trajectory/{user_id}/{chat_id}/findings.md

Evaluation framework

Focus on what system builders can change (instructions, tools, documentation), not on what the model should have known.

Instructions quality

Does the system prompt give the agent precise enough guidance?

• Agent guessing at workflow steps that instructions could have specified
• Agent ignoring instructions that exist (too buried or unclear)
• Missing guidance for a common question pattern
• Ambiguity that caused the agent to pick a suboptimal path

Tool adequacy

Do tools return clear, actionable output that makes the next decision obvious?

• Tool output missing information the agent needed next (row count, available levels, column names)
• Agent calling the same tool repeatedly to get information one call could have returned
• Agent working around a tool limitation using Bash/SQL when a dedicated tool or a small tool change would be cleaner
• Tool descriptions that are misleading or incomplete
• Fuzzy matching returning unhelpful results

Trajectory efficiency

Did the agent take unnecessary steps because of instruction or tool gaps?

• Steps that only exist because prior tool output was incomplete
• Exploratory steps that instructions could have eliminated
• Repeated queries that differ only in filter values the agent was searching for
• Trial-and-error discovery of something documentation could have stated
• NameError on a prior-turn dataframe may indicate shell state was lost (CLI restart, idle eviction) rather than agent misuse — don't count it as a tool error

Relevance

Is the user question within scope for the state statistician?

• Questions the agent shouldn't need to handle (general chat, non-data questions)
• Questions that are borderline — note whether the agent should redirect or attempt

Output format

Write findings to data/trajectory/{user_id}/{chat_id}/findings.md:

markdown
1# Review: Chat {chat_id}
2
3## Summary
4{1-3 sentences: what the user asked, overall assessment of how the system supported the agent}
5
6## Findings
7
8### {short title}
9**Dimension**: {Instructions | Tool | Trajectory | Documentation}
10**Turn**: {turn_idx}, Step {step_idx}
11**Observation**: {What happened — reference actual tool calls and results}
12**Suggestion**: {Concrete change to instructions, tool output, or documentation}
13**Impact**: {Steps saved, or what class of questions this helps}
14
15...
16
17## Verdict
18{The single most impactful improvement from this review}

Guidelines:

• Be concrete. Reference actual step numbers, tool calls, and results.
• Suggest specific changes. "Add row count to Sql tool output" not "improve tool output."
• Estimate impact. "Would save 2-3 steps for geographical queries" is useful.
• One finding per root cause. Group repeated issues across turns.
• Skip clean turns — only note what can be improved.

Agent environment (reference)

The reviewed agent (Rigsstatistikeren) operates in a sandboxed filesystem:

/subjects/{root}/{mid}/{leaf}.md   — subject overviews listing available tables
/fact/{root}/{mid}/{leaf}/{id}.md  — per-table docs: columns, joins, value ranges
/dim/                              — dimension table docs
/dashboard/                        — saved dashboard definitions
/skills/                           — guides for complex tasks (e.g., dashboard creation)

Tools: ColumnValues, Sql, Jupyter, Read, Write, Edit, Bash, UpdateUrl, Snapshot, WebSearch

Typical efficient trajectory for data analysis:

1. Identify subject area → Bash ls
2. Read subject overview → Read
3. Read table docs → Read
4. Check column values → ColumnValues
5. Query data → Sql with df_name
6. Visualize → Jupyter with show
7. Explain → final response