Optimize Model Compute (BigQuery Reservations)

Purpose

Automatically assign Dataform actions to BigQuery slot reservations based on priority and cost optimization strategy. Routes high-priority workloads to reserved slots while using on-demand pricing for low-priority tasks.

When to Use

• Assigning new models/actions to appropriate compute tiers (reserved vs on-demand)
• Rebalancing reservation assignments based on priority changes
• Optimizing costs by moving low-priority workloads to on-demand
• Ensuring critical pipelines get guaranteed compute resources

Configuration File

Reservations are configured in definitions/_reservations.js:

javascript
1const { autoAssignActions } = require("@masthead-data/dataform-package");
2
3const RESERVATION_CONFIG = [
4  {
5    tag: "reservation", // Human-readable identifier
6    reservation: "projects/.../reservations/...", // BigQuery reservation path
7    actions: [
8      // Models assigned to this tier
9      "httparchive.crawl.pages",
10      "httparchive.f1.pages_latest",
11    ],
12  },
13  {
14    tag: "on_demand",
15    reservation: "none", // On-demand pricing
16    actions: ["httparchive.sample_data.pages_10k"],
17  },
18];
19
20autoAssignActions(RESERVATION_CONFIG);

Implementation Steps

Step 1: Source Configuration

TODO: User will provide details on how to determine which models should use reserved vs on-demand compute

Step 2: Update Configuration

1. Open definitions/_reservations.js
2. Add or move actions between reservation tiers:

• Reserved slots (reservation: 'projects/...'): Critical, high-priority, SLA-sensitive workloads
• On-demand (reservation: 'none'): Low-priority, ad-hoc, or experimental workloads

Step 3: Verify Changes

bash
1# Check syntax
2dataform compile
3
4# Validate no duplicate assignments
5grep -r "\.actions" definitions/_reservations.js

Decision Criteria

Key Notes

• Each action should appear in only ONE reservation config
• File starts with _ to ensure it runs first in Dataform queue
• Changes take effect on next Dataform workflow run
• Package automatically handles global assignment (no per-file edits needed)

Package Reference

Using @masthead-data/dataform-package (see package.json)