Quality Metrics
<default_to_action> When measuring quality or building dashboards:
- MEASURE outcomes (bug escape rate, MTTD) not activities (test count)
- FOCUS on DORA metrics: Deployment frequency, Lead time, MTTD, MTTR, Change failure rate
- AVOID vanity metrics: 100% coverage means nothing if tests don't catch bugs
- SET thresholds that drive behavior (quality gates block bad code)
- TREND over time: Direction matters more than absolute numbers
Quick Metric Selection:
- Speed: Deployment frequency, lead time for changes
- Stability: Change failure rate, MTTR
- Quality: Bug escape rate, defect density, test effectiveness
- Process: Code review time, flaky test rate
Critical Success Factors:
- Metrics without action are theater
- What you measure is what you optimize
- Trends matter more than snapshots </default_to_action>
Quick Reference Card
When to Use
- Building quality dashboards
- Defining quality gates
- Evaluating testing effectiveness
- Justifying quality investments
Meaningful vs Vanity Metrics
| ✅ Meaningful | ❌ Vanity |
|---|---|
| Bug escape rate | Test case count |
| MTTD (detection) | Lines of test code |
| MTTR (recovery) | Test executions |
| Change failure rate | Coverage % (alone) |
| Lead time for changes | Requirements traced |
DORA Metrics
| Metric | Elite | High | Medium | Low |
|---|---|---|---|---|
| Deploy Frequency | On-demand | Weekly | Monthly | Yearly |
| Lead Time | < 1 hour | < 1 week | < 1 month | > 6 months |
| Change Failure Rate | < 5% | < 15% | < 30% | > 45% |
| MTTR | < 1 hour | < 1 day | < 1 week | > 1 month |
Quality Gate Thresholds
| Metric | Blocking Threshold | Warning |
|---|---|---|
| Test pass rate | 100% | - |
| Critical coverage | > 80% | > 70% |
| Security critical | 0 | - |
| Performance p95 | < 200ms | < 500ms |
| Flaky tests | < 2% | < 5% |
Core Metrics
Bug Escape Rate
Bug Escape Rate = (Production Bugs / Total Bugs Found) × 100
Target: < 10% (90% caught before production)
Test Effectiveness
Test Effectiveness = (Bugs Found by Tests / Total Bugs) × 100
Target: > 70%
Defect Density
Defect Density = Defects / KLOC
Good: < 1 defect per KLOC
Mean Time to Detect (MTTD)
MTTD = Time(Bug Reported) - Time(Bug Introduced)
Target: < 1 day for critical, < 1 week for others
Dashboard Design
typescript1// Agent generates quality dashboard 2await Task("Generate Dashboard", { 3 metrics: { 4 delivery: ['deployment-frequency', 'lead-time', 'change-failure-rate'], 5 quality: ['bug-escape-rate', 'test-effectiveness', 'defect-density'], 6 stability: ['mttd', 'mttr', 'availability'], 7 process: ['code-review-time', 'flaky-test-rate', 'coverage-trend'] 8 }, 9 visualization: 'grafana', 10 alerts: { 11 critical: { bug_escape_rate: '>20%', mttr: '>24h' }, 12 warning: { coverage: '<70%', flaky_rate: '>5%' } 13 } 14}, "qe-quality-analyzer");
Quality Gate Configuration
json1{ 2 "qualityGates": { 3 "commit": { 4 "coverage": { "min": 80, "blocking": true }, 5 "lint": { "errors": 0, "blocking": true } 6 }, 7 "pr": { 8 "tests": { "pass": "100%", "blocking": true }, 9 "security": { "critical": 0, "blocking": true }, 10 "coverage_delta": { "min": 0, "blocking": false } 11 }, 12 "release": { 13 "e2e": { "pass": "100%", "blocking": true }, 14 "performance_p95": { "max_ms": 200, "blocking": true }, 15 "bug_escape_rate": { "max": "10%", "blocking": false } 16 } 17 } 18}
Agent-Assisted Metrics
typescript1// Calculate quality trends 2await Task("Quality Trend Analysis", { 3 timeframe: '90d', 4 metrics: ['bug-escape-rate', 'mttd', 'test-effectiveness'], 5 compare: 'previous-90d', 6 predictNext: '30d' 7}, "qe-quality-analyzer"); 8 9// Evaluate quality gate 10await Task("Quality Gate Evaluation", { 11 buildId: 'build-123', 12 environment: 'staging', 13 metrics: currentMetrics, 14 policy: qualityPolicy 15}, "qe-quality-gate");
Agent Coordination Hints
Memory Namespace
aqe/quality-metrics/
├── dashboards/* - Dashboard configurations
├── trends/* - Historical metric data
├── gates/* - Gate evaluation results
└── alerts/* - Triggered alerts
Fleet Coordination
typescript1const metricsFleet = await FleetManager.coordinate({ 2 strategy: 'quality-metrics', 3 agents: [ 4 'qe-quality-analyzer', // Trend analysis 5 'qe-test-executor', // Test metrics 6 'qe-coverage-analyzer', // Coverage data 7 'qe-production-intelligence', // Production metrics 8 'qe-quality-gate' // Gate decisions 9 ], 10 topology: 'mesh' 11});
Common Traps
| Trap | Problem | Solution |
|---|---|---|
| Coverage worship | 100% coverage, bugs still escape | Measure bug escape rate instead |
| Test count focus | Many tests, slow feedback | Measure execution time |
| Activity metrics | Busy work, no outcomes | Measure outcomes (MTTD, MTTR) |
| Point-in-time | Snapshot without context | Track trends over time |
Related Skills
- agentic-quality-engineering - Agent coordination
- cicd-pipeline-qe-orchestrator - Quality gates
- risk-based-testing - Risk-informed metrics
- shift-right-testing - Production metrics
Remember
Measure outcomes, not activities. Bug escape rate > test count. MTTD/MTTR > coverage %. Trends > snapshots. Set gates that block bad code. What you measure is what you optimize.
With Agents: Agents track metrics automatically, analyze trends, trigger alerts, and make gate decisions. Use agents to maintain continuous quality visibility.
