TraceBench · Synthetic recorded-output demo

TraceBench / evaluation report

Offline deterministic checks, exact known costs, and supplied latency observations. Every result is available for inspection.

Suite fingerprint · SHA-256a99313d2d77360598bd69ac001f84a4562ac4dafd79ef199930fbe72a8f94733

Checks passed

4 / 6

2 failed · All checks must pass for a case to pass.

Supplied latency

150 ms mean

p50 140 ms · p95 210 ms
Min 100 ms · Max 210 ms

Coverage 6 / 6 · Missing 0.
Percentiles use nearest-rank.

Known token cost

0.0000327 USD

Priced cases 6 / 6.
Missing token observations 0 / 6.

Baseline comparison

Regressions 2Improvements 0Unchanged 4

Baseline passed 6 / 6 · Current passed 4 / 6. Comparison is by case ID.

Costs use only supplied prices and token observations. Partial coverage is not a total bill. Recorded latency is supplied by the caller; TraceBench does not time a provider or verify provenance. Deterministic checks do not establish semantic quality or factual correctness.

Case inspection

6 / 6 cases visible

Case inputs and expected values are not retained in this report schema. Recorded outputs are full JSON values; check reasons are JSON strings. Quotes and escapes preserve all text exactly.

en-greeting

FAILRegression

Latency: 100 ms · Tokens: 10 in / 4 out · Known cost: 0.0000022 USD

Recorded output · full JSON

"hello, world!"

Individual checks · 0 / 1 passed

exact · FAIL

"exact string differs (case and whitespace are significant)"

zh-greeting

PASS

Latency: 120 ms · Tokens: 12 in / 6 out · Known cost: 0.000003 USD

Recorded output · full JSON

"你好，世界。"

Individual checks · 1 / 1 passed

exact · PASS
```
"exact match"
```

en-summary

PASS

Latency: 180 ms · Tokens: 40 in / 14 out · Known cost: 0.0000082 USD

Recorded output · full JSON

"A community can invest in renewable energy."

Individual checks · 1 / 1 passed

contains · PASS
```
"substring found"
```

zh-location

FAILRegression

Latency: 140 ms · Tokens: 20 in / 9 out · Known cost: 0.0000047 USD

Recorded output · full JSON

"上海今天晴朗。"

Individual checks · 0 / 1 passed

contains · FAIL

"required literal substring not found in string output"

en-structured

PASS

Latency: 210 ms · Tokens: 30 in / 12 out · Known cost: 0.0000066 USD

Recorded output · full JSON

{
  "status": "ready",
  "count": 2
}

Individual checks · 1 / 1 passed

json_subset · PASS
```
"JSON subset matches"
```

zh-structured

PASS

Latency: 150 ms · Tokens: 35 in / 15 out · Known cost: 0.000008 USD

Recorded output · full JSON

{
  "城市": "上海",
  "项目": [
    "教育",
    "社区"
  ],
  "已验证": true
}

Individual checks · 1 / 1 passed

json_subset · PASS
```
"JSON subset matches"
```