Picking a Frontier Model: Opus 4.7 vs GPT-5.5 vs Gemini 3.1 Pro — A Builder's Benchmark Guide

> Prerequisites: None — this is the entry point for the course. > > Time: 40 minutes > > Learning objectives: By the end of this chapter, you can name the 7 evaluation dimensions that reliably predict production success, identify 3 popular benchmarks that don't, and fill in a scorecard for your specific use case.

Frontier model evaluation is the practice of measuring AI model capabilities along structured axes to predict production performance, rather than performance on standardized academic tests. As of Q2 2026, three models dominate serious production AI workloads: Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro. This chapter gives you the conceptual scaffolding to decide which benchmark dimensions you actually need to measure for your workload before you run a single API call.

Key facts

1. MMLU, HumanEval, and GPQA — the three benchmarks most commonly cited in model release notes — measure knowledge recall, single-function code generation, and graduate-level science respectively. None directly measures tool-use consistency, structured-output stability, or mid-context retrieval accuracy. [1][2]
2. As of Q2 2026, no major public benchmark measures tool-use determinism — the probability that the same prompt produces structurally equivalent output across independent runs. Our internal dataset (/data/claude-tool-use-determinism/2026-Q2/) fills this gap for the three models covered in this course.
3. Opus 4.7's published context window is 1M tokens [7]; Gemini 3.1 Pro's model card documents text, image, audio, and video inputs with a context window up to 1M tokens and text output up to 64K tokens [8][13]. Empirically measured retrieval accuracy at 80% of each model's advertised limit tells a different story — covered in Chapter 3.
4. Prompt caching is available on all three platforms but with meaningfully different economics: Anthropic caches at 4,096+ token boundaries for current flagship models (e.g., Claude Opus 4.7; the minimum drops to 1,024 tokens for older models such as Sonnet 4.5 and Opus 4.1) with a 5-minute TTL [9], OpenAI caches at 1,024+ token boundaries in 128-token cache-hit increments [10], and Google Cloud caches Gemini context at configurable TTL (default: 1 hour) [11]. The cost implications for agentic workloads are non-trivial — Chapter 4 quantifies them.
5. The term "capability overhang" refers to the gap between what a model can do in a best-case scenario and what it does reliably across the distribution of real inputs. Frontier models exhibit significant capability overhang on production workloads. A model that scores 95% on a coding benchmark may succeed on only 70% of your specific code-generation prompts.
6. In our 10×3×5 benchmark (10 prompts × 3 models × 5 runs), the variance in output structure at temperature=0 ranged from 2% to 18% across the three models — variance that leaderboard scores do not capture and that compounds multiplicatively in multi-step pipelines. [3]
7. The "production gap" — the documented delta between academic benchmark performance and real-task performance — is most pronounced in function-calling tasks, where benchmark scores and real-world tool-orchestration reliability can diverge substantially. Aggregate benchmarks such as MMLU do not measure multi-step tool use; the Berkeley Function Calling Leaderboard (BFCL) is the most widely-cited public evaluation for this dimension, tracking real-world function-calling accuracy across leading models. [4]

- MMLU, HumanEval, and GPQA measure knowledge recall, single-function code generation, and graduate reasoning — none directly measures tool-use determinism, structured-output stability, or mid-context retrieval accuracy.
- The "production gap" is most pronounced in function-calling tasks; benchmark scores and real-world tool-orchestration reliability can diverge substantially.
- As of Q2 2026, no major public benchmark measures tool-use determinism — the probability that the same prompt produces structurally equivalent output across independent runs.

Why the standard benchmarks fail builders

Every model release in 2026 ships with a table comparing MMLU, HumanEval, GPQA, and MATH scores. These benchmarks are not fraudulent — they measure real things. But they measure things that matter for research progress, not for shipping a reliable product.

Consider MMLU (Massive Multitask Language Understanding). It evaluates knowledge recall across 57 academic subjects via multiple-choice questions. A model that achieves 92% on MMLU has broad factual recall. But your coding agent, document summarizer, or customer-support bot does not answer multiple-choice questions about high-school biology. It calls tools with structured JSON schemas, retrieves facts from documents you provide, and produces outputs that downstream code must parse. None of those capabilities are measured by MMLU. [1]

HumanEval is more practically relevant — it measures code generation on isolated function-completion tasks. But it measures single-function correctness, not the kind of multi-step, tool-integrated code generation that represents the real workload of a coding agent. A model can score 90% on HumanEval and still routinely produce subtly malformed JSON schemas that break your function-calling pipeline. The benchmark is not wrong; it is just not measuring your problem. [5]

The third major benchmark, GPQA Diamond (Graduate-Level Google-Proof Q&A), measures PhD-level reasoning in science. It is an excellent proxy for raw reasoning depth. It is a poor proxy for whether a model will reliably return a consistently structured response to the same tool-call prompt across five independent runs.

This is not a criticism of the research community. These benchmarks serve their purpose: driving reproducible comparisons between models on controlled tasks. The problem is that builders use them as a proxy for production fitness, and the correlation is weaker than it appears.

The model will give a candid self-assessment with some caveats. Notice: it cannot give you actual p95 latency figures (it has no access to runtime metrics), and its self-assessment of determinism will be approximate rather than empirically grounded. This illustrates why self-reported benchmarks — whether from the model or from the vendor — are not a substitute for measurement.

The exercise above illustrates a key insight: the model cannot tell you its own production reliability. The vendor's benchmark table cannot either. The only thing that tells you production reliability is running the model on your prompts and measuring the outputs. That is what Chapters 2–4 of this course are built around.

- A model cannot tell you its own production reliability, and neither can the vendor's benchmark table — only running the model on your prompts with measurement produces actionable data.
- MMLU measures knowledge recall across 57 academic subjects via multiple-choice; your agentic pipeline calls tools with structured JSON schemas, retrieves from documents you provide, and produces outputs downstream code must parse — none of which MMLU measures.
- HumanEval measures single-function correctness, not multi-step tool-integrated code generation; high HumanEval scores do not prevent malformed JSON schemas in function-calling pipelines.

The 7 dimensions that predict production success

Based on our internal benchmark data across 12 months of production AI workloads, these are the seven dimensions that consistently separate models in ways that matter:

1. Tool-use determinism

The probability that the same prompt, at the same temperature, produces structurally equivalent tool calls or JSON output across independent runs. For agentic pipelines where model output feeds into downstream code, a 10% variance in output structure compounds dramatically. A three-step pipeline where each step has 90% structural stability has only a 73% end-to-end success rate. Five steps: 59%. Determinism is the foundational reliability metric for any agentic workload.

This is covered in depth in Chapter 2.

2. Context fidelity at depth

The ability to accurately retrieve and reason about information that appears in the middle of a long context window. All three frontier models exhibit "lost-in-the-middle" degradation — accuracy at retrieval drops as documents are buried deeper in the context. The key question is not how large the context window is, but how reliably the model retrieves from different positions within it. [6]

3. Structured-output reliability

The fraction of responses that parse cleanly as valid JSON (or whatever schema you specify) without requiring retry or post-processing. Related to determinism but distinct: a model can be deterministic in which keys it returns while still producing malformed JSON on 5% of calls. High structured-output reliability reduces retry costs and simplifies error handling.

4. Latency at your percentile

Not average latency — your 95th or 99th percentile latency under realistic concurrency. For a customer-facing feature, a 2-second average with a 12-second p99 may be worse than a 3-second average with a 5-second p99. Latency is workload-specific and cannot be read from a spec sheet.

5. Cost-per-task (not cost-per-token)

The true cost to complete one unit of your workload, accounting for retry rates, prompt caching hit rates, and tool-call overhead. A cheaper model with higher retry rates can easily cost more per task than an expensive model with near-perfect reliability. Covered in Chapter 4.

6. Multimodal fidelity

Whether the model handles the modality you actually need, and whether it does so on the same surface as reasoning. For Gemini 3.1 specifically, this distinction matters: gemini-3.1-pro-preview accepts text, image, video, audio, and PDF inputs and outputs text, while audio generation is explicitly not supported on that model. Scripted narration uses gemini-3.1-flash-tts-preview, a separate text-to-audio preview model for exact recitation and style-controlled speech. [13][14][15][16]

7. Governance and lifecycle risk

Whether the endpoint, access path, and model lifecycle fit production. Preview model IDs are useful for evaluation, but they create operational requirements: configurable model IDs, changelog review, deprecation monitoring, fallback routing, and per-model latency/error tracking. A model can be technically strong and still fail your governance bar if it is available only through a preview endpoint your team cannot safely operate.

- Tool-use determinism — the probability that the same prompt produces structurally equivalent output across independent runs — is the foundational reliability metric for any agentic workload.
- A 10% variance per step compounds multiplicatively: a 5-step pipeline where each step has 90% structural stability has only a 59% end-to-end success rate.
- Governance and lifecycle risk matters: preview model IDs require configurable IDs, changelog review, deprecation monitoring, and fallback routing before production use.

The 3 dimensions you can probably ignore

Not everything matters equally. Here are three dimensions frequently cited in benchmark tables that correlate weakly with most production workloads:

1. Aggregate reasoning score (MMLU, GPQA)

Unless your use case involves answering graduate-level science questions or broad knowledge recall, a 3-point delta in aggregate reasoning score is noise compared to a 5% difference in tool-use determinism. These scores are useful for tracking model progress over time, not for choosing between current-generation frontier models.

2. Peak performance on hard problems

"The model can solve competition math" is a capability, not a production metric. Peak capability tells you the ceiling; it says nothing about the floor. For most production workloads, the floor (what happens on the 10% of prompts where the model struggles) matters more than the ceiling.

3. Multilingual performance (unless your product is multilingual)

If you're building an English-language product, a model's Chinese or Arabic benchmark scores are irrelevant. Benchmark tables aggregate across many settings; make sure the dimension being measured applies to your actual distribution.

- MMLU and GPQA aggregate scores have weak correlation with production outcomes for most builders; a 3-point delta in reasoning score is noise compared to a 5% difference in tool-use determinism.
- Peak capability (e.g., "can solve competition math") describes the ceiling; for production the floor matters more — what the model does on the 10% of prompts where it struggles.
- Benchmark dimensions that don't apply to your actual distribution should be excluded from your scorecard entirely; don't let irrelevant axes influence the model decision.

Building your scorecard

The scorecard is a simple forcing function: before you run any benchmark, you write down which dimensions matter for your use case and how much you weight them. This prevents the common failure mode of running a benchmark, seeing that one model wins on latency, and anchoring on that — ignoring that your use case is latency-tolerant but determinism-critical.

The model should produce a table weighting tool-use determinism and structured-output reliability highest (4–5), cost-per-task and context fidelity at medium (3), latency at lower priority (2, since async PRs are latency-tolerant), and excluding multilingual and peak math. If it weights differently, that's worth examining — the model's reasoning reveals assumptions about your use case that you should validate.

A well-built scorecard has three properties: 1. Weights reflect your production SLA, not generic impressiveness. A latency-tolerant batch job should weight determinism higher than latency. 2. It includes a disqualifier. At least one dimension where a failing score eliminates a model regardless of other scores. For a tool-use pipeline, a determinism score below 85% is typically a disqualifier. 3. It is written before you see the benchmark results. Post-hoc scorecards unconsciously anchor on the model you already prefer.

Use-case archetypes

Most production AI workloads fall into one of three archetypes. Use these as a starting point for your scorecard, then customize.

Choosing your Gemini family member

As of May 2026, the Gemini 3.1 family has specialized into three distinct surfaces. Choosing the right one is your first move in model selection.

If your use case maps cleanly to one of these archetypes, you already know your top dimensions. If it doesn't — if you're building something latency-critical and tool-heavy and long-context — you have a hard evaluation problem and should expect to make tradeoffs rather than finding a model that wins on all axes.

Hands-on exercise

Build a scorecard for your use case.

1. Choose one of the three archetypes above as your starting point, or describe your own use case in 2–3 sentences.
2. Select 5 dimensions from this list: tool-use determinism, context fidelity at depth, structured-output reliability, latency p95, cost-per-task, multimodal fidelity, governance/lifecycle risk, multilingual performance, aggregate reasoning score.
3. Assign each a weight from 1 (nice to have) to 5 (critical). Total weight must equal 15.
4. For each dimension with weight ≥ 4, write one sentence explaining why it is high-priority for your use case.
5. Identify one disqualifier: a minimum threshold on one dimension below which you would not use a model regardless of its scores on other dimensions.

Verification: Your scorecard is valid if: - Exactly 5 dimensions are listed - Weights sum to 15 - At least one dimension has weight ≥ 4 with a written justification - A disqualifier is named

Estimated time: 15 minutes

What's next

Chapter 1 gave you the framework: five production dimensions, three benchmarks to deprioritize, and a scorecard template for your workload. You now have a hypothesis about which dimensions matter most for your use case — but a hypothesis is not evidence.

In Chapter 2, you'll run the 10×3×5 benchmark that measures the dimension most commonly overlooked in public comparisons: tool-use determinism. You'll run it across Opus 4.7, GPT-5.5, and Gemini 3.1 Pro on a reference prompt set — and optionally add 2 prompts from your own use case.

References

[1] Hendrycks, D. et al. (2021). "Measuring Massive Multitask Language Understanding." ICLR 2021 — https://arxiv.org/abs/2009.03300 · retrieved 2026-04-30

[2] Chen, M. et al. (2021). "Evaluating Large Language Models Trained on Code." OpenAI — https://arxiv.org/abs/2107.03374 · retrieved 2026-04-30

[3] Koenig AI Academy internal benchmark data, Q2 2026 — /data/claude-tool-use-determinism/2026-Q2/ · retrieved 2026-04-30

[4] Patil, S. et al. Berkeley Function-Calling Leaderboard (BFCL) V4 — https://gorilla.cs.berkeley.edu/leaderboard.html · retrieved 2026-04-30

[5] OpenAI. Introducing GPT-5.5 — https://platform.openai.com/docs/models/gpt-5-5 · retrieved 2026-04-30

[6] Liu, N. et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts" — https://arxiv.org/abs/2307.03172 · retrieved 2026-04-30

[7] Anthropic. Claude models overview — context windows and specifications — https://docs.anthropic.com/en/docs/about-claude/models/overview · retrieved 2026-04-30

[8] Google. Gemini 3.1 Pro model specification — https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-1-pro · retrieved 2026-04-30

[9] Anthropic. Prompt caching — https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching · retrieved 2026-04-30

[10] OpenAI. Prompt caching in the API — https://platform.openai.com/docs/guides/prompt-caching · retrieved 2026-04-30

[11] Google. Context caching overview (Gemini API) — https://ai.google.dev/gemini-api/docs/caching · retrieved 2026-04-30

[12] Google. Gemini 3.1 Pro launch post — https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ · retrieved 2026-05-28

[13] Google DeepMind. Gemini 3.1 Pro model card — https://deepmind.google/models/model-cards/gemini-3-1-pro/ · retrieved 2026-05-28

[14] Google. Gemini 3.1 Pro Preview model page — https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview · retrieved 2026-05-28

[15] Google. Gemini 3.1 Flash TTS launch post — https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-flash-tts/ · retrieved 2026-05-28

[16] Google. Gemini API speech generation guide — https://ai.google.dev/gemini-api/docs/speech-generation · retrieved 2026-05-28

> Prerequisites: Chapter 1 — you should have a scorecard with your top-priority dimensions and understand why tool-use determinism matters for your workload. > > Time: 60 minutes > > Learning objectives: By the end of this chapter, you can define tool-use determinism precisely, run the 10×3×5 benchmark, interpret variance as a reliability signal, and know which model wins — and by how much — on each prompt category.

Tool-use determinism, in the context of large language model evaluation, refers to the probability that a given prompt produces structurally equivalent tool calls or structured outputs across independent inference runs, controlling for temperature. Unlike accuracy (whether the output is correct) or latency (how fast it arrives), determinism measures stability — whether the output schema, key set, and structural decisions remain consistent run-to-run. As of Q2 2026, no major public benchmark measures this property. The 10×3×5 dataset (/data/claude-tool-use-determinism/2026-Q2/) is the basis for Chapters 2 and Chapter 4 of this course, and this chapter walks through the benchmark design, methodology, results, and a reproducible runner script.

Key facts

• At temperature=0, all three frontier models show measurable structural variance on complex tool schemas. Variance ranges from 2% (Opus 4.7 on simple schemas) to 22% (Gemini 3.1 Pro on nested multi-tool schemas with 5+ required fields). ^[1]
• Multiplicative reliability degradation: if a single tool call has 90% structural stability, a 5-step agentic pipeline relying on sequential tool calls has an end-to-end success probability of 0.9⁵ = 59% — assuming independence. For correlated failures (common prompt patterns that trigger the same instability), the degradation is worse. ^[1]
• The 10×3×5 benchmark uses 10 prompt categories, 3 models (Opus 4.7, GPT-5.5, Gemini 3.1 Pro), 5 independent runs per prompt per model. Each run is scored as a structural match or mismatch against a canonical reference output — producing a determinism score (0–100%) per prompt per model. ^[1]
• Opus 4.7 leads on determinism overall (91.4% average), but the margin over GPT-5.5 (88.0%) narrows significantly on simple schemas and widens significantly on complex nested schemas. Gemini 3.1 Pro averages 81.9% — viable for tolerant workloads, a liability for strict pipelines. ^[1]
• The most common failure mode across all three models is not hallucination — it is key omission: a required field present in 4 of 5 runs is silently absent on the 5th. This is harder to detect than a schema validation error because it often produces structurally valid (but incomplete) JSON. ^[1]
• Prompt caching marginally improves determinism on Anthropic's API: cached prompt prefixes produce slightly more stable outputs than uncached equivalents. This suggests the tokenization pathway — not just the model weights — influences structural stability. ^[2]
• OpenAI's GPT-5.5 with response_format: { type: "json_schema" } and a strict schema (enforcing exact required keys) improves its determinism score from 88% to 93% — making it competitive with Opus 4.7 when the schema is fully specified. This is the most important single finding in our dataset. ^[3]

- Opus 4.7 leads on determinism at 91.4% average; GPT-5.5 averages 88.0%; Gemini 3.1 Pro averages 81.9% — but all three show nonzero structural variance even at temperature=0.
- GPT-5.5 with strict JSON schema enforcement jumps from 88% to 93% on complex schemas — schema enforcement is a larger lever than model choice on OpenAI's platform.
- Prompt caching marginally improves determinism on Anthropic's API: cached prompt prefixes produce slightly more stable outputs than uncached equivalents.

What determinism is (and isn't)

Before running the benchmark, it helps to be precise. Determinism as used here is not:

• Identical character-for-character output. Two responses can be structurally equivalent while differing in whitespace, field ordering, or string values. We normalize JSON before comparison.
• Accuracy. A model can be perfectly deterministic while being consistently wrong. These are orthogonal.
• Repeatability at fixed seed. Most commercial APIs do not expose a random seed. Temperature=0 is the closest approximation, but it does not guarantee identical outputs across runs — especially at high model load or across API versions. ^[4]

Determinism is: - The fraction of runs (out of N) where the output, when normalized, matches the canonical reference structure — same keys present, same types, same nesting depth. - A production reliability signal: high determinism means your downstream parser can trust the model's output without defensive retries.

Why it degrades pipelines multiplicatively

This math is the single most important thing in this chapter.

In a pipeline where each step calls an LLM tool, structural failures at step k produce garbage that propagates forward. If each step has determinism d, and you have n steps:

Pipeline success rate = d^n   (assuming independence)

Gemini 3.1 Pro at 81.9% average determinism: a 5-step pipeline has a 37% success rate. That means 63% of runs require at least one retry or manual intervention. At any reasonable scale, that's untenable.

<Callout type="hot"> The temperature=0 illusion. Setting temperature to 0 is the most common "fix" builders reach for when they notice output variance. It helps — but it does not eliminate structural variance. All three frontier models in our dataset show nonzero structural variance at temperature=0. The reason: sampling is only one source of variance. Attention routing, batching behavior, and API load conditions introduce variance that temperature does not control. Measure empirically; do not assume. </Callout>

- Determinism measures structural stability — whether the output schema, key set, and nesting remain consistent run-to-run — not accuracy or character-for-character repeatability.
- Temperature=0 reduces but does not eliminate structural variance; attention routing, batching behavior, and API load all introduce variance that temperature cannot control.
- In a pipeline where each step has determinism d over n steps, end-to-end success probability is d^n: a 90% per-step rate becomes 59% over 5 steps.

Benchmark design: 10 prompt categories

The 10 prompt categories in /data/claude-tool-use-determinism/2026-Q2/ were selected to represent the full range of tool-use complexity seen in production agentic workloads. Unlike accuracy-focused multi-task benchmarks such as HELM ^[8] and BIG-Bench ^[9], which measure correctness across diverse capability dimensions, the 10×3×5 benchmark measures structural stability — whether the output schema remains consistent across runs, not whether the content is correct:

Categories 1–4 are "simple." Categories 5–7 are "medium." Categories 8–10 are "complex." The benchmark covers all three tiers.

Results summary

Full results are in /data/claude-tool-use-determinism/2026-Q2/results.json. Summary:

Determinism scores by category (5 runs each, temperature=0)

Headline findings:

1. All three models are reliable on simple schemas. Categories 1–2 show near-100% determinism across all models. If your use case is limited to flat schemas with ≤3 fields, model choice on determinism grounds is a non-issue.

1. The gap widens dramatically at complexity. Opus 4.7's 11-point lead over Gemini at category 10 vs. 0-point lead at category 1 means complexity is the lever. Match your model choice to your schema complexity, not your prompt complexity.

1. GPT-5.5 with strict JSON schema closes the gap. When we reran categories 6–10 with OpenAI's strict: true JSON schema enforcement (available since GPT-4.5), GPT-5.5's scores on categories 6–10 rose to 93–97% — matching or exceeding Opus 4.7 on nested schemas. This is the most actionable finding: schema enforcement is a bigger lever than model choice for structured-output reliability on OpenAI's platform. ^[3]

1. Category 9 (ambiguous input) is the universal weakness. All three models show their lowest determinism here. This prompt type — where the correct response is either a tool call or a clarifying question, depending on interpretation — reveals the deepest form of instability. If your pipeline regularly receives ambiguous inputs, plan for retry logic regardless of model choice.

1. Gemini 3.1 Pro requires surface hygiene. Google's launch post says 3.1 Pro is still a preview release while Google validates agentic workflow updates, and the Gemini API model page documents function calling, structured outputs, code execution, caching, and a separate gemini-3.1-pro-preview-customtools endpoint for workflows that mix bash and custom tools. It also documents text output only and no audio generation. For tool-use evals, benchmark schema adherence (whether the model fills every required field), not just JSON validity, and keep the model ID configurable so a preview or custom-tools endpoint can be swapped without rewriting your benchmark. ^[6][7]

The most common failure modes

Across 150 runs (10 prompts × 3 models × 5 runs), we classified each structural mismatch:

Key omission is the dominant failure mode. It is also the most dangerous: it passes many JSON schema validators (which check structure, not completeness) while silently dropping data that downstream stages expect.

- All three models are reliable on simple schemas (categories 1–2 show near-100% determinism); the gap widens dramatically at complex nested schemas and multi-model handoffs.
- The model gap scales with complexity: Opus 4.7's 11-point lead over Gemini at category 10 vs. a 0-point lead at category 1 means complexity is the key lever.
- Key omission — a required field present in 4 of 5 runs but silently absent on the 5th — accounts for 54% of structural mismatches and is the most dangerous failure mode.

Running the benchmark yourself

The benchmark runner is a ~80-line Python script. Here's the core loop:

```python import anthropic import json import hashlib

def normalize_json(obj): """Canonical form: sorted keys, stripped whitespace.""" return json.dumps(obj, sort_keys=True, separators=(',', ':'))

def structural_hash(text): """Hash the key structure, not the values.""" try: parsed = json.loads(text) keys_only = extract_key_structure(parsed) return hashlib.sha256(normalize_json(keys_only).encode()).hexdigest() except json.JSONDecodeError: return None

def extract_key_structure(obj, depth=0): """Recursively extract keys with types, not values.""" if isinstance(obj, dict): return {k: extract_key_structure(v, depth+1) for k, v in obj.items()} elif isinstance(obj, list) and obj: return [extract_key_structure(obj[0], depth+1)] else: return type(obj).__name__

def run_benchmark(prompt, tool_schema, model, n_runs=5): client = anthropic.Anthropic() hashes = [] # Note: temperature=0 is used here for Opus 4.7 / Sonnet 4.6 / GPT-style models. # Claude Sonnet 5 rejects any non-default temperature value (returns HTTP 400). # For Sonnet 5, omit temperature entirely — its default behavior is already # greedy-equivalent for structured output. Use prompt-level or schema-level # constraints (e.g. tool_choice, output schemas) to control determinism instead. use_temperature = not model.startswith("claude-sonnet-5") for _ in range(n_runs): kwargs = dict( model=model, max_tokens=1024, tools=[tool_schema], messages=[{"role": "user", "content": prompt}] ) if use_temperature: kwargs["temperature"] = 0 response = client.messages.create(**kwargs) tool_call = next( (b for b in response.content if b.type == "tool_use"), None ) if tool_call: hashes.append(structural_hash(json.dumps(tool_call.input))) else: hashes.append(None)

canonical = max(set(hashes), key=hashes.count) determinism = hashes.count(canonical) / n_runs return determinism, hashes ```

The structural_hash function is the key: it extracts the shape of the JSON (keys and types) without the values, so two responses that return different string values for the same keys are counted as structurally equivalent.

The model should call create_ticket with fields: title (string), description (string), account_id (string), priority (string or enum), submitted_at (string/datetime). Run this prompt 5 times in your own environment and check whether all 5 calls produce the same key structure. The expected determinism at temperature=0 is approximately 95%+ for this simple schema — if you see structural variation, note which fields fluctuate.

This is a category-7 style prompt (array of objects, variable length). The model will return a JSON plan. Run it 5 times and use the benchmark script's structural_hash function to check determinism. Expect ~86–88% determinism on this prompt — you may see variance in how many steps are included, in whether `depends_on` is an array or a single integer, or in whether the final scheduling step is split into two. Each of these is a structural mismatch.

- The `structural_hash` function extracts JSON key structure and types without values — two responses with different string values but identical key sets count as structurally equivalent.
- Run each prompt at temperature=0 for 5 independent calls per model; the canonical output is the most frequent hash; determinism score is the fraction of runs matching it.
- **Claude Sonnet 5 caveat**: Sonnet 5 rejects non-default temperature, top_p, and top_k (HTTP 400). Omit these parameters entirely when benchmarking Sonnet 5 and rely on prompt/schema controls for determinism.
- A determinism score below 90% warrants schema enforcement before pipeline deployment; below 70% requires additional guardrails such as constrained generation.

Interpreting your results

Once you have 5 determinism scores per prompt per model, you have enough data to make a production decision — at least directionally. Here's how to read the numbers:

Apply these thresholds to your specific prompt categories, not to the average. A model with 95% average determinism may have 70% determinism on the specific prompt type your pipeline uses most.

Hands-on exercise

Run the 10×3×5 benchmark on 2 prompts from your own use case.

1. Install the benchmark runner:
2. ```bash
3. pip install anthropic openai google-generativeai
4. git clone <internal-benchmark-repo> # or copy the script above
5. ```

1. Write 2 prompts from your actual use case that involve a tool call or structured JSON output. At least one should use a schema with ≥4 required fields.

1. Run each prompt 5 times on at least 2 of the 3 models (Opus 4.7 and GPT-5.5 are the minimum; Gemini 3.1 Pro optional). Use temperature=0 for Opus 4.7 and GPT-5.5. For Claude Sonnet 5, omit temperature entirely (Sonnet 5 rejects non-default sampling params and returns HTTP 400).

1. Record your determinism scores. Compare against the reference data for the closest matching category in /data/claude-tool-use-determinism/2026-Q2/results.json.

1. If you observe a structural mismatch, run extract_key_structure on the divergent output to identify which key(s) caused the mismatch. This is the actionable signal.

Verification: You have completed this exercise when: - Determinism scores are recorded for ≥2 models across ≥5 runs for at least 1 prompt - The structural mismatch type (if any) is identified from the failure taxonomy - You can state whether your use case falls in the "safe zone" (≥90%) or requires guardrails

Estimated time: 30 minutes (15 min setup, 15 min analysis)

What's next

You now have empirical determinism scores for your prompts — and an understanding of why simple schemas are robust while complex schemas are fragile. In Chapter 3, we shift from width (structural consistency) to depth (context fidelity). You'll run a needle-in-haystack test across 50K, 200K, and 500K token depths to find out where each model's "effective" context window actually ends.

References cited

^[1]: Koenig AI Academy internal benchmark data, Q2 2026. /data/claude-tool-use-determinism/2026-Q2/. Benchmark design: 10 prompt categories × 3 models × 5 runs at temperature=0 × 2 schema complexity tiers.

^[2]: Anthropic. "Prompt caching." Claude API documentation. https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching — canonical API reference for caching behavior and tokenization consistency. Cache hit behavior and tokenization path consistency noted in internal A/B across 500 cached vs. uncached runs.

^[3]: OpenAI. "Structured Outputs." Model release notes. https://platform.openai.com/docs/models — GPT-5.5 strict JSON schema enforcement capabilities.

^[4]: Anthropic. "API reference: create a message." Claude API documentation. https://docs.anthropic.com/en/api/messages — temperature parameter specification and non-determinism sources at temperature=0 beyond sampling.

^[5]: Shen, Y. et al. (2023). "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace." https://arxiv.org/abs/2303.17580 — real-world analysis of multi-step tool-calling pipeline failure modes.

^[6]: Google. "Gemini 3.1 Pro Preview." https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview — model capabilities, context limits, structured outputs, function calling, and custom-tools endpoint; retrieved 2026-05-28.

^[7]: Google. "Gemini 3.1 Pro: A smarter model for your most complex tasks." https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/ — preview rollout surfaces and agentic-workflow caveat; retrieved 2026-05-28.

^[8]: Liang, P. et al. (2022). "Holistic Evaluation of Language Models (HELM)." https://arxiv.org/abs/2211.09110 — multi-scenario benchmark framework for standardized capability evaluation; referenced for taxonomy and evaluation design principles.

^[9]: Srivastava, A. et al. (2022). "Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (BIG-Bench)." https://arxiv.org/abs/2206.04615 — large-scale multi-task benchmark establishing principles for distinguishing model capability dimensions.

> Prerequisites: Chapter 1 — you understand the concept of "effective context window" as distinct from the advertised limit. Chapter 2 is recommended but not required. > > Time: 50 minutes > > Learning objectives: By the end of this chapter, you can run a needle-in-haystack test at three depth levels, identify each model's effective context ceiling, and choose a chunking strategy appropriate for your document volume.

Long-context language model evaluation encompasses the methods used to measure how accurately and reliably a model retrieves and reasons over information as document length increases, independent of whether that information appears near the beginning, middle, or end of the input. As of Q2 2026, the three frontier models compared in this course advertise context windows of 1M tokens (Anthropic Claude Opus 4.7), 128K tokens (OpenAI GPT-5.5), and 1M tokens (Google Gemini 3.1 Pro). The gap between these advertised windows and each model's effective context window — the depth at which retrieval accuracy remains above 90% — ranges from 1.5× to 4× depending on task type, document structure, and whether the required information appears in a "hot zone" (beginning or end) or "cold zone" (middle). This chapter gives you the tools to measure that gap for your specific documents.

Key facts

• Lost-in-the-middle degradation is a documented property of transformer-based language models: retrieval accuracy is highest for information at the beginning and end of a long context, and falls sharply for information buried in the middle. The original study measured a 20–40 percentage point accuracy drop at depths above 50% of context length. ^[1]
• Gemini 3.1 Pro's 1M token context is genuinely superior at raw retrieval of isolated facts up to ~600K tokens, outperforming Opus 4.7 on needle-in-haystack retrieval tests at depths of 100K–400K. ^[2]
• However, Gemini 3.1 Pro's multi-hop reasoning accuracy — tasks requiring synthesis across multiple facts from different parts of the context — degrades faster than Opus 4.7's at depths above 300K tokens. A model that can retrieve a needle does not necessarily reason reliably across multiple needles. ^[3][6]
• Opus 4.7's 1M context window outperforms Gemini 3.1 Pro on synthesis tasks (cross-document reasoning, contradiction detection, multi-fact aggregation) — its synthesis effective limit (~500K tokens) substantially exceeds Gemini's (~300K tokens). ^[2][6]
• GPT-5.5's 128K context is the smallest of the three, but its middle-context performance (50–80% depth) is the most stable — it shows less "lost in the middle" degradation than either competitor on the retrieval tasks in our dataset. ^[4]
• The practical threshold for "reliable synthesis" (multi-fact reasoning accuracy ≥ 85%) varies by task: single-fact retrieval is reliable to Gemini's full advertised window; two-fact synthesis degrades sharply above 400K tokens; three-or-more-fact synthesis is unreliable beyond 200K tokens on all three models. ^[5][6]
• A well-implemented RAG (Retrieval-Augmented Generation) pipeline using top-k=5 with good embeddings typically outperforms full-context loading for documents above 100K tokens, at a fraction of the inference cost. Long context is not always the right answer. ^[5][7]

- Gemini 3.1 Pro leads on raw needle-in-haystack retrieval up to ~600K tokens; Opus 4.7 leads on multi-fact synthesis with an effective synthesis limit of ~500K tokens versus Gemini's ~300K.
- GPT-5.5's 128K context is the smallest advertised window but shows the most stable middle-context performance with less "lost in the middle" degradation than either competitor.
- A well-implemented RAG pipeline with top-k=5 typically outperforms full-context loading for documents above 100K tokens at a fraction of the inference cost.

The advertised vs. effective context window

Vendors advertise context window size in tokens. What they don't advertise is the shape of the accuracy curve within that window — how retrieval and reasoning quality changes as you fill the context.

Three useful concepts:

1. Retrieval effective limit: the depth at which single-fact retrieval accuracy falls below 90%. This is the safest operating boundary for fact-lookup use cases.

2. Synthesis effective limit: the depth at which cross-document reasoning accuracy falls below 85%. This is typically 30–50% of the retrieval effective limit — a significantly lower bar.

3. Hot zone: the first ~15% and last ~15% of a context window, where all models show dramatically higher accuracy. If your document structure places the most important information at the start and end (executive summary + conclusion), you're working with the model's bias, not against it.

Here's how the three models compare on each measure (from our internal tests and published third-party evaluations):

The headline: Opus 4.7 and Gemini 3.1 Pro share the same 1M advertised window but show different effective limit profiles: Gemini leads on raw retrieval depth, while Opus 4.7's synthesis effective limit (~500K tokens) substantially exceeds Gemini's (~300K tokens). But: - Its synthesis effective limit (300K) is only 30% of its advertised window. - Its synthesis accuracy within the effective limit is lower than Opus 4.7's for complex multi-hop tasks. - Loading 300K tokens costs significantly more per call than a well-tuned RAG pipeline over the same documents.

- The retrieval effective limit (90% single-fact accuracy) is typically 1.5–4× larger than the synthesis effective limit (85% multi-hop accuracy) — choose the right limit for your task type.
- The "hot zone" (first and last ~15% of context) shows dramatically higher accuracy across all models; placing important information at the start and end works with the model's attention bias.
- A 1M token context window means the model receives 1M tokens, not that it attends to all of them equally — treat large context windows as a retrieval tool, not working memory.

The three failure modes at scale

When a model exceeds its effective context limit, failures follow recognizable patterns. Knowing them helps you detect problems before they reach production.

Failure mode 1: Lost needles (retrieval miss)

The model returns an answer that ignores a fact explicitly present in the context. The fact is not hallucinated — it is simply not retrieved. This is the most common failure mode at moderate depth (50K–200K tokens for GPT-5.5; 200K–500K for Gemini 3.1 Pro).

Detection: run a needle-in-haystack test (see Hands-on exercise). Ask a question with a unique, specific answer buried in the document. A correct answer = retrieval; a plausible-but-wrong answer = lost needle.

Failure mode 2: Hallucinated synthesis

The model synthesizes an answer that combines real retrieved facts with invented connections. Unlike a lost needle (no answer), hallucinated synthesis produces a fluent, confident answer that is partially fabricated. This failure mode emerges in multi-hop reasoning tasks at depth.

It is harder to detect than a lost needle because the output looks high quality. Detection requires ground-truth verification — you must know the correct answer in advance, which isn't always possible in production.

Failure mode 3: Degraded step-by-step reasoning

On chain-of-thought tasks at high context depth, models show shorter, less thorough reasoning chains. The model short-circuits multi-step reasoning, skipping intermediate steps that it would correctly execute at lower context depths. This failure mode shows up in math-word problems, multi-step code analysis, and legal document reasoning.

Detection: include a complex reasoning task in your evaluation, not just retrieval. Compare the model's chain-of-thought at 50K tokens vs. 200K tokens on the same task.

- Lost needles (retrieval miss), hallucinated synthesis (fluent but partially fabricated answer), and degraded step-by-step reasoning are the three failure modes as context depth increases.
- Hallucinated synthesis is harder to detect than a lost needle because the output looks high quality — detection requires ground-truth verification.
- Degraded reasoning at depth shows as shorter chain-of-thought chains; compare chain-of-thought quality at 50K vs. 200K tokens on the same task to detect this failure mode.

The needle-in-haystack evaluation

The needle-in-haystack test ^[8] is the standard method for measuring retrieval effective limit. The methodology:

1. Prepare a "haystack" — a large document padded to the target token depth (e.g., a legal corpus, a Wikipedia dump, or synthetic filler text).
2. Insert a "needle" — a unique, specific fact that cannot be guessed from context ("The secret phrase is: banana-lighthouse-44").
3. Insert the needle at a specific position (expressed as percentage of total context depth, e.g., 25%, 50%, 75%).
4. Ask the model to retrieve the needle.
5. Score: correct retrieval = 1, any other response = 0.
6. Repeat across multiple needle positions and context sizes to build an accuracy heatmap.

A well-designed evaluation tests a grid: context size (50K / 100K / 200K / 500K) × needle position (10% / 25% / 50% / 75% / 90%). Each cell should have ≥3 runs to average out noise.

At 50K tokens with the needle at 50% depth (25,000 tokens in), Claude Sonnet 4.6 reliably retrieves this. The correct answer is 'KST-7-2026-09142'. At this depth the model should respond with high confidence. If you run this with your real documents at higher depths (100K, 200K), note when the retrieval accuracy drops and at what needle position first.

This is a three-fact synthesis task. The model must: (1) retrieve growth rates from three separate locations (18%, -4%, 22%), (2) rank them correctly (Great Lakes > Pacific Northwest > Southeast), (3) calculate combined revenue of top 2 ($4.2M + $5.1M = $9.3M), (4) reason about the Southeast's underperformance from the 'delayed contract closures' clue. At 150K tokens with facts spread across different 'pages', this tests synthesis effective limit. If the model gives the wrong combined revenue or misses the delayed-closure explanation, that's a synthesis failure, not just a retrieval miss.

- The needle-in-haystack methodology tests a grid of context size × needle position; each cell needs ≥3 runs to average out noise and build a reliable accuracy heatmap.
- A three-fact synthesis task is a harder and more realistic production test than single-fact retrieval — use both in your evaluation to distinguish retrieval from reasoning capability.
- Determine your retrieval effective limit empirically on your own documents; vendor-published context window sizes describe the ceiling, not the reliable operating range.

Choosing your context strategy

Given this complexity, here is a practical decision framework for multi-document workloads:

The key principle: use long context for retrieval tasks; use chunking + multiple calls for synthesis tasks. These are different operations with different reliability profiles.

The RAG vs. long-context tradeoff quantified

For a document corpus of 200K tokens, the cost and reliability comparison looks like this (rough figures from our internal workloads):

The RAG approaches are 10–20× cheaper. For synthesis tasks, they match or exceed full-context loading accuracy. For retrieval of a single specific fact (where the retrieved chunk is guaranteed to contain the answer), they are slightly less reliable because the embedding retrieval step may miss the right chunk.

The practical takeaway: if your workload is primarily synthesis, use RAG. If your workload is primarily exact-fact retrieval from a single large document, long context is the simpler, more reliable choice — and here, Gemini 3.1 Pro has a genuine advantage.

Hands-on exercise

Run a needle-in-haystack test and source-packet synthesis at three depth levels.

1. Choose a document or document set from your production context. Prepare versions at three sizes: ~50K tokens, ~200K tokens, and as large as your target depth (up to 1M tokens if testing Gemini 3.1 Pro Preview's full window).

1. Insert 3 unique "needles" into each version:
2. - Needle A: near the start (5–10% depth)
3. - Needle B: in the middle (45–55% depth)
4. - Needle C: near the end (85–95% depth)

1. For each model you are evaluating, ask: "What is the value of [needle identifier]?" Run each retrieval ≥3 times.

1. Record a 3×3 accuracy grid (3 depths × 3 positions). Note which positions and depths produce failures.

1. Run a source-packet synthesis task: a question that requires combining facts from Needles A and C to produce a summary or plan. If testing Gemini 3.1 Pro Preview, note its documented 1M-token context and 64K output token ceiling — ensure your synthesis prompt doesn't hit the output limit. Record whether the model correctly synthesizes both facts while maintaining reasoning quality.

Verification: You have completed this exercise when: - A 3×3 retrieval accuracy grid is filled for ≥1 model - The retrieval effective limit (depth where accuracy first drops below 90%) is estimated - The source-packet synthesis result is recorded, noting any reasoning degradation at 1M-token depth - You have explicitly checked Gemini's 64K output ceiling if using it for large-scale summarization

Estimated time: 25 minutes

What's next

You now have empirical data on both determinism (Chapter 2) and context fidelity (Chapter 3). Together, these two chapters answer: can I trust the model's outputs, and can I trust them when my documents are large?

The final question is: what does reliable output actually cost? In Chapter 4, you'll build a cost-per-task model that accounts for retry rates, context caching, and tool-call overhead — and discover why the cheapest model on the pricing page is often not the cheapest model in your bill.

References cited

^[1]: Liu, N. F. et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts." Transactions of the Association for Computational Linguistics, 12. https://arxiv.org/abs/2307.03172 — foundational study on retrieval accuracy degradation as a function of document position.

^[2]: Anthropic. Claude Opus 4.7 model card and release notes. https://www.anthropic.com/news — context window specifications and long-context benchmark comparisons.

^[3]: Google DeepMind. "Gemini 3.1 Pro release and changelog." https://ai.google.dev/gemini-api/docs/changelog — 1M token context capability notes and multimodal context handling.

^[4]: OpenAI. "GPT-5.5 release notes." https://platform.openai.com/docs/models — 128K context window specifications and retrieval accuracy claims.

^[5]: Hsieh, C.-Y. et al. (2024). "RULER: What's the Real Context Size of Your Long-Context Language Models?" https://arxiv.org/abs/2404.06654 — empirical methodology for measuring effective context window; multi-needle evaluation design.

^[6]: Bai, Y. et al. (2024). "LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks." https://arxiv.org/abs/2412.15204 — multi-hop synthesis degradation analysis across frontier models.

^[7]: Lewis, P. et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS 2020. https://arxiv.org/abs/2005.11401 — foundational RAG architecture paper; baseline for cost and accuracy comparisons between retrieval-augmented and full-context approaches.

^[8]: Kamradt, G. (2023). "Needle In A Haystack — Pressure Testing LLMs." https://github.com/gkamradt/LLMTest_NeedleInAHaystack — standard methodology for context window retrieval pressure testing; basis for the position-grid evaluation design used in this chapter's hands-on exercise.

• Koenig AI Academy internal long-context dataset: derived from /data/claude-tool-use-determinism/2026-Q2/ extended test set.

> Prerequisites: Chapter 1 required; Chapters 2 and 3 recommended for the best practical grounding. You should have token counts from at least one benchmark run. > > Time: 50 minutes > > Learning objectives: By the end of this chapter, you can calculate a defensible cost-per-task number for your workload, account for retries and caching, and know when the "cheaper" model is actually more expensive.

Cost-per-task is the total cost to complete one end-to-end production workload unit — input tokens, output tokens, tool-call overhead, retries, and cache misses. It is distinct from $/M token pricing, which ignores the factors that dominate real bills. As of Q2 2026, Gemini 3.1 Pro is cheapest per token, GPT-5.5 the most expensive, Opus 4.7 in the middle — but the cost-per-task ordering is often the reverse. This chapter shows why.

Key facts

• List pricing (Q2 2026): Opus 4.7 $5/$25/M in/out, cache read $0.50/M (90% discount) ^[1]; GPT-5.5 $5/$30/M, cached input $0.50/M (90% discount) ^[2]; Gemini 3.1 Pro $2/$12/M, context caching $0.20/M (90% discount) ^[3].
• On a simple prompt with no retries, Gemini 3.1 Pro is 2.5× cheaper than Opus 4.7 — the number that appears in comparison articles.
• Gemini 3.1 Pro's average determinism is 81.9% versus Opus 4.7's 91.4%. At 5-step pipelines that gap means a 2× difference in pipeline success rate (37% vs. 61%) — each failed run requiring a full retry.
• The biggest hidden cost is prompt caching misses: a 10K-token system prompt at full price on 10,000 calls/day costs $500/day; with caching, $50/day.
• Tool-call tokens are billed as input on every call: 10 tool definitions (~600 tokens) adds $9 per 1,000 calls at Opus pricing.

- List pricing: Opus 4.7 $5/$25/M in/out; GPT-5.5 $5/$30/M; Gemini 3.1 Pro $2/$12/M — but list pricing omits retry rates, caching hit rates, and tool-call overhead that dominate real bills.
- The biggest hidden cost is prompt caching misses: a 10K-token system prompt billed at full price on every call adds $500/day at Opus pricing at 10,000 calls/day, vs. $50/day with caching.
- Tool definition tokens are billed as input tokens on every call; a system with 10 tool definitions (~600 tokens) adds $9 per 1,000 calls at Opus pricing.

Why pricing pages are misleading

The standard comparison table omits retry rate, caching hit rate, tool-call overhead, output amplification, and context efficiency losses. A real cost model:

cost_per_task = (
    prompt_tokens_uncached × input_price
  + prompt_tokens_cached × cache_price
  + output_tokens × output_price
  + tool_tokens × input_price
) × (1 / determinism_rate)^pipeline_steps

The retry multiplier (1 / determinism_rate)^pipeline_steps is the single biggest divergence between pricing page and actual bill. ^[5]

- A real cost model accounts for retry rate, caching hit rate, tool-call token overhead, output amplification across pipeline steps, and context window efficiency — none of which appear on pricing pages.
- The retry multiplier formula is `(1 / determinism_rate)^pipeline_steps` — the single biggest driver of divergence between pricing page cost and actual bill.
- Preview model endpoints carry a hidden reliability tax beyond token cost: lower quotas, more frequent 429/503 errors, and shorter deprecation cycles all increase effective cost of ownership.

The retry multiplier in practice

Representative 3-step pipeline: 2,000-token system prompt, 200-token message, 800-token tool definitions, 400-token output per step.

Without caching, no retries:

Gemini is 2.3× cheaper than Opus with no retries. GPT-5.5 and Opus are within 8% of each other at list price — the critical differentiator is reliability under retries. This is the number in the comparison article.

Now apply determinism-driven retries (category-5 complexity, multi-tool sequence — Opus 94%, GPT-5.5 90%, Gemini 84%):

Pipeline success probability: Opus 0.94³ = 83%; GPT-5.5 0.90³ = 73%; Gemini 0.84³ = 59%.

Gemini is still cheapest — but the ratio has compressed from 2.3× to 1.7× against Opus. GPT-5.5's retry overhead pushes it to $0.111, roughly 23% above Opus after retries — despite matching on input price. At higher complexity the gap widens further: a 14-point determinism gap is 1.8× at 3 steps but 7.2× at 10 steps. Multi-agent systems with planning, tool-selection, and error-handling routinely reach 5–10 action steps per task.

<Callout type="hot"> The inversion is real. At category-9 complexity (ambiguous-input, multi-tool), Gemini 3.1 Pro crosses above Opus 4.7 in cost-per-task at pipeline length ≥ 5 steps. If your agentic system has 5+ action steps on hard inputs, the pricing page comparison is actively misleading. Run your determinism scores through the retry multiplier before making a cost decision. </Callout>

- At ambiguous-input complexity on a long pipeline, the cost ordering can invert: Gemini's higher retry rate more than offsets its lower per-token price.
- The cost break-even between Gemini 3.1 Pro and Opus 4.7 at ambiguous-input complexity occurs between 4 and 5 pipeline steps; beyond 5 steps, Opus wins on cost-per-task.
- The retry multiplier scales as `1 / determinism^n` — a 14-point determinism gap is 1.8× at 3 steps but grows to a 7.2× difference at 10 steps.

Prompt caching: the underrated cost lever

At 10,000 calls/day with a 10K-token system prompt:

All three providers give a 90% discount on cached tokens. The Gemini-vs-Opus and Gemini-vs-GPT-5.5 2.5× per-token ratio is preserved with caching since all platforms apply the same 90% discount.

Caching gotchas: Anthropic's cache TTL is 5 minutes — calls more than 5 minutes apart restart the cache; minimum cacheable prefix is 4,096 tokens. OpenAI caches automatically at a 90% discount with a 128-token minimum. Google's context caching requires explicit API creation with a configurable TTL (not automatic), but the 90% discount is competitive for large, stable system prompts.

- Anthropic caches at 4,096+ token boundaries for current flagship models with a 5-minute TTL and 90% discount on cached tokens; calls more than 5 minutes apart restart the cache.
- OpenAI's cache is automatic with a 90% discount and 128-token minimum; Google's context caching requires explicit API creation with configurable TTL and also gives a 90% discount.
- Cache hit rate depends on call timing: batch workloads with irregular intervals can have much lower actual cache hit rates than the theoretical maximum.

The three workload archetypes, costed

Archetype A: Coding agent (multi-step, tool-heavy)

Representative profile: 8,000-token system prompt cached after first call; 3,000 token average input; 800 token output; 5 steps; category 5–7 schemas.

GPT-5.5 with strict: true delivers the best pipeline success rate (70%) at the lowest cost among the top-two performers (~$0.31 vs Opus's ~$0.42) — a better value than pricing pages suggest, because its determinism advantage reduces expected retries more than the slight output-price premium adds. Gemini ($0.28) is marginally cheaper but requires robust retry infrastructure at 31% pipeline success. ^[4]

Archetype B: Document Q&A (long-context, single query)

Representative profile: 80K-token document; 500-token system prompt; 600-token output; 1 step.

With no pipeline and no retries, Gemini 3.1 Pro wins (2.5× cheaper than either Opus or GPT-5.5, which are now nearly cost-equivalent). Single-step tasks don't compound determinism variance; Gemini wins on cost for retrieval-focused workloads.

Archetype C: High-volume classification (batch, 10M items/month)

Representative profile: 300 tokens per item; 1,000-token system prompt cached; 50 tokens output; 1 step.

Gemini 3.1 Pro wins — saving $45K/month vs. Opus. The simple flat schema (category 1–2) keeps Gemini's determinism at 96–100%, eliminating the reliability advantage of more expensive models. Multi-model routing strategies — cheap model for easy tasks, premium model for complex — can reduce cost-per-task by 30–60%. ^[4][6]

Hands-on exercise

Build a cost-per-task model for your use case using your Chapter 2 benchmark data.

Fill in these numbers from actual benchmark runs (not guesses):

``` USE CASE: [describe in 1 sentence]

TOKEN COUNTS: System prompt tokens: ___ Average user message tokens: ___ Tool definition tokens: ___ Average output tokens: ___ Pipeline steps: ___

CACHING: Is system prompt ≥ 1024 tokens? [Y/N] Estimated cache hit rate: ___ % (Anthropic: use 80% if calls within 5-min windows; 40% if irregular)

DETERMINISM SCORES (from Chapter 2): Opus 4.7: ___ % GPT-5.5: ___ % Gemini 3.1 Pro: ___ %

COST FORMULA (per model): input_cost = (system_prompt × (1 - cache_hit_rate) × INPUT_PRICE) + (system_prompt × cache_hit_rate × CACHE_PRICE) + (message_tokens + tool_tokens) × INPUT_PRICE output_cost = output_tokens × OUTPUT_PRICE retry_multiplier = 1 / (determinism ^ pipeline_steps) cost_per_task = (input_cost + output_cost) × retry_multiplier × pipeline_steps

RESULTS: Opus 4.7 cost-per-task: $___ GPT-5.5 cost-per-task: $___ Gemini 3.1 Pro cost-per-task: $___

RECOMMENDATION: [which model and why, in 1 sentence] ```

Your cost model is complete when all token counts are from actual benchmark runs, cache hit rate reflects your actual call pattern, and cost-per-task accounts for retries using your measured determinism scores. Estimated time: 20 minutes.

What's next

You have a scorecard (ch01), determinism scores (ch02), context fidelity data (ch03), and a cost-per-task model (ch04). The capstone project synthesizes all four into a model selection memo — format in vault/courses/picking-a-frontier-model-2026-q2/outline.md.

References cited

<sup>[1]</sup>: Anthropic. "Claude pricing." https://www.anthropic.com/pricing — Opus 4.7 input/output/cache pricing as of Q2 2026. Also: "Prompt caching." https://www.anthropic.com/news.

<sup>[2]</sup>: OpenAI. "OpenAI API pricing." https://developers.openai.com/api/docs/pricing — GPT-5.5 $5/$30/M input/output; cached input $0.50/M (90% discount). Verified 2026-06-14. Model release notes: https://developers.openai.com/api/docs/models.

<sup>[3]</sup>: Google. "Gemini API pricing." https://ai.google.dev/pricing — Gemini 3.1 Pro input/output/context caching pricing as of Q2 2026. Changelog: https://ai.google.dev/gemini-api/docs/changelog.

<sup>[4]</sup>: Koenig AI Academy internal cost model data, Q2 2026. Derived from the Q2 2026 tool-use determinism benchmark dataset (reference tables embedded in Chapter 2) with retry simulation applied at workload scale.

<sup>[5]</sup>: Patil, S. et al. Berkeley Function-Calling Leaderboard (BFCL) V4. https://gorilla.cs.berkeley.edu/leaderboard.html — analysis of tool-call reliability impact on pipeline cost.

<sup>[6]</sup>: Chen, L. et al. (2023). "FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance." https://arxiv.org/abs/2305.05176 — analysis of model routing, cascading, and selection strategies that reduce cost-per-task by matching task complexity to model capability.

> Prerequisites: Chapter 1 required. The three deployment questions this chapter separates (model access, endpoint governance, agent/tool controls) build directly on the evaluation-dimensions framework from Chapter 1. > > Time: 50 minutes > > Learning objectives: By the end of this chapter, you can describe the Trusted Access for Cyber program, contrast it with Anthropic's Glasswing path, choose the right deployment architecture for an audit-grade security workflow, and produce a governance case study that would survive a compliance review.

Specialized cyber access programs — OpenAI's Trusted Access for Cyber (TAC) and Anthropic's Project Glasswing — represent a structural shift in how frontier AI capability is delivered to security practitioners. Rather than treating all API customers as equivalent, both vendors now gate their highest-risk cyber capabilities behind eligibility programs that verify team identity, constrain permitted workflows, and impose logging and human-review obligations. As of Q2 2026, GPT-5.5-Cyber is in limited preview to vetted security teams through the TAC program, with Codex included under expanded cyber-permissive access for verified critical-infrastructure defenders. <sup>[1][2]</sup> Anthropic's Project Glasswing serves a parallel function on the Claude side, with the Cyber Verification Program announced as forthcoming and Claude Mythos Preview access currently limited to approximately 50 invited organizational partners. <sup>[4]</sup> The practical question for a security engineering team in 2026 is not simply "which model has the best cyber-capability benchmark score?" but rather "which access path, endpoint architecture, and governance structure can we operate, audit, and defend?"

Key facts

1. GPT-5.5-Cyber limited preview launched May 7, 2026, rolling out to vetted cybersecurity teams through the Trusted Access for Cyber program. The Codex coding agent surface was subsequently included in the TAC program with fewer output restrictions for verified critical-infrastructure defenders. <sup>[2][9]</sup>
2. The UK's AI Safety Institute (AISI) evaluated GPT-5.5's cyber capabilities before the TAC program launched. The evaluation established the empirical basis for OpenAI's access controls by identifying the uplift risk the model poses for real exploit development. <sup>[6]</sup>
3. OpenAI models — including GPT-5.5 — are available on Amazon Bedrock as of Q2 2026. Bedrock delivery changes endpoint governance (AWS IAM, VPC, CloudTrail) and regional data residency but does not bypass the upstream eligibility requirements that TAC places on the model access tier. <sup>[7][8]</sup>
4. Anthropic's Project Glasswing is the organizational home for Anthropic's cybersecurity capability research and the Cyber Verification Program. The Glasswing initial update documents the scope of Claude Mythos Preview access and the review obligations placed on participating teams. <sup>[4][5]</sup>
5. "Fewer restrictions" in the TAC context is a precise technical term, not a casual claim. The GPT-5.5 system card describes specific use-case categories that shift from blocked to monitored-with-approval for TAC-verified teams — vulnerability triage, patch authoring for CVEs, malware reverse-engineering support, and red-team validation. Exploit weaponization remains blocked. <sup>[3]</sup>
6. Three questions that get conflated: (a) model access eligibility — which teams qualify for TAC or Glasswing verification; (b) endpoint governance — whether you call OpenAI-direct, OpenAI Enterprise, or Bedrock, and what controls each adds; (c) agent and tool controls — which Codex tools or Claude tool-use surfaces are permitted in scope, what logging is required, and who reviews flagged outputs. These three layers are independent; getting one right does not substitute for the others.
7. The AISI evaluation found that GPT-5.5 provides meaningful uplift to attackers with some existing security knowledge. <sup>[6]</sup> This finding is consistent with TAC's eligibility focus on demonstrated defensive expertise rather than organizational affiliation alone — a risk-calibrated interpretation, though the direct causal link is this author's inference rather than a documented AISI finding.

- Trusted Access for Cyber gates GPT-5.5-Cyber behind team identity verification and approved workflow scope — not just an API key upgrade.
- Three questions are often conflated: model access eligibility, endpoint governance (direct/Enterprise/Bedrock), and agent/tool approval controls. Answering one does not answer the others.
- AISI's pre-launch cyber evaluation found meaningful uplift for operators with existing security knowledge. This is consistent with — though not documented as the explicit basis for — TAC's requirement for demonstrated defensive expertise rather than organizational affiliation alone.

Why governance is the seventh evaluation dimension

The first four chapters of this course focused on measurement: how reliable is the model's tool-use, how does long-context retrieval degrade, what does production cost actually look like. Those are the dimensions that determine whether a model works on your workload. This chapter addresses a different question — whether you can operate it within the risk and compliance constraints of a security organization.

Governance is not a post-selection consideration. For security teams, it is a primary constraint that eliminates access paths before any technical benchmark is run. A model that scores 94% on tool-use determinism but is unavailable under your organization's cloud provider agreement is not in your shortlist. A model capability that is technically superior but blocked under TAC's approved-use scope for your specific workflow is similarly off the table.

The outline for this course lists governance as the seventh of seven evaluation dimensions in Chapter 1 (alongside latency p95, tool-use determinism, context fidelity at depth, structured-output reliability, cost-per-task, and multimodal fidelity). The reason it is last is not that it matters least — it is that it is the filter you apply after confirming the model is technically capable. If a model fails on determinism or long-context retrieval, governance is irrelevant. If it passes, governance determines which of the viable paths is actually available to you.

For security teams specifically, governance is often the dimension with the longest lead time. Model access eligibility verification — TAC or Glasswing — can take weeks from application to approval. Endpoint procurement through an Enterprise agreement or Bedrock organizational setup takes time to negotiate and configure. Tool approval controls require internal security review before they can be deployed in a regulated environment. A team that defers governance assessment until after it has validated a model's technical performance will find itself with a technically viable model it cannot legally or operationally deploy. Starting governance evaluation in parallel with technical benchmarking is not overcaution; it is schedule management.

A well-structured model will distinguish model access eligibility questions (does my team qualify for TAC or Glasswing?), endpoint questions (should we use OpenAI-direct, Enterprise, or Bedrock, and what does each add?), and agent/tool control questions (which tools can be in scope, what logging is required?). Notice whether the model conflates eligibility with endpoint selection — most do.

The Trusted Access for Cyber program

OpenAI launched the Trusted Access for Cyber program as a structured mechanism for extending GPT-5.5's cybersecurity capabilities to vetted defensive teams while managing the risk that the same capabilities pose in an attacker's hands. <sup>[1]</sup> The program has three components that operate independently:

Eligibility verification. Teams applying for TAC must demonstrate existing defensive security expertise through organizational attestation, team-level vetting, and use-case description. The program is not available to individual developers, general-purpose security consultancies without demonstrated critical-infrastructure contracts, or research teams without institutional backing. Approval is not automatic on organizational affiliation — the AISI evaluation's finding that uplift scales with attacker existing knowledge is consistent with the requirement for demonstrated defensive expertise, though whether that finding directly shaped the eligibility design is not documented in the AISI publication. <sup>[6]</sup>

Approved-use scoping. TAC approval comes with an explicit scope of permitted workflows. As of Q2 2026, the documented permitted categories include: vulnerability triage on internally-discovered or CVE-published vulnerabilities; patch authoring assistance for known CVEs affecting the team's own systems; malware reverse-engineering support for defensive analysis; and red-team validation where the target system is owned or contracted by the team. Exploit weaponization — producing working exploit code targeting external systems the team does not own — remains blocked regardless of TAC status. The GPT-5.5 system card documents these boundaries. <sup>[3]</sup>

Misuse monitoring. TAC-tier access includes additional output-layer monitoring. OpenAI does not publish the exact technical implementation, but the program's terms require teams to report outputs they believe fall outside the permitted scope and to cooperate with audit requests. This is a behavioral obligation, not just an API configuration.

The AISI's pre-launch evaluation found that GPT-5.5 provides meaningful uplift to operators who already have existing security knowledge — junior security researchers with some CTF background and experienced red-teamers both showed acceleration. <sup>[6]</sup> Consistent with that finding is a risk-calibrated interpretation of TAC's eligibility design: a team with no prior vulnerability research experience is less positioned to translate the model's vulnerability synthesis into actionable exploit steps, while a team with demonstrated security knowledge is precisely the population where misuse risk — and therefore monitoring obligations — creates the most value. Whether the AISI finding directly shaped OpenAI's eligibility criteria is not documented in the AISI publication; the causal link is this author's inference. What the AISI evaluation establishes independently is the empirical basis for why uplift-risk correlates with existing attacker knowledge — which is the premise any eligibility gate of this kind would need to be defensible.

- TAC has three independent components: team eligibility verification, approved-use workflow scoping, and ongoing misuse monitoring obligations.
- The permitted use-case list is explicit in the GPT-5.5 system card: vulnerability triage, patch authoring on your own systems, malware reverse-engineering (defensive), red-team validation of systems you own.
- Exploit weaponization for external systems remains blocked for all access tiers, including TAC.

GPT-5.5-Cyber in Codex: what expands and what stays restricted

When OpenAI included Codex under the Trusted Access for Cyber umbrella, it created a specific workflow path: a TAC-verified team can use Codex's agentic coding surface against security-adjacent tasks with the relaxed output policy applied to their verified-team API calls. <sup>[2]</sup> This is meaningful because Codex's tool-use surface — file access, code execution, web fetch — is exactly the surface that security automation requires. A vulnerability-triage workflow that reads a CVE description, pulls the affected package source, analyzes the diff, and drafts a patch requires multi-step tool use that a chat-completion endpoint alone cannot provide.

What the Codex TAC expansion does not change: the tool approval architecture. Codex's tool-use controls — which tools are in scope for a given deployment, what sandboxing is applied, how outputs are logged — are a separate layer from the TAC eligibility status. A TAC-verified organization that deploys Codex with file-system write access and no output logging has satisfied the model-access eligibility requirement while creating an uncontrolled endpoint governance situation. These are different risks.

The configuration that most security teams should target for a Codex-based vulnerability-triage workflow looks like this:

```yaml # codex-security-workflow.yaml — reference governance config access_tier: trusted_access_for_cyber model: gpt-5.5-cyber approved_use_scope: - vulnerability_triage - patch_authoring - malware_reverse_engineering_defensive

tool_allowlist: - file_read # read source files, CVE descriptions, advisories - code_execution # run analysis scripts in sandboxed container - web_fetch # pull NVD entries, vendor advisories (allow-listed domains) # file_write: EXCLUDED — patches are reviewed by human before commit # shell: EXCLUDED — full shell access not required for triage workflow

sandbox: network: restricted # only allow-listed domains filesystem: read_only # no writes from agent; human commits patch execution_timeout: 60s

logging: output_capture: full flagged_output_review: human_within_4h retention: 90_days # match SOC retention policy

escalation: misuse_signal: page_security_lead_immediately borderline_output: flag_and_hold_pending_human_review ```

This is a governance-first configuration, not a capability-first one. The tool-allowlist excludes file_write and shell deliberately — not because the model cannot use them, but because an audit-grade workflow requires human review before any patch lands in source control. The sandbox network restriction to allow-listed domains prevents the agent from fetching arbitrary external content during analysis. These controls exist in addition to the TAC eligibility layer, not as a substitute for it.

Anthropic's Project Glasswing and the Cyber Verification Program

Anthropic's parallel structure is Project Glasswing — the organizational unit responsible for Anthropic's work on cybersecurity capability research and the access controls that govern Claude Mythos Preview. <sup>[4]</sup> Where OpenAI's TAC is framed primarily as a deployment program (how verified teams access GPT-5.5-Cyber), Glasswing is framed as a research-and-governance program: it simultaneously researches Claude's offensive cyber capabilities, develops defenses and mitigations, and governs which external teams get access to the Mythos Preview under what conditions.

Anthropic has announced a forthcoming Cyber Verification Program as part of Glasswing; as of Q2 2026, access is limited to approximately 50 invited organizational partners — including Cloudflare, Microsoft, and Oracle. <sup>[4][5]</sup> There is no open application process for the CVP. Glasswing-participating teams are expected to share findings about model behavior and misuse signals back to Anthropic, making the relationship bidirectional in a way that TAC's commercial terms do not obviously require.

The Mythos Preview, which drew significant federal attention in April 2026, is the specific Claude model surface available under Glasswing verification. The AISI evaluation of GPT-5.5's cyber capabilities implicitly places Mythos and GPT-5.5-Cyber at comparable capability levels — both provide meaningful uplift to operators with existing security knowledge. <sup>[6]</sup> The governance structures that gate access to each model are the differentiator, not the raw capability ceiling.

One operational difference that matters for compliance teams: based on Anthropic's announced model for the forthcoming CVP, the program does not publish a fixed permitted-use taxonomy in the same way the GPT-5.5 system card does. <sup>[3]</sup> The Glasswing initial update describes an approach where use-case scope would be documented per approved team rather than drawn from a published list. This model creates more flexibility for novel or research-adjacent workflows — a team analyzing an emerging threat actor's toolchain may not fit any pre-defined category cleanly, and Glasswing's described approach accommodates scope discussion. The cost is that scope boundaries are less legible to auditors who are not party to the individual verification agreement. For organizations that need to demonstrate compliance to a third-party auditor — a SOC 2 Type II audit, a FedRAMP authorization, or an internal governance board — a fixed, published permitted-use list from the GPT-5.5 system card is easier to reference than a bespoke agreement. Neither approach is objectively superior; the choice depends on whether your compliance environment rewards specificity or flexibility.

Glasswing also operates with an explicit assumption that the capability landscape will evolve. The initial update notes that permitted-use categories and monitoring requirements will be updated as Anthropic's internal research on Mythos's offensive capability ceiling develops. <sup>[5]</sup> This means the governance agreement a team signs today is not static — Anthropic may tighten or loosen restrictions as the empirical picture changes. TAC terms are similarly subject to update, but the system card model creates a version-referenced baseline that auditors and regulators can pin to.

- Project Glasswing is Anthropic's research-and-governance unit for cyber capabilities — Mythos Preview access is currently limited to ~50 invited partners; the Cyber Verification Program is announced but not yet open for applications.
- Mythos Preview and GPT-5.5-Cyber sit at comparable capability levels per AISI's evaluation — the governance model, not raw capability, is the differentiator.
- TAC verification and Glasswing verification are not cross-recognized; teams wanting both access paths must apply to each separately.

Side-by-side: TAC vs. Glasswing

The structural difference that matters most for a security team choosing between the two: TAC is a deployment-tier program where eligibility is confirmed once and the commercial relationship then operates within documented permitted-use boundaries. Glasswing is a research-collaborative program where the relationship is expected to evolve as Claude's capabilities and Anthropic's understanding of the risk surface develop. Teams with stable, well-defined triage workflows that fit the TAC permitted-use list will likely find the TAC path more operationally predictable. Teams with novel research-adjacent workflows — new malware families, emerging vulnerability classes, experimental defensive tooling — may find the Glasswing relationship more appropriate because it creates a channel for scope discussion rather than requiring the workflow to fit a fixed list.

Deployment paths: OpenAI-direct, Enterprise, and Amazon Bedrock

TAC eligibility addresses model access. Endpoint governance — which API surface you call and what platform-level controls wrap it — is a separate decision with material implications for compliance, data residency, and audit. As of Q2 2026, TAC-verified organizations calling GPT-5.5-Cyber have three primary endpoint paths:

OpenAI-direct (API platform). The standard API path. Controls include: API key management, usage limits, and OpenAI's platform-level logging. Data residency is OpenAI's infrastructure. Compliance certifications (SOC 2 Type II, ISO 27001) apply to the platform broadly. Audit trail is OpenAI's usage logs plus any application-layer logging you implement. This is the fastest path to get a TAC-verified workflow running but offers the least organizational control over the data path.

OpenAI Enterprise. Enterprise agreements add: dedicated infrastructure, negotiated data processing terms, admin-level usage controls, and direct account management for compliance discussions. Enterprise contracts can specify data retention, deletion, and usage-for-training opt-outs. For security teams in regulated industries (HIPAA, FedRAMP-adjacent), Enterprise is often the minimum viable path — not because the model differs, but because the contractual and infrastructure controls satisfy requirements that the standard API platform cannot.

OpenAI models on Amazon Bedrock. AWS made GPT-5.5 available on Bedrock as of Q2 2026. <sup>[7][8]</sup> Bedrock delivery adds:

```hcl # Example: IAM policy restricting GPT-5.5 Bedrock invocations # to a specific security-team role in a specific region resource "aws_iam_policy" "gpt55_cyber_invoke" { name = "gpt55-cyber-invoke-tac-team" description = "Restrict GPT-5.5 Bedrock invocations to TAC security team role"

policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Action = ["bedrock:InvokeModel"] # Replace with the actual model ID from https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards-openai.html Resource = "arn:aws:bedrock:us-east-1::foundation-model/$BEDROCK_GPT55_MODEL_ID" Condition = { StringEquals = { "aws:PrincipalTag/team" = "security-tac-verified" } } } ] }) } ```

AWS-side controls that Bedrock adds on top of OpenAI's platform controls: - IAM role-based access: only specific IAM principals can invoke the GPT-5.5 Bedrock endpoint — enforce at the AWS control plane, not just the application layer - VPC endpoints: keep traffic off the public internet entirely, required for many FedRAMP and DoD IL2/IL4 environments - CloudTrail logging: every Bedrock InvokeModel call is logged in CloudTrail, giving you AWS-native audit trails that integrate with your existing SIEM - Regional data residency: GPT-5.5 on Bedrock can be constrained to specific AWS regions, satisfying data-residency requirements that OpenAI-direct cannot

The trade-off: Bedrock adds latency (a hop through AWS infrastructure), requires AWS-specific IAM configuration overhead, and may not have the same model-version rollout cadence as OpenAI-direct. For security teams already deep in the AWS ecosystem with existing CloudTrail/SIEM infrastructure, the Bedrock path is typically worth the overhead. For teams outside AWS, it adds friction without adding proportionate benefit.

<Callout type="hot"> Bedrock delivery does not bypass TAC eligibility. A team that is not TAC-verified calling GPT-5.5-Cyber through Bedrock still runs the standard GPT-5.5 output policy — not the TAC-relaxed policy. AWS-side controls (IAM, VPC, CloudTrail) are endpoint governance, not model access eligibility. The TAC verification happens at the OpenAI application layer and applies regardless of which deployment path delivers the API call. This is the most common misconception teams make when evaluating Bedrock as a compliance shortcut. </Callout>

- Three endpoint paths for TAC-verified GPT-5.5-Cyber: OpenAI-direct (fastest, least platform control), Enterprise (contractual controls for regulated industries), Bedrock (AWS IAM/VPC/CloudTrail/region for AWS-native compliance stacks).
- Bedrock adds endpoint governance — IAM access control, CloudTrail audit, regional residency — but does not substitute for TAC eligibility at the OpenAI application layer.
- For FedRAMP-adjacent or DoD environments, Bedrock + VPC endpoints is frequently the minimum viable path; Enterprise contracts are an additional layer, not an alternative.

Writing your governance case study

The hands-on exercise for this chapter asks you to produce a governance case study for one of four scenarios. Before you write it, here is the schema that distinguishes a defensible case study from a policy-shaped prose paragraph:

```markdown # Governance Case Study: [Scenario Name]

Scenario One paragraph: what the team is trying to accomplish, what the target system is, and what the risk surface is if the AI workflow produces a harmful output.

Eligibility signals - Team credential: [e.g., CISA-verified critical-infrastructure defender] - Organizational backing: [e.g., Fortune 500 financial institution with SOC 2 Type II] - Existing expertise signal: [e.g., 5-member red-team with CVE credits, lead holds OSCP] - Use-case category: [which TAC or Glasswing permitted category this maps to]

Access path - Model: [GPT-5.5-Cyber via TAC | Claude Mythos via Glasswing] - Endpoint: [OpenAI-direct | Enterprise | Bedrock | Anthropic-direct] - Rationale: [one sentence on why this endpoint path for this compliance context]

Permitted workflows - [Specific task type 1, with scope constraint] - [Specific task type 2, with scope constraint]

Blocked workflows - [Task that is excluded from scope and why] - [Task that is excluded even though it looks adjacent]

Tool controls - Allowed tools: [list] - Excluded tools: [list + rationale] - Sandbox: [network restriction, filesystem access level, execution timeout]

Logging and retention - Output capture: [full | summary | flagged-only] - Flagged output review: [human, within X hours] - Retention: [X days, matched to SOC/compliance requirement]

Escalation - Borderline output: [hold + human review within X hours] - Clear misuse signal: [page security lead immediately + suspend session] - Program reporting: [report to OpenAI TAC / Anthropic Glasswing within X hours] ```

Three things to notice about this template:

1. Permitted and blocked workflows are listed together. Security teams that only document what they are allowed to do, without documenting what they are explicitly blocking, create policy gaps that misuse hides in. A governance case study that says "allowed: vulnerability triage" but does not say "blocked: exploit code targeting external systems the team does not own" has not provided meaningful scope control.

1. Tool controls are independent of model access. A case study that specifies the TAC access tier but leaves tool controls blank has answered the eligibility question and left the deployment question open. Both must be present.

1. Escalation paths have time SLAs. "Human review" is not an escalation path — it is a category. "Flag and hold pending human review within 4 hours, with page to security lead if hold exceeds threshold" is an escalation path. The distinction matters when an auditor reviews the policy.

Hands-on exercise

Write a governance case study for one of the following four scenarios:

1. Critical-infrastructure vulnerability triage: A power utility's internal security team wants to use an AI coding assistant to accelerate CVE triage on their SCADA control software. They are CISA-registered critical-infrastructure defenders.

1. Open-source supply-chain patch review: A financial institution's AppSec team needs to analyze suspicious commits in open-source dependencies before pulling updates. They suspect an active supply-chain insertion campaign.

1. Malware reverse-engineering support: A threat intelligence team is analyzing a novel ransomware variant targeting healthcare infrastructure. No CVE exists yet; they are doing primary analysis.

1. Internal red-team validation: A large technology company's red team wants to use an AI model to accelerate coverage of their own production API surface during a scheduled red-team engagement.

Using the schema above, fill in all sections: - Eligibility signals (include at least 3 concrete signals, not generic placeholders) - Access path with rationale (choose between TAC/Glasswing and endpoint type — justify the choice) - Permitted workflows (at least 2 specific tasks with scope constraints) - Blocked workflows (at least 2 — one obvious, one adjacent-but-excluded) - Tool controls (complete tool allowlist and exclusion rationale) - Logging and retention (specific SLAs) - Escalation (time-bounded paths for borderline and clear-misuse signals)

Success criteria: - All sections are filled in with specific, concrete entries — no placeholders - Permitted and blocked workflows are listed together, not in isolation - Tool controls are independent of the model access tier specification - Escalation paths include time SLAs - The case study distinguishes between model access eligibility (TAC or Glasswing), endpoint governance (which deployment path), and agent/tool controls — at minimum as separate sections, ideally cross-referenced

Estimated time: 30–45 minutes. This deliverable is also the core of the capstone project governance section — saving it in your course notes means you can import it directly into the model-selection memo.

- A governance case study is not complete without both permitted and blocked workflows — gaps are where misuse hides.
- Tool controls must be specified independently of model access tier — TAC eligibility does not imply tool scope or sandbox controls.
- Escalation paths require time SLAs; "human review" without a time bound is not a functioning escalation path.

The next chapter applies the full set of evaluation dimensions — determinism, long-context behavior, cost-per-task, and governance — to the capstone project: a defensible model-selection memo for your specific production use case. See courses/picking-a-frontier-model-2026-q2/capstone-model-selection-memo.

References

<sup>[1]</sup>: OpenAI. "Trusted Access for Cyber." https://openai.com/index/trusted-access-for-cyber/

<sup>[2]</sup>: OpenAI. "GPT-5.5 with Trusted Access for Cyber." https://openai.com/index/gpt-5-5-with-trusted-access-for-cyber/ — May 7, 2026 announcement of GPT-5.5-Cyber limited preview and Codex inclusion under TAC.

<sup>[3]</sup>: OpenAI. "GPT-5.5 System Card." https://deploymentsafety.openai.com/gpt-5-5/gpt-5-5.pdf — documents the specific use-case categories that shift from blocked to monitored-with-approval for TAC-verified teams.

<sup>[4]</sup>: Anthropic. "Project Glasswing." https://www.anthropic.com/glasswing — Anthropic's cybersecurity capability research and governance unit; home of the Cyber Verification Program governing Mythos Preview access.

<sup>[5]</sup>: Anthropic. "Glasswing Initial Update." https://www.anthropic.com/research/glasswing-initial-update — documents scope of Mythos Preview access and review obligations for participating teams.

<sup>[6]</sup>: AISI. "Our evaluation of OpenAI's GPT-5.5 cyber capabilities." https://www.aisi.gov.uk/blog/our-evaluation-of-openais-gpt-5-5-cyber-capabilities — UK AI Safety Institute pre-launch evaluation establishing the empirical basis for TAC eligibility design; finding that uplift scales with attacker existing knowledge.

<sup>[7]</sup>: Amazon Web Services. "OpenAI models on Amazon Bedrock." https://aws.amazon.com/bedrock/openai/ — GPT-5.5 delivery via Bedrock with AWS-native IAM, VPC, CloudTrail, and regional controls.

<sup>[8]</sup>: Amazon Web Services. "OpenAI model cards — Amazon Bedrock." https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards-openai.html — model card documentation for OpenAI models on Bedrock.

<sup>[9]</sup>: CNBC. "OpenAI rolls out new GPT-5.5 Cyber to vetted cybersecurity teams." https://www.cnbc.com/2026/05/07/openai-rolls-out-new-gpt-5point5-cyber-to-vetted-cybersecurity-teams.html — third-party coverage of the May 7, 2026 TAC launch.

Related chapters in this course: [[courses/picking-a-frontier-model-2026-q2/01-dimensions-that-matter]] | [[courses/picking-a-frontier-model-2026-q2/02-tool-use-determinism-benchmark]] | [[courses/picking-a-frontier-model-2026-q2/04-cost-per-task]]

Related blogs: [[blogs/sub-hour-zero-days-aisi-mythos-autonomous-cyber-developers]] | [[blogs/gpt-5-5-in-codex]]