Provider certification
What you'll find here
The certification bar OpenAI, Anthropic, Gemini, and local model routes must clear before they ship as governed provider paths — requirements, families, the generated matrix, unsupported behavior, and the deterministic test posture.
Design rationale: ADR 0002 · Provider transport and mode families.
Provider metadata alone is not enough. A provider is MVP-ready only when tests and receipts show the runtime can safely use it in a governed workflow.
MVP requirements
Each certified provider path is checked across:
Chat
Streaming
Tool calls
Structured output
Usage metadata
Retryable errors
Redaction
Receipts
Budgets
Sandbox compatibility
Certification gate.
ProviderCertification records provider family · model
references · requirements that passed · requirements that are blocked
· policy envelope · evidence · receipts · documentation reference.
provider_certification_decision returns
certified only when every MVP requirement is supported
and none is blocked.
Generated matrix
craik provider certification
craik provider certification --provider-id provider_gemini
The generated craik.provider_certification_matrix records one row per
registered provider. Each row is machine-checkable and includes:
Provider id and family
Certification status
certified · fixture_only · unsupported.
Auth posture
Secret reference, no-secret local, or fixture-only.
Model support
Streaming, tools, and structured output
Receipts, budgets, and retry behavior
Sandbox compatibility
Live behavior
live_opt_in · operator_local_endpoint_required · fixture_by_default.
Hosted OpenAI, Anthropic, and Gemini rows are certified when they have
secret references, default model metadata, streaming/tool/structured
capabilities, receipt-producing runtime adapters, retry behavior, and
budget/quota references. Local model rows are certified for the
OpenAI-compatible local runtime path but mark live behavior as
operator-managed because endpoint availability is outside the registry.
The fixture provider remains explicitly fixture_only.
Matrix statuses
| Status | Meaning |
|---|---|
supported | Implemented by registry metadata and runtime code, with deterministic tests. |
fixture_only | Available only through deterministic fixture behavior; not a live model path. |
unsupported | Not declared for that provider row. |
Unsupported or fixture-only cells are intentional operator signals.
For example, provider_fixture_local is not a live route, while some
local presets do not declare streaming support even though they can use
the same OpenAI-compatible runtime shape for non-streaming execution.
Provider families
openaianthropicgeminichat_completionsfixturePublic metadata, receipts, docs, and certification fixtures must not include raw API keys, organization secrets, request bodies containing private task text, or provider console credentials.
Implementation boundary
Verify first, then implement.
Before implementing live API behavior, provider-specific assumptions should be verified against official provider documentation. Certification tests remain deterministic by default and use fixtures unless a live smoke profile is explicitly enabled.
MVP runtime certification
Deterministic tests against
craik.runtime.providers.provider_runtime certify OpenAI, Anthropic,
and OpenAI-compatible routes, while
craik.runtime.providers.provider_runtime_gemini certifies Gemini for:
Request payload construction
Chat · streaming · tools · structured output.
Response normalization
Text · tool calls · structured output · response ids · usage metadata.
Retry decisions
Throttling · transient failures · overloads.
Secret-reference-only configuration
Redacted provider receipts
Explicit live-access gating
Live provider calls remain disabled unless a caller constructs an
adapter with live_enabled=true and supplies credentials through an
external secret resolver. Local model rows additionally require the
operator to run the local endpoint named by the preset.