> ## Documentation Index > Fetch the complete documentation index at: https://docs.niadra.com/llms.txt > Use this file to discover all available pages before exploring further. # Report a run > The executions your CI ran; the answer carries the statistical verdict already decided: `pass`, `flaky`, `regression`, `pin_mismatch` or `infrastructure_error`. ## OpenAPI ````yaml openapi/en/cell.json POST /v1/scenario-runs openapi: 3.1.0 info: title: Niadra data API version: '1' description: >- Writing, context, history, objects, identity, privacy and governance of one space. Every space has a stable address, with the space and the region in its name. servers: - url: https://{space}.{region}.api.niadra.com variables: space: default: acme-prod description: The space, which comes in the source key. region: default: us-east-2 description: The region of the space, which also comes in the key. security: [] paths: /v1/scenario-runs: post: tags: - turns summary: Report a run description: >- The executions the company's CI ran; the answer carries the verdict already decided. **Authentication.** Source key with the `replay` scope, or a Console person token with the `integration`, `security` role (or admin). It answers only in a space with the `turns` feature on (the `features` document); in a space without it, 404. operationId: create_scenario_run_v1_scenario_runs_post parameters: - in: header name: Idempotency-Key required: true schema: maxLength: 256 minLength: 1 title: Idempotency-Key type: string description: >- Required. Kept 24 hours with the hash of the body: a retry with the same key returns the first answer, and the same key with a different body returns 409 `conflict`. requestBody: content: application/json: schema: $ref: '#/components/schemas/ScenarioRunCreate' required: true responses: '201': content: application/json: schema: $ref: '#/components/schemas/ScenarioRun' description: The run, with its verdict decided. '422': content: application/problem+json: schema: $ref: '#/components/schemas/Problem' description: The request does not match the contract. security: - sourceKey: [] - personToken: [] x-codeSamples: - lang: python label: Python source: >- # The Replayer reports the run for you and returns Niadra's verdict run = Replayer(niadra, build_agent, build=BUILD).run(["sc_01J9..."], runs=5, vary=["prompts"]) print(run.verdict) # pass, flaky, regression, pin_mismatch or infrastructure_error - lang: typescript label: TypeScript source: >- // The Replayer reports the run for you and resolves with Niadra's verdict const run = await new Replayer(niadra, buildAgent, { build: BUILD }).run(["sc_01J9..."], { runs: 5, vary: ["prompts"] }); console.log(run.verdict); // pass, flaky, regression, pin_mismatch or infrastructure_error components: schemas: ScenarioRunCreate: additionalProperties: false description: >- What the company's CI reports after running the scenarios N times: the statistics stay with Niadra. Every pin the recording requires must match each scenario's recorded build, except those in `vary`. properties: build: $ref: '#/components/schemas/TurnBuild' mode: default: hermetic_turn enum: - hermetic_turn - hermetic_conversation - era_memory title: Mode type: string results: items: $ref: '#/components/schemas/ReplayResult-Input' maxItems: 5000 minItems: 1 title: Results type: array runs: anyOf: - maximum: 100 minimum: 1 type: integer - type: 'null' title: Runs scenario_ids: items: maxLength: 512 minLength: 1 type: string maxItems: 200 minItems: 1 title: Scenario Ids type: array vary: items: enum: - prompts - corpus_digest - model - assembler - tool_schemas type: string maxItems: 5 title: Vary type: array required: - scenario_ids - build - results title: ScenarioRunCreate type: object ScenarioRun: additionalProperties: false properties: build: $ref: '#/components/schemas/TurnBuild' created_at: format: date-time title: Created At type: string mode: enum: - hermetic_turn - hermetic_conversation - era_memory title: Mode type: string run_id: maxLength: 512 minLength: 1 title: Run Id type: string status: enum: - pending - done title: Status type: string summary: $ref: '#/components/schemas/ScenarioRunSummary' vary: items: enum: - prompts - corpus_digest - model - assembler - tool_schemas type: string title: Vary type: array verdict: anyOf: - enum: - pass - flaky - infrastructure_error - pin_mismatch - regression type: string - type: 'null' description: The worst of the scenarios' verdicts. title: Verdict required: - run_id - status - mode - build - summary - created_at title: ScenarioRun type: object Problem: additionalProperties: false description: RFC 9457 problem details; `code` comes from the versioned error catalog. properties: code: title: Code type: string detail: anyOf: - type: string - type: 'null' default: null title: Detail request_id: anyOf: - type: string - type: 'null' default: null title: Request Id status: title: Status type: integer title: title: Title type: string type: default: about:blank title: Type type: string required: - title - status - code title: Problem type: object TurnBuild: additionalProperties: false properties: adapter: anyOf: - maxLength: 256 minLength: 1 type: string - type: 'null' description: The framework adapter and its version, if any. title: Adapter pins: $ref: '#/components/schemas/TurnPins' sdk: anyOf: - maxLength: 256 minLength: 1 type: string - type: 'null' description: E.g. `niadra-python/0.7.0`. title: Sdk title: TurnBuild type: object ReplayResult-Input: additionalProperties: false description: >- One execution of one case. `infrastructure_error` (the agent raised, a blob could not be fetched or did not match its digest, a timeout) is counted apart and never fails an assertion; `pin_mismatch` is a case the runner stopped at its pin check. properties: assertions: items: $ref: '#/components/schemas/AssertionOutcome' maxItems: 50 title: Assertions type: array case_id: anyOf: - maxLength: 512 minLength: 1 type: string - type: 'null' title: Case Id divergent_calls: default: 0 minimum: 0 title: Divergent Calls type: integer error: anyOf: - maxLength: 500 type: string - type: 'null' title: Error latency_ms: anyOf: - minimum: 0 type: integer - type: 'null' title: Latency Ms paraphrase: default: false description: The run's input was a paraphrase of the recorded one. title: Paraphrase type: boolean run: maximum: 100 minimum: 1 title: Run type: integer scenario_id: maxLength: 512 minLength: 1 title: Scenario Id type: string status: enum: - completed - pin_mismatch - infrastructure_error title: Status type: string turn_id: pattern: ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$ title: Turn Id type: string required: - scenario_id - turn_id - run - status title: ReplayResult type: object ScenarioRunSummary: additionalProperties: false properties: scenarios: items: $ref: '#/components/schemas/ScenarioVerdict' title: Scenarios type: array title: ScenarioRunSummary type: object TurnPins: additionalProperties: false description: >- What must be the same for a replay to reproduce this turn. The recording document says which are required; a turn without one of them is kept, marked not replayable. properties: assembler: anyOf: - maxLength: 128 minLength: 1 type: string - type: 'null' description: The version of the company's context assembler. title: Assembler corpus_digest: anyOf: - pattern: ^sha256:[0-9a-f]{64}$ type: string - type: 'null' description: >- Digest of the files the agent consults, computed in the SDK; never the files. title: Corpus Digest model: anyOf: - maxLength: 128 minLength: 1 type: string - type: 'null' description: The exact model the agent called. title: Model niadra: anyOf: - $ref: '#/components/schemas/TurnCompilerPins' - type: 'null' prompts: additionalProperties: maxLength: 128 minLength: 1 type: string description: Prompt name -> version, e.g. `core` -> `v16`. maxProperties: 50 propertyNames: maxLength: 256 minLength: 1 title: Prompts type: object tool_schemas: additionalProperties: pattern: ^sha256:[0-9a-f]{64}$ type: string description: Tool name -> hash of its schema. maxProperties: 200 propertyNames: maxLength: 256 minLength: 1 title: Tool Schemas type: object title: TurnPins type: object AssertionOutcome: additionalProperties: false properties: detail: anyOf: - maxLength: 500 type: string - type: 'null' title: Detail id: pattern: ^[a-z0-9][a-z0-9_.-]{0,63}$ title: Id type: string kind: enum: - tool_called - tool_not_called - hard_respected - no_denial_with_results - claims_traced - claims_match_state - no_promise_without_action - expected_in_topk - handoff_when - effect_once - budget - lexicon - tools_offered_match title: Kind type: string outcome: enum: - pass - fail - not_checked title: Outcome type: string required: - id - kind - outcome title: AssertionOutcome type: object ScenarioVerdict: additionalProperties: false properties: assertions: items: $ref: '#/components/schemas/AssertionStats' title: Assertions type: array baseline_run_id: anyOf: - maxLength: 512 minLength: 1 type: string - type: 'null' description: The earlier run compared against; none when it was the recording. title: Baseline Run Id completed: title: Completed type: integer infrastructure_errors: title: Infrastructure Errors type: integer needs_paraphrase: description: >- An assertion passed and failed across the runs: its next run must include paraphrases. title: Needs Paraphrase type: boolean pin_mismatches: title: Pin Mismatches type: integer scenario_id: maxLength: 512 minLength: 1 title: Scenario Id type: string verdict: enum: - pass - flaky - infrastructure_error - pin_mismatch - regression title: Verdict type: string required: - scenario_id - verdict - completed - infrastructure_errors - pin_mismatches - needs_paraphrase - assertions title: ScenarioVerdict type: object TurnCompilerPins: additionalProperties: false properties: compiler: anyOf: - maxLength: 128 minLength: 1 type: string - type: 'null' description: The context compiler's version, from the pack. title: Compiler pack_hash: anyOf: - maxLength: 128 minLength: 1 type: string - type: 'null' description: The hash of the pack this turn read. title: Pack Hash title: TurnCompilerPins type: object AssertionStats: additionalProperties: false description: >- One assertion over the run's completed executions, against its baseline. `p_value` is the one-sided Fisher exact test that the run passes less often than the baseline, rounded to 6 decimals. properties: baseline_failed: title: Baseline Failed type: integer baseline_passed: title: Baseline Passed type: integer drop: description: The baseline's pass rate minus the run's. title: Drop type: number failed: title: Failed type: integer flaky: title: Flaky type: boolean id: pattern: ^[a-z0-9][a-z0-9_.-]{0,63}$ title: Id type: string not_checked: title: Not Checked type: integer p_value: title: P Value type: number passed: title: Passed type: integer regression: title: Regression type: boolean required: - id - passed - failed - not_checked - baseline_passed - baseline_failed - drop - p_value - regression - flaky title: AssertionStats type: object securitySchemes: sourceKey: type: http scheme: bearer description: nia_sk_... personToken: type: http scheme: bearer bearerFormat: JWT ````