> ## Documentation Index
> Fetch the complete documentation index at: https://docs.niadra.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Report a counterfactual

> What the runner measured in your CI: whether an element of the constraints block changes what a tool returns, against the tool's own noise. Positions and overlaps, never items.



## OpenAPI

````yaml openapi/en/cell.json POST /v1/measure/counterfactual-runs
openapi: 3.1.0
info:
  title: Niadra data API
  version: '1'
  description: >-
    Writing, context, history, objects, identity, privacy and governance of one
    space. Every space has a stable address, with the space and the region in
    its name.
servers:
  - url: https://{space}.{region}.api.niadra.com
    variables:
      space:
        default: acme-prod
        description: The space, which comes in the source key.
      region:
        default: us-east-2
        description: The region of the space, which also comes in the key.
security: []
paths:
  /v1/measure/counterfactual-runs:
    post:
      tags:
        - signals
      summary: Report a counterfactual
      description: >-
        What a runner in the company's CI measured: whether an element changes
        what a tool returns, against the

        tool's own noise. Positions and overlaps only (the tool counterfactual
        spec).


        **Authentication.** Source key with the `replay` scope, or a Console
        person token with the `analysis` role (or admin). It answers only in a
        space with the `measurement` feature on (the `features` document); in a
        space without it, 404.
      operationId: counterfactual_run_v1_measure_counterfactual_runs_post
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CounterfactualRunCreate'
        required: true
      responses:
        '201':
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/CounterfactualRun'
          description: The report, computed over the completed cases.
        '422':
          content:
            application/problem+json:
              schema:
                $ref: '#/components/schemas/Problem'
          description: The request does not match the contract.
      security:
        - sourceKey: []
        - personToken: []
      x-codeSamples:
        - lang: python
          label: Python
          source: >-
            # niadra counterfactual --tools app.tools:TOOLS --tool
            search_products --element hard --scenario sc_01J9...

            # Runs each recorded call live, with and without the element, and
            reports positions and overlaps only
        - lang: typescript
          label: TypeScript
          source: >-
            // The Python SDK's `niadra counterfactual` command runs the cases;
            the report is read here

            const report = await niadra.api.counterfactualRunRead(runId);

            console.log(report.effect, report.limits);
components:
  schemas:
    CounterfactualRunCreate:
      additionalProperties: false
      description: >-
        What a runner measured in the company's CI for one tool and one element
        (the tool counterfactual

        spec).
      properties:
        cases:
          items:
            $ref: '#/components/schemas/CounterfactualCase'
          maxItems: 5000
          minItems: 1
          title: Cases
          type: array
        element:
          enum:
            - constraints
            - hard
            - size
            - exclude
          title: Element
          type: string
        k:
          default: 10
          description: The positions compared when a recorded list has no `visible_k`.
          maximum: 100
          minimum: 1
          title: K
          type: integer
        label:
          anyOf:
            - maxLength: 256
              minLength: 1
              type: string
            - type: 'null'
          description: A name for the run, such as the commit or the build it ran on.
          title: Label
        tool:
          pattern: ^[A-Za-z][A-Za-z0-9_.:-]{0,63}$
          title: Tool
          type: string
      required:
        - tool
        - element
        - cases
      title: CounterfactualRunCreate
      type: object
    CounterfactualRun:
      additionalProperties: false
      description: >-
        Whether the element changes what the tool returns, beyond the tool's own
        noise. It never says whether

        the result got better, nor how the model reacts: `limits` says so, and
        what else holds the answer

        back.
      properties:
        above_noise:
          title: Above Noise
          type: integer
        below_noise:
          description: Completed cases whose overlap is below their own noise.
          title: Below Noise
          type: integer
        cases:
          title: Cases
          type: integer
        completed:
          title: Completed
          type: integer
        created_at:
          format: date-time
          title: Created At
          type: string
        effect:
          anyOf:
            - type: number
            - type: 'null'
          description: >-
            `noise_floor - overlap`: how much of the first positions the element
            moves beyond noise.
          title: Effect
        element:
          enum:
            - constraints
            - hard
            - size
            - exclude
          title: Element
          type: string
        engaged:
          $ref: '#/components/schemas/CounterfactualEngagement'
        label:
          anyOf:
            - type: string
            - type: 'null'
          title: Label
        limits:
          items:
            enum:
              - not_quality
              - model_reaction_not_measured
              - trivial_for_hard
              - few_cases
              - noisy_tool
              - cases_skipped
              - dry_run
            type: string
          title: Limits
          type: array
        noise_floor:
          anyOf:
            - type: number
            - type: 'null'
          description: Mean overlap@k of the base with itself.
          title: Noise Floor
        overlap:
          anyOf:
            - type: number
            - type: 'null'
          description: Mean overlap@k of the base and the variant.
          title: Overlap
        p_value:
          anyOf:
            - type: number
            - type: 'null'
          description: >-
            The two-sided sign test of `below_noise` against `above_noise`: how
            likely a split this uneven is when the element changes nothing.
          title: P Value
        run_id:
          maxLength: 512
          minLength: 1
          title: Run Id
          type: string
        skipped:
          additionalProperties:
            type: integer
          description: The cases not completed.
          propertyNames:
            enum:
              - completed
              - no_dry_run
              - tool_error
              - infrastructure_error
          title: Skipped
          type: object
        ties:
          title: Ties
          type: integer
        tool:
          title: Tool
          type: string
      required:
        - run_id
        - tool
        - element
        - created_at
        - cases
        - completed
        - overlap
        - noise_floor
        - effect
        - below_noise
        - above_noise
        - ties
        - p_value
        - engaged
        - limits
      title: CounterfactualRun
      type: object
    Problem:
      additionalProperties: false
      description: RFC 9457 problem details; `code` comes from the versioned error catalog.
      properties:
        code:
          title: Code
          type: string
        detail:
          anyOf:
            - type: string
            - type: 'null'
          default: null
          title: Detail
        request_id:
          anyOf:
            - type: string
            - type: 'null'
          default: null
          title: Request Id
        status:
          title: Status
          type: integer
        title:
          title: Title
          type: string
        type:
          default: about:blank
          title: Type
          type: string
      required:
        - title
        - status
        - code
      title: Problem
      type: object
    CounterfactualCase:
      additionalProperties: false
      description: >-
        One recorded tool call run live three times at the same moment: twice
        with the element (the base,

        whose two lists measure the tool's own noise) and once without it (the
        variant). Positions and

        overlaps only: never an item, an argument or a result.
      properties:
        base_count:
          anyOf:
            - minimum: 0
              type: integer
            - type: 'null'
          description: Items in the base's first list.
          title: Base Count
        call_id:
          maxLength: 256
          minLength: 1
          title: Call Id
          type: string
        dry_run:
          default: false
          description: The calls ran as the tool's dry run.
          title: Dry Run
          type: boolean
        engaged:
          items:
            $ref: '#/components/schemas/CounterfactualEngaged'
          maxItems: 50
          title: Engaged
          type: array
        k:
          anyOf:
            - maximum: 100
              minimum: 1
              type: integer
            - type: 'null'
          description: >-
            The positions compared: the recorded list's `visible_k`, else the
            run's `k`.
          title: K
        noise:
          anyOf:
            - maximum: 1
              minimum: 0
              type: number
            - type: 'null'
          description: 'overlap@k of the base''s two lists: the tool''s own noise.'
          title: Noise
        overlap:
          anyOf:
            - maximum: 1
              minimum: 0
              type: number
            - type: 'null'
          description: overlap@k of the base's first list and the variant's.
          title: Overlap
        status:
          description: >-
            `no_dry_run` for a tool that writes state and has no dry run: it is
            never called. `tool_error` when a call failed,
            `infrastructure_error` when the runner could not run the case.
          enum:
            - completed
            - no_dry_run
            - tool_error
            - infrastructure_error
          title: Status
          type: string
        turn_id:
          pattern: ^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$
          title: Turn Id
          type: string
        variant_count:
          anyOf:
            - minimum: 0
              type: integer
            - type: 'null'
          description: Items in the variant's list.
          title: Variant Count
      required:
        - turn_id
        - call_id
        - status
      title: CounterfactualCase
      type: object
    CounterfactualEngagement:
      additionalProperties: false
      description: >-
        Where the items the person engaged with went without the element, over
        the completed cases.
      properties:
        gained:
          description: Within the variant's first `k` and not the base's.
          title: Gained
          type: integer
        items:
          title: Items
          type: integer
        kept:
          description: Within the first `k` of both lists.
          title: Kept
          type: integer
        lost:
          description: Within the base's first `k` and not the variant's.
          title: Lost
          type: integer
        mean_shift:
          anyOf:
            - type: number
            - type: 'null'
          description: >-
            The variant's position minus the base's, averaged over the items
            both lists hold; positive when they fell.
          title: Mean Shift
        shown:
          description: Within the first `k` of the base's list.
          title: Shown
          type: integer
      required:
        - items
        - shown
        - kept
        - lost
        - gained
      title: CounterfactualEngagement
      type: object
    CounterfactualEngaged:
      additionalProperties: false
      description: >-
        An item the person engaged with in the recorded turn, by its 1-based
        position in each live list;

        absent where the list does not hold it.
      properties:
        base:
          anyOf:
            - maximum: 10000
              minimum: 1
              type: integer
            - type: 'null'
          title: Base
        variant:
          anyOf:
            - maximum: 10000
              minimum: 1
              type: integer
            - type: 'null'
          title: Variant
      title: CounterfactualEngaged
      type: object
  securitySchemes:
    sourceKey:
      type: http
      scheme: bearer
      description: nia_sk_...
    personToken:
      type: http
      scheme: bearer
      bearerFormat: JWT

````