Skip to main content
POST
Python

Authorizations

Authorization
string
header
required

nia_sk_...

Body

application/json

What a runner measured in the company's CI for one tool and one element (the tool counterfactual spec).

cases
CounterfactualCase · object[]
required
Required array length: 1 - 5000 elements
element
enum<string>
required
Available options:
constraints,
hard,
size,
exclude
tool
string
required
Pattern: ^[A-Za-z][A-Za-z0-9_.:-]{0,63}$
k
integer
default:10

The positions compared when a recorded list has no visible_k.

Required range: 1 <= x <= 100
label
string | null

A name for the run, such as the commit or the build it ran on.

Required string length: 1 - 256

Response

The report, computed over the completed cases.

Whether the element changes what the tool returns, beyond the tool's own noise. It never says whether the result got better, nor how the model reacts: limits says so, and what else holds the answer back.

above_noise
integer
required
below_noise
integer
required

Completed cases whose overlap is below their own noise.

cases
integer
required
completed
integer
required
created_at
string<date-time>
required
effect
number | null
required

noise_floor - overlap: how much of the first positions the element moves beyond noise.

element
enum<string>
required
Available options:
constraints,
hard,
size,
exclude
engaged
CounterfactualEngagement · object
required

Where the items the person engaged with went without the element, over the completed cases.

limits
enum<string>[]
required
Available options:
not_quality,
model_reaction_not_measured,
trivial_for_hard,
few_cases,
noisy_tool,
cases_skipped,
dry_run
noise_floor
number | null
required

Mean overlap@k of the base with itself.

overlap
number | null
required

Mean overlap@k of the base and the variant.

p_value
number | null
required

The two-sided sign test of below_noise against above_noise: how likely a split this uneven is when the element changes nothing.

run_id
string
required
Required string length: 1 - 512
ties
integer
required
tool
string
required
label
string | null
skipped
Skipped · object

The cases not completed.