P-Value
The probability of observing results at least as extreme as the actual results, assuming the null hypothesis is true, used to assess the strength of evidence against the null hypothesis in statistical testing.
The p-value quantifies how surprising your data is under the assumption that there is no real effect. A p-value of 0.03 means there is a 3% probability of seeing a result this extreme (or more extreme) if the null hypothesis were true. It does not mean there is a 3% probability that the null hypothesis is true; this is a common and important misconception.
In practice, p-values below 0.05 are conventionally considered statistically significant, though this threshold is arbitrary. A p-value of 0.049 and 0.051 represent nearly identical evidence, yet one crosses the threshold and the other does not. This binary interpretation has led to calls for more nuanced statistical reporting, including confidence intervals and effect sizes.
For growth teams running A/B tests, understanding p-values prevents costly decision errors. A very small p-value (0.001) with a large effect size is strong evidence for shipping a change. A borderline p-value (0.04) with a small effect size might not justify the engineering cost of implementation. Bayesian alternatives provide more intuitive probability statements and are increasingly popular in modern experimentation platforms.
Related Terms
Cosine Similarity
A measure of similarity between two vectors based on the cosine of the angle between them, ranging from -1 (opposite) to 1 (identical), commonly used to compare embeddings.
Dimensionality Reduction
Techniques that reduce the number of dimensions in high-dimensional data while preserving meaningful structure, used for visualization, compression, and noise removal.
Batch Inference
Processing multiple ML predictions as a group at scheduled intervals rather than one-at-a-time on demand, optimizing for throughput and cost over latency.
Real-Time Inference
Generating ML predictions on-demand as requests arrive, typically with latency requirements under 200ms for user-facing features.
Data Pipeline
An automated sequence of data processing steps that moves data from source systems through transformations to destination systems, enabling reliable and repeatable data flows across an organization.
ETL (Extract, Transform, Load)
A data integration pattern that extracts data from source systems, transforms it into a structured format suitable for analysis, and loads it into a target data warehouse or database.