Measure medians, not one run

One run is noise: compare medians, and keep lab and field apart.

From nextjs-pagespeed.

Same page, different score

Press each button a few times. Nothing changes between runs.

One runBad
– Not run yet
    n = 1
    Median of 7Good
    – Not run yet
      median(n = 7)
      URL="https://www.example.com/pricing"
for i in 1 2 3 4 5; do
  curl -s "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=$URL&strategy=mobile" \
    | jq '.lighthouseResult.categories.performance.score * 100'
done | sort -n | sed -n '3p'

      Did my fix work?

      The fix saves 50 ms. Drag runs to 1; verdicts flip.

      1 run eachBad
      Before
      –
      After
      –
      Change
      –

      PR says –

      Median of 7 eachGood
      Before
      –
      After
      –
      Change
      –

      PR says –

      Outcome of 1,000 simulated before/after comparisons
      1,000 comparisonsCalled the fix a regressionMeasured change, 10th–90th pct
      1 run each––
      Median of 7––

      Lab run vs field p75

      Re-run each a few times. The lab jumps; field p75 holds.

      LCP, secondsSimulated
      Lab, one throttled run
      –
      Field p75, 200 users
      –
      Gap
      –
      good ≤ 2.5 spoor > 4 s
      curl -s -X POST "https://chromeuxreport.googleapis.com/v1/records:queryRecord?key=$CRUX_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://www.example.com/pricing","formFactor":"PHONE","metrics":["largest_contentful_paint"]}' \
  | jq '.record.metrics.largest_contentful_paint.percentiles.p75'

      Why it matters, pros and cons

      Why it matters

      One run swings with network and CPU jitter. Field p75 is what users got.

      Pros

      • Claims survive a reviewer's rerun.
      • Small real wins stop hiding in noise.
      • Field p75 reflects real devices.

      Cons

      • Each run costs time and API quota.
      • Field data lags 28 days and needs traffic.