Measured in public.
vetd replays the real git history of open-source projects. Each commit is checked incrementally, then rechecked from scratch by a separate tsgo, and every few commits by a stock full tsgo --noEmit. Failed runs are published too.
Summary
- Public repos run
- 4/6
- Median speedup vs full tsgo
- 3×
- Commits replayed
- 120 · 24 oracle
- Mismatches
- 231 /120
Speedup compares a full tsgo --noEmit of the repository at HEAD with vetd's median incremental check over replayed commits. Small repos type-check fast anyway, so their speedups are small. A mismatch is any diagnostic where vetd disagrees with the independent check.
Repositories
| Repository | Files | Full tsgo | vetd p50 | p95 | Commits | Full fallbacks | Oracle runs | Mismatches | Status |
|---|---|---|---|---|---|---|---|---|---|
| commerceNext.js App Router store · pnpm | 65 | 1.26 s | 432 ms | 846 ms | 30 | 17 | 6 | 116 | Mismatch |
| saas-starterNext.js SaaS starter (Drizzle, Stripe) · pnpm | 41 | 511 ms | 275 ms | 501 ms | 30 | 19 | 6 | 39 | Mismatch |
| honoWeb framework library, type-heavy · pnpm | 457 | 65 ms | — | — | — | — | — | — | Did not runat replay |
| excalidrawVite + React app, yarn monorepo · yarn | 695 | 302 ms | 1.40 s | 2.46 s | 30 | 11 | 6 | 8 | Mismatch |
| zodSchema library, pnpm monorepo, extreme generics · pnpm | — | — | — | — | — | — | — | — | Did not runat install |
| taxonomyNext.js 13 App Router app (Prisma, Contentlayer) · pnpm | 125 | 246 ms | 65 ms | 636 ms | 30 | 7 | 6 | 68 | Mismatch |
What made it slow
Where each repository's check time went, summed over its replayed commits, and why vetd fell back to a full check.
Iterations
Each round fixes the biggest bottleneck or bug the last one exposed, then re-measures the same commits. Times are p50 and p95 per commit.
No rounds recorded yet.
Every commit
Mode is incremental, trivia or full.
commerce30 commits · p50 432 ms · start 33 msMismatch
The per-commit rows load when this opens. They are also in benchmarks.json.
saas-starter30 commits · p50 275 ms · start 67 msMismatch
The per-commit rows load when this opens. They are also in benchmarks.json.
excalidraw30 commits · p50 1.40 s · start 473 msMismatch
The per-commit rows load when this opens. They are also in benchmarks.json.
taxonomy30 commits · p50 65 ms · start 62 msMismatch
The per-commit rows load when this opens. They are also in benchmarks.json.
Compatibility
Configuration errors stock TypeScript 7 reports for each repository, plus repositories vetd could not run.
commerceMismatch
TS5102: Option 'downlevelIteration' has been removed. Please remove it from your configuration. TS5108: Option 'moduleResolution=node10' has been removed. Please remove it from your configuration. TS5102: Option 'baseUrl' has been removed. Please remove it from your configuration.
saas-starterMismatch
TS5102: Option 'baseUrl' has been removed. Please remove it from your configuration.
honoDid not run
replay: ENOENT: no such file or directory, open 'bench/results/replay-hono.json'
excalidrawMismatch
TS5102: Option 'baseUrl' has been removed. Please remove it from your configuration.
zodDid not run
install: Command failed: pnpm install --ignore-scripts --frozen-lockfile=false Unsupported package manager specification (nub@0.8.3)
taxonomyMismatch
TS5108: Option 'target=ES5' has been removed. Please remove it from your configuration. TS5108: Option 'moduleResolution=node10' has been removed. Please remove it from your configuration. TS5102: Option 'baseUrl' has been removed. Please remove it from your configuration.
What the public corpus found
Bugs the first outside run exposed, all fixed.
- pnpmvetd could not find tsgo in any pnpm project. pnpm does not hoist the platform binary package. vetd now resolves it through
@typescript/native-preview's real path. - SymlinksA symlinked
node_modules(pnpm workspaces, linked worktrees) was indexed as a source file, and that crashed the oracle. Symlinks to ignored directories are now skipped. - ReplayThe history replay treated any
git showfailure as "file deleted", which could silently remove atsconfig.jsonmid-replay. A git failure now stops the run. - TS 7 configMany repositories still use
baseUrl,moduleResolution: nodeordownlevelIteration, which TypeScript 7 removed. Still open: see Compatibility.
Method
- Clone and install. Each repository from
bench/corpus.jsonis cloned in full and installed with--ignore-scripts. One tsgo, the same version vetd uses, serves the whole corpus. - Baseline. A stock
tsgo --noEmitat HEAD gives the time an agent pays today for a full check. - Replay. Recent commits that touch code are applied oldest first, each as one patch in a fresh fork, then committed so it becomes the next base.
- Verify. Every commit is rechecked from scratch in a separate tsgo process. Every Nth commit, a stock full check runs on a real tree holding the fork.
- Publish. Results, failures included, go to
site/benchmarks.json, which this page renders.
node bench/corpus.ts --dir /tmp/vetd-corpus --jobs 3 --commits 30 --oracle-every 5Add a repository by appending it to bench/corpus.json. A repository that fails to clone, install or start is still recorded, with the stage where it failed.