---
title: "vetd benchmarks on public repositories"
description: "vetd results on public open-source repositories: replayed commits, full tsgo times, independent checks and every failure."
url: http://localhost:4173/benchmarks
updated: 2026-11-11
---

# vetd benchmarks

> vetd replays real commits from public repositories. Each commit is checked by vetd, then checked again from scratch by a separate `tsgo`. Every few commits, a full `tsgo --noEmit` also runs. Failures are included. Raw data: [benchmarks.json](http://localhost:4173/benchmarks.json).

Run on 2026-11-11 with vetd `7371bf6+local` and tsgo `7.0.0-dev.20260707.2`, on darwin-arm64 (10 CPUs, 24 GB), 3 repositories in parallel, 30 commits per repository, a full `tsgo --noEmit` every 5 commits.

## Summary

| Measure | Value |
| --- | ---: |
| Repositories that ran | 4 of 6 |
| Median speedup over a full `tsgo --noEmit` | 2.4x |
| Commits checked | 120 |
| Full `tsgo --noEmit` runs | 24 |
| Mismatches (vetd disagreed with the separate check) | 231 |

Speedup = full `tsgo --noEmit` time divided by vetd's median check time. Small repositories are fast to check anyway, so vetd can be slower on them. A mismatch is a result where vetd and the separate check disagree. Each mismatch is a vetd bug.

## Repositories

| Repository | Kind | Files | Full tsgo | vetd p50 | vetd p95 | Commits | Full fallbacks | Mismatches | Status |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| [commerce](https://github.com/vercel/commerce/tree/f7521d57d29f0eb3d23c1399cf9bb299d61fcd21) | Next.js App Router store | 65 | 1.3 s | 432 ms | 846 ms | 30 | 17 | 116 | mismatch |
| [saas-starter](https://github.com/nextjs/saas-starter/tree/6e33e58b1e553a41fe22e6b941a7229a002de361) | Next.js SaaS starter (Drizzle, Stripe) | 41 | 511 ms | 275 ms | 501 ms | 30 | 19 | 39 | mismatch |
| [hono](https://github.com/honojs/hono/tree/61f7c59e50244ab4d2dd83c151928cf7ea448510) | Web framework library, type-heavy | 457 | 65 ms | n/a | n/a | n/a | n/a | n/a | did not run (at replay) |
| [excalidraw](https://github.com/excalidraw/excalidraw/tree/701b901a0f1ba3d0b390d9cdf63d112e25abac4b) | Vite + React app, yarn monorepo | 695 | 302 ms | 1.4 s | 2.5 s | 30 | 11 | 8 | mismatch |
| [zod](https://github.com/colinhacks/zod/tree/19c40d2072288b4cafec851e9d056a72a88e5973) | Schema library, pnpm monorepo, extreme generics | n/a | n/a | n/a | n/a | n/a | n/a | n/a | did not run (at install) |
| [taxonomy](https://github.com/shadcn-ui/taxonomy/tree/298a8857c7128a0d121e7f699dfd729f23b3966d) | Next.js 13 App Router app (Prisma, Contentlayer) | 125 | 246 ms | 65 ms | 636 ms | 30 | 7 | 68 | mismatch |

## Time per stage

Median time per commit:

| Repository | Type check p50 | Snapshot sync p50 | Framework rules p50 | Import graph p50 | Impact p50 | Report p50 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| commerce | 231 ms | 18 ms | 4 ms | <1 ms | <1 ms | 1 ms |
| saas-starter | 183 ms | 25 ms | 3 ms | <1 ms | <1 ms | <1 ms |
| excalidraw | 1.2 s | 30 ms | 3 ms | 7 ms | 51 ms | <1 ms |
| taxonomy | 46 ms | 19 ms | 1 ms | <1 ms | <1 ms | <1 ms |

## Why vetd checked the whole project

| Reason | Commits |
| --- | ---: |
| package.json changed: compiler/package configuration affects every file | 43 |
| a source file touches the global scope (script file, `declare global` or `declare module`) | 9 |
| tsconfig.json changed: compiler/package configuration affects every file | 2 |

## Compatibility

TypeScript 7 config errors per repository, and repositories that did not run:

| Repository | Finding |
| --- | --- |
| commerce | TS5102: Option 'downlevelIteration' has been removed. |
| commerce | TS5108: Option 'moduleResolution=node10' has been removed. |
| commerce | TS5102: Option 'baseUrl' has been removed. |
| saas-starter | TS5102: Option 'baseUrl' has been removed. |
| hono | Did not run, failed at replay: ENOENT: no such file or directory, open 'bench/results/replay-hono.json' |
| excalidraw | TS5102: Option 'baseUrl' has been removed. |
| zod | Did not run, failed at install: Unsupported package manager specification (nub@0.8.3) |
| taxonomy | TS5108: Option 'target=ES5' has been removed. |
| taxonomy | TS5108: Option 'moduleResolution=node10' has been removed. |
| taxonomy | TS5102: Option 'baseUrl' has been removed. |

## Bugs found

| Area | Problem | Status |
| --- | --- | --- |
| pnpm | vetd could not find `tsgo` in pnpm projects. It now finds it through `@typescript/native-preview` | Fixed |
| Symlinks | A symlinked `node_modules` was read as source code and crashed the check. Symlinks to ignored folders are now skipped | Fixed |
| Replay | A failed `git show` was treated as a deleted file. It now stops the run | Fixed |
| TS 7 config | Many repositories use `baseUrl`, `moduleResolution: node` or `downlevelIteration`, which TypeScript 7 removed | Open |

## Method

1. Clone each repository in `bench/corpus.json` and install with `--ignore-scripts`.
2. Time a full `tsgo --noEmit` at the latest commit.
3. Apply recent commits one by one, oldest first, and check each with vetd.
4. Check each commit again in a separate `tsgo`. Every Nth commit, also run a full `tsgo --noEmit`.
5. Save all results, including failures, to `site/benchmarks.json`.

```sh
node bench/corpus.ts --dir /tmp/vetd-corpus --jobs 3 --commits 30 --oracle-every 5
```

To add a repository, add it to `bench/corpus.json`. Repositories that fail to clone, install or start are recorded with the step that failed.

Per-commit results are in [benchmarks.json](http://localhost:4173/benchmarks.json) under `repos[].replay.steps`.