Extract Text From a Video — Vidstrackt (Open Source, On-Device)

You have a video that shows text on screen — a word flashing at a fixed point, a caption band on a screen recording, a slide deck someone screen-recorded and never sent you the source for — and you want the actual text back out as a string, not a transcript of the audio. That's a narrower problem than it sounds: audio transcription tools don't touch on-screen text at all, and generic OCR tools process one image, not a video's worth of frames that need grouping, deduplicating, and ordering.

Vidstrackt is a small open-source CLI that does exactly this, entirely on-device on a Mac, with no network calls and no API keys.

Install

brew install kegbenk/tap/vidstrackt

Or clone the repo and run ./vidstrackt.py directly — it needs ffmpeg and the Swift toolchain (xcode-select --install), and compiles its own OCR helper on first run.

What it does

./vidstrackt.py reel.mov                     # word mode: one word per screen -> a sentence
        ./vidstrackt.py captions.mp4 --mode phrase   # phrase mode: multi-word block per screen -> one line per screen
        ./vidstrackt.py talk.mp4 --mode scenes --scan   # scene mode: screenshot every visual cut, always start with --scan

Under the hood: ffmpeg samples frames at a fixed rate, Apple's Vision framework OCRs each frame on-device, and the observations are ordered top-to-bottom then left-to-right so a screen with several lines of text reconstructs as a coherent phrase instead of words in whatever order Vision happened to return them. Consecutive frames reading the same text are folded into one segment with fuzzy matching, so five frames of OCR jitter on the word "PLEROMA" don't become five words in the output.

There's a third mode, scenes, for a different problem: videos where the picture carries information the words don't — a creator naming a tool on camera and then cutting to the actual website. Scene mode finds the cuts, screenshots each one, OCRs it, and writes an agent-readable index (scenes.md + scenes.json) instead of dumping forty raw frames on whoever has to look at them.

What I actually verified

I ran the repo's own end-to-end suite before writing this page — no mocks, real ffmpeg and real Vision OCR against rendered test video:

./test_e2e.py
        [PASS] word mode round-trips 5 single words
        [PASS] phrase mode reconstructs multi-word / multi-line screens
        [PASS] scenes mode finds one scene per background
        ...
        All tests passed.

All 13 checks passed — word-mode round-tripping, phrase-mode reading order across multi-line screens, scene-cut detection and screenshot placement, and the legibility scorer that flags handwriting and OCR garble before an agent quotes it as fact. That legibility scorer is worth knowing about on its own: every scene carries a 0–1 score for how much of its OCR'd text is real dictionary words, so a whiteboard talk that OCRs as confident nonsense gets flagged (UNRELIABLE (legibility 0.31) — open the PNG, don't quote the line) instead of silently shipping garbage into a report.

Where it's actually used

I built Vidstrackt to solve a real problem in Velo's own marketing: verifying what a rendered RSVP demo video actually shows, frame by frame, against what the reader engine was supposed to display — the same word-at-a-time and multi-word-block modes Velo's reading engine uses when a speed-reading reel gets exported for Instagram or TikTok. Velo is the RSVP + audiobook-style reading app the tool was extracted from; if you're curious what the reels themselves are testing, RSVP_REELS_HOWTO.md in the repo has the full, battle-tested recipe for cropping and transcribing that kind of footage.

Known limitation

Straight from the repo's own README: identical adjacent segments shown back-to-back with no blank frame between them collapse into one segment. If a word or caption genuinely repeats twice in a row with nothing between, Vidstrackt reads it once — a transition frame (even a single blank one) between the repeats is what lets them survive as two. It's a real limitation of frame-diff-based segmentation, not a bug that's been quietly left unfixed; the README documents it plainly rather than pretending it away.

MIT licensed. Source, issues, and the full flag reference are on GitHub.

← All guides