Choosing Video Frames by Content Instead of by a Clock
The usual way to hand a video to a language model is to pull a handful of screenshots and let it reason over them. The trouble isn’t in reading any one frame; it’s in reading the gaps between them. A screenshot doesn’t say when it happened, how long after the last one, or what moved in between. So the model fills that in with a plausible story. VTL replaces those screenshots with a measured timeline. The first question worth asking is whether it actually sees more of a video than evenly spaced sampling does at the same cost.
It does, and the gap is large. Both methods get the same frame budget, run over five real videos: eighteen minutes of a narrated slide deck, two screen recordings, a scroll capture, and handheld phone footage. Then you count what each one misses. Across 28 shots, uniform sampling never looks at 9 of them. That is a third of the video, gone. It also spends 12 of its 40 frames re-photographing pictures that differ from the previous frame taken by under 7%. VTL misses no shot and wastes 2 frames. You can check this yourself. python3 tests/benchmark.py runs the comparison on generated fixtures, or on your own files if you pass them.
The mechanism is simple enough to disagree with. Uniform sampling puts a frame every T seconds, regardless of what is on screen. VTL puts frames where the picture changes, drops the near-duplicates, and enforces a floor. No shot goes without at least one frame, and no stretch longer than the coverage limit goes unobserved. The budget uniform sampling spends re-photographing a motionless slide, VTL spends on the shots uniform never reached.
Where it does not win
That is not the same as “smaller gaps everywhere,” and the benchmark is built to catch me if I claim it is. On the size of the largest unobserved gap, the two methods come out level. On one video, the dashboard capture, uniform sampling is the better of the two: a worst gap of 0.07 against VTL’s 0.14. That result is real, and it follows from the design. VTL concentrates its frames at the moments of change, and it deliberately declines to re-photograph a slide that is sitting still. Across the still part, that can leave a wider gap than even spacing would. The difference I will defend is not that the gaps are smaller. It is that VTL names its gaps in the timeline as frozen spans, where uniform sampling’s gaps are simply unexplained.
One honest caveat on the coverage number. “Shots never seen” is counted against VTL’s own shot boundaries, which are the thing under test, so it is not an independent oracle. It is fair for measuring coverage, because a shot boundary is a measured discontinuity whichever tool you ask. I would not push it further than that.
Why I trust the numbers
Most of these figures are checked against ground truth I can compute rather than eyeball. The camera-motion rates are asserted against a fixture whose every frame is rendered in Python, by moving a viewport along a closed-form path. The true pan, tilt and zoom rates are therefore known exactly. Pan and tilt land within about 10%. Zoom reads roughly 30% low: true +0.115, measured +0.080. That is not noise. Block displacement under a zoom is only a fraction of a proxy pixel near the frame centre, so the direction is reliable and the magnitude is a lower bound. Both facts are printed in the bundle, next to the measurement that produced them. On real footage there is no ground truth to assert against, so a second script measures the same clips with an independent algorithm: FFT phase correlation at 480 px, sharing no code with the converter’s block matching on a 96 px proxy. On a pan across a grass field the two agree, at -0.088 and -0.090.
Every one of those checks began by finding the code wrong. A fade from black fabricated a confident zoom_in on a static title card, because the matching is brightness-sensitive and read the brightening as the frame growing toward the camera. Aliasing on the downscaled proxy turned a true +0.94 px/frame drift into a confident -3.94, wrong in both sign and magnitude, until one blur pass before matching fixed it. OCR read the tool’s own timestamps, burned into each frame header, back out and reported them as text found in the video. None of these looked broken; they looked like measurements, which is exactly why the fixture has to carry an answer you already know.
On how it was built
The README is straight about this, and the post should be too. VTL was written with heavy AI assistance (Claude Code). The typing was assisted. The design judgement, and the “is this actually true?” loop that caught every bug above, were the work. Two of those bugs were found only because a fixture I had written was itself wrong. That is why the motion fixture renders each frame from a closed-form path instead of leaning on video filters. A fixture you have to debug is not a fixture.
The general point is the one the screenshots get wrong. When you sample a video on a fixed clock, the frames you get are an accident of the clock’s phase against the content, and the gaps between them go unexamined. VTL measures where the content actually changes, then states the worst gap it left rather than hoping there wasn’t one. That costs about 67 seconds for eighteen minutes of video. It turns those intervals from a guess into a number. The errors were never in the frames; they were always in the spaces between them.