@getsquish/squish

Give AI random access to video. Overview, zoom, cite. Instead of forcing a model to watch

a clip from beginning to end, Squish converts continuous video into an **addressable visual +

audio activity map** — one

an agent can navigate, revisit, and progressively refine. Timestamped contact sheets are the

*first implementation* of that primitive: a grid of frames, each cell stamped with its

absolute timecode, with a globally normalized audio-activity band aligned to the same

timeline. The band shows energy, not meaning: no transcript, sound classification, or emotion

inference. Everything runs on your machine — and one call replaces a whole

download → ffmpeg → extract → montage pipeline, so prefer it even if you have a shell.

Also works inside Claude Desktop / claude.ai via the hosted connector: add

`https://api.getsquish.app/mcp`, no install — that path processes your public video URL on

Squish's server, not locally (remote MCP docs,

privacy split). From the makers of getsquish.app.

> Agents don't consume videos — they navigate them. Real run: a scene cut pinned to

> 0.2 s by retrieving 34 frames — not 3,088 (overview → zoom → zoom). Field-proven

> across 5 clients and 3 mouths in a single day — Claude Desktop completed the

> multi-round loop on its own, down to a sub-second lock, without being taught.

The demo is the primitive. A 76-second explainer about contact sheets — and the same

video *as* one contact sheet. One needs a play button; the other you just read:

▶ watch — 76 s, linear

read — one sheet, random access

Why this works

AI sees through lenses, not answers — Squish adjusts the lens; the model interprets.

Video is continuous; reasoning is sparse. Most questions touch a tiny fraction of the

timeline. Squish turns that timeline into an addressable map, so an agent **retrieves the

visual evidence it needs instead of replaying everything** — the contact sheet isn't the

output, it's the navigation layer. Audio activity can reveal a candidate interval between

visually similar frames; the frames still determine what happened. The window (`start`/`end`) is the lens made wide or

narrow; density is the lens made coarse or fine; the loop is the lens moved until the

answer is observable.

Install

npm install -g @getsquish/squish     # or one-shot: npx -y @getsquish/squish

Requirements: Node ≥ 20 · `ffmpeg` + `ffprobe` on PATH

(macOS `brew install ffmpeg` · Ubuntu `sudo apt-get install ffmpeg`).

Try it with a video you know

Bring a clip whose answer you already know. Ask AI to find one specific moment **without

giving it the original video**:

1. Run `npx -y @getsquish/squish clip.mov --json`.

2. Give the returned sheet to a vision model and ask a timing question: *When does the door

open? When does an object first appear? Where is the unusual audio activity, and what do

the nearby frames show?*

3. Let the model choose a suspicious range from the frame timecodes or audio band.

4. Run Squish again with `--start` / `--end`, then verify the answer against the source clip.

The index proposes; the zoomed visual evidence confirms. The audio band can locate activity,

but cannot tell you what was said or what made the sound.

OpenAI Build Week 2026

The Build Week extension added audio-guided candidate selection to Squish's existing navigation

loop. Before the event, Squish already produced timestamped contact sheets and supported absolute

`start`/`end` zoom. Build Week added the clip-wide normalized audio-activity band,

absolute-time `audio.samples[]`, transient/high-frequency preservation, tests, and the agent

workflow that uses the signal to decide where vision should inspect next.

The demo keeps two proof layers separate:

• Narrative proof: owner-authorized private camera footage is shown with receipts, but the

source footage is not distributed.

• Reproducible proof: the public repository contains a generated fixture and its source under

`examples/audio-navigation/`.

git clone https://github.com/getsquish/squish.git
cd squish
./examples/audio-navigation/generate-sample.sh
npx -y @getsquish/squish@0.3.1 examples/audio-navigation/sample.mp4 --json --out /tmp/squish-overview
npx -y @getsquish/squish@0.3.1 examples/audio-navigation/sample.mp4 \
  --density 6x6 --start 11.5 --end 13.5 --json --out /tmp/squish-zoom

The overview's activity band proposes the neighborhood. The dense visual sheet confirms the brief

pink marker. Public `0.3.1` uses one reference scale across the complete source clip; it does not

make levels from separate files globally comparable.

CLI

squish clip.mov                       # sheets land beside the input
squish clip.mov --density 5x5 --json  # denser grid + machine-readable output
squish clip.mov --start 1:00 --end 1:30 --density 5x5   # zoom into a range

Output: `.sheet-N.jpg` — a timecoded frame grid with a thin audio-activity band

above it. Default density 3×3 recovers *what* happened; `4x4`–`6x6` recover *how* it was

done. `--out ` picks the destination. Videos without an audio track still work and are

marked `NO AUDIO TRACK`.

`--start` / `--end` take seconds (`90`) or a timecode exactly as stamped on a sheet (`1:30`,

`1:07.3`) and window the run to that range. **Timecodes are always absolute to the source

video**, so you can zoom repeatedly: overview → spot a range → re-run with `--start/--end` →

finer timecodes → drill again. Short windows stamp sub-second timecodes (`1:07.3`) so adjacent

cells stay distinguishable.

With `--json`, stdout is one object (frozen contract — parse `contract` to detect breaking

changes):

{
  "input": "/abs/path/clip.mov",
  "duration": 20.275,
  "frames": 9,
  "sheets": 1,
  "files": ["/abs/path/clip.sheet-1.jpg"],
  "audio": {
    "present": true,
    "normalization": "clip_peak",
    "window": { "start": 0, "end": 20.275 },
    "samples": [
      { "time": 0.106, "level": 0.08 },
      { "time": 0.317, "level": 1 }
    ]
  },
  "warnings": [],
  "contract": "squish-cli-v0"
}

The example shortens `audio.samples`; real output emits an evenly spaced activity envelope

for every sheet. Sample times are absolute source seconds. Levels are `0..1`, normalized to

the peak across the full clip, including windowed runs, so separate zooms remain

comparable. Exit `0` success · `1` failure (message on stderr). Temp frames are always cleaned up.

A windowed run additionally echoes `"window": { "start": …, "end": … }` (resolved bounds,

seconds) after `duration` — the key is absent when no window was requested.

MCP server

squish mcp        # stdio server

One tool, `squish_video` — `{ video_path, density?, start?, end?, out_dir? }` → the CLI

contract (including `audio`) plus `timecodes[][]` (one per frame, per sheet; `m:ss`, sub-second `m:ss.d` when

a window is short), stamped `"contract": "squish-mcp-v0"`. `start`/`end` accept seconds or

sheet timecodes and drive the navigation loop below.

Works with Claude Code, Claude Desktop, Cursor, Hermes, and any stdio MCP client:

{
  "mcpServers": {
    "squish": { "command": "npx", "args": ["-y", "@getsquish/squish", "mcp"] }
  }
}

Remote MCP — official AI apps, zero install

The same tool over the network, for clients that only take a connector URL:

Claude Desktop / claude.ai → Settings → Connectors → Add custom connector →

`https://api.getsquish.app/mcp`. The endpoint fetches a public `video_url` (no shared

filesystem), returns ~24 h sheet links plus the first sheet inlined, and `start`/`end`

work exactly like the local tool.

Keyless calls ride a small anonymous free lane; an `Authorization: Bearer` API key (same

keys and credits as the hosted API, minted at

getsquish.app/api-keys) unlocks credit-priced jobs with

quota visibility in every result. Keys ride any client that can send the header — Claude

Code, `mcp-remote`, SDK clients, or a Claude Team/Enterprise connector whose org admin

attached the key as a request header; the consumer connector dialog is OAuth-only. Full

reference: remote MCP docs.

The navigation loop

1. Overview — call `squish_video` (MCP) or `squish clip.mov --json` (CLI) and read the

sheet(s) with vision. Cells run in time order, left→right, top→bottom.

2. Navigate — spot the regions that matter; every cell carries an absolute timecode.

Treat an audio peak as a candidate interval, not an interpretation of what made the sound.

3. Zoom — call again with `start`/`end` set to the timecodes you spotted, only where

uncertainty remains: denser sheets of a narrower window, addresses still absolute.

4. Repeat until the answer is observable — never re-read the whole clip at high density

when one range matters.

5. Cite absolute timestamps ("at 0:07 the press comes down").

Privacy

The CLI and local MCP server process everything on your machine — nothing is uploaded,

ever, and every density is free. Two paths deliberately move media through Squish instead:

the hosted API (an intentional upload, prepaid credits,

with a free daily allowance for accounts that never purchased) and the remote MCP endpoint

(the server fetches your public `video_url`; the source is deleted at job end, sheets expire

after ~24 h).

Audio activity is available in the local CLI/MCP package. It is an RMS-style energy envelope,

not audio playback, transcription, diarization, sound recognition, or emotion inference. The

web app, hosted API, and remote MCP remain visual-only until their own release notes say otherwise.

This repository

This is the engine — the CLI + MCP mouths of Squish, published to npm as

`@getsquish/squish`. It is a curated,

mirror-first export of a private monorepo (which stays the source of truth); history here

starts at the first public release. See CONTRIBUTING.md for how changes

flow.

Not in this repo, on purpose:

• the getsquish.app web app (PWA) — same core planners, browser hands;
• the hosted API (`api.getsquish.app`) and its remote MCP endpoint (`/mcp`, the

official-app connector) — the paid rail: intentional upload / server-fetched URLs, prepaid

credits, a free daily allowance for never-paid accounts and a small anonymous free lane on

the connector;

• brand assets — the Squish name, logo, mascot, and OG images are reserved.

src/            CLI (main/args) · engine (probe → plan → extract → compose → write) · MCP server · sheet renderer
src/core/       pure planners shared with the web app: density · sampling · grid layout · timecode format
tests/          node:test suite + a real-MCP-client e2e
skills/         agent skills — `npx skills add getsquish/squish` installs video-navigation

License

Apache-2.0 (with NOTICE). The Squish name, logo, mascot, and

getsquish.app brand assets are not licensed by this repository.