How we got AI verse detection under 600ms

When we started building scripture detection, the best off-the-shelf approach took about four seconds end-to-end: half a second to transcribe a chunk of audio, a second to run it through an embedding model, another second to do a semantic search against a vector store of Bible verses, and another second-and-a-half of buffering and rendering.

Four seconds is unusable. By the time the verse appears on screen, the preacher has moved on. Our target was 600 milliseconds.

This post walks through how we got there.

The naive pipeline

The first version of our pipeline looked like this:

1. Buffer 8 seconds of audio.
2. Send the chunk to Whisper for transcription.
3. Embed the transcribed text.
4. Vector-search a database of 31,102 Bible verses (every verse in the KJV).
5. Score and pick the best match.
6. Render and send to projection.

Total latency: 3.8 seconds on a good day, 6+ on a bad one.

The first optimization: streaming transcription

Buffering 8 seconds of audio before transcribing was the biggest single source of latency. We replaced batched Whisper with streaming Whisper — feeding audio chunks every 200ms and getting partial transcripts back.

Streaming Whisper has a trade-off: the first version of each transcript chunk is often wrong, and gets corrected over the next few hundred milliseconds. We handle this by detecting at every revision, and quietly replacing the on-screen verse if the corrected transcript points to a different match.

This dropped average end-to-end latency to about 1.6 seconds.

The second optimization: domain-specific embeddings

Off-the-shelf embedding models treat "the Lord is my shepherd" and "I am the good shepherd" as semantically similar — they're both about shepherds. But they're different verses (Psalm 23:1 vs. John 10:11) and conflating them produces noisy results.

We fine-tuned a small (110M parameter) encoder specifically on a dataset of 400,000 paraphrased scripture references mined from sermon transcripts. It runs in 80ms on CPU and produces embeddings that cluster scripture references by their underlying verse, not their surface vocabulary.

Latency: 1.0 second.

The third optimization: not always searching

Most of the time, the preacher isn't quoting scripture. Running a full embed-and-search pipeline on every 200ms transcript chunk was wasteful.

We added a tiny upstream classifier — basically a 6-layer transformer that emits a single score from 0 to 1: "how likely is this chunk to contain a scripture reference?" It runs in under 5ms and lets us skip the full pipeline 80% of the time.

Latency: 720ms average, with a long tail at 1.1 seconds.

The fourth optimization: edge co-location

By this point, the bottleneck was network. Our pipeline was running in us-east-1, but our customers were in Lagos, Sydney, São Paulo. Adding 200ms of round-trip per stage adds up.

We moved the entire pipeline to our edge regions — the same regions that handle broadcast ingest. The classifier, encoder, transcription, and vector search all run in the same data center as your stream.

Latency: 540ms average. We ship with a 600ms SLO.

What's next

We're working on two things now: a larger encoder (~1B parameters) that handles cross-language references (a preacher quoting in Yoruba but expecting an English-translation projection), and on-device transcription so churches with unreliable internet can still get verse detection.

If this kind of work sounds interesting, we're hiring.