Chasing audio latency on a TV: 255 ms to 55 ms

Once the picture got fast, the sound got embarrassing. You'd click a video on the PC, the frame would show up on the TV in under 20 ms, and the audio would roll in about a quarter of a second later. We measured it at around 255 ms from capture to played on the TV. Lips moving, then words. On a monitor you work on that's annoying, and on a movie it's unwatchable.

We got it down to about 55 ms, in sync with the picture. It wasn't one fix. It was 4 separate problems stacked on top of each other, and each one was hiding the next.

The setup

On the PC, the sender captures what Windows plays with WASAPI loopback. It ships raw PCM, 48 kHz, 16-bit stereo, in 5 ms UDP datagrams, which is about 1.5 Mb/s. Every datagram is stamped with the same clock as the video frames, so the TV knows which sound belongs with which picture.

On the TV, a small jitter buffer holds the audio, between 20 and 80 ms depending on how bumpy the network is, and plays it on a clock. It never plays based on how full the buffer happens to be, because a fill-driven buffer just grows and grows. Then the audio goes out through AAudio, Android's low-latency audio API.

That design was right from the start. The trouble was underneath it.

1. A blocking write gets the slow lane

Our first TV player pushed audio into AAudio with plain blocking writes. It's the obvious way to do it, and it works, and it's slow. The Android docs say it plainly if you read far enough: a blocking write() never gets a FAST track, so you get high latency and big bursts no matter what you ask for.

So we moved to AAudio's data callback. AAudio now pulls audio from us exactly when the hardware needs it, and that's what unlocks the low-latency path. Same buffer, same clock, very different number.

2. The error that wasn't an error

Then the sound got choppy. Every burst or so, the stream would reopen itself and you'd hear a tiny gap.

It turned out to be the ndk crate, version 0.9. It maps every non-zero AAudio result to Err. That's fine for most calls, but write and set_buffer_size_in_frames return a frame count when they succeed. So a perfectly good write of 240 frames came back as an error, our code did the sensible thing with an error and reopened the stream, and that happened after every burst.

The fix is a small helper that reads the "error" back as the count it really is. We pinned the crate at that exact version and wrote down why, because the day a fixed ndk returns Ok(count), that helper has to go, and a different version of the bug would bring the choppy sound right back.

3. The TV that silently says no

This was the big one. AAudio can run in exclusive mode with MMAP, where the app writes straight into the hardware's buffer. That's the fastest path there is, so of course we asked for it.

On our MediaTek TV, exclusive MMAP always fails. In logcat you can see GET_STREAM_DESCRIPTION return FAILED_TRANSACTION. The nasty part is what happens next. AAudio doesn't fail the request. It quietly falls back to the old AudioTrack path and hands you back a working stream that just happens to be about 100 ms slower, and it never even tries shared MMAP.

So now the TV app checks what it actually got. If an exclusive request comes back as anything but exclusive, it closes that stream and opens a shared one on purpose. In shared mode the AAudio service makes the MMAP buffer itself and shares its endpoint, and that works on this TV. Output latency went from 107 ms to 25 ms on that change alone.

4. The OBS silence trick

The last problem was on the PC, and it was sneaky. Windows only delivers loopback packets while something is actually playing. When the PC goes quiet, the packets stop. When sound starts again, the first real packet shows up with a timestamp that jumps, and the TV's delay estimate jumps with it. You'd get a little lurch every time audio started after a pause.

The fix is the same one OBS uses. For the whole stream, the sender plays digital silence to the device it's capturing. Nobody hears it, but Windows sees something playing, so loopback packets keep coming all the time, the timestamps stay one continuous line, and the audio engine never goes idle. If that silent stream can't run, the sender falls back to sending its own silent datagrams on a 5 ms timer, in the same time base.

Where it landed

Stack those 4 fixes together and the audio goes from about 255 ms behind to about 55 ms from capture to played on the TV, locked to the picture the TV is actually showing. The play delay follows the video, so sound and frame land together instead of racing each other.

We also gave it a virtual "TV speaker". While you stream it becomes the Windows default, so you're capturing a clean device and your real speakers stay quiet, and when you stop, your old defaults come back.

What we took from it: check what the API actually gave you. AAudio handed us a slower stream without saying so, the crate reported success as an error, and we only caught both by measuring when the sound came out of the TV.

If you want to know how we got the picture fast in the first place, that's in Why we rewrote it in Rust.