Skip to the content.

# Abstract

High-fidelity video-to-audio synthesis is often computationally expensive, as state-of-the-art diffusion models require numerous iterative steps that impede real-time applications. To overcome this limitation, we introduce velocity consistency models built upon the Flow Matching framework, designed explicitly for few-step inference. Our core contribution lies in a novel training objective that learns a ground-truth average velocity field. This is achieved by supervising the model with a target that linearly interpolates its own velocity predictions at lower noise levels with the instantaneous velocity field given by the Flow Matching framework. We train our models from scratch, eliminating the need for a diffusion pre-training process. Crucially, we apply Classifier-Free Guidance (CFG) to our models during training, which substantially improves audio quality for few-step generation. Experimental results demonstrate that our model significantly reduces inference steps while achieving competitive generation fidelity compared with diffusion or Flow Matching models, bridging the critical gap between quality and efficiency for real-time video-to-audio synthesis.

# Demo

Since MF-MJT does not release public source code, we do not include its videos here. In the Experiments section of our paper, we compare mel-spectrograms using a demo directly from the MF-MJT webpage.

Note that for MMAudio-S-16k, “25x2 NFE” implies that Classifier-Free Guidance doubles the inference steps. Frieren-reflow & distill (rd) targets 1-NFE inference, while Frieren-reflow (r) 4NFE is designed for few-step generation (but not one-step).

All demos below come from the VGGSound test set. All baseline videos were generated following official implementations, each 8 seconds in length.

Ground Truth | SoundVCM-16k 4NFE | SoundVCM-16k 1NFE | MMAudio-S-16k 25x2NFE | Frieren-r-16k 4NFE | Frieren-rd-16k 1NFE
Prompt: striking bowling
Prompt: goose honking
Prompt: helicopter
Prompt: playing djembe
Prompt: donkey, ass braying
Prompt: playing accordion
Prompt: writing on blackboard with chalk
Prompt: tornado roaring
Prompt: race car, auto racing
Prompt: opening or closing car electric windows