# SoundVCM

This repository is built on **MMAudio** repository and contains the implementation of **SoundVCM**.


## Offline Demo Page

For immediate visualization of our results, we provide an **offline demo webpage** in the `./docs` directory. You can open it to watch to the generated samples and compare SoundVCM with other baselines directly.

## Prerequisites

The codebase supports **Python 3.10-3.13** and has been tested on Linux.

First, ensure you are in the project root directory:

```bash
cd SoundVCM
```

### 1. Setup Environments

```bash
conda create -n soundvcm python=3.13 -y
conda activate soundvcm
# Install PyTorch matching your CUDA version, e.g.
pip install "torch<=2.8" torchaudio torchvision -i https://download.pytorch.org/whl/cu130
pip install -r requirements.txt
```

Or, if you prefer uv:

```bash
uv venv --python 3.13
source .venv/bin/activate
# Install PyTorch matching your CUDA version, e.g.
uv pip install "torch<=2.8" torchaudio torchvision -i https://download.pytorch.org/whl/cu130
uv pip install -r requirements.txt
```

Finally, ensure `ffmpeg` (with version no higher than 6) is installed and accessible in your system path.

```bash
sudo apt install ffmpeg
# or, if you use Conda
conda install -c conda-forge "ffmpeg<7" -y
```

### 2. Prepare Datasets (Optional)

SoundVCM is trained and evaluated on datasets including [AudioSet](https://research.google.com/audioset/), [Freesound](https://github.com/LAION-AI/audio-dataset/blob/main/laion-audio-630k/README.md), [VGGSound](https://www.robots.ox.ac.uk/~vgg/data/vggsound/), [AudioCaps](https://audiocaps.github.io/), and [WavCaps](https://github.com/XinhaoMei/WavCaps). Please download them so that each dataset contains a directory of source video or audio files. Update the paths in `./cfg/data.yaml` to point to these directories. You may ensure the data filenames match the format in the `id` column of the TSV files located in `./tsv`.

**Note:** If you encounter difficulties configuring a specific dataset, you may comment it out in the configuration/code.

### 3. Setup Evaluation Metrics (for Evalation)

We adopt [av-benchmark](https://github.com/hkchengrex/av-benchmark) for evaluation. First, install `av-benchmark`:

```bash
git clone https://github.com/hkchengrex/av-benchmark.git
cd av-benchmark
pip install -e . # or, uv pip install -e .
cd ..
```

Then, download the [CLAP checkpoint](https://huggingface.co/lukewys/laion_clap/resolve/main/music_speech_audioset_epoch_15_esc_89.98.pt) and [Synchformer](https://github.com/hkchengrex/MMAudio/releases/download/v0.1/synchformer_state_dict.pth) and place them in `./av-benchmark/weights`. You can also run the following script (see part 4):

```bash
python util/download.py --av-benchmark "./av-benchmark"
```

### 4. Download External Model Weights (Optional)

Following [MMAudio](https://github.com/hkchengrex/MMAudio), our model builds upon several pretrained external models. These will be downloaded automatically when needed. You can also download them manually by

```bash
python util/download.py
```

## Demo

Due to size limit, we cannot provide the trained model weights directly. However, after training, you can run the inference script:

```bash
python demo.py \
    --duration 8 \
    --weight "path/to/your_model_weight.pt" \
    --prompt "your prompt (e.g. A dog barking)" \
    --video "path/to/your_video_path.mp4"
```

This will generate the foley audio based on the video visuals and the prompt, and produce a final video with the synchronized audio.

## Training

### 1. Latent Extraction

To reduce I/O overhead and accelerate training, we pre-extract dataset latents:

```bash
torchrun --nproc-per-node 8 latent/extract_audio.py
torchrun --nproc-per-node 8 latent/extract_video.py
```

This will generate HDF5 files (approx. 600 GB) in `./h5`.

### 2. Training

Configure the training settings in `./cfg/train.yaml`. Then start training:

```bash
torchrun --nproc-per-node 8 train.py
```

If an experiment is interrupted, locate the output directory for that run (defined in the config file; contains the preserved `cfg`, logs, and checkpoints). Pass this path to the `--rundir` argument to resume:

```bash
torchrun --nproc-per-node 8 train.py --rundir "path/to/your_rundir"
```

**Note:** If you encounter errors loading checkpoints, try replacing the latest checkpoint with the shadow backup and retry.

## Evaluation

### 1. Process Groundtruth

Download the precomputed cache `vggsound-test-eval-cache` from [Hugging Face](https://huggingface.co/datasets/hkchengrex/MMAudio-precomputed-results/tree/main) and add it to `gt_cache` in `./cfg/data.yaml`.

### 2. Generate Audio Samples

Modify `./cfg/eval.yaml` to specify your `weight_path` (ending with `ema_final.pth` or `last.pth`) and `output_name`. Then generate the samples:

```bash
torchrun --nproc-per-noode 8 eval.py
```

### 3. Compute metrics

Once the audios are generated (located at `${weight_path}/../${output_name}`), run the evaluation script:

```bash
cd av-benchmark
python evaluate.py \
    --gt_audio "path/to/gt_audio" \
    --gt_cache "path/to/vggsound-test-eval-cache" \
    --pred_audio "path/to/generated_audio" \
    --audio_length 8 \
    --num_workers 16 \
    --skip_clap
```
