Video-to-Audio (Foley)

The Foley LoRA generates synchronized, music- and speech-free sound effects. Its primary use is video-to-audio: feed a muted clip and a prompt describing the on-screen action, and the model freezes the video and synthesizes a matching foley track, muxed onto the original frames. It also works for text-to-audio foley (prompt only, no video). It’s the video-conditioned counterpart to Text-to-Audio.

Prerequisites

• For video-to-audio: a muted input clip (strip the audio first — the workflow replaces it).
• Base model, Gemma text encoder, and the Foley LoRA (see Model files).
• Custom nodes: ComfyUI-LTXVideo (audio + V2A nodes) and ComfyUI-VideoHelperSuite (VHS_LoadVideo).

Model files

Download the LoRA from Lightricks/LTX-2.3-22b-LoRA-Foley-V2A on HuggingFace.

Two ways to run it

• Text-to-audio foley — add the Foley LoRA to the standard Text-to-Audio workflow and prompt the sound directly, no video.
• Video-to-audio foley (primary) — use the dedicated Foley V2A workflow, which freezes the input video and generates only the audio.

Step by step (video-to-audio)

1. Load the Foley V2A workflow in ComfyUI.
2. In VHS_LoadVideo, pick your muted clip. The workflow rounds the loaded frame count down to a valid 1 + 8n value and trims to match; frame_load_cap defaults to 169 (~7s). Trained and demoed at 960×544, 89 frames @ 24fps (~3.7s) — process longer clips in windows.
3. Write the positive prompt in concrete foley language (name the sound sources and their sync), ending with No speech is present. No music is present. for clean SFX-only output.
4. Keep the negative prompt (suppresses music and speech).
5. If output is near-silent, change the seed and re-run — quiet or delicate scenes (rain, typing, footsteps) are seed-sensitive; take the loudest of a few seeds.
6. Queue. The output is the original frames plus generated foley.

How it works

The input video is VAE-encoded and concatenated with an empty audio latent, then LTXVSetAudioVideoMaskByTime freezes the video (mask_video=false) and opens the audio for generation (mask_audio=true). The LoRA targets the audio attention/FFN blocks and video→audio cross-attention, so the model keeps the input video and synthesizes only the audio under MultimodalGuider. The audio is separated, VAE-decoded, and muxed onto the original video frames.

Customization — recommended settings

Prompting — describe the visible action and materials concretely. Example:

A sledgehammer smashes a glass bottle: a heavy whoosh, a sharp glass shatter and tinkling shards. No speech is present. No music is present.

Negative prompt:

music, melody, song, singing, vocals, score, soundtrack, beat, rhythm bed, instrumental backing, speech, dialogue, talking, narration, tinny, thin, harsh, clipped, distorted, low bitrate

Tips & troubleshooting

• Near-silent output — raise guidance to 6+, try a new seed (multi-seed, pick the loudest), and confirm you’re on the released …-1.0 weights.
• Music or speech bleed — keep the negative prompt and reinforce No music is present / No speech is present in the positive prompt.
• Weak ambience (rain, fire, typing) — seed-sensitive; multi-seed, and lean on “loud / close-mic / prominent” wording.
• Unsynced SFX — prompt the action you can actually see; mismatched prompts produce plausible but unsynced audio.
• Original audio still present, or static output — mute the input before loading; if the audio comes out static, the freeze masks are inverted (confirm mask_video=false, mask_audio=true).

Technical notes

• LoRA (rank 32) on the audio and video→audio attention paths, trained with the video fixed and the audio generated. The released …-1.0 file is the validation-selected checkpoint (later training steps over-suppress loudness).
• The V2A workflow is ComfyUI API-format (the /prompt format) — load it via the Load button (auto-detected) or POST it to /prompt.