Usage Guides

Video-to-Audio (Foley)

The Foley LoRA generates synchronized, music- and speech-free sound effects. Its primary use is video-to-audio: feed a muted clip and a prompt describing the on-screen action, and the model freezes the video and synthesizes a matching foley track, muxed onto the original frames. It also works for text-to-audio foley (prompt only, no video). It’s the video-conditioned counterpart to Text-to-Audio.

Prerequisites

  • For video-to-audio: a muted input clip (strip the audio first, the workflow replaces it).
  • Base model, Gemma text encoder, and the Foley LoRA (see Model files).
  • Custom nodes: ComfyUI-LTXVideo (audio + V2A nodes) and ComfyUI-VideoHelperSuite (VHS_LoadVideo).

Model files

FileDescriptionPlacement
ltx-2.3-22b-lora-foley-v2a-1.0.safetensorsFoley V2A LoRAComfyUI/models/loras/
ltx-2.3-22b-dev.safetensorsBase modelComfyUI/models/checkpoints/

Download the LoRA from Lightricks/LTX-2.3-22b-LoRA-Foley-V2A on HuggingFace.

Two ways to run it

  • Text-to-audio foley — add the Foley LoRA to the standard Text-to-Audio workflow and prompt the sound directly, no video.
  • Video-to-audio foley (primary) — use the dedicated Foley V2A workflow, which freezes the input video and generates only the audio.

Video-to-audio step by step

  1. Load the Foley V2A workflow in ComfyUI.
  2. In VHS_LoadVideo, pick your muted clip.
  3. Write the positive prompt in concrete foley language (name the sound sources and their sync), ending with No speech is present. No music is present. for clean SFX-only output.
  4. Keep the negative prompt (suppresses music and speech).
  5. If output is near-silent, change the seed and re-run — quiet or delicate scenes (rain, typing, footsteps) are seed-sensitive; take the loudest of a few seeds.
  6. Queue. The output is the original frames plus generated foley.

How it works

The input video is VAE-encoded and concatenated with an empty audio latent, then LTXVSetAudioVideoMaskByTime freezes the video (mask_video=false) and opens the audio for generation (mask_audio=true). The LoRA targets the audio attention/FFN blocks and video→audio cross-attention, so the model keeps the input video and synthesizes only the audio under MultimodalGuider. The audio is separated, VAE-decoded, and muxed onto the original video frames.

SettingNodeValueNotes
Foley LoRA strengthLoraLoaderModelOnly0.8–1.0Demos used full strength; lower for subtler foley
Guidance scale (audio)GuiderParameters (audio)6.0Prefers 6–8, under 4 tends to collapse
StepsLTXVScheduler30
STGMultimodalGuiderscale 1.0, blocks [29], mode stg_av
Resolution & frames—960×544, 89 @ 24fpsTrained bucket
SeedRandomNoise42Multi-seed and pick the loudest on quiet scenes

Prompting — describe the visible action and materials concretely. Example:

A sledgehammer smashes a glass bottle: a heavy whoosh, a sharp glass shatter and tinkling shards. No speech is present. No music is present.

Negative prompt:

music, melody, song, singing, vocals, score, soundtrack, beat, rhythm bed, instrumental backing, speech, dialogue, talking, narration, tinny, thin, harsh, clipped, distorted, low bitrate

Tips & troubleshooting

  • Near-silent output — raise guidance to 6+, try a new seed (multi-seed, pick the loudest).
  • Music or speech bleed — keep the negative prompt and reinforce No music is present / No speech is present in the positive prompt.
  • Weak ambience (rain, fire, typing) — seed-sensitive; multi-seed, and lean on “loud / close-mic / prominent” wording.
  • Unsynced SFX — prompt the action you can actually see; mismatched prompts produce plausible but unsynced audio.
  • Original audio still present, or static output — mute the input before loading; if the audio comes out static, the freeze masks are inverted (confirm mask_video=false, mask_audio=true).