Union Control IC-LoRA

This guide walks you through the Union Control workflow: a ComfyUI workflow from LTX that generates video following the structure of a reference video. You provide a reference video, extract structural information from it (depth, edges, or pose), and the model uses that structure as a guide during generation. It supports both text-to-video and image-to-video.

The workflow uses the Union Control IC-LoRA: a single adapter that handles three control types (depth maps, canny edges, and pose skeletons) in one checkpoint.

This guide assumes you’re comfortable prompting, generating, and iterating in ComfyUI. If you haven’t used the default templates yet, start with the Text-to-Video or Image-to-Video guides first.

What you’ll need

Download the Union Control workflow and bring it into ComfyUI. Open the Workflow Overview panel; if any custom nodes or model files are missing, it will let you install or download them.

Model files:

Custom nodes: beyond ComfyUI-LTXVideo, this workflow uses the reference-video annotators ComfyUI-VideoDepthAnything and comfyui_controlnet_aux (depth / canny / pose). The annotators also download their own preprocessor model files on first run. Install any that show as missing via the ComfyUI Manager.

Step-by-step

1. Load your reference video

Load your source video in the LoadVideo node. This is the video whose structure the output will follow, providing the same layout, motion, or pose sequence, but with the visual content described by your prompt. The generation’s resolution, length, and frame rate all follow this reference video, so there’s no separate resolution to set.

2. Choose an annotation type

The Reference Video subgraph turns your reference into a control signal using an annotator. Depth (Video Depth Anything) is wired by default. To switch, open the subgraph and connect the Canny (CannyEdgePreprocessor) or Pose (DWPreprocessor) output into the scale-to-multiple Resize node instead of the Depth output. All three read the resized reference frames, and only the one wired into that Resize is used.

See Choosing the right annotation type below for which to pick.

3. Choose text-to-video or image-to-video

In the Input Parameters subgraph, the use image input toggle switches modes:

• Off — text-to-video. The model generates from your prompt, guided by the reference structure.
• On — image-to-video. Your image conditions the first frame, with subsequent frames following the reference structure. Load it in the LoadImage node.

4. Write your prompt

Describe the visual content you want, not the structure, which the reference already provides. If your reference shows a person walking and you’re using pose, prompt the style (“a knight in silver armor walking through a misty forest”), not the motion.

Optionally enable enhance positive prompt in Input Parameters to expand a short prompt with the Gemma 4 enhancer (the negative prompt is never enhanced). To save VRAM, you can offload encoding/enhancement to the LTX API (see the note in the workflow). See the Prompting Guide for additional prompting tips.

5. Generate

Click Run. The pipeline annotates the reference video, injects the guide frames via the IC-LoRA (LTXAddVideoICLoRAGuide), and generates in two stages: Stage 1 samples the joint audio-video latent in 8 distilled steps with the Union Control IC-LoRA applied; the guide frames are then cropped (LTXVCropGuides) so Stage 2 upscales the generated video 2× and refines it in 4 more steps. Audio and video are decoded and saved as an MP4.

6. Review and iterate

• Adjust the prompt to change visual style while keeping the same structure.
• Try a different annotation type if the current one isn’t capturing the structure you need.
• Change the reference video to guide with different motion or composition.
• Adjust IC-LoRA strength (below) to balance structural adherence against creative freedom.

Choosing the right annotation type

Depth (Video Depth Anything)

Estimates per-frame depth maps — a grayscale representation of spatial layout (lighter = closer, darker = farther). Works well for: landscapes, interiors, environments with clear depth layers, foreground subjects against distant backgrounds. Less effective for: flat compositions, scenes where all elements sit at similar depths, or subjects where edge detail matters more than spatial position. The VideoDepthAnythingProcess node has a configurable processing resolution.

Canny (edges)

Extracts visible contours and edges from each frame; the model follows these outlines. Works well for: architecture, product shots, geometric compositions, mechanical objects, cityscapes — anything with strong, well-defined edges. Less effective for: organic subjects with soft boundaries (clouds, fur, foliage) or heavily textured subjects where edge detection picks up noise. The CannyEdgePreprocessor node exposes low_threshold / high_threshold (defaults 92 / 200) — lower detects more edges, higher keeps only the strongest.

Pose (DWPose)

Detects human body keypoints and renders pose skeletons per frame. Works well for: dance, character animation, movement sequences — anything where human body position is the primary structure to transfer. Less effective for: scenes without people, heavy occlusion, or close-ups where full-body keypoints aren’t visible. The DWPreprocessor node can toggle face, hand, and body detection independently (all on by default).

Customization

IC-LoRA strength

The LTXICLoRALoaderModelOnly node loads the Union Control IC-LoRA at strength 1.0. Reducing it gives the model more creative freedom at the cost of structural adherence; 0.5–0.8 produces a softer guide where the model follows the general structure but takes more liberties. For finer control, including spatial and temporal masking, see IC-LoRA parameters.

CFG

The workflow uses CFG 1. The distilled model bakes guidance into distillation, so raising CFG doesn’t help the way it would with a standard diffusion model. If you experiment, stay in the 1.0–1.5 range.

Adding LoRAs

Standard LoRAs can be combined with the Union Control IC-LoRA to add style on top of structural control. Start with lower strengths (0.5–0.8) to avoid conflicts between style and the control signal. See the LoRA guide.