IC-LoRA

IC-LoRAs (In-Context LoRAs) are specialized adapters for LTX that condition video generation on a reference input, like an existing video clip or a control signal like a depth map. Each adapter is trained for a specific task: structural control, VFX, video restoration, creative transforms, or quality enhancement. Unlike standard LoRAs that modify style globally, IC-LoRAs perform targeted, reference-driven operations while preserving the original scene’s identity.

Comparison: IC-LoRA vs LoRA

When to use which:

• Need style/effect: Standard LoRA
• Need structural control, video restoration, VFX, or creative transforms: IC-LoRA
• Need both: Combine them

Available IC-LoRA Adapters

For documented IC-LoRA adapters — including Union Control, Motion Control, the Pixel Spatial Upscaler, and In-Outpainting — see the IC-LoRA Adapters page, which also lists every released adapter with links to their model cards.

Using IC-LoRAs in ComfyUI

Setup:

1. Install the ComfyUI-LTXVideo custom nodes. Some adapters need extra third-party nodes (e.g. the control-signal preprocessors used by Union Control) — the adapter’s own page or workflow lists them.
2. Download the IC-LoRA .safetensors from Hugging Face and place it in ComfyUI/models/loras/.
3. Download the LTX-2.5 model stack the workflow expects — the distilled transformer (ltx-2.5-22b-distilled-transformer-bf16), the Gemma 4 text encoder (gemma4-12b-with-proj-ltx-2.5-bf16), the Gemma 4 E2B prompt enhancer (gemma4_e2b_it_bf16), and the video + audio VAEs. See the model-files table in the workflow guide for the full list and placement.
4. Load an IC-LoRA workflow. Adapters that take a reference video directly use the generic video-to-video IC-LoRA workflow; adapters with dedicated workflows (Union Control, In-Outpainting) ship their own.

Key nodes:

• LTX IC-LoRA Loader Model Only — loads the IC-LoRA checkpoint and applies it to the model. Its latent_downscale_factor output is intentionally unused for adapters trained at reference_downscale_factor = 1 (most of them); see the Union Control workflow for handling other values.
• LTX Add Video IC-LoRA Guide — applies the reference input, with configurable attention_strength and attention_mask. Used by most IC-LoRA workflows.
• LTX Add Video IC-LoRA Guide Advanced — mask-aware version for In-Outpainting. Not needed for other IC-LoRAs.
• LTX Draw Tracks / LTX Sparse Track Editor — define spline-based motion trajectories for the Motion Control IC-LoRA.
• Gemma text encoder + enhancer — provide prompt conditioning; the enhancer optionally expands a short prompt before generation.

Frozen audio (optional): the generic 2.5 workflow can freeze the reference video’s audio (VAEEncodeAudio → LTXVSetAudioRefTokens) so the generated result stays consistent with the source’s sound. Remove it if you want the audio to change to fit the adapter you’re using.

Example workflow steps:

1. Load your reference — an extracted control signal (depth map, pose skeleton, edge map, motion trajectory) or a reference video clip, depending on the adapter.
2. Apply the IC-LoRA at strength 1.0 (strength details here).
3. Write a descriptive text prompt (optionally enhance it).
4. Generate — the generic workflow runs single-stage distilled at your set resolution.

Preparing Control Signals

Control-type IC-LoRAs (Union Control, Motion Control) require an extracted control signal — a depth map, pose skeleton, edge map, or motion trajectory. The quality of these signals directly impacts results.

Most other IC-LoRAs (VFX, utility, and creative types) take a reference video clip directly as input, with no preprocessing or signal extraction needed. For these, skip to Reference Video below.

Depth Maps

Tools:

• Depthcrafter
• Blender/3D software - For synthetic depth renders

Best practices:

• Use consistent depth range across all frames
• Ensure smooth temporal transitions (avoid flickering)
• Match resolution to target generation resolution

Pose Skeletons

Tools:

• OpenPose - Multi-person pose estimation
• MediaPipe Pose - Fast single-person pose
• DWPose - High-quality pose estimation

Best practices:

• Extract poses at consistent frame rate matching target generation
• Ensure pose keypoints are detected reliably across all frames
• Handle occlusions gracefully (interpolate missing keypoints)
• Use skeleton visualization format (lines connecting keypoints)

Format: Typically 17-18 keypoints for body skeleton rendered as visual overlay

Canny Edges

Tools:

• OpenCV Canny edge detection
• PIL/Pillow image processing
• ComfyUI Canny preprocessor nodes

Best practices:

• Adjust threshold values to capture essential edges without noise
• Maintain consistent edge thickness across frames
• Blur input slightly before edge detection to reduce noise

Sparse Track Conditioning

Used with the Motion Control IC-LoRA. Instead of providing a per-frame visual signal (like depth or pose), you define motion by placing keypoints at specific frames. The system interpolates smooth spline trajectories between them, rendered as trails of circles that the IC-LoRA follows.

Tools:

• LTX Draw Tracks node (ComfyUI) — visual canvas for drawing motion paths
• LTX Sparse Track Editor node (ComfyUI) — fine-tune keypoint positions and timing

Best practices:

• Start with 3-4 keypoints per track; add more only if the interpolated path doesn’t match your intent
• Keep trajectories physically plausible — sudden direction changes produce less natural results
• Match the track canvas resolution to your generation resolution

Reference Video

Most IC-LoRAs outside the Control category, such as VFX (Water Simulation), utility (Colorization, Decompression, Deblurring), and creative types (Day to Night, Instant Shave, Cross-Eyed) condition directly on a reference video clip. No signal extraction, preprocessing, or external tools are needed.

Best practices:

• Feed the reference video directly into the IC-LoRA workflow as-is
• Match the reference clip’s resolution and frame rate to your generation settings (25 fps is standard for most adapters)
• Some adapters use a trigger phrase prepended to the prompt (e.g. DEBLUR, REMOVEBEARD, COLORIZE, ADD WATER, ENHANCE QUALITY) — check the adapter’s README file for the correct trigger

Light Direction

Used with the Relight IC-LoRA. Rather than a signal extracted from the source, you composite a small shaded “light-direction ball” into the reference video’s top-right corner to form a rendered element whose lit side tells the model where the key light should come from.

Tools:

• SphereLightNode (ComfyUI) — renders and composites the ball in-graph; built into current ComfyUI
• An ffmpeg composite (Python and custom integrations) — render a ball and overlay it on every frame

Best practices:

• Keep the caption’s direction consistent with the ball — the model follows the ball
• The ball’s corner geometry and the rotation/elevation → direction mapping are covered in the Relight guide

IC-LoRA Parameters

Control Strength

LTX provides full strength control for IC-LoRAs through two complementary parameters:

attention_strength — Global strength scalar (0.0 to 1.0)

Controls overall IC-LoRA influence on generation. This works by scaling the cross-attention scores between the conditioning signal tokens and the noisy latents.

• 1.0 — Full adherence to the control signal (default, matches previous behavior)
• 0.5 — Balanced blend of control signal and free generation
• 0.0 — Control signal is completely ignored

attention_mask — Spatial or spatiotemporal mask (optional)

An optional mask that provides region-level control over where the IC-LoRA takes effect. The mask is multiplied by attention_strength to produce the final per-region strength.

• Spatial mask (H×W): Apply control to specific areas of each frame (e.g., apply depth control only to the foreground)
• Spatiotemporal mask (T×H×W): Vary control across both space and time (e.g., gradually introduce pose control over the first 2 seconds)
• Values range from 0.0 (no control) to 1.0 (full control)
• When omitted, uniform full strength is applied everywhere

How it works under the hood:

The strength control operates at the attention layer level. During cross-attention between the noisy video latents and the IC-LoRA conditioning tokens, the attention scores are scaled by attention_strength × attention_mask. This means the control signal’s influence can be precisely dialed in globally, regionally, or both.

Practical examples:

• Soft depth guidance: Set attention_strength to 0.6 for depth IC-LoRA to get approximate spatial structure while allowing the model more creative freedom
• Foreground-only pose control: Use a spatial attention_mask that’s 1.0 on the character region and 0.0 on the background
• Gradual control fade-in: Use a spatiotemporal mask that ramps from 0.0 to 1.0 over the first 8 frames
• Multi-region control: Combine with spatial masks to apply different IC-LoRA strengths to different parts of the frame

Resolution and Frame Rate

• Match control signal resolution to generation resolution
• Control FPS should match target generation FPS
• For best results: 704x1216 at 24-30 FPS
• IC-LoRAs work at various resolutions but quality depends on control signal clarity

Upscaling

The Pixel Spatial Upscaler is a reference-video IC-LoRA that raises a clip’s resolution by 2× or 4×, synthesizing new spatial detail rather than interpolating pixels. It is a creative upscaler, not a pixel-accurate refiner or denoiser.

The recommended pattern is draft-then-upscale:

1. Generate your video at a low base resolution (around 280p) until the composition and motion are right.
2. Feed that clip as the reference and run the 2× or 4× upscaler to produce the final high-resolution output.

Choose the 2× variant for moderate upscaling or the 4× variant for larger resolution jumps. Fidelity to the source is tunable through LoRA strength, guidance, and step count: lower values stay closer to the reference, higher values allow more synthesized detail. It runs on the standard IC-LoRA video-to-video workflow — connect the low-resolution clip as the reference video.

Best Practices

Control Signal Quality

• Use high-quality control extraction tools
• Ensure temporal consistency (smooth transitions between frames)
• Match control resolution to generation resolution
• Pre-process control signals to remove noise and artifacts

Prompt Alignment

• For control-type IC-LoRAs, describe visual style, not control type (“ornate architecture” not “depth map shows…”)
• For IC-LoRAs with a trigger phrase, prepend it to the prompt before any scene description
• Align prompt with the reference input’s motion and composition
• Be specific about materials, lighting, and atmosphere
• Avoid contradicting the structure or content of the reference

Performance Optimization

• IC-LoRAs add minimal overhead (less than 10% compute)
• Works with FP8 quantized models
• Compatible with distilled models for faster generation

Quality Validation

• Always test IC-LoRA with simple control signals first
• Verify control is being respected before complex generations
• Compare with and without IC-LoRA to assess control strength
• Iterate on control signal quality before increasing generation complexity

Training Custom IC-LoRAs

Create your own IC-LoRA control adapters using the LTX Trainer.

IC-LoRA Troubleshooting

Control Signal Not Being Followed

Symptoms: Generated video ignores control structure

Solutions:

• Verify IC-LoRA is loaded correctly (check adapter name)
• Check attention_strength value — if set below 1.0, control will be proportionally weaker
• If using attention_mask, verify it covers the intended regions (mask values of 0.0 will disable control in those areas)
• Check control signal format matches expected input (resolution, channels)
• Validate control signal has sufficient contrast/detail
• Verify model compatibility (IC-LoRA version matches base model)

Temporal Flickering or Inconsistency

Symptoms: Unstable motion, frame-to-frame inconsistencies

Solutions:

• Smooth control signal temporal transitions (use interpolation)
• Increase inference steps (try 40-50 instead of 30)
• Ensure control signal FPS matches generation FPS
• Apply temporal filtering to control signal before generation
• Check for abrupt changes in control signal values

Poor Quality or Artifacts

Symptoms: Visual artifacts, degraded quality, unwanted textures

Solutions:

• Improve control signal quality (better extraction tools)
• Ensure control signal resolution is adequate
• Adjust prompt to better describe desired style
• Check that control signal doesn’t have noise or compression artifacts

Control Too Strong or Rigid

Symptoms: Output looks constrained, lacks natural variation

Solutions:

• Reduce attention_strength to 0.5-0.8 for a softer control effect
• Use a spatial attention_mask to limit control to specific regions
• Adjust your control signal to be less restrictive (e.g., lighter edges)
• Use more flexible prompts that allow style variation

Memory Issues

Symptoms: Out of memory errors during generation

Solutions:

• Use FP8 quantized models to reduce VRAM
• Reduce generation resolution

Resources

• IC-LoRA Adapters — All official IC-LoRA adapters with checkpoints, compatibility, and workflows
• LTX Trainer — Train custom IC-LoRA control adapters