🎬 COMPLETE ROADMAP: Building Image↔Video AI Models & Services

1.1 What Are These Tasks?

Image-to-Video (I2V)

• Definition: Generating a temporally coherent video sequence from one or more static images as conditioning input
• Core Challenge: Hallucinating plausible motion, depth, occlusion, lighting dynamics that are consistent with the source image
• Examples: Animating a portrait photo, generating camera panning from a landscape, adding realistic rain to a still scene

Video-to-Image (V2I)

• Definition: Extracting, summarizing, reconstructing, or stylizing still images from video sequences
• Sub-tasks:
  • Key-frame extraction
  • Video frame interpolation (super-resolution in time)
  • Video-to-style-transfer (apply image style to every frame)
  • Video summarization to single composite image
  • Depth map / segmentation map extraction per frame
  • Video inpainting → still output

1.2 The Unified Vision: Spatiotemporal Synthesis

Both tasks are fundamentally about spatiotemporal modeling:

• Spatial: Understanding scene geometry, objects, textures, lighting
• Temporal: Understanding motion fields, optical flow, causality, physics

PHASE 0 — Mathematical Foundations (Weeks 1–6)

2.0.1 Linear Algebra (Essential)

• Vectors, matrices, tensors (3D/4D for video)
• Eigenvalues, SVD, PCA — used in feature decomposition
• Matrix factorization — used in optical flow and compression
• Resources: Gilbert Strang's MIT OCW Linear Algebra, 3Blue1Brown series

2.0.2 Probability & Statistics

• Probability distributions: Gaussian, Categorical, Bernoulli
• Bayesian inference — core to diffusion models
• KL divergence, Jensen-Shannon divergence — used in VAE, GAN losses
• Maximum Likelihood Estimation (MLE)
• Monte Carlo methods, importance sampling
• Resources: Bishop's "Pattern Recognition and Machine Learning" Ch.1–2

2.0.3 Calculus & Optimization

• Partial derivatives, chain rule (backpropagation)
• Gradient descent variants: SGD, Adam, AdamW, LAMB
• Second-order methods (Newton, L-BFGS)
• Stochastic differential equations (SDEs) — for diffusion models
• Resources: "Deep Learning" by Goodfellow, Bengio & Courville

2.0.4 Signal Processing

• Fourier Transform, Discrete Cosine Transform (DCT) — video compression
• Convolution and correlation
• Nyquist theorem — temporal sampling for video
• Wavelet transforms — multi-scale feature extraction
• Resources: Oppenheim's "Discrete-Time Signal Processing"

2.0.5 Information Theory

• Entropy, cross-entropy — classification losses
• Mutual information — used in contrastive learning
• Rate-distortion theory — video codecs
• Resources: Cover & Thomas "Elements of Information Theory"

PHASE 1 — Deep Learning Core (Weeks 7–16)

2.1.1 Neural Network Fundamentals

• Perceptron → MLP → Universal Approximation Theorem
• Activation functions: ReLU, GELU, Swish, SiLU
• Normalization: BatchNorm, LayerNorm, GroupNorm, RMSNorm
• Regularization: Dropout, Weight Decay, Spectral Norm
• Loss functions: MSE, MAE, Perceptual loss, SSIM, LPIPS

2.1.2 Convolutional Neural Networks (CNN)

• Conv2D → Conv3D (for video)
• Depthwise separable convolutions
• Dilated/Atrous convolutions
• Transposed convolutions (deconvolution) — upsampling in generators
• ResNet, VGG, EfficientNet architectures
• Feature Pyramid Networks (FPN)
• Key Paper: "Deep Residual Learning" — He et al. (2015)

2.1.3 Recurrent Neural Networks (RNN)

• Vanilla RNN, LSTM, GRU — temporal modeling
• Bidirectional RNNs
• Sequence-to-sequence models
• ConvLSTM — spatial + temporal in one module
• Application: Early video prediction models

2.1.4 Attention Mechanisms & Transformers

• Self-attention, cross-attention, multi-head attention
• Positional encodings: sinusoidal, RoPE, ALiBi
• Vision Transformer (ViT)
• Swin Transformer — hierarchical vision transformer
• Video Swin Transformer — extends to temporal dimension
• Flash Attention — memory-efficient attention
• Key Papers: "Attention is All You Need" (Vaswani 2017), ViT (Dosovitskiy 2020)

2.1.5 Generative Models — Core Theory

Variational Autoencoders (VAE)

• Encoder-decoder structure
• Reparameterization trick
• ELBO loss = Reconstruction + KL divergence
• KL annealing
• Role in Video AI: Compress video frames to latent space

Generative Adversarial Networks (GAN)

• Generator vs Discriminator adversarial training
• Mode collapse problem and solutions
• WGAN, WGAN-GP (gradient penalty)
• Progressive growing (ProGAN)
• StyleGAN, StyleGAN2, StyleGAN3
• Conditional GAN (cGAN), Pix2Pix, CycleGAN
• Temporal discriminators for video
• Key Papers: Goodfellow 2014, Karras 2019/2020/2021

Normalizing Flows

• Invertible transformations
• GLOW, RealNVP
• Used for exact likelihood computation

Diffusion Models (DDPM, Score Matching)

• Forward process: gradually add Gaussian noise
• Reverse process: learn to denoise
• DDPM (Ho et al., 2020)
• Score-based generative models (Song et al.)
• DDIM — deterministic, faster sampling
• Latent Diffusion Models (LDM) — work in VAE latent space
• This is the dominant paradigm today for I2V

2.1.6 Contrastive & Self-Supervised Learning

• SimCLR, MoCo, BYOL
• CLIP (Contrastive Language-Image Pretraining) — text-image alignment
• DINO, DINOv2 — self-supervised ViT features
• Application: Building rich image/video embeddings for conditioning

PHASE 2 — Computer Vision Specialization (Weeks 17–26)

2.2.1 Image Understanding

• Object detection: YOLO family, DETR, Faster RCNN
• Semantic segmentation: UNet, DeepLab, Mask2Former
• Instance segmentation: Mask RCNN, SAM (Segment Anything Model)
• Depth estimation: MiDaS, DPT, ZoeDepth
• Image matting and compositing
• Super-resolution: SRCNN, ESRGAN, Real-ESRGAN

2.2.2 Optical Flow & Motion Estimation

• Classical: Lucas-Kanade, Horn-Schunck, Farnebäck
• Deep learning: FlowNet, PWCNet, RAFT (Recurrent All-Pairs Field Transforms)
• RAFT is state-of-the-art for dense optical flow
• Scene flow (3D motion estimation)
• Motion segmentation
• Application: Understanding what should move in I2V generation

2.2.3 Video Understanding

• Action recognition: SlowFast, I3D, TimeSformer
• Temporal localization
• Video object tracking: SORT, DeepSORT, ByteTrack
• Video object segmentation: DAVIS benchmark models
• Scene understanding in video

2.2.4 3D Vision (Critical for Advanced I2V)

• Camera models: pinhole, intrinsics/extrinsics
• Structure from Motion (SfM)
• Neural Radiance Fields (NeRF) — 3D scene representation
• Instant-NGP — fast NeRF
• 3D Gaussian Splatting — real-time 3D rendering
• Depth-conditioned generation
• Application: Camera motion control in I2V (e.g., moving camera viewpoint)

2.2.5 Image/Video Quality Metrics

• PSNR (Peak Signal-to-Noise Ratio)
• SSIM (Structural Similarity Index)
• LPIPS (Learned Perceptual Image Patch Similarity)
• FID (Fréchet Inception Distance) — for images
• FVD (Fréchet Video Distance) — for videos
• IS (Inception Score)
• CLIP Score — semantic alignment with text
• DOVER, BVQA — video quality assessment

PHASE 3 — Core I2V & V2I Model Architectures (Weeks 27–40)

2.3.1 Video Generation Fundamentals

Temporal Architecture Choices:

• 3D Convolutions: Process space+time together (C3D, I3D)
• Pseudo-3D (P3D): Decompose 3D conv into 2D spatial + 1D temporal
• Conv + RNN Hybrid: CNN features fed into LSTM
• Full Transformer: Spatial + temporal attention (Video Transformer)
• Factorized Attention: Separate spatial and temporal attention heads

Key I2V Conditioning Methods:

• Image as first frame: Concatenate with noise
• Image embeddings via CLIP: Text-like conditioning
• Image + optical flow: Motion-guided generation
• ControlNet-style conditioning: Structural guidance
• Reference attention: Cross-attention to reference image tokens

2.3.2 Diffusion-Based Video Models (Dominant Approach)

Latent Video Diffusion Models (LVDM)

• Encode all frames into latent space using 3D VAE
• Apply diffusion in compressed latent space
• Key advantage: 10–100× more memory efficient
• Temporal attention and 3D U-Net backbone

Video Diffusion Models (VDM) — Ho et al. 2022

• Extended DDPM to video
• Joint distribution over all frames
• Hierarchical generation (keyframes → interpolation)

AnimateDiff

• Plug-and-play motion module for Stable Diffusion
• Trains motion module separately on video data
• Works with existing SD image checkpoints
• Architecture: Insert temporal attention blocks into SD U-Net

Stable Video Diffusion (SVD) — Stability AI

• Fine-tuned from Stable Diffusion image model
• Image conditioning via CLIP + VAE
• 25-frame generation at various resolutions
• Key insight: Multi-stage training (text→image→video)

CogVideoX — Zhipu AI

• Full 3D attention model
• Expert transformer blocks
• 3D causal VAE
• Trained with video-text pairs
• Open source, competitive with proprietary models

Open-Sora, Open-Sora-Plan

• Community implementations of Sora-like architectures
• DiT (Diffusion Transformer) backbone
• Variable length, resolution, aspect ratio

Architecture Deep Dive: Video DiT (Diffusion Transformer for Video)

• Replace U-Net with Transformer backbone
• Patch tokens from video frames (space-time patches)
• 3D RoPE positional encoding
• Full 3D attention or factorized temporal+spatial
• Scalable: more parameters → better quality

2.3.3 Video-to-Image Architectures

Frame Extraction & Processing Pipeline

• Keyframe detection algorithms: histogram difference, SSIM drop, shot boundary detection
• Thumbnail generation systems
• Adaptive sampling (dense for action, sparse for static)

Video Super-Resolution → High-Res Stills

• EDVR (Enhanced Deformable Video Restoration)
• BasicVSR, BasicVSR++ — recurrent video SR
• Real-BasicVSR — for real-world degradation
• RVRT (Recurrent Video Restoration Transformer)

Video Style Transfer

• AdaIN (Adaptive Instance Normalization) applied per-frame
• ReReVST — temporally consistent style transfer
• Optical-flow-guided consistency

Video Inpainting

• STTN (Spatial-Temporal Transformer Network)
• ProPainter — propagation-based video inpainting
• Applications: watermark removal, object removal, background replacement

Video Summarization

• Encoder-decoder with attention over frame sequence
• Clustering-based: K-means over CNN features
• Submodular optimization for frame selection

PHASE 4 — Advanced Conditioning & Control (Weeks 41–50)

2.4.1 Text-to-Video Pathway (Prerequisite for Full I2V Pipeline)

• CLIP/T5 text encoder → conditioning signal
• Cross-attention for text guidance
• Classifier-Free Guidance (CFG) for controllability
• Text-guided motion: "the dog runs left"

2.4.2 ControlNet for Video

• Depth maps, edge maps, pose as structural conditions
• Temporal consistency of control signals
• Video ControlNet: extends ControlNet to temporal domain
• Application: Consistent character animation from pose sequence

2.4.3 IP-Adapter (Image Prompt Adapter)

• Inject image features into cross-attention
• Decoupled from text conditioning
• Works with any SD checkpoint
• Application: Strong image reference in I2V

2.4.4 Camera Control

• CameraCtrl: encode camera trajectories
• MotionCtrl: unified motion control
• ViewCrafter: novel view synthesis for video
• 3D-aware video generation using camera intrinsics/extrinsics
• Plücker coordinates for camera representation

2.4.5 Motion Control

• Drag-based motion (DragNUWA, DragAnything)
• Flow-guided generation
• Trajectory-conditioned animation
• Physics-based motion priors

2.4.6 Audio-Driven Video

• Lip sync: SadTalker, Wav2Lip, EchoMimic
• Full-body audio-driven animation
• EMO (Emote Portrait Alive)
• Hallo, Hallo2 series

PHASE 5 — Training Infrastructure (Weeks 51–60)

2.5.1 Data Pipeline

• Video dataset collection and curation
• Scene cut detection (PySceneDetect, TransNetV2)
• Aesthetic scoring (LAION aesthetics predictor)
• OCR filtering (remove text-heavy frames)
• Motion filtering (optical flow magnitude)
• Deduplication (perceptual hashing, embedding similarity)
• Caption generation (CogVLM, LLaVA, GPT-4V for dense captions)

2.5.2 Distributed Training

• Data parallelism: DDP (DistributedDataParallel)
• Model parallelism: Tensor Parallelism, Pipeline Parallelism
• DeepSpeed ZeRO (Zero Redundancy Optimizer): ZeRO-1, 2, 3
• FSDP (Fully Sharded Data Parallel)
• Gradient checkpointing (activation recomputation)
• Mixed precision: FP16, BF16, FP8 (emerging)
• Flash Attention 2/3 — memory efficient attention

2.5.3 Training Strategies

• Pretraining on image data → fine-tune on video
• Curriculum learning: start with short videos, scale up
• Progressive resolution training
• Flow matching (replacing DDPM noise scheduler)
• Rectified Flow — straight-path ODE, faster training convergence
• Min-SNR weighting — balanced loss across noise levels

2.5.4 Fine-tuning Methods

• LoRA (Low-Rank Adaptation) — efficient fine-tuning
• DreamBooth for video — personalized video generation
• Textual Inversion
• DoRA, AdaLoRA — improved LoRA variants

PHASE 6 — Inference Optimization & Deployment (Weeks 61–70)

2.6.1 Sampling Acceleration

• DDIM (50 steps → deterministic)
• DPM-Solver, DPM-Solver++ (20 steps)
• UniPC (10 steps)
• DDPM with fewer steps via distillation
• Consistency Models (1–4 steps)
• LCM (Latent Consistency Models)
• Adversarial Diffusion Distillation (ADD) — used in SDXL-Turbo

2.6.2 Model Compression

• Quantization: INT8, INT4 (GPTQ, AWQ for transformers)
• Pruning: structured and unstructured
• Knowledge distillation
• TensorRT optimization
• ONNX export for cross-platform deployment

2.6.3 Efficient Serving

• Batching strategies for diffusion models
• Continuous batching for transformer decoders
• KV-cache for transformer video models
• Model caching and hot-loading
• Speculative decoding for consistency models

2.6.4 Infrastructure Stack

• NVIDIA Triton Inference Server
• vLLM (for transformer-based video models)
• ComfyUI backend for pipeline orchestration
• BentoML, Ray Serve for scalable serving
• FastAPI + Celery + Redis for async job queues
• Docker + Kubernetes for container orchestration

3.1 Core Algorithm Families

Generative Algorithms

Algorithm	Type	Best For	Year
DDPM	Diffusion	High-quality generation	2020
DDIM	Diffusion	Fast inference	2020
LDM	Latent Diffusion	Memory efficient	2022
Flow Matching	ODE-based	Stable training	2022
Rectified Flow	ODE-based	Fast convergence	2022
DiT	Transformer Diffusion	Scalable quality	2022
Consistency Models	Distillation	1-step generation	2023
GAN (StyleGAN3)	Adversarial	Video coherence	2021
VideoVAE (3D-VAE)	Compression	Temporal latent	2023

Motion & Flow Algorithms

Algorithm	Type	Application
RAFT	Deep Optical Flow	Motion extraction
FlowFormer	Transformer Flow	High-quality flow
GMFlow	Global Matching Flow	Efficiency
UniMatch	Unified Flow+Stereo	Multi-task
Scene Flow	3D Motion	Depth-aware motion

Temporal Consistency Algorithms

Method	Principle
Optical Flow Warping	Warp previous frame features
Temporal Attention	Attend across frame tokens
ConvLSTM	Recurrent spatial states
Deformable Convolutions	Adaptive receptive fields
Cross-frame Attention	Direct token communication

3.2 Key Techniques

For Image-to-Video

1. Reference Attention: Store image features in KV cache, all video frames attend to image
2. Dual-stream Architecture: Separate image encoder + video decoder
3. Anchor Frame Conditioning: First/last frame conditioning
4. Pose-guided Animation: Extract pose from image, drive motion
5. Flow Prediction Module: Predict optical flow, then synthesize frames
6. Temporal Self-Attention Inflation: Extend 2D attention to temporal
7. 3D VAE Encoding: Encode video as 3D latent tensor
8. CLIP Visual Conditioning: Global image semantics as guidance
9. CFG (Classifier-Free Guidance): Balance faithfulness vs creativity
10. Noise Augmentation: Add noise to conditioning image for robustness

For Video-to-Image

1. Deformable Convolution Alignment: Align frames before aggregation
2. Non-local Means across frames: Temporal denoising
3. Sliding Window Processing: Handle long videos
4. Propagation-based Inpainting: Propagate known pixels across time
5. Recurrent Feature Propagation: LSTM over frame features
6. Keyframe Selection via Clustering: Representative frame extraction
7. Temporal Super-Resolution: Hallucinate intermediate frames

3.3 Complete Tool Ecosystem

Deep Learning Frameworks

• PyTorch (primary for research + production)
• JAX / Flax (Google TPU, high-performance)
• TensorFlow / Keras (legacy, enterprise)
• MXNet (AWS, less common)

Video & Image Processing

• OpenCV — classical computer vision
• FFmpeg — video encoding/decoding/processing
• Decord — fast GPU video decoding
• torchvision / torchcodec — PyTorch video loading
• imageio, Pillow, scikit-image — image manipulation
• PyAV — Python FFmpeg bindings
• moviepy — programmatic video editing

Diffusion Model Libraries

• Diffusers (HuggingFace) — modular diffusion implementations
• ComfyUI — node-based pipeline builder
• Automatic1111 (AUTOMATIC1111/stable-diffusion-webui) — UI for SD
• InvokeAI — professional creative tool
• kohya_ss — fine-tuning scripts

Training Infrastructure

• DeepSpeed — distributed training, ZeRO optimizer
• Accelerate (HuggingFace) — simple distributed training wrapper
• FSDP (PyTorch native) — fully sharded data parallel
• Megatron-LM — NVIDIA's large-scale training
• Lightning (PyTorch Lightning) — structured training loops
• Wandb / TensorBoard — experiment tracking
• MLflow — ML lifecycle management
• DVC — data version control

Data Tools

• LAION datasets — large-scale image/video datasets
• WebDataset — efficient streaming for large datasets
• FFCV — fast computer vision data loading
• Albumentations — image augmentation
• vidaug — video augmentation
• PySceneDetect — scene cut detection
• Whisper — audio transcription for captions

Cloud & GPU Platforms

• NVIDIA A100, H100, H200 — primary training GPUs
• AWS (SageMaker, EC2 p4/p5) — cloud training
• Google Cloud (TPU v4, v5, A100 VMs)
• Azure (ND A100 clusters)
• Lambda Labs — affordable GPU cloud
• Vast.ai — marketplace GPU rental
• RunPod — GPU pods for inference/fine-tuning

Serving & Deployment

• FastAPI — async Python API framework
• Celery + Redis/RabbitMQ — async task queue
• NVIDIA Triton — inference server
• TorchServe — PyTorch model serving
• BentoML — ML model serving framework
• Ray Serve — scalable model serving
• Docker + Kubernetes — containerized deployment
• AWS Lambda + S3 — serverless for pre/post-processing

Monitoring & Observability

• Prometheus + Grafana — metrics and dashboards
• Datadog — APM and infrastructure monitoring
• Sentry — error tracking
• OpenTelemetry — distributed tracing

4.1 Forward Engineering: Scratch to Production

STEP 1: Environment Setup

# System Requirements
# Ubuntu 22.04 LTS (recommended)
# CUDA 12.1+, cuDNN 8.9+
# Python 3.10+

# Environment
conda create -n video_ai python=3.10
conda activate video_ai

# Core packages
pip install torch torchvision torchaudio --index-url https://cuda.pytorch.org/whl/cu121
pip install diffusers transformers accelerate
pip install opencv-python-headless decord
pip install einops timm xformers
pip install deepspeed wandb

# Video tools
apt-get install ffmpeg libavcodec-dev
pip install ffmpeg-python moviepy

STEP 2: Data Collection & Preprocessing

Dataset Sources for Training:

• WebVid-10M — 10M web video clips with captions
• Panda-70M — 70M high-quality video clips
• InternVid — 234M video clips
• LAION-5B — images (for pre-training)
• HD-VILA-100M — 100M high-definition clips
• OpenVid-1M — curated 1M clips for fine-tuning

Preprocessing Pipeline:

Raw Videos
    ↓
Scene Cut Detection (TransNetV2)
    ↓
Quality Filtering (BRISQUE/CLIP score)
    ↓
Motion Filtering (optical flow magnitude)
    ↓
Resolution Check (≥256x256)
    ↓
Duration Filtering (2–30 seconds)
    ↓
Caption Generation (LLaVA/CogVLM)
    ↓
Deduplication (perceptual hashing)
    ↓
Shard into WebDataset format
    ↓
Upload to distributed storage (S3/GCS)

STEP 3: Model Architecture Design

Minimal I2V Architecture (Start Here):

Input: Image (3, H, W) + Noise latent (C, T, H//8, W//8)
         ↓
Image Encoder (VAE encoder):  → image_latent (C, H//8, W//8)
         ↓
Reference Features (image_latent → projected to cross-attn keys/values)
         ↓
3D U-Net Backbone:
  Down Blocks (ResBlock3D + Temporal Attn + Cross Attn)
  Middle Block (ResBlock3D + Full Attn)
  Up Blocks (ResBlock3D + Temporal Attn + Cross Attn)
         ↓
Output: Predicted noise (C, T, H//8, W//8)
         ↓
VAE Decoder → Video frames (3, T, H, W)

U-Net 3D Block Design:

class TemporalResBlock(nn.Module):
    """Spatial ResBlock + Temporal attention"""
    def __init__(self, channels, num_frames):
        self.spatial_resblock = ResBlock2D(channels)
        self.temporal_attn = TemporalAttention(channels, num_frames)
        self.norm = GroupNorm(32, channels)
    
    def forward(self, x):
        # x: (B, C, T, H, W)
        B, C, T, H, W = x.shape
        # Process spatially
        x = rearrange(x, 'b c t h w -> (b t) c h w')
        x = self.spatial_resblock(x)
        x = rearrange(x, '(b t) c h w -> b c t h w', b=B)
        # Process temporally
        x = rearrange(x, 'b c t h w -> (b h w) t c')
        x = self.temporal_attn(x)
        x = rearrange(x, '(b h w) t c -> b c t h w', b=B, h=H)
        return x

STEP 4: Training Loop Design

# Simplified I2V Training Loop

def train_step(batch, model, scheduler, optimizer):
    images = batch['image']       # (B, 3, H, W) - conditioning
    videos = batch['video']       # (B, 3, T, H, W) - target
    
    # 1. Encode to latent space
    with torch.no_grad():
        image_latent = vae.encode(images).latent_dist.sample() * 0.18215
        video_latents = vae.encode(
            rearrange(videos, 'b c t h w -> (b t) c h w')
        ).latent_dist.sample() * 0.18215
        video_latents = rearrange(video_latents, '(b t) c h w -> b c t h w', b=B)
    
    # 2. Sample noise and timestep
    noise = torch.randn_like(video_latents)
    timesteps = torch.randint(0, scheduler.num_train_timesteps, (B,))
    
    # 3. Add noise (forward diffusion process)
    noisy_latents = scheduler.add_noise(video_latents, noise, timesteps)
    
    # 4. Get image conditioning
    image_embeds = image_encoder(images)  # CLIP features
    
    # 5. Predict noise
    noise_pred = model(noisy_latents, timesteps, 
                       encoder_hidden_states=image_embeds,
                       image_latent=image_latent)
    
    # 6. Compute loss (v-prediction or epsilon)
    if scheduler.prediction_type == 'epsilon':
        target = noise
    elif scheduler.prediction_type == 'v_prediction':
        target = scheduler.get_velocity(video_latents, noise, timesteps)
    
    loss = F.mse_loss(noise_pred, target, reduction='none')
    
    # 7. Min-SNR weighting for balanced training
    snr = compute_snr(timesteps)
    mse_loss_weights = torch.stack([snr, 5 * torch.ones_like(snr)], dim=1).min(dim=1)[0] / snr
    loss = (loss.mean(dim=list(range(1, len(loss.shape)))) * mse_loss_weights).mean()
    
    # 8. Backprop
    optimizer.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    
    return loss.item()

STEP 5: Inference Pipeline

def image_to_video_inference(
    image_path: str,
    prompt: str = "",
    num_frames: int = 25,
    height: int = 576,
    width: int = 1024,
    num_inference_steps: int = 25,
    fps: int = 7,
    motion_bucket_id: int = 127,
    guidance_scale: float = 7.5
):
    # Load pipeline
    pipe = StableVideoDiffusionPipeline.from_pretrained(
        "stabilityai/stable-video-diffusion-img2vid-xt",
        torch_dtype=torch.float16, variant="fp16"
    )
    pipe.to("cuda")
    pipe.enable_model_cpu_offload()  # Memory optimization
    
    # Load and preprocess image
    image = Image.open(image_path).convert("RGB")
    image = image.resize((width, height))
    
    # Generate video
    generator = torch.manual_seed(42)
    frames = pipe(
        image,
        decode_chunk_size=8,    # Process 8 frames at a time
        generator=generator,
        motion_bucket_id=motion_bucket_id,
        noise_aug_strength=0.02,
        num_frames=num_frames,
        num_inference_steps=num_inference_steps,
    ).frames[0]
    
    # Export to MP4
    export_to_video(frames, "output.mp4", fps=fps)
    return frames

STEP 6: Evaluation System

class VideoQualityEvaluator:
    def __init__(self):
        self.fvd_model = load_i3d_model()
        self.clip_model = load_clip_model()
    
    def compute_fvd(self, real_videos, generated_videos):
        """Fréchet Video Distance"""
        real_feats = self.extract_i3d_features(real_videos)
        gen_feats = self.extract_i3d_features(generated_videos)
        return frechet_distance(real_feats, gen_feats)
    
    def compute_clip_consistency(self, frames):
        """Frame-to-frame CLIP feature consistency"""
        embeddings = [self.clip_model.encode_image(f) for f in frames]
        similarities = [cos_sim(embeddings[i], embeddings[i+1]) 
                       for i in range(len(embeddings)-1)]
        return np.mean(similarities)
    
    def compute_motion_smoothness(self, frames):
        """Optical flow magnitude variance"""
        flows = [compute_optical_flow(frames[i], frames[i+1]) 
                for i in range(len(frames)-1)]
        return np.mean([np.std(f) for f in flows])

4.2 Reverse Engineering Method

What is Reverse Engineering in AI? Starting from a working model and dissecting it to understand its internals — then applying insights to build your own.

Step 1: Obtain and Run Reference Model

# Download Stable Video Diffusion
git clone https://github.com/Stability-AI/generative-models
cd generative-models
pip install -e .

# Run inference
python scripts/sampling/simple_video_sample.py \
    --input_path assets/test_image.png \
    --output_folder outputs/

Step 2: Inspect Model Architecture

import torch
from diffusers import StableVideoDiffusionPipeline

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid"
)

# Print full architecture
print(pipe.unet)

# Count parameters
total_params = sum(p.numel() for p in pipe.unet.parameters())
print(f"UNet params: {total_params/1e9:.2f}B")

# Inspect individual blocks
for name, module in pipe.unet.named_modules():
    print(f"{name}: {type(module).__name__}")

Step 3: Hook-Based Feature Extraction

# Extract intermediate activations to understand information flow
activations = {}

def hook_fn(name):
    def hook(module, input, output):
        activations[name] = output.detach()
    return hook

# Register hooks
for name, module in pipe.unet.named_modules():
    if 'temporal_attn' in name:
        module.register_forward_hook(hook_fn(name))

# Run inference
frames = pipe(image).frames

# Visualize temporal attention patterns
for key, feat in activations.items():
    print(f"{key}: {feat.shape}")
    # Visualize attention maps
    visualize_attention(feat, key)

Step 4: Ablation Study

• Remove temporal attention → measure FVD increase
• Disable image conditioning → measure semantic drift
• Change noise scheduler → measure speed/quality tradeoff
• Reduce U-Net channels → measure capacity vs efficiency

Step 5: Identify Transferable Components

From SVD reverse engineering, key learnings:

• The 3D VAE temporal compression is the most critical component
• Reference attention (image → all frames) beats simple concat
• Noise augmentation on input image is critical for robustness
• Motion bucket ID is a clever scalar conditioning for motion magnitude

Step 6: Rebuild with Modifications

Use the insights to design your custom model with improvements.

5.1 Core Working Principles

How Image-to-Video Works (Step by Step)

PHASE A: ENCODING
═══════════════════
Input Image (RGB, H×W)
    → VAE Encoder → Latent z_image (C, H/8, W/8)
    → CLIP Image Encoder → Global semantic embedding e_clip (1, 1024)

PHASE B: NOISE INITIALIZATION
══════════════════════════════
T frames of pure Gaussian noise: z_T (C, T, H/8, W/8)
Concatenate z_image to z_T as conditioning (channel-wise or cross-attn)

PHASE C: ITERATIVE DENOISING (Reverse Diffusion)
══════════════════════════════════════════════════
For t = T, T-1, ..., 1:
    input = concat([z_t, z_image_broadcasted])  (C×2, T, H/8, W/8)
    
    predicted_noise = UNet3D(
        input,
        timestep=t,
        image_embed=e_clip,
        reference_features=from_image_encoder
    )
    
    z_{t-1} = scheduler.step(predicted_noise, t, z_t)

    # CFG: blend conditional and unconditional predictions
    ε_uncond = UNet3D(input, t, image_embed=zeros, ...)
    ε_cond = UNet3D(input, t, image_embed=e_clip, ...)
    ε_final = ε_uncond + cfg_scale × (ε_cond - ε_uncond)

PHASE D: DECODING
══════════════════
Final latent z_0 (C, T, H/8, W/8)
    → Decode frame by frame: VAE Decoder(z_0[:, t, :, :])
    → Output: T frames of RGB video (3, T, H, W)

Why Does This Work? The Mathematics

Score Function: The model learns the score ∇_x log p(x), which points toward regions of higher data density.

Denoising: At each step, the model takes a noisy video latent and moves it towards the manifold of real videos, conditioned on the source image.

Temporal Coherence: Temporal attention ensures that tokens from different time steps can directly communicate, preventing frame-to-frame flickering. The attention weights encode "what should persist across time" vs "what should change."

5.2 Architecture Comparison

Architecture 1: 3D U-Net (Most Common Today)

Input Latent: (B, C, T, H, W)
    ↓
[Down Block 1]  : ResBlock3D → TemporalAttn → SpatialAttn → CrossAttn(img)
    ↓ Downsample(spatial)
[Down Block 2]  : ResBlock3D → TemporalAttn → SpatialAttn → CrossAttn(img)
    ↓ Downsample(spatial)
[Down Block 3]  : ResBlock3D → TemporalAttn → SpatialAttn → CrossAttn(img)
    ↓
[Middle Block]  : ResBlock3D → Full3DAttn → ResBlock3D
    ↓
[Up Block 3]    : ResBlock3D (+ skip) → TemporalAttn → SpatialAttn → CrossAttn(img)
    ↑ Upsample(spatial)
[Up Block 2]    : ResBlock3D (+ skip) → TemporalAttn → SpatialAttn → CrossAttn(img)
    ↑ Upsample(spatial)
[Up Block 1]    : ResBlock3D (+ skip) → TemporalAttn → SpatialAttn → CrossAttn(img)
    ↓
Output Conv → Predicted noise: (B, C, T, H, W)

Pros: Well-established, good inductive bias for local features, compatible with SD weights via inflation

Cons: Limited global temporal modeling, quadratic memory with resolution

Architecture 2: Video DiT (Emerging Standard)

Video Patches: (B, N_space × N_time, D)
Where N_space = (H/p)(W/p), N_time = T/pt

Patch Embedding (3D patchify)
    ↓
[DiT Block × N]:
    LayerNorm
    → Full 3D Self-Attention (or factorized spatial+temporal)
    → LayerNorm  
    → Cross-Attention with image/text conditioning
    → LayerNorm
    → MLP (4× expand, GELU, 4× contract)
    → AdaLayerNorm modulation (timestep + conditioning)
    ↓
Unpatchify → Predicted noise: (B, C, T, H, W)

Pros: Global attention, scales with compute, no inductive bias constraints

Cons: Quadratic in sequence length, requires longer training from scratch

Architecture 3: Mamba / SSM-based (Emerging)

• State Space Models for linear-complexity temporal modeling
• VideoMamba architecture
• Promising for very long videos

5.3 3D VAE Architecture (Critical Component)

VIDEO ENCODER (3D Causal VAE)
═════════════════════════════
Input Video: (B, 3, T, H, W)

CausalConv3D blocks (causal = no future leakage in time dim)
  → (B, C1, T, H/2, W/2)
Temporal Downsampling (if T > 1)
  → (B, C2, T/4, H/4, W/4)
Spatial Downsampling
  → (B, C3, T/4, H/8, W/8)
  
μ, σ heads → Latent z: (B, 16, T/4, H/8, W/8)

Compression ratio: 4× temporal, 8× spatial, RGB→16ch
Typical: 256×256×16 video → 32×32×4 latent per frame

5.4 Hardware Requirements

Training Hardware

Minimum Viable (Prototype/Research)

Component	Spec	Notes
GPU	2× NVIDIA A100 80GB	Minimum for 256×256 video
CPU	AMD EPYC 7742 or Intel Xeon	64+ cores
RAM	256GB DDR4	For data loading
Storage	10TB NVMe SSD	Dataset + checkpoints
Network	100Gbps InfiniBand	Multi-node training
Cost/month	~$6,000 (cloud)	AWS p4d.24xlarge

Production Training Setup

Component	Spec	Notes
GPU	64× H100 80GB (8 nodes)	Large model training
Interconnect	NVLink 3.0 + InfiniBand NDR	Critical for efficiency
CPU	2× AMD EPYC 9654 per node	High core count
RAM	2TB DDR5 per node
Storage	100TB all-NVMe shared storage	Lustre/GPFS
Cost/month	~$500,000+	Hyperscale training

Memory Calculations

Model: ~3B parameter UNet3D
Parameters: 3B × 4B (fp32) = 12GB
  Or: 3B × 2B (fp16) = 6GB

Optimizer states (AdamW): 3× model = 36GB (fp32 master weights)

Activations per sample (example):
  Video: 16 frames × 64×64 latent × 4 channels × 2B = 512MB
  Attention: T×H×W × T×H×W attention matrices → scales quadratically!

Gradient checkpointing: Trade 30% speed for ~60% activation memory

Minimum GPU memory per device: 40–80GB for small models

Inference Hardware

Consumer / Developer

Setup	GPU	Memory	Speed	Cost
Laptop	RTX 4090	24GB	5fps (512×512)	$1,600
Desktop	RTX 3090	24GB	3fps (512×512)	$700
Workstation	2× A5000	48GB	8fps (768×768)	$3,000

Production Inference

Setup	GPU	Memory	Throughput	Cost/month
Single inference	A10G 24GB	24GB	1 video/20s	$1.20/hr
Batch inference	A100 80GB	80GB	4 videos/20s	$3.20/hr
High throughput	H100 80GB	80GB	8 videos/20s	$6.50/hr

Memory Optimization Techniques

1. CPU Offloading: Non-active model parts in RAM
2. Sequential CPU Offloading: Layer-by-layer on CPU
3. xFormers / Flash Attention: Reduce attention memory O(N²) → O(N)
4. Sliced VAE Decoding: Decode one frame at a time
5. BF16 / FP16: Half precision (2× memory savings)
6. 8-bit Quantization: (bitsandbytes) ~4× memory savings

6.1 2024–2025 State of the Art

Proprietary Models (Reference Benchmarks)

Model	Company	Capability	Notes
Sora	OpenAI	60s, 1080p	Transformer + Flow Matching, sparse 3D attention
Veo 2	Google DeepMind	4K, physics-aware	Better temporal coherence, camera control
Kling 1.6	Kuaishou	2min, cinematic	Strong Chinese-language I2V
Gen-3 Alpha	Runway	High quality, fast	Professional creative tool
Dream Machine 1.5	Luma AI	Realistic motion	Good for product videos
Hailuo MiniMax	MiniMax	High quality I2V	Very competitive pricing

Open-Source Frontier

Model	Params	License	Key Innovation
CogVideoX-5B	5B	Apache 2.0	Expert transformer, 3D causal VAE
Open-Sora 1.2	1.1B	Apache 2.0	Any resolution/duration
HunyuanVideo	13B	Tencent	Dual-stream architecture
Wan2.1	14B	Apache 2.0	State-of-the-art I2V open source
LTX-Video	2B	Lightricks	Real-time inference capability
AnimateDiff V3	~1.5B	Apache 2.0	SD-compatible motion modules
SV3D	1B	Stability AI	3D object video orbit generation

6.2 Key Technical Innovations (2024–2025)

Flow Matching (Dominant Training Paradigm)

• Replaces DDPM noise scheduling
• Trains model to predict velocity (direction from noise to data)
• Optimal transport flow: straight-line paths in probability space
• Why better: More stable training, faster inference, better quality
• Used in: Sora, Stable Diffusion 3, CogVideoX

DiT Scaling Laws for Video

• Larger DiT = proportionally better quality
• Quality scales predictably with compute
• Sparse attention patterns (like Sora's spacetime patches) enable longer videos
• Window attention + global attention hybrid

3D Causal VAE

• Temporal causality in VAE encoder/decoder
• No information leakage from future frames during encoding
• Enables streaming inference
• CogVideoX, HunyuanVideo use this

World Models

• Genie 2 (DeepMind): Interactive world generation
• GameNGen: Playing games via neural simulation
• Video generation as physics simulation substrate
• I2V as the backbone for world model interfaces

Native Long Video Generation

• Context window extension for video transformers
• RoPE temporal dimension interpolation
• Sliding window inference for arbitrarily long videos
• Memory-efficient attention for 1000+ frame sequences

Real-Time Inference

• LTX-Video: Generation faster than playback speed
• Consistency distillation for video (4-step generation)
• Adversarial distillation (AnimateLCM)
• Caching of KV states across denoising steps (TeaCache, PAB)

6.3 Emerging Research Directions

Physically-Based Video Generation

• Integrating physics simulators as priors
• Fluid dynamics, rigid body physics in generation
• PhysGen, PhysDreamer research direction

4D Generation (Video + 3D)

• Generate consistent 3D across time
• Gaussian splatting + video generation
• Shape4D, 4D-fy research

Video Foundation Models

• Single model for generation + understanding + editing
• Unified video + image + text space
• Video-GPT style next-token prediction

Autonomous Camera Control

• Free-form text-described camera trajectories
• Learning from cinematography datasets
• Integration with real camera hardware

8.1 Service Architecture

Tier 1: API Service

Client → API Gateway (Kong/AWS API GW)
       → Auth Service (JWT validation)
       → Rate Limiter (Redis)
       → Job Queue (Celery)
       → GPU Worker Pool (auto-scaling)
       → Storage (S3 / GCS)
       → CDN (CloudFront)
       → Webhook / Polling for results

Tier 2: Web Application

Next.js Frontend
  ↓ REST API calls
FastAPI Backend
  ↓ Async job dispatch
Celery Workers (GPU instances)
  ↓ Results stored
PostgreSQL (metadata) + S3 (video files)
  ↓ CDN delivery
CloudFront → End users

8.2 Pricing Models

Model	Example	Pros	Cons
Per-second of video	$0.10/sec	Simple, fair	Unpredictable revenue
Credit bundles	100 credits/$9.99	Encourages bulk buy	Complex to manage
Subscription	$20/mo for 100 videos	Predictable revenue	Unused credits waste
Enterprise API	$500+/mo + usage	High value	Sales cycle

8.3 Technology Cost Estimation

Cost per video generation (2 seconds, 512×512, SVD):
  GPU time: ~15s on A10G = $0.005
  Storage: 2MB video = $0.0001
  Bandwidth: 2MB × 2 (in+out) = $0.0002
  Total COGS: ~$0.006 per video

Recommended price: $0.05–0.20/video (8–30× margin)

9.1 Foundational Papers (Must Read)

Diffusion Models

• DDPM: "Denoising Diffusion Probabilistic Models" — Ho et al., NeurIPS 2020
• DDIM: "Denoising Diffusion Implicit Models" — Song et al., ICLR 2021
• LDM: "High-Resolution Image Synthesis with Latent Diffusion Models" — Rombach et al., CVPR 2022
• DiT: "Scalable Diffusion Models with Transformers" — Peebles & Xie, ICCV 2023
• Flow Matching: "Flow Matching for Generative Modeling" — Lipman et al., ICLR 2023

Video Generation

• VDM: "Video Diffusion Models" — Ho et al., NeurIPS 2022
• SVD: "Stable Video Diffusion" — Blattmann et al., arXiv 2023
• CogVideoX: "CogVideoX: Text-to-Video Diffusion Models with an Expert Transformer" — Yang et al., 2024
• AnimateDiff: "AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning" — Guo et al., ICLR 2024
• Sora Technical Report: "Video generation models as world simulators" — OpenAI, 2024

Motion & Control

• RAFT: "RAFT: Recurrent All-Pairs Field Transforms for Optical Flow" — Teed & Deng, ECCV 2020
• ControlNet: "Adding Conditional Control to Text-to-Image Diffusion Models" — Zhang et al., ICCV 2023
• CameraCtrl: "CameraCtrl: Enabling Camera Controllability for Text-to-Video Generation" — He et al., 2024
• DragAnything: "DragAnything: Motion Control for Anything using Entity Representation" — Wu et al., 2024

9.2 Open Source Repositories

Core Models

• https://github.com/Stability-AI/generative-models (SVD)
• https://github.com/hpcaitech/Open-Sora (Open-Sora)
• https://github.com/THUDM/CogVideo (CogVideoX)
• https://github.com/guoyww/AnimateDiff (AnimateDiff)
• https://github.com/tencent/HunyuanVideo

Infrastructure

• https://github.com/huggingface/diffusers
• https://github.com/microsoft/DeepSpeed
• https://github.com/comfyanonymous/ComfyUI
• https://github.com/lllyasviel/ControlNet

Evaluation

• https://github.com/universome/fvd (FVD metric)
• https://github.com/richzhang/PerceptualSimilarity (LPIPS)

9.3 Datasets

Dataset	Size	Type	License
WebVid-10M	10M clips	Web videos + captions	Research
Panda-70M	70M clips	High quality	Research
InternVid	234M clips	Diverse	Research
UCF-101	13K clips	Action recognition	Public
Kinetics-400/600/700	400K clips	Actions	Research
DAVIS	90 sequences	Segmentation	Public
LAION-5B	5B images	Image-text pairs	CC-BY

9.4 Benchmarks

Benchmark	Task	Metric
UCF-FVD	Video generation	FVD ↓
MSR-VTT	Text-to-video	CLIP-Sim ↑
EvalCrafter	Multi-aspect evaluation	Composite
VBench	16 quality dimensions	VBench Score
DAVIS	Video object seg	J&F Score
Sintel	Optical flow	EPE ↓

9.5 Learning Resources

Courses

• Fast.ai Part 2: Diffusion models from scratch (highly recommended)
• Stanford CS231n: CNN for Visual Recognition
• Stanford CS25: Transformers United (video lectures free)
• MIT 6.S191: Introduction to Deep Learning

Books

• "Deep Learning" — Goodfellow, Bengio, Courville (free online)
• "Pattern Recognition and Machine Learning" — Bishop
• "Understanding Deep Learning" — Simon Prince (free online, 2023)
• "Probabilistic Machine Learning" — Kevin Murphy (free online)

Communities

• Hugging Face Discord — active diffusion model community
• Reddit r/StableDiffusion — practical tips and new releases
• Papers With Code — track latest SOTA
• Yannic Kilcher YouTube — paper explanations
• Andrej Karpathy YouTube — deep fundamentals