Video Processing
From raw footage to polished final output.
What is Video Processing?
Video processing encompasses all operations performed on video data: capturing, editing, encoding, transcoding, filtering, and exporting. It transforms raw footage into finished video through a series of computational steps.
Modern video processing uses both CPU and GPU acceleration. GPUs excel at parallel operations like encoding, filtering, and effects. CPUs handle logic, sequencing, and complex branching operations.
The Video Pipeline
Typical Workflow
- Capture — Record from camera, screen, or import existing footage
- Ingest — Import into editing software, organize media
- Edit — Cut, arrange, add transitions, B-roll, music
- Effects — Color grading, filters, text, overlays
- Export/Encode — Compress to final format with target codec
- Distribution — Upload to platform, optimize for web
Video Codecs
A codec (coder/decoder) compresses and decompresses video data. Different codecs balance compression efficiency, quality, and compatibility.
| Codec | Type | Compression | Quality | Use Case |
|---|---|---|---|---|
| H.264/AVC | Lossy | Good | Good | Universal compatibility, web video |
| H.265/HEVC | Lossy | Better | Better | 4K video, smaller files |
| VP9 | Lossy | Better | Good | YouTube, WebM |
| AV1 | Lossy | Best | Best | Next-gen streaming, royalty-free |
| ProRes | Visually lossless | Low | Excellent | Editing intermediate, color grading |
| DNxHD | Visually lossless | Low | Excellent | Editing intermediate, broadcast |
Frame Rates
Frame rate (fps) determines both smoothness and file size. Higher frame rates capture more motion but require more data.
| Frame Rate | Standard | Use Case |
|---|---|---|
| 24 fps | Film | Cinematic, narrative |
| 25 fps | PAL | European broadcast |
| 30 fps | NTSC | Standard video, web |
| 60 fps | High frame rate | Smooth motion, gaming, sports |
| 120+ fps | Slow motion | Slow-mo, VFX, sports analysis |
Resolution & Aspect Ratios
Common Resolutions
- 720p (1280×720) — HD, web video, small file size
- 1080p (1920×1080) — Full HD, standard for most content
- 1440p (2560×1440) — 2K, high-quality web and streaming
- 4K (3840×2160) — Ultra HD, professional production
- 8K (7680×4320) — Emerging standard, massive file sizes
Aspect Ratios
- 16:9 — Widescreen, YouTube, modern displays
- 4:3 — Traditional TV, some web content
- 1:1 — Square, Instagram, TikTok
- 9:16 — Vertical, TikTok, Instagram Stories, YouTube Shorts
- 21:9 — Ultrawide, cinematic
Video Processing Operations
Transcoding
Converting from one codec/format to another. H.264 → H.265 for smaller files, or ProRes → H.264 for web delivery.
Trimming & Cutting
Removing sections without re-encoding (fast) or re-encoding with transitions. Essential for editing.
Scaling
Resizing video: 4K → 1080p for web, or upscaling with AI enhancement. Quality depends on algorithm.
Frame Rate Conversion
Changing fps: 24fps → 60fps with interpolation, or 60fps → 24fps with frame dropping. Can introduce motion artifacts.
Color Grading
Adjusting color, contrast, saturation for mood and consistency. LUTs (Look-Up Tables) enable rapid color style application.
Stabilization
Reducing camera shake using motion analysis. Software analyzes frame-to-frame motion and applies counter-transformation.
Tools for Video Processing
- FFmpeg — Command-line Swiss Army knife. Transcoding, filtering, streaming. Free, open-source, ubiquitous.
- Adobe Premiere Pro — Industry-standard NLE (non-linear editor). Professional workflows, extensive effects.
- DaVinci Resolve — Professional editing + industry-leading color grading. Free version is extremely capable.
- HandBrake — Open-source transcoder. Simple UI, great for converting to web-friendly formats.
- Shutter Encoder — FFmpeg GUI with presets. Easier than raw FFmpeg for common tasks.
- DeltaSketch — Capture from camera/video, apply real-time effects, export as video via FFmpeg.
Optimization for Web
Web video must balance quality and file size. Key strategies:
- Use H.264 or H.265 — Best browser support and compression
- Target 1080p or lower — 4K is overkill for most web content
- Two-pass encoding — Better quality at same bitrate
- Variable bitrate (VBR) — Allocates more bits to complex scenes
- CRF (Constant Rate Factor) — Quality-based encoding (18–28 is typical)
- Consider WebM/VP9 — Better compression than H.264, supported by most browsers
- Use a CDN — Deliver video from servers close to viewers
- Adaptive bitrate streaming — HLS or DASH for varying connection speeds
Common Video Processing Tasks
Extract Frames from Video
FFmpeg command: ffmpeg -i input.mp4 -vf fps=1 frames/frame_%04d.png — extracts 1 frame per second.
Convert Video to GIF
FFmpeg + palettegen: ffmpeg -i input.mp4 -vf "fps=15,scale=480:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" output.gif
Compress Video
FFmpeg with CRF: ffmpeg -i input.mp4 -c:v libx264 -crf 23 -c:a aac -b:a 128k output.mp4
Create Time-Lapse from Video
Extract frames at intervals, then reassemble at higher frame rate. DeltaSketch automates this workflow.