Encoding video

Egami.Video encodes VP8 and VP9 into WebM, Matroska and IVF (VP9 into MP4 too), with alpha in WebM and Matroska, and AV1 into WebM, Matroska, MP4, IVF, raw OBU and Annex B files. VP8's frame coding is Egami's Vp8StreamEncoder, VP9's Egami.Video's own and AV1's Egami's Av1StreamEncoder; for all three, Egami.Video decides how each frame is coded: where keyframes go, which frames become references, and how many bits each frame gets. Motion JPEG and uncompressed video encode too, without these settings: every frame is a keyframe.

Writing VP8

Set VideoWriterOptions.Codec to VideoCodecs.Vp8. Quality runs from 0 to 100 (default 60, the scale of Egami's AVIF writer) and Effort from 0 (fastest) to 10 (smallest files, default 4). Frames given as 8-bit 4:2:0 planes are coded as they are; anything else is converted to BT.601 limited range first. Frames with alpha get a second VP8 stream for the alpha plane, as WebM carries it.

// VP8 in WebM at constant quality: Quality 0 to 100 (default 60) and Effort 0 to 10 (default 4).
VideoWriterOptions options = new() { Codec = VideoCodecs.Vp8, FrameRate = new Rational(30, 1), Quality = 70, Effort = 4 };
using (VideoWriter writer = VideoWriter.Create("duck.webm", options))
{
    for (int i = 0; i < 30; i++)
    {
        using Raster<Rgb8> frame = duck.Derive(ops => ops.Crop(new Rect(i * 4, i * 2, 320, 240)));
        writer.WriteFrame(frame);
    }
}

// Expert settings: no hidden alternate reference frames, and quantizer limits.
VideoWriterOptions tuned = options with { Vp8 = new Vp8EncoderOptions { AlternateReference = false, MinQuantizer = 10, MaxQuantizer = 90 } };

Vp8EncoderOptions holds the expert settings: quantizer limits, alternate reference frames on or off, golden frame spacing, the temporal filter's frames and strength, token partitions, error resilience and alpha.

Writing VP9

Set VideoWriterOptions.Codec to VideoCodecs.Vp9. Quality and Effort work as for VP8 (default 60 and 4); Quantizer runs from 0 to 255, and 0 is lossless. The encoder searches each 64x64 superblock's partition down to 4x4, its intra modes, motion vectors to 1/8 pixel with three interpolation filters, compound prediction from two references and its transform sizes by rate and distortion, quantizes with a trellis, and picks the loop filter level per frame. Hidden alternate references go out in a superframe with the next shown frame, so every packet shows one frame.

// VP9 in WebM at constant quality, on the same Quality and Effort scales as VP8.
VideoWriterOptions options = new() { Codec = VideoCodecs.Vp9, FrameRate = new Rational(30, 1), Quality = 70, Effort = 2 };
using (VideoWriter writer = VideoWriter.Create("duck-vp9.webm", options))
{
    for (int i = 0; i < 10; i++)
    {
        using Raster<Rgb8> frame = duck.Derive(ops => ops.Crop(new Rect(i * 4, i * 2, 320, 240)));
        writer.WriteFrame(frame);
    }
}

// Expert settings: two tile columns, adaptive quantization, quantizer limits on VP9's 0 to 255 scale.
VideoWriterOptions tuned = options with
{
    Vp9 = new Vp9EncoderOptions { TileColumns = 2, AdaptiveQuantization = true, MinQuantizer = 20, MaxQuantizer = 200 },
};

Vp9EncoderOptions holds the expert settings: quantizer limits, alternate reference frames on or off, golden frame spacing, the temporal filter's frames and strength, tile columns, adaptive quantization by block detail (libvpx's aq-mode 1), error resilience, the bit depth and alpha. Frames are split into up to four tile columns by default (each at least 256 pixels wide), which the encoder searches on separate threads, superblock rows running as a wavefront within them.

Bit depths, chroma and HDR

8-bit 4:2:0 planes are coded as they are (profile 0); 4:2:2 and 4:4:4 planes as profile 1; 10 and 12-bit planes as profile 2 (4:2:0) or 3. Vp9EncoderOptions.BitDepth codes another depth, and VideoEncoderOptions.ChromaFormat sets the chroma format for RGB frames. The track's color comes from the frames: their code points (primaries, transfer, matrix, range), and their static HDR metadata (mastering display, content light levels), which WebM and Matroska keep in the Colour element and MP4 in its colr, mdcv and clli boxes. Frames with alpha get a second, 8-bit 4:2:0 VP9 stream for the alpha plane. Alternate references of other formats than 8-bit 4:2:0 are their frame unfiltered.

// 10-bit planes code profile 2 (4:2:0) or 3 (4:2:2, 4:4:4); the frames' code points and HDR metadata go into the
// container's Colour (WebM, Matroska) or colr, mdcv and clli boxes (MP4).
ColorCodePoints pq = new(9, 16, 9, false);
HdrMetadata hdr = new() { MasteringMaxLuminance = 1000, MasteringMinLuminance = 0.005, MaxContentLightLevel = 1000, MaxFrameAverageLightLevel = 400 };
VideoWriterOptions options = new() { Codec = VideoCodecs.Vp9, Quality = 80, Effort = 1 };
using (VideoWriter writer = VideoWriter.Create("hdr.mp4", options))
{
    for (int i = 0; i < 5; i++)
    {
        YuvRaster<ushort> planes = YuvRaster.Create(64, 48, ChromaFormat.Yuv420, 10, new YuvColor(YuvMatrix.Bt2020Ncl, YuvRange.Limited, ChromaSiting.Center));
        using YuvVideoFrame frame = new(planes, new MediaTime(i, new Rational(1, 30)), color: pq) { Hdr = hdr };
        writer.WriteFrame(frame);
    }
}

Writing AV1

Set VideoWriterOptions.Codec to VideoCodecs.Av1. Quality and Effort work as for VP8 and VP9 (default 60 and 4), and equal qualities give the same quantizer step as VP9's; Quantizer is AV1's base_q_idx, 0 to 255, and 0 is lossless. Each packet is a temporal unit of OBUs, as MP4 and Matroska store them, and the track's av1C comes from the sequence header.

The frame coding is Av1StreamEncoder's: partitions, intra and inter modes from up to seven references with compound prediction, transform types and sizes chosen by rate and distortion, a trellis, delta quantizers per superblock, deblocking levels searched over every level at once, CDEF strengths and, on frames others predict from, Wiener loop restoration; from effort 5, 6 and 7 on it adds compound masks, overlapped and warped motion, dual filters and global motion. Egami.Video plans the references: each group of 16 frames is a pyramid. Its last frame, blended with its neighbours along their motion, is coded first as a hidden alternate reference, then a hidden frame halfway to it, halfway again and so on down to pairs, five layers in all; the frames left between them predict from both sides and nothing predicts from them. No hidden frame costs a frame of its own when its time comes: AV1 shows it again with show_existing_frame, a packet of a few bytes. Before a group is coded, it and the next run through Av1StreamEncoder.Analyze, which finds how much of each frame the frames after it reuse: the more, the finer the alternate reference's and the keyframe's quantizers, and the finer the superblocks that matter most. Deeper layers get coarser quantizers and spend fewer bits on what only they show. The temporal filter, on keyframes and alternate references, blends as much as each picture's noise and the frame's quantizer call for.

// AV1 in MP4 at constant quality, on the same Quality and Effort scales as VP8 and VP9. Each group's hidden alternate
// reference, and a hidden frame halfway to it, are shown later with show_existing_frame.
VideoWriterOptions options = new() { Codec = VideoCodecs.Av1, FrameRate = new Rational(30, 1), Quality = 70, Effort = 1 };
using (VideoWriter writer = VideoWriter.Create("duck-av1.mp4", options))
{
    for (int i = 0; i < 10; i++)
    {
        using Raster<Rgb8> frame = duck.Derive(ops => ops.Crop(new Rect(i * 4, i * 2, 320, 240)));
        writer.WriteFrame(frame);
    }
}

// Expert settings: no pyramid, no adaptive quantization, quantizer limits on AV1's 0 to 255 base_q_idx scale.
VideoWriterOptions tuned = options with
{
    Av1 = new Av1EncoderOptions { Pyramid = false, AdaptiveQuantization = false, MinQuantizer = 20, MaxQuantizer = 200 },
};

Av1EncoderOptions holds the expert settings: quantizer limits, alternate references and the pyramid on or off, golden frame spacing, the temporal filter's frames and strength (adaptive by default), adaptive quantization (from the temporal model on the base frames, by block detail on the others), screen content tools, error resilience and the bit depth. AV1 codes no alpha in video files (Matroska and MP4 have no standard place for it); animated AVIF, through Egami's AvifWriter, does.

Profiles and HDR

8 and 10-bit 4:2:0 planes make Main profile streams, 4:4:4 High and 12-bit or 4:2:2 Professional, all coded as they are; Av1EncoderOptions.BitDepth codes another depth and VideoEncoderOptions.ChromaFormat sets the chroma format for RGB frames, which are converted to BT.601 limited range. The frames' code points (primaries, transfer, matrix, range) and the chroma siting of 4:2:0 planes go into the sequence header and the container. Their static HDR metadata goes into content light level and mastering display metadata OBUs on every keyframe, and into the container as for VP9; a frame's HDR10+ metadata goes into its own temporal unit as an ITU-T T.35 metadata OBU. The level is the smallest whose picture size and sample rate hold the stream.

// 10-bit 4:2:0 planes code the Main profile (4:4:4 High; 12 bits or 4:2:2 Professional). The frames' code points go into
// the sequence header and the container, their HDR metadata into metadata OBUs on every keyframe and into the container
// (Matroska's Colour, MP4's mdcv and clli).
ColorCodePoints pq = new(9, 16, 9, false);
HdrMetadata hdr = new() { MasteringMaxLuminance = 1000, MasteringMinLuminance = 0.005, MaxContentLightLevel = 1000, MaxFrameAverageLightLevel = 400 };
VideoWriterOptions options = new() { Codec = VideoCodecs.Av1, Quality = 80, Effort = 1 };
using (VideoWriter writer = VideoWriter.Create("hdr-av1.mkv", options))
{
    for (int i = 0; i < 5; i++)
    {
        YuvRaster<ushort> planes = YuvRaster.Create(64, 48, ChromaFormat.Yuv420, 10, new YuvColor(YuvMatrix.Bt2020Ncl, YuvRange.Limited, ChromaSiting.TopLeft));
        using YuvVideoFrame frame = new(planes, new MediaTime(i, new Rational(1, 30)), color: pq) { Hdr = hdr };
        writer.WriteFrame(frame);
    }
}

Writing FFV1

VideoCodecs.Ffv1 writes FFV1 version 3 (RFC 9043), the lossless codec archives use, in Matroska or AVI: a decode gives back every sample. Frames given as planes keep their layout (4:2:0, 4:2:2, 4:4:4, 4:1:1, 4:1:0 or gray, with or without alpha, 8 to 16 bits); RGB rasters, and 4:4:4 planes tagged YuvMatrix.Identity, are coded as RGB. The first frame sets the stream's layout; a later frame of another layout is converted to it, and each frame's FieldOrder is coded with it.

The defaults are the archival profile: every frame a keyframe, slice CRCs on (so Ffv1Codec.Verify can check the file), the range coder with a custom state table, and a slice grid picked from the frame size (16 slices at 1080p). KeyframeInterval spaces keyframes further apart, Effort 7 and up picks the large context model, and Ffv1EncoderOptions sets the slices, the coder (Ffv1Coder.GolombRice is faster and larger), the context model and BitDepth for planes that hold a 9, 11, 13, 14 or 15-bit source shifted up. Quality does not apply, and setting it reports Ffv1.QualityIgnored. Slices are coded in parallel, with the same file at any thread count. Files come within 1 % of FFmpeg's size at the same settings, and FFmpeg decodes them to the same samples.

// FFV1 keeps every sample: RGB frames are coded as RGB and planes keep their layout and depth. The defaults are the
// archival profile (version 3, every frame a keyframe, slice CRCs, the range coder with a custom table).
VideoWriterOptions options = new() { Codec = VideoCodecs.Ffv1, FrameRate = new Rational(25, 1) };
using (VideoWriter writer = VideoWriter.Create("master.mkv", options))
{
    for (int i = 0; i < 5; i++)
    {
        using Raster<Rgb8> frame = duck.Derive(ops => ops.Crop(new Rect(i * 4, i * 2, 320, 240)));
        writer.WriteFrame(frame);
    }
}

// Expert settings: more slices for more threads, Golomb-Rice for speed, a keyframe every 10 frames for smaller files.
VideoWriterOptions fast = options with
{
    KeyframeInterval = 10,
    Ffv1 = new Ffv1EncoderOptions { Slices = 16, Coder = Ffv1Coder.GolombRice, ContextModel = Ffv1ContextModel.Small },
};

// Check the archive without decoding it.
bool intact = Ffv1Codec.Verify("master.mkv").IsIntact;

How bits are spent

RateControl picks one of four modes:

Mode	What it does	For
`Quality` (default)	Every frame aims at `Quality`; files are as large as the content needs.	Files
`Bitrate`	An average `Bitrate`, spread over frames by how hard they are; capped by `MaxBitrate`.	Files of a known size, streaming
`ConstantBitrate`	`Bitrate` with a decoder buffer of `BufferSize` bits (one second by default) that never runs dry.	Live streams, WebRTC
`Quantizer`	The same quantizer for every frame (`Quantizer`, or the one `Quality` maps to).	Tests, comparisons

// An average of 1 Mbit/s, the bits spread by how hard each frame is; ConstantBitrate keeps a decoder buffer from running dry.
VideoWriterOptions options = new()
{
    Codec = VideoCodecs.Vp8,
    RateControl = RateControlMode.Bitrate,
    Bitrate = 1_000_000,
};
using (VideoWriter writer = VideoWriter.Create("duck.mkv", options))
{
    for (int i = 0; i < 30; i++)
    {
        writer.WriteFrame(duck);
    }
}

Two passes

A two-pass encode first reads the whole clip to learn how hard each frame is, then spreads the bitrate over all of it. VideoWriter.AnalyzeFirstPass runs the first pass over frames or a file and returns FirstPassStats, which can be saved and loaded; pass it as FirstPassStats with RateControlMode.Bitrate.

// Pass one reads the clip and learns how hard each frame is; save it to run the passes apart.
FirstPassStats stats = VideoWriter.AnalyzeFirstPass("clip.ivf");
stats.Save("clip.stats");

// Pass two spreads 500 kbit/s over the whole clip by what pass one learned.
VideoWriterOptions options = new()
{
    Codec = VideoCodecs.Vp8,
    RateControl = RateControlMode.Bitrate,
    Bitrate = 500_000,
    FirstPassStats = FirstPassStats.Load("clip.stats"),
};
using (VideoReader reader = VideoReader.Open("clip.ivf"))
using (VideoWriter writer = VideoWriter.Create("clip.webm", options))
{
    foreach (YuvVideoFrame frame in reader.ReadFrames())
    {
        using (frame)
        {
            writer.WriteFrame(frame);
        }
    }
}

Keyframes, groups and latency

The encoder looks 25 frames ahead (LatencyMode.Quality, the default). It analyzes each frame on a half-size copy as it arrives and starts a keyframe at a scene cut, at least every 10 seconds (KeyframeInterval sets another spacing, in frames), and wherever EncodeFrameOptions.ForceKeyframe asks. Between keyframes it codes groups of 4 to 16 frames: the group's last frame is blended with its neighbours along their motion into a hidden alternate reference, coded first at a finer step, so the frames before it can predict from it. That happens only where the group's first frame still predicts its last well (still content); with motion, zooms or fades the group just ends in a golden frame. A VP8 hidden frame's packet carries the timestamp of the shown frame after it, and gives no frame when decoded; VP9 and AV1 put their hidden frames in the next shown frame's packet. Turn SceneCutKeyframes off for keyframes at a fixed spacing.

LatencyMode.Realtime looks nowhere ahead: each packet comes out as soon as its frame goes in, there are no hidden frames, frames keep no probability updates for the next (so a lost packet spoils one frame), and 720p and larger VP8 frames get two or four token partitions so decoders can use more threads.

// Live: no lookahead, so each packet comes out as soon as its frame goes in, at a constant bitrate.
VideoEncoderOptions options = new()
{
    Width = 320,
    Height = 240,
    LatencyMode = LatencyMode.Realtime,
    RateControl = RateControlMode.ConstantBitrate,
    Bitrate = 300_000,
};
using IVideoEncoder encoder = VideoCodecs.Vp8.CreateEncoder(options, MediaContext.Default);
using IMuxer muxer = MediaFile.CreateMuxer(stream, "webm", [encoder.Track], leaveOpen: true);
for (int i = 0; i < 60; i++)
{
    using VideoFrame<Rgb8> frame = new(Raster.Create(320, 240, new Rgb8((byte)i, 90, 160)), new MediaTime(i, new Rational(1, 30)));

    // A viewer joined: start a keyframe so they can decode from here.
    encoder.Send(frame, new EncodeFrameOptions { ForceKeyframe = i == 30 });
    while (encoder.TryReceive(out EncodedPacket? packet))
    {
        using (packet)
        {
            muxer.WritePacket(packet);
        }
    }
}

encoder.Flush();
muxer.Finish();

Threads and repeatability

The output depends only on the frames and the options, never on the number of threads or the machine, so a test can compare hashes. EgamiContext.MaxParallelism on the MediaContext sets how many threads the encoder uses. VP9's tile columns depend on the frame width and Vp9EncoderOptions.TileColumns, never on the threads; AV1's are one per 4096 samples of width, so frames up to 4K are one tile, and its threads help little within a frame.

Table of Contents