H.264

Egami.Video.H264 decodes H.264 (AVC), the codec of most MP4 and MOV files from phones, cameras and the web. It is a separate, opt-in package because H.264 is covered by patents (see Patents); Egami.Video itself contains no H.264 decoder and copies H.264 tracks through untouched when this package is not registered.

Profiles: Constrained Baseline, Main, High, High 10 and High 4:2:2, at 8 and 10 bits, 4:0:0, 4:2:0 and 4:2:2, and their Intra profiles. CAVLC and CABAC, B-frames with spatial and temporal direct prediction, weighted prediction, the 8x8 transform and scaling matrices, long-term references and several slices per frame.
Interlaced streams: field pictures (PAFF) and MBAFF frames, whose macroblock pairs are each two frame or two field macroblocks. A frame's two fields come out woven into one frame, as FFmpeg gives them; see Interlaced video.
Containers: MP4, MOV, Matroska and AVI through Egami.Video's readers, and raw Annex B .h264 files through H264Format.
Frames equal FFmpeg's decode sample for sample on every clip in the test corpus and on the interlaced and FRExt conformance bitstreams of ITU-T H.264.1.
Not decoded: 4:4:4 and lossless (High 4:4:4), and the Extended profile's data partitions, FMO, ASO and SP and SI slices. They give UnsupportedFormatException.

Registering

WithH264() (from H264MediaContextExtensions) adds the decoder, H264Codec, and the raw stream reader, H264Format, to a MediaContext. Nothing registers them on its own; set the context as the default with MediaContext.SetDefault to use it everywhere.

// Opt in to H.264 once: WithH264() adds the decoder and the raw .h264 reader to a context. MediaContext.SetDefault makes it
// the default for every call.
MediaContext context = MediaContext.Default.WithH264();
using VideoReader reader = VideoReader.Open("h264.mp4", context: context);
Console.WriteLine($"{reader.Track.CodecId} {reader.Track.Width}x{reader.Track.Height}");

Decoding

Frames come out in display order through VideoReader, like any other codec, as YuvVideoFrames with the stream's own planes (8-bit samples in bytes, 10-bit in ushorts) or converted to a pixel type. The colour comes from the stream's VUI, then the container, and otherwise BT.601 for 576 lines and fewer and BT.709 above; HDR10 mastering display and content light level SEI become the frame's Hdr, and HDR10+ user data SEI its Hdr10Plus.

// Frames come out in display order, B-frames included; a broken slice is concealed from the frame before.
using VideoReader reader = VideoReader.Open("h264.mp4", new VideoReaderOptions { Errors = DecodeErrorMode.Conceal }, context);
int count = 0;
foreach (VideoFrame<Rgb8> frame in reader.ReadFrames<Rgb8>())
{
    using (frame)
    {
        if (count++ == 0)
        {
            frame.Raster.Save("first.png");
        }
    }
}

Broken data follows VideoReaderOptions.Errors. A slice that cannot be read reports H264.CorruptFrame, and the macroblocks it leaves out are filled from the previous frame (Conceal, and Skip, the default) or the frame is finished and the error thrown (Throw). Decoding that starts in the middle of a stream waits for the next IDR or recovery point; a reference frame that is missing is replaced by the nearest one.

Slices decode in parallel when a frame has several, and deblocking runs on EgamiContext.MaxParallelism threads as a wave down the frame.

Interlaced video

An interlaced frame holds two fields, every other row each, filmed one after the other. The decoder gives the frame with both fields woven together and says how on the frame: FieldOrder is TopFieldFirst or BottomFieldFirst for an interlaced frame and Progressive otherwise, and RepeatFields counts the fields a frame is shown for beyond its two (3:2 pulldown and frame doubling). Both come from the picture timing SEI's pic_struct when the stream has it, else from how the frame was coded (two field pictures or an MBAFF frame) and the order of its fields, as ffprobe reports them. Deinterlacing is up to you; InterlaceDetector finds interlaced stretches in decoded video.

// An interlaced frame comes out with its two fields woven; FieldOrder says which was filmed first.
using VideoReader reader = VideoReader.Open("interlaced.h264", context: context);
foreach (VideoFrame frame in reader.ReadFrames())
{
    using (frame)
    {
        Console.WriteLine($"{frame.Timestamp}: {frame.FieldOrder}, {frame.RepeatFields} repeated fields");
        orders.Add(frame.FieldOrder);
    }
}

A field without its second field, as a cut stream can end, comes out with its rows doubled into the missing field and reports H264.UnpairedField.

Raw .h264 files

H264Format reads raw Annex B streams (.h264, .264 and .avc), as encoders and ffmpeg -f h264 write them. Such a stream has no timestamps, so frames are timed by the stream's VUI frame rate, or 25 frames a second, unless FrameRate is set. Seeking goes back to the IDR frame before the target.

// A raw .h264 stream has no timestamps: frames are timed by the stream's own frame rate, or 25 a second, unless you
// register the format with a rate of your own.
MediaContext context = MediaContext.Default.WithH264().WithContainer(new H264Format { FrameRate = new Rational(30, 1) });
using VideoReader reader = VideoReader.Open("camera.h264", context: context);
Console.WriteLine($"{reader.Track.Width}x{reader.Track.Height} at {reader.Track.FrameRate} fps");

// Remuxing keeps the stream as it is, here into MP4.
MediaFile.Remux("camera.h264", "camera.mp4", context: context);

Patents