Table of Contents

Planar YUV

A YuvRaster holds a picture as separate luma (Y) and chroma (U, V) planes, plus an optional alpha plane: the way video codecs, AVIF, lossy WebP and JPEG store it. Use it to take frames from a decoder or a camera, to scale video before converting it, and to read and write JPEG and AVIF without the RGB step. Everything else is done on a Raster after conversion.

Creating and wrapping

// A 1080p frame as video stores it: full-size luma, chroma at half width and height, BT.709 at limited range.
using YuvRaster<byte> frame = YuvRaster.Create(1920, 1080, ChromaFormat.Yuv420, YuvColor.Bt709Limited);

// 10-bit HDR planes hold samples 0 to 1023 in ushorts; an alpha plane is optional.
using YuvRaster<ushort> hdr = YuvRaster.Create(3840, 2160, ChromaFormat.Yuv420, 10, YuvColor.Bt2020Limited, alpha: true);
Console.WriteLine($"Chroma planes are {frame.U!.Width}×{frame.U.Height}");

Each plane is a YuvPlane<TSample>. YuvRaster<byte> holds 8-bit samples; YuvRaster<ushort> holds 10-, 12- or 16-bit samples, stored low-bit aligned (a 10-bit sample is 0 to 1023), as libvpx, dav1d and libavif hand them out. Chroma planes are half the luma width (4:2:0 and 4:2:2) and half the height (4:2:0), rounded up; 4:0:0 has no chroma at all. 4:1:1 (NTSC DV) has chroma at a quarter of the width, and 4:1:0 at a quarter of the width and height, also rounded up. The planes come from the context's memory source in one block, so pooling, SafetyLimits and the usage counters cover them.

using YuvRaster<byte> yuv = YuvRaster.Create(640, 480, ChromaFormat.Yuv420, YuvColor.Jpeg);

// Planes are read and written by row, like a raster's pixels.
for (int y = 0; y < yuv.Y.Height; y++)
{
    yuv.Y.Row(y).Fill((byte)(y / 2));
}

yuv.U!.Row(0)[0] = 128;

// An 8-bit plane can be viewed as Gray8 pixels, without copying.
PixelPlane<Gray8> luma = yuv.Y.AsGray8();

Frames a decoder or the operating system already holds are wrapped without a copy:

// A decoder's 10-bit frame, rows padded to 64 samples: wrapped without a copy. Disposing the raster disposes the owner.
int width = 50, height = 30, stride = 64, chromaStride = 32;
ushort[] y = new ushort[stride * height], u = new ushort[chromaStride * 15], v = new ushort[chromaStride * 15];
using YuvRaster<ushort> wrapped = YuvRaster.Wrap<ushort>(
    y, stride, u, chromaStride, v, chromaStride, width, height, ChromaFormat.Yuv420, bitDepth: 10, YuvColor.Bt2020Limited, owner: null);

4:1:1 and 4:1:0 planes wrap, crop, resize and convert like the others:

// An NTSC DV frame: 4:1:1, chroma a quarter of the luma width at full height, on the left column of each four.
byte[] y = new byte[720 * 480], u = new byte[180 * 480], v = new byte[180 * 480];
using YuvRaster<byte> dv = YuvRaster.Wrap<byte>(y, 0, u, 0, v, 0, 720, 480, ChromaFormat.Yuv411, bitDepth: 8, YuvColor.Bt601Limited);
using Raster<Rgb8> rgb = dv.ToRaster<Rgb8>();

What the samples mean

// How samples turn into RGB is always stated: a preset, or the codes a video container carries (ITU-T H.273).
YuvColor jpeg = YuvColor.Jpeg;                                         // BT.601, full range, chroma centered
YuvColor custom = new(YuvMatrix.Bt709, YuvRange.Full, ChromaSiting.Left);
if (YuvColor.TryFromH273(matrixCoefficients: 9, fullRange: false, ChromaSiting.TopLeft, out YuvColor fromStream))
{
    Console.WriteLine($"{fromStream.Matrix}, {fromStream.Range}; back to code {fromStream.ToH273()}");
}

// Or straight from the code points an AVIF or PNG cICP box carries.
Egami.Colors.ColorCodePoints hdr10 = new(Primaries: 9, TransferCharacteristics: 16, MatrixCoefficients: 9, FullRange: false);
hdr10.TryGetYuvColor(ChromaSiting.TopLeft, out YuvColor fromBox);
Part Values Notes
YuvMatrix Bt601, Bt709, Bt2020Ncl, Identity Kr and Kb from ITU-T H.273. Identity stores G, B, R as Y, U, V (AV1 lossless)
YuvRange Full, Limited Limited is luma 16 to 235 and chroma 16 to 240 at 8 bits, scaled for more
ChromaSiting Center, Left, TopLeft JPEG and WebP center chroma; BT.601 and BT.709 video put it on the left column; BT.2020 on the top-left sample

There is no default guess: every load, wrap and conversion states a YuvColor. YuvColor.TryFromH273 reads the matrix_coefficients code and full-range flag that AV1, VP9, HEVC and CICP boxes carry, and refuses codes Egami does not know; ColorCodePoints.TryGetYuvColor does the same from code points already read.

Converting

using Raster photo = Raster.Load("duck.png");

// RGB to planes and back. RGB comes out in the picture's own transfer (BT.709 video gives BT.709-encoded RGB).
using YuvRaster<byte> yuv = YuvRaster.FromRaster(photo, ChromaFormat.Yuv420, YuvColor.Bt709Limited);
using Raster rgba = yuv.ToRaster();                        // Rgba8 for 8-bit planes, Rgba16 for more
using Raster<Rgb8> rgb = yuv.ToRaster<Rgb8>();             // or any pixel type

FromRaster drops alpha unless asked: pass alpha: true for an alpha plane, opaque when the source has none.

Conversion is within one level of an exact double-precision implementation of H.273, at every depth, and identical for at least 99% of samples. RGB comes out in the picture's own transfer function: BT.709 video gives BT.709-encoded RGB, PQ gives PQ-encoded RGB. Linearizing or tone mapping is a separate step (see color management).

Chroma is filtered by its siting both ways: bilinear upsampling (the 3:1 triangle filter for centered chroma, the average between co-sited samples otherwise), and averaging that matches the siting on the way back. 4:1:1 and 4:1:0 use the same rules over four luma samples: bilinear weights in eighths, and a box of four (centered) or a 1-2-3-4-3-2-1 triangle (co-sited) on the way back. The fast options repeat or pick samples instead:

using Raster photo = Raster.Load("duck.png");

// The defaults filter chroma by its siting both ways; Nearest and Fast trade edge quality for speed.
YuvConvertOptions fast = new() { Upsampling = ChromaUpsampling.Nearest, Downsampling = ChromaDownsampling.Fast };
using YuvRaster<byte> yuv = YuvRaster.FromRaster(photo, ChromaFormat.Yuv420, YuvColor.Bt601Limited, options: fast);
using Raster<Rgba8> back = yuv.ToRaster<Rgba8>(fast);

A player converts every frame into the same raster; after the first frame nothing is allocated:

// A player converts every frame into the same raster: after the first frame, nothing is allocated.
using YuvRaster<byte> frame = YuvRaster.Create(1280, 720, ChromaFormat.Yuv420, YuvColor.Bt709Limited);
using Raster<Bgra8> screen = Raster.Create<Bgra8>(1280, 720);
for (int i = 0; i < 3; i++)
{
    // ... the decoder writes frame's planes ...
    frame.CopyTo(screen);
}

// And the other way, for an encoder: CopyFrom fills existing planes.
frame.CopyFrom(screen);

Decoder frames in NV12 or P010 copy into the planes first; see Interleaved layouts.

Interleaved layouts

Media Foundation, MediaCodec, GPU decoders and webcams hand out NV12, NV21, P010, P016, YUYV or UYVY. One SIMD pass copies them to and from the planes:

// An NV12 frame from Media Foundation or MediaCodec: Y rows, then interleaved U, V rows, both at the same stride.
int width = 640, height = 360, stride = 704;
byte[] nv12 = new byte[stride * height * 3 / 2];
using YuvRaster<byte> frame = YuvRaster.Create(width, height, ChromaFormat.Yuv420, YuvColor.Bt709Limited);
frame.CopyFromPacked(YuvPackedFormat.Nv12, nv12, stride);

// A webcam's YUYV, 4:2:2 in one plane, the other way.
using YuvRaster<byte> webcam = YuvRaster.Create(width, height, ChromaFormat.Yuv422, YuvColor.Bt601Limited);
byte[] yuyv = new byte[width * 2 * height];
webcam.CopyToPacked(YuvPackedFormat.Yuyv, yuyv);

JPEG without the RGB step

// A JPEG's own planes, as decoded: no chroma upsampling and no color conversion.
YuvRasterInfo info = YuvRaster.Probe("duck.jpg");
Console.WriteLine($"{info.Info.Width}×{info.Info.Height}, planes: {info.ChromaFormat}");
using YuvRaster planes = YuvRaster.Load("duck.jpg");

// Saving JPEG-colored 8-bit planes skips color conversion and downsampling; other planes are converted first.
planes.Save("copy.jpg", new JpegWriterOptions { Quality = 90 });

YuvRaster.Load gives a YCbCr JPEG's planes exactly as libjpeg-turbo's raw output does: 4:2:0, 4:2:2 or 4:4:4 at the file's own sampling (other samplings are replicated to 4:4:4), gray as 4:0:0, also at reduced scale. CMYK, YCCK and RGB JPEGs, and formats without planes, throw NotSupportedException: load those with Raster.Load. Saving 8-bit BT.601 full-range planes as JPEG encodes them straight, with the same bytes as libjpeg-turbo's encoder from planes; 4:1:1 is written as 4:2:2 and 4:1:0 as 4:2:0. Saving planes as AVIF encodes them straight too, at 8, 10 or 12 bits with any matrix, range and siting (see the AVIF guide); AV1 has no 4:1:1 or 4:1:0, so those are saved from RGB.

Loading planes skips chroma upsampling and color conversion, so it is faster than Raster.Load on one thread. On several threads both are limited by the same Huffman decoding and take about the same time.

Converting loaded planes with ToRaster may differ by one level, in a small share of pixels, from Raster.Load, which keeps the JPEG decoder's own libjpeg-turbo color code.

Crop and resize

using YuvRaster<byte> frame = YuvRaster.Create(1920, 1080, ChromaFormat.Yuv420, YuvColor.Bt709Limited);

// Scale the planes before converting, as video does; crops start on a chroma sample (even x and y at 4:2:0).
using YuvRaster<byte> small = frame.Resize(1280, 720, ResampleFilter.Bicubic);
using YuvRaster<byte> center = small.Crop(new Rect(320, 180, 640, 360));

// A player scales every frame into the same raster.
using YuvRaster<byte> screen = YuvRaster.Create(960, 540, ChromaFormat.Yuv420, YuvColor.Bt709Limited);
frame.ResizeTo(screen);

// Everything else (rotate, color adjustments, drawing) runs on a Raster after conversion.
using Raster<Rgba8> rgb = center.ToRaster<Rgba8>();
rgb.Apply(ops => ops.Rotate(90));

Resize runs each plane through the same filters as RasterOps.Resize and clamps to the bit depth; the chroma format, siting and color stay. For now it is somewhat slower than resizing the same picture as Rgba8, whose resize has a faster horizontal pass. ResizeTo does the same into a raster you already have, so a player scales every frame into one raster. Crop must start on a chroma sample, so chroma is copied exactly.

Edit this page