Table of Contents

Video editing

Egami.Video's editing library, in Egami.Video.Editing, joins clips with transitions, draws clips over clips, keys out green screens, lays clips side by side and runs Egami operations on the result, encoding it all again in one pass. An edit is a description: VideoEdit holds an EditCanvas, a main EditTrack of Clips with a Transition between each two, Layers drawn over it and EditEffects. Nothing is decoded until it renders, and then only the frames in use are held. The egami-video commands join, fade, overlay, pip, stack and edit build the same edits (see the command line).

Joining clips

Clips play one after another; each TrackItem on the track is a clip and the transition into it. They may differ in size, frame rate, pixel format and sound: the canvas sets the output's size, frame rate and audio format (from the first clip when there is no canvas), each clip is fitted with ClipFit (Contain with bars by default, Cover, Stretch or None) over the canvas Background, each output frame shows the source frame nearest its time, and the sound is resampled and remixed; a clip without sound adds silence. Clip.Trim cuts a clip on the exact frame and sample, and FadeIn and FadeOut fade the picture from and to a colour and the sound from and to silence. Render writes the result with a VideoWriter, whose settings EditRenderOptions.Writer carries; the sound goes into Opus unless another audio codec is set.

// Three clips of different sizes and rates, fitted to one 320x180 canvas at 25 fps, with fades at both ends.
VideoEdit edit = new(new EditCanvas(320, 180, new Rational(25, 1)) { Fit = ClipFit.Contain });
edit.Track
    .Add(Clip.Open("intro.avi").FadeIn(MediaTime.FromSeconds(0.5)))
    .Add(Clip.Open("talk.avi").Trim(MediaTime.FromSeconds(0.5), MediaTime.FromSeconds(1.5)))
    .Add(Clip.Open("outro.avi").FadeOut(MediaTime.FromSeconds(0.5)));
EditPlan plan = edit.Render("joined.mkv", new EditRenderOptions { Writer = new VideoWriterOptions { Codec = MotionJpegCodec.Instance, Quality = 80 } });
Console.WriteLine($"{plan.FrameCount} frames, {plan.Duration.Seconds} s");

Plan lays the timeline out in whole frames before anything is decoded and returns an EditPlan: every PlannedClip and PlannedLayer with its frames, the length, and warnings about what it fixed up. An edit that cannot render, such as one with a trim past a clip's end or a transition longer than a clip, throws with every reason. ToJson and VideoEdit.FromJson save and load an edit as the JSON file the command line's edit runs.

// Lay out the timeline before rendering, and save the edit as the JSON file 'egami-video edit' runs.
VideoEdit edit = new(new EditCanvas(320, 180, new Rational(25, 1)));
edit.Track.Add(Clip.Open("intro.avi")).Add(Clip.Open("talk.avi"), Transition.Fade(MediaTime.FromSeconds(1)));
EditPlan plan = edit.Plan();
foreach (PlannedClip clip in plan.Clips)
{
    Console.WriteLine($"clip {clip.Index}: frames {clip.StartFrame} to {clip.StartFrame + clip.FrameCount - 1}, {clip.SourceSize.Width}x{clip.SourceSize.Height}");
}

foreach (string warning in plan.Warnings)
{
    Console.WriteLine(warning);
}

File.WriteAllText("edit.json", edit.ToJson());
VideoEdit again = VideoEdit.FromJson(File.ReadAllText("edit.json"));

Transitions

A Transition overlaps the two clips it joins by its Duration, so the edit gets shorter by it, and crossfades their sound over the same time (AudioCurve, equal power by default). TransitionKind has ten built-in kinds: Fade (a cross-dissolve), FadeThroughColor, Dissolve, Wipe and Slide in a TransitionDirection, CircleOpen, CircleClose, Clock, Zoom, Pixelize and Blur. A wipe can have a soft edge (Softness), and any transition can ease its progress (Easing). The fades and wipes match FFmpeg's xfade to within its rounding and one pixel of the edge.

// A dissolve, then a soft wipe with eased progress: each overlaps the two clips by its length, sound crossfading too.
VideoEdit edit = new(new EditCanvas(320, 180, new Rational(25, 1)));
edit.Track.Add(Clip.Open("intro.avi"))
    .Add(Clip.Open("talk.avi"), Transition.Dissolve(MediaTime.FromSeconds(0.5)))
    .Add(Clip.Open("outro.avi"), Transition.Wipe(TransitionDirection.Left, MediaTime.FromSeconds(0.5)) with { Softness = 0.1, Easing = Easing.EaseInOut });
EditPlan plan = edit.Render("transitions.mkv");

An app's own transition implements ITransition, which draws one frame from the two clips' frames and the progress:

// An app's own transition: the incoming clip appears from the top down.
VideoEdit edit = new(new EditCanvas(320, 180, new Rational(25, 1)));
edit.Track.Add(Clip.Open("intro.avi")).Add(Clip.Open("outro.avi"), Transition.Custom(new Blinds(), MediaTime.FromSeconds(1)));
edit.Render("blinds.mkv");

private sealed class Blinds : ITransition
{
    public void Render(Raster<Rgba8> outgoing, Raster<Rgba8> incoming, double progress, Raster<Rgba8> target)
    {
        int rows = (int)Math.Round(progress * target.Height);
        for (int y = 0; y < target.Height; y++)
        {
            (y < rows ? incoming : outgoing).Row(y).CopyTo(target.Row(y));
        }
    }
}

Layers

A Layer draws a clip over the main track: placed in a rectangle (Place(Rect)) or at an Anchor with a margin and a scale, with an opacity and an Egami BlendMode, inside a time window (Between), with fades. Animate moves its LayerProperty values (X, Y, Scale, Rotation and Opacity) through Keyframes, timed from the layer's start. A layer ends with its clip, or holds the last frame (HoldLastFrame). Its sound is mixed in unless its clip is muted.

// Picture-in-picture: talk.avi in the bottom-right corner from 0.5 s to 1.5 s, sliding in from the right as it fades in.
VideoEdit edit = new(new EditCanvas(320, 180, new Rational(25, 1)));
edit.Track.Add(Clip.Open("intro.avi"));
edit.Layers.Add(Layer.Of(Clip.Open("talk.avi").Muted())
    .Place(Anchor.BottomRight, margin: 8, scale: 0.3)
    .Between(MediaTime.FromSeconds(0.5), MediaTime.FromSeconds(1.5))
    .FadeIn(MediaTime.FromSeconds(0.25))
    .Animate(LayerProperty.X, new Keyframe(MediaTime.Zero, 60, Easing.EaseOut), new Keyframe(MediaTime.FromSeconds(0.25), 0)));
edit.Render("pip.mkv");

Layer.Key runs a ChromaKey on the layer's frames: Egami's chroma key takes out the pixels near the screen colour in the CbCr plane, so uneven lighting on the screen matters little, and despill takes the screen's colour out of hair and edges.

// A green-screen clip over another: the key takes the studio green out and despill cleans its edges.
VideoEdit edit = new(new EditCanvas(320, 180, new Rational(25, 1)));
edit.Track.Add(Clip.Open("intro.avi"));
edit.Layers.Add(Layer.Of(Clip.Open("presenter-green.avi"))
    .Key(new ChromaKey(Rgba8.FromHex("#00b140")) { Similarity = 0.12f, Blend = 0.06f, Despill = 0.6f })
    .Place(new Rect(160, 90, 160, 90)));
edit.Render("keyed.mkv");

Stacks

Stack.Horizontal, Stack.Vertical and Stack.Grid (the three StackLayouts) make one clip of several shown at once, like FFmpeg's hstack, vstack and xstack. StackOptions sets a gap, the background, how grid cells fit, whether the stack ends with its shortest clip, and whether the sound is the first clip's or every clip's mixed. A stack is a clip, so it can be trimmed, faded and joined with a transition.

// Two clips side by side with a 4-pixel gap, each scaled to the first one's height; a stack is a clip like any other.
VideoEdit edit = new();
edit.Track.Add(Stack.Horizontal([Clip.Open("intro.avi"), Clip.Open("talk.avi")], new StackOptions { Gap = 4, MixAudio = true }));
EditPlan plan = edit.Render("side-by-side.mkv");
Console.WriteLine($"{plan.Canvas.Width}x{plan.Canvas.Height}");

Speed, reverse, loops and freezes

Clip.Speed plays a clip faster or slower, from a tenth of its speed to ten times: the picture takes the nearest frame, and the sound keeps its pitch through V9.1's Tempo (or follows it, like a tape, with keepPitch: false). Reverse plays a clip backwards, picture and sound, decoding one keyframe interval at a time, so memory stays bounded however long the clip is. Loop plays it several times, Freeze holds the frame at a time for a while (a ClipFreeze, with silence), and Hold keeps the last frame after the end. They apply in that order: speed, reverse, loops, freezes, then the hold. Frame counts and timestamps match FFmpeg's setpts with fps, reverse, loop and tpad.

// Twice as fast with the pitch kept, then backwards, then twice in a row with a frame frozen for half a second.
VideoEdit edit = new(new EditCanvas(320, 180, new Rational(25, 1)));
edit.Track
    .Add(Clip.Open("talk.avi").Speed(2))
    .Add(Clip.Open("intro.avi").Reverse())
    .Add(Clip.Open("outro.avi").Trim(MediaTime.Zero, MediaTime.FromSeconds(1)).Loop(2).Freeze(MediaTime.FromSeconds(0.5), MediaTime.FromSeconds(0.5)).Hold(MediaTime.FromSeconds(1)));
EditPlan plan = edit.Render("timing.mkv");

Stills, colours and an app's own frames

Clip.Still shows an image for a time, Clip.Animation plays an animated GIF, WebP or APNG at its own delays, and Clip.Color is one colour the canvas size. PanZoom moves the part of the picture shown from one rectangle to another (a PanZoomPath, in fractions of the picture), drawn with a sub-pixel transform from the full picture on every frame, so slow zooms do not jitter as whole-pixel crops do. Slideshow.Create builds an edit of images with transitions and, optionally, a pan and zoom for each from a seed (SlideshowOptions). Clip.FromSource takes an app's own IFrameSource, such as titles drawn with Egami.Drawing.

// Photos as a slideshow with fades and a slow pan and zoom, then a title an app draws, on black.
VideoEdit show = Slideshow.Create(["duck.png", "egret.png"], new SlideshowOptions
{
    Each = MediaTime.FromSeconds(3),
    PanZoom = true,
    Canvas = new EditCanvas(320, 180, new Rational(25, 1)) { Fit = ClipFit.Cover },
});
show.Track.Add(Clip.Color(Rgba8.Black, MediaTime.FromSeconds(0.5)), Transition.Fade(MediaTime.FromSeconds(0.5)));
show.Track.Add(Clip.FromSource(new Title()));
show.Track.Add(Clip.Still("duck.png", MediaTime.FromSeconds(2))
    .PanZoom(new System.Drawing.RectangleF(0, 0, 1, 1), new System.Drawing.RectangleF(0.3f, 0.3f, 0.4f, 0.4f), Easing.EaseInOut));
show.Render("slideshow.mkv");

private sealed class Title : IFrameSource
{
    public Size Size => new(320, 180);

    public MediaTime Duration => MediaTime.FromSeconds(1);

    public void Render(MediaTime time, Raster<Rgba8> target)
    {
        // A bar that grows across the frame: draw anything here, such as text with Egami.Drawing.
        int width = (int)(target.Width * time.Seconds / Duration.Seconds);
        for (int y = 0; y < target.Height; y++)
        {
            Span<Rgba8> row = target.Row(y);
            row.Fill(Rgba8.Black);
            if (y is > 80 and < 100)
            {
                row[..width].Fill(Rgba8.White);
            }
        }
    }
}

Effects

EditEffect.Apply runs Egami operations or any IRasterOperation on each frame, and EditEffect.Of runs an app's own IFrameEffect, which gets the frame's time and its progress through the effect's window in a FrameEffectContext. Effects attach to the whole edit (VideoEdit.Effects), to a clip (Clip.WithEffect) or to a layer (Layer.WithEffect), and Between limits one to a time window. An effect keeps the frame size.

// Blur the whole edit from 0.5 s to 1 s, and darken one clip with an app's own effect that knows the time.
VideoEdit edit = new(new EditCanvas(320, 180, new Rational(25, 1)));
edit.Track.Add(Clip.Open("intro.avi").WithEffect(EditEffect.Of(new Darken())));
edit.Effects.Add(EditEffect.Apply(ops => ops.Blur(4), "blur 4").Between(MediaTime.FromSeconds(0.5), MediaTime.FromSeconds(1)));
edit.Render("effects.mkv");

private sealed class Darken : IFrameEffect
{
    public void Apply(Raster<Rgba8> frame, FrameEffectContext context)
    {
        float keep = 1 - (0.5f * (float)context.Progress);
        for (int y = 0; y < frame.Height; y++)
        {
            foreach (ref Rgba8 p in frame.Row(y))
            {
                p = new Rgba8((byte)(p.R * keep), (byte)(p.G * keep), (byte)(p.B * keep), p.A);
            }
        }
    }
}

Plug-in points

An app adds its own pieces at five places, each an interface it implements: ITransition draws a transition between two clips' frames, IFrameEffect changes a frame knowing its time and progress, IRasterOperation (Egami's own operation interface) runs through EditEffect.Apply as an effect, IFrameSource makes a clip's frames, such as titles drawn by the app, and ChromaKey settings key a layer. All of them work on Egami rasters, so they need nothing from the video side.

A model on every frame

A machine-learning model, such as a style transfer or a denoiser, plugs in as an IRasterOperation. The adapter below turns each frame into a 1x3xHxW tensor of red, green and blue from 0 to 1, hands it to a model, and writes the model's tensor back. Here the model is a stand-in written in C# (a sepia tone), so the sample runs with the tests:

/// <summary>
/// Runs a model on each frame: the frame goes in as a 1x3xHxW tensor of red, green and blue from 0 to 1, and the model's
/// tensor of the same shape comes back into the frame. Alpha is kept.
/// </summary>
public sealed class ModelOperation(Func<float[], int, int, float[]> model) : IRasterOperation
{
    public Size OutputSize(Size input) => input;

    public void Run<TPixel>(OperationInput<TPixel> input)
        where TPixel : unmanaged, IPixelType<TPixel>
    {
        int width = input.Source.Width, height = input.Source.Height, plane = width * height;
        float[] tensor = new float[3 * plane];
        float[] alpha = new float[plane];
        for (int y = 0; y < height; y++)
        {
            Span<TPixel> row = input.Source.Row(y);
            for (int x = 0; x < width; x++)
            {
                Vector4 v = row[x].ToScaled();
                int i = (y * width) + x;
                tensor[i] = v.X;
                tensor[plane + i] = v.Y;
                tensor[(2 * plane) + i] = v.Z;
                alpha[i] = v.W;
            }
        }

        float[] result = model(tensor, width, height);
        if (result.Length != tensor.Length)
        {
            throw new InvalidOperationException($"The model returned {result.Length} values, not {tensor.Length}.");
        }

        for (int y = 0; y < height; y++)
        {
            Span<TPixel> row = input.Destination.Row(y);
            for (int x = 0; x < width; x++)
            {
                int i = (y * width) + x;
                row[x] = TPixel.FromScaled(Vector4.Clamp(new Vector4(result[i], result[plane + i], result[(2 * plane) + i], alpha[i]), Vector4.Zero, Vector4.One));
            }
        }
    }
}

/// <summary>A stand-in for a style model: a sepia tone, as a tensor in and a tensor out.</summary>
public static float[] Sepia(float[] tensor, int width, int height)
{
    int plane = width * height;
    float[] output = new float[tensor.Length];
    for (int i = 0; i < plane; i++)
    {
        float r = tensor[i], g = tensor[plane + i], b = tensor[(2 * plane) + i];
        output[i] = (0.393f * r) + (0.769f * g) + (0.189f * b);
        output[plane + i] = (0.349f * r) + (0.686f * g) + (0.168f * b);
        output[(2 * plane) + i] = (0.272f * r) + (0.534f * g) + (0.131f * b);
    }

    return output;
}

// A model as an effect: every frame of the edit from 0.2 s goes through it.
VideoEdit edit = new();
edit.Track.Add(Clip.Open("clip.mkv"));
edit.Effects.Add(EditEffect.Apply(new ModelOperation(Sepia), "sepia model").Between(MediaTime.FromSeconds(0.2)));
edit.Render("styled.mkv");

With ONNX Runtime, an app's own reference to the Microsoft.ML.OnnxRuntime package (Egami.Video has none), the stand-in becomes a call into an InferenceSession. These lines are app code shown as a sketch: they are not compiled or run with Egami's tests, and the input name and shape depend on the model.

// Untested app code: needs the Microsoft.ML.OnnxRuntime package and a model file.
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;

using InferenceSession session = new("style.onnx");
string inputName = session.InputMetadata.Keys.First();

float[] RunModel(float[] tensor, int width, int height)
{
    DenseTensor<float> input = new(tensor, [1, 3, height, width]);
    using IDisposableReadOnlyCollection<DisposableNamedOnnxValue> results = session.Run([NamedOnnxValue.CreateFromTensor(inputName, input)]);
    return results.First().AsTensor<float>().ToArray();
}

edit.Effects.Add(EditEffect.Apply(new ModelOperation(RunModel), "style model"));

Smart render

EditRenderOptions.SmartRender (the command line's --smart) copies the untouched parts of clips instead of encoding them again. Where one clip fills the picture with no transition, fade, layer, effect or change of time, its source has the canvas's size and frame rate, and its stream matches what the encoder writes (VP8, VP9, AV1 or Motion JPEG with the same profile, bit depth, chroma, color and, for AV1, sequence header), its packets are copied byte for byte from the first keyframe on, and the encoder starts again with a keyframe after each copied part. A source that does not match is encoded as usual, with a MediaWarning. The sound is always mixed and encoded. It is off by default.

Interlaced clips

Clip.Deinterlace deinterlaces a clip's frames as the edit decodes them, with the Deinterlacer of the video guide: adaptive or bob, at the frame or the field rate. It is off by default, so edits render as they always did; turn it on for interlaced sources, such as broadcast H.264. At the field rate a canvas at the field rate (50 fps for 50i) shows every field. The setting is kept in the edit's JSON, and --deinterlace sets it on the CLI's editing commands. A deinterlaced clip is always encoded again, never copied by smart render.

// Deinterlace a broadcast clip as the edit decodes it; one frame per field fills a 50 fps canvas with every field.
VideoEdit edit = new(new EditCanvas(320, 240, new Rational(50, 1)));
edit.Track.Add(Clip.Open("movie.webm").Deinterlace(new DeinterlacerOptions { Rate = DeinterlaceRate.Field }));
edit.Render("smooth.mkv");

Stabilizing

Stabilizing takes two passes, like FFmpeg's vidstabdetect and vidstabtransform. Stabilizer.Analyze measures the camera's motion from each frame to the next: it tracks corners with pyramidal Lucas-Kanade on a copy of the luma no wider than MotionAnalysisOptions.AnalysisWidth and fits a turn, a zoom and a move to them with RANSAC. The CameraMotion it returns saves as JSON, so the slow pass runs once; its Frames are FrameMotions, the move, turn and zoom from the frame before. Clip.Stabilize then smooths the camera's path with a Gaussian over StabilizeOptions.Smoothing frames either way and warps each frame onto the smooth path; the clip's Stabilization holds both as a ClipStabilization. StabilizeEdges says what fills the edges a correction uncovers: Zoom (the least zoom that hides them in every frame, up to StabilizeOptions.MaxZoom, 1.5 by default), Black or Blur. The command line's stabilize does both passes, and --save-motion and --motion split them.

// Two passes: measure the camera's motion, then render the clip warped onto a smoothed path, zoomed to hide the edges.
CameraMotion motion = Stabilizer.Analyze("talk.avi");
File.WriteAllText("talk.motion.json", motion.ToJson());
VideoEdit edit = new();
edit.Track.Add(Clip.Open("talk.avi").Stabilize(motion, new StabilizeOptions { Smoothing = 20, Edges = StabilizeEdges.Zoom }));
edit.Render("steady.mkv");
Edit this page