Table of Contents

Audio filters

Egami.Video's audio filters change level, time, tone and placement of decoded audio as it streams: gain and gain envelopes, fades and crossfades, mixing and panning, delay, padding, trimming and new timestamps, tempo and speed changes, EQ, dynamics (compressor, gate and limiter), loudness normalization and silence removal. They live in Egami.Video.Audio.Filters and work on the AudioFrames that AudioReader and the decoders hand out (see Audio).

The filter loop

Every filter is an IAudioFilter with the same calls as Resampler: Send a frame, TryReceive the frames that are ready, Flush at the end of the input and Reset after a seek. Latency says how much input a filter holds back, such as the lookahead of Tempo. Frames of any sample format go in and are converted once to planar float; frames come out as SampleFormat.F32Planar with exact timestamps. The filters that keep the length work on the frame in place, so a steady stream allocates no sample memory. Times you give a filter, such as where a fade starts or a trim cuts, are the frames' own timestamps, not counted from the first sample, as in FFmpeg: audio that starts at 1.4 s fades at 1.4 s only if you say so. Put a Retime first to count from zero.

AudioFilterChain runs several filters on one input in one pass and is itself a filter. It owns the filters added to it. Egami.Video's filters derive from AudioFilter, which takes the rate and channels from the first frame and refuses frames that differ; to put a filter of your own in a chain, implement IAudioFilter.

// Fade in over 2 s, then turn down 3 dB: frames of any format go in, F32 planar frames come out.
using AudioFilterChain chain = new AudioFilterChain(48000, ChannelLayout.Stereo)
    .Add(new AudioFade(FadeDirection.In, MediaTime.Zero, MediaTime.FromSeconds(2)))
    .Add(new Gain(-3));
foreach (AudioFrame frame in frames)
{
    chain.Send(frame);
    while (chain.TryReceive(out AudioFrame? output))
    {
        samples += output.SampleCount; // write it to an AudioWriter or encoder here
        output.Dispose();
    }
}

chain.Flush();

Gain

Gain changes the level by a fixed number of Decibels or a linear Factor. Changing either between frames moves to the new gain over 5 ms, so it does not click; unity gain passes samples through unchanged. Its Envelope, a GainEnvelope, changes the gain over time: a list of Keyframes on the frames' time line, values in decibels, each with an Easing (linear, ease in, ease out, ease in and out, or hold) to the next. The same Keyframe type animates layers in video editing.

// Duck to -12 dB from 1 s to 2 s, easing down and back up over a quarter second.
using Gain duck = new()
{
    Envelope = new GainEnvelope(
        new Keyframe(MediaTime.FromSeconds(0.75), 0, Easing.EaseInOut),
        new Keyframe(MediaTime.FromSeconds(1), -12),
        new Keyframe(MediaTime.FromSeconds(2), -12, Easing.EaseInOut),
        new Keyframe(MediaTime.FromSeconds(2.25), 0)),
};

Fades and crossfades

AudioFade fades in (FadeDirection.In, silent before its start) or out (FadeDirection.Out, silent after its end) over a stretch of the time line, and reaches full level or silence on the exact sample. FadeCurve has five shapes with the same gains as FFmpeg's afade curves: Linear (tri, the default), EqualPower (qsin), Logarithmic (log), Exponential (exp) and SCurve (hsin).

// A 3 s clip: fade out over its last second along an S curve.
using AudioFade fadeOut = new(FadeDirection.Out, MediaTime.FromSeconds(2), MediaTime.FromSeconds(1), FadeCurve.SCurve);

Crossfade joins two streams with an overlap, like FFmpeg's acrossfade: the first stream's end fades out as the second's start fades in, EqualPower by default so unrelated material keeps its loudness. It has two inputs, so it takes Send(input, frame) and Complete(input) instead of the single-input calls.

// Join two clips with a 1 s equal-power overlap: 3 s + 2 s - 1 s = 4 s of output.
using Crossfade crossfade = new(MediaTime.FromSeconds(1));
foreach (AudioFrame frame in first)
{
    crossfade.Send(0, frame);
}

crossfade.Complete(0);
foreach (AudioFrame frame in second)
{
    crossfade.Send(1, frame);
}

crossfade.Complete(1);
int samples = 0;
while (crossfade.TryReceive(out AudioFrame? output))
{
    samples += output.SampleCount;
    output.Dispose();
}

Mixing and panning

AudioMixer adds several inputs, each times its weight (SetWeight), and places each by its timestamps, so a clip that starts later comes in at its time. Inputs at another rate or with other channels are resampled and remixed on the way in. It does not divide by the number of inputs unless Normalize is set, so a voice over music keeps its level. MixDuration says when the mix ends: with the longest input (the default), the shortest, or input 0.

Pan places a mono input in stereo with the constant-power law (-3 dB in each channel at the centre), turns one side of a stereo input down with Position or Balance, swaps the channels and flips their polarity. For any other channel matrix use ChannelMixer.

// A mono voice at 44.1 kHz over stereo music at 48 kHz, the music at half level: the voice is panned a little left,
// then resampled and placed by its timestamps on the way into the mixer.
using Pan pan = new(-0.3);
using AudioMixer mixer = new(48000, ChannelLayout.Stereo, inputs: 2);
mixer.SetWeight(0, 0.5);
foreach (AudioFrame frame in music)
{
    mixer.Send(0, frame);
}

foreach (AudioFrame frame in voice)
{
    pan.Send(frame);
    while (pan.TryReceive(out AudioFrame? panned))
    {
        mixer.Send(1, panned);
        panned.Dispose();
    }
}

mixer.Flush();

Timing

Delay puts silence in front of each channel, by one delay or one per channel, and hands out the delayed tail on Flush. Pad adds silence at the end, a fixed amount or up to a length (Pad.ToLength). Trim keeps the part between two times, cut on the exact sample, and keeps its timestamps; Retime gives the stream new timestamps that start at a time and run on without gaps.

// Keep 0.5 s to 2.5 s, start it at zero, delay the right channel 20 ms and add half a second of silence.
using AudioFilterChain chain = new AudioFilterChain(48000, ChannelLayout.Stereo)
    .Add(new Trim(MediaTime.FromSeconds(0.5), MediaTime.FromSeconds(2.5)))
    .Add(new Retime())
    .Add(new Delay([MediaTime.Zero, MediaTime.FromSeconds(0.02)]))
    .Add(new Pad(MediaTime.FromSeconds(0.5)));

Tempo and speed

Tempo plays faster or slower and keeps the pitch, from a quarter of the speed to four times, with WSOLA (waveform similarity overlap-add). TempoPreset.Music uses longer windows for steady rhythm and low notes, TempoPreset.Speech shorter ones for speech. The output lasts the input's length divided by the factor, to the sample. Speed changes speed and pitch together by resampling, like a tape played faster, and costs less.

// A quarter faster with the pitch kept (3 s becomes 2.4 s); Speed would raise the pitch too, like a tape.
using Tempo tempo = new(1.25, TempoPreset.Speech);
using Speed speed = new(1.25);

EQ

Biquad is a second-order filter from the Audio EQ Cookbook in one of eight BiquadKinds: low and high pass, band pass, notch, all pass, peaking and low and high shelves. Its width is a Q (0.7071, Biquad.ButterworthQ, by default) or, with Biquad.FromBandwidth, a bandwidth in octaves; peaking and shelves take a Gain in decibels. Butterworth is a steeper low or high pass, order 2 to 8 (12 to 48 dB per octave), flat in the pass band and -3 dB at its corner. ParametricEqualizer runs a list of EqualizerBands as one filter. All of them run in double precision in place, and refuse a frequency at or above half the sample rate.

// Rumble out below 80 Hz (24 dB per octave), a little warmth, less harshness and some air.
using AudioFilterChain chain = new AudioFilterChain(48000, ChannelLayout.Stereo)
    .Add(new Butterworth(BiquadKind.HighPass, 80, order: 4))
    .Add(new ParametricEqualizer(
        new EqualizerBand(BiquadKind.LowShelf, 150, Gain: 2),
        new EqualizerBand(BiquadKind.Peaking, 3000, Gain: -4, Q: 1.5),
        new EqualizerBand(BiquadKind.HighShelf, 10000, Gain: 3)))
    .Add(Biquad.FromBandwidth(BiquadKind.Notch, 50, octaves: 0.1)); // mains hum

Dynamics

Compressor turns the level down above a Threshold by a Ratio, with a soft Knee, Attack and Release times and Makeup gain; Gate turns it down below a threshold, by up to its Range, to keep noise out of the pauses. Both measure the loudest channel (LevelDetection.Rms by default, or Peak) and give every channel the same gain. Limiter looks 5 ms ahead, so no sample goes over its Ceiling; with TruePeak it also catches the peaks between samples by oversampling, as ITU-R BS.1770 measures them. Its output is held back by its Latency and keeps the input's timestamps.

// A voice chain: gate the room noise, even out the level 4:1 above -20 dBFS, then keep true peaks under -1 dBTP.
using AudioFilterChain chain = new AudioFilterChain(48000, ChannelLayout.Stereo)
    .Add(new Gate { Threshold = -50, Range = -20 })
    .Add(new Compressor { Threshold = -20, Ratio = 4, Makeup = 4 })
    .Add(new Limiter { Ceiling = -1, TruePeak = true });

With Sidechain on, a compressor takes its level from another signal given to SendSidechain, lined up by timestamps, so music ducks under a voice. Input waits until the sidechain covers it; CompleteSidechain says no more comes.

// Music ducks under a voice-over: the voice's level drives the music's compressor.
using Compressor ducker = new() { Sidechain = true, Threshold = -30, Ratio = 8, Release = MediaTime.FromSeconds(0.4) };
for (int i = 0; i < music.Count; i++)
{
    ducker.SendSidechain(voice[i]);
    ducker.Send(music[i]);
    while (ducker.TryReceive(out AudioFrame? output))
    {
        samples += output.SampleCount; // mix it with the voice in an AudioMixer here
        output.Dispose();
    }
}

ducker.CompleteSidechain();
ducker.Flush();

Loudness

LoudnessNormalizer brings audio to a LoudnessTarget: an integrated loudness, a true-peak ceiling and, for the dynamic mode, a loudness range. The presets are EbuR128 (-23 LUFS, -1 dBTP, the default), AtscA85 (-24, -2), Streaming (-14, -1) and Podcast (-16, -1). Given a first pass's LoudnessReport from LoudnessMeter, it is linear: one gain, so the dynamics stay as they are, and a true-peak Limiter only when that gain would take the peaks over the ceiling. AudioFilters.NormalizeLoudnessAsync runs both passes over every audio track of a file, copying the video and other tracks. LoudnessNormalizer.Dynamic works in one pass, for input that cannot be read twice: it follows the loudness with 1.5 s of lookahead and keeps it within the target's range, so it changes the dynamics and lands less exactly on the target.

// Two passes over a file: measure each audio track, then apply one gain (and a true-peak limiter only if needed).
IReadOnlyList<LoudnessNormalization> done = await Egami.Video.Audio.Filters.AudioFilters.NormalizeLoudnessAsync(input, output, LoudnessTarget.Podcast);
Console.WriteLine($"{done[0].Measured.Integrated:F1} LUFS, {done[0].Gain:+0.0;-0.0} dB, limited: {done[0].Limited}");

// The same as a filter, from your own first pass; or one pass, following the loudness as it goes.
LoudnessMeter meter = new();
foreach (AudioFrame frame in Music())
{
    using (frame)
    {
        meter.Add(frame);
    }
}

using LoudnessNormalizer linear = new(LoudnessTarget.EbuR128, meter.Report());
using LoudnessNormalizer live = LoudnessNormalizer.Dynamic(LoudnessTarget.Streaming);

AudioFilters.ApplyAsync runs filters of your own on every audio track of a file the same way. Filtered tracks are encoded with the codec you give, else their own when the container holds it, else the container's usual one.

Silence

SilenceRemover cuts silence out: a sample is silent when the loudest channel's RMS over 20 ms is under its Threshold (-50 dBFS by default). SilenceRemoval.Trim cuts the start and the end; SilenceRemoval.Shorten also cuts pauses longer than MaxGap (1 s) to KeepGap (0.3 s). Every cut fades or crossfades over 10 ms. The result is shorter than the input and its timestamps run on without gaps, so use it on audio alone, not on a track that has to stay in step with video.

// Cut a voice recording's lead-in and tail, and pauses over 1 s down to 0.3 s. Audio only: the result is shorter.
using SilenceRemover remover = new(SilenceRemoval.Shorten) { Threshold = -45 };

On the command line

egami-video audio runs these filters with --gain, --fade-in, --fade-out, --tempo, --normalize, --trim-silence and --af, which takes ffmpeg's -af syntax (highpass=f=80,acompressor=threshold=-20dB:ratio=3,loudnorm); encode and run take them for the audio they add or copy, and egami-video ffmpeg turns an -af chain into the same command. See the egami-video command.

Edit this page