Audio mixing with Pixel Bender

Time to have some advanced fun with Pixel Bender. Recently someone in the community complained to us that mixing 13 mp3 tracks using the dynamic sound playback feature in Flash Player 10 does not really work. Well, true with the sample project he gave us. Doing dynamic sound playback is generally tricky to get right. I can provide a few tips though.

1. Pick the right mp3 encoding format

It's important you pick a format which consumes the least amount of CPU time for decoding. Specifically you should always choose 44.1Khz as the sample rate for your mp3 files. Why? The Flash Player will otherwise have to re-sample and filter your audio which takes away precious CPU cycles.

The tricky part here is that mp3 encoders usually pick the sample rate automatically, including Adobe Audition. Especially at a bit rate of 64kb or less it will try to switch to 24Khz or 22Khz. You can override this at least for CBR in Audition using the advanced settings at export time if you need to.

2. Keep things simple

Do all your processing in one function if you can. Function calls are expensive generally. Try to read and write data only once. Ideally your mixing code should look something like this if you use pure ActionScript:

var buffer:Vector.<ByteArray> = new Vector.<ByteArray>(NUM_TRACKS);
var sound:Vector.<Sound> = new Vector.<Sound>(NUM_TRACKS);

function onSoundData(sampleDataEvent:SampleDataEvent) : void
{
  for (var i:int = 0; i <  NUM_TRACKS; i++) {
    buffer[i].position = 0;
    sound[i].extract(buffer[i], BUFFER_SIZE); 
    buffer[i].position = 0;
  }

  for (var j:int = 0; j < BUFFER_SIZE*2; j++) 
  {
    var val:Number = 0;
    for (var k:int = 0; k < NUM_TRACKS; k++) 
    {
      val += buffer[k].readFloat();
    }
    sampleDataEvent.data.writeFloat(val);
  }
}

You will notice that you will spend a lot of time in this function. So...

3. Use Pixel Bender to mix sounds

I have talked to some who have tried to use Pixel Bender for audio processing. They had little success most of the time. Truth is, our tools are not ready yet. But with some patience and using the assembler for creating .pbj files I posted recently you can make it happen today.

One problematic issue is that right now the Pixel Bender toolkit is designed to handle image data. What does that mean? The toolkit limits you to float3 and float4 output types right now which is not really what you want. Now you might think you could just use float4. Not so. You will notice that Flash Player 10 has a pretty bad bug which makes it not work when float4 types are used for output. I am really angry we did not catch this sooner, hopefully we can address this bug sooner than later. What you are left with is using pure Pixel Bender assembly code for now which allows you to use a float2 output type.

For my experiment I took the Adobe Audition theme sample project and exported all tracks as separate .mp3 files, 15 in total. Incidentally that is also the maximum amount of inputs you can use for a single shader. The goal was to mix all 15 tracks in real time using the dynamic sound playback feature.

Here is the Pixel Bender assembly code I used to create my .pbj file:

  version 1
  name "SoundMixer"
  kernel "namespace", "adobe"
  kernel "vendor", "Adobe Systems"
  kernel "version", 1
  kernel "description", "A generic sound mixer with volume control"

  parameter "_OutCoord", float2, f0.rg, in

  texture "track0", t0.rg
  texture "track1", t1.rg
  texture "track2", t2.rg
  texture "track3", t3.rg
  texture "track4", t4.rg
  texture "track5", t5.rg
  texture "track6", t6.rg
  texture "track7", t7.rg
  texture "track8", t8.rg
  texture "track9", t9.rg
  texture "track10", t10.rg
  texture "track11", t11.rg
  texture "track12", t12.rg
  texture "track13", t13.rg
  texture "track14", t14.rg

  parameter "volume0", float2, f3.rg, in
  meta "defaultValue", 1, 1
  parameter "volume1", float2, f4.rg, in
  meta "defaultValue", 1, 1
  parameter "volume2", float2, f5.rg, in
  meta "defaultValue", 1, 1
  parameter "volume3", float2, f6.rg, in
  meta "defaultValue", 1, 1
  parameter "volume4", float2, f7.rg, in
  meta "defaultValue", 1, 1
  parameter "volume5", float2, f8.rg, in
  meta "defaultValue", 1, 1
  parameter "volume6", float2, f9.rg, in
  meta "defaultValue", 1, 1
  parameter "volume7", float2, f10.rg, in
  meta "defaultValue", 1, 1
  parameter "volume8", float2, f11.rg, in
  meta "defaultValue", 1, 1
  parameter "volume9", float2, f12.rg, in
  meta "defaultValue", 1, 1
  parameter "volume10", float2, f13.rg, in
  meta "defaultValue", 1, 1
  parameter "volume11", float2, f14.rg, in
  meta "defaultValue", 1, 1
  parameter "volume12", float2, f15.rg, in
  meta "defaultValue", 1, 1
  parameter "volume13", float2, f16.rg, in
  meta "defaultValue", 1, 1
  parameter "volume14", float2, f17.rg, in
  meta "defaultValue", 1, 1

  parameter "output", float2, f1.rg, out

;----------------------------------------------------------

  texn f1.rg, f0.rg, t0
  mul f1.rg, f3.rg
  texn f2.rg, f0.rg, t1
  mul f2.rg, f4.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t2
  mul f2.rg, f5.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t3
  mul f2.rg, f6.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t4
  mul f2.rg, f7.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t5
  mul f2.rg, f8.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t6
  mul f2.rg, f9.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t7
  mul f2.rg, f10.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t8
  mul f2.rg, f11.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t9
  mul f2.rg, f12.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t10
  mul f2.rg, f13.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t11
  mul f2.rg, f14.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t12
  mul f2.rg, f15.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t13
  mul f2.rg, f16.rg
  add f1.rg, f2.rg
  texn f2.rg, f0.rg, t14
  mul f2.rg, f17.rg
  add f1.rg, f2.rg

Looks complicated, but in fact this does nothing more that the above ActionScript code, with unrolled loops. As an extra you can control the volume on each track.

To use the shader I wrote this little piece (note that this is incomplete code, it will not compile):

// Create shader
[Embed(source="mixer.pbj", mimeType="application/octet-stream")]
var MixerShader:Class;
var mixerShader:Shader = new Shader(new MixerShader());

// buffers will become shader inputs
var buffer:Vector.<ByteArray> = new Vector.<ByteArray>(15);
// volume control volume on each track, 1.0 is full volume
var volume:Vector.<Number> = new Vector.<Number>(15);

// initialize the shader inputs and volume values
for (var j:int = 0; j < 15; j++) {
  volume[j]=1.0;
  buffer[j] = new ByteArray();
  // set so shader will always work even we have not enough tracks
  buffer[j].length = BUFFER_SIZE*4*2; 
  mixerShader.data["track"+j]["width"] = 1024;
  mixerShader.data["track"+j]["height"] = BUFFER_SIZE/1024;
  mixerShader.data["track"+j]["input"] = buffer[j];
}

function onSoundData(e:SampleDataEvent) : void
{
  // extract the mp3 data into our shader inputs
  for (var i:int = 0; i < NUM_TRACKS; i++) {
    buffer[i].position = 0;
    sounds[i].extract(buffer[i], BUFFER_SIZE); 
    buffer[i].position = 0;
  }
  // update the volume value in the shader
  for (var k:int = 0; k < NUM_TRACKS; k++)  {
    mixerShader.data["volume"+k]["value"] = [ volume[k], volume[k] ];
  }
  // mix!
  var mixerJob:ShaderJob = new ShaderJob(mixerShader, e.data, 1024, BUFFER_SIZE/1024);
  mixerJob.start(true);
}

Compared to the pure AS3 version this runs twice as fast overall. On my Core 2 Mac mixing the 15 tracks consumes about 24% of one CPU. So if you are doing simple audio mixing like this Pixel Bender is a good choice. YMMV depending on what application we are talking about and how much processing you need to do on the audio.