Wednesday, May 14, 2008

Adobe Is Making Some Noise Part 3

[Update: I have updated the code sample to match the API changes in build 10.0.1.525]

Along with the new event to drive dynamically generated audio playback there is one gaping hole: Where do you get source audio from? Sure, you can load raw audio from external sources through ByteArray but that is less than efficient. What you really need is a way to have sound assets you can access from your ActionScript code.

In Flash Player 10 code named Astro the Sound object will have one more method which is designed to work together with the "samplesCallback" event handler. It will extract raw sound data from an existing sound asset. That means any mp3 file you have in the library or load externally can be accessed and processed. Let's look at an actual code example. I am passing through audio from an existing sound object to another sound object doing the dynamic audio playback:

var mp3sound:Sound = new Sound();
var dynamicSound:Sound = new Sound();
var samples:ByteArray = new ByteArray();

function sampleData(event:SampleDataEvent):void {
samples.position = 0;
var len:Number = mp3sound.extract(samples,1777);
if ( len < 1777 ) {
// seamless loop
len += mp3sound.extract(samples,1777-len,0);
}
samples.position = 0;
for ( var c:int=0; c < len; c++ ) {
var left:Number = samples.readFloat();
var right:Number = samples.readFloat();
event.data.writeFloat(left);
event.data.writeFloat(right);
}
}

private function loadComplete(event:Event):void {
dynamicSound.addEventListener("sampleData",sampleData);
dynamicSound.play();
}

mp3sound.addEventListener(Event.COMPLETE, loadCompleteMP3);
mp3sound.load(new URLRequest("sound.mp3"));

Notice the extract() call here. This function will extract raw sound data from any existing sound object. The format returned is always 44100Khz stereo, the number format is 32-bit floating point which you can read and write using ByteArray.readFloat and ByteArray.writeFloat. The floating point values are normalized between -1.0 and 1.0. Here is the full prototype of this function:

function extract(target:ByteArray,
length:Number,
startPosition:Number = -1 ):Number;
  • target: A ByteArray object in which the extracted sound samples should be placed.
  • length: The number of sound samples to extract. A sample contains both the left and right channels -- that is, two 32-bit floating point values.
  • startPosition: The sample at which extraction should begin. If you don't specify a value, the first call to extract() starts at the beginning of the sound; subsequent calls without a value for startPosition progress sequentially through the sound.
  • extract() returns the number of samples which could be retrieved. This might be less than the length you requested at the very end of a sound.


With what we provide in Flash Player 10 we hope that we are addressing the most pressing needs of what you want to do with sound. I will likely be just a matter of time until we'll see high level frameworks done by the community on the magnitude of something like Papervision3D. The next couple of years should be very interesting indeed when it comes to sound on the web.

What's missing? Unfortunately some features did not make it into Flash Player 10: Extracting audio data from a microphone and extracting audio from a NetStream object. We are aware that both features are highly desirable, but for various reasons it was not possible to make this happen in this release.

24 Comments:

Blogger mdk said...

brilliant.

finally my flash player dreams are coming true.

Thursday, May 15, 2008 1:22:00 AM  
Blogger iiley said...

thanks for the good thing, that's awesome.

Thursday, May 15, 2008 1:38:00 AM  
Blogger Erki Esken said...

"Extracting audio data from a microphone..."

Is this something you're looking for adding in version 10 point release or more likely version 11 (i.e. a lot of engineering effort)?

Thursday, May 15, 2008 3:05:00 AM  
Blogger Seb said...

Hi Tinic!

This is indeed great news, and will definitely open up a lot of possibilities!

I'm also one of those annoying people who could really do with microphone access, as we are looking at a lot of very cool collaborations with large music groups and realtime projections (similar to the flash digital fireworks http://www.sebleedelisle.com/?p=178 but sound triggered).

So you think there's any possibility of microphone data access? Would be good to know. Hate having to use the java socket server approach!

cheers!

Seb Lee-Delisle

Thursday, May 15, 2008 3:41:00 AM  
Blogger Neave said...

That's fantastic news Tinic, and a great post too - thanks for taking us through the gritty details.

I too would love microphone support, but I guess we'll just have to wait a little bit longer.

Thursday, May 15, 2008 4:04:00 AM  
Blogger Matt Rix said...

Hey Tinic,

Is there any way to generate sounds in advance(ie. not in realtime) using any of these new features? I currently have a Flash audio app where I load a wav file and then "cache" a bunch of pitch shifted versions of it in a specific musical scale. Will I still have to do this sort of thing using the old technique?

Thursday, May 15, 2008 6:38:00 AM  
OpenID glenatcrash said...

So, does this mean, if I load an mp3 file, and choose to grab bytes from it, and play it, it will loop without the usual mp3 hiccup or glitch when trying to loop mp3 files? Or at the very least, can we REMOVE the silence that mp3 files insert and thus make a seamelessly looping mp3?

Thursday, May 15, 2008 7:05:00 AM  
OpenID FactoryJustin said...

This is so exciting. Thanks for the great posts and the hard work.

A slight correction in the example code - loadComplete and loadCompleteMP3 are conflicting - they should both be one or the other.

Thursday, May 15, 2008 12:07:00 PM  
Blogger efnx said...

So when can we expect to use said Sound improvements in our own projects?

Thursday, May 15, 2008 3:01:00 PM  
Blogger Anatole said...

Tinic,
Is there a chance that at least event with Microphone sound data in uncompressed format we can listen to be added to Microphone object? How about pacifica codecs?
Thank you
Anatole

Friday, May 16, 2008 10:42:00 AM  
Blogger Olivier Groulx said...

Hi,

I've been playing with this since this morning. Very cool stuff. However, I think there is something wrong with the third parameter of the extract function, or else I'm doing something wrong. Basically, I'm trying to increment the start position myself.

My code is exactly like yours, except that I keep a "position" variable for and increment it of "len" each time.

samples.position = 0;
var len:Number = mp3Sound.extract(samples, 1024, temppos);
temppos += len;

samples.position = 0;

for (var j:int = 0; j < len; j++ )
{
dynamicSound.samplesCallbackData.writeFloat(samples.readFloat());
dynamicSound.samplesCallbackData.writeFloat(samples.readFloat());
}

I don't know how to describe the sound that I get from this, however I know it's not right :)

Monday, May 19, 2008 8:58:00 PM  
Blogger Frank said...

I wonder if it now will be possible to generate a static waveform of a whole soundfile before its played?
If so, then Flash 10 will be ordered at my work :)

Tuesday, May 20, 2008 6:21:00 AM  
Blogger dorkie dork from dorktown said...

I appreciate the new features but I don't think I can wait any longer for some sound features.

* getting byte data from a microphone
* creating seemless loops
* setting the start and end points for a sound
* the other most important features at adobemakesomenoise.com

thanks but please please please delay flash player 10 to add these in. because without these it will be 2 more years to wait...

Tuesday, May 20, 2008 10:59:00 PM  
Blogger spender said...

Good stuff Flash Player team!

I've been doing some experimentation with these new features. Take a look here:

http://www.flexiblefactory.co.uk/flexible/?p=37

Wednesday, May 21, 2008 5:49:00 AM  
Blogger jeanphilippe said...

Hi,
Can you show me the way to extract a part of a sound and save it on desktop with filereference save() ?
Thanks

Friday, May 23, 2008 2:40:00 AM  
Blogger joshspoon said...

That's nice. Will this work on recording dynamic flash created sounds? I'm working on building a keyboard: http://tinyurl.com/5tjgro

Sunday, May 25, 2008 1:30:00 AM  
Blogger Jack said...

Would it be possible to do any dsp on the audio the way VST or DirectX plugiins do?

Tuesday, June 03, 2008 7:00:00 PM  
Blogger Jack said...

Do you think someone with DSP knowledge could now create something akin to the VST or DirectX plugins for music/audio applications, such as reverb, echo, flanging etc?

Tuesday, June 17, 2008 6:49:00 AM  
Blogger sc said...

yes tinic great stuff. but i know what u and adobe and savvy people here know: the one who gets low/no latency mic data into flash for spectral analysis wins. wins big. wins so big they could make anyone sell their souls (or at least their cars) to get it. So if adobe is trying to screw us, like making an FMS-type solution to this problem that is priced similarly to a car, tell them no, tell them to hire more people like you, and delay the release of Flash CS4/10 as long as nessecary. The flash community would grow at a much more feverish pace if it developed a more symbiotic relationship with the indie music community...
if u dont get this solution done for the upcoming release, all the people who are on the fence about diving deeper into flash rather than learning processing/MaxMSP/etc... will hop off, and wave goodbye to what could have been one of the most important technologies driving the future convergence of art and music.

Monday, July 21, 2008 8:59:00 AM  
Blogger Jim Cheng said...

Tinic,

This is great news and opens a lot of doors for building novel audio applications in Flash Player 10. However, I've run into a rather painful issue with the new extract method.

Specifically, the performance of the extract call becomes painfully slow if the startPosition parameter is not -1 (the default). On my Mac Book Pro, I'm seeing about 8ms per call to extract 2048 samples from a sound object, compared to about 1ms when the startPosition parameter is absent or is set to the default value.

Unfortunately, this is so poor that if I attempt to call the extract method with a startPosition value inside of a SampleDataEvent handler, the sample buffer will empty completely and result in an audible click.

I've filed this as bug #FP-518 on JIRA with a simplified test case if you'd like to take a look.

Monday, August 04, 2008 5:07:00 PM  
Blogger joshspoon said...

I posted a dynamic piano with the new SampleDataEvent: http://tinyurl.com/65bzmx

Friday, September 05, 2008 7:32:00 AM  
Blogger Miguel said...

Is there hope that flash 10 will play the following wav format one day?
CCITT A-Law, CCITT u-Law, DSP Group TrueSpeech(TM), elemediaTM AX2400P music codec, IMA ADPCM, Microsoft ADPCM, MSN Audio, and GSM 6.10.
Ideally, I would love to be able to just read in audio stream like in java and play any of those.

Thursday, November 13, 2008 8:56:00 PM  
Anonymous DJ said...

Tinic,

Is it possible to display video loaded thru FileReference.load()? I got it to work for images using Loader.loadBytes(), but what about video?

Thanks.

Monday, November 17, 2008 4:20:00 PM  
Anonymous nyca said...

This is good news. Thanks Flash Player team!

First BC, now this.. great!!

Monday, March 08, 2010 10:51:00 PM  

Post a Comment

<< Home