Wednesday, October 31, 2007

New File Extensions and MIME Types

There have been a few questions around file types and mime types which should be used for the new video container format. To summarize this post, we will promote new file extensions and will stick with standard MIME types.

The new file extensions and MIME types will be the following:

File ExtensionFTYPMIME TypeDescription
.f4v'F4V 'video/mp4Video for Adobe Flash Player
.f4p'F4P 'video/mp4Protected Media for Adobe Flash Player
.f4a'F4A 'audio/mp4Audio for Adobe Flash Player
.f4b'F4B 'audio/mp4Audio Book for Adobe Flash Player

These are pretty much in sync with what Apple does offer for their downloads. Why new file extensions? It will be an easy way to distinguish files which can be played back by the Flash Player. There are simply too many instances where .mov and .mp4 files can not be played back, or vise versa, a file compatible with the Flash Player might not play back in QuickTime, an iPod or other video device. I will also be working on a technical document to exactly outline what the Flash Player does support and what it does not support although my previous posts already pointed out that we are very close to full support of the H.264 standard with the exception of Extended profile and FRExt.

Why didn't we stick with .FLV? Technical reasons, mostly. Various technologies, including our own products expect that .FLV files have a certain file structure. In case of Flash CS3 a file which was renamed from .MP4 to .FLV would stall the application at import time. Not a good experience. In case of FMS file type plug ins can only handle unique file extensions. In this context it made it impossible to have files with the new container format using the .FLV file extension. We expect other tools to have similar issues.

It's not yet clear when our video tools will start using these extensions, although I expect it to happen sooner than later. The first one to be used will clearly be .f4v, the other ones are in 'reserved' state for now and not destined for a particular product right now. Why do I talk about this now in this case? Well, as we ready the Flash Player for release we did add new icons and file extension registration of these file types into the standalone Flash Player. So it's better to make this announcement now as you will discover this on your own very soon anyway.

If you ask what the FTYP column is, it is the ftyp atom within ISO 14496-12 files. If you have a custom encoder and are targeting the Flash Player you should add one of the ftyp major brand 4CCs mentioned above. This will make it much easier for servers to handle and recognize these files. And to prevent you from panicking now: the Flash Player will not even look at this atom. This atom is nothing more than a hint for tools handling ISO14496-12 files. A list of known ftyp major brand 4CCs can be found here. We'll hopefully make this list also at some point.

It might be a good time to update your IIS mime-type entries to include the new file extensions, otherwise IIS will refuse to play serve up these files. This is the same process as for flv files described in Technote 19439, but using the file extensions and mime types mentioned above.

Update, since I am apparently not clear enough and have to repeat myself: the Flash Player will not look at the file extension when loading files. It just means that if you are targeting the Flash Player or AIR we suggest to use these file extensions.

Thursday, October 04, 2007

The obligatory post on Hydra

Now that MAX 2007 in Chicago is over and the Flash Player team is somewhat settling for the final landing of Flash Player 9 Update 3 I've got to talk a little about Adobe Image Foundation, code named Hydra.

I have not been involved in this project as much as I would have liked to, with Bob Archer and Kevin Goldsmith driving most of the specifications and communication so far, nevertheless some technical questions in the context of Flash have been asked I should be able to answer. But let me make this post a little more approachable and explain the basics.

What is Hydra? If I would have to explain it in one sentence I would say: It is a shader language specifically tuned for 2D pixel graphics.

What is a shader language? Most of you will not know the answer to this. Let me try to put it into terms you might understand, with the assumption that you already have played with BitmapData objects in ActionScript 2.

Scenario: You have a bitmap and you want to exchange the red channel with the green channel. Easy enough, in Flash 8 or newer you can actually use the ColorMatrixFilter to do this. Lets say you do not have that filter, how would you do it? Well, this is what I would write in ActionScript 2:
import flash.display.*;

// create a green bitmap
var bitmap:BitmapData = new BitmapData(550,440,false,0x00FF00);

// go through the whole bitmap pixel by pixel
for(var y:Number=0; y<bitmap.height; y++) {
for(var x:Number=0; x<bitmap.width; x++) {

// get a single pixel from our bitmap
var pixel:Number = bitmap.getPixel(x, y);

var red:Number = (pixel>>16)&0xFF; // extract red
var green:Number = (pixel>> 8)&0xFF; // extract green
var blue:Number = (pixel )&0xFF; // extract blue

// create a new pixel with the red and green channels flipped
var newpixel:Number = (green<<16)|
(red << 8)|
(blue );

// replace the ol' one with the new one
bitmap.setPixel(x, y, newpixel);
// now the bitmap is pure red

// show it
bitmap_mc.attachBitmap(bitmap, 2, "auto", true);
That sure looks ugly and slow. Now lets express the same as above using Hydra, and believe me, it will do exactly the same as the above AS2 code:

kernel FlipRedGreen
void evaluatePixel(in image3 source, out pixel3 result)
result.grb = sample(source, outCoord());
There are several things you will notice:

  • There are no for loops to go over the pixels. That's right, shader languages only express what it inside the loop and assume that this is the operation you want to perform on every pixel.

  • There are a bunch of strange types here! Not really: image3 stands for a bitmap with RGB data and pixel3 stands for an individual RGB pixel. If you would want to include alpha you could say image4 and pixel4.

  • What is sample()? It is the same as BitmapData.getPixel() and returns a pixel, in this case a pixel3 type.

  • What is outCoord()? It is the same as the x and y positions in the AS2 code but it returns a single value containing x and y without the need to specify them seperately.

  • How does the actual flip work? Now that is some of the beauty which comes with using a shader language. You can perform all kinds of operations directly with the component identifiers.

    In this case we say 'result.grb =' which will flip the red and green channel. I could write 'result =' or 'result.rgb =' to just copy the pixel without a change. Or I could write any other combination like .bgr, .gbr to flip the different components.

    If you really want to get fancy you can also write something like this:
    'result = sample(source, outCoord()).rrr;' or
    'result = sample(source, outCoord()).bbr;'.
    In this case we make the component choice on the other side of the assigment.

    In essence, no more bit fiddling anymore, you have a high level access to the actual component data.

This is one of the most simple examples I can think of, Hydra is much more powerful and expressive than that. So I invite you to take a look and get comfortable with it, this is something which will stay with us for a long time to come. And possibly not only for graphics, but let me talk about that in a future post. :-) I also promise that I will have another post specifically talking about the technical aspects of the Flash Player implementation. I can think of those and more:

  • How will I be able to access this in my ActionScript code?

  • How much faster will it really be than ActionScript code?

  • Where will I be able to use Hydra? Just filters and blend modes, or more?

  • Will Flash Authoring support Hydra?

  • Will the Flash Player take advantage of hardware acceleration?

  • Will this work in the Flash Player even if there is no hardware acceleration available?

I know you are clamoring for answers, though for most questions we have not figured them out ourselves just yet.

There is obviously one more question: "Instead of working on new features, when do you get your act together and ship a 64-bit version of the player?" :-) I always have a good laugh when we get this one, especially since it has been answered many times already.

Monday, October 01, 2007

Getting ready for showtime

MAX is now underway and to celebrate this we just released another prerelease of Flash Player 9 Update 3 code named 'Moviestar', the version is have been numerous changes and fixes since the last beta, especially in the area of MP4, H.264 and AAC support. I figured I should summarize the bugs we fixed. In this area only as I was not able to keep track of the other stuff. As a warning, this is highly technical stuff, but I know that some wanted this information.

  • A frame ordering issue when using more than 1 reference frame has been fixed. There should be no limit anymore on how many reference frames can be used.

  • One multi core systems H.264 streams using the loop filter showed artifacts along the slice boundaries. This has been fixed by switching from sliced based multi threaded decoding to picture based multi threaded decoding.

  • Overlaying flash content would sometimes cause black artifacts during a seek because the underlying YUV buffer was invalid.

  • AAC streams would sometimes not play back if it contained SBR information. This usually happened when switching between SBR and non-SBR streams.

  • The H.264 codec would crash on octo (8) core machines.

  • AAC streams with sample rates higher than 44.1Khz could crash the Flash Player.

  • Corrupt or fuzzed AAC streams could crash the Flash Player.

  • The H.264 codec now supports FMO decoding, PicAFF and MBAFF decoding should be improved.

  • H.264 decoding should generally be about 10-15% faster because of the removal of a full frame copy. Note that this is only the case if a square pixel ratio is used, the video stream is not interlaced or cropped on the top, left or right.

  • The PPC version of the H.264 codec would call kill() during certain situations which resulted in a user experience similar to a crash. The reason behind this was the incorrect behavior of assert() in release builds under CodeWarrior.

  • The H.264 codec now has AltiVec versions for all the functions which are accelerated by MMX and SSE2 SIMD instructions on x86. Note that due to the extremely low available memory bandwidth of G4 and G5 PowerPC processors the overall performance will never match x86 CPUs. Refer to our system requirements page which should be up once the final version of MovieStar is released.

  • The mp4 parser will now parse variable chunk sizes correctly. Usually this bug manifested itself in bad or accelerated AAC audio playback.

  • The mp4 parser should now be much more robust when encountering corrupt or fuzzed files. Two new NetStream events have been added to capture invalid or unsupported files: NetStream.Play.FileStructureInvalid and NetStream.Play.NoSupportedTrackFound.

  • The H.264 codec could crash when trying to decode unsupported profiles like Extended, High 4:2:2 and High 4:4:4.

  • Past onImageData and onTextData events are now resent when seeking. This f.ex. allows previewing of cover art when scrubbing pod casts and will assure you will get the current timed text for any particular time. Note that also means that you will receive a lot of these events during a seek.

  • supported AAC aot types are now correctly parsed and validated, meaning we allow aot types 1, 2 and 5.

  • Some encoders (like Handbrake) embed aspect ratio information into the matrix of the track header box atom, which is inconsistent with ISO 14496-12. Despite this the Flash Player will now apply this matrix on the reported width and height in the onMetaData object. This does not affect the actual video stream size which can be different from what is reported in onMetaData and in this case usually points to a bug in the encoder.

  • Artifacts on the right and the bottom when mixing cropping and aspect ratios in H.264 streams should be gone.

  • Along with times, the seek points array will now also contain a byte offset which allows to determine if a particular seek point is reachable during a progressive download.

What is still outstanding before the final release?

  • There is an injection concerning scrubbing H.264 files. Currently no images are shown during scrubbing. You have to resume playback before the video is updated. There is a one line fix I have which did not make it in time for the build.

  • When playing back video using FMS you can end up in a situation where you get 1 black frame when seeking. Another minimal low risk change will address this for the final build.

  • AAC streams using SBR do not play back in certain situations if the original sample rate is higher than 24Khz.
What are the known issues we will not address?
  • We still do not support Extended, High 4:2:2 or High 4:4:4 profiles. Unfortunately the most commonly non-professional codec out there which is x264 does not adhere to standard terminologies when specifying encoding options. This means you will likely run into a case were you encode using options which are part of the profiles we do not support. The most common option you will want to use is 4:4:4 support to obtain screen casts without YUV artifacts. This is not a bug, although I am sure we will continue to get reports on that. I strongly suggest you stick with Adobe's AME encoder which uses standardized terminologies defined in the H.264 specifications.

  • On the more controversial side we decided to postpone support for H.264 color profiles as the risk of adding tons of new SIMD code was too high at this point in time. Like MPEG-2 default H.264 streams use the color profile standardized under ITU-R BT.709 and in addition allows the selection of 6 more less frequently used color profiles. The streams will be displayed under the assumption that the content is ITU-R BT.601 meaning it will be a little brighter and slightly color shifted if the encoder did use ITU-R BT.709. Video buffs will frown upon this, but most end users will probably not notice it. We'll try to address this in one of the next versions of the Flash Players. I realize that it is important. For designers this also means that color matching with other Flash content will be more tricky for now.

  • A more sophisticated way of displaying field coded streams which we had prototyped but were not able to fully test will not make it into the final release. So field coded streams will simply be frame blended.

  • Corrupt AAC or H.264 streams or audio dropouts caused by the underlying OS can cause bad audio/video sync. This is due to an architectural limitation of the Flash Player which we will try to remedy for the next major version of the Flash Player, code named Astro.