Adobe Open Screen Project

Unless you have been living under a rock you probably saw the announcement Adobe made this week:

http://www.adobe.com/openscreenproject/

This is a significant move by Adobe to bring the Adobe® Flash® Player to a much broader audience than ever before. By doing this we are responding to the requests from customers throughout the different industries which want to leverage the Flash platform.

So, yes I haven't been blogging in a while, that has been due to my busy schedule (I'll be more specific soon). But since I've worked on fixing up the FLV and SWF specs a little bit for this project I figured I should add my own personal thoughts.

As the site above mentions this project essentially consists of the following changes:

• Removing restrictions on use of the SWF and FLV/F4V specifications
• Publishing the device porting layer APIs for Adobe Flash Player
• Publishing the Adobe Flash® Cast™ protocol and the AMF protocol for robust data services
• Removing licensing fees – making next major releases of Adobe Flash Player and Adobe AIR for devices free

All of these are very significant for a lot of customers although individual parties might think that it is one particular item which matters most to them. I assure you that all of them are extremely important.

Bringing a platform like the Adobe Flash Player to devices has its challenges. One of them revolves around documentation. By making the file format and API documentation open, the entry barrier for a lot of developers becomes much lower. In the past it could have been a struggle to get the necessary documentation to the teams which need them. This problem is now gone.

In many circumstances technologies Adobe provided were not compatible with how a particular device or infrastructure was set up. Given the old licensing restrictions it could prove difficult to find technical solutions. By making the specifications open and without restrictions a particular vendor might instead choose a clean rewrite of some parts if they see fit with the result that it integrates better into their eco-system.

Now there have been questions if there are any gotchas in what Adobe has announced concerning the SWF and FLV specifications. There IS in fact a license, it is on page two of the specifications. You might be shocked however by how small it is. It's essentially BSD license style damages disclaimer plus some additional information about trademarks and such. Read it before you use it.

The specifications are available here as PDF files:

http://www.adobe.com/devnet/swf/
http://www.adobe.com/devnet/flv/

Does that mean you can implement your own clone which implements what is in these specifications without fear of Adobe coming after you? Yes. Notice that I am careful how I ask this question though. Adobe still holds all the trademarks around this technology, you can not use any of those in your clone or anything related to it. Unless we give the OK to do so obviously.

If you find inaccuracies or missing information in these specifications let us know, we can integrate the feedback right away so they can appear in the next revision of these documents.

Adobe Flash Player 9 Update 3 released

We made it! We just shipped Adobe Flash Player 9 Update 3, the final version number is 9.0.115.0. As usual a number of different flavors of the web plugin are available, the main install page is here. And yes, we also shipped the Linux x86-32 version. We have done this for several versions now and we will not stop. The only exception is the Solaris version which is still in testing. I can't tell you though when a new player for Flash CS3 will be available since I do not know.

This version is a significant milestone, and I expect it will be adopted fairly quickly once content using H.264 and AAC starts appearing. Emmy Huang has some additional information and links you should look at.

Here is a condensed list of bugs which have been fixed since the last release candidate (9.0.64.0). As you can see these have been mostly fixes for crashers and backwards compatibility issues. It's a pretty long list which explains why 2 months have passed since the last release candidate. Maybe you'll spot a few you were affected by:

• 212379 audio playback in swf off sync.
• 212294 on (release) events not firing on sprites
• 212224 Heavy FMS Seeking in Vista can cause bad audio and crash
• 211921 Crash when paging through a Flex app and closing the player
• 211898 FileReference IO errors when uploading multiple files
• 211840 playlist does not advance after end of commercial
• 211836 [MP4] If you seek in the NetStream before a movie that has the moov atom at the end has completely downloaded, you can crash.
• 211813 BitmapData draw ColorTransform not affecting all text on all platforms
• 211759 inputted simplified Chinese characters are not correct inside of Flash Player
• 211727 Japanese/RHEL4/Firefox 1.0 -- hang when selecting print dialog
• 211725 Player crashes when Premiere Express is connected to a server that times out
• 211700 Add support for Opera XEmbed
• 211672 the pop-up print panel layout is incorrect when R-click on the red button and choose ‘print…’
• 211625 Flash player crashes while playing servierside playlists containing MP4 files
• 211593 Crash in NavigateToURL when using local relative URL
• 211584 can't input anything to flashplayer for Simplified Chinese Linux OS
• 211582 New AS3 NetStream.receiveVideoFps() should be NetStream.receiveVideoFPS()
• 211558 Flash Player movies soft-freeze
• 211504 getURL loading with UTF8 Japanese text data crashes IE6 on WinXP J
• 211498 crash if flash unloads while in a JS function called by ExternalInterface
• 211493 Audio locks up and starts looping on Linux, requiring the page with the player to be shutdown
• 211491 AS3/2 keyDown event (Code Keys) not firing for NumPad on Linux
• 211485 Flash previewer hangs the browser
• 211403 Found an audio sync issue after a pause and seek
• 211367 Memory leak when subscribing live stream with buffer time 0 published by Flash Media Encoder 2 in MP3 audio format
• 211362 One more time: An A/V Sync issue
• 211346 extra lines appear when a radial gradient is on top of a radial gradient
• 211343 Video does not resume after pause and seek
• 211298 Accessibility Plugin reads text from dynamic text field after tf.text has been set to ""
• 211264 Fms seeking while paused won't always generate an image
• 211263 FMS Video streams when seeking can go out of A/V sync
• 211186 H264 files : Loss of audio sync during the playback after seek
• 211171 site no longer automatically advances to next song
• 211120 new file icons for F4V, F4P, and F4A for the standalone player to register
• 211070 Intermittent (8/10) Crash on site
• 211067 NetStream.Seek.InvalidTime NetStatus event coming in different order from other events, breaking legacy content
• 211051 File Upload broken on leopard. FileReference throws cancel event when a file is selected for upload in the file-browsing dialog
• 211017 Scrubbing streaming flv doesn't work (possible injection)
• 210983 Incorrect sprite size and position on several platforms
• 210962 Player crashes when finish streaming a mp4 movie.
• 210957 Bitmap.draw() producing erratic behavior when subsequently drawn in bitmapfill
• 210915 Blur filter behavior uses black on edges instead of nearest pixel
• 210909 Fullscreen not painting over screen
• 210907 perform execstack and strip commands on executables
• 210901 [MP4] Handbrake videos made with certain settings play but halt batch at end of playback
• 210871 Smart Buffering: Few issues when streaming on2 files from Flash Media Server 2.0.4.
• 210854 scrolling images show stuttered or jumpy or jerky movement on playback in Vista
• 210846 [MP4] Errors in the 'ilst' parser
• 210844 Audio stutters
• 210809 widget hangs the browser
• 210808 Player crashes by scrubbing the playhead when streaming a mp4 file under rtmps, rtmpt and rtmpe connections.
• 210780 All the strings for the linux context menu leak [Memory Leak]
• 210769 Repeated use of shared objects in an event handler causes slow script dialog and hosed Settings UI to appear
• 210767 [LNX RHEL4] Crashes interacting with site
• 210746 When a release swf loads a debug swf, flash player doesn't look for the debugger
• 210720 Loader acts strangely in FullScreen mode
• 210717 Keyboard non functioning with the Netscape
• 210716 full screen does not respond to mouse events or display context menu
• 210715 Selecting printer dropdown cause hang with plugin on linux
• 210679 Certain videos look bad when played on Mac
• 210667 Context menu in full screen does not display, intermittently hangs
• 210645 streaming checks may unnecessarily reject valid socket policy files
• 210642 -root option sends sometimes a gtk warning, and sometimes a segfault
• 210641 Null Objects when iterating through children
• 210636 non-default locations not rejected for missing Content-Type
• 210593 Cannot open Settings manager page after clicking advance settings.
• 210574 Full episode videos don't stay in FullScreen Mode in Firefox
• 210573 Smart buffering: Artifacts with episodes
• 210562 dragging performs much poorer than 9r48
• 210561 Smart Buffering: NetStream.Play does not function after NetStream.Pause is called in this app.
• 210545 Netstream.receiveVideo parameter change is breaking existing content
• 210538 [MP4] Pauses due to buffer running out cause loss of sound sync
• 210532 Animation is skipped, causing audio and swf playback off sync.
• 210501 Enhanced Seeking driving audio/video out of sync on lower bandwidth like DSL
• 210485 Pause/Unpause on MP3 Files doesn't play from the paused location when setBufferTime is used.
• 210476 extraneous reload of content after close of sub-pane
• 210451 Player Crashing while playing a video through a serverside stream
• 210434 Player crashes when streaming this bad flv under an AS3 streaming application.
• 210433 massive performance degradation after being in fullscreen for a long time
• 210380 Refreshing the browser when AS3 content is loaded will cause the browser to crash.
• 210335 Flash content does not display
• 210333 Can't input local words with IME on linux
• 210331 [MP4] Lots of screen artifacts on certain trailers. PPC Only
• 210292 fscommand ("fullscreen", "false") crashes the Projector on Linux
• 210245 Can't input any words with IME
• 210209 [MP4]: When unpausing after seek while paused, video is black for one frame
• 210170 [MP4] AVC Profile 110 videos all appear purple on MacIntel machines
• 210097 Multiple SWFs hang page
• 210039 Leaking file descriptors to Flash resources
• 209833 Intermittent Flash Player 9.x crashes in Internet Explorer 6 and 7
• 209513 Changing IME conversion mode does not work on Mac X.5
• 208369 Consecutive shared object flush calls causes slow script error when used repeatedly in non-mouse/keyboard events
• 208212 Crash when loading external swf file in AS3
• 208125 page with 9 Flash video instances generates "R6025 - pure virtual function call" and crashes IE
• 208039 Flash plugin logs keyboard in Safari on Mac OS X
• 207349 Linux stalls on random test cases
• 206369 ByteArray.readMultiByte returns garbled characters from a binary file containing Win-1252 encoded characters.
• 147787 removeMovieClip should not fail for objects > kDisplayClonedEnd in depth

New File Extensions and MIME Types

There have been a few questions around file types and mime types which should be used for the new video container format. To summarize this post, we will promote new file extensions and will stick with standard MIME types.

The new file extensions and MIME types will be the following:

File Extension	FTYP	MIME Type	Description
.f4v	'F4V '	video/mp4	Video for Adobe Flash Player
.f4p	'F4P '	video/mp4	Protected Media for Adobe Flash Player
.f4a	'F4A '	audio/mp4	Audio for Adobe Flash Player
.f4b	'F4B '	audio/mp4	Audio Book for Adobe Flash Player

These are pretty much in sync with what Apple does offer for their downloads. Why new file extensions? It will be an easy way to distinguish files which can be played back by the Flash Player. There are simply too many instances where .mov and .mp4 files can not be played back, or vise versa, a file compatible with the Flash Player might not play back in QuickTime, an iPod or other video device. I will also be working on a technical document to exactly outline what the Flash Player does support and what it does not support although my previous posts already pointed out that we are very close to full support of the H.264 standard with the exception of Extended profile and FRExt.

Why didn't we stick with .FLV? Technical reasons, mostly. Various technologies, including our own products expect that .FLV files have a certain file structure. In case of Flash CS3 a file which was renamed from .MP4 to .FLV would stall the application at import time. Not a good experience. In case of FMS file type plug ins can only handle unique file extensions. In this context it made it impossible to have files with the new container format using the .FLV file extension. We expect other tools to have similar issues.

It's not yet clear when our video tools will start using these extensions, although I expect it to happen sooner than later. The first one to be used will clearly be .f4v, the other ones are in 'reserved' state for now and not destined for a particular product right now. Why do I talk about this now in this case? Well, as we ready the Flash Player for release we did add new icons and file extension registration of these file types into the standalone Flash Player. So it's better to make this announcement now as you will discover this on your own very soon anyway.

If you ask what the FTYP column is, it is the ftyp atom within ISO 14496-12 files. If you have a custom encoder and are targeting the Flash Player you should add one of the ftyp major brand 4CCs mentioned above. This will make it much easier for servers to handle and recognize these files. And to prevent you from panicking now: the Flash Player will not even look at this atom. This atom is nothing more than a hint for tools handling ISO14496-12 files. A list of known ftyp major brand 4CCs can be found here. We'll hopefully make this list also at some point.

It might be a good time to update your IIS mime-type entries to include the new file extensions, otherwise IIS will refuse to play serve up these files. This is the same process as for flv files described in Technote 19439, but using the file extensions and mime types mentioned above.

Update, since I am apparently not clear enough and have to repeat myself: the Flash Player will not look at the file extension when loading files. It just means that if you are targeting the Flash Player or AIR we suggest to use these file extensions.

The obligatory post on Hydra

Now that MAX 2007 in Chicago is over and the Flash Player team is somewhat settling for the final landing of Flash Player 9 Update 3 I've got to talk a little about Adobe Image Foundation, code named Hydra.

I have not been involved in this project as much as I would have liked to, with Bob Archer and Kevin Goldsmith driving most of the specifications and communication so far, nevertheless some technical questions in the context of Flash have been asked I should be able to answer. But let me make this post a little more approachable and explain the basics.

What is Hydra? If I would have to explain it in one sentence I would say: It is a shader language specifically tuned for 2D pixel graphics.

What is a shader language? Most of you will not know the answer to this. Let me try to put it into terms you might understand, with the assumption that you already have played with BitmapData objects in ActionScript 2.

Scenario: You have a bitmap and you want to exchange the red channel with the green channel. Easy enough, in Flash 8 or newer you can actually use the ColorMatrixFilter to do this. Lets say you do not have that filter, how would you do it? Well, this is what I would write in ActionScript 2:

import flash.display.*;

// create a green bitmap
var bitmap:BitmapData = new BitmapData(550,440,false,0x00FF00); 

// go through the whole bitmap pixel by pixel
for(var y:Number=0; y<bitmap.height; y++) {
    for(var x:Number=0; x<bitmap.width; x++) {

        // get a single pixel from our bitmap
        var pixel:Number = bitmap.getPixel(x, y);

        var red:Number   = (pixel>>16)&0xFF; // extract red
        var green:Number = (pixel>> 8)&0xFF; // extract green
        var blue:Number  = (pixel    )&0xFF; // extract blue

        // create a new pixel with the red and green channels flipped
        var newpixel:Number = (green<<16)|
                              (red  << 8)|
                              (blue     );

        // replace the ol' one with the new one
        bitmap.setPixel(x, y, newpixel);
    }
}
// now the bitmap is pure red

// show it
this.createEmptyMovieClip("bitmap_mc",this.getNextHighestDepth());
bitmap_mc.attachBitmap(bitmap, 2, "auto", true);

That sure looks ugly and slow. Now lets express the same as above using Hydra, and believe me, it will do exactly the same as the above AS2 code:

kernel FlipRedGreen
{
    void evaluatePixel(in image3 source, out pixel3 result)
    {
        result.grb = sample(source, outCoord());
    }
}

There are several things you will notice:

• There are no for loops to go over the pixels. That's right, shader languages only express what it inside the loop and assume that this is the operation you want to perform on every pixel.


• There are a bunch of strange types here! Not really: image3 stands for a bitmap with RGB data and pixel3 stands for an individual RGB pixel. If you would want to include alpha you could say image4 and pixel4.


• What is sample()? It is the same as BitmapData.getPixel() and returns a pixel, in this case a pixel3 type.


• What is outCoord()? It is the same as the x and y positions in the AS2 code but it returns a single value containing x and y without the need to specify them seperately.


• How does the actual flip work? Now that is some of the beauty which comes with using a shader language. You can perform all kinds of operations directly with the component identifiers.
  
  In this case we say 'result.grb =' which will flip the red and green channel. I could write 'result =' or 'result.rgb =' to just copy the pixel without a change. Or I could write any other combination like .bgr, .gbr to flip the different components.
  
  If you really want to get fancy you can also write something like this:
  'result = sample(source, outCoord()).rrr;' or
  'result = sample(source, outCoord()).bbr;'.
  In this case we make the component choice on the other side of the assigment.
  
  In essence, no more bit fiddling anymore, you have a high level access to the actual component data.

This is one of the most simple examples I can think of, Hydra is much more powerful and expressive than that. So I invite you to take a look and get comfortable with it, this is something which will stay with us for a long time to come. And possibly not only for graphics, but let me talk about that in a future post. :-) I also promise that I will have another post specifically talking about the technical aspects of the Flash Player implementation. I can think of those and more:

• How will I be able to access this in my ActionScript code?


• How much faster will it really be than ActionScript code?


• Where will I be able to use Hydra? Just filters and blend modes, or more?


• Will Flash Authoring support Hydra?


• Will the Flash Player take advantage of hardware acceleration?


• Will this work in the Flash Player even if there is no hardware acceleration available?

I know you are clamoring for answers, though for most questions we have not figured them out ourselves just yet.

There is obviously one more question: "Instead of working on new features, when do you get your act together and ship a 64-bit version of the player?" :-) I always have a good laugh when we get this one, especially since it has been answered many times already.

Getting ready for showtime

MAX is now underway and to celebrate this we just released another prerelease of Flash Player 9 Update 3 code named 'Moviestar', the version is 9.0.64.0.There have been numerous changes and fixes since the last beta, especially in the area of MP4, H.264 and AAC support. I figured I should summarize the bugs we fixed. In this area only as I was not able to keep track of the other stuff. As a warning, this is highly technical stuff, but I know that some wanted this information.

• A frame ordering issue when using more than 1 reference frame has been fixed. There should be no limit anymore on how many reference frames can be used.


• One multi core systems H.264 streams using the loop filter showed artifacts along the slice boundaries. This has been fixed by switching from sliced based multi threaded decoding to picture based multi threaded decoding.


• Overlaying flash content would sometimes cause black artifacts during a seek because the underlying YUV buffer was invalid.


• AAC streams would sometimes not play back if it contained SBR information. This usually happened when switching between SBR and non-SBR streams.


• The H.264 codec would crash on octo (8) core machines.


• AAC streams with sample rates higher than 44.1Khz could crash the Flash Player.


• Corrupt or fuzzed AAC streams could crash the Flash Player.


• The H.264 codec now supports FMO decoding, PicAFF and MBAFF decoding should be improved.


• H.264 decoding should generally be about 10-15% faster because of the removal of a full frame copy. Note that this is only the case if a square pixel ratio is used, the video stream is not interlaced or cropped on the top, left or right.


• The PPC version of the H.264 codec would call kill() during certain situations which resulted in a user experience similar to a crash. The reason behind this was the incorrect behavior of assert() in release builds under CodeWarrior.


• The H.264 codec now has AltiVec versions for all the functions which are accelerated by MMX and SSE2 SIMD instructions on x86. Note that due to the extremely low available memory bandwidth of G4 and G5 PowerPC processors the overall performance will never match x86 CPUs. Refer to our system requirements page which should be up once the final version of MovieStar is released.


• The mp4 parser will now parse variable chunk sizes correctly. Usually this bug manifested itself in bad or accelerated AAC audio playback.


• The mp4 parser should now be much more robust when encountering corrupt or fuzzed files. Two new NetStream events have been added to capture invalid or unsupported files: NetStream.Play.FileStructureInvalid and NetStream.Play.NoSupportedTrackFound.


• The H.264 codec could crash when trying to decode unsupported profiles like Extended, High 4:2:2 and High 4:4:4.


• Past onImageData and onTextData events are now resent when seeking. This f.ex. allows previewing of cover art when scrubbing pod casts and will assure you will get the current timed text for any particular time. Note that also means that you will receive a lot of these events during a seek.


• supported AAC aot types are now correctly parsed and validated, meaning we allow aot types 1, 2 and 5.


• Some encoders (like Handbrake) embed aspect ratio information into the matrix of the track header box atom, which is inconsistent with ISO 14496-12. Despite this the Flash Player will now apply this matrix on the reported width and height in the onMetaData object. This does not affect the actual video stream size which can be different from what is reported in onMetaData and in this case usually points to a bug in the encoder.


• Artifacts on the right and the bottom when mixing cropping and aspect ratios in H.264 streams should be gone.


• Along with times, the seek points array will now also contain a byte offset which allows to determine if a particular seek point is reachable during a progressive download.

What is still outstanding before the final release?

• There is an injection concerning scrubbing H.264 files. Currently no images are shown during scrubbing. You have to resume playback before the video is updated. There is a one line fix I have which did not make it in time for the 9.0.64.0 build.


• When playing back video using FMS you can end up in a situation where you get 1 black frame when seeking. Another minimal low risk change will address this for the final build.


• AAC streams using SBR do not play back in certain situations if the original sample rate is higher than 24Khz.

What are the known issues we will not address?

• We still do not support Extended, High 4:2:2 or High 4:4:4 profiles. Unfortunately the most commonly non-professional codec out there which is x264 does not adhere to standard terminologies when specifying encoding options. This means you will likely run into a case were you encode using options which are part of the profiles we do not support. The most common option you will want to use is 4:4:4 support to obtain screen casts without YUV artifacts. This is not a bug, although I am sure we will continue to get reports on that. I strongly suggest you stick with Adobe's AME encoder which uses standardized terminologies defined in the H.264 specifications.


• On the more controversial side we decided to postpone support for H.264 color profiles as the risk of adding tons of new SIMD code was too high at this point in time. Like MPEG-2 default H.264 streams use the color profile standardized under ITU-R BT.709 and in addition allows the selection of 6 more less frequently used color profiles. The streams will be displayed under the assumption that the content is ITU-R BT.601 meaning it will be a little brighter and slightly color shifted if the encoder did use ITU-R BT.709. Video buffs will frown upon this, but most end users will probably not notice it. We'll try to address this in one of the next versions of the Flash Players. I realize that it is important. For designers this also means that color matching with other Flash content will be more tricky for now.


• A more sophisticated way of displaying field coded streams which we had prototyped but were not able to fully test will not make it into the final release. So field coded streams will simply be frame blended.


• Corrupt AAC or H.264 streams or audio dropouts caused by the underlying OS can cause bad audio/video sync. This is due to an architectural limitation of the Flash Player which we will try to remedy for the next major version of the Flash Player, code named Astro.

What just happened to video on the web?

That's a question you should ask with the announcement we made tonight. I think a lot will change. This is probably one of my longest and information packed posts ever, but I think it is important we put down all cards on the table. Lets summarize what new functionality Flash Player 9 Update 3 Beta 2 contains (for the impatient: It will be available on labs.adobe.com this afternoon):

• An file format parser implementing parts of ISO 14496-12. In terms you might understand this means a very limited sub set of MPEG-4, 3GP and QuickTime movie support.


• Support for the 3GPP timed text specification 3GPP TS 26.245. Essentially this is a standardized subtitle format within 3GP files.


• Partial parsing support for the 'ilst' atom which is the ID3 equivalent iTunes uses to store meta data. This really more a de-facto standard which came through the ubiquity of iTunes, there is no official documentation on the format. Look here for an incomplete list of supported tags iTunes does use.


• A software based H.264 codec with the ability to decode Base, Mainline and High profiles. This is also an ISO standard with the identifier being ISO 14496-10.


• An AAC decoder supporting AAC Main, AAC LC and SBR (also known as HE-AAC). The corresponding ISO specification is ISO 14496-3.

That's pretty much what we say publicly. Truth is that these specifications are so complex that no one supports 100% of it. I realize that it will be important for Adobe to communicate exactly what is and what is not supported. We are working on this and will be trying to help novices and experts alike. For those who scream murder and accuse us of going with incomplete standards support let me tell you that ISO 14496-12 specifically allows for the definition of sub sets. 3GP is one of those. We did not extend or add proprietary extensions whatsoever to the mentioned standards above, it is a pure sub set.

Why now? Short answer: Because you wanted it. Long answer: We've been working on this for a while and this was planned to be part of the next major revision of the Flash Player. What was unexpected was how impatient a lot of our customers are :-) It seems many are trying to make choices when it comes to video technologies right now. We wanted to make sure that we would offer the best possible choices to them and set a signal that we are willing to embrace industry standards. No one believed that we would make this happen.

Unfortunately, and we realized while working on this: along with adopting industry standards also comes completely new terminology which seems designed to confuse non-insiders. This makes it difficult to pin down exactly what it is what we did and how you might benefit from it. It took me several months to just understand the basics in the ISO specifications. By now I might have lost the ability to boil it down into simple terms everyone can understand. But I'll try anyway. :-)

Lets talk about actual functionality you can leverage in the Flash Player. Now I am getting really technical:

• You can load and play .mp4,.m4v,.m4a,.mov and .3gp files using the same NetStream API you use to load FLV files now. We did not add any sort of new API in the Flash Player. All your existing video playback front ends will work as they are. As long as they do not look at the file extension that is, though renaming the files to use the .flv file extension might help your component. The Flash Player itself does not care about file extensions, you can feed it .txt files for all it matters. The Flash Player always looks inside the file to determine what type of file it is.


• A new version of FMS is upcoming and will support the new file format. This is powerful stuff. Simply drop video files you might have encoded using one of the countless tools out there onto the server and it'll stream. Even if the moov atom is at the end of the file. Ah, that is something I have to mention as you are 100% likely to fall into this trap:


• If you use progressive download instead of FMS make sure that the moov atom (which is the index information in MPEG-4 files) is at the beginning of the file. Otherwise you have to wait until the file is completely downloaded before it is played back. You can use tools like qt-faststart.c written by our own Mike Melanson to fix your files so that the index is at the beginning of the file. Unfortunately our tools (Premiere and AfterEffects etc.) currently place the index at the end of the file so this tool might become essential for you, at least for now. We are working hard to fix this in our video tools. There is nothing we can do in the Flash Player and iTunes/QuickTime does behave the same way.


• The Flash Player will display the first supported video and audio track it finds in a file. Subsequent audio and video tracks are ignored and not selectable right now. This covers the majority of files out there on the web, only in rare instances do you have additional audio tracks f.ex. But I believe that for the web you would rather create several versions of a file anyway to save bandwidth. One of next major revisions of the Flash Player will add new APIs to enhance this most likely. Our goal was not to add any new APIs for this release.


• Video needs to be in H.264 format only. MPEG-4 Part 2 (Xvid, DivX etc.) video is not supported, H.263 video is not supported, Sorenson Video is not supported. Keep in mind that a lot of pod casts are still using MPEG-4 Part 2. So do not be surprised if you do not see any video. We should be close to 100% compliant to the H.264 standard, all Base, Main, High and High 10 bit streams should play. Extended, High 4:2:2 and High 4:4:4 profiles are not officially supported at this time. They might or might not work depending on what features are used. We have no artificial lower limit on B-frames or any problems with B-pyramids like other players do. We also decode field coded streams, although this beta displays the images progressively using the weave method. The final release will be blending the two fields. There are still a couple of bugs with frame ordering/timing I need to fix in the Flash Player itself for the final release. And there is also a problem with files using the loop filter on dual core machines which causes horizontal artifacts along slice boundaries, which is my bad. The fix for this did not make it into this beta. Overall though and leaving out the bugs I listed here which are my fault, the H.264 decoder is a remarkable piece of engineering, it is provided to us by MainConcept. It weights in at less than 100KB of compressed code which is quite an achievement for such a complicated standard.
  


• Audio can be either AAC Main, AAC LC or SBR, corresponding to audio object types 0, 1 and 2. We also support the '.mp3' sample type meaning tracks with mp3 audio. MP3inMP4 which intends to do multi-channel mp3 playback within mp4 files is not supported. Also, the old QuickTime specific style of embedding AAC and MP3 data is not supported. It is unlikely though that you will run into these kind of files.


• 3gp timed text tracks. Any number of text tracks are supported and all the information, including esoteric stuff like karaoke meta data is dumped in 'onMetaData' and a new 'onTextData' NetStream callback. Language information in the individual tracks is also reported. That means you can have sub titles in several languages. Study the 3GPP TS 26.245 specification to see what information is available. Note that you have to take care of the formatting and placement of the text yourself, the Flash Player will do nothing here. Time for you to start working on one of those components which do that. You can use MP4Box to inject text data into existing files.


• Meta data stored in the 'ilst' atom. This is usually present in iTunes files. It contains ID3 like information and is reported in the onMetaData callback as key/value pairs in a mixed array with the name 'tags'. ID3V2 is not supported right now. An incomplete list and link to tools which can edit these tags is available here.


• Since these files contain an index unlike old FLV files, we can provide a list of save seek points, e.g. times you can seek to without having the play head jump around. You'll get this information through the onMetaData callback in an array with the name 'seekpoints'. On the downside, some files are missing this information which also means that these files are not seekable at all! This is very different from the traditional FLV file format which is rather based on the notion of key frames to determine the seek points.


• Unencrypted audio book files contain chapter information. We expose this in the onMetaData callback as an array of objects with name 'chapters'.


• Image tracks encoded in JPEG, GIF and PNG are accessible. Unfortunately only in AS3 as I pass this information as a byte arrays through a new callback 'onImageData'. You can simply take that byte array and use the Loader class to display the images. Most often these images represent cover artwork for audio files. TIFF image tracks are not supported, you might come across files using this. Also note that we support the 'covr' meta data stored in iTunes files, these are also accessible as byte arrays.


• Will it be possible to place H.264 streams into the traditional FLV file structure? It will, but we strongly encourage everyone to embrace the new standard file format. There are functional limits with the FLV structure when streaming H.264 which we could not overcome without a redesign of the file format. This is one reason we are moving away from the traditional FLV file structure. Specifically dealing with sequence headers and enders is tricky with FLV streams.


• Will it be possible to place AAC streams into an FLV file structure? Yes, though the same limitations as for H.264 apply.


• Will the Flash Player play back multi channel AAC files? It will play them, though the sound is mixed down to two channels and resampled to 44.1Khz. We are targeting multi channel playback for one of the next major revisions of the Flash Player. This requires complete redesign of the sound engine in the Flash Player which dates from circa 1996 and has not been improved since.


• Will the Flash Player be limited to 11Khz, 22Khz and 44.1Khz sampling rates like for MP3? No, we support all sampling rates from 8Khz to 96Khz. I implemented a 32 tap Kaiser Bessel based FIR filter which resamples the sound to 44.1Khz, retaining high quality. The most common sample rate combinations have a hard coded number of phases. In case of a 48000 to 44100 Hz conversion the filter has 147 phases f.ex. Even better: Flash Player Update 3 Beta 2 now can play back any MP3 sampling rate leveraging the same code I implemented for AAC. No more chipmunks. Ever. Err, this is actually kind of major as I have seen complaints about this bug for years :-) I fixed this problem in the AS3 Sound class, though it was using very low quality resampling. This change I made this time will fix it even for AS2 and sound in FLV files while retaining excellent quality.


• Will it be possible to place On2 VP6 streams into the new file format? Not right now, we are still trying to figure out if it is possible for us to support this.


• Can you play files protected by FairPlay? No.


• Do we support MPEG-4 BIFS or other esoteric stuff (scripting, VRML etc.) from the MPEG-4 Systems specification? No. Whatever is not listed above we do not support.


• Do we support SMIL? No. You can easily write your own SMIL parser in ActionScript though.


• Can you use the Sound class to play back AAC/.mp4a files? No, you have to use the NetStream class. We are now getting into a situation where there is not much difference between audio and video files anymore. They are the same essentially. Hence we figured we should not further add confusion and allow to do things ten different ways which would also increase the Flash Player binary size. My guess is that we will enhance the Sound class in the future but it might go into a different direction and will not be dedicated to pure playback of static files anymore.


• Here is a list of data which is reported in onMetaData:
  
  duration - Obvious. But unlike for FLV files this field will always be present.
  
  videocodecid - For H.264 we report 'avc1'.
  
  audiocodecid - For AAC we report 'mp4a', for MP3 we report '.mp3'.
  
  avcprofile - 66, 77, 88, 100, 110, 122 or 144 which corresponds to the H.264 profiles.
  
  avclevel - A number between 10 and 51. Consult this list to find out more.
  
  aottype - Either 0, 1 or 2. This corresponds to AAC Main, AAC LC and SBR audio types.
  
  moovposition - The offset in bytes of the moov atom in a file.
  
  trackinfo - An array of objects containing various infomation about all the tracks in a file.
  
  chapters - As mentioned above information about chapters in audiobooks.
  
  seekpoints - As mentioned above times you can directly feed into NetStream.seek();
  
  videoframerate - The frame rate of the video if a monotone frame rate is used. Most videos will have a monotone frame rate.
  
  audiosamplerate - The original sampling rate of the audio track.
  
  audiochannels - The original number of channels of the audio track.
  
  tags - As mentioned above ID3 like tag information.
  

Here are some good links to get an understanding of what MPEG-4, H.264 and AAC are:

http://forum.doom9.org/showthread.php?s&threadid=62723
http://forum.doom9.org/showthread.php?t=96059
http://en.wikipedia.org/wiki/H264
http://en.wikipedia.org/wiki/Advanced_Audio_Coding
http://daringfireball.net/2007/04/some_facts_about_aac

Let's put together some thought up scenarios I would imagine are important:

• You created a pod cast for iTunes and happily distribute over this channel. Now you want to add value to it and easily make it accessible over the web without special plug-ins, reaching an audience which does not have QuickTime installed. Well, this new feature will allow you to do this. You can take your existing podcast in .m4a format and present it on any web page through the Flash Player. Add more value by adding interactivity and branding if you want to. The possibilities are endless.


• Your media company has made or is about to make a significant investment into web video or video archiving. You are wondering what format you should choose. Video for Flash reaches everyone now, but the format is not an 'industry standard' so you have the fear that content you will create will become obsolete and unsupported at some point. Flash Player 9 Update 3 comes to the rescue: MPEG-4 is an extremely well documented ISO standard and completely vendor independent. And by using the Flash Player now you get instant gratification for viewers.


• You want to get best the possible quality out of your video and do not want to be tied to a particular encoding solution. You also like open source software to do all of the work you need to do to encode video. A combination of libfaad, x264 and MP4Box which are all licensed under the GPL will do exactly that, albeit with little usability and requiring lots of expertise. But it will now play just fine through the most distributed run time in the world, the Adobe Flash Player.

Those are immediate benefits, there are plenty more when we look ahead. Let me mention a few of them:

• H.264 will be supported natively by most new graphics cards. NVidia, ATI and Intel have made a commitments to have full support for it. This means better than HD video on your PC will become possible in the not so distant future.


• There are hardware based H.264 encoders which encode at better than real time. This is important if you need to be quick to market like f.ex news organisations.


• Digital TV, especially in Europe is quickly adopting H.264. The interoperability with the web will open new doors for a lot of media companies.


• AAC SBR offers demonstrable advantages over plain MP3, think 5.1 channel surround sound f.ex. While the Flash Player does only support 2 channels output at this time, there is opportunity to go beyond that.

And last but not least here are some things I will not give a complete answer to since they are begging for controversy:

• Comparing H.264 against other video codecs, might it be performance or quality. I've looked at the comparisons out there, they are at best subjective, most of the times outright marketing bull and almost always completely biased. My take is: Take a good and well accepted encoder and compare the results yourself. Your mileage will vary. And that is fine. Quality is not the main reason Flash Player 9 Update 3 has H.264 support.


• Tell you if On2 VP6 is better or worse than H.264. Truth is that they have different strengths, not only performance and quality wise. It totally depends on your individual situation of what fits best. The Adobe Flash Player now offers more choice which is more important than anything else.


• I am not in a position able to explain to you why we will not allow 3rd party streaming servers to stream H.264 video or AAC audio into the Flash Player. What I can tell you is that we do not allow this without proper licensing. Refer to Adobe's friendly Flash Media Server sales staff for more information.


• I can also not help you with anything regarding broadcast fees for commerical use of H.264 and AAC streams. Please refer to the FAQ Adobe provides which usually point to contacts at MPEG-LA and Via Licensing. A summary of licencing terms for H.264 is available here.

Multi core support

As I mentioned in Flash Player Update 3 we finally realized that multi-core CPUs are here to stay. So why not follow the times and take advantage of it? As most of you hard core Flash developers know, rendering is a huge bottleneck. I've seen a couple of blog post complaining that their second core/CPU is not doing anything when they run the Flash Player. Well people, this is about to change in this update.

Flash Player Update 3 takes advantage of multiple CPUs in many different areas. The first one is obvious: The vector rasterizer. The Flash Player will split the rasterization workload by dividing the stage into horizontal stripes. In case you have to 2 cores/CPUs, the top half is rendered by CPU 1, the bottom half by CPU 2.

How much performance improvements will you see? That depends, unlike with a 3D ray tracer the workload is not totally independent and there is some additional overhead to handle multiple threads. As a rule of thumb you could say that you get an additional 25-33% performance improvement per CPU. That does not sound like a lot, but keep in mind that the Flash Player still does some stuff which can not be multi-threaded.

The second area we focused on a are bitmap filters. If you deal with large movie clips and use lots of filters on them you will see a nice performance jump. Again the workload is split into horizontal stripes. The following filters are supported:

• DropShadowFilter


• GlowFilter


• GradientGlowFilter


• BevelFilter


• GradientBevelFilter


• BlurFilter


• ConvolutionFilter


• ColorMatrixFilter


• DisplacementMapFilter

Third, compositing and color transformations of bitmap cached object are also accelerated. The BitmapData APIs are NOT multi-threaded yet, hopefully we get around doing this at some point.

Fourth, VP6 video decoding now happens in a separate thread, independent of rendering and post processing of the video. In most cases that means a dramatic improvement in performance, especially if you did reach a limit performance wise with your current content despite the fact that you had a dual core machine. On my Athlon X2 4800 I can now play a true 1920x1080 pixels video at 30 frames/sec without a single dropped frame. Really cool if you ask me. It was a reason for me to buy a new toy: a Canon HV20 which can do 1080p at 24 frames/sec (actual pixel resolution is 1440x1080). Perfect for the web. Now if these video files would not clog up my drive, they are really huge...

Why is the Sorenson Spark codec not multi-threaded? Because of its design: It does not have a dual YUV buffer setup which allows us to separate blitting and video decoding.

Since I am talking about VP6 already let me mention some of the more generic improvements in VP6:

• I added a new gaussian noise filter which is turned on when post processing is active. It lessens the effect of blockiness and makes videos more look like movies. If you are turned off by this change, please let me know as soon as possible. In most of the cases it is seen as a improvement, but you designers should tell us if things still look okay.


• We fixed a long standing issue with VP6's automatic post processing selection on Athlon X2 class CPUs. It never worked correctly on these machines. This is due to a broken CPU driver on stock installs of Windows which affects the QueryPerformanceCounter Win32 call. Incidentally the new version of QuickTime and the Real Media Player seem to have problems related to this API, playback stutters heavily on my machine with some content because of it. I usually fix this problem by setting the process affinity to a single CPU in the Task Manager. That's what you get for not testing on AMD CPUs :-)


• Entropy decoding and some parts of the decoder should be faster, you should see a generic 10% improvement in CPU usage.

Mip map what?

True to my reputation here at Adobe I added a miniature feature into Flash Player 9 Update 3 (9.0.60.120). It is Mip mapping. The Wikipedia entry is nice, but there is an in depth article I highly recommend to anyone interested in the subject: Part 1 and Part 2. As described in the article we use the fast and simple box filtering as apposed to the more complex methods which are not really suitable for real-time rendering. So no Kaiser in the Flash Player. Neither do we have trilinear or anisotropic filtering, but then again they make more sense with real 3D graphics. And since we are still stuck in software only land for bitmap rendering I was not interested in a performance loss either.

Why do I call it a miniature feature? Because the whole feature occupies less than 1Kb of compressed code in the binary. This is the kind of changes we engineers in the Flash Player love. Extremely small in code size, large in impact.

If you feel kind of overwhelmed with the descriptions in these articles let me summarize that the feature does in more simple words: This feature improves the quality and performance of downscaled bitmaps. Not slightly downscaled, but anything which is scaled down by more than 50%. Mip maps are nothing more than precomputed smaller and higher quality versions of an original bitmap. They are used instead of the original bitmap when something is downscaled a lot.

A few words about the Flash Player implementation and its limitation might be worth talking about here:

• Mip maps are only created for 'static' bitmaps, e.g. anything like a JPEG, GIF or PNG you display via loadMovie(), a bitmap in the library or a BitmapData object. They do not apply to filtered objects or bitmap cached movie clips.


• In case of video things are more tricky. If smoothing is turned on the mip mapping applies since this is overall faster than rendering the original bitmap. For non-smoothed video mip mapping is not used since it would make things much slower.


• Mip maps level generation stops when it encounters an odd width or height. What does that mean? In the most extreme case it means that if you have a bitmap with an odd width or height pixel value to begin with, no mip map can be generated at all and therefore you will not see any benefit. This is unfortunately hard technical limitation. That also means that 'perfect' mip maps are generated from bitmaps which have a width and height which are 2^n, e.g. 256x256, 512x512, 1024x1024 etc. On average though it is enough to have bitmaps sizes which are dividable by 8, meaning you get at least 8 levels, f.ex. 640x128 -> 320x64 -> 160x32 -> 80x16 -> 40x8 -> 20x4 -> 10x2 -> 5x1.


• The quality improvements are more visible when smoothing in turned on for a particular bitmap.

The most immediate effect of this feature can be seen with Papervision3D. The Rhino demo f.ex. shows much less aliasing in the textures when you rotate. Also, a couple of frames per second more are displayed with this demo.

Good and well, but how does 'normal' Flash content benefit? Any time you create something like an image gallery with small thumbnails which are based on larger bitmaps you will see better bitmap quality and slightly higher performance.

Will this feature ever make things look worse? Not unless you do something really dumb like downscaling heavily aliased images and still expect it to look pixelized.

I made up a quick and dirty demo which shows the reduced aliasing effect: http://www.kaourantin.net/swf/mipmap.html Make sure you have Flash Player Update 3 Beta 1 installed, otherwise the two samples will look exactly the same.

What about control on this feature? Can you select the threshold or select your own mip map bitmaps? No, right now everything is automatic. That might change in a next major revision of the Flash Player. And the threshold value, well, let me simply tell you that we use the standard OpenGL value which is <= 0.5. If you want to know how this actually works, here is I would express it in pseudo ActionScript 3, you graphic experts will get it right away:

var bitmapData:Array; // contains the mip map bitmaps
var m:Matrix; // the affine matrix to used for display

var i:int = 0;
while ( Math.sqrt(m.a*m.a+m.b*m.b) <= 0.5 &&
        (bitmapData[i].width % 1) == 0 &&
        (bitmapData[i].height % 1) == 0) {

   // switch to next mip map level
   i++;
   m.a /= 2.0;
   m.b /= 2.0;
   m.c /= 2.0;
   m.d /= 2.0;
   m.tx /= 2.0;
   m.ty /= 2.0;

}
mc.draw(bitmapData[i],m);

At last I should mention that mip maps apply to any SWF version 9 or newer content. SWF version 8 or lower will not use mip maps since we feared that this would impact on backwards compatibility.

Flash Player Update 3 Beta 1

Flash Player Update 3 Beta 1 (build 9.0.60.120) is now ready for download here. This is probably the first time we have done a dot release as big as this one. We've been extremely busy over the past few months since Flash Player Update 2 (build 9.0.r45), so now it is finally time to talk about the improvements we have made. There are tons of them, so I'll use this post to summarize those I know something about and I have been working on. Stay tuned for more detailed posts of mine explaining the technical details behind these and how you will be able to leverage them:

• Mip map support for all bitmaps for Flash 9 or newer content. This improves the quality and rendering performance of downscaled bitmaps. Perfect for thumbnails and such. Even better, Papervision 3D content now automatically looks better and should be slightly faster when large textures are used.
  
  
• Multi-threaded vector renderer. Now we take advantage of up to 4 Cores/CPUs for vector rendering.
  
  
• Multi-threaded bitmap filters. Same as above but this applies to bitmap functionality specifically instead of the core vector rasterizer only.
  
  
• Multi-threaded video decoding. The VP6 video codec will now run in a separate thread if a multi-core system is detected which leaves the main thread to do rendering and post processing of the video. With this true 1080p video is now possible on most modern dual core machines. Also, the responsiveness is improved with this change. The Sorenson codec on the other hand did not get this change for technical reasons.
  
  
• Full-screen mode with hardware scaling. Probably the biggest new feature in the Flash Player Update 3. This leverages DirectX on Windows and OpenGL on OSX. There is an new API to control the behavior which was required since we could not change current behavior and we wanted to give the maximum flexibility possible. I know you are probably eager to use this feature, we will post more information on this on labs.adobe.com soon (Update: Link to labs page is active). I'll also will give you much more technical details in an upcoming blog post.
  
  
• Less tearing in the new full screen mode. We now have some code which will try to do VBL syncing. It's still a work in progress but we hope we can fix the remaining issues.
  
  
• Going into full screen mode has a zoom transition effect. The beta does not work perfectly right now, but we want to get feedback if this is acceptable to end users. We will not expose an API to access/control this, either we'll leave it in and fix the remaining bugs or it is out. Also you might notice that this even affects the current full screen mode, something we will remove in the final release.
  
  
• The Linux plugin now uses the XEmbed protocol. This is work in progress. The downside is that konqueror and Opera do not support this right now, so the Flash plugin will not work until these vendors update their plugin support. Also we are seeing decreased performance because GTK lacks somewhat in the the basic graphics API department. I'll explain in a later post.
  
  
• Tons of performance tweaks and bug fixes. Looking the the current bug database statistic we fixed 371 bugs since 9.0.r45. Fixed really means fixed, it does not include duplicates, unreproducible bugs etc.
  
  
• Much more.

A word of warning, this is a beta version! Do not use this version in a production environment. There are several known issues with the new features and while they might work on your machine they will not on others. We are obviously interested in the others and are looking for any issues you might encounter.

The limits of software rendering

I have been following the progress of Papervision3D recently, the 3D engine for Flash. Here at Adobe we are always interested in applications which challenge the Flash Player performance wise. We spent some time to profile some demonstrations to see if there is anything we can do performance wise. The short answer is: not really.

What is the problem? Well, there is a limit what software can do on current CPUs. The limit can be squarely attributed to memory bandwidth and the size of L1 and L2 caches. I thought I'd explain the point by showing some actual profiling data I collected with the demonstration mentioned above. You can reproduce the same results including seeing the actual assembly instructions with any 3rd party profiler tool out there.

For testing I used a pretty beefy system, an AMD Athlon 64 X2 4800+. CodeAnalyst was my profiler of choice since this is what AMD provides for their CPUs, Intels VTune does the job just as well though if you have an Intel CPU.

Here are the top three loops which show up in the profiler and together take about 40% of overall CPU time:

The first loop does the affine bitmap blitting, meaning it handles rotation and scaling. This loop takes about 20-25% of overall time. Its characteristic is that memory access is almost always non sequential, meaning you get lots of L1 and L2 cache misses.



The second loop does a simple copy from one buffer to another. Some will recognize this code, this was adopted from the AMD Athlon Processor x86 Code Optimization Guide. This loop takes about 10-15% of overall time.



The third does a blur for a radius which is power of two, I have presented this code in a blog post before. This loop takes another 10-15% of overall time.



So what can you determine from the above? Look at the CPU clock column. This is the number of total cycles spent on an individual instruction. Any memory access, especially random access like the affine bitmap rendering is the most expensive operation. It really does not matter at that point, without GPU hardware support there is pretty much nothing we can do anymore on modern CPUs when it comes to graphics performance.

Obviously there are other areas we can improve, the other 60% of overall time have maybe another 10% overall performance bump in store if we fundamentally change some algorithms and fix bottlenecks in Tamarin. Although in this example ActionScript execution really does not account for much, it is really all related to graphics code. In the end the conclusion remains: It is memory access time which kills us.

What is even more surprising though for some is that even multi core systems will not help dramatically. In most cases the memory bus and even the L2 cache are shared among the cores.