Google Video: Pragmatism at work

I've been following the news about Google switching to Flash Video lately and I am watching with interest their technical implementation of it. While I can mostly only make assumptions some things are interesting.

Google loves Open Source solutions, the cost savings for them are certainly immense. The original implementation of Google Video was using a custom plugin which was based on VLC. VLC in itself is heavily lending code from the ffmpeg project to decode and encode a number of video streams, an effort originally driven by Fabrice Bellard, while most of the real interesting stuff is now done by Michael Niedermayer. ffmpeg incidentally has support for Flash 7 video, supporting the .FLV and .SWF file formats. So the step from going from a VLC based solution to using FLV was probably a small one. At least on the encoding and other server side tasks.

What Google was really lacking with the VLC plugin is in my opinion penetration and a stable client. I tried the VLC plugin and ran into some minor problems on some machines, making me think it would have taken major engineering efforts to make it really stable (granted the Flash Player has its problems too, although more and more problems are fixed as we go along). IMO these are probably the main two reasons Google picked Flash. I do not think that porting the VLC plugin to Mac or Linux would have been such a big deal since the NetScape plugin API is supported on all of them.

Another small issue could have been the license fees which come with encoding MPEG-4 and MPEG-2 streams. Google did not use these format and actively disabled handling of these codecs. Still, the source code for encoding with non patent encumbered file formats is pretty much shared and intermixed with the MPEG-4 and MPEG-2 code, and so probably legally questionable in some countries. Do take my word for it though, this is just an assumption.

There are more reasons though why using Flash Video makes sense for such a large undertaking I think. One of these is that the .FLV file format is by design using the KISS (Keep it simple stupid) approach. It's neither offering the high fidelity or the flexibility file formats like QuickTime or Windows Media offer. But it does what it does well and that is playing back simple video streams with some meta information. In fact it's so simple that was able to write a perl script in less than 20 minutes which determines the length of a .FLV file, not having touched Perl for a really long time. Some customer had troubles interpreting the public FLV file format specification, so this helped them tremendously, they ported it to Java though for performance reasons. ;-) With some simple changes this can be modified to do splitting, merging and many other tasks.

As to Google not using the Flash Media Server, I got to give them some slack. Their whole backbone is probably based on serving HTTP streams, so not handling RTP at this early stage makes somehow sense. I hope though that at some point they'll feel more comfortable with Flash and see the benefits of going RTP once more complex client experiences are planned. On my side I would love to see Google Mail offering the ability to record and view video messages and Google Talk could easily be integrated into that experience also. Flash has everything you need to do so, on most of the common platforms. Certainly on more platforms that are currently supported by Google Earth and Google Desktop ;-) And new ones will be added too...

Garbage collection in Flash Player 8

While this was not a feature I worked on I still want to point it out. Flash Player 8 contains a new garbage collection (GC) system for ActionScript. Due to the nature of the beast this is somehow a feature which does not get too much Ahhs and Ohhs. It is one of those important under the hood things you simply never think about or should have to think about. But it is one of the most challenging and complicated feature an engineer can work on, it took a whole release. It is simply amazing to watch an engineer look at some raw dump of memory and tell me what type of object it contains. I am sure that when he does day to day life mental arithmetic he uses hexadecimal instead of decimal. :-)

Flash Player 7 and lower used reference counting to keep track of ActionScript objects. That means that each object created in ActionScript had a tracking counter attached to it. If that counter became zero the object would be released and memory was freed. Every object means not only objects like MovieClips, Arrays etc, but also simple types like Numbers, Strings and Booleans. This is a very simple system and worked well in the past. With the advent of large applications done in Flash this model has extreme drawbacks: High memory usage since even simple types need a reference count, low performance since incrementing and decrementing the reference count takes time and the potential for circular references with the result that memory might never be freed (although weak references somewhat reduced this issue).

Flash Player 8 features a mark and sweep GC, very much like most of the implementations of the Java run time. While I could try explaining all the details (probably with little success due to my limited involvement), there is an excellent Wikipedia article on this subject if you are interested. One of the drawbacks of the mark and sweep model is that the collection of objects is not immediate but deferred until a later time. This means that it could block the player for an undetermined amount of time if there is a lot to collect. Early versions of the Java run time had this problem if you remember, sometimes everything would freeze up for many seconds. To get around this problem Flash Player 8 implements an incremental GC. With the new GC in place you should see higher ActionScript performance and less peak memory usage. I can't give you exact number since it depends on the application, but in some very special cases we have seen memory usage decrease of 40%.

The true challenge of this project was to take the existing reference counting system and move it over to the mark and sweep model in the existing code base without breaking old content. I do not know if something like this had ever been attempted before, but I amazed how well it works now. I am also really excited that the Flash Player is growing up and getting closer to other run times out there like Java.

Movie clip and Graphic symbol differences strike again

Ever wonder why you can't apply filters to graphic symbols in Flash 8? This is certainly a limitation animators will run into again and again. While there could have been a technical solution, this feature was essentially cut due to time constraints and the involved complexity.

They key part is to understand what graphic symbols are. Although they seem to handle the same way as movie clips, they really only exist in the authoring environment. When you publish to SWF they are flattened out to plain shapes. That means the player would not know how to apply the filters as the sense of object hierarchy is gone due to the flattening.

One of the technical solutions for this would have been to export graphic symbols as movie clips behind the scenes. The problem with this is that a movie clip time line is not synchronized with the parent timeline, it uses its own timing. The whole point of graphic symbols is exactly that: have animations play synchronized on the same time line. And the whole point of movie clips is that they are not synchronized and can be individually controlled. So somehow we needed a mix between these two symbol types.

I am disappointed that we were not able to fix this in this release, but in the future (meaning I give no guarantee this will ever be implemented, when it will be implememented, how it will be implemented etc. You get the picture...) I'll be looking into a solution where you can tie movie clip time lines together which would resolve this problem. And while at it, why do we not add the same thing for NetStream objects? ;-) In ActionScript this could look like this:

MovieClip.syncTo(movieClip:MovieClip, 
    startFrame:Number, endFrame:Number):Void;
MovieClip.syncTo(netStream:NetStream, 
    startTime:Number, endTime:Number):Void;

One of the side effects of this solution is that some calls like MovieClip.stop(), MovieClip.play() etc. would have no effect anymore until you untie the time lines. Let me know what think.

Beta releases

Here and there I am browsing forums and blogs to see what people a doing with Flash 8. During those visits I sometimes come across posts of frustration. A typical comment would be something like this: "I wasn't worried since it was still a beta, but it seems Macromedia did not fix this bug for the final release!". Somehow that leaves me with frustration too since we genuinely try to address as many bugs as possible during the betas. But there seem to be a few misconceptions of what public betas of the Flash Player are really about. I am not sure if that is a communication problem on our side, but let me put out a few blunt points:

• To debunk a common myth: Public beta versions do not contain any type of special debug code or information. What you get as a beta is essentially the same thing you'll get upon release.


• Public betas are considered functionally complete. Otherwise we'd call it a 'preview' or 'alpha' release. If you see something missing or non functional, it'll be missing and not working in the final release unless you tell us.


• Public betas are fully performance optimized. If something runs slower, it'll be running slower in the final release unless you tell us.


• Public betas will break existing content. It always does despite our best efforts of manually testing huge amounts of existing sites and having large automated test systems. Unless reported to us chances are high it will remain broken.


• Public betas are released months before the final release. This is because a bug life cycle can average 2 weeks. If a bug is reported to us 2 weeks or less before a release, chances are high it will not be fixed unless it is catastrophic (meaning a security issue or crasher).


• While posting about bugs on blogs and forums is informative to the community, we sometimes miss these as we are extremely busy during beta time as you can imagine. So please, please take a few seconds and use the bug reports forms or if you really can't stand the impersonal feel of this, email someone at Macromedia about your problem. Before the final release. :-)

Microsoft Sparkle

I am currently attending the PDC 2005 in Los Angeles and the big news for me is obviously Microsoft Sparkle. One word summarizes my reaction: Respect. Not only did they make a demo, they even offer a hands on lab where you can try it out yourself. Talking to some of the engineers reveals that they have been doing some amazing work in the past 4 years (It seems it's a team of about 40 people which has been working on this). Sparkle looks to me like a mix of Macromedia Flash, Macromedia DreamWeaver, Adobe Illustrator and Macromedia Director.

Looking at the features I can see that they copied a bunch of ideas from competing products and technologies:

• XAML. A straight rip off the SVG standard (Update: what I really mean here is that the code generated by Sparkle looks VERY similar to SVG. I am sure there are subtle differences though. Saying it's a rip off is a little inflammatory, I admit. ;-).


• Code view. Straight from design/code view feature in DreamWeaver, but instead of HTML they show the XAML code. It works amazingly well and the code generated looks very clean (unlike FrontPage or Word ;-).


• Timeline. The timeline is very AfterEffects like, although very much simplified. The timeline allows you to dig down on every property and manipulate it. They are no nested timelines like in Flash.


• Media integration. This looks very much like what Director offers. You can insert video and 3D with a few clicks and manipulate it easily.


• UI elements. This looks very much like Flash components which first appeared in Flash MX. All the controls are stylable in CSS fashion.


• Layout control classes. Looks very much like Java Swing.

But there is much much more, I made a list of features for my personal reference and I am simply amazed how complete everything seems. What is even more impressive is that Microsoft did eat their own dog food and implemented the Sparkle UI using the Windows Presentation Foundation. Very much in the classic way: A compiler is not a real compiler until it can compile itself. :-)

Overall the innovation here is not so much the individual features, but the integration of them into one single package. With the addition of WPF/E I suspect this will be final nail on the coffin for SVG.

The first small scale beta program will start in 3 months from what I just heard. More updates and details will follow...

Using Flash Player 8 filters for good

I am pretty sure that a lot of code purists out there will doubt that Flash Player 8 brings anything useful to the table apart from fancy new graphics. But some of the filters can be used for much more important things which become critical when you try to deploy Flash applications to large audiences.

We've had many discussion internally of how we could make the Flash Player more accessible for disabled users and I can already tell you that there are plenty of ideas which will eventually make their way into the player. The list of things to do is really long. But for the time being let me show you a little example of how a filter can make something which used to be difficult very easy now. If you ever had to deal with accessibility for the visually impaired you might know that Flash does not follow the high contrast settings of Windows. This is a real problem and prevents the deployment of Flash applications in many cases. While native integration is still missing in Flash Player 8 you can offer clients a way of enabling this, using the ColorMatrixFilter.

Here is a little example. Click on the "High Contrast & Scale" checkbox and you should see right away what's going on. This was simply not possible in previous releases without writing complex code to handle color schemes. With enough end user controls (allowing to set saturation, contrast & brightness) it can be extended to apply to any type of Flash application, not matter how weird the colors might be, including video. I think this is very powerful. In this particular example the ColorMatrixFilter setup looks like this:

var highContrastFilter = new flash.filters.ColorMatrixFilter ();
highContrastFilter.matrix = new Array (
 -2, -2, -2, 0, 768*2, 
 -2, -2, -2, 0, 768*2, 
 -2, -2, -2, 0, 768*2, 
  0,  0,  0, 1, 0); 

If you use your imagination I am sure that you will find many more useful applications for the new Flash Player 8 features which are not only designer oriented.

Some word on blend modes in Flash Player 8

Blend modes is one of those features who were long thought impossible to do in the Flash Player. And to be frank, it was indeed the most difficult feature I worked on during this release. Something to understand here is that blend modes are not implemented through the bitmap caching feature like the filters, but it's actually part of the core vector renderer. The two main advantages here are that using blend modes will not use excessive amounts of memory and use the shortest path possible in the rendering pipeline making it acceptable speed wise.

Nevertheless there are some things which need to be understood about this feature. It has limitations performance wise. In fact anytime you use a blend mode, all of the vector content in a particular region of your movie will be handled as it was partially transparent. And as you might know using transparent vectors can be a bit slow if used too gratiously. Experiment and see where they are useful, but do not overuse them. Sometimes precompositing or using bitmap caching might help.

The leads me to the list of modes we currently support. Some of them are better understood than others:

• Multiply


• Screen


• Darken


• Lighten


• Hardlight


• Overlay


• Difference


• Add


• Subtract

We used two sources to implement these blend modes. The main source was the SVG 1.2 draft spec which presents the algorithms working with premultiplied alpha color values which Flash needs. The other one is much easier to understand and is a rather complete list of all known blend modes. This guy did an excellent job of putting this list together. As a side note: I forwarded this link to the FireWorks team a long time ago after I saw that the original plan was to only implement two new modes. Guess what, the new FireWorks 8 blend modes look VERY much like the ones listed on this page now ;-) It was the main source of information for them. Funny how much things have changed since the days were you had to buy expensive books to access this kind of information.

There are two special modes here which are f.ex. not part of Photoshop and that are the 'Add' and 'Subtract' modes. These will do exactly what they say and simply add or subtract color values, alpha, red, green and blue color components separately. I think that the 'Add' blend mode will probably be most used one overall since it's the fastest one (even faster than normal alpha compositing!) and will allow great lighting effects which were essentially only possible in OpenGL or Direct3D. In PhotoShop most users use 'Screen' as a replacement which is very similar.

Some of you who have used Director in a past life and maybe now might think that these are simply 'ink' effects. Well, not really. Flash rather followed the path of SVG, Photoshop and other more modern graphics applications. Directors 'ink' effects are IMO rather binary transfer modes that remind me of the QuickDraw era. Sticking to these blend modes is also more in line with what can be easily implemented in graphics hardware.

For those computer scientist out there you might wonder why we did not include a full Porter-Duff compositing model into the Flash Player. Frankly I think this model is pretty much irrelevant for most designers and I had no desire to add a very complex feature which does not bring anything useful to the table. In addition we would have had to implement a JIT for compositing functions just to make it fast enough and that was frankly out of the scope for this release.

There are some 'special' blend modes I should talk about:

• Layer


• Invert


• Alpha


• Erase

The layer blend mode is probably the most interesting here. What does it do? Well, have you ever tried to place several stacked bitmap images into a movie clip and apply a alpha color transform on it? If you used Flash in the past, the alpha color transform would be applied separately to each image instead of being applied to the whole stack of images taken together. This means that images in the back would show through opaque images stacked on top. Now if you set a movie clip to use the layer blend mode, the images inside it will first be composed together and then the alpha color transform is applied on top of it. Once you get a hang of this you'll see that this makes Flash more compatible with applications like FireWorks or PhotoShop which work this way all the time.

The invert blend mode is a special version of the difference blend mode. I added this since a lot of times you simply want to invert an area (for selection f.ex.) without having to fiddle with the color of the top movie clip to get the same effect. It's also a tad faster than the difference mode.

That leaves alpha and erase. If you apply an alpha blend mode on a movie clip it will extract the alpha of that movie clip and try to apply it to the background. The erase mode will invert the alpha before applying it to the background. Now, and that probably the hardest part to understand, you can not change the alpha of the root stage. It's always opaque. In that sense this is very different from FireWorks and Photoshop which will show you a pattern where the transparent areas show through. This means applying an alpha blend mode on a movie clip on the root timeline will have NO effect. Instead you need a movie clip with the layer blend mode set (or any other blend mode) and move your movie clips which should be alpha blended inside it. A movie clip with the layer blend mode set is allowed to have a transparent background as it will be recomposited on the background later. I hope this makes it clear somehow. Our documentation guys certainly had a hard time understanding this, I am still looking for better ways of explaining it...