Thursday, February 11, 2010

Core Animation

Amazing, so many things have happened in the Flash Player engineering team over the past year. Lots I would love to talk about. But the purpose of this post is to deep dive into a subject Kevin Lynch touched upon recently, specifically Mac performance and his comment about Core Animation. Whenever performance is mentioned in the context of Flash it gathers a lot of the attention and some of the technical background is lost in the PR.

So what's the deal with Core Animation in Flash Player 10.1? Let's look at how Apple's documentation summarizes what Core Animation does:

Core Animation is an Objective-C framework that combines a high-performance compositing engine with a simple to use animation programming interface.

Sounds like perfect match for Flash does it not? So yes, Flash Player 10.1 is attempting to leverage this framework to work around a few specific technical issues we've had in Safari and all other browsers on OS X.

The drawing model jungle on OS X

Before going into more specifics of why we are going towards Core Animation lets get an overview about how plugins on OS X draw into the browser window. There 4 possible ways (compared to one on Windows):

  1. QuickDraw. Default mode used by Opera, older Firefox and Safari versions.
  2. Quartz 2D (a.k.a. Core Graphics). Supported by newer versions of Firefox and Safari.
  3. OpenGL. No browser I know of supports this properly today.
  4. Core Animation. Only available in Safari 4 + OS X 10.6 right now, with caveats in the current version.

In addition to these drawing models designers can embed Flash content in 3 different ways by specifying wmode:
  1. Normal
  2. Opaque
  3. Transparent
Normal means that you can't have overlapping HTML sitting on top of your SWF, Opaque allows it and Transparent means that the SWF is transparent and underlying HTML content will show through. Taking all these variables into account we come up with these tables which shows when a particular drawing model is used (and subject for change before we release Flash Player 10.1):

Flash Player 10.0:
Safari 4Firefox 3Opera 10
NormalQuartz 2DQuickDrawQuickDraw
OpaqueQuartz 2DQuickDrawQuickDraw
TransparentQuartz 2DQuickDrawQuickDraw


Flash Player 10.1:
Safari 4 (*)Firefox 3Opera 10
NormalCore AnimationQuartz 2DQuickDraw
OpaqueQuartz 2D(**)Quartz 2DQuickDraw
TransparentQuartz 2D(**) Quartz 2DQuickDraw
(*) Actually using nightly builds of WebKit because support for Core Animation is work in progress.
(**) Core Animation is used when the SWF is the front most object on the HTML page.

What are the issues with Quartz 2D?

The basic premise of Quartz 2D as Apple describes it:

Quartz 2D is an advanced, two-dimensional drawing engine available for iPhone application development and to all Mac OS X application environments outside of the kernel. Quartz 2D provides low-level, lightweight 2D rendering with unmatched output fidelity regardless of display or printing device.

Quartz 2D is not designed for multimedia applications, like animation or video playback. That's where OpenGL, Core Video, Core Animation shine. Safari's use of Quartz 2D to draw HTML content makes perfect sense as its content is static in most cases. Everything works well until Flash comes into the picture. For instance when the Flash Player plays a SWF using the Quartz 2D drawing model is has to do so with the full involvement of the browser. The sequence of events looks like this (you can follow the stack traces in Shark):
  1. Whenever the Flash Player is ready to display a new frame, the Flash Player requests a refresh of its region using NPN_InvalidateRect.
  2. The browser adds the the rectangle provided by the Flash Player to its dirty region.
  3. The browser traverses its own display list (the HTML DOM) and paints every node which is part of the dirty region.
  4. When the browser finds a node with a Flash Player instance it first draws the HTML background and then posts an event to the Flash Player to tell it that it has to paint over the requested region now.
  5. The Flash Player then finally draws its frame.

So far so good, makes sense I hope. So what's the technical issue? Think of a fairly complex HTML page, for instance a page with a CSS gradient in the background. Add to add a SWF which runs at 30 frames/sec. You will see that a lot of time is spent in the browser, not in the Flash Player. This is where Core Animation kicks in: step 3 and 4 pretty much go away (as long as the SWF is the top most object).

Core Animation in the Flash Player

Flash Player 10.1 implements the Core Animation drawing model to fix this technical issue, among others. Instead of using a CGImageRef + CGContextDrawImage to get the bits to screen we pass a CAOpenGLLayer to Safari and use an OpenGL texture of type GL_TEXTURE_RECTANGLE_ARB to get our bits to the screen.

The support for the Core Animation drawing model was originally driven by Apple and we have worked feverishly to finish the engineering work on both sides. Yes that's right: This was and is a joint effort between Apple and Adobe engineers. Given the now almost perfect integration of Core Animation plugins into Safari I hope that future versions of the Flash Player will take advantage of more capabilities of OpenGL. And that without the requirement of setting any special wmode. I am pretty stoked about it.

As of today (2/10/2010) we are getting closer to having it stable enough for public consumption. That means though: You will need Flash Player 10.1, OS X 10.6 and updated version of Safari (or the nightly WebKit build), otherwise you will not see anything.

What difference does it really make?

This is by no means panacea for all performance issues in the Flash Player. Far from it. But it is a small step to a larger goal which is to improve the experience in the browser with the ever more complex web content out there. That said here is a comparison between Flash Player 10.0 and Flash 10.1 using this test case (this only works in Safari). Keep in mind that that is an extreme test case which has little to do with real world web content.

Flash Player 10.0 + nightly WebKit + OS X 10.6


Flash Player 10.1 + nightly WebKit + OS X 10.6


PS: You might have noticed that Core Animation is a Cocoa API. Yes, Flash Player 10.1 is a true Cocoa app now (with a Carbon fallback to support Firefox and Opera which are not Cocoa yet).

90 Comments:

Blogger mvonballmo said...

Opera 10.5 is a Cocoa application; any idea of whether it will support Core Animation?

Wednesday, February 10, 2010 11:08:00 PM  
Blogger Martijn said...

I cannot call this construction "lazy" ;) A very nice read, very clear (to me).

Wednesday, February 10, 2010 11:12:00 PM  
Anonymous Bazard said...

Congratulations! ^^

Wednesday, February 10, 2010 11:50:00 PM  
Blogger vhbit said...

What about Chrome browser? Since it uses WebKit engine can we expect that it will take advantage of Core Animation too?

Thursday, February 11, 2010 12:18:00 AM  
Blogger Tinic Uro said...

mvonballmo: Opera does not support Cocoa plugins yet. Once it does Flash Player 10.1 will automatically switch to Cocoa mode, i.e. it is a runtime check in the Flash Player.

vhbit: Chrome does not support Core Animation. Once it does Flash Player 10.1 will automatically take advantage of it. Like for Cocoa mode this is a simple runtime check, i.e. the Flash Player asks the browser if that drawing model is available. If it is it will use it.

Thursday, February 11, 2010 12:31:00 AM  
Blogger Christian Giordano said...

Which are the differences, in performance, between Quartz2D and QuickDraw?

Thursday, February 11, 2010 12:34:00 AM  
Anonymous Tek said...

The problem with Quartz 2D will still be major with the advertising companies who don't understand performance concerns. They near all publish animations with wmode=transparent by default.

Should we not change how the player works by always initializing it by fixing wmode to windowless by default?

In this case only the code would allow the browser to change the animation wmode to transparent or opaque. The Flash Player ActionScript API may include a Stage.wmodeEnabled property or even if possible Stage.wmode property.

Don't know if the Flash Player could force the wmode property of its DOM container? But it seems a good idea to fix this problem and to fix the problem of animation running in the background (I mean inactive tabs or window).

Thursday, February 11, 2010 1:26:00 AM  
Blogger Nass D. Luth said...

Hello sir. Great job! This work is not "lazy" at all :P

"OpenGL. No browser I know of supports this properly today"

You said that in order for Flash Player to use OpenGL more (that's mean GPU?) we need to wait for the browser's support itself. However i have seen implementation of web plugins that use GPU to create hardware accelerated graphic (real 3d) such as Unity and O3D (Google).

Any comment on this?

Thursday, February 11, 2010 1:45:00 AM  
Blogger splix said...

Is there any plans to add graphics acceleration into Linux version of Flash plugin?

Thursday, February 11, 2010 2:17:00 AM  
Anonymous Armetiz said...

For a minor update, this is a real improvment !

Thomas Tourlourat

Thursday, February 11, 2010 2:19:00 AM  
Blogger Vic said...

So lets say I do same project in Flash and in Unity. Both running on PC or Mac. One of them is 10x faster. Why?

Thursday, February 11, 2010 4:35:00 AM  
Blogger Vic said...

So lets say I do same project in Flash and in Unity. Both running on PC or Mac. One of them is 10x faster. Why?

Thursday, February 11, 2010 4:35:00 AM  
Blogger Maxim Porges said...

Thanks for the detailed overview - this makes a lot of sense. Looking forward to continued improvements in FP performance on OS X.

Thursday, February 11, 2010 5:07:00 AM  
Blogger Matthew Fabb said...

Thanks for the great detailed post Tinic! I love reading behind-the-scenes technical details like this.

It would be great to see more posts like this describing the technical reasons why there's performance differences in Flash between browsers.

Thursday, February 11, 2010 5:29:00 AM  
Anonymous Anonymous said...

Sigh. It's us Linux users who need help from companies to create the code for new APIs in X! OSX and Windows already have hundreds of engineers working furiously to implement new functionality in advance...

Thursday, February 11, 2010 5:39:00 AM  
Anonymous Anonymous said...

(Oops, forgot to say thanks for the 64 bit version of Flash and to add the obvious licence caveat that goes with providing plumbing work on Linux thus making it unattractive to companies)

Thursday, February 11, 2010 5:42:00 AM  
Blogger George Profenza said...

This is a bit over my head, but drawing Flash content into a CAOpenGLLayer, would be handled internally by the Player, right ?

This might sound silly, but,
t any point would it be possible for Flash Developers to pack OPENGL instructions and send that to the Player ? As in being able to implement a custom 3D engine using OPENGL and sending the projected 2d result as GL_TEXTURE_RECTANGLE_ARB to the player ?

Thursday, February 11, 2010 6:05:00 AM  
Anonymous Bryn said...

Great explanation Tinic. Will these Core Animation benefits pass through to the WebKit browser inside Adobe AIR (which I think currently uses CoreGraphics (Quartz 2D) but follows the same WebKit branch as the Safari 4 beta)? Thanks

Thursday, February 11, 2010 6:56:00 AM  
Blogger Paul Neave said...

"OpenGL. No browser I know of supports this properly today"

Following up on Nass's point, the new WebKit (Chrome and Safari) nightly builds and Firefox 3.7 are developing native OpenGL support via JavaScript. So will Flash also be able to implement OpenGL's power eventually?

Thanks a lot, Tinic.

Thursday, February 11, 2010 8:11:00 AM  
Blogger CaptainN said...

Very nice info. This explains why current versions of Flash on Mac never have that "flash through dhtml menus" problem that it has on Windows. Is there any chance of getting a way to change that wmode flag at runtime (either in JS or AS)?

Thursday, February 11, 2010 8:41:00 AM  
Blogger skuwamoto said...

Is there any chance that using Core Animation will get rid of the vsync tearing that is so apparent in that demo? :-)

Thursday, February 11, 2010 10:26:00 AM  
Anonymous Anonymous said...

@nass

Tinic didn't say other plug-ins didn't use OpenGL. He's specifically referring to the NPDrawingModelOpenGL browser feature.

The plug-ins like you mentioned rely on an unsupported technique that dangle OpenGL surfaces on the browser window, without any involvement from the browser. They are responsible for when to manually show and hide the surface when switching tabs, etc. A design that relies on unsupported windowing/browser view hierarchies assumptions and is unfortunately prone to breaking and is not compatible out-of-process design.

NPDrawingModelCoreAnimation plug-ins (like Quicktime and Flash 10.1) are entirely encapsulated from all this.

Thursday, February 11, 2010 12:18:00 PM  
Blogger James Smith said...

When is Flash Player 10.1 scheduled to be out of Beta? It also fixes the long-standing LocalConnection bug on macs. Great work, glad to see this progress!

Thursday, February 11, 2010 3:02:00 PM  
Blogger dewman said...

Awesome! Finally, a reason to upgrade to Snow Leopard : )

Thursday, February 11, 2010 5:28:00 PM  
Anonymous Anonymous said...

Why is Flash 10 still using QuickDraw for Firefox but not for Safari? Firefox has had support for NPNVsupportsCoreGraphicsBool since 3.0 was released in June 2008. Flash 10 wasn't released until October 2008.

Thursday, February 11, 2010 5:47:00 PM  
Blogger Tinic Uro said...

@Anonymous It turns out that QuickDraw is faster than Quartz 2D, especially for very small region updates (20-30% for the isolated BitBlt call vs. CGContextDrawImage). We've made unrelated improvements in performance in FP10.1 so that we break even now which is better than regression.

Thursday, February 11, 2010 5:53:00 PM  
Blogger zwaldowski said...

What about 64-bit mode, in more than Safari? Now that it's pure Cocoa for Safari, 64-bit should be cake. In Firefox, there's a bit of a chicken-egg problem: since Flash would use Carbon in Firefox, developers there aren't really pushing for it even though it's ready and works (I'm typing from it now).

Thursday, February 11, 2010 8:37:00 PM  
Anonymous Anonymous said...

You should teach Mike Melanson a thing or two about writing informative posts without dripping spite, insults and hatefulness.

Friday, February 12, 2010 10:58:00 PM  
Anonymous Anonymous said...

You're using QuickDraw in 2010??? How lazy can Adobe be, this 30 year old API has been deprecated since 2007.

Why has VLC beat the pants off Flash playing FLV or any type of video for the last 10 years on OS X?

Saturday, February 13, 2010 12:39:00 AM  
Blogger Tuomas A said...

Fantastic to see you blogging again! Keep it up!

Saturday, February 13, 2010 10:08:00 AM  
Anonymous Anonymous said...

If I am not mistaken, Apple also offers the Core Animation drawing model on Safari 4 running on Leopard. Will Adobe support the new drawing model on Leopard as well?

Saturday, February 13, 2010 11:06:00 AM  
Blogger jamiegau said...

READERS PLEASE BACK THIS ISSUE SO WE GET A RESPONCE POST ON IT.

It is great to see Adobe is putting a lot of effort into Flash performance for Mac based technology. (Desktop, Iphone, iPad)
For too long now the Techorati have been shooting Flash down, and for good reason.
This post seems to indicate that Flash performance issue where based around exposed API's rather then Adobe's need.

One MAJOR topic you failed to mention was the support for hardware accelerated Video.

I am actually happy to see the VIDEO tag, however, HTML5 will never support some features we can do in Flash. Flash as a second rate, non-hardware accelerated, citezen on a Mac, Flash will NEVER be taken seriously.

Adobe has indicated, in other public blogs, that the reason for not supporting hardware acceleration is a APPLE decision for not exposing the API's needed. That it is not Adobe being "LAZY".

Us as the flash community deserve a clearer understanding of this issue from Adobe.
Why it is not supported currently?
Why it is not mentioned above and looks like it will not be supported in 10.1?

The above is the main ammunition used by Apple and its well manipulated Technorati as to displace Flash.

As developers who have put our life into the future of Flash, we deserve to know whats going on here.

James

Saturday, February 13, 2010 12:25:00 PM  
Anonymous Zbigniew L. said...

Why Linux flash does not use cairo library for drawing and picture compositing? Firefox3 and Seamonkey2 uses cairo for page rendering (text, pictures, svg) on all platforms (Windows/Linux/Mac). Cairo library operates on RGB model. Cairo has many hardware accelerated backends. On Linux it uses XRender or OpenGL for fast drawing.
XRender acceleration is near perfect on Nvidia in Linux. Why you do not use it directly or via cairo?

Saturday, February 13, 2010 12:43:00 PM  
Anonymous Anonymous said...

@Tek: I built a series of reusable, dynamically-sized ad containers for a major ad network last year. I made sure to use the window wmode in every browser I could. (I think I used transparent to fix a drawing issue in IE7).

@Tinic: I'm creating a SWF that applies multiple Pixel Bender shaders to a video. Should I be using window, direct, or GPU as the wmode?

Saturday, February 13, 2010 4:25:00 PM  
Blogger Petter said...

jamiegau,

That's not quite true, Apple's OS isn't a looked platform where they can't do nothing without the blessing of Apple. It's actually very open. They could do the same thing they did in Windows, where they told the graphics vendors - driver makers to support the specific functionalities they needed. It was up to them to support it, not Microsoft. It's not that DXVA enabled drivers of the bat support Adobe's flash video acceleration. They needed to implement support for it. So the driver developers did as they communicated.

nVidia and ATi/AMD can and do (occasionally) release their own drivers outside of Apples channels. The CUDA driver for example are such. Apple don't need to implement any CUDA APIs into their frameworks and toolkits. It's a good damn beta for gods sake. So they could have released experimental support that's enabled with say nVidia drivers. It's just that they don't seem to communicate. So the Windows and embedded flash player folks do demand the needed functionality from the driver makers (not from Microsoft) and get it, while the other gang who does OS X and Linux doesn't seem to be doing any thing else then to complain. They don't drive development or issue demands or even communicate what they want.

Sure they might want a single working API (not that hard to communicate and demand one) when it's released, but not even developing a solution in the lab doesn't look good. Thus likely the final release won't support any acceleration.

On OS X it's meant that you should use Quicktime just as you should use Media Foundation on Windows (Vista & 7). Adobe don't utilize Media Foundations framework or the older and still supported DirectShow. It's doesn't use a EVR renderer/overlay. If they do it all outside of the platforms APIs for video, they can't blame them when standardized solutions aren't available for them. However on Windows they apparently could just communicate what they needed with the graphics vendors. Normally the EVR renderer is required for DXVA 2.0 acceleration. Thus Microsoft does not supply an API they can use as is.

Sunday, February 14, 2010 6:18:00 AM  
Blogger Og2t said...

Hold on, I don't quite get something, why it's possible to get stunning display performance with Java applets (i.e. Processing) and Shockwave but not with Flash Player?

Monday, February 15, 2010 1:22:00 AM  
Blogger jamiegau said...

@peter, You seem to know a lot about this issue. But.. Could you please be more detailed about why would Adobe NOT simply use the API's exposed by the OS.
DirectShow, MediaFoundation on Windows, and Quicktime on Mac. Same as the tag implementations are doing under HTML5 implementations?

The Vendor API will take advantage of the best and most current acceleration technologies as they become available. Adobe just need to support the API.

Why was it not done this way? Why use some of these direct methods?

It sounds quite silly to me.

James

Monday, February 15, 2010 3:52:00 AM  
Anonymous Erwan said...

Sounds very promising!

Wednesday, February 17, 2010 12:22:00 PM  
Blogger John Mason said...

Thanks for the post and diving into the technical side of this.

Core Animation has been out for some time now (around 3 years from my research). Why did it take so long to incorporate it into Flash Player? I would have figure you guys would have wanted to jumped on that fairly quickly given the performance gains.

Wednesday, February 17, 2010 12:54:00 PM  
Blogger Tinic Uro said...

@John Mason: because without browser support it is not possible for a plugin to use Core Animation. While Safari has had support since version 4 there were numerous issues which now have been resolved.

Wednesday, February 17, 2010 12:58:00 PM  
Blogger Tinic Uro said...

@jamiegau: QuickTime is inadequate for the Flash Player for about 100 or more reasons. Examples: - It does not support RTMFP or other required protocols by the Flash Player. - It does not provide low level access to an H.264 decoder. - You can not read back RGB data with reasonable performance. - Pre 10.6 only supported baseline and some of main profile. etc. etc. etc.

Wednesday, February 17, 2010 1:02:00 PM  
Blogger Tinic Uro said...

@Og2t: If you would have examined the example I posted for 1 minute you would have realized that it is simply not possible with either Java or ShockWave to do the same. Java can not render a transparent background (i.e. it does not have support for wmode=transparent). It also does not support DTML z layering. Neither does ShockWave. And ShockWave remains mostly a QuickDraw based plugin meaning it will never run as a 64bit plugin.

Wednesday, February 17, 2010 1:11:00 PM  
Blogger Tinic Uro said...

"You're using QuickDraw in 2010???"

Some browsers still do not support Quartz 2D or Core Animation, especially the PowerPC versions. Do you want the plugin to simply stop working?

Add to that that QuickDraw turns out to be faster than Quartz 2D in some case.

Wednesday, February 17, 2010 1:13:00 PM  
Anonymous Pesti said...

Tinic, we need your help over here at the linux camp! Mike just can't handle it any more. He realized linux is bad voodoo and reduced his commitment to nagging and whining.

Wednesday, February 17, 2010 2:47:00 PM  
Blogger Petter said...

jamiegau, short answer is because Flash Player is not a video player and Adobe don't want to do it that way.

A bit longer bit still short. They need to read back the RGB-video stream (as Tinic responded) and do some magic with it. So they can have their overlays, text and ads. There's nothing stopping them to turn FP (flash player) into something that will open as a native video-player. MainConcept which does the decoders in FP do offer decoders, even directshow decoder and demuxer/importer for playing flash flv/mp4 videos. There are other decoders who can do flash videos too. Including doing RTMP. But sites like youtube uses HTTP. There's even experimental hardware acceleration in other libraries.

However it's up to Adobe to decide whatever they want to do. Thankfully as most sites already just play mp4 H264 videos with the FP we probably will have alternatives such as the HTML5 tag. Then there's just the problem of licensing a player/decoder if you have none, Adobe does that for you today. On Mac Apple has already done it. On Windows Google (Chrome) does it. On linux gstreamer codecs will probably work. Or as Google does it with Chrome you can use that legit. There's commercial codecs if one needs one. It's not an replacement however.

Any way if they didn't read back the RGB-data they can do bitstream (hw) acceleration on the H264 videos flash player plays and it would work no matter which platform or framework. DirectShow, Media Foundation, DXVA1.0, DXVA2.0, Quicktime's accelerated H264 decoder, Broadcom cards/library (works in FP's work flow too), VDPAU for unices, VA-API, OpenMAX decoders and so on. But as said, they don't want to output the video via these APIs/frameworks. As they are not really building a video player.

Wednesday, February 17, 2010 6:48:00 PM  
Blogger Tinic Uro said...

@ Petter: One of the most informed comments I have read in a while.

Wednesday, February 17, 2010 7:59:00 PM  
Anonymous Anonymous said...

So no comment on which wmode is recommended for processor-intensive video playback with filters in 10.1?

Wednesday, February 17, 2010 10:42:00 PM  
Blogger Christian Giordano said...

Ok, Core Animation could have been used only recently because Safari 4, but what about Flash Player standalone? Which renderer does/did it use? And Adobe AIR hybrid apps, the ones with webkit? I presume both will use Core Animation, isn't it?

Thanks for all the info, chr

Wednesday, February 17, 2010 11:24:00 PM  
Blogger Tinic Uro said...

@Christian Giordano: The standalone player has been using OpenGL for many years now, no QuickDraw or Quartz2D.

AIR is more focused on applications with static interfaces and custom window chrome. That's where Quartz 2D makes more sense. For some AIR application types Core Animation would make sense, but I can't tell you when that support will be added.

Thursday, February 18, 2010 10:04:00 AM  
Blogger Gavin said...

Most of this is way above my understanding, but my basic problem is that each time there is a Flash upgrade the performance decreases dramatically. Yes, you may be able to place transparent ads on top of the video, but if the video refreshes so slowly it becomes unwatchable then it somewhat defeats the object.

Isn't there any way of including backwards compatibility into new versions? Flash upgrades have made my 4-year-old 2GHz PPC Mac obsolete, at the same time turning thousands of high-spec machines into landfill, at a huge economic and environmental cost.

"You will need Flash Player 10.1, OS X 10.6 and updated version of Safari (or the nightly WebKit build), otherwise you will not see anything."
- the final nail in the coffin, as pre-Intel Macs are stuck at OS X 10.5.

Two years ago I could happily watch any video content on the net without any stutter. Toady I have the choice of using Flash 8 (best) or Flash 9 and getting "This content requires the latest version of Flash Player" messages on many sites, include Google Street View, or using Flash 10 and not being comfortably able to watch any video. There is also no way (that I know of) of having Flash 8 and Flash 10 installed on the same machine, even with two different user accounts; upgrading Flash affects all installations, and re-installing Flash 8 is a mammoth task , involving uninstalling Flash 10 first.

Surely it must be possible to include a 'simple mode' which is compatible with older versions of Flash as alternative content? Using the same technology that detects older versions and gives the "Please install the latest version..." message you could then give the option to load the compatible content.

Saturday, February 20, 2010 3:42:00 AM  
Blogger The_Decryptor said...

Is there an easy way to see if the plugin is using Cocoa or Carbon?

It'd be interesting to have a test page that could tell you want APIs were involved.

Saturday, February 20, 2010 8:28:00 PM  
Blogger Tinic Uro said...

@The_Decryptor: Install the developer tools and run Shark on the Flash plugin. Among the symbols you will see stuff like this:

0.1% 0.1% QuartzCore x_heap_malloc
0.1% 0.1% QuartzCore CALayerSetNextAnimationTime
0.1% 0.1% QuartzCore CALayerRelease
0.1% 0.1% QuartzCore CALayerCollectAnimations_(CALayer*, double, double*, collect_closure*)
0.1% 0.1% QuartzCore CA::Transaction::get_value(unsigned int, _CAValueType, void*)
0.1% 0.1% QuartzCore CA::Transaction::ensure_(CA::Transaction*)
0.1% 0.1% QuartzCore CA::Transaction::commit()
0.1% 0.1% QuartzCore CA::Render::Surface::retain_iosurface() const
0.1% 0.1% QuartzCore CA::Context::retain_context_with_id(unsigned int)
0.1% 0.1% QuartzCore -[CALayer setContents:]
0.1% 0.1% QuartzCore -[CALayer contents]
0.1% 0.1% QuartzCore -[CALayer bounds]

Saturday, February 20, 2010 8:35:00 PM  
Blogger Tom Dibble said...

Petter -

So if Adobe wanted to make the Hulu / YouTube experience better performance-wise (a big 'if') on the Mac, it would seem to make sense for them to switch between the non-overlay approach (using Quicktime APIs) versus overlayable approach (using QuickDraw/etc) as needed, right? Is there a problem with switching from one to another within the plugin?

I mean, with Hulu on, 95% of the time or better, there's no transparent overlay on the screen. There's just the large video frame playing back H.264 which has been streamed in, with some controls along the edges. There's the couple seconds here and there where the playback controls come on the screen (which aren't transparent, so would Flash need to thunk down from Quicktime for that?), and I suppose every now and then there's an overlay ad (but I honestly can't remember this happening on Hulu).

It seems the "best of both worlds" approach. Use the high-performance approach where it is possible, which just happens to be the vast majority of the playback time, and then thunking down to the CPU-heating mode only when necessary. Yeah, it wouldn't end up helping those places which have a semi-transparent overlay on the screen the whole time, but it seems like it would go a very long way towards reducing the ill-will towards Flash in general that you see right now.

Saturday, February 20, 2010 8:45:00 PM  
Blogger The_Decryptor said...

While not as easy as I would have liked, Shark did work, showed it using Quartz 2D for Firefox and Core Animation for Safari (And hopefully soon Firefox as well, although there's some crash bugs I think)

Didn't see any signs about Cocoa or Carbon though, which is what I'm really interested in (Since Firefox supports the Cocoa plugin event model, so I'd like to see if Flash is using it or the older Carbon model)

Also, the gradient syntax you've specified on the demo page is outdated for Firefox, it switched just before the 3.6 release to a newer syntax that was proposed.

Saturday, February 20, 2010 9:44:00 PM  
Anonymous Anonymous said...

Was it necessary to see S. Jobs beatching on Flash to finally see some (very capable) Adobe answers.
Too much politics for a simple engineering problem.

Saturday, February 20, 2010 10:37:00 PM  
Blogger scroll said...

Ok so your getting around to it! What do you want, a medal?

More like the dog house for bringing up QuickDraw to somehow claim that the OSX graphics environment is a jungle - a very desperate attempt to present false complexity to Apple's OS.

Admit it, Jobs caught Adobe with its pants down. It's just too bad the picture is now banned from the app store!

Saturday, February 20, 2010 10:55:00 PM  
Anonymous Alex Bischoff said...

Just to be sure, I presume Adobe is aware of Firefox's work toward supporting the Core Animation drawing model? (bug 497225)

Saturday, February 20, 2010 10:59:00 PM  
Blogger Tinic Uro said...

@scroll: a medal would be nice, yes. I'll ask Steve about that next time I see him.

Saturday, February 20, 2010 11:02:00 PM  
Blogger Tinic Uro said...

@Alex Bischoff: Yes we know about this planned feature. There are still some technical issues which need to be ironed out before a patch can land.

Saturday, February 20, 2010 11:08:00 PM  
Blogger ffx said...

Hey Tinic,

could you also add CoreAnimation to 10.5? Why it is 10.6 only?

Also, could you write a bit about the new WMODEs like "direct" and "gpu"? Are there any changes from 10.0 -> 10.1 ?

Thanks!

Saturday, February 20, 2010 11:38:00 PM  
Blogger Jonas said...

You guys seem to be doing good work, but Adobe made a huge mistake to not carefully program Flash on the Mac from the very beginning. You just thought that because Macs were a small market that it didn't matter. Mac users have always hated Flash because it was a CPU hog, so no WONDER Apple has this long history with Flash pissing on the Mac and so they say "if Flash is terrible on the Mac think how it will be on the iPhone". You guys thought Macs didn't matter, you made Flash slow, and now you're paying the price.

Saturday, February 20, 2010 11:46:00 PM  
Blogger Tinic Uro said...

@ffx: While 10.5 was the first OS X version with Core Animation support it did have some problems (specifically with CAOpenGLLayer) which seem to have been addressed in 10.6. Sure, it probably could be fixed up, but I'd rather see everyone looking forward, not backwards.

'Direct' and 'GPU' wmode are now served using Core Animation/CAOpenGLLayer. There is no reason to distinguish on OS X anymore. Different story on other platforms obviously.

Saturday, February 20, 2010 11:51:00 PM  
Anonymous Anonymous said...

A simple colored lines and circle drawing "Web-plugin-Application" with hundreds and hundreds of open security holes shouldn't have access to a bottom layer API or graphics card memory! Adobe, you don't get it! Do you?

Flash is dead!

I really think about uninstalling the Flash-Plugin once for all, rather to install the 10.1 version...

Sunday, February 21, 2010 12:03:00 AM  
Blogger Tinic Uro said...

@Jonas: That's quite an accusation. Have you ever seen the code?

Sunday, February 21, 2010 12:04:00 AM  
Anonymous Anonymous said...

thank you for censoring my statement.

and hail the communist parties remembering tiananmen beijing.

Sunday, February 21, 2010 12:05:00 AM  
Blogger The EAM-Dude said...

But CoreAnimation allows you to to layer ads and other stuff on top of H.264 so I still dont see why Adobe couldnt just use the native API:s. Especially as HTTP Live Streaming of H.264 will become very common.

On the other hand I am getting more and more irritated that we no longer have a free and open web - that so much content depend on a proprietary, locked binary that only one company controls.

This is the real danger/problem. Not if Adobe is lazy or not. Or if Flash is dead or not. It is o.k. to use Flash as long as the site provides an alternative for devices that runs on open standards.

The release of the iPad will make this problem very clear.

Sunday, February 21, 2010 12:08:00 AM  
Blogger Tinic Uro said...

@Anonymous: I do not censor anything on this blog. But I reject comments with profanity because it is the right thing to do.

Sunday, February 21, 2010 12:10:00 AM  
Blogger Tinic Uro said...

@The EAM-Dude: "On the other hand I am getting more and more irritated that we no longer have a free and open web - that so much content depend on a proprietary, locked binary that only one company controls."

It certainly would be great to see Chrome, Firefox and Opera on the iPad.

Sunday, February 21, 2010 12:20:00 AM  
Blogger Will said...

What does the majority of the internet use with respect to wmode? Do the major sites (Youtube, Hulu etc.) use Normal mode? And do Flash ads predominantly use Normal?

Sunday, February 21, 2010 12:53:00 AM  
Blogger Tinic Uro said...

@Will: The answer to that is complicated. In many cases publishers will choose wmode=opaque or wmode=transparent because they want to float HTML on top (usually interstitials). It's hard to convince publishers that this is a bad idea for performance.

Sunday, February 21, 2010 12:57:00 AM  
Blogger Мої вірші said...

Gavin, I would recommend you using Click2Flash on your PowerPC Mac. It will not speed Flash up directly but indirectly by cutting off those flash ads that eat you CPU and you probably don't need them anyway.

Sunday, February 21, 2010 1:52:00 AM  
Anonymous Christian said...

It's easy to see why wmode=transparent is the obvoius choice for many sites. Take a hybrid webpage with a horizontal html navigation and dropdown menus for the second level. Directly below a Flash app which overlaps with the dropdowns. Pretty common, isn't it?

Sunday, February 21, 2010 1:59:00 AM  
Anonymous Seba said...

Thanks for the nice read, Tinic!

Got a question, tho - I truly hope you won’t forget the PPC Mac users out there. It’s the last Flash Player version for them and they are in DIRE need for some playback improvements to keep them going.

Sunday, February 21, 2010 6:44:00 AM  
Blogger Gavin said...

Мої вірші: thanks. I do already use ClickToFlash and it's a very useful tool (and a great example of how Flash are shooting themselves in the foot: the fact that programmers go to the trouble of creating a tool specifically to block Flash content shows how much of a problem it is).

However, unfortunately it doesn't solve the problem that I can no longer watch most video content on the web.

Sunday, February 21, 2010 8:37:00 AM  
Blogger Petter said...

@Tom Dibble

Yes you can use Core Animation to overlay controls on a video, however I'm not sure (I'm not working for Adobe or on Flash) it would be a great fit as it's more then a video player and the decoders and demuxers aren't quicktime decoders and demuxers any way, which would mean substantial work. Not only for Adobe but for their partners. It would be easier if Hulu could just provide a H.264 video in MP4 on HTTP (which can also play inside flash player) and use HTML5/qt directly for that. But I'm guessing they want some sort of DRM (I'm not in the US so I haven't cared for the details surrounding Hulu).

Also it's not like QTX is fully ready. It would be more complex to implement such a setup for RTMP too.

I'm sure they (Hulu) don't want to create a new streaming platform for just Mac users however. (Which would be the easiest.). So I guess it's flash or nothing.

Tinic is the man who could answer the technical questions here, that might not be as widely known any way. However they have clearly made the decision to not use the native APIs for video. There's no question that you could playback the videos with the native players. The MP4s need no special decoders or demuxers. Even if the streaming part might be harder. However it's clear they don't just want to be a video delivery runtime.

Sunday, February 21, 2010 1:26:00 PM  
Anonymous edgema said...

With all the talk of APIs and CPU cycles, why ignore RAM? Why does the new 10.1 player need to use 55% MORE memory than the previous one?

Sunday, February 21, 2010 1:55:00 PM  
Anonymous Anonymous said...

In this upcoming release of Flash, will you finally take out the call to Debugger() please?!? ( http://www.stinkbot.com/blog/archives/69 )

Sunday, February 21, 2010 9:06:00 PM  
Blogger Radical said...

Around when will beta 3 be released?

Sunday, February 21, 2010 10:08:00 PM  
Blogger ffx said...

"Sure, it probably could be fixed up, but I'd rather see everyone looking forward, not backwards."

Tinic, but I think a lot of people still use Leopard. If you look at statistics, it's still much more used than Snow Leopard.

Also Snow Leopard does not support PPC and has not the best gui performance if you look at some pro apps (for example Logic Audio). So there are enough reasons for people to stay at Leopard which is quite solid since 10.5.7.

Maybe your problem is fixed in Leopard 10.5.7? Please consider adding your new features to Leopard, too. Thanks.

Monday, February 22, 2010 1:39:00 AM  
Anonymous Tim said...

I assume that when you say there's only one way to draw on Windows that you mean there is only one way that browsers currently use. Is this correct?

Otherwise the drawing model jungle on Windows looks like this: GDI, GDI+, DirectX, OpenGL. Both OSes have their modern and legacy drawing APIs.

It's neither Adobe's nor Apple's fault (excepting Safari) if the browser maker is stuck using deprecated APIs.

Monday, February 22, 2010 3:38:00 AM  
Blogger Petter said...

Actually your forgetting a few, you who referred to Windows. Direct2D, deprecated DirectDraw and WPF is available too.

Direct2D is what browsers should support now. DX including legacy thus is actually Direct3D (in all it's versions), Direct2D which is the new, DirectDraw which is the old. There's also a few oddities such as DirectAnimation. Not to mention that Adobe had to tell the graphics vendors to support flash and give low-level access to the video hardware acceleration. It's not something they could do via Microsofts APIs by default. Becasue they don't render the video in EVR (Vista/7) or use directshow/media foundation at all for video decoding.

Monday, February 22, 2010 8:03:00 AM  
Blogger Steven Fisher said...

Looks promising. What about subpixel text? Or does Flash already handle that manually anyway?

Monday, February 22, 2010 10:45:00 AM  
Blogger enriquepr said...

Tinic, this post was very well written. You have done a great job outlining some of the behind the scenes things that can be done to improve the flash player for everyone.

As I'm sure you know, ignore the naysayers and finger pointers. They have no idea how far flash has truly come over the years. There is alot of focus on the animosity towards flash but it should be redirected as energy to inform developers of good practices that will help utilize all the advancements being made in the flash player. That coupled with making good decisions when to use flash will really help bring the next generation of the web forward. To those that are slinging dirt and praising HTML5 as your savior, just go to your own respective forum of supporters and convert the negativity into positive influence.

I, for one, am optimistic for 10.1. It will open new doors for us on the personal computer platform as well as the mobile. A few of the unofficial demos of it running on android have been very inspiring!

Keep it up, Tinic. I look forward to some more posts!

Monday, February 22, 2010 10:58:00 AM  
Anonymous Pierre Lebeaupin said...

A little nitpick: what you call "Quartz 2D" (esp. in the support by various browsers) isn't so much Quartz 2D as it, plus the window content/control compositing model of Cocoa, where the controls are redrawn back to front; as opposed to the new Core Animation model where each control renders in its own Core Animation raster layer, making control recomposition much faster (at the expense of memory). In fact, most apps that use CoreAnimation draw into these layers using Quartz 2D, however an OpenGL layer makes sense for blitting content such as video.

Monday, February 22, 2010 11:05:00 AM  
Blogger casa_poker said...

A lovely informative post, thank you.

Curious if there will be a public Beta3 of the 10.1 plugin?

Monday, February 22, 2010 11:41:00 AM  
Blogger Andrew said...

Hi,

I grabbed Flash 10.1 Beta 3 today, and combined that with a recent WebKit nightly, but performance doesn't seem to be any better. Is there any way that I can tell that the Core Animation sauce is working?

Wednesday, February 24, 2010 11:08:00 AM  
Blogger ffx said...

The yesterday released 10.1 beta 3 runs a bit slower on my machine than recent 10.0... As well there seems to be a problem with screen sync now.

Please do some improvements for all OSX computers. Thank you.

Wednesday, February 24, 2010 2:31:00 PM  
Blogger ffx said...

And in 10.1 beta 3 videos look not as good as in 10.0. I can see much more compression artifacts.

My machine is a Leopard Core2Duo HD2600.

Thursday, February 25, 2010 8:02:00 AM  
Blogger ffx said...

wmode = "direct" is still a pain in Firefox Mac (3.6.2). The best seems to be wmode "opaque" for Firefox Mac. Could you improve direct mode for Firefox?

Thursday, February 25, 2010 9:13:00 AM  
Blogger Zav said...

Just make it stop crashing Safari and stop making it a resource pig. Flash is the #1 reason Safari crashes for me. It makes me hate Flash.

And if Werner's still on the team, tell him Zav said Hi.

Sunday, February 28, 2010 2:35:00 PM  

Post a Comment

<< Home