Friday, September 05, 2008

Pixel Bender .pbj files

If you have been playing with Pixel Bender in Flash Player 10 you know the workflow:
  • Create your .pbk in the Pixel Bender Toolkit.
  • Export a .pbj binary from the Pixel Bender Toolkit.
  • Embed or dynamically load the .pbj file in your ActionScript.
There is still some mystery around .pbj files, i.e. the file format is neither documented nor is it clear what exactly it contains. While I can't offer documentation on the file format at this time (although that will happen eventually) what I can offer is an assembler and disassembler I quickly hacked together. I am mostly using for this my own debugging purposes. For those who want to tweak Pixel Bender in Flash to the max this is a really good way to go.

Please note that this is neither officially supported by Adobe nor do I guarantee any correctness or completeness of these two tools. The binary format could change at any time, you should not rely on it. I am really throwing this out to the world for educational purposes. There is no documentation on the syntax or format nor how to put together kernels from scratch. It's really up to you to make any sense out of it and I do not expect that any Adobe will ever use this syntax since I just made it up myself. Since these are a quite advanced tools I also assume you know how to compile C++ command line tools yourself ;-)

Enough disclaimers, here is the meat:

http://www.kaourantin.net/source/pbjtools/apbj.cpp
http://www.kaourantin.net/source/pbjtools/dpbj.cpp

But for the lazy I have pre-compiled two Windows binaries of these two command line tools. On OSX you can simply compile these using 'g++ apbj.cpp -o apbj' and 'g++ dpbj.cpp -o dpbj' if you have the developer tools installed on your system. Here are the Windows binaries:

http://www.kaourantin.net/source/pbjtools/apbj.zip
http://www.kaourantin.net/source/pbjtools/dpbj.zip

It should be fairly clear on how to use these if you run them in a command prompt. One of the goals was to allow perfect round tripping, i.e. disassemble->assemble->disassemble without any information loss. I hope I succeeded in this. I also know that someone has already put together an ActionScript version of this which is quite easily done.

Here is some sample output of the disassembler, the .pbj file was based on a Pixel Bender kernel created by Mr.doob (and btw, I fixed the bug he noticed in his blog entry ;-):


version 1
name "NewFilter"
kernel "namespace", "Hypnotic"
kernel "vendor", "Mr.doob"
kernel "version", 1
kernel "description", "Hypnotic effect"

parameter "_OutCoord", float2, f0.rg, in

texture "src", t0

parameter "dst", float4, f1, out

parameter "imgSize", float2, f0.ba, in
meta "defaultValue", 512, 512
meta "minValue", 0, 0
meta "maxValue", 512, 512

parameter "center", float2, f2.rg, in
meta "defaultValue", 256, 256
meta "minValue", 0, 0
meta "maxValue", 512, 512

parameter "offset", float2, f2.ba, in

;----------------------------------------------------------

mov f3.rg, f0.rg
sub f3.rg, f2.rg
rcp f3.ba, f0.ba
mul f3.ba, f3.rg
mov f3.rg, f3.ba
set f3.b, 3.14159
mov f3.a, f3.g
atan2 f3.a, f3.r
mov f4.r, f3.a
set f3.a, 2
mov f4.g, f3.r
pow f4.g, f3.a
set f3.a, 2
mov f4.b, f3.g
pow f4.b, f3.a
mov f3.a, f4.g
add f3.a, f4.b
sqr f4.g, f3.a
mov f3.a, f4.g
set f4.g, 0
set f4.b, 0
set f4.a, 0
add f4.g, f2.b
add f4.b, f2.a
cos f5.r, f4.r
rcp f5.g, f3.a
mul f5.g, f5.r
add f4.g, f5.g
sin f5.r, f4.r
rcp f5.g, f3.a
mul f5.g, f5.r
add f4.b, f5.g
set f5.r, 1
set f5.g, 0.1
mov f5.b, f3.a
pow f5.b, f5.g
rcp f5.g, f5.b
mul f5.g, f5.r
add f4.a, f5.g
mul f4.g, f0.b
mul f4.b, f0.a
set f5.r, 0
ltn f4.g, f5.r
mov i1.r, i0.r

if i1.r

set f5.r, 0
sub f5.r, f4.g
rcp f5.g, f0.b
mul f5.g, f5.r
ceil f5.r, f5.g
mov f5.g, f0.b
mul f5.g, f5.r
add f4.g, f5.g

end

set f5.r, 0
ltn f4.b, f5.r
mov i1.r, i0.r

if i1.r

set f5.r, 0
sub f5.r, f4.b
rcp f5.g, f0.a
mul f5.g, f5.r
ceil f5.r, f5.g
mov f5.g, f0.a
mul f5.g, f5.r
add f4.b, f5.g

end

ltn f0.b, f4.g
mov i1.r, i0.r

if i1.r

rcp f5.r, f0.b
mul f5.r, f4.g
floor f5.g, f5.r
mov f5.r, f0.b
mul f5.r, f5.g
sub f4.g, f5.r

end

ltn f0.a, f4.b
mov i1.r, i0.r

if i1.r

rcp f5.r, f0.a
mul f5.r, f4.b
floor f5.g, f5.r
mov f5.r, f0.a
mul f5.r, f5.g
sub f4.b, f5.r

end

mov f5.r, f4.g
mov f5.g, f4.b
texn f6, f5.rg, t0
mov f1, f6
mul f1.rgb, f4.aaa

Wednesday, September 03, 2008

On Performance

With the release of Google Chrome I see blogs and articles blaming the Flash Player for poor performance and somehow linking this to the fact that it is not open source. Time to clarify a few bits. I'll start with classic comments:

"Flash hogs my CPU!"

1. HTML != Flash

HTML is a static document format. Flash (TM) content is in its core a classic multimedia format and most Flash content is still purely passive media.

What does that mean? When rendering HTML pages CPU usage only peaks for a very short of amount of time, essentially one single frame in Flash terms. After that almost no resources apart from memory are required. If you do not interact with the HTML page at all, no CPU time is required.

How does Flash compare? Most animated Flash content like rich media advertisement continues to use CPU resources to drive animation, video and/or sound. As opposed to static HTML which has exactly 1 frame, Flash content can have an infinite amount of frames which are played back over time.

Flash is great to provide experiences you could not get otherwise. Animation, video and sound are functions the browser does not (yet) provide, or at least they are not used to the same extend yet by designers. Once the browser will be used to play the same type of multimedia content you will face the same resource usage issues. It takes CPU cycles to decode video, sound and render animation. This is just a fact of life, we are however improving how much is used release after release, something benchmarks can back up.

So, there is a fundamental difference in media type. HTML is static, Flash is not. To put it in terms you might be able to understand:

If you take a picture and print it out you use energy only once and then can continue to view the picture forever without consuming any further energy. If you record a movie you will need some form of machine to play it back which will continue consume energy in form of a projector. The Flash Player is a projector.

"You are so full of it, AJAX does not hog my CPU!"

2. AJAX != Flash, but when done correctly AJAX can be the same as Flash

In most practical instances AJAX is used to drive RIAs. Examples include Gmail, Google Maps and many others. One fundamental property of good applications is that they only respond to either network activity or user input. Peak CPU usage is limited to these events. In general, if you do not touch the browser page no CPU time is required.

Compare this again to Flash animations, video and sound which in many cases remain passive experiences with no requirement for external events to drive the content. This will obviously use CPU resources continuously.

Now, it is perfectly possible to implement a Flash RIA application (that usually means using Flex) which uses the same or even less peak CPU than a AJAX RIA and only responds to network and user input. Flash is a flexible multiple paradigm platform, it depends on what the designer/programmer wanted to do. Unfortunately we at Adobe tend to see of a lot of RIAs which do not follow that principle and add lots of moving sugar to their applications which do little to improve actual usability.

Following good coding practices Flash can yield equivalent or better results than AJAX for many types of RIAs. Another benefit is that writing RIAs in Flash is truly cross browser as there is one Flash Player implementation only.

"Bull, SVG and Canvas show that it can be done better"

3. SVG/Canvas != Flash

Have you ever seen SVG or the canvas tag being used to implement anything else than static (1-frame) content? Have you ever seen rich media advertisement done using SVG or the Canvas tag? I mean not some demo page but actual deployed content. If so you will realize that the same resource usage issues apply.

"You are clueless, why does Flash suck up CPU time when it is on a hidden tab?"

4. Easy shortcuts do not work

Believe it or not but we and the browser vendors have tried to disable/pause/stop Flash content when a tab is hidden. The results were disastrous user experience wise to say the least. Disabling Flash to get any benefit CPU resource wise means the following:
  • Sound will have to stop
  • Any network transfer will have to stop
  • ActionScript execution will have to stop

Each one of these affect CPU resource usage and would affect user experience if we would turn it off. However the Flash Player does not render anything if it is on a hidden tab, we only execute the operations mentioned in the above list.

There is one exception to the rendering optimization: WMODE. If you use WMODE the Flash Player has no way of knowing if it is hidden or not and will continue to do a full render. Do not use WMODE. Unfortunately lots of rich media advertisement I see out there continues to enable this for no apparent reason.

"Flash sucks!"

5. You can help to educate web designers so common mistakes are not made

Huge help would be to adopt strict policies especially for rich media advertisement. I like the rules Google has put forward for Flash ads. Quoting:

"Animation Length: Animated ads are restricted to a maximum of 15 seconds (at a 15-20 fps frame rate), after which point they must remain static. These ads must also comply with the other animation policies."

Personally I would go even further and request the following:

  • After the animation has played no CPU resources should be used, ActionScript should be on a stop() command.
  • Mouse tracking or other event handling is not allowed unless you activate the banner with a mouse click.
  • DO NOT USE WMODE UNLESS YOU ABSOLUTELY NEED TRANSPARENCY! I can't stress that enough. Given the architecture of plugins there is no way for the Flash Player to know if Flash content is on a hidden tab or not and disable rendering properly. If you use WMODE the Flash Player will continue to suck up CPU cycles as if the tab was visible. In addition WMODE is much slower than the normal mode.
These simple rules would address almost all the complaints we hear about. Adobe has unfortunately only limited influence on what content gets deployed, in this case it is really up to the community to balk at the web sites putting up content which impacts user experience negatively.

--------------------

Like with any powerful technology it is easy to shoot yourself in the foot and with the ease of use of Flash that is unfortunately too common.

Despite of that we are working with all browser vendors to improve performance and user experience whenever possible. There are differences between browsers and our goal is to close this gap once and for all. We are for example looking forward to work together with Google to improve Flash performance in Google Chrome.

On our (Adobe) side we are also looking forward to improve Flash performance further. Flash Player 10 for instance is making the first steps towards hardware accelerated rendering which will provide a huge boost in rendering performance. On the scripting side Tamarin-tracing will improve scripting performance dramatically. This is work we share with the Mozilla foundation which will use the same core libraries under the TaceMonkey project. The latest benchmarks are quite remarkable.