Builds, builds and more builds

Following up some requests to see more what happens behind scenes here at Adobe I thought I talk a little bit about our daily build process. There is not too much terribly exciting as we follow common industry practices, but some numbers may be surprising for some.

First and foremost everything here centers around SCM (Source code management). Without it nothing would get done. Every kind code and documentation we create is manipulated with it. The tool of choice for us is Perforce and after the merger between Macromedia and Adobe we were happy to find out that Adobe was using it all across the company. It might not have the fancy features other newer solutions have, but it's always been rock solid, fast and has excellent client tools. A long time ago (Flash 3&4) we were using CVS which proved to be too primitive in the end though (and yes I know, everyone is migrating to subversion at this point).

Every time we make a change to the Flash Player we face the challenge that every single project and target still needs to compile and work (for those who know, we compile with the '-Werror' option). Normally it takes a few hours to rebuild all of our projects, but we employ a few tools to help engineers find build issues. One such tool is IncrediBuild which allows to check all Windows builds within a few minutes. Also, every time an engineer checks in a change, a server will automatically start builds of all targets and send out an email if a build failed including the offending change list number. On OS X we use distcc which is part of XCode to achieve the same. A luxury we do not have with CodeWarrior (typically a full build of one target in CodeWarrior takes 20-30 minutes). We do not have a distcc setup for Linux yet, but Mike is working hard on making this happen right now. That's the reason he ordered a few Gentoo servers recently.

What is a target? Every project has 4 targets: "Release", "Release Debugger", "Debug" (update: left this one out originally) and "Debug Debugger". The "Debugger" targets contain code for debugging ActionScript whereas the "Debug" targets are non-optimized binaries with a full set of symbols for debugging C++ code. Every project we support has these targets and are build daily. Our set of standard projects we build every day are:

• Flash.ocx (Win32 x86)


• NPSWF32.dll (Win32 x86)


• authplay.dll (Win32 x86)


• standalone.exe (Win32 x86)


• FlashPlayer.plugin (PPC 32-bit 10.1 or newer)


• FlashPlayer.plugin (Universal 32-bit 10.4 or newer)


• AuthPlay.bundle (Universal 32-bit 10.4 or newer)


• Standalone.app (Universal 32-bit 10.4 or newer)


• libflashplayer.so (Linux x86 32-bit)


• gflashplayer (Linux x86 32-bit)

That makes 40 binaries which are build overnight plus a wide variation of installation packages for different deployment scenarios. Quite a lot of stuff and that only represents our mainline builds. We keep separate branches for all older versions and special ongoing projects (which I can't talk about ;-). So in the worst case we could end up with a couple of hundred binaries and installer packages on a given day. Why do we do this? Typically QE (quality engineering) needs to confirm fixed bugs in every binary. If injections are found it is valuable to be able to quickly find the build a bug was injected by testing older builds.

Talking about bugs, we use like almost everyone in the industry a bug database which not only holds information on bugs, but also engineering tasks, ECRs (Enhancement/feature requests, usually from users) and other technical information which needs to be tracked and followed up on, f.ex. in tech notes. Currently the system consists of a SQL database, ColdFusion with an easy to use forms based HTML front-end. Nothing really special. But this is probably the one tool everyone in the team uses, including our beta testers, although they see a somewhat different interface.

Something which might not be well know, but every time you submit a bug report or feature request through the wish form we have a human verify the submission and add a new entry to our bug database. It's not automatic, as you can imagine we get a lot of spam and inappropriate stuff through this medium. Although sometimes feedback can be outright amusing...

Universal binary update, sound woes...

I am still busily working on the universal binary beta. So far the beta feedback has been very sparse (probably because penetration of Intel machines is still low), with half the complaints being around sound playback. The good news is that we can fix it for the Intel binary, the PowerPC binary will still fail in certain situations as we are uncomfortable changing the PowerPC binary too much at this point.

What situations? Well, until now the Flash Player was using the Carbon Sound Manager to output sound. It turns out if you change some system settings which some audio application like to do it breaks that API. It seems that the QuickTime player is also using the Sound Manager, so you do not get any sound playback there either when playing back a .mov.

To reproduce the bug in the universal binary beta (9.0r18 or earlier) go to Applications->Utilities->Audio MIDI Setup and change the properties for the 'Build-in Output', changing the format from 44100.0 Hz to f.ex. 96000.0 Hz. I used OS X 10.4.7 with QuickTime 7.1.1. Now, you'll notice that there is suddenly no audio anymore with any Flash content.

A user reported that Audacity is a source of this issue and indeed, if you just launch Audacity it always changes the system wide setting to 96000.0 Hz without reverting it to the old value when you quit. So when using older applications, use Audacity and suddenly have no sound output make sure to double check this setting.

What's the fix? Well, the easy way for me would be to fault the Sound Manager, it should support any sampling rate or have Audacity not change this setting unless asked for. On the other hand I know that the Sound Manager is some extremely old API which is basically there to support legacy applications only. Therefore I switched the whole sound output to use Core Audio which is probably the Right Thing to do at this point. Although as I said, we are uncomfortable with drastic changes like this, so we will not touch the PowerPC binary for now.

Udpate: QE has informed me that is is apparently not an issue with native PowerPC G4 and G5 machines as their sound hardware does not support anything higher than 48000.0 Hz with which Sound Manager has an issue. This seems to be the reason this only happens on Intel hardware. Well, except if you have a special sound card which supports 96000.0 Hz.

Random bits on current status

Update (10/31/06): I still see tons of people reading this old entry. Please read this for up to date information. A beta for Linux is now available.

Flash Player 9 is finally done and some of us can finally take a long deserved break. As I have been part of handling the engineering on the OS X universal binary I have been more busy than others after the release together with those who are handling the other ports and versions which need to get out. The OS X universal binary is in beta and I expect people to find lots of bugs. I've made numerous changes to the code base to make it work better with XCode/gcc. These changes will be moved it into the Linux builds, with some minor exceptions. It's certainly good to see that people seem to be happier with the performance compared to 8.0r27. Although I have to say, we probably hit the top limit of what is possible with the current architecture of the Flash Player. Everything else will have to wait until the next major version. My focus during this beta period will be purely on bugs.

I have been following the progress of the Linux version closely and have even managed to add a couple of change lists specifically for this version, even though I broke the build badly on Mike's Gentoo box recently. I decided to use Ubuntu to do my Linux builds since I am a .deb/apt-get/dselect guy and each one of the other engineers is using a different distribution. Amazing how many small good bugs pop up because of this.

Another few good bits come from a very well known engineer:

http://www.gibix.net/dokuwiki/en:projects:nspluginwrapper

In case you have missed this, it is a wrapper to run Netscape 32 bit plugins in a 64 bit environment. Some of us will probably soon try this in their spare time and see if we can make it work this way, despite its obvious limitations. Hopefully the requirement for low level OS constructs in Flash Player 9 (GC/JIT/mmap/exceptions/pthreads etc.) will not prevent it from working.

You might wonder why I bring this up obviously. Well, yes I know this will initiate a flame war, but there will be definitely no 64 bit version of the Linux plugin initially, at least not for the beta. Neither have we planned to have a Linux PowerPC or even ARM version for the upcoming beta. There is no Windows 64 bit version either right now, I have already talked about some (but not all) of the technical the reasons in a previous blog entry and why it's not just a simple recompile or fixing 'pointer casting bugs'. It will happen eventually and we are actively working on it, but the time table is completely unknown to me at this point. The focus of ours is to first reach a majority of Linux users with a stable and usable build to get as much beta feedback as possible.

Fun personal side note: I see that Gwenolé Beauchesne is maintaining the SheepShaver project. Did you know that I did draw the SheepShaver icon you see here? Really. This was back in my BeOS days. And yes, I suck as a pixeler ;-)

What else is there in the short term? I'll be actively working on updating some of my AS3 code which I posted here on my blog and rerelease it in some form, with hopes that people will pick it up. Also, next gen stuff is always in my mind.