Builds, builds and more builds
First and foremost everything here centers around SCM (Source code management). Without it nothing would get done. Every kind code and documentation we create is manipulated with it. The tool of choice for us is Perforce and after the merger between Macromedia and Adobe we were happy to find out that Adobe was using it all across the company. It might not have the fancy features other newer solutions have, but it's always been rock solid, fast and has excellent client tools. A long time ago (Flash 3&4) we were using CVS which proved to be too primitive in the end though (and yes I know, everyone is migrating to subversion at this point).
Every time we make a change to the Flash Player we face the challenge that every single project and target still needs to compile and work (for those who know, we compile with the '-Werror' option). Normally it takes a few hours to rebuild all of our projects, but we employ a few tools to help engineers find build issues. One such tool is IncrediBuild which allows to check all Windows builds within a few minutes. Also, every time an engineer checks in a change, a server will automatically start builds of all targets and send out an email if a build failed including the offending change list number. On OS X we use distcc which is part of XCode to achieve the same. A luxury we do not have with CodeWarrior (typically a full build of one target in CodeWarrior takes 20-30 minutes). We do not have a distcc setup for Linux yet, but Mike is working hard on making this happen right now. That's the reason he ordered a few Gentoo servers recently.
What is a target? Every project has 4 targets: "Release", "Release Debugger", "Debug" (update: left this one out originally) and "Debug Debugger". The "Debugger" targets contain code for debugging ActionScript whereas the "Debug" targets are non-optimized binaries with a full set of symbols for debugging C++ code. Every project we support has these targets and are build daily. Our set of standard projects we build every day are:
- Flash.ocx (Win32 x86)
- NPSWF32.dll (Win32 x86)
- authplay.dll (Win32 x86)
- standalone.exe (Win32 x86)
- FlashPlayer.plugin (PPC 32-bit 10.1 or newer)
- FlashPlayer.plugin (Universal 32-bit 10.4 or newer)
- AuthPlay.bundle (Universal 32-bit 10.4 or newer)
- Standalone.app (Universal 32-bit 10.4 or newer)
- libflashplayer.so (Linux x86 32-bit)
- gflashplayer (Linux x86 32-bit)
That makes 40 binaries which are build overnight plus a wide variation of installation packages for different deployment scenarios. Quite a lot of stuff and that only represents our mainline builds. We keep separate branches for all older versions and special ongoing projects (which I can't talk about ;-). So in the worst case we could end up with a couple of hundred binaries and installer packages on a given day. Why do we do this? Typically QE (quality engineering) needs to confirm fixed bugs in every binary. If injections are found it is valuable to be able to quickly find the build a bug was injected by testing older builds.
Talking about bugs, we use like almost everyone in the industry a bug database which not only holds information on bugs, but also engineering tasks, ECRs (Enhancement/feature requests, usually from users) and other technical information which needs to be tracked and followed up on, f.ex. in tech notes. Currently the system consists of a SQL database, ColdFusion with an easy to use forms based HTML front-end. Nothing really special. But this is probably the one tool everyone in the team uses, including our beta testers, although they see a somewhat different interface.
Something which might not be well know, but every time you submit a bug report or feature request through the wish form we have a human verify the submission and add a new entry to our bug database. It's not automatic, as you can imagine we get a lot of spam and inappropriate stuff through this medium. Although sometimes feedback can be outright amusing...