Builds, builds and more builds
Following up some requests to see more what happens behind scenes here at Adobe I thought I talk a little bit about our daily build process. There is not too much terribly exciting as we follow common industry practices, but some numbers may be surprising for some.
First and foremost everything here centers around SCM (Source code management). Without it nothing would get done. Every kind code and documentation we create is manipulated with it. The tool of choice for us is Perforce and after the merger between Macromedia and Adobe we were happy to find out that Adobe was using it all across the company. It might not have the fancy features other newer solutions have, but it's always been rock solid, fast and has excellent client tools. A long time ago (Flash 3&4) we were using CVS which proved to be too primitive in the end though (and yes I know, everyone is migrating to subversion at this point).
Every time we make a change to the Flash Player we face the challenge that every single project and target still needs to compile and work (for those who know, we compile with the '-Werror' option). Normally it takes a few hours to rebuild all of our projects, but we employ a few tools to help engineers find build issues. One such tool is IncrediBuild which allows to check all Windows builds within a few minutes. Also, every time an engineer checks in a change, a server will automatically start builds of all targets and send out an email if a build failed including the offending change list number. On OS X we use distcc which is part of XCode to achieve the same. A luxury we do not have with CodeWarrior (typically a full build of one target in CodeWarrior takes 20-30 minutes). We do not have a distcc setup for Linux yet, but Mike is working hard on making this happen right now. That's the reason he ordered a few Gentoo servers recently.
What is a target? Every project has 4 targets: "Release", "Release Debugger", "Debug" (update: left this one out originally) and "Debug Debugger". The "Debugger" targets contain code for debugging ActionScript whereas the "Debug" targets are non-optimized binaries with a full set of symbols for debugging C++ code. Every project we support has these targets and are build daily. Our set of standard projects we build every day are:
That makes 40 binaries which are build overnight plus a wide variation of installation packages for different deployment scenarios. Quite a lot of stuff and that only represents our mainline builds. We keep separate branches for all older versions and special ongoing projects (which I can't talk about ;-). So in the worst case we could end up with a couple of hundred binaries and installer packages on a given day. Why do we do this? Typically QE (quality engineering) needs to confirm fixed bugs in every binary. If injections are found it is valuable to be able to quickly find the build a bug was injected by testing older builds.
Talking about bugs, we use like almost everyone in the industry a bug database which not only holds information on bugs, but also engineering tasks, ECRs (Enhancement/feature requests, usually from users) and other technical information which needs to be tracked and followed up on, f.ex. in tech notes. Currently the system consists of a SQL database, ColdFusion with an easy to use forms based HTML front-end. Nothing really special. But this is probably the one tool everyone in the team uses, including our beta testers, although they see a somewhat different interface.
Something which might not be well know, but every time you submit a bug report or feature request through the wish form we have a human verify the submission and add a new entry to our bug database. It's not automatic, as you can imagine we get a lot of spam and inappropriate stuff through this medium. Although sometimes feedback can be outright amusing...
First and foremost everything here centers around SCM (Source code management). Without it nothing would get done. Every kind code and documentation we create is manipulated with it. The tool of choice for us is Perforce and after the merger between Macromedia and Adobe we were happy to find out that Adobe was using it all across the company. It might not have the fancy features other newer solutions have, but it's always been rock solid, fast and has excellent client tools. A long time ago (Flash 3&4) we were using CVS which proved to be too primitive in the end though (and yes I know, everyone is migrating to subversion at this point).
Every time we make a change to the Flash Player we face the challenge that every single project and target still needs to compile and work (for those who know, we compile with the '-Werror' option). Normally it takes a few hours to rebuild all of our projects, but we employ a few tools to help engineers find build issues. One such tool is IncrediBuild which allows to check all Windows builds within a few minutes. Also, every time an engineer checks in a change, a server will automatically start builds of all targets and send out an email if a build failed including the offending change list number. On OS X we use distcc which is part of XCode to achieve the same. A luxury we do not have with CodeWarrior (typically a full build of one target in CodeWarrior takes 20-30 minutes). We do not have a distcc setup for Linux yet, but Mike is working hard on making this happen right now. That's the reason he ordered a few Gentoo servers recently.
What is a target? Every project has 4 targets: "Release", "Release Debugger", "Debug" (update: left this one out originally) and "Debug Debugger". The "Debugger" targets contain code for debugging ActionScript whereas the "Debug" targets are non-optimized binaries with a full set of symbols for debugging C++ code. Every project we support has these targets and are build daily. Our set of standard projects we build every day are:
- Flash.ocx (Win32 x86)
- NPSWF32.dll (Win32 x86)
- authplay.dll (Win32 x86)
- standalone.exe (Win32 x86)
- FlashPlayer.plugin (PPC 32-bit 10.1 or newer)
- FlashPlayer.plugin (Universal 32-bit 10.4 or newer)
- AuthPlay.bundle (Universal 32-bit 10.4 or newer)
- Standalone.app (Universal 32-bit 10.4 or newer)
- libflashplayer.so (Linux x86 32-bit)
- gflashplayer (Linux x86 32-bit)
That makes 40 binaries which are build overnight plus a wide variation of installation packages for different deployment scenarios. Quite a lot of stuff and that only represents our mainline builds. We keep separate branches for all older versions and special ongoing projects (which I can't talk about ;-). So in the worst case we could end up with a couple of hundred binaries and installer packages on a given day. Why do we do this? Typically QE (quality engineering) needs to confirm fixed bugs in every binary. If injections are found it is valuable to be able to quickly find the build a bug was injected by testing older builds.
Talking about bugs, we use like almost everyone in the industry a bug database which not only holds information on bugs, but also engineering tasks, ECRs (Enhancement/feature requests, usually from users) and other technical information which needs to be tracked and followed up on, f.ex. in tech notes. Currently the system consists of a SQL database, ColdFusion with an easy to use forms based HTML front-end. Nothing really special. But this is probably the one tool everyone in the team uses, including our beta testers, although they see a somewhat different interface.
Something which might not be well know, but every time you submit a bug report or feature request through the wish form we have a human verify the submission and add a new entry to our bug database. It's not automatic, as you can imagine we get a lot of spam and inappropriate stuff through this medium. Although sometimes feedback can be outright amusing...
posted by Tinic Uro at 12:04 PM
8 Comments:
This is very interesting info.. thanks.
I noticed 2 targets for the linux player.. one an .so (So I assume the plugin) and one not. Does that mean a stand-alone linux flash player as well as a plugin is being worked on?
Hello.
I've seen that you are talking about the player for linux. What is this gflasplayer. Is it the v9 player for it?
gflashplayer is a standalone flash player using GTK (hence the g at the beginning, both GTK1.2 an GTK2.x are supported). It is not not even close to being on par with the plug-in version right now (most of the networking, RTMP, file access etc. are non-functional). I can't make any sort of definite announcements or predict the future concerning this project.
Right now it is mostly used for developer to debug specific low level issues (AS3 x86 JIT, garbage collection, rendering bugs etc.) as sometimes it is not possible to do this in a browser environment when using gdb.
Nevertheless our Linux devs will continue to improve it. We know that people out there want it, sometimes as badly as the plug-in version.
Hello,
You said 'Every project has 4 targets: "Release", "Release Debugger" and "Debug Debugger".'
Four targets but three mentioned, so I was wondering what is the fourth one.
As after you mentioned it is 10 set of standard projects giving 40 binaries, it doesn't seem that 4 mentioned previously is not a typing error.
So I am just curious about what is this fourth unmentioned one.
Thanks for sharing your internal processes and experience.
The 4th one is "Debug"
Is there a Flash Debug Player for Intel Macs? I just got an intel mac and the debug player installer doesn't work and the player it points you to on the web isn't a debug player. I'm trying to use it for Flex applications I work with.
I like articles like this. Thanks!
Well done. Keep up the great work. Best regards!
Post a Comment
<< Home