A Modest Proposal

Chris Seawood and John Keiser
May 9, 2003

The mozilla project has produced a set of stable top-level libraries that are used by many outside projects, and its is time for its directory structure and build system to evolve to match this reality.  In particular, the Gecko Runtime Engine (GRE) is being embedded in multiple projects but it is still inexplicably necessary to build SeaMonkey (its main embedder) to get the GRE.

The Problems

There are a number of problems with our current build system and directory structure that we aim to solve:

- Inadequate separation of GRE and Seamonkey
- Unnecessarily complicated top-level Mozilla hierarchy (non-standalone libraries live at the same level with things like Rhino)
- Other embedders (like Firebird) require Seamonkey to be built to create the GRE, creating long build times
- Cross-breeding headers and interfaces between Seamonkey and GRE, making things difficult for other embedders
- It is currently impossible to update Firebird and Gecko separately because they are under the same tree.  For example, Firebird might want to pull GECKO_LATEST_STABLE the way it pulls the stable build of NSPR, allowing for major instable changes to be dumped to Gecko without breaking production nightlies.
- Release schedules of embedders in the tree are tied to the circadian rhythm of the GRE itself rather than allowing the Firebird owners to choose when to move the Gecko tag.

Given that we are moving from the old SeaMonkey / Mozilla browser code to the Firebird browser, it would be best to make this move so that Firebird is just a second embedder rather than replicating the problems we had with Seamonkey or (worse) building Seamonkey underneath when we build Firebird.

The Proposal

We propose to create a set of top-level directories that represent first-order libraries (like gre).  /mozilla becomes a sort of hosting directory, and the top-level libraries each have their own make systems (most of which would use our current autoconf build system) instead of having a top-level make that makes multiple libraries at once.

Building will be handled by building each library separately and pointing to the libraries it depends on with --with-gecko=../gecko or --with-js=../js, etc.  This allows you to build against a gecko from a different source tree; if --with- is not specified searching for the dependent libraries in the current tree will be automatic.  Ease of building is solved with client.mk-like solutions (like gecko/gecko.mk and mozilla/apps/browser/browser.mk).  A mozilla/client.mk will be provided that pulls and builds the default browser suite and all necessary libraries.  So for most people, the build process won't change.

Here is how we envision the tree looking after this change:
/mozilla: hosting directory for all Mozilla-related stuff
	# Gecko and libraries built on top of it
	/gecko: the web rendering engine (GRE)
	/toolkit: the XUL Runtime Engine (XRE) toolkit
	/xpinstall: the XPInstall library

	# Products
        /apps: Mozilla Application Suite
        /apps/browser: Mozilla Browser
        /apps/mailnews: Mozilla Mail/News
        /apps/composer: Mozilla Composer

        /seamonkey: the old Seamonkey browser
	/seamonkey/xpfe: the Seamonkey browser
        /seamonkey/mailnews: Seamonkey mailnews

        /camino: the Camino browser

	# Non-Gecko-dependent libraries and utilities
	/ipc: the IPC daemon
	/netwerk: the Necko network library
	/db: database implementations, including Mork
	/xpcom: the XPCOM library
        /js: SpiderMonkey JavaScript Engine
        /directory/c-sdk: Directory Service
        /security/nss:    NSS
        /other-licenses: ical, libart
	/nsprpub: NSPR
	/string: String

        /config: the build configuration directory
For example, the default build for Firebird would be apps/browser/browser.mk, which would which would pull and build the Toolkit and GRE at toolkit/ and gecko/, which would pull and build the base libraries /nsprpub, /js, /directory/c-sdk, and /security/nss (unless switches were specified to use other versions of these libraries and sdks elsewhere on the disk).

Directories that do not belong at the top-level will be cvs removed and their histories copied into the appropriate top-level directory.  For example, mozilla/content/ and mozilla/layout/ will be moved to mozilla/gre/content/ and mozilla/gre/layout/.

Stuff that still has to work:
- pulling & building in a single step: client.mks handle this
- mozconfig: you can still specify a single mozconfig (even the same one as before) and each library will take the options from it that it needs.
- objdir: objdir builds will still work; a parallel directory structure under objdir/ will be created by client.mk and the other .mk, like: objdir/gecko, objdir/apps/browser, objdir/nsprpub, etc.  However, there will be multiple configure/make systems since these are now parallel top-level libraries.
- parallel builds: will still work.
- cross-compilation: will still work.
- static builds: will still work; need to test to make sure :)
- tinderboxen: will need preparation and tweaking.
- lxr: will need tweaks to what directories it indexes and a full index rebuild
- bonsai: will need directory tweaks as well and a full index rebuild
- bugzilla: should not change.

Dependency Breaking

There are a number of dependencies that still need to be broken for this to happen, including:
- dependency between layout and bookmarks
- dependency between xpconnect and caps (not necessarily necessary for this to happen, but would be nice)
- dependency between netwerk and the entire Gecko world

When

First thing, we need to make a mockup and get module owner feedback on the proposal.  This thing can be implemented fairly quickly; we just need to notify people.

Chris and I would like to do this in the beginning-of-1.5alpha timeframe, which will help support the goal of moving to Firebird as the default build.  During this time there will be about a day of downtime in which cvs checkins will not be allowed.

Despot will need to be updated at the same time, and we will get Bonsai and LXR updated ASAP after that (the respective admins will be primed).  It seems like this is a good time to fix the list of module owners in Despot if someone wants to do that, but it is not necessary for this change.

We will get a modified tinderbox client working against the new tree before any of this occurs to verify that the client works and we will have full coverage, and we will work with the tinderbox client owners to upgrade their clients.

Discussion

  1. What about my patches?  Will those still apply?
    All patches will still apply with some massaging (change of locations) unless they touch build files.  But the massaging will take some work.  We will supply a Perl script that can change a patch to work with the new world, but some patches (notably build changes and patches with new files) may not carry over completely.  We feel the benefits we will get out of this are worth the pain.
  2. I'd love to use this time to rearrange some directories and files whose locations have been bugging me!
    We'd love to do it too.  Send us email and we'll get it done.
  3. Why aren't camino/ and seamonkey/ under apps/?
    apps/ is for the Mozilla Application Suite, not a generic place for all apps..
  4. What about if I want to build with configure and make?  How does this affect me?
    You will still be able to build with configure and make, but since we now consider apps/browser, gre, nspr, nss and directory as separate top-level libraries, you will have to build each one separately.  (We are considering methods to make this automatic without subverting our joyful liberation from the monolithic build system.)
  5. Why so many top-level libraries?
    Other applications use nspr, nss, directory and js directly from our tree as standalone apps.  This makes them top-level.
  6. How will this affect the Firebird and Mozilla roadmap?
    This actually arose from discussion about the roadmap and build support for the impending move to Firebird.  Moving GRE into its own gre/ directory and build process will make it extremely easy to make Firebird the default build when drivers decides to do so.
  7. Can I still build older releases of Mozilla if you're moving these directories around?
    Yes.  Directories that get moved (like layout/ and content/) will only be cvs removed, not actually physically moved, so everything will look exactly the same as it used to for old releases.
  8. Can I use my old tree and just pull?
    Not recommended or tested or supported.  That said, you possibly can; but it's really not worth it since you have to repull the gre anyway now that most of its source has moved under mozilla/gre/
  9. Will my old objdir / mozconfig configuration work?
    Yes.  objdirs and mozconfigs will be supported.
  10. Have you thought about Bonsai, LXR, Bugzilla, Despot and Tinderbox?
    Of course.
  11. To whom do I bitch that things have changed and are not the same?
    Try http://www.mozillaquest.com/ .  If you have a real problem ask on IRC or file a bug.
  12. A whole day of downtime?  I can't live without my hourly cvs pull.
    You can check out timeless's variable name spelling checkin the day after tomorrow, how about you enjoy the fresh air before it all goes the way of the SUV.