Tinderbox 3

Tinderbox 3 is a new Tinderbox server and client designed to be compatible with the old but have several new spiffy features:

Database-backed, http protocol for communication
Tinderboxen upload builds to the server when they finish building
Patches can be uploaded to the server and distributed to all tbox clients for easy, widespread testing of an idea
Builds have much quicker cycle time and pick up changes quicker (thanks to fast-update and not building when there are no changes)
Clients can be controlled from the server: .mozconfig and other configuration, as well as commands ("kick", "clobber", "checkout" and "build" supported currently)
Client setup is brain-dead--as long as compiler and basic unix tools are there, it will check out and build, including setting up .mozconfig and other options specified by the server
Clients auto-upgrade when there is a new tbox client version
Server is entirely adminstered via a web interface
login/password security is enabled for sheriffing and build comments (you can use the login/password for your Bugzilla account).

All of these features conspire to make Tinderbox easy to install and administer, easy to donate machine time to, extremely useful for testing and verification, and fun for the whole family! All server control features are of course configurable by the client, for those who lack trust.

Client Installation

Prerequisites

You will need a system that has the prerequisites on mozilla.org to be capable of building. You will also need libwww-perl. Linux already comes with this generally. To check whether you have it you can run "

perl -e 'use LWP::UserAgent; print
$LWP::UserAgent::VERSION, "\n"'

". If that prints a version number you are good to go.

Installing LWP::UserAgent on cygwin (Windows)

For cygwin Perl you need to do the following (not sure about ActiveState Perl - ed.):

Install gcc on cygwin (run setup.exe and find it under Development)
Type "perl -MCPAN -e shell"
Set up CPAN with default options
Type "install Bundle::LWP" from the shell.

Also of special interest on Windows only: Mozilla's binary build creator needs the find program from cygwin to come first in your path, so make sure c:\cygwin\bin (or /usr/bin) is in your path before the win32 system directories.

Installing LWP::UserAgent on Mac

For Mac you need to do the following (not sure about ActiveState Perl - ed.):

Run fink install libwww-pm

Of special interest on Mac only: currently Mozilla requires wget for fast-update to work. This should be rectified soon.

Installation

If LWP::UserAgent is installed, installation is cake:

Download tinderclient.pl.
Go to a directory where you want the client to place the tree.
Type "perl tinderclient.pl --url="<url to tinderbox server>" treename your_machine_name"

You should now be building and sending status to the server. It will create a mozilla/ directory and build in there.

Server Installation

Prerequisites

Installation

Install Postgres. Many Linux systems have this by default.
Create a Postgres DB where the Tinderbox information will be stored. (createuser <apache_user> as the postgres user (fill in the apache username), then createdb tbox as the Apache user)
Download and install the server: untar it anywhere you like, and copy the scripts/ directory (and subdirectories) to a cgi-bin executable location of your choice. (tar xzvf tbox3.tgz; cp -R tbox3/scripts/* /var/www/cgi-bin). NOTE: if you are installing in a homedir and this installation requires
Set up the Tinderbox database. (cd tbox3/sql; ./setup-postgres.pl tbox as the Apache user)
Point Tinderbox at the database by editing the $dbname,$username and $password parameters in Tinderbox3/DB.pm (you can use blank for username and password if you created the database as the Apache user).
Set up your Bugzilla username as super-user by editing the @superusers variable in Tinderbox3/Login.pm.
Set up the cleanup and bonsai updater scripts (scripts/tbox_updater)as a cron job. NOTE: you should comment out the tbox_build_quota.pl line until you are uploading builds to your server.

Congratulations! You now have a mozilla.org-ready Tinderbox.

Configuration

Once you have installed the server and connected to it through http, you are ready to add a tree and configure it.

Adding a Tree

Go to <your tinderbox url>/admin.pl.
Click "Add Tree".
Set a name and change options as you like. You can change all these options later; the initial ones are configured to work optimally with mozilla.org.
Type your login and password in the Login / Password boxes at the bottom of the screen and hit Submit Query.
You have a new tree! You can view it by clicking the View link that should have appeared in the resulting screen.

Customizing Your Tree

You can customize your tree at <your tinderbox url>/admin.pl by clicking on Sheriff or Edit links. The Sheriff link is where you can edit the style and body of the tree. The Edit link is where you edit hard code configuration options. You won't need to mess with most of these if you are doing Mozilla; but the .mozconfig and upgrade_url parameters are of interest. NOTE: you should update upgrade_url to point to a place where you will put updated tinderclient.pl's. You can modify this on the fly when it comes time for your first upgrade, though.

Printing CVS Checkins

To print CVS checkins like mozilla.org, you need to add a Bonsai Monitor. Go to Edit your tree and click "New Bonsai" at the bottom. The resulting screen has everything necessary to monitor for trunk checkins to mozilla; you can just hit Submit Query if that's what you want. Otherwise edit to your heart's content. NOTE: make sure you have set up the tbox_updater cron job or your Bonsai monitor will not show anything. tbox_updater calls the script that actually grabs (and caches) the checkin information from Bonsai.

Uploading Patches

Of particular interest is the Upload Patch button on the Edit screen. This is where you will upload patches for your clients to apply, and delete them from the system when you are done. You can also obsolete a patch--i.e. leave the patch up but keep clients from accessing it.

Managing Machines

Once a machine has made its presence known to Tinderbox it will show in the Machines section of the Edit screen. Click on the machine you want to edit there. From there you can make it visible / invisible to the main tinderbox page, give it commands (for example, putting "clobber" in the commands box will cause the machine to do a full rebuild next cycle even if it is a dep build, and "kick" will cause the machine to restart building as soon as possible) and edit configuration (the configuration variables on the bottom and .mozconfig are the same as the ones you can edit on the tree, but things you edit here will only be sent to the specified machine--this allows you to customize what different machines build).

Uploading Builds

A little extra configuration is necessary to start your Tinderbox client uploading builds to a server.

You need to be able to ssh to the machine where the builds will be stored.
You need ssh and scp installed on the client.
Set up no-password authentication for ssh from this host to the server. (It's actually easy and is damn useful outside of this use :)
Add these parameters to the tinderclient.pl line: --upload_ssh_loc=user@server --upload_ssh_dir=<dir on server> --uploaded_url=http://www.example.com/location_where_builds_will_be/%s (the % s will be replaced with the build name)

Cleaning Up Builds on the Server

Your admin might get mad at you if you just blindly upload gigs and gigs of data to the server, so you will want to delete some old builds. Fortunately tbox3 comes with a script to intelligently delete old builds--so that there is general coverage.

Ensure the server where the builds are stored has LWP::UserAgent installed (see client install instructions above if not).
Grab tbox3/client/tbox_build_quota.pl from tbox3.tgz and put it on the server somewhere.
Create a file on the server defining where your various builds are stored, called quota_dirs.txt (it can be called anything), in the following format:
```
<glob pattern to get files> <url to build that is referenced on server with %s>
```
For example, on my server:
```
/home/jkeiser/public_html/builds/johnserver/*.tar.gz http://jkeiser.no-ip.com/~jkeiser/builds/johnserver/%s
```
You may specify multiple lines in this file for multiple uploaded build directories.
Important: the URL should be the same as the --uploaded_url parameter the client uses. This URL exists so that the script can notify the server that old builds are being deleted so it can take the links down from the page.
Run the client like this: tbox_build_quota.pl --url=<url to tinderbox server> --start=0 --end=0 --quota=<quota size in Mb> quota_dirs.txt
Set up a cron job to run the script periodically.

What it does

This script tries to do two things:

(a) keep multiple build machines even--i.e. if there are 6 machines uploading builds to this machine, it will try to keep the builds at the same frequency for them--one build per hour, or one build per three hours, etc. This way each machine has the same coverage.
(b) keep the coverage spread evenly over the period. To do this, when it deletes a build it deletes the build with the least

To accomplish this, it iterates through all the builds when they are over quota and finds the one that is the closest to the two surrounding builds--i.e. if there is a build a half hour between two other builds on the same machine, it will be deleted before a build that is an hour between two other builds.

Fine-tuning quota

If you specify a different --start and --end, the script will apply that quota only to builds start hours ago through end hours ago. You can use this to have multiple quotas for different time periods (these are cumulative--three 500M quotas equals 1.5G), For example, you might want the first day's builds to always be kept around, the next 30 day's builds to have 500M quota, the previous 3 months to have another 500M quota, and no builds at all beyond 3 months (so that it is likely many builds will be deleted). To do this you would run the script twice (you don't run it at all for the :

tbox_build_quota.pl --url=<url> --start=24 --end=48
--quota=500 quota_dirs.txt

tbox_build_quota.pl --url=<url> --start=48 --end=748 --quota=500
quota_dirs.txt

tbox_build_quota.pl --url=<url> --start=748 --end=2908
--quota=500 quota_dirs.txt

tbox_build_quota.pl --url=<url> --start=2908 --end=0 --quota=0
quota_dirs.txt

Note that you can say "all builds beyond 3 months have 500M quota" by changing that last line to --quota=500.