Tinderbox

From DreamHost
Revision as of 21:44, 1 October 2009 by Heikki (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Tinderbox

"Tinderbox is a software suite that provides continuous integration capability. Tinderbox allows developers to manage software builds and to correlate build failures on various platforms and configurations with particular code changes."

The Tinderbox server (these instructions are for the server) displays the status of each build slave. The build slaves report whether builds passed or failed, and if tests passed or failed. Full logs from builds and tests can be shown.

These instructions describe how to get Tinderbox2[1] running on a shared Dreamhost account.

Preparation

It is recommended to run Tinderbox2 under a separate account from your other accounts, so first go and create a new shell account with the panel. It is recommended you set quotas, as Tinderbox server can receive large amounts of data.

Next, create a domain under the new user we just created. Let's assume you created user tbox and domain tinderbox.example.com.

Finally, create a mail username under the new domain (something that is hard to guess would be good to reduce spam), for example: tboxbuildshereplease@tinderbox.example.com. It is a good idea to set up reasonable quota for that account just in case. As for other settings: no daily warnings, remove read when 250, remove read when older than 30, remove even unread, do not save removed, no emails when removed. We also need to set up an email filter from the panel for this account: when To contains tboxbuildshereplease@tinderbox.example.com, forward to shell account tbox. We'll need this when we finish the setup.

Download

Login as tbox user, and create some directories:

$ mkdir -p $HOME/soft $HOME/webapps/tinderbox

Let's download Tinderbox2 sources

$ cd $HOME/soft
$ export CVSROOT=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
$ cvs login 
  (password is "anonymous")
$ cvs checkout mozilla/webtools/tinderbox2
$ mv mozilla/webtools/tinderbox2 $HOME/soft/.
$ rm -fr mozilla

You may want to read the various documentation files included as these are just quick instructions.

Configuration

Next we need to run configure (remember to change path and domain as appropriate for your username and domain):

$ cd $HOME/soft/tinderbox2
$ ./configure --prefix $HOME/webapps/tbox2/prefix --cgibin-prefix=$HOME/webapps/tbox2/cgi-bin --html-prefix=$HOME/tinderbox.example.com
$ make
$ make compile_bin_code
$ make install

Then we'll need to copy the images over, and create a symbolic link for the cgi-bin directory:

$ cp -a src/gif ~/tinderbox.example.com/
$ ln -s $HOME/webapps/tbox2/cgi-bin ~/tinderbox.example.com/cgi-bin

Next we need to create local configuration files:

$ mkdir -p $HOME/webapps/tbox2/prefix/local_conf
$ cp $HOME/webapps/tbox2/prefix/default_conf/TinderConfig.pm $HOME/webapps/tbox2/prefix/local_conf/
$ cp $HOME/webapps/tbox2/prefix/default_conf/TreeConfig.pm $HOME/webapps/tbox2/prefix/local_conf/

Next you need to edit the local_conf/TinderConfig.pm file:

  • Remove line ':/home/kestes/mozilla/webtools/tinderbox2/build/test/vcsim'.
  • Change $TINDERBOX_UID and $TINDERBOX_GID to what the command "id" shows for uid and gid
  • Change $URL_BIN to http://tinderbox.example.com/cgi-bin
  • Change $TINDERBOX_HTML_DIR to /home/tbox/tinderbox.example.com
  • Change $TINDERBOX_DATA_DIR to /home/tbox/webapps/tbox2/tbox2-data
  • Change $TINDERBOX_CGIBIN_DIR to /home/tbox/webapps/tbox2/cgi-bin/
  • Change $GIF_URL to http://tinderbox.example.com/gif
  • Change $ERROR_LOG to /home/tbox/webapps/tbox2/tinderbox2.log
  • Change $LOCK_FILE to $TINDERBOX_DATA_DIR."/tinderd.lock";
  • Uncomment line 'TinderDB::Time_Local',
  • Assuming you want to use Subversion, comment line 'TinderDB::VC_Perforce', and add a line after it: 'TinderDB::VC_SVN',
  • Uncomment line 'VCDisplay::None',
  • Comment line 'VCDisplay::Bonsai',
  • Change $BT_URL to what is applicable for you
  • You might also want to change $DEFAULT_DISPLAY_HOURS top 8 and $LOG_PERFORMANCE to 1

Next you need to edit local_conf/TreeConfig.pm file:

  • Comment out all dummy projects for testing (Project_A, Project_B, Project_C)
  • Add your real projects, for example if you had a "Dummy" project:
           'Dummy' =>  {
                   root => 'http://svn.example.com/dummy',
                   module => 'trunk',
                   branch => '',
                  },
  • Change $VCMAIL_SUFFIX to what is applicable for you
  • Comment out tree groups (jim, fred)

Generate Pages

Tinderbox2 has a script that needs to be run to generate the HTML pages. This should run without errors:

~/tinderbox.example.com/cgi-bin/tinder.cgi --daemon-mode

You should now see generated pages when you browse to http://tinderbox.example.com. If you run into any problems, check the $HOME/webapps/tbox2/tinderbox2.log file.

Create Administrator Account

Once the HTML pages come up ok, it is time to check that cgi-bin also works and set up the administrator:

  • Your project should be the first link under "Select one of the following trees:"; click it
  • Click Administrate this tree
  • Put in your email address in both the Email address field and Add new administrator fields, and hit submit
  • Navigate back to the admin page, fill in your email address again, and set a new password

Now you are set up to administer your Tinderbox.

Cronjobs for Automatic Updates

Tinderbox2 does not have long running server processes. Instead, the HTML pages need to be generated by a script, which is typically run by a cronjob.

Issue the following command:

$ crontab -e

and add these lines to generate the HTML pages every 5 minutes and clean up logs once a day:

MAILTO="your_email@example.com"

*/5 * * * *     $HOME/webapps/tbox2/cgi-bin/tinder.cgi --daemon-mode
6 6 * * *       $HOME/webapps/tbox2/prefix/bin/rmlogs

Receiving Incoming Build Logs

The Tinderbox2 documentation states that the processmail_builds script should be able to receive form posts, but this seems to result in internal server errors. Typically Tinderbox2 is set up to receive build logs by email, so that is what we will do as well.

First, a procedure you can use to test that processmail_builds works and the build logs are good... Run the following command twice:

$ python -c "import time;print int(time.time())"

And take the output from it and replace the starttime and timenow fields below and send yourself that in email:

tinderbox: tree: Dummy
tinderbox: starttime: 1225315543
tinderbox: timenow: 1225315654
tinderbox: status: success
tinderbox: buildname: ubuntu-linux
tinderbox: errorparser: unix
tinderbox: administrator: dummy@example.com
tinderbox: END

this
is my
build log

Once you have received the email, view the source and save it in a file. Then, feed the source into the processmail_builds script like so:

$ $HOME/webapps/tbox2/prefix/bin/processmail_builds < emailbuild.eml

Now, run tinder.cgi script and you should see a green column and cell in the Dummy tree: http://tinderbox.example.com/Dummy/status.html

It is time to set up the machinery to catch incoming emails. You should already have set up the incoming email account and filter to deliver to the tbox user. Now it is just a matter of adding two files to make sure build emails get routed to the processmail_builds script:

$ cat >~/.forward.postfix
"|/usr/bin/procmail -t"

Notice the quotes, and hit Ctrl+C to finish. Then procmail:

$ cat >~/.procmailrc
MAILDIR=$HOME/Maildir
LOGFILE=$HOME/procmail.log

:0:
* ^To:.*tboxbuildshereplease@.*tinderbox.example.com
|$HOME/webapps/tbox2/prefix/bin/processmail_builds

Test your setup by running the python command shown above twice, replacing the starttime and timenow fields with the values, and send the build log now to tboxbuildshereplease@tinderbox.example.com. Run tinder.cgi and now you should have two green cells under the "ubuntu-linux" column (that column name can be almost anything, and can be decided by the build slaves).

Build Slaves

Now you just need some people who have computers that are on the net most of the time willing to set up build slaves for your project. The build slaves can be written in any language in any manner you want, as long as they can send those build emails. There are some samples in the Tinderbox2 sources. Note: The slave must wait at least 6 minutes before starting a new build after the last one!