Document toolboxDocument toolbox

Elastic Bamboo

Owned by Nicole Deflaux (Unlicensed)
Last updated: Jan 31, 2012 by John HillVersion comment

On This page

On Related Pages

Elastic Bamboo

We currently have MySQL, R, and Python2.7 running on our Elastic Bamboo host for the purposes of integration testing. This is accomplished by the customise-extras.sh shell script which is run at boot time for the hosts.

How to modify the EBS snapshot used

If you want to import a database dump, install Crowd, etc., here is how to modify that script:

  1. Log onto the currently running elastic instance (or spawn one if none are running). http://confluence.atlassian.com/display/BAMBOO/Accessing+an+Elastic+Instance
    • Note that the ssh key is on belltown:/work/platform/PasswordsAndCredentials/AtlassianAccountAWSCredentials/elasticbamboo.pk
    • Note that if you want to log onto the AWS console, Mike's username and password is on belltown:/work/platform/PasswordsAndCredentials/AtlassianAccountAWSCredentials/Mike'sPassword
  2. From the shell try out the commands you want to run at boot time
  3. Run you build and make sure it works
  4. Check your changes to customize-extras.sh into source control PLFM/trunk/tools/bamboo/bin/customise-extras.sh
  5. On the bamboo host, overwrite /mnt/bamboo-ebs/bin/customise-extras.sh with your updated script.
  6. Follow these instructions to create snapshot and configure bamboo to use it:
    1. Clean off any build artifacts
      • /mnt/bamboo-ebs/bin/mysql -u root -p drop database bambooa; create database bambooa
      • rm -rf /mnt/bamboo-ebs/bamboo-agent/build-dir/PLFM-TRUNK-JOB1
    2. Shutdown MySQL /mnt/bamboo-ebs/bin/mysqladmin -u root -p shutdown
    3. Do this stuff Updating your EBS snapshot to create the snapshot
  7. Shutdown the host, and kick off a new build. This should start a new bamboo host that will run your updated script.
  8. Make sure the build passes!

Installing custom MySQL, R, Python, and dependencies

Mysql, R, and Python are all binary builds which reside in the /mnt/bamboo-ebs structure. The primary binaries for all reside in /mnt/bamboo-ebs/bin/. In order to access these builds, you need to have the following in your environment:

PATH=/mnt/bamboo-ebs/bin:$PATH
LD_LIBRARY_PATH=/mnt/bamboo-ebs/lib:$LD_LIBRARY_PATH

These environmental variables are inserted into the environment via the /mnt/bamboo-ebs/customize-extras.sh script which is run by the elastic bamboo system.

Mysql

Before installed mysql, the libaio package needs to be installed. Only a few files are needed to run mysql, however. So after installing via yum (yum install libaio), the following command can be used to move the files to their permanent home in /mnt/bamboo-ebs/lib/ (mv /usr/lib/libaio* /mnt/bamboo-ebs/lib/) to satisfy this dependency during future boots.

Mysql was upgraded using the binary distribution located here: http://dev.mysql.com/downloads/mysql/#downloads, listed as Linux - Generic 2.6 (x86, 32 bit), Compressed Tar Archive. This is untarred into /mnt/bamboo-ebs/. From with the resultant directory: /bin, /share/, and /lib should have their contents moved to the same locations in /mnt/bamboo-ebs/. The rest of the tar can be deleted. The bin/mysqld_safe file needs to be modified, having all occurances of the following paths replaced: /var/lib/mysql with /mnt/bamboo-ebs/data/ and /usr/bin with /mny/bamboo-ebs/bin. After this the customize-extras.sh script will be able to launch mysql on boot. In addition, the default databases need to be created using the mysql_install_db script, and the test2 database create as explained in how to modify the ebs snapshot above.
Python

Python requires gcc-c++ for a full compile. This is also required by R later on, but neither require any of the files installed to run. This can installed via yum (yum install gcc-c++).

Python was installed from scratch from the source located here: http://www.python.org/download/releases/ , using version 2.7.2. Download this source to the ebs and untar, then compile using the following flags:

./configure --prefix=/mnt/bamboo-ebs/;make;make install

R

R requires several dependencies, though some of them are only required for packages used in the Synapse platform code. The first is the gcc-gfortran packages, installed via yum (yum install gcc-gfortran). This package installs a library which is used during the R runtime, and needs to copied to the library location (cp /usr/lib/libgfortran* /mnt/bamboo-ebs/lib/). The second is libreadline-devel, which isn't required for runtime, only for compile (yum install readline-devel). The libraries for readline are already part of the default image, only the headers are required here.

Rcurl requires that a recent copy of Curl be installed. This isn't available via the standard fedora distribution, so it needs to be compiled from scratch. The source can be obtained here (http://curl.haxx.se/download.html), and it should be untarred and compiled via the following command:

./configure --prefix=/mnt/bamboo-ebs/;make;make install

R itself can be obtained from here (http://cran.sixsigmaonline.org/src/base/R-2/R-2.13.0.tar.gz), and should be untarred and compiled via the following command:

./configure --prefix=/mnt/bamboo-ebs --with-x=no --enable-R-shlib;make;make install

After installation, R needs certain packages installed for the build to work. The following commands should work perfectly:

# R

R version 2.13.0 (2011-04-13)R version 2.13.0 (2011-04-13)
...
Copyright (C) 2011 The R Foundation for Statistical ComputingISBN 3-900051-07-0
Platform: i686-pc-linux-gnu (32-bit> install.packages('rJava')
...
> install.packages ("rjson")
...
> install.packages ("RCurl")
...
> install.packages ("RUnit")
...

After the above EBS changes have been made, the source directories and tar.gz files should all be deleted. Then the snapshot should be saved as detailed here: Updating your ebs snapshot.

For more information, see: