
********
Buildbot
********

Buildbot masters and slaves
===========================

Master
------

From the root of the git checkout (this will use the master.cfg from CoolProp)::

    virtualenv buildbot-sandbox
    source buildbot-sandbox/bin/activate
    pip install sqlalchemy==0.7.10 buildbot
    cd dev/buildbot
    buildbot create-master master
    buildbot start master

The file ``buildbot-private.py`` (which is a python module with the passwords for the slaves as well as
the buildbot website), should also be placed in the master folder next to master.cfg.  Alternatively,
you can put the ``buildbot_private.py`` in another folder on the master's computer and make a soft-link
in the master folder to point to the buildbot_private.py file.

If you want to completely restart the master, you can do::

    buildbot restart master

but usually a::

    buildbot reconfig master

will do the job since it will just reparse the configuration file without signing you out of the server

To ensure that the buildbot server stays online, you can make a script with the contents::

    buildbot start /path/to/master_folder

and add it to a cron job


Slaves
------

To start a slave connected to a buildbot master at IP address 10.0.0.2 (default for
host for VirtualBox), with a slave named ``a-slave`` and passsword ``pass``,
run the command::

    virtualenv a-slave-sandbox
    source a-slave-sandbox/bin/activate
    pip install sqlalchemy==0.7.10 buildbot-slave
    buildslave create-slave a-slave coolprop.dreamhosters.com:port a-slave pass
    buildslave start a-slave

If the master is somewhere else, just change the IP address.  As of Sept, 2014, the
master was at www.coolprop.dreamhosters.com.  The buildbot_private.py on the master
holds the required passwords.

OSX Virtualbox host
-------------------

Thanks to http://stackoverflow.com/questions/1261975/addressing-localhost-from-a-virtualbox-virtual-machine and the comment of spsaucier you basically need to do the following, copied verbatim:

To enable this on OSX I had to do the following:

1. Shut your virtual machine down.
2. Go to ``VirtualBox Preferences -> Network -> Host-only Networks ->`` click the "+" icon. Click OK.
3.Select your box and click the "Settings" icon -> Network -> Adapter 2 -> On the "Attached to:" dropdown, select "Host-only Adapter" and your network (vboxnet0) should show up below by default. Click OK.
4. Once you start your box up again, you should be able to access localhost at http://10.0.2.2/

You can refer to it by localhost and access other localhosted sites by adding their references to the hosts file (C:\windows\system32\drivers\etc\hosts) like the following::

	10.0.2.2    localhost
	10.0.2.2    subdomain.localhost
    

Python slaves
-------------

Based on the miniconda Python ecosystem, you can create your own virtual
environments for building the Python wheels. This requires the following
steps on a Windows machine::

    conda create -n CoolProp27 python=2
    conda create -n CoolProp34 python=3
    conda install -n CoolProp27 cython pip pywin32
    conda install -n CoolProp34 cython pip pywin32

    activate CoolProp27
    pip install wheel
    deactivate
    activate CoolProp34
    pip install wheel
    deactivate

Please repeat the steps above for both 32bit and 64bit Python environments.

On a Linux system, things only change a little bit::

    conda create -n CoolProp27 python=2
    conda create -n CoolProp34 python=3
    conda install -n CoolProp27 cython pip
    conda install -n CoolProp34 cython pip

    source activate CoolProp27
    pip install wheel
    deactivate
    source activate CoolProp34
    pip install wheel
    deactivate

Please make sure that the standard shell ``/bin/sh`` used by the builbot is
bash or zsh. We make use of the ``source`` command, which is not part of the
POSIX specification.

At the moment, it is not possible to use several slaves for the same build job.
We have to find a new way to generate the configuration.

Information on building the single wrappers can be found on
:ref:`this dedicated page<wrapper_common_prereqs>`.

For uploading generated binary python files to PYPI, you should create a file ``~\.pypirc`` with the contents::

	[distutils]
	index-servers=
	    pypi
	    test

	[test]
	repository = https://testpypi.python.org/pypi
	username = user
	password = XXXXXXXXXXXXXXXX

	[pypi]
	repository = https://pypi.python.org/pypi
	username = user
	password = XXXXXXXXXXXXXXXX

Buildbot as a service (Windows)
-------------------------------

On Windows, you create a batch script that activates your virtual environment
and starts the buildslave::

    @echo off
    call "C:\Program Files (x86)\Miniconda32_27\Scripts\activate.bat" Buildbot
    buildslave start "C:\CoolProp-slave"

This script can then be added to the system services via::

    sc create <serviceName> binpath= <pathToBatFile> DisplayName= "CoolProp Buildbot" start= auto

You might want to run ``services.msc`` to edit the user that runs the service. If
you are tired of the error messages from the non-returning script, you could
also use a service wrapper like `NSSM <http://nssm.cc/>`_ to start the script.

Buildbot and launchd (Mac OS)
-----------------------------
As written in the `Buildbot Wiki <http://trac.buildbot.net/wiki/UsingLaunchd>`_,
you can start your slaves automatically with a so called ``plist`` or property list.
Place the example content below in a file called ``/Library/LaunchDaemons/org.coolprop.a-slave.plist``
and make sure it is owned by the user ``root`` and the group ``wheel``::

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
        <key>StandardOutPath</key>
        <string>org.coolprop.a-slave.log</string>
        <key>StandardErrorPath</key>
        <string>org.coolprop.a-slave-err.log</string>
        <key>Label</key>
        <string>org.coolprop.a-slave</string>
        <key>Program</key>
        <string>/Users/buildbot/bin/a-slave.command</string>
        <key>RunAtLoad</key>
        <true/>
        <key>KeepAlive</key>
        <dict>
            <key>SuccessfulExit</key>
            <false/>
        </dict>
        <key>GroupName</key>
        <string>staff</string>
        <key>UserName</key>
        <string>buildbot</string>
        <key>WorkingDirectory</key>
        <string>/Users/buildbot/slave/logs</string>
        <key>SessionCreate</key>
        <true/>
    </dict>
    </plist>

Please change the file above according to your needs and pay special attention
to username and path definitions. The script ``a-slave.command`` that is called
by ``launchd`` could look like this one::

    #!/bin/bash
    #
    # Description: This file call the control script to start and
    #              stop the buildbot slave. It stays open when being
    #              called and waits for a signal to terminate running
    #              and endless while-loop. After catching a signal
    #              to terminate, it shuts down the build slave and
    #              returns. It is a wrapper for another Bash script
    #              allowing us to use launchd on MacOS.
    #
    # Author: Jorrit Wronski <jowr@mek.dtu.dk>
    #
    # Please remove the "Author" lines above and replace them
    # with your own name if you copy and modify this script.
    #
    # If you experience any problems with the PATH variable on OSX,
    # this setting might be for you:
    if [ -x /usr/libexec/path_helper ]; then
      eval `/usr/libexec/path_helper -s`
    fi
    #
    CTRLSCRI="/Users/username/a-slave.bsh"
    #
    trap "$CTRLSCRI stop; exit 0; " TERM SIGINT SIGTERM
    #
    $CTRLSCRI start & wait
    # Just idle for one hour and keep the process alive
    # waiting for SIGTERM.
    while : ; do
      sleep 3600 & wait
    done
    #
    echo "The endless loop terminated, something is wrong here."
    exit 1

Note that this script calls another Bash script that does the actual work. We hope
to simplify maintenance by using a common control script for Linux and MacOS as
shown in :ref:`slavescript`.

Or alternatively, you can just launch buildslave directly if you do not use conda environment::

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
        <key>KeepAlive</key>
        <true/>
        <key>Label</key>
        <string>com.start.buildbot</string>
        <key>ProgramArguments</key>
        <array>
            <string>/Users/Ian/anaconda/bin/buildslave</string>
            <string>restart</string>
            <string>slave</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>StandardErrorPath</key>
        <string>/Users/Ian/.buildbot_stderr</string>
        <key>StandardOutPath</key>
        <string>/Users/Ian/.buildbot_stdout</string>
        <key>UserName</key>
        <string>Ian</string>
        <key>WorkingDirectory</key>
        <string>/Users/Ian</string>
    </dict>
    </plist>

Buildbot as a daemon (Linux)
----------------------------

On Linux, you can add the following lines to the end of your ``~/.profile`` file (similar
ideas apply on other platforms) to start the slave automatically at user log in::

    # Connect to the buildbot master
    buildslave start ~/slave

... or even better, you install a service that gets started and shutdown together with
your computer. For Debian/Ubuntu, we recommend a script like::

    #! /bin/sh
    ### BEGIN INIT INFO
    # Provides:          buildslave
    # Required-Start:    $remote_fs $syslog
    # Required-Stop:     $remote_fs $syslog
    # Default-Start:     2 3 4 5
    # Default-Stop:      0 1 6
    # Short-Description: A script to start the buildbot slave at boot time
    # Description:       This file activates the virtual environment and starts
    #                    the buildbot slaves. It also shuts them down if the
    #                    system is halted. Place it in /etc/init.d.
    ### END INIT INFO

    # Author: Jorrit Wronski <jowr@mek.dtu.dk>
    #
    # Please remove the "Author" lines above and replace them
    # with your own name if you copy and modify this script.

    EXECUSER=username
    NAME="a-slave"
    CTRLSCRI="/home/username/$NAME.bsh"

    # Load the VERBOSE setting and other rcS variables
    . /lib/init/vars.sh

    # Define LSB log_* functions.
    # Depend on lsb-base (>= 3.2-14) to ensure that this file is present
    # and status_of_proc is working.
    . /lib/lsb/init-functions

    #
    # Function that starts the daemon/service
    #
    do_start(){
      sudo -u $EXECUSER $CTRLSCRI start
      #start-stop-daemon --start --user $EXECUSER --chuid $EXECUSER --startas $CTRLSCRI -- start
      RETVAL="$?"
      return "$RETVAL"
    }

    #
    # Function that stops the daemon/service
    #

    # Function that stops the daemon/service
    #
    do_stop() {
      #start-stop-daemon --stop --user $EXECUSER --startas
      sudo -u $EXECUSER $CTRLSCRI stop
      RETVAL="$?"
      return "$RETVAL"
    }

    case "$1" in
    start)
        log_action_msg "Starting $NAME"
        do_start
        ;;
    stop)
        log_action_msg "Stopping $NAME"
        do_stop
        ;;
    restart)
        log_action_msg "Restarting $NAME"
        do_stop
        do_start
        ;;
    *)
        log_action_msg "Usage: $0 {start|stop|restart}"
        exit 2
        ;;
    esac
    exit 0

Which then can be added to the scheduler with ``update-rc.d buildslave defaults``.
This should gracefully terminate the bot at shutdown and restart it again after reboot.
To disable the service, run ``update-rc.d -f buildslave remove``. You can enable and
disable the daemon by runnning ``update-rc.d buildslave enable|disable``.

.. _slavescript:

Buildbot slave management (Mac OS and Linux)
--------------------------------------------

Note that the two examples above call a user-script to activate the virtual
environment and start the buildslave. Such a script could look like this::

    #!/bin/bash
    #
    # Description: This file activates the virtual environment and starts
    #              the buildbot slaves. It is also used to shut them down
    #              during system shutdown.
    #
    # Author: Jorrit Wronski <jowr@mek.dtu.dk>
    #
    # Please remove the "Author" lines above and replace them
    # with your own name if you copy and modify this script.
    #
    VIRTENV="/home/username/a-slave-sandbox"
    SLAVEDIR="/home/username/a-slave"
    #
    ## For virtualenv
    #ACTICM="source $VIRTENV/bin/activate"
    ##DEACCM="source $VIRTENV/bin/deactivate"
    #
    # For miniconda
    MINICO="/home/username/miniconda/bin/activate"
    ACTICM="source $MINICO $VIRTENV"
    #DEACCM="source deactivate"
    #
    # Carry out specific functions when asked to by the system
    case "$1" in
      start)
        echo "Starting script buildbotslave "
        $ACTICM
        buildslave start $SLAVEDIR
        #$DEACCM
        ;;
      stop)
        echo "Stopping script buildbotslave"
        $ACTICM
        buildslave stop $SLAVEDIR
        #$DEACCM
        ;;
      restart)
        echo "Restarting script buildbotslave"
        $ACTICM
        buildslave restart $SLAVEDIR
        #$DEACCM
        ;;
      *)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
    esac
    exit 0




Setting MIME type handler
=========================

To change the MIME types on the server so that unknown file types will map properly to ``application/octet-stream``, modify the ``buildbot.tac`` file to add the following block::

    from twisted.web.static import File

    webdir = File("public_html")
    webdir.contentTypes['.mexw32'] = 'application/octet-stream'
    webdir.contentTypes['.mexw64'] = 'application/octet-stream'
    webdir.contentTypes['.mexmaci64'] = 'application/octet-stream'
    webdir.contentTypes['.jnilib'] = 'application/octet-stream'
    webdir.contentTypes['.mexa64'] = 'application/octet-stream'
    webdir.contentTypes['.oct'] = 'application/octet-stream'
    webdir.contentTypes['.whl'] = 'application/octet-stream'
    webdir.contentTypes['.dylib'] = 'application/octet-stream'
    ...

and then do a ``buildbot restart master``


Starting virtualbox images at boot
==================================

Create a daemon entry in Libray/LaunchDaemons.  Make sure you use full paths to VBoxManage::

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
        <key>GroupName</key>
        <string>staff</string>
        <key>InitGroups</key>
        <true/>
        <key>KeepAlive</key>
        <false/>
        <key>Label</key>
        <string>com.start.windows.vm</string>
        <key>ProgramArguments</key>
        <array>
            <string>/usr/bin/Vboxmanage</string>
            <string>startvm</string>
            <string>xp</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>StandardErrorPath</key>
        <string>/Users/Ian/.virtualbox_window_stderr</string>
        <key>StandardOutPath</key>
        <string>/Users/Ian/.virtualbox_windows_stdout</string>
        <key>UserName</key>
        <string>Ian</string>
    </dict>
    </plist>


Documentation Builds
====================

Some parts of the documentation are quite involved. That is why we decided not
to rebuild the whole documentation after every commit. There is a special python
script that runs a day and performs the most expensive jobs during
documentation rebuild. This covers the generation of validation figures for all
fluids and the fitting reports for the incompressible fluids.

If you have some tasks that take a long time, make sure to add them to that
special script in ``Web/scripts/__init__.py``. This helps us to keep the continuous
integration servers running with an acceptable latency with regard to the commits
to the git repository. However, if you are unlucky and your commit coincides with
figure generation, you will experience a long
delay between your commit and the appearance of the freshly generated documentation
on the website. You can follow the progress in the logfiles on the buildbot master though.
