Uncategorized

Establishing Python CI Build in Jenkins

Setting up project build in Jenkins is very easy – there are a lot of plugins fow varioous languages, notification, pre and post build actions. But the most important path is Build Action itself.

Build Action for Python Project Build with Unit Test and Code Analysis

PYENV_HOME=$WORKSPACE/.pyenv/

# Delete previously built virtualenv
if [ -d $PYENV_HOME ]; then
    rm -rf $PYENV_HOME
fi

# Create virtualenv and install necessary packages
virtualenv --no-site-packages $PYENV_HOME
. $PYENV_HOME/bin/activate
pip install --quiet -r requirements.txt 
pip install --quiet nosexcover
pip install --quiet pylint
nosetests --with-xunit --cover-package=myapp --cover-erase --with-xcoverage ./test 
pylint -f parseable myapp/ | tee pylint.out

Build Action for Python Module Build with Unit Test and Code Analysis

PYENV_HOME=$WORKSPACE/.pyenv/

# Delete previously built virtualenv
if [ -d $PYENV_HOME ]; then
    rm -rf $PYENV_HOME
fi

# Create virtualenv and install necessary packages
virtualenv --no-site-packages $PYENV_HOME
. $PYENV_HOME/bin/activate
pip install --quiet nosexcover
pip install --quiet pylint
pip install --quiet $WORKSPACE/  # where your setup.py lives
nosetests --with-xunit --cover-package=myapp --cover-erase --with-xcoverage ./tests
pylint -f parseable myapp/ | tee pylint.out

Post-Build Actions for Both Configurations

**/nosetests.xml
Uncategorized

Selection of Appropriate Continuous Integration Solution for the Projects

CI service providing builds, tests and deployment is must when you want to speedup your work and get rid of useless work. There exist various solutions these days. All of them do almost same, but not every with same requirements.

We had following needs:

  • Easy to configure
  • Notifications, connection to other services like Bitbucker, Slack
  • Support for Python, C++, C#
  • Run on own linux-based virtual machine together with other services (wiki)

Exisiting solutions:

  • TeamCity

Even I was or I’m using all of them and because of linux-based machine, my hidden favorite was TeamCity. It looks more “professional” that Jenkins :).

Setup

So, let’s setup it. I have established the smallest Azure virtual machine:

Standard Virtual Machine A0 (Shared core, 768 MB memory) with Ubuntu Server 14.04 LTS. I thought it is enought – there will be wiki which uses almost no resources and CI itself, which consume more resources only it is doing something – building, testing or deploying. I have established TeamCity, run it and … nothing. OutOfMemory on the machine. Even I have set different arguments for the JVM,TC neeed more. So I have found that minimal requirements are 750MB only for the JVM (https://confluence.jetbrains.com/display/TCD8/Installing+and+Configuring+the+TeamCity+Server#InstallingandConfiguringtheTeamCityServer-SettingUpMemorysettingsforTeamCityServer).

Ok, lets set better machine – Standard Virtual Machine A1 (1 core, 1.75 GB memory). TC was running. But the ‘feeling’ from the browsing was ‘slow’. So I established simple build of the Python project. It was running fine and the build was successful. Nice. But the second build failed – not enought memory. What?! Next Azure VM A2 has 3.5 GB memory which is needless!

Sorry TeamCity. The last change was Jenkins. Set A0 again, setup build, run and … success! Run it again, again, again and fail. Check the logs … not enought memory. Ok, after five minutes of cursing on greedy JVM, I setup A1 back. Jenkins web was ‘very fast’. Seems good, lets try to run multiple builds, multiple times to be sure. So after 20 successful builds, no memory execption and check of VM memory consumption, it seems it is enought (but I will see in the future)!

Conclusion

I have to say that it was surprising to me that even you want to have own CI for a few of  your projects and run it ourself, you need a quite strong machine to be able to have proper running CI service.