IBM Bluemix, IPython Notebook via the Anaconda distribution

Written a few months back now, but may help someone at some point…

Work in progress – No warranty/use at own risk.

Objective – How to run the Eurex VSTOXX walk-through using iPython Notebook on IBM Bluemix including using HDF5 hierarchical datasets

I was inspired by the following blogs, giving the basis for moving forward.

Blog input – “Welcome to the Expand VSTOXX Tutorials based on Python” http://www.eurexchange.com/vstoxx/

The packaging requirements are Python 2.7.x, scipy, numpy, pandas, ipython, matplotlib and pytables, h5py for HDF5 support.

Blog input – “Run IPython Notebook on IBM Bluemix by Peter Parente, IBM Developerworks”

http://www.ibm.com/developerworks/cloud/library/cl-ipython-app/index.html

After investigation, I’ve determined I’d like to use Anaconda. Anaconda is a Scientific Python distribution, comprising 195+ packages for science, math, engineering, data analysis. Anaconda Distribution

Which Bluemix build pack? I’m still working on the pros/cons of each build pack…I can’t say I understand them. My simple criteria for this at this stage – it works and successfully allows my application to run.

The choices I found:
https://github.com/cloudfoundry/buildpack-python.git
https://github.com/kennethreitz/conda-buildpack.git
https://github.com/heroku/heroku-buildpack-python
https://github.com/joshuamckenty/heroku-buildpack-python

The joshuamckenty buildpack seemed to be the most highly regarded amongst users, however I just struggled to get the additional packages to work e.g. pytables, scipy, h5py so really wanted to leverage the Anaconda distribution. Anaconda is the binaries only distribution, Miniconda is the source code distribution.

Step 1. Install the local Anaconda Python distribution.

I installed the Anaconda installation locally on my local machine (OSX based), plus I assume you have already installed the CF command line, you have access to Bluemix, Github/JazzHub etc.

Step 2. Clone Peter Parente’s project as the basis for your project

Clone the source code of the parente | ipython-on-bluemix project on IBM DevOps Services:

git clone https://hub.jazz.net/git/parente/ipython-on-bluemix

Step 3. Edit Peter Parente’s project to support a different buildpack, different requirements

Change the build pack from https://github.com/joshuamckenty/heroku-buildpack-python to https://github.com/kennethreitz/conda-buildpack.git

applications:
- name: ipython
  memory: 1G
  instances: 1
  host: ipython-${random-word}
  domain: mybluemix.net
  path: .
  buildpack: https://github.com/kennethreitz/conda-buildpack.git

The requirements.txt file contains the list of libraries the buildpack will install using the pip installer. You can add or remove libraries here. As we are using Anaconda, with binaries only, we will have not for requirements.txt.

Add a new file, conda-requirements.txt and the following packages:

ipython-notebook
dateutil
scipy
pytables
h5py
numpy
pandas
matplotlib

Copying the guidance from Peter…

Finally, Procfile instructs Bluemix on how to run your application once it’s built:

web: ipython profile create; echo "c.NotebookApp.password = '$( python -c 'from IPython.lib import passwd; import os; print passwd(os.environ.get("PASSWORD", ""))' )'" >> ~/.ipython/profile_default/ipython_notebook_config.py; ipython notebook --port=$VCAP_APP_PORT --ip=$VCAP_APP_HOST --no-mathjax

This is the most complicated piece of the solution. (It’s also the most messy, because it must be a single line.) Breaking it down, the code:

Creates an IPython user profile.
Generates a secure hash of the PASSWORD environment variable and inserts it into the profile.
Starts the notebook server using the port and host information provided by Bluemix.

Step 4. Push the app bundle to Bluemix

cd ipython-on-bluemix
cf api https://api.ng.bluemix.net
cf login
cf push --no-start
cf set-env ipython PASSWORD mysecretpassword
cf start ipython

In another terminal, run this to check the installation.

cf logs ipython

You can confirm that the notebook server is running properly by querying its status:

cf app ipython
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s