Zero Carbon Food

I’m an advocate of trying something first hand before consulting or advising others. Unfortunately my dabble into Bitcoin didn’t go too well – learned a lot, lost a bit, However, on the Equity Crowd Funding side and experience of EIS taxation, so far so good. I’m an investor, via CrowdCube, of Zero Carbon Food in Clapham….a very interesting endeavour which is a glimpse into the future of urban farming.

http://www.zerocarbonfood.co.uk/investors/

FullSizeRender

IMG_3983

IMG_3986

IMG_3988

IMG_3990

IMG_3991

IMG_3992

IMG_3993

IBM Bluemix, IPython Notebook via the Anaconda distribution

Written a few months back now, but may help someone at some point…

Work in progress – No warranty/use at own risk.

Objective – How to run the Eurex VSTOXX walk-through using iPython Notebook on IBM Bluemix including using HDF5 hierarchical datasets

I was inspired by the following blogs, giving the basis for moving forward.

Blog input – “Welcome to the Expand VSTOXX Tutorials based on Python” http://www.eurexchange.com/vstoxx/

The packaging requirements are Python 2.7.x, scipy, numpy, pandas, ipython, matplotlib and pytables, h5py for HDF5 support.

Blog input – “Run IPython Notebook on IBM Bluemix by Peter Parente, IBM Developerworks”

http://www.ibm.com/developerworks/cloud/library/cl-ipython-app/index.html

After investigation, I’ve determined I’d like to use Anaconda. Anaconda is a Scientific Python distribution, comprising 195+ packages for science, math, engineering, data analysis. Anaconda Distribution

Which Bluemix build pack? I’m still working on the pros/cons of each build pack…I can’t say I understand them. My simple criteria for this at this stage – it works and successfully allows my application to run.

The choices I found:
https://github.com/cloudfoundry/buildpack-python.git
https://github.com/kennethreitz/conda-buildpack.git
https://github.com/heroku/heroku-buildpack-python
https://github.com/joshuamckenty/heroku-buildpack-python

The joshuamckenty buildpack seemed to be the most highly regarded amongst users, however I just struggled to get the additional packages to work e.g. pytables, scipy, h5py so really wanted to leverage the Anaconda distribution. Anaconda is the binaries only distribution, Miniconda is the source code distribution.

Step 1. Install the local Anaconda Python distribution.

I installed the Anaconda installation locally on my local machine (OSX based), plus I assume you have already installed the CF command line, you have access to Bluemix, Github/JazzHub etc.

Step 2. Clone Peter Parente’s project as the basis for your project

Clone the source code of the parente | ipython-on-bluemix project on IBM DevOps Services:

git clone https://hub.jazz.net/git/parente/ipython-on-bluemix

Step 3. Edit Peter Parente’s project to support a different buildpack, different requirements

Change the build pack from https://github.com/joshuamckenty/heroku-buildpack-python to https://github.com/kennethreitz/conda-buildpack.git

applications:
- name: ipython
  memory: 1G
  instances: 1
  host: ipython-${random-word}
  domain: mybluemix.net
  path: .
  buildpack: https://github.com/kennethreitz/conda-buildpack.git

The requirements.txt file contains the list of libraries the buildpack will install using the pip installer. You can add or remove libraries here. As we are using Anaconda, with binaries only, we will have not for requirements.txt.

Add a new file, conda-requirements.txt and the following packages:

ipython-notebook
dateutil
scipy
pytables
h5py
numpy
pandas
matplotlib

Copying the guidance from Peter…

Finally, Procfile instructs Bluemix on how to run your application once it’s built:

web: ipython profile create; echo "c.NotebookApp.password = '$( python -c 'from IPython.lib import passwd; import os; print passwd(os.environ.get("PASSWORD", ""))' )'" >> ~/.ipython/profile_default/ipython_notebook_config.py; ipython notebook --port=$VCAP_APP_PORT --ip=$VCAP_APP_HOST --no-mathjax

This is the most complicated piece of the solution. (It’s also the most messy, because it must be a single line.) Breaking it down, the code:

Creates an IPython user profile.
Generates a secure hash of the PASSWORD environment variable and inserts it into the profile.
Starts the notebook server using the port and host information provided by Bluemix.

Step 4. Push the app bundle to Bluemix

cd ipython-on-bluemix
cf api https://api.ng.bluemix.net
cf login
cf push --no-start
cf set-env ipython PASSWORD mysecretpassword
cf start ipython

In another terminal, run this to check the installation.

cf logs ipython

You can confirm that the notebook server is running properly by querying its status:

cf app ipython

Adding a BLU/Analytics Warehouse Service to iPython

Written a few months back now, but may help someone at some point…

Create a BLU Warehouse in Bluemix, and bind to the Application.

cf create-service AnalyticsWarehouse Small AW-ipython
cf services
cf bind-service ipython AW-ipython

Script to run to download the DB2 Server driver, unpack it and set environment settings for the ibm_db install later.

#!/usr/bin/env bash
#

echo "Downloading IBM Data Server driver..."
curl "www.ng.bluemix.net/docs/Services/BluStratus/samples/clidriver.tgz" -s -o ./clidriver.tgz
tar xzf ./clidriver.tgz -C ./

# Delete the archive
rm -rf ./clidriver.tgz
mv ./clidriver /home/vcap/app/.heroku/clidriver

# Set environment variables, so the ibm_db install works correctly
export IBM_DB_HOME="/home/vcap/clidriver"
export IBM_DB_LIB="/home/vcap/app/.heroku/clidriver/lib"
export LD_LIBRARY_PATH="/app/.heroku/clidriver/lib:$LD_LIBRARY_PATH"

Python scripts to install the ibm_db package.

import os
!pip install ibm_db
import ibm_db

Test application code to inspect the Bluemix VCAP_SERVICES variable and connect to the remote DB2 instance.

import urllib
import datetime
import json


# get service information if on Bluemix 
if 'VCAP_SERVICES' in os.environ: 
  db2info = json.loads(os.environ['VCAP_SERVICES'])['AnalyticsWarehouse'][0]
  db2cred = db2info["credentials"] 

# connect to DB2  
db2conn = ibm_db.connect("DATABASE="+db2cred['db']+";HOSTNAME="+db2cred['hostname']+";PORT="+str(db2cred['port'])+";UID="+db2cred['username']+";PWD="+db2cred['password']+";","","") 
 
# we have a DB2 connection
if db2conn:  
    stmt = ibm_db.exec_immediate(db2conn,"select host_name,os_name,os_version,total_cpus,configured_cpus, total_memory,os_kernel_version,os_arch_type, os_release,os_full_version from sysibmadm.env_sys_info")  

# fetch the result  
result = ibm_db.fetch_assoc(stmt)   

result

Other useful resources:

https://hub.jazz.net/project/ibmdatabase/bluacceleration/overview