• Bellatrix in Windows: http://mainlydata.kubadev.com/ec2/bellatrix-windows-users-two-things-to-remember/

• Bellatrix provisioning commands: http://mainlydata.kubadev.com/ec2/bellatrix-provisioning-commands/

thanks Richard Shea!

Some time ago I wrote some lengthy instructions on how to configure Django, NGinx and Green Unicorn on an brand new Ubuntu EC2 instance. Now, I want to show how to automate the same tasks using a command line tool called Bellatrix. We will also provide a Jenkins (formerly Hudson) pipeline so we have, if we want, a nice display for the UI.

What is Bellatrix?

Bellatrix is a set of (magic) command line utilities for EC2. It wraps the boto library in order to provide a set of easy to use commands that will help you with the common EC2 operations (start, stop, list, provision, burn, etc.) You can find more information about Bellatrix on this link.

We will use Bellatrix to help us provisioning a blank Ubuntu EC2 image (AMI) by using these commands:

1. pip install bellatrix
2. bellatrix start ami
3. bellatrix provision ubuntu_django_nginx_gunicorn.py
4. bellatrix burn instance_id
5. bellatrix terminate instance_id

1. Installing Bellatrix

First, install Bellatrix by typing:

pip install bellatrix

You can also install Bellatrix inside a virtualenv environment without modifying any of the following steps. Our Jenkins example (published at the end) uses virtualenv on every step.

Now that you have Bellatrix installed, you will need to provide the AWS EC2 credentials in the configuration directory. Please follow these instructions: http://bellatrix.readthedocs.org/en/latest/commands_use_tut.html#setting-your-aws-credentials in order to set your environment.

2. Starting an EC2 instance

We are ready to start an EC2 instance:

bellatrix start ami-baba68d3 ec2-keypair

Parameters explanation

• ami-baba68d3 is the last Ubuntu Oneiric AMI published in http://cloud.ubuntu.com/ami Just in case you don’t know, an AMI (Amazon Machine Image) is just a snapshot of a pre-configured machine.
• ec2-keypair is the name of the key pair that Bellatrix (and you) will use to connect to your EC2 instance.

 



More information on the bellatrix start command can be found here: http://bellatrix.readthedocs.org/en/latest/commands_use_tut.html#starting-an-ec2-instance

The above command will provide an output like this one:

bellatrix start ami-baba68d3 ec2-keypair
+ bellatrix start ami-baba68d3 key
2012-04-01 15:06:15,848 INFO starting EC2 instance...
2012-04-01 15:06:15,848 INFO ami:ami-baba68d3 type:t1.micro key_name:key security_groups:default new size:None
2012-04-01 15:06:17,017 INFO starting image: ami-baba68d3 key key type t1.micro shutdown_behavior terminate new size None
2012-04-01 15:06:17,709 INFO we got 1 instance (should be only one).
2012-04-01 15:06:17,709 INFO tagging instance:i-864c56e2 key:Name value:Bellatrix started me
2012-04-01 15:06:21,252 INFO instance:i-864c56e2 was successfully tagged with: key:Name value:Bellatrix started me
2012-04-01 15:06:21,252 INFO getting the dns name for instance: i-864c56e2 time out is: 300 seconds...
2012-04-01 15:06:41,216 INFO DNS name for i-864c56e2 is ec2-23-20-206-220.compute-1.amazonaws.com
2012-04-01 15:06:41,217 INFO waiting until instance: i-864c56e2 is ready. Time out is: 300 seconds...
2012-04-01 15:06:41,217 INFO Instance i-864c56e2 is running

Please note the DNS name: ec2-23-20-206-220.compute-1.amazonaws.com and the instance id: i-864c56e2 from the output above. We will use both in the commands below. By the way, given the DNS name we can infer the (less verbose) public IP. In this case is 23.20.206.220. I am sure you already got the rule.

3. bellatrix provision

Once you have your instance up and running, it’s time to configure it! Automation here is the key to a reliable output. Our bellatrix provision command will set-up Django, Nginx, Green Unicorn and Upstart in your formerly blank Ubuntu.

First of all you need to get the provisioning configuration file:

wget https://bitbucket.org/deccico/bellatrix_configs/raw/tip/bellatrix_configs/ubuntu_django_nginx_gunicorn.py

If you are interested in getting also the Jenkins jobs for this article or looking at other provisioning examples you can instead clone the whole project:

hg clone ssh://hg@bitbucket.org/deccico/bellatrix_configs

bellatrix provision executes a list of commands. Since every command is just a simple list of Python strings it’s very easy to add your own just by looking at the available ones. Bellatrix already provides an interesting set of generic, ready to use commands that cover the most common operations.

• Bellatrix commands: – https://bitbucket.org/adeccico/bellatrix/src/tip/bellatrix/lib/cmds.py
• bellatrix provision documentation: – http://bellatrix.readthedocs.org/en/latest/commands_use_tut.html#provisioning-an-ec2-instance-or-any-host

Command Execution

Now that you know more about the provision command, is time to execute it:

bellatrix provision ubuntu_django_nginx_gunicorn.py ubuntu 23.20.206.220 --private_key=~/.bellatrix/ec2.pk

Parameters explanation

• ubuntu_django_nginx_gunicorn.py – Provisioning script. It is possible to execute a configuration inside a directory, so in case you cloned the whole project you can use  “bellatrix_configs/ubuntu_django_nginx_gunicorn.py” as this parameter.
• ubuntu – This is just the user name of the remote host.
• 23.20.206.220 – The host where we will execute the provisioning command. We can use the DNS name or the IP that we got in the second step.
• –private_key=~/.bellatrix/ec2.pk - The private key that correspond to the ec2-keypair specified in the start command.

4. Saving your work into a new AMI

Once the provision command performed its magic, we only need to save the current state into a new AMI, so every time we get a new instance from this new AMI the configuration will be ready to use. The command in this case will be:

bellatrix burn i-864c56e2  ubuntu_django_nginx_gunicorn_x64 --wait=true

Parameters

• i-864c56e2  -  Instance id captured in the start command.
• ubuntu_django_nginx_gunicorn_x64 – Name of the new AMI. Bellatrix will add a timestamp to it.
• –wait=true – Burning a instance takes some time. By default the burn command will finish in some seconds after it gets the new AMI code. Nevertheless the AMI won’t be ready until the burning process finishes so if we specify this option, this command will return only when we get the AMI code and the burning process is done .

Documentation about the burn command: http://bellatrix.readthedocs.org/en/latest/commands_use_tut.html#saving-the-state-of-an-instance-into-a-new-amazon-ami

5. Shutting the instance down

After we get the new AMI we can safely terminate the EC2 instance with this command:

bellatrix terminate i-864c56e2

As noticed here, a terminated instance won’t generate any cost.

Bonus: Adding some Jenkins magic

In case you plan to execute these commands more than once (like for any new Ubuntu release) you can find the Jenkins pipeline ready to be used here. Remember to copy the jobs to your Jenkins “jobs” directory and to restart Jenkins afterwards.

Green Unicorn is a Python WSGI HTTP Server for UNIX. Its combination with the high performance HTTP server NGinx is gaining lot of momentum in the Python community.

From “man gunicorn”…

Green Unicorn (gunicorn) is an HTTP/WSGI server designed to serve
fast clients or sleepy applications. That is to say; behind a
buffering front-end server such as nginx or lighttpd.

We are describing here how to combine a Django application with Green Unicorn and Nginx within a pristine EC2 Ubuntu 11.10 image.

Getting an Ubuntu instance

First of all we are going to http://cloud.ubuntu.com/ami/ to check the last released ami. As described in the image, we are looking for the last 64 bits, Ubuntu 11.10 EB2 image in the US-East region, which so far is ami-bf62a9d6.

• Why Ubuntu 11.10 ? Because is the last stable release.
• Why EBS? Because is faster and with more options like on the fly snapshots or easy image resizing than instance-store.
• Why US-EAST? Because is cheaper and usually with more features than the other regions.
• We also choose 64 bits so we can eventually expand to a bigger instance.

Starting your free EC2 instance

So far there is a nice offer for new customers where you can get a free EC2 Linux Micro Instance usage (613 MB of memory and 32-bit and 64-bit platform support) for one year. Please open your account and then launch your instance as described in the following screen-shots:

Launching and configuring your Ubuntu box

At this point you should have your EC2 or your own Ubuntu box up and running. If you are using EC2 you should ssh into it with this command:

ssh -i you_private_key.pem ubuntu@DNS_NAME

Then, please follow this steps to start setting your environment:

#install Python Package Installer
sudo apt-get install python-pip
#upgrade PIP itself
sudo pip install pip --upgrade
#install Green Unicorn
#install Virtualenv to generate our own isolated environment
sudo pip install virtualenv
#Installing NGINX
sudo apt-get install nginx
#Creating our Virtualenv environment
virtualenv --no-site-packages django_app
cd django_app
#activating the environment
source bin/activate
#installing Green Unicorn and Django
pip install django gunicorn
#Creating a Django project
django-admin.py startproject app
cd app
#start Django application with Green Unicorn
gunicorn_django -b 0.0.0.0:8000

Now go to your browser and check if your application is working as expected:



Finally, just Ctrl+C and “deactivate” so you can continue with Nginx…

Setting up Nginx

Now we will set up Nginx in order to have it listening in the port 80, serving static files and working with Gunicorn and Django in order to serve dynamic content.

#prepare directories
sudo mkdir -p /opt/django/logs/nginx/
#Create directories and softlinks for static content and templates
mkdir $HOME/django_app/static
mkdir $HOME/django_app/templates
sudo ln -s $HOME/django_app/static /opt/django
#download and set up Nginx configuration. Basically it will listen in port 80 and forward to port 8000 dynamic content
sudo mv /etc/nginx/sites-available/default /etc/nginx/sites-available/default.backup
wget https://bitbucket.org/deccico/django_gunicorn/raw/tip/server/etc/nginx/sites-available/default
sudo cp default /etc/nginx/sites-available/default

Testing Static resources

Configuring settings.py

You need to tell Django where your templates are, for that reason, add your absolute “templates” directory to your /home/ubuntu/django_app/app/settings.py file

Now look for the TEMPLATE_DIRS section and add your template directory

TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
'/home/ubuntu/django_app/templates',
)

Configuring urls.py

/home/ubuntu/django_app/app/urls.py will help us telling Django to return our template page when we introduce the /test_static request.

To make things easier we are using a dynamic view:

from django.conf.urls.defaults import patterns
from django.views.generic.simple import direct_to_template

urlpatterns = patterns('',
(r'^test_static/$',             direct_to_template, {'template': 'test_static.html'}),
)

Preparing test template

We need to prepare our template. Please create this file /home/ubuntu/django_app/templates/test_static.html with the following content:

<!--Load static will recover the static prefix variable and saving it in STATIC_PREFIX. We do this only once.-->

{% load static %}

{% get_static_prefix as STATIC_PREFIX %}

<!--Here we are just showing how easy is to recover static resources . -->
Test image
<img src="{{ STATIC_PREFIX }}django.png" />

After creating the template file you will need an actual image You can download one from the same repo.

cd /home/ubuntu/django_app/static

wget https://bitbucket.org/deccico/django_gunicorn/raw/tip/static/django.png

Restarting Nginx and voilà

Finally, we just need to restart Nginx and test everything is working.

#First we restart Nginx

sudo service nginx restart

Now we will start our Django application again:

cd /home/ubuntu/django_app/app
#activating the environment
source ../bin/activate
#start Django application with Green Unicorn
gunicorn_django -b 0.0.0.0:8000

Then we just go to our browser and request test_static in port 80. In my case the full address is: http://ec2-107-20-88-133.compute-1.amazonaws.com/test_static

As you can see above, we got Nginx serving the image while Django give us the content.

Type “Ctrl + C” and then “deactivate” again so we can be ready for the next step…

Setting Green Unicorn automatic start

We will use Upstart so we don’t have to worry about starting our application whenever we need to restart the server. One more configuration file and a bootstrap script will do the job here:

In /home/ubuntu/django_app/run.sh we create:

#!/bin/bash
set -e
LOGFILE=/var/log/gunicorn/django_app.log
LOGDIR=$(dirname $LOGFILE)
NUM_WORKERS=3  #recommended formula here is 1 + 2 * NUM_CORES

#we don't want to run this as root..
USER=www-data
GROUP=www-data

cd /home/ubuntu/django_app
source bin/activate
cd app
test -d $LOGDIR || mkdir -p $LOGDIR
exec gunicorn_django -w $NUM_WORKERS \
  --log-level=debug \
  --log-file=$LOGFILE 2>>$LOGFILE \
  --user=$USER --group=$GROUP

After creating the file, please remember to execute chmod a+x /home/ubuntu/django_app/run.sh

On the other hand, Upstart configuration file have to be located in: /etc/init/django_app.conf with the following content:

description "Django Application"

start on runlevel [2345]

stop on runlevel [06]

respawn

respawn limit 10 5

exec /home/ubuntu/django_app/run.sh

Now we can control our application with

sudo start django_app

sudo stop django_app

To sum up

The puropose of this post is to provide a path to make the installation and configuration of this stack easier. While some things could be changed or improved the focus was to stress how easily we can configure everything from scratch rather than exploring or discussing different options.

The code and configuration files of this post can be found here: https://bitbucket.org/deccico/django_gunicorn Have fun!

References

• https://docs.djangoproject.com/en/1.3/ref/generic-views/
• http://upstart.ubuntu.com/getting-started.html
• http://senko.net/en/django-nginx-gunicorn/
• https://help.ubuntu.com/community/EC2StartersGuide
• https://help.ubuntu.com/community/UEC/Images
• https://help.ubuntu.com/community/UbuntuCloudInfrastructure
• http://cloud.ubuntu.com/ami/

Update: these instructions are now automated in this new post

In this snippet I will show how to use the Tweepy library to get statistics from Tweeter.

The script will read a sample feed from the Twitter Streaming API (http://dev.twitter.com/pages/streaming_api) and perform the following actions:

* Reads the sample feed
* Notes the number of retweets seen
* Tracks the number of times posts have been retweeted, and
* Produces an hourly report of the 20 most frequently retweeted posts

#!/usr/bin/env python

"""
Tweepy library proof of concept

@author: Adrian Deccico
"""

from getpass import getpass
import hashlib
import time
import re
import operator
import logging
import codecs
import tweepy

logging.basicConfig(level = logging.INFO)

class TweetListener(tweepy.StreamListener):

    #we use this pattern to decide if a post is a retweet or not, given that retweet fields of the feed don't work
    __retweet_pattern = "^(rt|retweet).*$"

    #statistics
    count = 0
    found = 0
    hour_ranking = {}
    retweets = {}

    #constants
    TOP_TWEETS = 20 #number of tweets to display in each hour

    def on_status(self, status):
        """callback that will process new tweets"""
        try:
            self.count += 1
            text = status.text
            #check wether we got a retweet or not
            logging.debug('count %s found %s - %s' % (self.count, self.found, text))
            if re.match(self.__retweet_pattern, text, re.IGNORECASE) == None:
                return

            self.found += 1

            if text not in self.retweets.keys():
                twitt_times = 1
            else:
                twitt_times = self.retweets[text] + 1

            self.retweets[text] = twitt_times

            hour = status.created_at.strftime("%Y%m%d%H")

            logging.info("hour: %s - times: %s - %s" % (hour, twitt_times, text))
            logging.info("Number of retweets found:%s" % self.found)

            if hour in self.hour_ranking.keys():
                if text in self.hour_ranking[hour].keys():
                    number = self.hour_ranking[hour][text] + 1
                else:
                    number = 1
            else:
                number = 1
                self.hour_ranking[hour] = {}

            logging.debug("adding %s to key %s" %(number,text))
            self.hour_ranking[hour][text] = number
            self.printHourlyReport() 

        except:
            logging.exception("error while analyzing tweets")

    def printHourlyReport(self):
        """Print an hourly statistic file in results.txt"""
        logging.debug("updating statistics file")
        with codecs.open("results.txt", "w", "utf-8") as f:
            for h in sorted(self.hour_ranking):
                logging.debug(h + " " + str(type(h)))
                f.write("Top %s tweets at: %s \n" % (self.TOP_TWEETS, h))
                count = self.TOP_TWEETS
                h_dict = self.hour_ranking[h]
                #sort retweets by times and then by text
                for t in sorted(h_dict, key=lambda k: (-h_dict[k], k)):
                    line = "%s time%s - %s \n" % (h_dict[t],"s" if h_dict[t]>1 else "",t)
                    f.write(line)
                    count -= 1
                    if count == 0:
                        break
                f.write("-------------------------------\n\n")

    def on_error(self, status_code):
        logging.error('An error has occured! Status code = %s' % status_code)
        return True  # keep stream alive

    def on_timeout(self):
        logging.info('Time out event')

def main():
    # Prompt for login credentials and setup stream object
    username = raw_input('Twitter username: ')
    password = getpass('Twitter password: ')
    stream = tweepy.Stream(username, password, TweetListener(), timeout=None)

    logging.info("Analyzing Tweeter sample feed. Results.txt will be updated in order to reflect the top 20 retweets of each hour.\n")
    stream.sample()

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        logging.info('\nExecution finished!')

How to organize and avoid concurrency problems in big projects, splitted in multiple jobs and chained to each other in Hudson?

If you have a lengthy process, it will be a good practice to split it in multiple tasks, having each calling the next one, like in the (somewhat simplified) image on the left.

Problem number one: How to avoid Clean and Build (for instance) running at the same time?

As (almost) always, “you have a plugin for that” in Hudson. This time is Locks and Latches, which will let you create a lock, shared by all the jobs you want. Only the job that has the lock, will be executed.

So far, so good, but, where is the problem?

As you may guess, Clean could be called multiple times, while Build, Publish, or the same Clean are being executed. How could this happen? If you trigger Clean with a SCM (like SVN) this could be an expected situation, as you have no way to prioritize the jobs in Hudson. This is explained in bug 833.

The following Groovy script, checks if there is any other job waiting for an executor or being executed, preventing Clean to destroy the others jobs data. (Note: you will need the Groovy plugin and then create a “system Groovy script”.)

//Jobs currently building or in the queue
jobs = hudson.model.Hudson.instance.items.findAll{job -> job.isBuilding()} + hudson.model.Hudson.instance.items.findAll{job -> job.isInQueue()}

//Keep only the names from the collected job objects
jobs = jobs.collect{x -> x.name}

//get current job name
currentJob = Thread.currentThread().executable.toString().split()[0]

//cut current job from the jobs list
jobs -= currentJob

//decide if there is any running (or waiting) job
res = true
if (jobs.size()  > 0)
{
    res = false
    println "The following jobs are being executed or waiting for an executor"
    jobs.each{println it}
}
else
{
    println "Concurrency OK"
}

return res

In case you have another jobs in your Hudson instance, that have nothing to do with our “Clean -> Build -> Publish” project, you can restrict to our project, the concurrency checking. You just have to do a slight modification to our Groovy script:

//Jobs currently building or in the queue
jobs = hudson.model.Hudson.instance.items.findAll{job -> job.isBuilding()} + hudson.model.Hudson.instance.items.findAll{job -> job.isInQueue()}

//Keep only the names from the collected job objects
jobs = jobs.collect{x -> x.name}

//list of jobs that belong to the same project
project_jobs = ["Compile", "Publish"]

//cut current job from the jobs list
jobs = project_jobs.intersect(jobs)

//decide if there is any running (or waiting) job
res = true
if (jobs.size()  > 0)
{
    res = false
    println "The following jobs are being executed or waiting for an executor"
    jobs.each{println it}
}
else
{
    println "Concurrency OK"
}

return res

Hi, here are all the the links collected from #agilespain2010 posted in Twitter after applying this regex. ^(.*)(http://\S+)(.*)$ plus ordering + plus filtering within a small script.

The complete list until Jun-18-2010:

http://twitter.com/#search?q=%23agilespain2010

Hi!, this a clean and simple sample test of a web app using Open ID login in Google App Engine + Django.

http://bitbucket.org/deccico/djangoappengine-openid-sample

This application combines with slight modifications, the following projects:

• Djangoappengine: http://www.allbuttonspressed.com/projects/djangoappengine
• Django-nonrel http://www.allbuttonspressed.com/projects/django-nonrel – Changes to Text fields to Chars, due to lack of filtering support in the firsts
• OpenID integration: https://launchpad.net/django-openid-auth – Small change in the login view template, in order to work with standard Cross site forgery protection
• Python OpenID library: http://github.com/openid/python-openid

Just remember to add {% csrf_token %} within the login form of your own view.

enjoy!

Hi, below there is a little snippet that calculates the closest point to a segment. It is written in C++ but could be easily translated to any other language. Disclaimer: given its age, it use hungarian notation, and some names are in spanish, but it was deeply tested and works ok.

//--------------------------------------------------------------------------------------------------------
//author: Adrián Deccico - http://adrian.org.ar
/*
Description: calculate the closest point belonging to a given segment.
Parameters:
nSx1, nSy1 =  First point of the segment
nSx2, int nSy2 = Second point of the segment
nPx, int nPy = The point to analize
*nCercaX,  *nCercaY = The closest point to the given point (nPx,nPy) that belongs to the segment
Returns: The distance between the given point and the point found of the segment
*/
double ElPuntoMasCercanoAUnSegmento(double nSx1, double nSy1, double nSx2,
double nSy2, double nPx, double nPy,
double *nCercaX,  double *nCercaY)
{

//caracterizo la recta del segmento dado Y=m*X + b
double m, b;

bool bEsHorizontalOVertical = false;

//el segmento es vertical
if (nSx2 == nSx1){
    *nCercaX = nSx1;
    *nCercaY = nPy;
    bEsHorizontalOVertical = true;
}

//el segmento es horizontal
if (nSy2 == nSy1){
    *nCercaY = nSy1;
    *nCercaX = nPx;
    bEsHorizontalOVertical = true;
}

//el segmento está caracterizado por una recta \"AX + BY + C = 0\" con
// A y B distintos de cero
if (!bEsHorizontalOVertical)
{
    m= (nSy2 - nSy1) / (nSx2 - nSx1);
    b = nSy1 - (nSx1 * m);

    //caracterizo la recta que une al punto dado con el punto más
    // cercano dentro del segmento Y=m1*X + b1
    double m1, b1;

    m1 = -1 / m;
    b1 = -m1 * nPx + nPy;

    //encuentro el punto de intersección
    *nCercaX =  (b1 - b) / (m - m1);
    *nCercaY =  m * *nCercaX + b;
}

//verificar que el punto encontrado está dentro del segmento
if (*nCercaX < min(nSx1,nSx2)
||  *nCercaX > max(nSx1,nSx2)
||  *nCercaY < min(nSy1,nSy2)
||  *nCercaY > max(nSy1,nSy2) )
{
    //el punto más cercano será uno de los dos extremos (el más cercano al punto)
    if (dDistanciaEntrePuntos(nSx1, nSy1, nPx, nPy) <= dDistanciaEntrePuntos(nSx2, nSy2, nPx, nPy))
    {
        *nCercaX = nSx1;
        *nCercaY = nSy1;
    }
    else
    {
        *nCercaX = nSx2;
        *nCercaY = nSy2;
    }
}

return dDistanciaEntrePuntos(nPx, nPy, *nCercaX , *nCercaY);

}
//--------------------------------------------------------------------------------------------

//------------------------------------------------------------------------
//return the distance between thwo points in a x,y plane.
double dDistanciaEntrePuntos(double lOldX, double lOldY,
double lNewX, double lNewY)
{
    double x = lNewX - lOldX;
    double y = lNewY - lOldY;

    return sqrt(pow(x,2) + pow(y,2));
}
//------------------------------------------------------------------------

#the first group contains the expression we want to replace
pat = "word1\s(.*)\sword2"
test = "word1 will never be a word2"
repl = "replace"

import re
m = re.search(pat,test)

if m and m.groups() > 0:
  line = test[0:m.start(1)] + repl + test[m.end(1):len(test)]
  print line
else:
  print "the pattern didn't capture any text"

This will print: ‘word1 will never be a word2‘

The group to be replaced could be located in any position of the string.