SST version 0.2.1 has been released.

SST (selenium-simple-test) is a web test framework that uses Python to generate functional browser-based tests.

SST version 0.2.1 is on PyPI: http://pypi.python.org/pypi/sst

install or upgrade with:

pip install -U sst

Changelog: http://testutils.org/sst/changelog.html
SST Docs: http://testutils.org/sst
SST on Launchpad: https://launchpad.net/selenium-simple-test

SST downloads | (Jan 1 2012 - April 23 2012)

1600+ downloads from PyPI since initial release.

This post shows code examples in Python (2.7) for sending data to Graphite.

Once you have a Graphite server setup, with Carbon running/collecting, you need to send it data for graphing.

Basically, you write a program to collect numeric values and send them to Graphite's backend aggregator (Carbon).

To send data, you create a socket connection to the graphite/carbon server and send a message (string) in the format:

"metric_path value timestamp\n"

• `metric_path`: arbitrary namespace containing substrings delimited by dots. The most general name is at the left and the most specific is at the right.
• `value`: numeric value to store.
• `timestamp`: epoch time.
• messages must end with a trailing newline.
• multiple messages maybe be batched and sent in a single socket operation. each message is delimited by a newline, with a trailing newline at the end of the message batch.

Example message:

"foo.bar.baz 42 74857843\n"

Let's look at some (Python 2.7) code for sending data to graphite...

Here is a simple client that sends a single message to graphite.

Code:

#!/usr/bin/env python import socket import time CARBON_SERVER = '0.0.0.0' CARBON_PORT = 2003 message = 'foo.bar.baz 42 %d\n' % int(time.time()) print 'sending message:\n%s' % message sock = socket.socket() sock.connect((CARBON_SERVER, CARBON_PORT)) sock.sendall(message) sock.close()

Here is a command line client that sends a single message to graphite:

Usage:

$ python client-cli.py metric_path value

Code:

#!/usr/bin/env python import argparse import socket import time CARBON_SERVER = '0.0.0.0' CARBON_PORT = 2003 parser = argparse.ArgumentParser() parser.add_argument('metric_path') parser.add_argument('value') args = parser.parse_args() if __name__ == '__main__': timestamp = int(time.time()) message = '%s %s %d\n' % (args.metric_path, args.value, timestamp) print 'sending message:\n%s' % message sock = socket.socket() sock.connect((CARBON_SERVER, CARBON_PORT)) sock.sendall(message) sock.close()

Here is a client that collects load average (Linux-only) and sends a batch of 3 messages (1min/5min/15min loadavg) to graphite. It will run continuously in a loop until killed. (adjust the delay for faster/slower collection interval):

#!/usr/bin/env python import platform import socket import time CARBON_SERVER = '0.0.0.0' CARBON_PORT = 2003 DELAY = 15 # secs def get_loadavgs(): with open('/proc/loadavg') as f: return f.read().strip().split()[:3] def send_msg(message): print 'sending message:\n%s' % message sock = socket.socket() sock.connect((CARBON_SERVER, CARBON_PORT)) sock.sendall(message) sock.close() if __name__ == '__main__': node = platform.node().replace('.', '-') while True: timestamp = int(time.time()) loadavgs = get_loadavgs() lines = [ 'system.%s.loadavg_1min %s %d' % (node, loadavgs[0], timestamp), 'system.%s.loadavg_5min %s %d' % (node, loadavgs[1], timestamp), 'system.%s.loadavg_15min %s %d' % (node, loadavgs[2], timestamp) ] message = '\n'.join(lines) + '\n' send_msg(message) time.sleep(DELAY)

Resources:

• Graphite Docs
• Graphite Docs - Getting Your Data Into Graphite
• Installing Graphite 0.9.9 on Ubuntu 12.04 LTS
• Installing and configuring Graphite

I just setup a Graphite server on Ubuntu 12.04 (Precise).

Here are some instructions for getting it all working (using Apache as web server).

It follows these steps:

• install system dependencies (apache, django, dev libs, etc)
• install Whisper (db lib)
• install and configure Carbon (data aggregator)
• install Graphite (django webapp)
• configure Apache (http server)
• create initial database
• start Carbon (data aggregator)

Once that is done, you should be able to visit the host in your web browser and see the Graphite UI.

Setup Instructions:

############################# # INSTALL SYSTEM DEPENDENCIES ############################# $ sudo apt-get install apache2 libapache2 libapache2-mod-wsgi / libapache2-mod-python memcached python-dev python-cairo-dev / python-django python-ldap python-memcache python-pysqlite2 / python-pip sqlite3 erlang-os-mon erlang-snmp rabbitmq-server $ sudo pip install django-tagging ################# # INSTALL WHISPER ################# $ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/whisper-0.9.9.tar.gz ################################################ # INSTALL AND CONFIGURE CARBON (data aggregator) ################################################ $ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/carbon-0.9.9.tar.gz $ cd /opt/graphite/conf/ $ sudo cp carbon.conf.example carbon.conf $ sudo cp storage-schemas.conf.example storage-schemas.conf ########################### # INSTALL GRAPHITE (webapp) ########################### $ sudo pip install http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz or $ wget http://launchpad.net/graphite/0.9/0.9.9/+download/graphite-web-0.9.9.tar.gz $ tar -zxvf graphite-web-0.9.9.tar.gz $ mv graphite-web-0.9.9 graphite $ cd graphite $ sudo python check-dependencies.py $ sudo python setup.py install ################## # CONFIGURE APACHE ################## $ cd graphite/examples $ sudo cp example-graphite-vhost.conf /etc/apache2/sites-available/default $ sudo cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi $ sudo mkdir /etc/httpd $ sudo mkdir /etc/httpd/wsgi $ sudo /etc/init.d/apache2 reload ######################### # CREATE INITIAL DATABASE ######################### $ cd /opt/graphite/webapp/graphite/ $ sudo python manage.py syncdb $ sudo chown -R www-data:www-data /opt/graphite/storage/ $ sudo /etc/init.d/apache2 restart $ sudo cp local_settings.py.example local_settings.py ################################ # START CARBON (data aggregator) ################################ $ cd /opt/graphite/ $ sudo ./bin/carbon-cache.py start

Resources:

• http://graphite.readthedocs.org/en/latest/install.html
• http://graphite.wikidot.com/installation
• http://geek.michaelgrace.org/2011/09/how-to-install-graphite-on-ubuntu/

* works on my machine, Ubuntu 12.04

I'm doing some work with Graphite in Python. Here is a quick overview of what Graphite is...

Graphite provides real-time visualization and storage of numeric time-series data.

Links:

• Project: https://launchpad.net/graphite
• Docs: http://graphite.readthedocs.org

Graphite does two things:

• Store numeric time-series data
• Render graphs of this data on demand

Graphite consists of a storage backend and a web-based visualization frontend. Client applications send streams of numeric time-series data to the Graphite backend (called carbon), where it gets stored in fixed-size database files similar in design to RRD. The web frontend provides 2 distinct user interfaces for visualizing this data in graphs as well as a simple URL-based API for direct graph generation.

Graphite consists of 3 software components:

• carbon - a Twisted daemon that listens for time-series data
• whisper - a simple database library for storing time-series data (similar in design to RRD)
• graphite webapp - A Django webapp that renders graphs on-demand using Cairo

I am bringing some of my [lightly read] Python books to give away at the next Boston Python User Group meetup.

Boston Python User Group - April Project Night
When: Thursday, April 12, 2012, 6:30 PM
Where: Microsoft NERD, Cambridge

Books:

• Pro Python (Alchin)
• Head First Python (Barry)
• Programming in Python 3 (Summerfield)
• Python Programming Patterns (Christopher)
• Hello World! (Sande)

If you are interested in getting a free book, come to #bostonpython Project Night on April 12!

I uploaded a 5 minute video/screencast showing how to install and setup SST Web Test Framework on Ubuntu (Precise Pangolin 12.04).

I step through: creating a virtualenv, installing SST from PyPI, and creating a basic automated web test:

http://www.youtube.com/watch?v=LpSvGmglZPI

the following steps are essentially a transcript of what I did...

install system package dependencies:

$ sudo apt-get install python-virtualenv xvfb

create a "virtualenv":

$ virtualenv ENV

active the virtualenv:

$ cd ENV $ source bin/activate (ENV)$

* notice your prompt changed, signifying the virtualenv is active

install SST using `pip`:

(ENV)$ pip install sst

Now SST is installed. You can check the version of SST:

$ sst-run -V

particle visualization of Python core development commits: Jan 1, 2010 - Mar 06, 2012

http://www.youtube.com/watch?v=IQPuU_YtN8Q

data source is the commit log from cpython mercurial trunk:

$ hg clone http://hg.python.org/cpython

... trimmed to show development activity since Jan 1, 2010.

(images making up this video were rendered with: Codeswarm)

SST - Web Test Framework: http://testutils.org/sst

The latest release of SST (0.2.0) adds the ability to capture HAR (HTTP Archive format) output for pageload performance tracing/profiling.

New SST doc section: Performance tracing with Browsermob Proxy (HAR)

The HAR format is based on JSON, and is used by tools that consume/produce data collected by monitoring HTTP communication. These files contain a log of HTTP client/server conversation and can be used for additional analysis of page load performance.

The capture is achieved by routing browser requests through BrowserMob Proxy, which records web page loads while your tests run. SST will launch the proxy and save output to .har files if you enable the `--browsermob` command line option. HAR files are saved in the results directory for each page load.

HAR files can be viewed/analyzed with various tools, such as `harviewer`:

• http://www.softwareishard.com/har/viewer/
• http://pcapperf.appspot.com/
• http://code.google.com/p/harviewer/

I created a screencast demo showing it all together:

Automated Web Page Profiling : SST + BrowserMob

SST (selenium-simple-test) is a web test framework that uses Python to generate functional browser-based tests.

Version 0.2.0 Released!

SST version 0.2.0 is on PyPI: http://pypi.python.org/pypi/sst

install or upgrade with:

pip install -U sst

Changelog: http://testutils.org/sst/changelog.html
SST Docs: http://testutils.org/sst
SST on Launchpad: https://launchpad.net/selenium-simple-test

Development Update

So... what's up with SST development? is it active?

I ran Gource against the trunk branch and created an awesome visualization (with soundtrack!) to show off development activity:

video: link (with soundtrack)
video: link (no soundtrack)

The magic incantation of `gource` to produce the video:

$ gource \ -s .3 \ -1280x720 \ --auto-skip-seconds .3 \ --multi-sampling \ --stop-at-end \ --hide mouse,progress \ --file-idle-time 0 \ --max-files 0 \ --background-colour 222222 \ --font-size 20 \ --logo docs/assets/sst-logo_small.png \ --title "SST Development - lp:selenium-simple-test" \ --output-ppm-stream - \ --output-framerate 30 \ | ffmpeg -y -r 30 -f image2pipe -vcodec ppm -i - -b 2048K movie.mp4

(rendered on Ubuntu 11.10, audio mixed with Pitivi)

There are `python-matplotlib` and `python-numpy` packages in the Debian/Ubuntu repos.

However, if you want to run in a virtualenv (with no-site-packages), and pip install these packages from PyPI, you need some system dependencies installed first to build with:

$ sudo apt-get install build-essential python-dev libfreetype6-dev libpng-dev python-virtualenv

Then, you can create a virtualenv, and the installers for Numpy and Matplotlib will work:

$ virtualenv env $ cd env $ source bin/activate (env)$ pip install numpy matplotlib ... ... Successfully installed numpy matplotlib Cleaning up...

(tested on Ubuntu Oneiric 11.10 and Ubuntu Precise 12.04 alpha)