This is a guest blog post by Rob Martin (@version2beta) originally published in Software Developers Journal in August 2013.
Note: This article was originally published on Librato, which has since been merged with SolarWinds®AppOptics™. Here’s how to monitor Python performance using AppOptics.
Like many people, I confused the Heisenberg Uncertainty Principle with the Observer Effect. The Heisenberg Uncertainty Principle asserts that we cannot accurately measure pairs of physical properties of particles. That is, if we know one value, the other is unknowable. This is best illustrated by the story of Heisenberg being pulled over by a police officer. The officer asks Heisenberg if he knows how fast he was driving. “No, but I know where I am,” says Heisenberg. The officer says, “Sir, you were driving 76 miles per hour.” Heisenberg replies, “Great. now I’m lost.”
On the other hand, the observer effect describes the impact on a system of measuring things within that system. A common example is measuring tire pressure. It’s hard to do without having a bit of air leak out, thereby affecting the pressure of the tire. The more we measure, the flatter the tire gets.
Instruments that measure code performance are subject to the observer effect too. The mere process of inserting code (or reflecting on existing code) to measure our application’s performance will also affect the performance of the application. If we naively code against a cloud-based metrics tool such as SolarWinds Librato™ or New Relic, we risk significant impact on our performance waiting for blocking I/O as we record the measurement to a remote location.
I needed a library for measuring code performance, and wanted to minimize the observer effect. This problem suggested an asynchronous programming pattern that records in real time, but stores the result within another thread or process.
ZeroMQ
Enter ZeroMQ. On the surface, it would seem that ZeroMQ is an extremely fast (“zero” time) message queue, but I find it to be a bit of a misnomer. It’s not a message broker like Rabbit-MQ. It doesn’t support the Advanced Message Queuing Protocol (AMPQ). There’s no management interface. It doesn’t persist messages to a disk, and if you don’t have a subscriber, all of the publisher’s messages are dropped by default. You can’t inspect messages or get statistics on the queue – at least not without writing your own management layer.
ZeroMQ is more like a brilliant and fast socket library with built-in support for a wide variety of asynchronous patterns. This makes ZeroMQ an ideal message dispatcher when you don’t need complex broker features. For my metrics library, I didn’t need those features, but I did need speed.
Building a Code Instrumentation Library
The features I wanted were simple:
- Easy instrumentation for timing and counting.
- Highly efficient operation within the instrumented code.
- The ability to instrument multiple programs, processes, and threads in one common system. These processes are the publishers of metrics.
- Support for multiple back-end systems to consume the metrics. These processes are the subscribers to the metrics.
Because ZeroMQ was my message dispatcher and Librato.com was my primary target for recording and aggregating these metrics, I chose the portmanteau Zibrato as my project name.
The Zibrato Library for Code Instrumentation
Before we get into the architecture of the library, let’s take a quick look at the API. The Zibrato library provides three ways to instrument your Python code:
- Timers – Zibrato provides a decorator that can time any defined function or method, and a context manager that works with any code block.
from zibrato import Zibrato
z = Zibrato()
# decorated function
@z.time_me(level = ‘debug’, name = ‘myfunct_
timer’, source = ‘myprog’)
def myfunct():
time_consuming_operations()
# context manager
with z.Time_me(level = ‘debug’, name = ‘timer_name’):
slow_function_to_time()
- Counters – Zibrato provides a counter method as a
decorator or as a context manager.
from zibrato import Zibrato z = Zibrato() # decorated function @z.count_me(level = ‘info’, name = ‘myfunct_ counter’, value = 5) # inc by 5 def myfunctc(): pass # context manager with z.Count_me(level = ‘info’, name = ‘counter_ name’, source = ‘deathstar’): pass
- Gauges – Finally, Zibrato can be used to insert an arbitrary value into the back end at any point in the code.
from zibrato import Zibrato z = Zibrato() # Zibrato gauge z.gauge(level = 'crit', name = 'gauge_name', value=123)
This is just a quick overview of how Zibrato is used to instrument code. For more information, check out the library at Pypi or look at my Github.com repository.
The Architecture
In order to accomplish the goals behind the API, Zibrato is divided into three parts:
- The Zibrato library, which implements the API described above and publishes metrics to the message queue. As a user, I can have zero or more instrumented processes all communicating with my
message queue. - A message broker, which subscribes to zero or more publishers of metrics and in turn republishes the metrics to zero or more backend subscribers.
- Zero or more back-end providers, which subscribe to the message broker to receive metrics, then in turn do whatever is appropriate with them. In my application, the back-end provider sends the messages to my Librato account.
This is referred to as an “extended pubsub pattern”. The message broker (sometimes called a message bus) is core to this topology. It provides support for multiple publishers, and filters which messages the back end providers will receive.
To implement Zibrato on a server, I generally use supervisord to start the broker and my Librato backend. Then any code that is instrumented can connect to the broker, and the backend will receive whatever messages meets its filter and forward them to Librato. The back end aggregates messages and performs the blocking I/O portion of the work, communicating across the network to Librato.com, in a completely separate process from the instrumented code.
Using ZeroMQ
Clearly, ZeroMQ is the special sauce in the Zibrato architecture, doing the heavy lifting of messages from the instrumented code to the back-end providers. It provides the asynchronicity.
If you don’t already have ZeroMQ installed, it’s easy to do with pip:
pip install --upgrade python-dev pyzmq
If you’re running Anaconda Python from Continuum Analytics, pyzmq is already installed.
Creating a Message Broker
Our broker subscribes to publishers and then publishes to subscribers. ZeroMQ refers to this type of device as a “forwarder”. It’s fairly trivial to do this in Python:
- First, we create a ZeroMQ context, which is basically a thread-safe container for our sockets that allows us to cleanly shut everything down when we’re done with it.
- Then, we create a TCP socket to serve as a subscriber to the instrumented code. This could have been done using a Unix domain socket (similar to a named pipe) or a PGM multicast IP socket, but I wanted the ability to connect to the broker on a specific IP address and port, even if the broker is running on a different server from the instrumented code. Potential gotcha: by default, a subscriber filters out all messages, so we have to tell it what messages to receive. An empty string tells it to receive all messages.
- Next, we create a TCP socket to serve as a publisher for the backends. Again, I made a design decision allowing backends to run on separate servers.
- Finally, we combine our subscriber socket and our publisher socket into a ZeroMQ forwarder device.
Here’s the code:
import zmq # Get the ZeroMQ Context context = zmq.Context() # Create a subscriber subscriber = context.socket(zmq.SUB) subscriber.bind(‘tcp://127.0.0.1:5550’) # Subscribe to all messages subscriber.setsockopt(zmq.SUBSCRIBE, ‘’) # Create a publisher publisher = context.socket(zmq.PUB) publisher.bind(‘tcp://127.0.0.1:5551’) # Combine the subscriber and the publisher into a forwarder zmq.device(zmq.FORWARDER, subscriber, publisher)
This code gets us started. Now any ZeroMQ publisher written in any programming language running on localhost can send messages to our broker. Likewise any subscriber on localhost can receive those messages.
If instead of 127.0.0.1 we created our sockets on an IP address accessible over the network, the broker would be able to connect to publishers and subscribers on other machines, too.
Building a Subscriber
A broker without any listeners has little purpose in life. Let’s create a simple subscriber that receives messages and prints them to standard output. We’ll create this in a separate Python script—it runs separately from the broker. Here’s how to create a subscriber:
- First, we connect to our ZeroMQ context described above.
- Then we create our listener’s socket.
- Next, we set a filter to which our socket subscribes. Our call to receive messages from the message broker will only return values that start with this filter. An empty string subscribes to all messages, but the default is to subscribe to no messages, so we have to set it to something—even if it’s an empty string.
- Finally, we set up a loop to keep on listening until
something comes in.
Our code might look like this:
import zmq # Get the ZeroMQ context context = zmq.Context() # Create a socket socket = context.socket(zmq.SUB) socket.connect(‘tcp://127.0.0.1:5551’) # Subscribe to all messages socket.setsockopt(zmq.SUBSCRIBE, ‘’) # Keep on keeping on while True: print socket.recv()
This code gives us a listener that will receive anything
the broker forwards and print it to STDOUT. Now all
we need is someone who will give the broker some
messages to forward.
Building a Publisher
If a tree falls in the forest and there’s no one to hear it, does it make any noise? I don’t know the answer to this, but I do know that a message broker without any publishers lives a pretty quiet life.
Connecting our broker to a publisher is easy to do. In a third Python script, we’ll create a publisher that sends messages to the broker. If we do it right, the subscriber we created in the last section will receive these messages and print them to STDOUT.
Here’s how to build a publisher:
- First, we connect to our ZeroMQ context, just like
we did above. - Next, we create our publisher socket.
- Finally, we send a message from our publisher to the message broker.
Here’s the code:
import zmq
# Get the ZeroMQ context
context = zmq.Context()
# Create a socket
socket = context.socket(zmq.PUB) socket.connect(‘tcp://127.0.0.1:5550’)
# Publish a little Isaac Newton
socket.send('I stand on the shoulders of giants.')
With these three components, we have a publisher that sends messages, a subscriber that receives messages, and a broker that forwards messages from any connected publisher to any connected subscriber.
Putting It All Together
Zibrato is based on the extended PubSub pattern described above, and uses the same three basic components.
On the front end, we have the Zibrato class that provides the instrumentation methods. This is our ZeroMQ publisher. The code itself is quite light, and all it needs to do is get the message to our broker, so the net impact on performance is pretty low.
In the middle, we have a broker coded much like the example above. It is a simple forwarder that accepts connections from multiple publishers and forwards messages to multiple subscribers.
On the back end, we have a library that implements the Librato HTTP API using the Python Requests library to store the metrics we’ve recorded. It flushes the data to Librato on a set schedule, including rolling up the counters. The Librato backend inherits from a standard back end class, so it’s easy to implement other kinds of backends too—like simply outputting to a log file or sending to a central Statsd server.
Credits
Isaac Newton once said, “If I have seen further it is by standing on the shoulders of giants.” At best I squat and risk falling off the giants’ shoulders, so I prefer the earlier quote from Isaiah di Trani. “Who sees further a dwarf or a giant? Surely a giant for his eyes are situated at a higher level than those of the dwarf. But if the dwarf is placed on the shoulders of the giant who sees further? … So too we are dwarfs astride the shoulders of giants. We master their wisdom and move beyond it.”
ZeroMQ is the brilliant work of Pieter Hintjens and iMatix Corporation. It is a powerful and flexible messaging platform and I highly recommend it for asynchronous applications. I also recommend reading Pieter’s ZeroMQ Guide. It’s lengthy and comprehensive, but is quite accessible and even an enjoyable read.
I first became aware of Librato on the Ruby Rogues podcast #62 featuring Joe Ruscio, Librato’s CTO and cofounder. They’ve done an excellent job of making metrics easy. Librato offers free development accounts and a free month of production with very reasonable pricing thereafter.
Zibrato was initially inspired by Etsy’s Statsd package, a Node.js service that, coupled with Graphite (written in Python and Django), provides a full asynchronous instrumentation stack. On a related note, check out Steve Ivy. He has not only written a Python library for interfacing with Statsd, he’s also reimplemented it in Python.
I use Kenneth Reitz’ Request library: HTTP for Humans. This library makes web interactions painless.
My testing setup is greatly benefitted from Gary Bernhardt’s Expecter package, and in the future I’ll probably refactor my tests to also use his Dingus library. Since I first learned BDD in Ruby, Gary was very helpful in bridging my knowledge gap from Ruby to Python.
I’ve become increasingly impressed by base version of Continuum Analytics’ Anaconda Python. It now goes on each of my development machines and virtual development environments.
Special thanks to Tracy Harms who spent several days pair programming with me on the Zibrato project. His feedback and insight were invaluable.
ROB MARTIN
Rob Martin is a developer at i.TV in Provo, Utah, U.S. One of the cooler parts of his job is that he’s expected to learn every language used in their stack. Before i.TV, he did Ruby at a Python shop, Python at a PHP house, and Perl on the factory floor. Online, Rob Martin is known as version2beta in most places. Follow him on Twitter: @version2beta, Github.com, and on his blog.
