Apr 6 ’11
Lightweight Messaging Middleware
No man is an island, and these days that goes for information systems, too. Getting systems to talk with one another reliably, efficiently, and simply has been the goal of multiple efforts over the years. Approaches have varied from mailboxes, simple socket programming, Distributed Computing Environment (DCE), Common Object Request Broker Architecture (CORBA), and Message-Oriented-Middleware (MOM). Associated with each approach are attributes of complexity, cost, and standards.
There are many good commercial, proprietary messaging offerings available today, and also some good open source offerings, one of which we’ll explore in this article. Several open source MOM options are available such as RabbitMQ and Apache’s ActiveMQ. A recent addition to the open source field is ZeroMQ or 0MQ.
The key attributes a messaging system should support include:
- Low overhead; small footprint and minimal processor requirements
- Ease of setup
- The ability to send messages when the receiver isn’t yet ready.
Both RabbitMQ and ActiveMQ provide a complete “out of the box” system: just configure and go. 0MQ is more a library of routines you can use to “roll your own” messaging system, although that’s a simplistic description. 0MQ also has adapters (aka devices) ready to be incorporated into a service. The 0MQ authors define 0MQ with these terms:
- Exploits socket library as a concurrency framework
- Carries messages across in process, Inter-Process Communication (IPC), Transmission Control Protocol (TCP), and multicast
- Connects n-to-n via fan-out, publish/subscribe, pipeline, request-reply
- Is fast enough for clustered products and supercomputing
- Uses asynchronous I/O for scalable, multi-core message-passing applications
- Enjoys a large, active open source community
- Supports more than 20 languages, including C, C++, Java, Microsoft .NET, and Python
- Runs on most operating systems, including Linux, Windows, and OS X
- Licensed under GNU Lesser General Public License (LGPL) free software with full commercial support.
To this set of definitions, we’d add that 0MQ has a small footprint with componentry that enables development of sophisticated messaging systems.
Writing distributed programs is one of the hardest things programmers do. We seem poorly suited to reason about processes running in parallel, and the normal tools such as mutexes, conditional variables, and semaphores can sometimes be fragile and hard to use.
0MQ is a messaging library used programmatically that provides a “pimped” socket interface (i.e., “sockets on steroids”) that supports building messaging systems (see Nicholas Piël’s excellent tutorial “ZeroMQ an Introduction” at http://nichol.as/zeromq-an-introduction).
Martin Lucina and Martin Sustrik, two of the main forces behind 0MQ, put it best in their January 20, 2010, article “0MQ: A New Approach to Messaging” (http://lwn.net/Articles/370307/):
“The low-level nature of the socket API leads developers to re-implement[ing] the same functionality on top of sockets over and over again. Alternatives exist in the form of various ‘I/O frameworks’ and ‘enterprise messaging systems’ but both of these approaches have their own set of drawbacks. The former are generally bound to certain programming languages or paradigms, while the latter tend to be bloated, proprietary solutions with resident daemons that hog system resources.”
0MQ opts not to add a new set of Application Programming Interfaces (APIs) but extends the existing socket API already familiar to programmers. Indeed, the semantics are only trivially different. This also translates into performance that matches and sometimes exceeds traditional socket programming.
Socket programmers are used to treating data as a stream with no inherent record structure. They’ll typically prefix a message with a length field, and once that length field is received, they use that length to issue the recv() call until that length has been met. So, the code fragment ends up looking something like the top portion of Figure 1 (error conditions ignored).
With 0MQ, the programming, as shown in the bottom half of Figure 1, is simpler. Conventional sockets typically present a synchronous interface to either connection-oriented reliable byte streams or connectionless unreliable datagrams. 0MQ sockets present an abstraction of an asynchronous message queue, with the exact queuing semantics dependant on the socket type in use. Where conventional sockets transfer streams of bytes or discrete datagrams, 0MQ sockets transfer discrete messages. 0MQ dictates no internal format for the message (e.g., additional headers, etc.) but treats the contents as an opaque “blob.”
According to “0MQ – The Guide” (http://zguide.zeromq.org/chapter:all), other attributes of messages include:
- 0MQ sends and receives messages atomically (i.e., you get a whole message or you don’t get one at all).
- 0MQ doesn’t necessarily send messages right away, but rather later, at some indeterminate time.
- You can send zero-length messages (e.g., for sending a signal from one thread to another).
- A message must fit in memory. If you want to send files of arbitrary sizes, you need to break them into pieces and send each piece as a separate message.
The programmer has a choice of transport mechanisms for his or her messages, but the message handling APIs are identical. The programmer informs 0MQ of the socket type to use at the start of processing, then uses the same set of calls to send/receive/poll the connections (see Figures 2 and 3).
The high water mark is a hard limit on the maximum number of outstanding messages 0MQ can queue in memory for any single peer with which the specified socket is communicating. 0MQ separates the types of messaging from the underlying transport protocol and provides a simple URL-like syntax for specifying that (e.g., tcp://host.ex.org:9999/). You can use several types of transports when you call zmq_bind and zmq_connect:
- tcp:// is a plain old TCP socket with a host and port number.
- ipc:// uses UNIX inter-process communication such as domain sockets, MQ, or whatever is available.
- inproc:// is an in-process transport that passes messages via memory directly between threads sharing a single 0MQ context.
- pgm:// is reliable multicast messaging that uses raw IP layering and requires special privileges.
- epgm:// is an encapsulated version that uses regular User Datagram Protocol (UDP) to do reliable multicast messaging.
0MQ divorces who binds and connects from the previously mentioned configurations. This means that, unlike classic sockets, who connects and who binds doesn’t matter for the direction or kind of message. All that really matters is whether the connection makes sense for your application.
Conventional sockets allow only strict one-to-one, many-to-one (many clients, one server), or, in some cases, one-to-many (multicast) relationships. With the exception of ZMQ::PAIR, 0MQ sockets may be connected to multiple endpoints using zmq_connect(), while simultaneously accepting incoming connections from multiple endpoints bound to the socket using zmq_bind(). This allows many-to-many relationships.
The authors describe it best: “Traditional network programming is built on the general assumption that one socket talks to one connection, one peer. There are multicast protocols, but they’re exotic. When we assume ‘one socket = one connection,’ we scale our architectures in certain ways. We create threads of logic where each thread works with one socket, one peer. We place intelligence and state in these threads.
“In the 0MQ universe, sockets are clever multi-threaded applications that manage a whole set of connections automagically for you. You can’t see, work with, open, close, or attach state to these connections. Whether you use blocking send or receive, or poll, all you can talk to is the socket, not the connections it manages for you. The connections are private and invisible, and this is the key to 0MQ’s scalability.
“Your code, talking to a socket, can then handle any number of connections across whatever network protocols are around, without change. A messaging pattern sitting in 0MQ can scale more cheaply than a messaging pattern sitting in your application code.”
Low Overhead and Fast Messaging
Formula One racers say it’s easier to make a fast car reliable than a reliable car fast. 0MQ is quite fast, but doesn’t have the features that give such programs as WebSphere MQ their reputation for reliability. However, the component-like qualities of the 0MQ APIs and supporting routines enable reliability to be built into an application.
0MQ implements a mechanism of batch messaging that allows larger blocks of data to be sent over a link at one time for better network utilization. The algorithm used doesn’t adversely affect latency, so using 0MQ may result in better throughput than, for example, using raw sockets.
An important factor is to minimize resources the messaging system requires. Many other MOM solutions are Java-based and have a huge memory footprint. 0MQ may live in as little as two resident pages. For those writing applications to run in virtual machines or in embedded devices, this is a significant advantage.
0MQ handles I/O asynchronously in background threads. So, when you issue a zmq_send(), that operation doesn’t really send the data; it places it in a queue, which a background thread will service (perform the network I/O). These background threads communicate with application threads using lock-free data structures. This means applications need no locks, semaphores, or other wait states: You’re able to control the number of I/O threads servicing your application when you initialize a socket.
Active Open Source Community
0MQ has a large, active community with interaction available via:
- Mailing lists
- Internet Relay Chat (IRC) bug tracking
- Project laboratory.
Multiple Language Support
Language bindings exist for:
- Chicken Scheme
- Common Lisp
- C# (.NET & Mono)
0MQ is available on multiple platforms, including Linux, Windows, Solaris, and OpenVMS.
While in its basic state, 0MQ is used to construct message-switching applications; the APIs can be used to create your own brokering mechanisms.
0MQ has a concept called devices, used to extend the operation, scale, or function of the messaging system you’re constructing. They tend to be used to connect a set of front-end sockets to a set of back-end sockets. They allow an application to be stretched over as many intermediate nodes as needed, and are placed in the topology where it makes sense for the application.
These devices can do intermediation of addresses, services, queues, or any other abstraction you care to define above the message and socket layers. 0MQ provides three simple devices an application may invoke using the zmq_device() API:
- Queue: a request-reply broker
- Forwarder: a pub-sub proxy server
- Streamer: a forwarder for pipeline flows.
It’s likely that application writers will create their own devices to provide functions such as brokering, routing, logging, interfacing non-secure environments to secure applications, etc.
How We Use It
We’ve exploited 0MQ in a relatively straightforward data-gathering system. A central server listens for work from any number of clients that could be running on the same processor complex or on boxes distributed on LAN or WAN connections. The clients periodically gather data, package it, and use the ZMQ::REQ socket type to ship the data to the server. The server sends an acknowledgement to the client and any updated configuration information.
The 0MQ Guide, “Mongrel2 Documentation” (see http://mongrel2.org/doc/tip/docs/manual/book.wiki#x1-640005.2) describes how the software can be used to provide function-rich systems. The guide also provides an excellent way to understand how the APIs are used and how components can be put together. Here are a couple of commercial users of the suite:
- Mongrel2 – a language-independent Web server
- Solvians IT-Solutions GmbH creates Web projects in the financial and stock market environment. They report: “0MQ is used as a MOM for all components in the system … the new design method is capable of processing current dataflow from our client up to 10 times faster than the old system.”
The Python scripts in Figure 4 illustrate the pub/sub socket types. The publisher continuously sends randomly generated temperature and relative humidity figures for a randomly chosen ZIP code.
A client program subscribes to the port that the publisher is sending its weather information on, but specifies a ZIP code as a filter. 0MQ will only satisfy client receive requests based on that filter. Once it has received five weather readings, it will calculate an average, report it, and then exit (see Figure 5, which also shows the launch of the publisher and then four subscribers, each with a different ZIP code).
Getting the Code
Fedora 14 for System z includes 0MQ 2.0.7; there have been many significant enhancements since then. The latest released 0MQ version as of this writing is 2.1.1. Hopefully, the update repository will be at that level soon. We’ve built 2.1.1 RPMs for SUSE SLES10, SLES11, Centos 4.4, and Fedora 14 from the 0MQ git repository. Python bindings are available using the easy_install pyzmq command.
References and Further Reading
“PGM Reliable Transport Protocol Specification,” www.rfc-editor.org/rfc/rfc3208.txt
“0MQ – Less is More,” www.zeromq.org/
“Broker vs. Brokerless,” www.zeromq.org/whitepapers:brokerless
“zmq Man page,” http://api.zeromq.org/zmq.html