Centralised Event Logging with RabbitFeed

Here at Simply Business, we are continually striving to improve our infrastructure for business intelligence. We are currently transitioning from a traditional ETL system built on SSIS to an event-based analytics system using Snowplow, Redshift, and Looker. The goal of this movement is to bring our approach to business intelligence in line with the architectural principles we've adopted such as Agile, TDD, and Git as configuration.

One of the key pieces required to support the shift to event-based analytics is a tool for centralised event logging. Our initial experiment was to build a centralised event logging tool around our existing RabbitMQ infrastructure. We'd been using a home-grown tool for feeding data from our Rails apps into our BI system for the past two years built around publishing/subscribing with RabbitMQ and the Bunny gem. We took the best bits from this codebase and built a sleek new gem called RabbitFeed. RabbitFeed plugs into your Rails (or plain Ruby) apps giving them the power to log and consume events!

Main Principles

RabbitFeed is built around the following beliefs:

1. The payload of the event should be well-defined. Business owners, data analysts, and developers work together to define the event payload.
2. The payload of the event can change over time. Subscribers must be able to handle different versions of the event.
3. Event processing should be resilient to network connectivity problems and server crashes.
4. The infrastructure for publishing/subscribing can be swapped from RabbitMQ to a different infrastructure with no change to the interface.
5. Event publishing requires no knowledge or concern for where or how the events are used downstream.
6. The subscriber defines from which applications and which types of events it will consume.
7. Publishers and subscribers should be developed via test-driven development.

Why RabbitFeed?

RabbitFeed provides the following features that, when combined, set it apart from similar tools:

Take advantage of your existing RabbitMQ infrastructure

There is no need to install and administer an additional messaging system if you're already using RabbitMQ.

Test-driven development of publishers and subscribers

RabbitFeed provides a custom RSpec matcher allowing you to assert your application publishes the expected events. It also provides a test helper that simulates incoming events allowing you to assert that the events are handled as expected.

Configuration lives in your application and can be put into source control

Due to the distributed nature of RabbitFeed, each application in your ecosystem will define its own configuration for RabbitFeed. This configuration is in plain Ruby and sits within the application's codebase. This means that when you deploy your application, any changes to how it publishes or subscribes with RabbitFeed get deployed as well.

Configuration is in plain English

RabbitFeed offers two simple DSLs that can be used to configure publishing and subscribing respectively:

Publishing Events

RabbitFeed includes a DSL for defining the payload of your events, which provides a business-friendly way for non-techies to collaborate with the developers and analysts when defining the event. RabbitFeed converts this into a schema against which each event is validated when published.

The syntax of the DSL is shown below:

#  Say we have an application called 'online_shop'
#  This application publishes an event every time a purchase is made
EventDefinitions do
  define_event('customer_purchases_item', version: '1.0.0') do
    defined_as do
      'A customer has purchased an item from the store'
    end
    payload_contains do
      field('customer_id',
              type: 'string',
        definition: 'The unique identifier...')
      field('item_id',
              type: 'string',
        definition: 'The unique identifier...')
      field('price',
              type: 'int',
        definition: 'The price the customer paid...')
      field('purchased_at',
              type: 'int',
        definition: 'The time at which the customer...')
    end
  end
end

Publishing the event looks like this:

require 'rabbit_feed'
RabbitFeed::Producer.publish_event 'customer_purchases_item', {
  'customer_id'  => customer.id,
  'item_id'      => item.id,
  'price'        => item.price,
  'purchased_at' => Time.now.to_i,
}

Subscribing

RabbitFeed provides a DSL for defining which events a consumer will subscribe to and how the events will be handled. This provides a platform-agnostic means to route events from the publisher to one or more subscribers. RabbitFeed uses this information to automatically configure queues and routing within RabbitMQ when the consumer is initialised.

The syntax of the DSL is shown below:

#  Say we have a sales bell in the office,
#  and we want to ring the sales bell every time
#  a sale is made in our online shop
EventRouting do
  accept_from('online_shop') do
    event('customer_purchases_item') do |event|
      if event.price < 10000
        sales_bell.ring_for_small_sale
      else
        sales_bell.ring_for_large_sale
      end
    end
  end
end

Summary

Since incorporating RabbitFeed into our applications, we've been able to build a real-time sales dashboard and sales leaderboard in just a few weeks. These things would have been nearly impossible to do on our old ETL infrastructure. If you'd like to see how you can leverage centralised event logging to revolutionise your company's approach to business intelligence, look no further than RabbitFeed!