Here at Simply Business, we are continually striving to improve our infrastructure for business intelligence. We are currently transitioning from a traditional ETL system built on SSIS to an event-based analytics system using Snowplow, Redshift, and Looker. The goal of this movement is to bring our approach to business intelligence in line with the architectural principles we've adopted such as Agile, TDD, and Git as configuration.
One of the key pieces required to support the shift to event-based analytics is a tool for centralised event logging. Our initial experiment was to build a centralised event logging tool around our existing RabbitMQ infrastructure. We'd been using a home-grown tool for feeding data from our Rails apps into our BI system for the past two years built around publishing/subscribing with RabbitMQ and the Bunny gem. We took the best bits from this codebase and built a sleek new gem called RabbitFeed. RabbitFeed plugs into your Rails (or plain Ruby) apps giving them the power to log and consume events!
RabbitFeed is built around the following beliefs:
RabbitFeed provides the following features that, when combined, set it apart from similar tools:
There is no need to install and administer an additional messaging system if you're already using RabbitMQ.
RabbitFeed provides a custom RSpec matcher allowing you to assert your application publishes the expected events. It also provides a test helper that simulates incoming events allowing you to assert that the events are handled as expected.
Due to the distributed nature of RabbitFeed, each application in your ecosystem will define its own configuration for RabbitFeed. This configuration is in plain Ruby and sits within the application's codebase. This means that when you deploy your application, any changes to how it publishes or subscribes with RabbitFeed get deployed as well.
RabbitFeed offers two simple DSLs that can be used to configure publishing and subscribing respectively:
RabbitFeed includes a DSL for defining the payload of your events, which provides a business-friendly way for non-techies to collaborate with the developers and analysts when defining the event. RabbitFeed converts this into a schema against which each event is validated when published.
The syntax of the DSL is shown below:
# Say we have an application called 'online_shop'
# This application publishes an event every time a purchase is made
EventDefinitions do
define_event('customer_purchases_item', version: '1.0.0') do
defined_as do
'A customer has purchased an item from the store'
end
payload_contains do
field('customer_id',
type: 'string',
definition: 'The unique identifier...')
field('item_id',
type: 'string',
definition: 'The unique identifier...')
field('price',
type: 'int',
definition: 'The price the customer paid...')
field('purchased_at',
type: 'int',
definition: 'The time at which the customer...')
end
end
end
Publishing the event looks like this:
require 'rabbit_feed'
RabbitFeed::Producer.publish_event 'customer_purchases_item', {
'customer_id' => customer.id,
'item_id' => item.id,
'price' => item.price,
'purchased_at' => Time.now.to_i,
}
RabbitFeed provides a DSL for defining which events a consumer will subscribe to and how the events will be handled. This provides a platform-agnostic means to route events from the publisher to one or more subscribers. RabbitFeed uses this information to automatically configure queues and routing within RabbitMQ when the consumer is initialised.
The syntax of the DSL is shown below:
# Say we have a sales bell in the office,
# and we want to ring the sales bell every time
# a sale is made in our online shop
EventRouting do
accept_from('online_shop') do
event('customer_purchases_item') do |event|
if event.price < 10000
sales_bell.ring_for_small_sale
else
sales_bell.ring_for_large_sale
end
end
end
end
Since incorporating RabbitFeed into our applications, we've been able to build a real-time sales dashboard and sales leaderboard in just a few weeks. These things would have been nearly impossible to do on our old ETL infrastructure. If you'd like to see how you can leverage centralised event logging to revolutionise your company's approach to business intelligence, look no further than RabbitFeed!
Want to know more about what it's like to work in tech at Simply Business? Read about our approach to tech, then check out our current vacancies.
Find out moreWe create this content for general information purposes and it should not be taken as advice. Always take professional advice. Read our full disclaimer
Keep up to date with Simply Business. Subscribe to our monthly newsletter and follow us on social media.
Subscribe to our newsletter6th Floor99 Gresham StreetLondonEC2V 7NG
Sol House29 St Katherine's StreetNorthamptonNN1 2QZ
© Copyright 2023 Simply Business. All Rights Reserved. Simply Business is a trading name of Xbridge Limited which is authorised and regulated by the Financial Conduct Authority (Financial Services Registration No: 313348). Xbridge Limited (No: 3967717) has its registered office at 6th Floor, 99 Gresham Street, London, EC2V 7NG.