Myth of the Monolith

or the Solar System Pattern

flickr photo shared by Image Editor under a Creative Commons ( BY ) license

According to a number of sources, Microservices are the latest and greatest way to set up web sites. They're said to be better than SOA and an improvement on the hated monolith. I won't go into describing either the Microservices Pattern or the dreaded Monolith Pattern as they are described in the links above.

What I will do is try to summarise why monoliths are believed to be 'bad' and microservices are believed to be 'good'. That will give me a basis to introduce 'The Real World'. Not one of the longest running TV shows in history but the place we all live and work in.

After I've thoroughly debunked the myth of the monolith (don't hate me for being over-confident), I'll point out the real pattern that exists in most businesses. That pattern is the Solar System Pattern. And trust me, you're probably already using it in your company.

Why the monolith is so 'bad'

There are a number of reasons that the monolith is thought to be 'bad'. And all these downsides are in fact true.

1. Overly complex Monoliths are believed to be overly complex, leading to a whole host of problems. Complicated things are, well, complicated.
2. Hard to maintain Because monoliths are complicated, they are hard to maintain. A whole ecosystem of people with various specialisms are required to feed and nurture the beast. Monoliths 'do too much'.
3. Hard to reason about Because they are complicated, they are hard to reason about. When a change is needed, too many aspects get pulled into the change. Sometimes hundreds of files need to be updated to add a cross-cutting concern. This leads to people making overly tactical changes.
4. Hard to deploy They are hard to deploy because they need larger servers, and they need loads of configuration for their many many dependencies.
5. Hard to develop concurrently A complex system requires more people to maintain it, meaning that developers start to clash with each other, requiring ever more exotic mechanisms for maintaining reasonable pull requests.
6. Overly shared ownership Because so many people are working on the code together, they can have a weakened sense of pride and ownership which has consequences in uptime and code quality among other problems.
7. Easy to accumulate technical debt Because of all the above, it is easy to miss signs of technical debt, and easy to justify not dealing with it as there is a lot of work to keep things in a good place.
8. Testing coverage is critical If automated test coverage and quality isn't good, the testing surface area becomes unmanageable.

Why microservices are so 'good'

Microservices are believed to be the solution. Simple, separate, and easy to deploy, these little gems allow many small teams to collaborate with each other within an organization without ever having to worry that there is a clash. Each team owns a service, operating and deploying it on their own schedule.

The benefits are:

1. Separation of concerns Microservices don't need to know about other services. They can be developed in isolation and a new release doesn't require deploying the entire company server farm at once.
2. Allows people to get the domain space in their heads Given the surface area of a microservice is small, developers and product owners can quickly pick up the code and functionality and understands what needs doing.
3. Small interfaces Takes the SOLID Principles and applies them to services. Interfaces between services (just like objects), should be constrained, not leak data, be responsible for themselves and not provide their guts to others.
4. Scalable Small services that are separable are much easier to scale.
5. Can choose the best tools and technologies Because services are small, developers can choose technologies based on the best tool for the job (which unfortunately says nothing about ability to manage and maintain said diversity)
6. Ownership Small teams can truly own a microservice which means it will get the attention it needs.

The problem: the real world

And now for the problem. Does the real world accept our principles of beautiful and coherent design? Does it allow us to break the world into small pieces? And, really, should it?

Of course, you answer. The case (above) is compelling.

However, many of the current blog posts about microservices exist in the following matrix:

1. Usually added to a monolith People are focussed on the journey to getting to microservices. They have taken functions that could be extracted and turned them into services. These functions can be things like search and PDF generation.
2. It's the future Getting rid of the monolith is coming in the future, but is not yet here.
3. Performance Most often, companies move because of performance demands, rather than architectural cleanliness. Is the architectural cleanliness a post hoc justification?
4. Workloads Businesses separate their technologies due to different workloads such as data warehousing or telephony systems rather than logical separation.

The problems

The first problem in microservices is actually the easy one, but is still hugely hard. It's all about managing a fleet of services. Managing dozens of servers, even with Docker and modern cloud farms, is still hard. They need to be clustered, kept running and scaled. This can take a lot of time to get right. And reliability of distributed systems is hard. See here and here for a flavor.

And then there's the hard problem. Most businesses have complex business domains that interlock tightly. Exploding these into microservices does not produce separate concerns as the domain objects interact. This leads quite quickly to tightly coupled microservices with wide interfaces, the exact opposite of what is desired.

Introducing the Solar System Pattern

Most real businesses have a hefty domain model with many complex interactions, and a number of systems that need to collaborate over the domain model. Those systems are forced to split on technology grounds because they process completely different workloads, such as online order processing, business intelligence, lead management and real-time personalization. Unfortunately, despite the best efforts of microservices, each of these workloads needs access to the full suite of data. Can you imagine the CFO agreeing not to see the sales numbers because they shouldn't be shared between system boundaries for architectural reasons?

The Solar System Pattern is about accepting the real world. It acknowledges that most businesses will have a complex domain model, that for technical reasons will need to be distributed to a number of systems.

OK, so why Solar System?

Those multiple systems, that need to have full or substantial knowledge of the domain model, I call Suns. They are the core of a solar system. And in reality, it is very rare for our techie solar systems to have just one sun. The most typical small company binary system looks like 2 suns:

1. The main transactional system
2. The data warehouse/big data system

Even slightly larger companies, such as ours, have many suns, all collaborating over the same domain model.

And solar systems have planets. These are the microservices that circle around the sun(s). They are generally smaller and more self-contained with less required knowledge of the domain model. They peel off functionality that really doesn't need to understand large bits of the domain model. They can really adhere to the single responsibility principle.

Despite efforts to limit the amount of domain knowledge that suns need, they are always bigger, more complex and share more data with each other than the microservices that orbit them.

The Suns

flickr photo shared by rmatthendrick under a Creative Commons ( BY-ND ) license

A bit more detail.

What is a sun in the Solar System Pattern? It's a monolith. It's an encapsulation of a complex domain model where the different components don't easily separate from each other.

In a nutshell, a sun is a system that holds data and manipulates a substantial portion of a domain model.

For example in our domain (we are the UK's largest business insurance broker), one of our suns (we have more than one) is our main policy management system. It is a 'quote and buy' system and is used to manage all our customers quotes and policies. It handles when a customer wants to change any aspect of their policy, and deals with all the complexities. Our consultants use it to help our customers. And it provides an array of functionality. We frequently look to split functions out, but find it hard to see good seams.

Inside this sun, sits our domain model. Our domain model has customers with quotes and policies. The policies have endorsements and changes and policy documents. Our customers have answered a series of questions about their insurance needs that led to their being offered and then purchasing a policy. Those questions are based on a questionnaire that was answered. All of these elements are part of our domain model.

And on and on. All of these elements interlink in a domain defined way. We've even created an internal site to document it called Simplypedia. There you can look up what a Policy actually is to us and which other domain objects it connects with.

Sure it would be nice for policies to relate to customers in completely independent ways, but they simply don't (for example, a policy provides insurance to a customer, and therefore needs to know who that customer is).

As I mentioned, we actually have multiple suns, since our quote and buy and CRM system interacts with a number of other systems. For example, it interacts with our Big Data analytics system which wants to analyse the full content of our customer, policy and other domain objects. In order to do that, the analytics system requires a very wide interface with our first sun, the quote and buy system.

And what about the lead management system that prioritises who we contact based on the answers our future customers gave us? It needs all the data as well. It uses that data to prioritise who we contact.

The planets (aka microservices)

The microservices in our solar system are components with limited or well defined interfaces. We've got a PDF generator that crunches HTML. That's a well defined interface. We've got a mail queue system that sends out quote emails that get pushed to it from the main transactional sun. These are tight and well contained interfaces and we can just deploy them at will. And in fact, we hardly ever have to deploy them.

Domain model changes are a challenge

One of the big challenges of real businesses that are iterating rapidly (or not so rapidly) is to keep the domain model up to date. Domain changes are typically cross-cutting concerns and suffer from having to be implemented everywhere that the domain model objects that are changing are used. In practice, this means updating each Sun to communicate, store and interact with the changed domain model. The more places you have to do this, the harder it is.

For example, we recently changed the name of Product in our domain model to Vertical. That small change needed to be rolled out to every system (all Suns) that knew about Product so that it now knew about Vertical. This meant for us: our quote and buy system, our analytics system, our lead management system and our renewals system to call out the major ones. That entailed the modification of APIs, data migration, schema changes and other machinations in order to flow the change through. If we hadn't done the work, we would have had systems with inconsistent domain models, a sure cause for extensive pain. For example, it leads to people not understand a legacy term that is used in only one system and therefore misusing the data.

More generically, the cost of not flowing the domain changes through is that different systems will have different representations of the business. Changes to the domain model are inherent to all businesses, and the maintenance issue is inescapable. They used to call this metadata management but now we call it domain driven design. Microservices don't help here, other than if they are able to avoid knowing about the domain model entirely, in which case they don't need updating. If the microservices are intimate with the domain model, then the more microservices there are, the more changes are needed.

Especially since the reasons we have multiple suns (let alone microservices) is that we need to separate them for technical (functional or performance) reasons.

Summary

There's no getting around your domain model. Typically it is tightly linked and just doing object oriented decomposition on it is challenge enough. When you start to want to separate it into multiple microservices, you start having to think about distributed transactions and numerous other mechanisms, just to be able to create 'small' services. This can be worth it, but typically because performance or some other metric has become too much of a problem. Martin Fowler says that complexity is that measure, but I'm not convinced. Artificial boundaries between services don't mean that the services have been correctly separated, and even worse, if a cross-service domain change comes along, the amount of work can skyrocket in a microservices architecture.

So dismantle your majestic monolith at your peril. You've got a domain model to keep alive.