(in the voice of The Clash) Should I break or should I mono (lith)?

(originally posted on LinkedIN)

Been at Synopsys 5+ months now and having a wonderful time onboarding and learning about the scale challenges this company has to solve for, but I really felt like I should do a brain dump on one of my favorite projects at my previous company (LinkedIN) - breaking up the frontend API service monolith.

In both of my previous employers, breaking up the monolith was a major, prioritized project. The reasons why, the end state and solution were different, based on the needs of the company/org.

It's always break, break, break... (or why we decided to break up the monolith)

1. The codebase was massive (4 million lines of hand written code, another 1-1.5 million lines of generated code)

2. Catered to most of the lines of businesses at LinkedIN. This meant that with thousands of developers committing code, deploying and releasing this monolith became a bottleneck. While we had a true north of multiple deployments/day, we were satisfying ourselves with 1 per day. From a product velocity perspective, this was unforgivable.

3. Size of codebase and active developers also meant that we did thousands of builds a week. Parallelization and scaling CI infrastructure become a forever project. As you would guess, this had diminishing returns. For example, the first few projects helped bring our CI from 65 minutes to 15 minutes. But subsequent projects could only hold back regressions (up to a point)

4. To make things worse, this service's size and scale was unique within LinkedIN - as a comparison, the next largest service was 1/10th the size and an order of active developers fewer. This translated to frequent prioritization calls between my business org and central developer productivity orgs.

...If you do, there will be trouble... (or Gotchas to watch out for)

Rather than talk through the specific solution we implemented, since that would be very LinkedIN specific, I wanted to highlight the caveats, pitfalls and gotchas to watch for, should your company decide to break up any monolith you own:

1. Single Responsibility Principle: If you do break up the single monolith the typical guidance is to break it up into purposeful microservices. The gotcha to remember: the definition of "purpose" is ambiguous. So when choosing the microservice to carve out from the monolith, have an explicit definition that specifically limits the total number of services you will end up with.

2. Services or Microservices: Related to #1 above, while the "industry standard" is to break it up into microservices, remember organizational boundaries are around capabilities (e.g. Messaging, Notifications, Sales, Supplier) rather than microservices - so find a suitable, long term domain-specific definitions for your "services", which may be more than a microservice.

3. Permanent Overhead: operational (i.e. availability/uptime or support in production) or developer (ensuring dependencies are fresh, builds are green, service is in active state and receives sufficiently frequent commits, etc.) or talent (e.g. developers now need to understand understand how the deployment process works)

4. Talent gaps: Management should account for new skills that the engineers on the team need to pick up. With the monolith, most organizations would probably have a central team (call them release engineers, devops, SRE, what have you) that deploys and releases it. BUT with this breakup, the individual teams need to know how to troubleshoot and fix the broken deployment pipelines especially since it would be all automated (it is all automated, right :D) and only rarely does it need manual push.

5. Forever project: This large a scale, usually means the project will take years and patience will wane. So set aggressive goals in the first few quarters and ensure your team isn't too pi**ed off when compromises are needed (perfection being the enemy of good enough) or if you don't hit 100% (smaller monolith is still better than a large one). (in my own project, we hit 85% and called it good enough).

...It'll be here till the end of time... (or In conclusion)

Breaking up monolith as a project should not be considered lightly. Having done it successfully twice now, the only thing I would change is to ensure there is ONE single owner of the architecture and operations of the monolith. That owner's role should be strategic and high enough in the reporting chain to make decisions for an entire org (rest of the pieces will work themselves out).