Database veteran Martin Gaffney isn’t impressed by companies that claim to modernise the database for the cloud, but don’t deliver
By-lined to Martin Gaffney, Vice President – EMEA Yugabyte
There’s currently a lot of pressure on IT teams to perform so-called ‘database modernisation.’ The claim is that this is urgently needed to deliver better overall IT performance and availability, reduce hardware and shrink licence costs—and most desirable of all, provide a stable, agile and future-proofed database operation.
Without doubt, if you execute a database modernisation strategy in the right way you will get all that and probably even more. The problem is that the kind of database ‘modernisation’ most suppliers are touting cannot meet these promises: best case scenario, you might end up with 50% of that list.
Why do I make this pessimistic statement? Well, businesses have finally started to leverage the cloud and done sterling work on application and infrastructure modernisation, but they have not sufficiently attacked the last, and actually the biggest challenge: the database.
This last step is often more complex than the others. Therefore, CIOs are receiving vague promises that providers will “do something.” They aren’t really sure what that “something” actually is, because neither are many of those providers!
A database cloud muddle
Any cloud vendor worth its salt will have the basics in place, like a commitment to a microservices-based architecture and extensive use of containers. However, they are almost certainly still trying to run your core business data on Oracle, with Sun hardware, DB2 on IBM, or on Microsoft SQL Server with some flavour of Linux in its data centre.
That is far from optimal. You can just about make it work, but although that kind of monolithic database runs fine on an in-house cluster, it just doesn’t make sense in the cloud. Monolithic databases simply don’t work well with the kind of cloud applications businesses want to run.
Those systems are great, but were designed for a very different type of IT architecture than cloud. To work (just about) in the cloud, CIOs need to spend significant effort pumping data into them, which massively increases application transaction latency. You can do that until your FD starts asking in the senior management team meetings why she’s still paying for big software contracts, when the board was told moving to the cloud meant leaving all that ongoing OpEx behind.
This brings us back with a bump to the key database modernisation conundrum. You just can’t ‘lift and shift’ Oracle-style business databases in one go into the cloud: that approach doesn’t work, and it certainly doesn’t scale. Putting your transactional data platform into the cloud demands a fresh approach. For a start, you need the biggest cloud server you can get your hands on to run it, while everything else in your new cloud topology is instead running on little connected networks of small, commodity boxes.
The way until now most people have adopted, is to try and put a monolithic database into the cloud by splitting it into lots of little ones… which is a kind of database modernisation, but only of a very limited sort. If you aim to game the cloud by chunking up your core business database into lots of smaller ones, you simply end up with multiple additional requirements, like keeping multiple copies and back-ups synched. Machines get overloaded, and you will be constantly buying more licences to make it work. Reliability is also a big headache, as you’ll constantly be switching between databases, and potentially, not reconciling your transactions in time.
Problems with transaction consistency are not problems you want
Patently, none of this is conducive for serious, transaction-heavy business processes. But of course, you don’t have to try and split the database. So, I hear you ask, is all this avoided by going NoSQL instead? As this class of database was designed to natively cloud scale-out, plus has high availability and resilience built-in, this must be the right database modernisation route…
Well, yes and no, I’m afraid. NoSQL, while cloud native, isn’t great at application transactional consistency, for example. Yes, absolutely, lots of applications will work fine on NoSQL… if they do not rely on the level of transactional/ACID (atomicity, consistency, isolation, durability) transactions serious ecommerce, welfare and financial services applications demand. With a transaction service, you need to ensure that if a customer’s money comes out of an account it will land on time in the right place.
The transaction users I’ve seen who went NoSQL end up with stressed-out engineering teams constantly trying to work around the basic issue that transactions cannot be guaranteed to be consistently applied in a NoSQL database. We ended up in a place where cloud database modernisation was attempted, but not delivered, as neither monolithic SQL or NoSQL worked for us. Both involve compromise, tend to incur engineering time to patch together, and overall just create too much technical debt to be sustainable in the long term.
Yes, some cloud companies offer a distributed SQL database solution that can give you great results—but that will only work in that one single cloud environment. If your core business workflow is on AWS, but you want to move into GCP or Azure either 100% or for one problematic geography, you can’t share the load between them—so you’re still stuck.
The good news is that there is a way to deliver true database modernisation. One which is both open source SQL and distributed at its core. Adopting it means decreased risk and cost, and helps you evade the increased complexity, brittleness, and time spent on constant patching and database hacking.
Is distributed SQL the on-ramp to genuine database modernisation?
A recipe for long-term success and true cloud ROI, is distributed SQL. My company offers such a solution in the form of distributed SQL based on open source PostgreSQL – YugabyteDB. Many customers who tried out the limited first wave of database modernisation techniques I’ve described above have switched to distributed SQL. They actually modernised and future-proofed their database just as the database modernisation brochures and sales emails say they can—but genuinely so. They managed to eliminate their technical debt headaches, and liberated their developers, allowing them to focus on business problems instead of database ones–completing the last mile to full enterprise cloud multi-application exploitation.
Given that full enterprise cloud multi-application exploitation is why you moved to the cloud, this suggests to me that real database modernisation, not just the ‘database modernisation’ fudge, is the best route of travel for forward-looking enterprise data architects.
