Mark Balkenende, Senior Director Product Marketing at Matillion
Businesses today aren’t short of buzzwords and philosophies to describe their approach to data, with DataOps and Data Fabric among the most popular. Sometimes it can be tricky to differentiate these concepts and decide which helps businesses compete with data and execute their digital strategy most effectively. On the surface either approach – or even both in tandem – can provide a solid foundation for a company’s data strategy, but dig deeper and it becomes clear that they offer unique solutions to different organisational problems.
Both approaches are driven by businesses’ urgent need to cut through data sprawl and deliver meaningful insights as close to real-time as possible. It’s something many find difficult, with countless firms still struggling with the negative impact of slow data migration and maintenance. Our research for example suggests two-thirds (66 per cent) of data decision-makers and data team members believe their organisation is wasting time on data preparation. Certain data types are becoming blind spots for data teams too and compounding the issue, with 44 per cent of these teams admitting that dealing with a diverse range of data types is a challenge.
All of these points indicate that existing data migration practices are not only outdated and a barrier to team productivity, but also a risk to business’ bottom line. So, what’s needed to effectively harness the power of data as a strategic asset?
First, leaders need to recognise which elements of their data infrastructure – from data types to volumes – are likely to offer the best foundation for data utilisation and overall success. That will then help to inform whether a DataOps or Data Fabric approach is the right option for them and their organisation. Both tend to be leveraged when businesses are experiencing bottlenecks in delivering their current data and analytics services, so truly understanding how they can use data to its fullest potential is essential.
Assessing the advantages
DataOps: The concept of DataOps comes from DevOps, which most organisations have already adopted. Unlike DevOps, however, DataOps takes the best of the common practices in modern software development – including continuous integration, unit testing, and automated deployment – and applies them to data projects. When combined, these processes empower data engineers and users to assemble data pipelines from end-to-end and centrally manage the DataOps project.
DataOps has the potential to instil greater confidence in teams to make changes to data without needing to worry about affecting its quality or delivering it late. It’s potentially transformative because it allows organisations to move from analysis to insight more seamlessly while potentially reducing the total cost of data operations by 39 per cent, according to IDC.
Data Fabric: Similarly, the role of a ‘data fabric’ is to make it easier to streamline and handle enterprise data. The concept tends to be a better fit for organisations investing heavily in advanced technology and have mature data stacks which require different data integration styles. A Data Fabric contains metadata on all existing and new data coming into a business ecosystem – put simply, this is data about data – and uses that information to take action and continuously improve the data integration design and delivery process.
Organisations often want a full picture of the lifecycle of their data, so they can centrally see, manage and share information across their data estate. Data Fabrics enable that unified view of an organisation’s data so users can work together to share metadata – data that describes other data – and knowledge about core data, with very little input required from data teams.
Choosing which approach to adopt first
Every organisation is unique, as is every data strategy but typically it’s businesses with clearly defined data roadblocks across the enterprise that have more to gain from a DataOps approach in the first instance. It’s likely to show the largest benefit in the shortest amount of time by helping to streamline manual processes or fragile integration points that hinder data teams on a daily basis.
That said, in cases where an organisation’s data delivery process is slow to reach customers, then a data Fabric approach may prove more effective, as the firm has a more urgent need for a flexible, rapid, and reliable data delivery method.
A dual strategy for maximum efficiency
A DataOps approach is considered an essential component of a wider ‘Data Fabric’, meaning once an organisation has successfully implemented DataOps and set up solid data governance practices, they have the foundation to build on a Data Fabric. The latter approach uses semantics and knowledge graphs to build augmented data integrations, and DataOps processes are then built from the output of lower levels of the data fabric – allowing the two to work fluidly with one another.
Businesses are increasingly opting for the dual approach to maximise the use of their data, and for IT leaders in those organisations it’s strongly advisable to follow these steps:
- Instil the discipline of DataOps. It’s a discipline rather than a technology, and when you apply the technology without the discipline, you risk failure. So, start with a data platform to give yourself future growth potential.
- Make metadata available to everyone. Every tool in the business can benefit from it.
- Invest in pioneering tools and integrate them using metadata. Cover the basics — such as data ingestion, ETL, analytics and crucial end-to-end testing of the pipeline.
- Choose ingestion tools that maximise the capture of metadata at the entry point. Ensuring the business doesn’t lose the context of the original source – i.e. where data originates from – is often one of the most crucial pieces of metadata.
Empower organisations for innovation
As explored above, DataOps complements the overall Data Fabric approach, making the two approaches stronger and more effective in driving business insights when implemented together. When re-evaluating the makeup of your organisation’s data strategy, it should be a case of asking yourself “where do I start,” rather than “which strategy should I implement,” so you can give their business users back the power they need to innovate with data, and make better decisions, faster.