By Itamar Ben Hemo, the CEO and co-founder of Rivery
Many businesses fail to implement effective data projects, with research from Gartner estimating that 85% of data science projects fail to go into production or to generate any value. A solid data strategy is crucial to the digital transformation of businesses. IDG research also suggests that just one in three data science projects will succeed. The urgency and ability to leverage data has never been more important.
As new data job roles, technologies and organisational cultures emerge, businesses are presented with the opportunity to extract more value from the data available to them. In response, many businesses are looking to employ data experts to interpret advanced analytics and develop more robust insights. However, relying on data experts alone can prove problematic on the road to digital transformation. The number of obstacles that data experts face requires other alternatives. To overcome this, businesses must look to train internal talent to become data experts, and transform their business’s ability to gain insights from data.
The role of data experts vs business users
There is a growing trend for the artificial divisions between data experts and business users to break down, with data experts becoming more business-minded and business users learning to ‘self-serve’ with data. One aspect of this is the rise of roles such as ‘analytics engineer’, which help to bridge the gap between IT and data consumers within an organisation. Analytics engineers collaborate with the team to analyse the data, to ensure that the business can use the high-quality insights generated from their work. Together with wider teams, these engineers help to set up and activate a truly modern data stack.
The need to train existing employees
Rather than relying solely on hiring qualified data experts, business leaders should aim to train their existing workers with data skills: this can help to keep costs and overheads down. Data literacy courses are already becoming common in many companies, and large organisations such as Bloomberg and Adobe are going further, with in-house digital academies dedicated to training workers in how to use data.
Training analysts to use low-code or no-code tools for data management costs far less than hiring a data scientist. By removing bottlenecks in daily data operations, teams that need analytic dashboards to make decisions for campaigns don’t have to wait and can focus more on revenue-generating activities.
Training existing employees is particularly powerful because they combine newly acquired data skills with their existing domain expertise to extract maximum value from the data. These ‘data citizens’ will be able to extract value from data without waiting for a separate team of data experts or scientists to do it for them.
Unlocking business value through new tools
Democratising access to data within your organisation and unlocking the business value of data requires the right tools. Data management is critical to ensure data is delivered to the right team within your business, in a condition where it can be used, without the bottlenecks and delays that can come from relying on a central data team.
Data management deployments automate procedures into one framework, making it simpler for business users to extract value. Along with tools such as data quality management, data validation ensures that data meets the standards required by business users.
Perhaps even more important is Reverse ETL, which turns the normal job of data warehouses on their head to direct a stream of valuable data directly to the teams which need it most. Reverse ETL reverses the traditional process by which data is loaded into a data warehouse, by first extracting it from a data warehouse and then loading it into your operational systems.
In Reverse ETL, the data is loaded from the data warehouse and then fed directly into business software such as ERP (Enterprise Resource Planning) or CRM (Customer Relationship Management). Sales or marketing teams have data delivered directly into the applications they use in their daily work, meaning there’s less training required to understand it.
For example, this can be used to deliver personalised offers based on purchase history or more precisely targeted marketing campaigns. It’s key to breaking down the barriers between data and the data consumers within a company, removing the burden from overworked specialist data teams.
Delivering a self-serve approach to data
Along with these technological changes and job role evolutions around data, there is also a new organisational approach to how data works within companies; a data mesh. In short, data mesh offers a decentralised and ‘self-serve’ approach to delivering data throughout an organisation. Rather than relying on a centralised data team – where the warehouse is controlled by hyper-specialised experts – data is organised via shared protocols, in order to serve the business users who need it most.
The significance of this is that it helps empower teams to access the correct data they need, right when they require it, via the distribution of data ownership across the organisation. Whilst the concept of data mesh isn’t necessarily new, the key to operationalising this approach effectively is the introduction of a platform or universal interoperability layer that facilitates the connection of domains and associated data assets within it. Companies can then use a platform that will help them to connect all the dots and manage the entire operation, in order to fully operationalise the approach.
Being aware of the business value of data is no longer enough. Companies need to start adopting a “data as a product” approach as data mesh will be the core to enabling the application of the product life cycle to data deliverables. By applying product thinking to datasets, a data mesh approach will ensure that the discoverability, security and explorability of datasets are retained. Teams are then better prepared to swiftly derive the most important insights from their data.
Getting the right data access, for the right people
It is critical that teams are able to access the systems and processes that they’re already using, whilst providing users with the tools that they need to self-serve. If analysts and engineers are unable to access the data they require, a central data team may then become a bottleneck. Data experts should work in collaboration with other teams from a majority of functions to provide users with the skills that they need to self-serve. This approach will ensure that businesses can evolve to have data citizens throughout the company that are able to self-serve data as a product, without the need for hyper-specialisation. Through these new approaches businesses will be able to gain the insights that they need to act on data in real time and make insightful decisions.
