By Krzysztof Sopyla, Head of Machine Learning and Data Engineering at STX Next,
Krzysztof Sopyla
Uptake of artificial intelligence (AI) and machine learning (ML) technologies has increased significantly in recent years. In fact, STX Next’s 2021 CTO report found that around two-thirds of CTOs have now implemented ML in to their businesses and a further three-quarters believe it is the most likely technology to come to prominence in the next two years.
For businesses of all industries and sizes, the volume of structured and unstructured data created every day has grown exponentially, in large part due to the increased usage of technology to support business. For organisations and end users, AI- and ML-powered products can assist in processing it, thus streamlining operations, eliminating human error and saving on resources.
In order to maximise the potential of ML, teams must be fully aware of the complexity and unique challenges that come with these types of projects, and take the necessary steps to overcome them.
Understand the business goal
Before any coding work starts, it’s vital to have a detailed understanding of what is desired by the client and be able to communicate that to the rest of the team, who might not always be present at every meeting.
This means understanding a business’s purpose and budgets, and then being able to recommend the best course of action. There could be a simple solution to a client’s problem that takes a project in a completely new direction. At the same time, budget constraints could mean that you need to cut corners in some areas and prioritise others.
It is important to understand how it may need to be tailored to fit the needs of different regions. This should be carefully considered and planned out before the development process begins.
Keep a record of everything
Documenting properly is key to budgeting and knowing exactly where money is spent. Where projects might be funded from different sources, it’s important to keep a record of what costs have been incurred.
Having the right documentation also makes it easier to perform maintenance on projects further down the line.
Recognise bad data and data bias
Data sets can be vast, complex and diverse. ML is crucial to organising this information and accurately aligning data with the relevant markets, so that it can be used in a way that will add value to the business.
The reality is, when a business first begins working with customers, the right data is often not available or in the wrong format. Collecting and annotating data is a very expensive process, and we rely on data engineers to put this right. This is why it’s so crucial to prepare data accordingly, so that when the moment for expansion arrives, businesses are equipped with all the information needed to make the right decisions.
Make use of your subject matter experts
If possible, ensure that you have access to an expert in the field. In an ML project, the code will often replace something that was previously performed by a human. It’s important then that you have someone who can confidently tell you whether your code can perform to the expected standard.
There is no feedback like feedback from someone who will actually use the product every day.
Manage goals effectively
Managing the expectations of your client as well as those placed on your teams is a delicate task. While it’s a good idea to set goals for yourself and the people you’re working with, be cautious because you can quickly find yourself burned out if you fall behind.
It’s worth being aware of the limitations and the fact that for different ML tasks, the simpler the solution, the better. This will mean no big data processing, no complicated transformations of that data before it goes into the model and ultimately, a much simpler implementation.
Manage performance
When deploying a model to production, performance matters. Once deployed, technical teams should take time to measure and test performance, and use this data to make any necessary improvements and changes.
If a product is deployed internationally, monitoring should be localised so that performance can be reviewed across different regions. Occasionally, businesses will have to prioritise speed or accuracy, but this can be managed according to the judgement of team members, clients and relevant experts.
ML’s potential to create efficiencies and streamline processes frees up businesses to take on more projects and extend operations into new markets. Getting to grips with this tech is certainly a challenge, but one that can facilitate global expansion if the right amount of attention is given to mastering and maximising its potential.
