ML Strategy — what can make it successful? — part 1
Why strategise?
Machine Learning projects can be a lengthy endeavour. Processing, visualising, and finally training the data can take weeks or months. After working on your project for some time, you achieved a decent accuracy, but it might not be good enough for your application. You decide to diversify your datasets, try a different algorithm or tune the hyperparameters.
However, the problem is that if you choose poorly, you can spend months going into a direction that is not necessarily going to do you any good. So, it is very important to analyse a machine learning problem in a way that will point you to the right direction and allow for you to find the most promising things to try.
Orthogonalisation
For a supervised learning system to do well, you predominantly need to focus on the following elements:
1. doing well on the training set,
2. doing well on the dev set,
3. doing well on the test set,
4. having a decent cost function for your model to perform well in real world.
If your algorithm is not fitting the training set well on the cost function, you might want to tune the algorithm. Or even train a bigger network. Another option would be to switch to a better optimisation algorithm, like the Adam algorithm.
On the other hand, if the algorithm is not fitting the dev set well, you might want to look into regularisation. And if it does well on the dev set but not the test set, it probably means you’ve overtuned to your dev set, and you need to go back and find a bigger dev set.
And finally, if it does well on the test set, but it isn’t working as well in real life, then what that means is that you want to go back and change either the dev set or the cost function.
This is just a taste of the orthogonalization process. What’s the trickiest in this process is figuring out exactly what’s wrong, and then have exactly one parameter, or a specific set of parameters that helps to just solve that problem that is limiting the performance of machine learning system.
Single Number Evaluation Metric
Comparatively, your progress can be much faster if you have a single real number evaluation metric that lets you quickly tell if the new thing you just tried is working better or worse than your last idea. Therefore, it is often recommended to pick a classifier rather than to use two numbers: precision and recall.
In the machine learning literature, the standard way to combine precision and
recall is something called an F1 score. And in mathematics, this function is called the harmonic mean of precision P and recall R. But less formally, you can think of this as some way that averages precision and recall.
Evaluation metric allows you to quickly tell if classifier A or classifier B is better, and therefore having a dev set plus single number evaluation metric distance to speed up iterating. It speeds up this iterative process of improving your machine learning algorithm. So having a single number evaluation metric can really improve your efficiency or the efficiency of making those decisions.
Satisficing and Optimizing Metrics
It’s not always easy to combine all the things you care about into a single row number evaluation metric. In those cases, it is sometimes useful to set up satisficing as well as optimizing matrices.
One thing you could do is combine accuracy and running time into an overall evaluation metric. It may seem a bit artificial to combine accuracy and running time using a formula like a linear weighted sum of these two things. There’s something else you could do instead: choose a classifier that maximizes accuracy but subject to that the running time.
For instance, if accuracy is the optimising metric, you want to maximize accuracy. You want to do as well as possible on accuracy, but that running time is what we call a satisficing metric. Meaning that it just must be good enough, it just needs to be less than some number, and beyond that you don’t really care. This will be a reasonable way to trade off or to put together accuracy as well as running time.
In summary, while structuring your ML project there is a lot of things you need to pay close attention to. One of the most vital things is the optimising and satisficing metrics. Using those, you can have an almost automatic way of quickly looking at multiple core size and picking the best one.
