DevOps Pipeline for a Machine Learning Project
- Adding machine learning into this life cycle brings new challenges and changes in a DevOps pipeline.
- We have changed the testing to 25% of the overall data set and keep strict separation of training and testing data for all cycles.
- The third challenge every machine learning application faces in CI/CD cycle while applying to DevOps is the time needed to train the classifier.
- Pros: – – • Easy fit to existing process – – • Full visibility of changes either in code or in data – – Cons: – – • Slow process – – • A lot of “failing builds” for no apparent reason or change in code – – • Complex…
- The main driver for the separation of machine learning is the size of the data set.
Machine learning is getting more and more popular in applications and software products, from accounting to hot dog recognition apps. When you add machine lea…
@analyticbridge: DevOps Pipeline for a Machine Learning Project
Machine learning is getting more and more popular in applications and software products, from accounting to hot dog recognition apps. When you add machine learning techniques to exciting projects, you need to be ready for a number of difficulties. The Statsbot team asked Boris Tvaroska to tell us how to prepare a DevOps pipeline for an ML based project.
There is no shortage in tutorials and beginner training for data science. Most of them focus on “report” data science. A one-time activity is needed to dig into a data set, clean it, process it, optimize hyperparameters, call .fit() method, and present the results. On the other hand, SaaS or mobile apps are never finished, always changing and upgrading complex sets of algorithms, data, and configuration.
There are two different types of applications:
Machine learning often expands functionality of existing applications — recommendations on a web shop, utterances classification in a chat bot, etc. It means it will be part of the bigger cycle of adding new features, fixing bugs, or other reasons for frequent changes in overall code.
One of my last projects was to build a productivity bot — help knowledge workers in the large company to find and execute the right process. It has a lot of software engineering — integration, monitoring, forwarding the chat to the person, several machine learning components, etc. The main one was intent recognition where we decided to use our own instead of online services (such as wit.ai, luis.ai, api.ai) because we were…