#LEARNING MYTHS UPDATE#
#LEARNING MYTHS DRIVER#
#LEARNING MYTHS SOFTWARE#
#LEARNING MYTHS SERIES#

In the words of Josh Wills, a former staff engineer at Google and Director of Data Engineer at Slack, вЂњ weвЂ™re always trying to bring new models into production just as fast as humanly possible.вЂќ Myth #5: Most ML engineers donвЂ™t need to worry about scale Etsy deployed 50 times/day, Netflix 1000s times/day, AWS every 11.7 seconds. Even back in 2015, people were already constantly pushing out updates. HereвЂ™s an area of machine learning where we can learn from existing DevOps best practices.

#LEARNING MYTHS UPDATE#

Since a model performance decays over time, we want to update it as fast as possible. The right question should be: вЂњHow often CAN I update my models?вЂќ People tend to ask me: вЂњHow often SHOULD I update my models?вЂќ ItвЂ™s the wrong question to ask. Myth #4: You wonвЂ™t need to update your models as much One tip for addressing this: train models on data generated 6 months ago, 2 months ago, 1 month ago & test on current data to see how much worse their performance gets over time. Therefore, ML systems perform best right after training. On top of that, ML systems suffer from whatвЂ™s known as concept drift: the data your model runs inference on drifts further and further away from the data it was trained on.

#LEARNING MYTHS SOFTWARE#

The phenomenon in which a software program degrades over time even if nothing seems to have changed is known as вЂњsoftware rotвЂќ or вЂњbit rotвЂќ. Human-Centric Machine Learning Infrastructure (Ville Tuulos, InfoQ 2019) Myth #3: If we donвЂ™t do anything, model performance remains the same Here are some of the tasks that Netflix uses machine learning to solve. I canвЂ™t find the number of ML models that Google, Amazon, or Facebook has in production, but it must be in the order of thousands, if not tens of thousands. In fact, Uber has thousands of models in production. So with 20 countries and 10 models for each country, you already have 200 models. If this app operates in 20 countries, and until you can have models that generalize across different user-profiles, cultures, and languages, each country would need its own set of models.

#LEARNING MYTHS DRIVER#

It needs a model each to predict rider demand, driver availability, ETA, optimal price, fraudulent transaction, customer churn, etc. An application has different features, each feature requires its own model. In reality, companies have many, many ML models. Subsequently, the infrastructure and plans they have in mind donвЂ™t work for actual applications, because it can only support one or two models. Many people from academic backgrounds IвЂ™ve talked to tend to also think of machine learning production in the context of a single model. When doing academic projects, I was advised to choose a small problem to focus on, which usually led to a single model. Myth #2: You only deploy one or two ML models at a time Debugging a machine learning model during training is hard. Setting up the infrastructure so that the right person can be immediately notified when something went wrong and figuring what went wrong is hard. Making your model available to millions of users with a latency of milliseconds and 99% uptime is hard. You can create a couple of endpoints, push your model to AWS, create an app with Streamlit or Dash - if youвЂ™re familiar with these tools, it can take an hour. If you want to deploy a model for your friends to play with, itвЂ™s straightforward. IвЂ™ve heard many data scientists in their early careers telling me how worried they are about not being able to deploy a model since they donвЂ™t get to learn that in school. Therefore, in this summary post, IвЂ™d like to go over the six production myths I often encounter when talking to people who have only experience with machine learning in an academic environment. The talk was originally about the principles of good machine learning design, but most questions I received from the audience was about the myths of machine learning production. It was great to see the growing interest of the academic community in building practical AI applications.

#LEARNING MYTHS SERIES#

In November, I had the opportunity to come back to Stanford to participate in MLSys Seminars, a series about Machine Learning Systems. Takeaways from MLSys Seminars with Chip Huyen

Learning myths