In every company I've worked at that does ML work, the data scientists building the models don't have a strong software delivery background. They aren't building a production system with fail safes and redundancy. Their focus is the model itself. Consequently, when they stand up services in front of these models they are not even close to "production grade". I'd hope that a company focused on ML (versus an enterprise just dipping their toes into ML) would have solved these problems, but I wonder if it's related.
I have to frequently work with Data Scientists to learn their ML models and scale them in our production environments. A lot of times the models are not even statistics, it's just linear algebra. They have a tendency to go for approximate algorithms when an exact algorithm could be written. I largely blame the company for this and not the data scientists. Every company wants to be in the ML game but may not have the volume and variety of data to warrant a use case. When your favorite hammer is ML then every data problem is a nail.
I work in industry (sort of) and this is very true.
One of the reasons for this is that the people designing the systems don't know how to do anything else.
There's such a huge, huge gap between using ML to do the same thing everyone else is doing (say... image classification, or maybe regression on volume vs sales...) and solving unique business problems using ML.
This is sadly so consistent with what I'm seeing at a big corporation. We are working so hard to make a centralized ML platform, get our data up to par, etc. but so many ML projects either have no chance of succeeding or have so little business value that they're not worth pursuing. Everyone on the development team for the project I'm working on is silently in agreement that our model would be better off being replaced by a well-managed rules engine, but every time we bring up these concerns, they're effectively disregarded.
There are obviously places in my company where ML is making an enormous impact, it's just not something that's fit for every single place where decisions need to be made. Sometimes doing some analysis to inform blunt rules works just as well - without the overhead of ML model management.
It sounds so painful as someone all-in this area. But I have to agree on a task of overdoing with fancy models. Nevertheless, the most common ML algo in industry is still linear regression along with boostraping.
Sure, and I understand this sentiment. But in practice/industry, it's best not to build/deploy models you don't understand. Edge cases in ML models that are fully automating business decisions can be pretty dangerous. I think the likelihood of a penicillin level discovery happening when I'm trying to train a model to make better marketing budget decisions for a company is quite low.
I recently got the assignment to "do ML" on some data. I hadn't done anything in the area before, and a couple of things surprised me:
1. Most of your time is spent transforming data. Very little is spent building models.
2. Most of the eye-grabbing stuff that makes headlines is inapplicable. My application involves decisions that are expensive and can be safety critical. The models themselves have to be simple enough to be reasoned about, or they're no use.
You might argue that this means what I'm actually doing is statistics.
I completely agree with this sentiment, I've seen a lot of people throw ML at problems because they don't know much mathematics. Especially when you have a lot of data, I can understand the allure of just wiring up the input & output to generate the model.
I think there's a lot of truth to this. I'm new to ML, still going through the ropes on some online courses, but already I can see that, once I get a bit of muscle memory in setting up models etc, there's a whole lot of power and efficiency to be unlocked by using simple models - specifically in CS/X and Marketing. Obviously model quality matters, so you have to have proper monitoring etc, but this stuff is low hanging fruit and should enable teams to be so much more efficient.
This is good humoured satire and reminds me of my first job as a data scientist straight out of maths school. I was in a team who’s very existence was to advocate for ML everywhere and anywhere, without much success because everyone (including the stakeholders) failed to start from the fundamentals of the problem.
Luckily I got out, upped my game and corrected course. Now in my current role it is all about starting from basics and letting our insights as well as the problem solving process (aka engineering) guide our solution. The modelling approach we thought we’d do in the first 3 months turned out to be absolutely wrong 7 months later.
Don’t get me wrong that ML is a waste of time. If you need to do predictive modelling to solve your problem and if the complexity is high enough that an ordinary linear model won’t suffice (they usually don’t) or you need to model uncertainty as well, then ML may very well get you from point A to B. And achieving that step can be a worthwhile, rewarding and challenging endeavour.
The issue is when ML becomes a substitute for the fundamental problem solving or is treated as a solution when it isn’t.
I’ve had the opposite experience. After finishing grad school and working in industry for a long time now, applying ML to computer vision, NLP, causal inference and quant finance for production systems, the vast, vast majority of problems are such that the training and test set are defined by very precise business operating constraints and it’s known with a high degree of certainty that future examples to which the model is required to generalize will also be from the same distribution reflected in model training, and any differences are so minor as to be practically ignorable all the time, and requiring really special evidence to show the super rare cases when it cannot be ignored.
(Those cases also are most often just solved by collecting new/ more training & testing data and re-training / fine-tuning / calibrating the model to some slight shift in the distribution, and require no formal modeling techniques related to tricky generalization distribution differences).
This has been true across commercial ML products or services at large tech companies, small startups, quant finance firms, defense research labs, and academic labs. Across projects ranging from face recognition, high-traffic search engine and related services like machine translation, trending models, search ranking, dimensionality reduction models in time series forecasting, reverse image search, audio similarity models, advertising models, and many more.
It just turns out that in industry, the problems that have meaningful utility to the business are almost always defined by intrinsic constraints that nearly guarantee there won’t be a meaningful difference in the distribution of future prediction-time examples compared with training & testing distribution.
ML needs data engineering like electrical devices need electrical infrastructure (i.e. generators, transmission lines, transformers, last mile lines).
It turns out companies have wildly different maturities and proficiencies with these precursors, in addition to simple company ages. Many from having under-invested in actual (not consulting BS) technology transformation and skilling for decades.
Consequently, ML is ridiculous to consider and doomed to failure for company A. While company B can toss a simple model at their well-architected data systems and get immediate ROI.
This is underappreciated, because VPs and consultants are not typically in the business of saying "Our systems are out of date and have poor hygiene, and we can't do this new thing because of that."
They should have used the Google excuse for the middle step.
Construct an algorithm that tells you what you want, based on the data that you're not supposed to be using, and then throw your hands up when anybody asks and say "Nobody really knows how ML models work."
Or before machine learning was an option: management consultants.
I mean in the sense that using ML for a problem often requires just trying a dozen different modeling techniques, then a bunch of a hyper-parameter searching, then a bunch of stochastic tuning…
reply