I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form.
My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive any key metric you want in the direction you want to. Short-term, it seems to work great.
Long-term, it fails. Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. You can
(and should) invest in metrics that are more indicative of the long-term. And you should use data to help guide and improve your intuition.
But data is not a substitute for good judgment, or for a deep understanding of your users and their problems, or of "where the puck is going". It's just a tool. It's a very powerful tool, but if it's your main or only tool, you will lose.
I liked this article and agree with the point - when it comes to innovation, new products, and new markets, instinct and experience could probably help more than data.
Using data for decision making is more relevant for more mature processes where the data exists to learn from. Being data driven is especially important for data intensive sectors where the complexity of issues cannot be fathomed just by being brilliant, it's here you need to be able to analyse the intricacies using the data to find possible answers.
Being data-driven is detrimental when it replaces common-sense (talking to users, collecting feedback, improving based on feedback).
Before the internet (and being able to track every single action), successful companies were built. It can be done. Using data to drive decisions has some value, but it's not the end-all solution, it's merely a piece of the puzzle.
Does anyone actually buy data driven? Seems like a great methodology to adopt if you need to absolve yourself of responsibility for your failures. Most questions people try to apply “data driven” to are so ridiculously complex I’d need to see some incredible methodology and ground breaking understanding of human behavior to put any faith in them. There’s just too many unknowns and confounds.
Data driven is great when you’re monitoring computer performance, but that’s a domain humans have nearly built from the ground up. And even then it can still be very hard to utilize that data. Trying to apply the same to systems we barely understand seems fraught with error.
Great article, but I think it somewhat misunderstands the impetus for the concept. "Data has its place" sounds obvious precisely because "data-driven" has been such a successful concept. The alternative perspective, which used to be very common in our industry and still pops up from time to time, is that metrics are something you write for debugging and business decisions are made by gut feeling or abstract philosophical analysis. (Most software companies had to make decisions this way in the pre-cloud era, because it wasn't usually feasible to collect usage metrics.)
Totally agree. Data is not a substitute for good judgment, or for deeply understanding your users and their pain points or your industry. You can use data to guide decisions but as soon as you try to rely on it completely, the part of your brain that does critical reasoning just turns off.
The problem with "data driven" approaches is that they often miss the context.
Which can make their results absolute garbage.
Like data driven can tell you a button is "not used often" it can't tell you that a button is "essential to be fast available in some safety critical situations". (Or for other examples, that a feature is not unused because people don't want it but because it's hidden or bad designed.)
But somehow I still meat people which believe that poorly data driven approaches will yield the best results, which as far as I can tell is complete unrealistic. (Which doesn't mean in any way you shouldn't also use data for decisions, just be aware that data just shows a part of an picture and can often be very misleading.)
I agree that data-driven decisions are less common than they should be. However, there is such a thing as over-dependence on data, and this piece might be toying with the line. You don't have enough resources to test every little thing - sometimes you have to use your intuition and move on.
There’s long vs short term, but I’d also add ease of calculating a decision’s value. Ironically I think a lot of the business and tech world is hamstrung by “data-driven” decision making which assumes
1. You have all the necessary data and
2. You are interpreting it correctly and completely
This is almost never true, so instead “data driven” is mostly “data covering-your-ass.” Maybe the future will yield leaders more capable of wielding data less like a cudgel, but I’m not optimistic.
Data-driven choices are largely for optimizing choices, not discovering them. Being data-driven can be just as dangerous intentionally ignoring data. Companies “win” by focusing on what matters, with the right people, at the right time.
There is one, and only one reason to be 'data driven'. Or to test one's hypotheses, for that matter. And that is to make sure you aren't fooling yourself; that you haven't fallen prey to the myriad cognitive biases; to prevent your preconceived notions from clouding your judgement of reality, aka what is actually going on.
While data always provides more information, the less strong your prior beliefs, the less informative your experiment will be -- If you believe something and it turns out to be majorly false, you get a nice shift in expectations. If you believe in something and it turns out to be very true, you gain lots of information in terms of quantifying the effect you are looking into.
If you are Google, looking to eck out every last 1/1000th of a penny on ads, yeah, maybe a/b testing the shade of blue of a button can be justified.
The more other companies are "Data Driven" [like the somewhat unfortunate examples the author chose], as opposed to "Hypothesis Driven", the more there is room for somebody else to fry bigger fish.
In other words, it's not the "data's" fault, it is ours.
What irks me about data-driven decision making are two things:
1. Correctly reasoning from data is a very difficult task, needing advanced skills, requiring deep understanding of statistics. Most companies have nobody with such skill set on board.
2. It provides a fake air of legitimacy to otherwise arbitrary decisions. Whether by mistake (confirming your own biases with data) or on purpose, you can use data to justify whatever decision you already want to make, few people know enough statistics to call you on your bullshit.
This is take I don't agree with on a real problem, which is the usefulness of data science in modern business, and for which processes in the business.
I try to explain.
Recommender systems add tremendous value, millions when not billions in revenue, to any retail business, streaming business. Is it by chance that all the online retailers had to set up a recommender system?
A/B tests are not needed? If A/B testing and actions taken looking the results would not be useful, don't you think that business would know it?
Forecasts in business are nearly irrelevant, or rubbish, you say.
Like in the sense that in complex systems is irrelevant to know what's going to happen next, that is better just to play it along at best?
Having worked in companies in media, retail, healthcare, and telco for which forecasting is crucial for planning the use of resources, negotiation of contracts etc., this comment is off-base.
Going from abstract to practical, I have worked recently with one telco giant on one of the services they offer.
Their forecast of subscribers done by drawing a straight line that follows the latest data is making them leave tens of millions on the table each year.
I proposed to make some A/B testing on their "coupon" strategy and they were not interested, preferring to send a "coupon" to everybody and their grandfather because they did not want to study for a week how to set up an experiment.
It is mostly laziness. Clearly to know if A/B testing is working or not (say increase engagement, revenue etc.), one has to try it. But they don't, because they are mentally lazy.
One thing I agree with is that most legacy business are not ready to adopt a data-driven modern way of doing business, due to stiff processes, old-timers that would like to go on with their business like they were doing 30 years ago, and general fear of changing.
The fact that the legacy businesses are still "working" may give the impression that all these data-driven "baloney" is subtracting rather than adding to the business. Why do we need forecasts, they say, I cannot believe that these people want to use data, isn't our expertise enough? Don't you see we are the still among the 5 top companies that do what we do?
But their position in the market is not due to sound business practices, but due to their dominant position in the market. But then the dominant position goes away and the old-timers finally recognize that A/B testing, forecasting etc. combined with domain-knowledge are a plus for the business.
Maybe you don't need 1,000 DS, maybe you need 20 (practitioners, not "researchers"), that depends on the business. Clearly the mom and pop business does not need any data science.
I'd like to see a solid argument for a new style of management that uses statistics in new ways. For the most part, my experience with the phrase "data driven" has been a negative one. Most of the time, when I have a client that claims to be "data driven", they are using a style of argument to avoid direct, honest conversations. When I wrote "When companies make a fetish of being data driven they reward a passive aggressive style" I did my best to explain what I've seen:
"As far as I know, there has never been a company that said “We want the worst informed people to make the decisions” so in a sense all companies have always valued data. But they didn’t make a fetish out of it. They simply expected people to be well informed, and to make intelligent arguments, based on what they know. That would have been true at General Motors in 1950. That much has probably been true at most companies for centuries. When management says that the company is going to be “data driven” they are implicitly asking for a particular type of interaction to happen in meetings, an elaborate dance where people hide their emotions and quote statistics."
Like any good idea that sees wide adoption, being "data-driven" has jumped the shark to cargo-cult methodology in most places.
The truth is that data is only powerful if applied judiciously with solid knowledge of statistical fundamentals and careful thinking about causality (which will may not be practically falsifiable!). "Data" can also be misread and applied in big powerpoints with a reckless disregard for reality.
Enlightenment is understanding enough about data to use it correctly, but also acknowledging its limitations and that to create a successful business you also need vision and insight about potentials and trends for which accurate data does not and can not exist (except as a post-hoc trailing indicator).
Decision driven from bad data is IMO worse than data-less decision.
Data grants authority to a decision that gut feel driven one doesn't. It is hard to argue against evidence as it should be, but that assume a certain level of quality in the evidence.
Second, if practice doesn't match the expected outcome, the first thing you will look at is what the team is doing wrong, not review the decision as not working.
That said, parent is far from unique in his skepticism, so I think the problem is more often reversed in the industry. Having some data, even flawed can help your company decide to try something new.
The article wasn't arguing against being data driven, it was warning how you can fool yourself into thinking you are being data driven when in fact you are simply following intuition.
My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive any key metric you want in the direction you want to. Short-term, it seems to work great.
Long-term, it fails. Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. You can (and should) invest in metrics that are more indicative of the long-term. And you should use data to help guide and improve your intuition.
But data is not a substitute for good judgment, or for a deep understanding of your users and their problems, or of "where the puck is going". It's just a tool. It's a very powerful tool, but if it's your main or only tool, you will lose.
reply