Yeah, but giving them metrics won't fix that. Even forcing them to cite the metrics in their decisions won't force a culture change. And even if you could somehow convince/force them to earnestly intend to be data-driven, they'd probably still fail.
In ostensibly data-driven workplaces, the decision makers pick and choose which metrics to consider, influence how those metrics are measured, determine which exceptions and extenuating circumstances to consider, etc. The end result is often just as subjective as what you started with, except now the decision makers can conveniently pretend that the result comes from an objective process rather than their personal whims. It becomes a method for disclaiming responsibility.
Sure, no need to give me examples because I've seen plenty on my own.
Still, I don't think we should abandon the idea of basing our decisions on data just yet. Bad metrics are efficiently worsening products and work culture, but the solution is better metrics, not going back to HiPPO.
I work as a data scientist on an extremely data driven team at an extremely data driven company, and I couldn't agree more strongly. Everyone I work with would agree too. "Numbers aren't everything!" says the person whose job it is to deal with business problems via numbers.
But this advice is so hard to implement. It's like the whole null hypothesis p-value debacle going on in academic publishing right now. Sure we can agree it's no good, but there's no standard alternative.
Standards are powerful. They break lots of ties. All else equal, if you have a hunch and I have a measurement tied to objective reality, people will side with me. Nobody ever got fired for choosing Java. Nobody ever got fired for doing the thing that made The Metrics go up.
It's so reassuring to just say "We still disagree, so let's just test it." We're submitting our dispute to adjudication via Science by resolving via The Metrics! It's much harder to be objective without these kinds of numbers, and I like being objective! I want the scientific method to tell me I did the right thing and I'm a good person.
Because without that, when we're trying to figure out what to do, it's just us, you know? It feels like guessing, and that's hard.
I agree with you completely except on one part. Metrics is not the only solution to this. The other option, more difficult but more effective if you can pull it off, is for individual contributors to not make these mistakes in the first place.
Having executives fix the bad decisions by individual contributors is sort of treating the symptoms rather than the underlying cause. It adds extra work because now everyone has to produce numbers about their work in addition to work -- and some of the numbers might not be better than tea leaves, statistically speaking.
So how do you get ICs to make the correct decision? Training, ownership, free flow of information about strategy and market, involving everyone in setting the direction, etc. The opposite of the instinct of an executive going by the numbers.
----
To stave off an eventual misunderstanding: I'm not saying you shouldn't measure what you do, operationalise your definitions, define in advance when you reject your hypotheses etc. I'm just saying that producing these numbers for yourself and your peers is far more effective than doing it for someone removed from the day-to-day business because they don't have the same context and nuance.
I agree with everything you say. And bad numbers are awful. At some point of organizational size, executives can’t stay on top of everything and make good decisions without quantitative metrics.
That doesn’t mean making everyone full time metric gatherers. It doesn’t mean looking at the data uncritically. It doesn’t mean ignoring what you can’t measure. It does mean smartly measuring what you can.
This, to me, is a general problem with a purely quantified, metrics driven approach to management, not just limited to NPS. I'm not demonizing these approaches, just saying that you have to balance humanity with data.
I like to look at this kind of thing as the one of the dark sides/dark patterns of using data for decisionmaking.
Qualitative measures are important, as is maintaining as much humanity as possible, to have a balanced and healthy culture. Being solely metrics/data driven can lead to cold, heartless, damaging culture (might be efficient or make profit, but very dehumanizing).
This take is probably going to be controversial here, but I suspect that most metrics don't accomplish anything beyond giving control freak managers a sense of control or insight.
Most complex processes can't be reduced to a handful of simple variables - it's oversimplification at its worst. The best you can do is use metrics for a jumping-off point for where something /might/ be going wrong and thus start engaging with actual humans (or reading code/logs/some other source of feedback). Too often I've had to deal with management who go straight from metrics to decisions and end up making bad decisions (or wasting everyone's time shuffling paper to generate good looking metrics).
> People with management experience can decide these things based on intuition. It's practically impossible to quantify anything in this industry, but 'data-driven decisionmaking' is fetishized to the extreme.
The answer to "it's practically impossible to quantify anything in this industry" is not "therefore we should just give up and rely on managerial intuition with zero accountability or measurement", but rather "therefore we should improve our ability to quantify things and pair that quantification and measurement with human judgment and oversight".
My take on this is without metrics it's all subjective which is in many ways worse. Everyone, including your team managers have biases. Do they make decisions based on feelings rather than data?
Is data really necessary? Can't you just describe the behavior you think is lacking, why you think that behavior is important, and then ask them what they think?
If you can't come to consensus with your employee about what aspects of their job matter, then you have a bigger problem than poor performance.
I don't think having data or metrics is really critical here. Some people like to think in metrics and that's great if you like it, but I don't think that's universal, necessary, or sufficient for a relationship that facilitates good performance.
I think you might have misunderstood my point. I said our management culture leans on metrics so heavily as a way to mitigate the effects of racism (and sexism) in managerial decision making. Not that the metrics are racist.
It's harder for us to rely on systems that leave more room for personal judgement without oversight because we worry that decisions not attached to hard metrics are more likely to be subject to invidious biases. The charitable take is that the drive for metrics is motivated by a sincere attempt to mitigate these biases. The more cynical take I offered was saying that people use it as a way to cover their tracks against charges of discrimination.
Right, but you can't simultaneously say that pursuit of the metrics will lead the organization away from its holistic goals while also saying that managers will ignore what the metrics are telling them by interpreting them to confirm whatever they wanted to begin with. Something can't be both inert and poisonous.
Using metrics to gauge how well the enterprise is doing on the whole and to influence policy decisions and strategy is absolutely good and desirable. It's only when metrics become rules or quotas or measures to compare people or departments as a policy that things go bad.
Sadly I've been in many organizations that have a kind of "vibes for me, data for thee" approach to things. Meaning, "data drives decisions" is the official philosophy ... until it isn't.
It takes a certain amount of rigor and product maturity to make metric-based decisions work — and some types of applications are better suited to this than others.
Most of the metrics that I have been judged by over the course of my career (e.g. lines of code, story point velocity, etc) are absolute bullshit, and easily manipulated. And yet... I continue to build "dashboards" for VP's to stare at, and continue to be measured by deeply flawed metrics.
I believe that every rational thinking person would agree that tracking racial and ethnic and sexual headcount percentages doesn't necessarily mean what they purport to mean. But... what are you gonna do? We just need metrics, any metrics. Bad metrics are better than no metrics, or at least this seems to be the deeply entrenched mindset of business.
Bad metrics can wreck corporate cultures. Let’s attack the other straw man: metric-free orgs.
Symptoms:
- Resources (and credit and promotions) are given to the most articulate and outgoing rather than the most effective. (Marketing budget goes to the extrovert rather than the manager producing the best ROI)
- When dates slip, they slip a lot. The night before a release it’s announced that there will be a 3 month delay. (Bad for anyone having to explain “If it’s 3 months, why couldn’t you tell me that sooner?” to a customer)
- Unanticipated earnings and revenue misses. (Same issue as above, but bad to anyone talking to investors)
- Poor prioritization. Effort is spent on squeaky wheels versus high value problems. (Engineers spend time helping “that nice person in accounting” versus solving a customer problem that brings in hard money)
With all the things an exec needs to worry about, they can’t effectively operate without good data to drive decisions. Data needs to have human interpretation as well. Of course bad data can be worse than no data, but that’s another story.
I think the real message here is not that metrics are bad, but that they are misused. Imagine if every time you went to the doctor with a fever and they took your temperature, the doctor prescribed an ice bath to bring your temperature down. You wouldn't conclude that thermometers are evil, you'd switch doctors. Same goes for most of the metrics here.
Metrics are useful to navigate BY, not to navigate TO. If you have skilled and experienced managers, you can get a lot of value out of all of the metrics listed in the article.
In ostensibly data-driven workplaces, the decision makers pick and choose which metrics to consider, influence how those metrics are measured, determine which exceptions and extenuating circumstances to consider, etc. The end result is often just as subjective as what you started with, except now the decision makers can conveniently pretend that the result comes from an objective process rather than their personal whims. It becomes a method for disclaiming responsibility.
reply