> I took a look at the data. The data schema is disorganized to the point that a lot of janitorial work would be necessary to get it useable and perform any analysis or visualization.
>>> based on an analysis of all the publicly available data This aim is not as ambitious as it sounds, because there is not a great deal of data publicly available.
Amusing and revealing at the same time. Bet they did not take long to agree on that senrence
> No, all the choices have already been made. The data have already been destroyed
You don't know that. You'd need an investigation to conclude that. Using it as an excuse not to investigate seems like assuming the conclusion.
Even if some data have been destroyed, it doesn't follow that every last piece of data everywhere has. Who knows what might turn out to be significant? There may well be relevant data in many countries, too, since the research was international.
> The giant piles of dirty data are that way because for thirty years no one has considered them worth cleaning up. How will they create such astounding amounts of unexpected value?
It was left that way because we didn't have the tools to process it. Imagine the amount of unprocessed video data that we can now annotate pretty accurately. What's the value of that data now?
> , few people have the time or want to make the effort to comb through and analyze original sources.
I mean, unless it's your profession, you're not. At best, you're reading an article (with summarized data that you hope was aggregated correctly) in a journal. To the best of my knowledge, the raw datasets that those are based on are rarely shared.
> I have a pet theory that a large portion of the decay in our society can be traced back to the layers of abstraction between stakeholders and the things (and people) they have power over.
My pet theory is a specific instance of yours: MBAs wielding spreadsheets (and, more generally, analysts wielding databases) is the abstraction layer.
Buried in over a million graves.
reply