It appears you're referring to when I said "(assuming a highly controlled, completely reproducible experiment actually can be set up in social/economic sciences)". I'm not making this assumption, I'm questioning if a meaningful experiment can actually be formulated and performed.
If a meaningful experiment can not be performed (and I'm not saying that it can), then examining and observing how people born into rich families operate would give better, or at least no worse, information than an uncontrolled, non-reproducible experiment would. And it might actually be cheaper to study those with money already rather than give people money just to be able to study them.
It would be more accurate so say that they've resisted looking for one. And even that's a bit unfair, since doing proper experiments (with basic features like a control group, never mind replication or blinding) in sociology, economics, or political 'science' is somewhere between crushingly expensive and outright impossible.
They're not actually simple and we don't actually get to play god with populations of actual people interacting in the real world under different conditions. We can devise experiments that (imperfectly) try to compare the results of different legal and other environments on outcomes. And some compelling results in economics come about from creating especially interesting "experiments" in this manner.
We are almost to the point where an entity like Facebook or Amazon could do controlled economic experiments. Or they could gather so much data, that when filtering them based on the experimental criteria, whatever remains could show a significant result.
Aside from the technical possibility of doing economic experiments, there is also the ethics of doing them. You are not only involving humans, but doing so on a grand scale. To prove a point, you would not only need to possibly send an economy into a recession, but you would have to do so dozens of times to get significant results. The people you are experimenting upon might get a little murdery.
I'm not sure that partial experiments which test none of the putative downsides are of much benefit to a debate though.
I certainly would contest, for example, the suggestion that Neil Armstrong walking on a sidewalk would have represented a moonwalking experiment, because we didn't need to experiment to learn that astronauts are capable of perambulation on earth or learn anything about low gravity from it. Similarly, if you wish to test the hypothesis that a welfare state would be best reorganized on the to each irrespective of his need principle with taxes adjusted accordingly, you're not creating evidence for it by learning that unusually poor people are better off after receiving one off windfall of $7500.
The assumption that poor people are better off with more money is already baked into the status quo solution.
- pick participants at random (regardless of employment status)
- give money to all the participants
- make all participants pay higher taxes to finance
For this to be realistic, the participants would not have a choice whether or not to participate. So yeah, I can't for the life of me think why we "can't ever design such experiment right".
I think the gold standard is experimenting for physical systems. Then you can control for whatever you want. For medicine it would be double blind study (select participants, split them in 2 groups, then you know the groups are the same, give one group the medicine, give the other group placebo).
And then we get to social sciences - like economics - very hard to make an experiment. You can't just play with the interest rate to write a paper...
I do not see why experimentation is impossible if we are in a position where there literally is no alternative other than doing 'experiments' except refusing to address the results and change tactics except under extreme political pressure. Things like 'we believe if taxes are lowered on businesses, it will result in them hiring more workers', after it has been tried for 30 years and has resoundingly shown to be a collassal untruth, actively destroying the middle class and leading to tremendous instability even among those who do remain employed, at least if we could say it was an experiment and we were uncertain of the outcome, we could say 'well that didn't work let's try something different' instead of it being a political battle fueled by people just trying to trick others into thinking that reality is different from what it is.
In a situation that affects so many people, it may seem unwise to do experiments and try things where we can not say with certainty what effects they might have, though we admit they may be substantial. The ethics are very dim on that front. But the alternative has to be considered. We do not have 'generally accepted treatment' to turn to and rely on. We only have other experiments, albeit while not calling them experiments and not treating them as if their outcomes are in doubt, simply forging ahead with blinders on. I can't imagine an argument that would justify that as more ethical than earnestly doing experiments.
Hardly simple. You are talking about running huge social experiments based on guesses about what might work better. It would be interesting to see the results of such experiments, but extremely difficult to convince people to participate.
I could see this working in mathematics and many branches of computer science. How would this work in environments that require large outlays of money to do experiments? Do you just hand untested individuals large sums of money and leave them alone?
I think the point is to at least try to have a control for the experiment. Social experiments like these are going to be difficult to eliminate all other variables, but this is probably as close as we are likely to achieve.
I was once an experimental physicist and never psychologist. In physics there is a lot of trial and error before you can get the experiment to do anything at all. Anyone who reads your paper needs fewer trials.
So I was going to agree with you that in fields where experiments work differently, reproduction can cost as much as the original study. But on second thought, that's wrong: and indeed it is the whole point of the thread.
There is always trial-and-error. You simply don't know if your theory is correct. And there is always risk of a methodolical weakness that you only understand after seeing the data. The trouble is after you have already spent all your time and grant money on your study, you have every incentive to paper over these issues. Indeed even if you are scrupulously honest, you will have to shove study in the proverbial file drawer - because you don't have the resources to try another iteration.
So I think it's still true that a new experiment costs more than reproduction, and many of the problems with studies arise from trying to pretend that cost doesn't exist.
Yes. If we take a look at social sciences struggling with experimental method, we'll see, that they get better with time. When experiments do not work, people somehow figure out how to do experiments better, and it changes things. It gives a hope that experimental method could falsify itself, so when it stops working we will be able to notice it. So if we do not see it to fail, we could assume that it works.
I think that in the hard sciences, things are very cut and dry - or at least that is the goal of a study. For example, in physics, the goal of experiments is to prove with a very high certainty that something is true or false.
Humans on the other hand are unpredictable. So if you run an experiment that says X, it may or may not replicate later, depending on hidden variables and assumptions.
Consider the famous marshmallow test. The latest studies suggest that it is not willpower but actually affluence that is the bigger determinant factor. [1] So that means that in future studies, they probably need to consider this variable and design the experiment in a way that they can control for it.
What is interesting is that for all the differences and intervening variables, humans can be studied and exhibit very predictable patterns. A good example of that is the study of power. [2] The Prince was written in 1532 and its principle continue to be just as valid today!
It's tricky, because "experiment" doesn't necessarily mean you're following the scientific method and its levels of rigor.
Some people "experiment" with leaving produce at the side of the road and having a trust based money box for people to pay. If someone takes all of the produce without paying one day, the experimenter isn't necessarily going to analyze it scientifically and be happy at the seeming failure, but instead think "I can't trust people enough - experiment's over."
This is a great point, but at the same time how do we answer this question when all our experiments last 5 years or less? To answer this question you have to run an experiment for roughly 30 years, ensuring that people born into this system are taken through college (i.e. the dependent years, which is usually considered <25yrs of age) and that you have enough children being born during this period. We know that influences before the age of 8 have significant factors in establishing things like educational culture for children, and this is highly associated with socioeconomic status.
So if we're saying that an experiment has to run for at least 30 years: 1) why aren't we getting on that? 2) are short term studies enough to warrant bypassing the long term small scale study and precede to a large scale study?
It appears you're referring to when I said "(assuming a highly controlled, completely reproducible experiment actually can be set up in social/economic sciences)". I'm not making this assumption, I'm questioning if a meaningful experiment can actually be formulated and performed.
If a meaningful experiment can not be performed (and I'm not saying that it can), then examining and observing how people born into rich families operate would give better, or at least no worse, information than an uncontrolled, non-reproducible experiment would. And it might actually be cheaper to study those with money already rather than give people money just to be able to study them.
reply