A few tidbits, before I forget...
Beware of drawing conclusions based on insufficient evidence.
-The Simpsons, Much Apu About Nothing
Homer: Not a bear in sight. The Bear Patrol must be working like a charm.
Lisa: That's specious reasoning, Dad.
Homer: Thank you, dear.
Lisa: By your logic I could claim that this rock keeps tigers away.
Homer: Oh, how does it work?
Lisa: It doesn't work.
Lisa: It's just a stupid rock.
Lisa: But I don't see any tigers around, do you?
Homer: Lisa, I want to buy your rock.
Another thing to beware of is cargo cult science.
I think the educational and psychological studies I mentioned are examples of what I would like to call cargo cult science. In the South Seas there is a cargo cult of people. During the war they saw airplanes with lots of good materials, and they want the same thing to happen now. So they've arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head to headphones and bars of bamboo sticking out like antennas -- he's the controller -- and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land.
Philosophically, it's very difficult to know "why" something happens, as the word implies intent. Forget about that, though - when I say "why does B follow A", I mean "what are the links in the chain of causation from A to B". The more fine-grained our understanding is of the steps between A and B, the more powerful our knowledge is.
Think of a child with a lightswitch. They may grasp that when they flip the switch one way, the light goes on, and when they flip it the other way, the light goes off. Their black-box understanding of the system shows them this much - it's testable, it's repeatable, and it's quantifiable. In other words, it's science.
Then one day the bulb burns out. The child's understanding of the system is inadequate to explain this. Frustration results.
We could add another rule to the child's list of observations - "flip switch up, light turns on. flip switch down, light turns off. if the light stays off, replace the bulb and it will turn on again". But this understanding does not help the child to understand the danger of replacing the bulb when the socket is live, or the possibility of a damaged switch or a power failure. We can patch the model with more prescriptive statements, or we can teach the child about electricity.
-1 Corinthians 13:9-12 NIV
For we know in part and we prophesy in part, but when perfection comes, the imperfect disappears. When I was a child, I talked like a child, I thought like a child, I reasoned like a child. When I became a man, I put childish ways behind me. Now we see but a poor reflection as in a mirror; then we shall see face to face. Now I know in part; then I shall know fully, even as I am fully known.
This is the goal.
A fine-grained understanding is powerful in other ways, too. Not only is it more versatile, but it gives us the power to monitor the validity of our approach and assumptions in different ways, and to optimize our techniques based on intermediate results rather than long-term outcomes.
Consider a diet that claims to cause improved body composition. The diet provides a justification that involves hormone levels and sensitivity. This means that we can test the hypothesis that the diet presents directly - measure testosterone and insulin levels, insulin sensitivity, etc. - because we have a theory-based understanding of how testosterone and insulin levels correspond to our goals (or at least, we think we do). We don't have to wait eight weeks and measure our waistline and take the author's reasoning for granted - we can suck it up, get the bloodwork done and see for ourselves. And then we can make adjustments, and get more immediate feedback than a half-percent of body fat six weeks down the road.
Measurability is king.