Stories, Conjectures, Evidence and Proof

OK, we need to clear a few things up here.  People throw around words like "evidence" or "proof" a mite too loosely for my liking.  In an earlier post, I talked about evidence for the "hypothesis" (yes, scare quotes) that some UFO sightings are alien spacecraft, and why it's not so straightforward to say that.  And yet, almost everyday we hear claims of proof or evidence both for and against.  Let's talk a bit about what that means.

 

Short Version

 Evidence isn't just evidence - it is evidence for or against something.  Before that, it's just data, or worse, stories.  In order for data to serve as evidence we need to be able to fullfill three conditions:

1. We need a decent idea of what that something is and how we expect it to behave, i.e. a Hypothesis.
2. We need at least an approximate idea of how the data we have or propose to gather could be produced if the hypothesis was true (and, if it false).
3. We need to bend over backwards to honestly ask and answer the question: what if I'm wrong?  You can be wrong about the hypothesis, or you can be wrong about the evidence, and data can be just noisy or biased.

When you have all these things in place, and the odds that the data we have if your hypothesis is true are greater than the odds of the same data if the hypothesis is false, then you can claim that there is evidence for your hypothesis.  Not proof - proof means that your hypothesis is logically unavoidable, which is a much stronger (and almost certainly wrong) claim.

This leads us to problem we get with stories as evidence.  Also, with such conjectures as ghosts, alien spaceships, or cryptids  - we can't fulfill the first two conditions listed above, so are lost in claiming that some data we have is evidence for that.

Less Short Version

None of the ideas here are new, but for some people, this may serve as a useful new mental lens.

Baye's Theorem

We are really talking about Bayes' theorem, which has been around for quite a while now, and is by itself an easily proven mathematical fact.  The math itself is only important here in one aspect: Bayes' theorem gives us a simple way to update the probability that a hypothesis is true, given that you know how probable the evidence is given the hypothesis, and you have some idea of how probable the hypothesis is to begin with.  You might want to run through that in your head one more time if it's unfamiliar: if we know how likely we are to get the evidence we have given that the hypothesis is right, we can update our estimate of how likely the hypothesis is  - using a method that does not introduce logical contradictions.

 

The Wet Driveway Mystery

I hope that's not too confusing.  I am going to introduce an example to make this more concrete.
Let's say you look out your window in the morning and you see that your driveway is wet, which surprises you, because you didn't think it was very likely to rain.  However, you have just updated your assessment of the probability that it rained last night (the hypothesis), because you see the driveway is wet.  Now, the probability that the driveway would be wet given that it rained (probability of evidence given hypothesis) is nearly 100%, but it could also possibly get wet some other way, so we increase our estimate of the probability that it rained last night, but not to 100%.

 

Thinking in terms of Odds

Bayes theorem in terms of Odds

Probably a better way to understand it is to take Bayes' theorem and do a little high school math on it. When we talk about the odds of a hypothesis, I think you know what it means; for example, the odds that a horse will win or lose a race. If we say that the odds are 10:1 against a horse winning, we mean that probability that the horse will not win is 10 times the probability that it will.  Now Bayes' theorem tells us that the odds that a hypothesis is true can be updated by knowing the ratio of the probability of the evidence given the truth of the hypothesis (what if I'm right?) to the probability of the evidence given the falsehood of the hypothesis (what if I'm wrong?).

Back to the driveway example.  Let's say it's the summer dry season, and you know that your next door neighbor likes to run his sprinklers at the crack of dawn, and usually places them so that they sprinkle your driveway.  Therefore, the water on your driveway would almost as likely be from sprinklers as from rain.  That is, the likelihood ratio for rain is only a little more than 1.  Therefore, you would only slightly increase your estimate of the odds for your hypothesis that it rained last night.  If you looked out your window again and saw that the street was dry, that makes the likelihood ratio much less than 1, and now your odds for the rain hypothesis drop drastically.  You wouldn't want to bet on that horse.  The wet driveway was evidence for rain (before we remembered the neighbor's sprinkler), and the dry street was evidence against it.

 

Why Simple Stories Fail

So now the problem with stories should be evident, and why stories need independent corroboration and hard evidence to back them up. The likelihood ratio for a story, however fascinating or well told, is not much greater than 1, because stories can be made up.  The probability we would have a story if someone just made it up is 1, and of course it is entirely plausible that people perpetrate hoaxes.  It is therefore very tough for the "the story is true" hypothesis to win out in the likelihood ratio.

 

Roughing it Far From the Comforts of Paradigms

There is also a problem when you simply don't know enough to form a hypothesis, or even make a defensible estimate of the likelihood ratio.  This is when you are in a state called "pre-paradigmatic",  and there's not much hope of getting to that likelihood ratio with modest adjustment on what you already know.  Is it impossible to learn anything then from your data?

 

Be Wrong

No, but you may have to narrow your ambitions, or take some risk with a hypothesis that is very specific and very probably wrong. This latter is exactly what physics and astronomy are going through now with dark matter.  They sense they are probably wrong about what it is, but can make a good case that there is something there, and they expect to learn as much from being wrong as being right.  It's not about the answers so much, as the questions.



The Dream of an Open Channel by PaulCarr is licensed under a Creative Commons Attribution- ShareAlike 3.0 Unported License.