OK, we need to clear a few things up here. People throw around words like "evidence" or "proof" a mite too loosely for my liking. In an earlier post, I talked about evidence for the "hypothesis" (yes, scare quotes) that some UFO sightings are alien spacecraft, and why it's not so straightforward to say that. And yet, almost everyday we hear claims of proof or evidence both for and against. Let's talk a bit about what that means.

### Short Version

Evidence isn't just evidence - it is evidence

*for*or*against*something. Before that, it's just data, or worse, stories. In order for data to serve as evidence we need to be able to fullfill three conditions:- We need a decent idea of what that something is and how we expect it to behave, i.e. a Hypothesis.
- We need at least an approximate idea of how the data we have or propose to gather could be produced if the hypothesis was true (and, if it false).
- We need to bend over backwards to honestly ask and answer the question: what if I'm wrong? You can be wrong about the hypothesis, or you can be wrong about the evidence, and data can be just noisy or biased.

*odds that the data we have if your hypothesis is true are greater than the odds of the same data if the hypothesis is false*, then you can claim that there is evidence for your hypothesis. Not proof - proof means that your hypothesis is logically unavoidable, which is a much stronger (and almost certainly wrong) claim.

This leads us to problem we get with stories as evidence. Also, with such conjectures as ghosts, alien spaceships, or cryptids - we can't fulfill the first two conditions listed above, so are lost in claiming that some data we have is evidence for that.

### Less Short Version

None of the ideas here are new, but for some people, this may serve as a useful new mental lens.Baye's Theorem |

#### The Wet Driveway Mystery

Let's say you look out your window in the morning and you see that your driveway is wet, which surprises you, because you didn't think it was very likely to rain. However, you have just updated your assessment of the probability that it rained last night (the hypothesis), because you see the driveway is wet. Now, the probability that the driveway would be wet given that it rained (probability of evidence given hypothesis) is nearly 100%, but it could also possibly get wet some other way, so we increase our estimate of the probability that it rained last night, but not to 100%.

#### Thinking in terms of Odds

Bayes theorem in terms of Odds |

*odds*of a hypothesis, I think you know what it means; for example, the odds that a horse will win or lose a race. If we say that the odds are 10:1

*against*a horse winning, we mean that probability that the horse will not win is 10 times the probability that it will. Now Bayes' theorem tells us that the odds that a hypothesis is true can be updated by knowing the ratio of the probability of the evidence given the truth of the hypothesis (what if I'm right?) to the probability of the evidence given the falsehood of the hypothesis (what if I'm wrong?).

Back to the driveway example. Let's say it's the summer dry season, and you know that your next door neighbor likes to run his sprinklers at the crack of dawn, and usually places them so that they sprinkle your driveway. Therefore, the water on your driveway would almost as likely be from sprinklers as from rain. That is, the likelihood ratio for rain is only a little more than 1. Therefore, you would only slightly increase your estimate of the odds for your hypothesis that it rained last night. If you looked out your window again and saw that the street was dry, that makes the likelihood ratio much less than 1, and now your odds for the rain hypothesis drop drastically. You wouldn't want to bet on that horse. The wet driveway was evidence for rain (before we remembered the neighbor's sprinkler), and the dry street was evidence against it.

#### Why Simple Stories Fail

So now the problem with stories should be evident, and why stories need independent corroboration and hard evidence to back them up. The likelihood ratio for a story, however fascinating or well told, is not much greater than 1, because stories can be made up. The probability we would have a story if someone just made it up is 1, and of course it is entirely plausible that people perpetrate hoaxes. It is therefore very tough for the "the story is true" hypothesis to win out in the likelihood ratio.#### Roughing it Far From the Comforts of Paradigms

There is also a problem when you simply don't know enough to form a hypothesis, or even make a defensible estimate of the likelihood ratio. This is when you are in a state called "pre-paradigmatic", and there's not much hope of getting to that likelihood ratio with modest adjustment on what you already know. Is it impossible to learn anything then from your data?#### Be Wrong

No, but you may have to narrow your ambitions, or take some risk with a hypothesis that is very specific and very probably wrong. This latter is exactly what physics and astronomy are going through now with dark matter. They sense they are probably wrong about what it is, but can make a good case that there is something there, and they expect to learn as much from being wrong as being right. It's not about the answers so much, as the questions.The Dream of an Open Channel by PaulCarr is licensed under a Creative Commons Attribution- ShareAlike 3.0 Unported License.

Dear Paul

ReplyDeleteI liked this and re-posted it at my own blog

http://gmopundit.blogspot.com.au/2012/11/dream-of-open-channel-stories.html

(Found you via google Plus)

Thanks for the repost.

Delete