Showing posts with label Bayes. Show all posts
Showing posts with label Bayes. Show all posts

Tuesday, December 31, 2013

When is absence of evidence = evidence of absence?

You often hear the old mantra "absence of evidence is not evidence of absence," but I think that this is an oversimplification. The truth is, sometimes it is, and sometimes it isn't. It depends on the experiment, and also how well you understand the implication of the results.

There are times when you can make an excellent case that something is absent because there is no evidence of it. If you are is small, well-lit room, you probably don't need to look under many things to convince any reasonable person that there is no tiger there. In general, you need two things:
  1. You need a solid argument for what sort of evidence you would expect if what you are looking for is there, and how that would be different from the null hypothesis (not there). 
  2. A set of data that tends to falsify the hypothesis, thus advancing the null hypothesis.
We're not talking about "proof" here. I would just as soon we not talk about proof much at all, unless the topic is logic or mathematics. Rather, simply raising or lowering the odds of the null hypothesis, i.e. that the claim in question is not true.

Tuesday, November 13, 2012

Stories, Conjectures, Evidence and Proof


OK, we need to clear a few things up here.  People throw around words like "evidence" or "proof" a mite too loosely for my liking.  In an earlier post, I talked about evidence for the "hypothesis" (yes, scare quotes) that some UFO sightings are alien spacecraft, and why it's not so straightforward to say that.  And yet, almost everyday we hear claims of proof or evidence both for and against.  Let's talk a bit about what that means.

 

Short Version

 Evidence isn't just evidence - it is evidence for or against something.  Before that, it's just data, or worse, stories.  In order for data to serve as evidence we need to be able to fullfill three conditions:
  1. We need a decent idea of what that something is and how we expect it to behave, i.e. a Hypothesis.
  2. We need at least an approximate idea of how the data we have or propose to gather could be produced if the hypothesis was true (and, if it false).
  3. We need to bend over backwards to honestly ask and answer the question: what if I'm wrong?  You can be wrong about the hypothesis, or you can be wrong about the evidence, and data can be just noisy or biased.
When you have all these things in place, and the odds that the data we have if your hypothesis is true are greater than the odds of the same data if the hypothesis is false, then you can claim that there is evidence for your hypothesis.  Not proof - proof means that your hypothesis is logically unavoidable, which is a much stronger (and almost certainly wrong) claim.

This leads us to problem we get with stories as evidence.  Also, with such conjectures as ghosts, alien spaceships, or cryptids  - we can't fulfill the first two conditions listed above, so are lost in claiming that some data we have is evidence for that.