When is absence of evidence = evidence of absence?

You often hear the old mantra "absence of evidence is not evidence of absence," but I think that this is an oversimplification. The truth is, sometimes it is, and sometimes it isn't. It depends on the experiment, and also how well you understand the implication of the results.

There are times when you can make an excellent case that something is absent because there is no evidence of it. If you are is small, well-lit room, you probably don't need to look under many things to convince any reasonable person that there is no tiger there. In general, you need two things:

1. You need a solid argument for what sort of evidence you would expect if what you are looking for is there, and how that would be different from the null hypothesis (not there). 
2. A set of data that tends to falsify the hypothesis, thus advancing the null hypothesis.

We're not talking about "proof" here. I would just as soon we not talk about proof much at all, unless the topic is logic or mathematics. Rather, simply raising or lowering the odds of the null hypothesis, i.e. that the claim in question is not true.

The Skeptical UFO Investigator, Part 1.

This post is a companion to the talk I gave at Balticon 47, and is for people who would rather not watch the video, or would like more more details not in the video.

I've been investigating UFOs for over a year now, and plan to continue. Rather than explain exactly why this is, I am going to try and show you, one bit at a time. Much of the interest is driven by the human element, but perhaps not all of it.

As I have explained before, I don't know whether the Earth has ever been visited by non human intelligences from elsewhere, and I think it's highly unlikely that anyone else does either. I am skeptical we would know it when such an encounter was happening, though right in front of us. Our perceptions of such events would be fragmentary and distorted, and we would likely interpret them as magic, or supernatural.

Encounters of that very sort have been recorded all through human history, and continue to this day. Puzzling, often frightening, and frequently life changing events take place every day. Many of these, it turns out, are due to known bugs in the human perceptual and conceptual machinery, and others are merely hoaxes. UFO investigation has a more than 60 year history, and the truth continues to elude. I want to know why this is.

Let's go and have a look, I say, and let's see for ourselves what is going on.  Woody Allen once famously said that 80 percent of life is just showing up.  Let's show up where the saucers are seen and look for whatever patterns are there.

Stories, Conjectures, Evidence and Proof

OK, we need to clear a few things up here.  People throw around words like "evidence" or "proof" a mite too loosely for my liking.  In an earlier post, I talked about evidence for the "hypothesis" (yes, scare quotes) that some UFO sightings are alien spacecraft, and why it's not so straightforward to say that.  And yet, almost everyday we hear claims of proof or evidence both for and against.  Let's talk a bit about what that means.

 

Short Version

 Evidence isn't just evidence - it is evidence for or against something.  Before that, it's just data, or worse, stories.  In order for data to serve as evidence we need to be able to fullfill three conditions:

1. We need a decent idea of what that something is and how we expect it to behave, i.e. a Hypothesis.
2. We need at least an approximate idea of how the data we have or propose to gather could be produced if the hypothesis was true (and, if it false).
3. We need to bend over backwards to honestly ask and answer the question: what if I'm wrong?  You can be wrong about the hypothesis, or you can be wrong about the evidence, and data can be just noisy or biased.

When you have all these things in place, and the odds that the data we have if your hypothesis is true are greater than the odds of the same data if the hypothesis is false, then you can claim that there is evidence for your hypothesis.  Not proof - proof means that your hypothesis is logically unavoidable, which is a much stronger (and almost certainly wrong) claim.

This leads us to problem we get with stories as evidence.  Also, with such conjectures as ghosts, alien spaceships, or cryptids  - we can't fulfill the first two conditions listed above, so are lost in claiming that some data we have is evidence for that.

Doubt is a Core Value

It is our responsibility as scientists, knowing the great progress and great value of a satisfactory philosophy of ignorance, the great progress that is the the fruit of freedom of thought, to proclaim the value of this freedom, to teach how doubt is not to be feared but welcomed and discussed, and to demand this freedom as our duty to all coming generations.

- - - Richard Feynman

Here is how things go wrong: beliefs hold the believer. The believer acts, thinks and perceives the world through his beliefs and in order to support them, and is cognitively and morally blinded and enslaved; unable to inquire outside a narrow, nonthreatening sphere, and willing to use any means whatever to propagate the One Truth.  Then belief exists for it's own sake, and it can escape all moral restraints, with terrible consequences that are only too well known.The problem is not so much the belief systems themselves, but with the banishment of the one great value: doubt.