Path: EDN Asia >> Design Centre >> Test & Measurement >> Scientific results evaluation: Irreproducible results
Test & Measurement Share print

Scientific results evaluation: Irreproducible results

29 Mar 2016  | Ransom Stephens

Share this page with your friends

Let's start with a simple example. In figure 2, the background noise barely fluctuates and the signal is quite pronounced. Count the number of signal events above the noise to get NSIGNAL—don't put any more effort into it than you would while reading "I Freaking Love Science" or the New York Times. Then. count the number of background events below the signal to get NBACKGROUND. Because NSIGNAL/NBACKGROUND = 6, It's a 6-sigma effect; the sort of signal that random processes could conspire to in less than one in every twenty-trillion repetitions of the experiment (c.f., Table 1 and Table 2 in Part 2 of this series), convincing but still in need of independent verification.

Figure 2: In this plot, the signal is obvious.

Now do the same thing for the six plots in Figure 1 and determine which have significant signals. Your calculations aren't likely to be the same as mine, but they're probably close enough.

Because I made the six plots with a simple simulation, I can peer under the rug and tell you that all six have the same signal and the same background. The only thing that differs is the role of random fluctuations. That is, I used the same parameters to create each plot except for the initial random number seed. I chose the six plots from 30 different runs and (I admit it!) I picked two where the signal looked drowned by background, two where it peaked over the noise, and two that looked like most of the other 24 runs.

A perfect measurement of this system would show a 3-σ effect, which means that the signal would disappear under the noise less than 0.3% of the time. Our estimates differ over the six plots, not because the rule-of-thumb is inadequate, but because random fluctuations shift things around in ways that, to our pattern-predicting brains, don't look random. In other words, our results might be irreproducible, not because of any systematic bias, but because this is how people interact with nature.

On the other hand, the "crisis" of irreproducible results (at least those not caused by fraud) could be averted if (a) the experimenters always quoted both systematic and statistical uncertainties, and (b) if journalists reported both uncertainties in a way that advised the reader where the results belong on the scale from inconclusive (evidence for) to conclusive (discovery of). But, that would de-sensationalise the results and reduce the click-bait so editors won't allow it, except here at EDNA?

Generally, when reading science journalism, it's useful to keep in mind the immortal words of Miles Dylan, from his book, Everything: "There's more to it than that."

About the author
Ransom Stephens is a technologist, science writer, novelist, and Raiders fan.

 First Page Previous Page 1 • 2

Want to more of this to be delivered to you for FREE?

Subscribe to EDN Asia alerts and receive the latest design ideas and product news in your inbox.

Got to make sure you're not a robot. Please enter the code displayed on the right.

Time to activate your subscription - it's easy!

We have sent an activate request to your registerd e-email. Simply click on the link to activate your subscription.

We're doing this to protect your privacy and ensure you successfully receive your e-mail alerts.

Add New Comment
Visitor (To avoid code verification, simply login or register with us. It is fast and free!)
*Verify code:
Tech Impact

Regional Roundup
Control this smart glass with the blink of an eye
K-Glass 2 detects users' eye movements to point the cursor to recognise computer icons or objects in the Internet, and uses winks for commands. The researchers call this interface the "i-Mouse."

GlobalFoundries extends grants to Singapore students
ARM, Tencent Games team up to improve mobile gaming

News | Products | Design Features | Regional Roundup | Tech Impact