Path: EDN Asia >> Design Centre >> Test & Measurement >> Scientific results evaluation: Irreproducible results
Test & Measurement Share print

Scientific results evaluation: Irreproducible results

29 Mar 2016  | Ransom Stephens

Share this page with your friends

In the last few years, many scientific results that society considers crucial have been difficult if not impossible to reproduce. The phenomenon has been called everything from a widespread problem to a crisis.

In Part 1, we discussed the important but difficult process of estimating unknown measurement biases called systematic uncertainties and how to deal with them when interpreting results. Systematic uncertainties can be estimated by comparing results of a given measurement performed with different techniques. In Part 2, we developed rules of thumb for quick and dirty estimation of statistical uncertainties and statistical significance; statistical uncertainty is roughly the square root of the number of observations, N, and statistical significance is roughly the ratio of a signal to the square root of its background noise, NSIGNAL/NBACKGROUND.

In Part 3, the last part of this series (as far as you and I know, anyway). I'll show you how easy it is for the demons of random processes to wipe out otherwise convincing results. As in Part 1 and Part 2, we'll assume researchers are sincere and leave fraud for another time.

Before we get in too deep, let me disclaim. While we should maintain our sceptical scrutiny, many irreproducible results come from attempts by neuroscientists, sociologists, and psychologists to reverse engineer the brain by analysing the behaviour of fewer than 1000 people, usually fewer than fifty. They're doing something far more fraught with uncertainty than anything in the physical sciences or engineering. Plus, the instrumentation available just isn't up to the task. For example, neuroscience experiments rely on fMRI (functional magnetic resonance imaging) that has spatial resolution of a few millimeters and temporal resolution of seconds. The equipment attemps to measure the behaviour of neurons, whose axons have diameters of a thousandth of a millimeter as they exchange signals in dozens of milliseconds—like searching for a needle in a haystack with a backhoe.
Find the signal!
The six graphics in figure 1 show the number of times that something happens, the vertical axis, at some time or place or energy, the horizontal axis. In which of these plots has something special happened at a specific time, place, or energy?

Can you find the signal without being tricked by the noise? Or, equivalently, can you determine whether the signal you see is real or just a fluctuation of the noise? In which plots do you see nothing but noise, which show evidence for a signal, and which have enough evidence for you to say that it's conclusive?

Figure 1: Where is the signal in each of these plots and would you consider that signal significant evidence for the existence of something other than noise?

1 • 2 Next Page Last Page

Want to more of this to be delivered to you for FREE?

Subscribe to EDN Asia alerts and receive the latest design ideas and product news in your inbox.

Got to make sure you're not a robot. Please enter the code displayed on the right.

Time to activate your subscription - it's easy!

We have sent an activate request to your registerd e-email. Simply click on the link to activate your subscription.

We're doing this to protect your privacy and ensure you successfully receive your e-mail alerts.

Add New Comment
Visitor (To avoid code verification, simply login or register with us. It is fast and free!)
*Verify code:
Tech Impact

Regional Roundup
Control this smart glass with the blink of an eye
K-Glass 2 detects users' eye movements to point the cursor to recognise computer icons or objects in the Internet, and uses winks for commands. The researchers call this interface the "i-Mouse."

GlobalFoundries extends grants to Singapore students
ARM, Tencent Games team up to improve mobile gaming

News | Products | Design Features | Regional Roundup | Tech Impact