How reliable is that research result?

Latest response in a series of letters to the Times of Malta, following two
excellent companion articles for M.E. Awareness Week. I suggest, for conext,
you view all 6 replies here
http://www.mefreeforall.org/2008-Apr-Jun.448.0.html#c3634

If you wish to join in, the e-mail address is daily@timesofmalta.com

Cheers
John
drjohngreensmith@mefreeforall.org


*The Times of Malta Letters*.

What Albert Cilia-Vincenti finds "strange" and "doubt-instilling"
(*Understanding
and treatment of ME sufferers, Letters, 6 June 2008*) about Rebecca
Sultana's response to Professor Bassant Puri's work with VegEPA for M.E.
sufferers (*Research into ME, Letters, 30 May 2008*), I call healthy
questioning of experimental results and, not strange, but a familiar method
of the research scientist.

Her curiosity is particularly pertinent in a field with too much ignorance,
misinformation, subjective opinion and, regrettably, some exploitation,
impeding progress to a better understanding of the physical cause of
M.E. (*Myalgic
Encephalomyelitis*) and, perhaps, a cure.

Remarkably, all three of us think VegEPA is worth a try and may have some
beneficial effects but whereas Albert, unreservedly recommends buying pots
of it and Prof. Puri's book, Rebecca is more cautious and I will remain
unconvinced until his work has been, independently, repeated again and
again.

In any research, the onus is on the researcher to prove their hypothesis by
rigorously testing it, in order to show that their results are valid
(actually measure what they say they are measuring) and reliable (will be
consistently reproduced if the experiment is repeated over and over).

For those not familiar with the language or methodology of research, think
of the researcher's job as that of the prosecutor in a legal case. The
defendant is innocent until proved guilty, no matter how things may seem, to
any individual onlooker, before the trial starts. Likewise, the hypothesis
remains untested and deserves a fair trial.

The defence is there to ensure that the prosecution does not make any
crucial mistakes, whether accidentally or, deliberately, to sway opinion in
one favoured direction. The defence certainly does not have to make any
concessions, nor do the more neutral dispassionate observers in the public
gallery have to take anything on faith. In the same way, peer researchers,
those with an interest in the outcome of the work and the general public are
not only entitled but should be encouraged to scrutinise every detail of the
work.

Just as it is the job of the prosecutor to prove their case "beyond
reasonable doubt", so the researcher must say that, if this experiment is
repeated 100 times, the probability that they will get the same result is 95
times (that is, 95% or p<0.05) or, better still, 99 times (p<0.01).

If the prosecuting lawyer has a very strong case, you can't shut them up;
they will pile up every piece of evidence they can find to make their case
the strongest possible. Similarly, with the researcher: The strength of the
results, in support of their hypothesis, is like that of a wall they have
built. The more robust they believe it to be, the more likely they are to
invite you - even challenge you - to take a sledgehammer to it. It is when
they discourage you from merely leaning on it that you should, quite
rightly, question its construction and how useful it is for the purpose.

The strength of confidence in the results of studies is important because it
is on the basis of these that we may follow the recommendation to take, or
decline, a particular treatment, which may have serious long-term
consequences.

Yours sincerely
drjohngreensmith@mefreeforall.org
Dr John H Greensmith
ME Free For All. org

* * *

One thing that we have found repeatedly over the years is that those researchers who want to prove that CFS is psychiatric in nature will load up their studies with patients who actually have depression, some even going so far as to say that certain symptoms that are not shared will disqualify you from being a research subject, in order to not have their "CFS" research contaminated by True CFS patients.

The original diagnostic criteria for CFS required depression to be ruled out before making the diagnosis, so it's obvious that the diagnostic criteria were being ignored when researchers started using patients with current depression in order to make pronouncements about CFS.

Cort Johnson is particularly good at ferretting out "CFS" researchers who ignore the diagnostic criteria in order to reach the results they want to reach.