Search This Blog

Tuesday 26 February 2013

Be a Pro not a Dufus

The wonderful Irish Up has written the article below and has let me share it.  It is useful for ALL areas of scientific research.

Sometimes I feel like a lot of us feel overwhelmed, bewildered, or just plain turned off by the debates around EDs, Evidence Based Medicine, different therapies & treatments, and all the research and data we throw around this joint. In a good way! Even if sometimes our passions make the discussions contentious, *that* the discussions are happening is very valuable, IMHO. But I think they could be even *more* valuable.

What I'd like to do on this thread is give readers a practical tutorial with the following goals:
  • Explain the basics of medical research;
  • Explain the relationship between research and the practice of medicine;
  • Give readers the tools to critique Mainstream Media (MSM) reporting of primary research;
  • Give readers the tools to evaluate a primary research paper for their own understanding.
On FEAST-ED, and here on the Forum, there is a lot of focus on Evidence Based Medicine and "The Research". Some might have a vision of a world where all ED research was well designed with clear outcomes that are plainly explained, and these would form a pool of data - evidence - that would lead to uniformly effective treatment. And that would be swell, but that is not the world we live in.

The truth is that no research is perfect, some research is poorly designed or executed, and no treatment for any disease or condition is uniformly effective. We think of Medicine as a Science. It is NOT, it is an Art that is *informed* by science. And actually, it's a GOOD thing that Medicine does not rely 100% on science - it would be bad if a discipline as complicated and encompassing as Medicine had to wait for and rely on only one source of information. That said, the science behind Medicine is REALLY IMPORTANT!


Step 1: Understanding Medical Research -Different Kinds of Research Give You Different Kinds of Information.

The first thing is to understand that the "knowledge base" of medicine comes from many sources. Each of these has strengths and weaknesses. The FIRST thing to do when you are looking at a piece of research is to identify which category it falls under. Then you have an idea of what kind of information the study design is good at getting, and what the limitations are. This is by no means exhaustive, but it's a good starting point :

  1. Primary bench science: this would be the kind of experiments NOT run on people at all, but more like the stuff most of us did in biology or chemistry in school. This would include experiments on animals.
    • Pros: Valuable for understanding basic mechanisms, lots of control over conditions, you can easily repeat what was done.
    • Cons: hard to know how well what happens in a lab will match what happens in a real live human being, lab conditions do not replicate real world conditions very well.
  2. Case reports details a single patient or small series of patients. These describe what the patient(s) showed up complaining of, their medical history, physical exam, symptoms, the treatment(s) administered and the outcome of the patient. Prior to formally applying the Scientific Method to medical research, Case reports comprised the bulk of medical knowledge. They are still extremely valuable for detailing rare diseases and conditions,novel treatments, unusual outcomes, or other anomolous or singular patient care events.
    • Pros: Pretty much the only way to study rare or singular events, good for descriptive purposes.
    • Cons: very small numbers, not repeatable, hard to know how generalizable (how applicable to another person or situation) the results or findings are.
  3. Observational studies which involve no "experimental" manipulation of patients in any way, but rely on identifying a set of patients with some shared quality - a diagnosis, an exposure, a treatment, an outcome - and either looking back in time to see what happened to them before the diagnosis/treatment/outcome of interested (in a retrospective study design) or forward to see what happens AFTER your event of interest (a prospective design). The important part here is that you aren't changing anything that happened or will happen to the people you are studying.
    • Pros: Inexpensive to do, easier to get large numbers, the study is already looking at "real life" situations, low risks to study participants.
    • Cons: Can't control all of the conditions, can only show relationships but CANNOT show which thing causes what, can be hard to replicate. 
  4. Randomized, controlled, clinical trials which involve carefully selecting people to get a treatment of your specific design. RCTs involve active participation from the subjects because you are deliberately manipulating what happens to them. Also very important: The treatments administered in RCTs have the PRIMARY GOAL of benefiting OTHER people! Now, the risks to the patient have to be acceptable - you can't do something worse to a person than what would happen if they *weren't* in your study. But since you're conducting an experiment, you don't KNOW if you're doing anything better, so really, the people who get the MOST benefit from an RCT are the people who get treatments that come AFTER the results of the study are known.
    • Pros: You can control conditions and participants so you don't have contamination from unknown sources. You can "prove" causality. Easy to duplicate
    • Cons: Expensive, study participation limits can limit generalizability.
RCTs are called the "Gold Standard of Proof". What that means is that RCTs are the only experimental design that gives you live people getting real treatments or exposures in such a way that you can *show* that what you did CAUSED the findings you observed.

BUT! Not every problem can be investigated with an RCT. And you DON"T need an RCT if the preponderance of evidence reaches a certain tipping point. Real life example:
 - Smoking has never been "proven" to cause lung cancer in humans. This is because it would be unethical to design a study where you make one group smoke. The preponderance of the evidence from primary bench science, animal studies, and observational human studies is that smoking *does* cause lung cancer. This did not stop scientists in the employ of tobacco companies from loudly proclaiming that the connection wasn't "proven" for decades. They were technically right, but ethically reprehensible.

The other issue is that RCTs participants are often very unlike the larger group of people who would get the treatment, and that the treatment is administered under ideal, and not real world conditions. Because RCTs can be very small in terms of numbers of participants, it can be hard to get detailed data about subgroups of patients - men vs. women, responders vs. not responders, that kind of thing.

So, take home of part 1: When reading a report about research, look for what kind of study it was. Think about the plusses and minuses of that kind of study. Then look at the results being reported and see what kind of "yeah but" (or caveat) might apply.

For further interesting comment, please read the italics that Carrie has added on her blog and/or follow the thread on the forum

No comments:

Post a Comment