Oh I agree. I’m coming at this from the medicine side though and there’s some specific concerns and criticisms for Cochrane reviews and the meta analysis method in general, though on the whole I generally agree with what they’re going for. A really simplistic example but I think illustrates the point well would be this:
A meta analysis of the effectiveness of parachutes concludes there is no evidence to suggest their effectiveness, due to lack of high quality double blind randomized controlled trials.
I’m a huge proponent of evidence based medicine, but sometimes people are quick to point to a Cochrane review or other systematic review as biblical scientific dogma that cannot be defied, and don’t consider the nuances of the situation. Like what went into the meta analysis and why? What data might be missing from the meta review and why? Is only including double blind randomized controlled trials the best research method to answer the specific question being addressed? Was everything included together actually comparable? Was the specific question the review asked the right question to ask? How broad can this finding actually be applied?
Part of the problem in this case though was the Cochrane reviews findings being somewhat misrepresented in the popular media by those not familiar with the format or the nuances. No doctor is gonna read that review and rip off their masks when treating covid, flu, or other airborne or droplet diseases. Masks were used in the hospital long before covid and will continue to do so, for good reason.
what went into the meta analysis and why? What data might be missing from the meta review and why? Is only including double blind randomized controlled trials the best research method to answer the specific question being addressed? Was everything included together actually comparable?
The critiques / rebuttals to the masking review typcially ask these questions. This review almost falls apart after considering the answers.
If there’s not enough information to form a solid conclusion, maybe they have no business analyzing it until there is. At the very least, if they’re going to include partially unrelated studies, then reflect that in the title / opening statement. Don’t say the analysis is of apples, then analyze fruit in general.
Oh I agree. I’m coming at this from the medicine side though and there’s some specific concerns and criticisms for Cochrane reviews and the meta analysis method in general, though on the whole I generally agree with what they’re going for. A really simplistic example but I think illustrates the point well would be this:
https://www.bmj.com/content/327/7429/1459
A meta analysis of the effectiveness of parachutes concludes there is no evidence to suggest their effectiveness, due to lack of high quality double blind randomized controlled trials.
I’m a huge proponent of evidence based medicine, but sometimes people are quick to point to a Cochrane review or other systematic review as biblical scientific dogma that cannot be defied, and don’t consider the nuances of the situation. Like what went into the meta analysis and why? What data might be missing from the meta review and why? Is only including double blind randomized controlled trials the best research method to answer the specific question being addressed? Was everything included together actually comparable? Was the specific question the review asked the right question to ask? How broad can this finding actually be applied?
Part of the problem in this case though was the Cochrane reviews findings being somewhat misrepresented in the popular media by those not familiar with the format or the nuances. No doctor is gonna read that review and rip off their masks when treating covid, flu, or other airborne or droplet diseases. Masks were used in the hospital long before covid and will continue to do so, for good reason.
The critiques / rebuttals to the masking review typcially ask these questions. This review almost falls apart after considering the answers.
If there’s not enough information to form a solid conclusion, maybe they have no business analyzing it until there is. At the very least, if they’re going to include partially unrelated studies, then reflect that in the title / opening statement. Don’t say the analysis is of apples, then analyze fruit in general.