We’re taught to treat systematic reviews and meta-analyses as the gold standard. They sit at the top of the evidence pyramid, so when one lands in our inbox saying an intervention works, we tend to believe it and move on. But what if that instinct is exactly the problem?
A new editorial from two of the most respected methodologists in sports medicine makes an uncomfortable argument: the explosion of meta-analyses in our field has outpaced the rigor behind them. Sport-related meta-analyses now make up a striking share of everything published, and when you look under the hood, a lot of them are built on shaky ground. In one recent example, only a single review out of nearly forty was at low risk of bias.
On this week’s episode, I break down the specific red flags that separate a meta-analysis you can trust from one you can’t, including one common practice that leading evidence organizations have flatly called “unacceptable.” I also get into an example that hits close to home for a lot of us: a widely prescribed injury-prevention exercise whose evidence looks very different once the analysis is done right.
If you make clinical decisions based on the research you read, and we all do, this one will change how you read it. It won’t take a statistics degree, just a handful of questions you can ask of any review before you let it change your practice.
Check out this week’s podcast for the full breakdown.
To view more episodes, subscribe, and ask your questions, go to mikereinold.com/askmikereinold.
#AskMikeReinold Episode 396: Drowning in Evidence: How to Read a Meta-Analysis Without Getting Fooled
Listen and Subscribe to Podcast
You can use the player below to listen to the podcast or subscribe. If you are enjoying the podcast, PLEASE click here to leave us a review in iTunes, it will really mean a lot to us. THANKS!
Show Notes
• Drowning in Evidence: The Meta‐Analysis Boom and the Rigor It Demands
Transcript
Mike Reinold:
Welcome back, everybody, to the latest episode of the Ask Mike Reinold Show. We are here with another journal article review episode. I’m joined with Lenny Macrina, Dave Tilley, Brendan Gates, Noah Sprinkle, and Anthony Videtto. We’re here with another journal article. And this one’s a little different, and I think that’s okay. We said we’re going to review some journal articles that make a big impact on our clinical day-to-day, I guess.
And I think a lot of people assume that means research studies where we look at some findings of things that come out. But we have an interesting one today, and it’s actually an editorial. So it’s not a research study that we’re going to draw conclusions from, but an editorial. And the reason why we picked this one for this episode is, I think we all agree when we start talking about this, we agree to the concept here, but I think this is a very important editorial piece and an opinion for our profession right now because we’re going down a road that is misleading, I will say.
So let’s start with this. The article we’re talking about, and the links will be in the show notes so you can check it out, is Drowning in Evidence: The Meta-Analysis Boom And The Rigor It Demands. And this is from the Scandinavian Journal of Medicine and Science and Sport from 2026. And it’s from Franco Impellizzeri and Ian Schrier, who are two well-established and excellent researchers and academic people. They know what they’re talking about.
So when they put together an editorial with their opinion, and trust me, this is a strong opinion piece from them. When they put that together, though, I think that’s something that we need to listen to a little bit. So just real quick, it’s an editorial. It’s not long, so you should definitely check it out. But what they did essentially was this. They start off by saying, “Just because a study is a meta-analysis does not mean that it’s a great study.” And gosh, I remember putting this on Twitter back when it was Twitter, back before it was X, and then everybody yelled at me again, which is part of why I got off Twitter.
But I remember putting on there, “Putting a bunch of crappy studies together does not make one good study. It just makes a big crappy study.” And I guess maybe not everybody wanted to hear that. But that’s what I think sometimes with these meta-analyses. So what they determined here was this, is we’re seeing a rise in meta-analyses. If you look at the percentage of meta-analyses that are being conducted, it’s actually crazy the percentage that are being done in sports medicine.
It’s very large. Sports-related meta-analyses now make up roughly 16% of all meta-analyses. And think about that. Sports is not 16% of medicine. So for some reason in our field, we’re really shifting towards that. And we can talk about why, because I do think there’s some reasons why, but I’ll let everybody else chime in on that. But essentially we’re seeing this. And what’s happening is I think a lot of people are saying, since it’s a meta-analysis, this thing must be perfect.
So what they said is you have to break this down. You have to look at a meta-analysis and determine if it’s a good study. And what they did is they came up with a four-step system that they recommend you do. And by the way, this isn’t necessarily on how to perform a meta-analysis. This is on how to interpret it as well. So it’s for all of us, because you should read these meta-analyses before you determine if you think this is something you should apply to your clinical practice.
So the first thing they came up with was this framework. They come as these four steps. They call it identify, appraise, synthesize, and then interpret. And they gave steps for each one of those. And I really urge you to check that out because you should read each one of those things. But what they’re saying basically is: did they identify a good research question? Was it performed with proper methodology?
And they gave some really good examples of how you can perform a sloppy meta-analysis versus a good meta-analysis. You should then appraise it, and not just appraise it for the outcomes necessarily, or more importantly, the quality, which is something that everybody does, but they bring up the point that we often leave out the risk of bias. And they make a strong point towards that. So you should check that out.
And then for synthesize, they bring up a couple examples, but the one I thought was really cool was they talk about the concept of vote counting. So as they say, six out of seven studies showed that the Nordic hamstring exercise is good at preventing an injury. So it sounds like, wow, that’s the majority of studies, but you don’t know the population of each one, and you don’t know the percentage of those there.
So it’s a sly way of doing it, and then obviously putting that all together for interpretation. So again, what’s the clinical take home for us? To summarize this right here is, just because it’s a meta-analysis, doesn’t mean it’s a great research study. And everybody thinks these meta-analyses now are like scripture. That if a meta-analysis says something, you can go for it. That’s the way to do it.
And we’re starting to see even people on social media like the rehab bros. I think I just came up with that. We say finance bros, it’s like rehab bros. It’s those young, new rehab guys on social media that want to be the contrarians. They want to say nothing works, or they want to confirm their bias with a study, and you put that together. So I’ll leave it at that. But I think if you’re truly into research and you want to make sure that you’re reading stuff on your own, you should check out this article and follow their four-step process so that way you can identify a good meta-analyses versus a bad one because there are a lot of bad ones out there right now.
So what do you guys think? We’ve been talking about this a lot in the background, and we’ve been talking about how there’s so many studies out there. How do you guys handle this? What are your thoughts? What’d you think of this article? Dave, you want to start?
Dave Tilley:
Yeah, I’ll just jump in as fresh in mind because I gave an in-service to the students a few weeks ago, and this was on my mind a lot because I treat a lot of people for back pain, for example, and did a huge PubMed review last year. And what this reflected for me is, what is your system for analyzing evidence and applying stuff in the clinic? When someone sits in front of you with a certain problem, back pain, knee pain, whatever, ACL stuff, Lenny talks a lot about graphs and quads versus patellars, what is your system for gathering information and delivering that to a person who’s in front of you, who your job is to educate them on what is available out there?
That’s part of our profession. So if your system for educating somebody on quad tendon graph versus patellar graft, LET or not, is Instagram, Facebook, and an occasional Google search, that’s not a very good system for analyzing high-level evidence. The hard thing about our profession, especially in our situation where a lot of our time is with clinic, and we don’t get paid lunches, paid hours for research and stuff, that should just be your work you do on your own.
Your system, unfortunately, has to be waking up before a clinic, getting two articles, drinking coffee, going through in depth, reading the full study, looking at the methodology, and then taking an evaluation of “does this apply?” Is the study good? Do they have good internal, external validity? Does this apply to the people that I see in front of me? And how confident do I feel that this is something I can educate somebody about and use in the clinic?
So I think the sentence in here that stuck out to me is, if the studies are so biased that the true effect could differ substantially from our estimate in either direction, how can one support the claim that the intervention works? And that’s what I was alluding to, which is if I look at a bunch of stuff for spinal fractures and someone says hamstring stretching is good for back pain, and that’s the headline article… And I just read that on Google, but I didn’t look at the fact that it was seven kids in the middle of Switzerland who play handball and they’re from eight to 12 years old…
How does that apply to the person that I’m treating? It’s a college gymnast with back pain. It’s tough. So that was what stuck out to me, is unfortunately, you need to have a system in place where you’re willing to do the work to read articles and analyze your methodology before you take conclusions. And in a meta-analysis, you have to look at their methods inside the analysis to say, did the studies, like you said, go inside? Is it worth it?
Is it worth my time? Was it high quality? If six out of seven studies in the meta-analysis are mixed populations with mixed interventions, how in the world can I apply that to the person in front of me?
Mike Reinold:
Agreed. And you brought up a couple of things on meta-analyses that I think are really interesting with how you talked about six, seven patients. You brought that up. I wanted to be very clear about that. But more importantly for me, look it up right now. I want everybody to go look this up right now. Do a lit review right now on the effectiveness of manual therapy for shoulder pain. And to me, this highlights the problem with systematic reviews like these meta-analyses kind of things.
You’re going to find that the patient population was 18 to 65 years old, then they had shoulder pain, and they had manual therapy. They don’t define what manual therapy is, what shoulder pain is. And my gosh, if an 18 and a 65-year-old have the same kind of shoulder pain, that’s crazy. But who performed a manual therapy, blah, blah, blah, blah, blah.
We’ll go down that list all freaking day. But you’ll find the rehab bros, I’m going to trademark that, on Instagram basically saying, “See, manual therapy doesn’t work for shoulder pain.” So it’s hard to draw conclusions based on that. So, to Dave’s point, it’s hard to draw conclusions if maybe it’s not the right population, but it’s also hard to draw conclusions when we have such diluted crap out there.
Of course nothing’s going to work. So guess what the outcome is? Well, guess what their conclusion is? Well, the results are questionable. That’s what they’ll say, or we’re not 100% sure. And people say, well, see, there’s no evidence that it works. So you got to be really careful. You can’t put together a bunch of crappy articles. I guess that’s that main point. But what else? Who else wants to jump in on this one? This is a good topic, but who’s got some more for me? Anthony, you want to jump in?
Anthony Videtto:
Yeah, I guess I’ll add that sometimes when I’m going to the literature or looking for research, I might find that I’m a little bit more lazy than most people. And I find myself hovering towards meta-analyses a lot because I think, oh, well, these people, these authors are synthesizing a lot of data for me on a topic. And I can appreciate that because maybe I don’t have a lot of time to try to find an answer to a question I have that I want to present to a patient that day, let’s say.
And so I think I just take meta-analyses as scripture, like you said, because these authors did all the synthesis for me. I’m just trusting that they did a good job at including good studies and getting rid of the bad studies. So I guess this is maybe a little bit more eye-opening for me, that I probably need to do a better job at actually diving into the studies that were included in these papers, but then have this framework that this paper’s talking about in terms of “how am I identifying a good meta-analysis versus a bad meta-analysis?”
So as someone who might just be looking for a quick answer, I probably need to dive into this stuff a little bit more personally, but I feel like a lot of people out there might relate to that kind of outcome as well.
Mike Reinold:
Yeah. If you feel that way, I bet you tons of people do as well. And I think you’re right. I think we all want that. We all look for that. And then sometimes we’re disappointed. But can we get serious? Can we talk? What’s the dirty secret of meta-analyses? Do we know this? Do you guys know? All right, so let’s start with this. There’s two dirty secrets. Two dirty secrets. This is a tell-all podcast episode. Dirty secret number one. Do you know why so many are being conducted? What do you guys think? Anyone jump in?
Dave Tilley:
They get published faster. They’re bigger.
Mike Reinold:
Yep. They get published fast.
Anthony Videtto:
They’re easier to conduct.
Mike Reinold:
Yeah. So you’re seeing a lot of fellows, residents, research assistants, academic institutes… You could sit in front of a computer and do this. You don’t need a lab, you don’t need IRB, you don’t need to go in there, and you could pump these things out. So you can get so many publications. So that’s part of why we’re doing it. What’s the flip to that? Why are the journals publishing so many? Dirty secret number two. I’m going to whisper a little ASMR for that.
Anthony Videtto:
People read them more?
Mike Reinold:
People read them more, they download them more, and it helps with the ranking of the journal. Becomes popular. So authors want to do them, journals want to publish them, and then rehab bros on Instagram want to say nothing works. And that’s the dirty cycle of meta-analyses right now. So look, we can do better.
I don’t expect everybody in the world to go out there and just do these huge lit reviews all the time. But you know what? We’re at a point now where we can do better. We have websites like Open Evidence. If you’re not using Open Evidence right now, which is an AI tool that’s specific to medical journals to help you to go through and search through stuff, you should be. But we can do better now. And the other thing is to just follow the right people online. You should not be getting your research updates and your Con Ed from Instagram.
And trust me, everybody here on this podcast, we teach via social media. But hopefully you know that we also teach, and we also publish, and we also conduct research, and we’re actually really clinicians. I think that’s why people like us and trust us. But just realize that not everybody that you’re learning from may be in that boat. So be careful with these meta-analyses traps.
And thank you so much to these authors for publishing this editorial because now it’s not just us saying that, but you actually have quality researchers that are well respected in the field that throw this editorial out there because the world needs to see it. Just because it’s a meta-analysis doesn’t mean it’s quality. So we have to be very careful with the interpretation of that. Okay? Awesome. So, good review. Thanks guys. Appreciate everybody’s input on that one, and thank you so much for that.
You can check out the article in the show links on the website. You really should because I do want you to read it. It’s quick. It’s an editorial, so you can fill it in some small gaps in your schedule, but check it out. And be sure to subscribe so you can get future updates of our next episode. See you on the next episode.
