On this episode of the #AskMikeReinold show, I’m joined by Phil Page to discuss the current state of scientific evidence and journal articles in the physical therapy world. Not all evidence is good evidence. Here’s how to stay current with the best research.
For those interested in learning more, Phil has a presentation for my Inner Circle Online Mentorship group that digs in really deep on these topics.
To view more episodes, subscribe, and ask your questions, go to mikereinold.com/askmikereinold.
#AskMikeReinold Episode 211: Not All Research is the Same: The Difference Between Good and Bad Research
Listen and Subscribe to Podcast
You can use the player below to listen to the podcast or subscribe. If you are enjoying the podcast, PLEASE click here to leave us a review in iTunes, it will really mean a lot to us. THANKS!
Show Notes
- See Phil’s presentations on understanding and reading research in my Inner Circle
- Tips on Reading Research Articles
- What is Evidence Based Practice
Transcript
Mike Reinold: Welcome back everybody to the latest episode of the Ask Mike Reinold show. I am with my good friend Phil Page. Phil and I just spent the last couple hours or so now kind of filming some presentations for my Inner Circle on research, which is one of the areas that I think Phil is awesome at. He’s obviously one of the people I’ve looked to over my career on how to learn, how to read journal articles and digest information from them and understanding good research and bad research. And I’ve seen him speak and do a bunch of things over the years. So Phil’s the man for that and I’ve been trying to get what for like, what was it two years you think Phil?
Phil Page: Two years.
Mike Reinold: Yeah, two years. I’m just like, “Hey man, you got to fill in these presentations for my Inner Circle. I think everybody would love them.” So anyway, thank you for doing that. If you’re not a member of my Inner Circle, you’ve got to check it out. But I also wanted to do a special podcast episode because him and I were talking and I have a lot of questions. So Phil, before we get started a little bit, just tell everybody a little bit about yourself.
Phil Page: Yeah, thanks Mike. So I am Research Director and Assistant Professor in physical therapy at Franciscan University here in Baton Rouge, Louisiana. It’s a new program. We’re just finishing up our second cohort. So it’s a new school and I spent the last 20 years working with Performance Health as their Director of Research and Education. And I’ve been a research geek since my first year of college. Many, many years ago as an undergraduate student athletic trainer, I did a research project as a freshmen, believe it or not. And I got the research bug and I’ve had it ever since. And so it’s really just been a fantastic ride for me to now be Research Director. It’s like a kid in a candy store, right? So I’ve got 40 students every year, they’re doing independent research projects. So at the end of the cohort, I’ve got 80 students involved in research at one time. I just I love, I just love this. It’s so much fun.
Mike Reinold: One of the things I give you a lot of credit for though is that you’re a PhD and you’re a research based kind of person here, but you’re very clinical. And I think sometimes we don’t have both of those put together. So I think you get to see a research article and the things that are being published from multiple perspectives because of that. And I think that’s super helpful. So that was neat. So I’m going to start off with this question.
Phil Page: Okay.
Mike Reinold: I want to go from here. I’m going to kind of ask you the questions, I think.
Phil Page: Sure.
Mike Reinold: I don’t know what this is. This is a weird episode, but we’re going to go with it. I remember when we were first getting started and we were younger in this profession, journal articles were golden. They were evidence, they were science. Everything about it was always perfect. So you went to a journal article and said, “This is perfect.” I feel like nowadays where I don’t know if things are becoming so diluted or we just have so many journals, but I feel like there’s a big difference now between a publication and evidence. Right? There’s so many publications out there. I’ve read so many articles over the last several years, especially the last five or so, that I just think this is not good research. This is bad research. This is not helping our profession.
Phil Page: Yep.
Mike Reinold: This is not helping us. What are your thoughts on this? Is it just me? Am I crazy thinking that?
Phil Page: No Mike, it’s tough. And I keep up with the literature. I’m fortunate enough to do that as my job. Right? So everyday clinicians that are out there working eight to five and seeing patients, it’s tough to keep up with the literature. And I get frustrated that there is so much out there first of all, that you have to sift through. Right? And a part of it, Mike obviously, is that there is that, I’ll publish or perish.” Right? So there’s this big push for academics to have to publish their works. I will tell you that, at least in the physical therapy world, it’s not that big of a push because in the [inaudible 00:04:15] criteria for physical therapy school, I think we only need two publications every five years or something like that. So in a way that’s good because it’s not flooding the market with all this stuff that may not be good. Right?
Phil Page: Now at what you’re seeing the other part of this, which I think is the bigger driver, is there’s two things. One is the international side of things where there’s more global influence with the internet that then ties into those journals that are for-profit journals. And you may call those predatory journals or I think there’s a subgroup called pseudo journals if you will, pseudo predatory journals where there’s now a market fee for publication. Right? Now, I’m not going to say that fee for publication is a bad thing because it does offer free access. But there are people that take advantage of that and that they allow articles to get through with very shoddy peer review. Okay?
Mike Reinold: Right.
Phil Page: So that’s one side of it, is that these articles these days, pseudo journals. You look at them and they’ll have very close names to the things that we’re reading every day and they want to make it look good. All you need is a word processing thing and an internet page to make it look like a journal. Right? The other thing is there’s a lack of quality reviewers out there. And there are some articles that get through the review process and I’m not sure how it does, but again there’s a shortage of people that are volunteering to review. Because a lot of us that are actually over here writing and teaching and clinically working, we don’t have time to add all these reviews. And I get review requests from journals I’ve never heard of before.
Mike Reinold: Right.
Phil Page: Like, “Would you mind spending a couple of hours to help out my journal that I’m profiting from?” Now I review all the time for our journals because I think it’s a service that we have to do. So there’s a lot of factors going into it Mike, and it’s really tough. But now what that means, Mike, is that you can’t rely on that article. You’ve got to be the one that actually looks at it and is able to evaluate it.
Mike Reinold: Right. And it’s funny because I was going to ask you about these predatory journals a little bit here too. From my perspective, I think you’re being nice. So you can be, you’re the guest on the podcast. You have to be a little bit nicer. There are, I agree with your sub tiers, there are predatory journals that are just terrible. They are just awful. And I mean, there’s typos. You can tell them it’s like really obvious when it’s a predatory journal. But I will say that yes, I like the for-profit journals. I like that they offer free access to the things. The peer review process is terrible-
Phil Page: Yeah.
Mike Reinold: -even for the best ones. Absolutely terrible in the peer review process. So even some of the bigger name ones from reputable journals that are now doing an online version, I’m getting away from because I continue to see poor methodology. So look at this point in time, there’s a zillion research articles out there. Remember we used to hoard those JOSBTs.
Phil Page: Yeah.
Mike Reinold: And that was like, you know what? You’d get that for the month and you’d read the whole thing. There are so many articles there right now that I just feel like we can answer any question we want and get whichever answer we want the harder we look. If you’re looking, does A do B, you can find yes and you can find no. Right?
Phil Page: Yeah.
Mike Reinold: So that’s why I think we just got to take a huge step back and just start sticking to the main journals. Right? Like the top ones in our profession. So based on that, what are your favorite journals that you read? Because I know a lot of my listeners and people that watch the podcast actually want to know this. They want to say like, “Alright. If there’s some good journals and bad journals, what are your favorites?” We won’t talk about the bad ones, but what are your favorite journals?
Phil Page: Again, it goes to … I think it goes to your interests. Right? So I am not treating patients every day. So my interest is maybe more on the biomechanical analysis side. Right? So journals like Journal of Electromyography and Kinesiology. I mean, that’s not something that every clinician is going to want to read but I’m using it as an extreme example. But I love, I’m biased, International Journals for Sports Physical Therapy. Full disclosure, I’m an Assistant Editor of that. But what I love about that journal and what Dr. Voight and Barb, we put together this journal that was practical and it was applied.
Phil Page: Now there’s that balance Mike, right? So the problem is if I want to do a really good internally valid study, I call those petri dish studies. They’re sterile and they’re well controlled, but what’s the external validity? There is none. So you have to give and take, right? So sometimes these articles may not be very strong scientifically or internally valid, but you’d give it up with that externally. So a lot of what we do with the IJSPT is you’ll see, it’s not a lot of hard science, it’s practical science. And there’s going to be bias, there’s going to be some issues with that.
Phil Page: And quite honestly we get a ton, I can’t tell you how many people, how many article submissions we get. I think Ashley Campbell had told me that we get like a hundred a month. Mike Voight could probably tell us. I mean, the numbers is insane. Do you know why I think we get so many? Open access.
Mike Reinold: Yeah. Because people can read it.
Phil Page: Pub med.
Mike Reinold: Right well, so pub med open access. But I think you nailed it. It’s a more clinical journal.
Phil Page: Yes.
Mike Reinold: And there’s a lot of differences out there. Over my career, I’ve seen the transition. I remember when PT Journal, which I haven’t read in over 15 years. Right? So I remember when PT Journal made that transition. I remember when JOSBT started to make that transition of less clinical. And I remember … so like Kevin Wilke and myself, we used to write tons of current concept papers and we’d always go JOSBT or something like that. And every CSM, APTA meeting, we would … like Gee Seminole, the editor at the time, he would always come home and be like, “Hey, Mike you had the most read article of the year again.” Right? Like everybody loves your article. And I was like, “Thanks Gee.” It’s clinical implications.
Phil Page: Yeah.
Mike Reinold: And I remember now JOSBT is kind of starting to get away with that. In the discussion, if you start talking like, “Well, I think this means,” or, “The clinical implications of this means this,” sometimes the editors say, “Hey, let’s not talk about that.” And I think people miss that, I think they want to get that and that’s why International Journal of Sports Physical Therapy is fantastic. I think sports health is doing a good job with that.
Phil Page: Yes.
Mike Reinold: That’s become one of my favorites now. For pure science too, for the people keeping track, I think AJSM, General Sport … excuse me, Journal Shoulder Elbow Surgery, Arthroscopy, The British Journal of Sports Medicine is doing a really good job. Those are some of our go tos. And look, there’s a million articles out there. You can read articles all day every day if you want. None of us have time to do that. Just pick like four of those and just stay current and you’re going to be in the upper 99% or 1%, whatever the good percent is. You’re going to be in the one of our field. Right? Does that make sense?
Phil Page: Yeah. And it goes, I also like more of the science stuff. Right? So MSSE, Mid-size Sports Exercise.
Mike Reinold: Yeah, good one.
Phil Page: Journal of Strength and conditioning Research tend to have some good stuff in it. But one of the things that I do is I just play the field. Right? So I actually subscribe to the table of contents for a lot of different journal articles and then I just look through every month and I go, “Yep. No, no, no.” And then that way I don’t really have to focus on one journal per se, with having a wide variety of them. But you do want to make sure that, as you said, that you’re focusing your topic. That’s the key thing.
Mike Reinold: Right.
Phil Page: I get all over the place, man. I got, “Oh, there’s a shoulder article. Oh, there’s a ankle. Oh my gosh, there’s an EMG article over. Oh my gosh.” But going back to what IJSBT does, and I agree with you 100%, they have that wide. We’ll do anything from an EMG study to an epidemiological study and some journals like JOSBT have focused in on these specific topics. So be careful in what you choose and don’t limit yourself. And again, follow Twitter with these updates too because there’ll be journals out there that I’ve never heard of or articles that I’ve never seen. And I’ll see one tweet about it and I’ll go, “Hey, I would’ve missed that if I had just stuck with one journal.” Right?
Mike Reinold: Right, right. Yeah. And getting together with some like-minded people on Twitter, for example. It’s like they’re helping you because we’re all going to read research together and tweet the articles you like. Right? So that way then you’re getting some help from others. And I always find things that yourself or Dan Loren’s or somebody like that is tweeting out and I’m like, “That’s amazing.” So one question, or another question I have for you, because I have a lot of questions, but for this episode at least. I want to talk about some of my disappointments in research with this episode a little bit.
Phil Page: Okay.
Mike Reinold: I am currently getting very sick of systematic reviews-
Phil Page: Oh.
Mike Reinold: -and metaanalyses. I’m getting very sick of them Phil. We are starting to dilute information and have these gigantic reviews that always end with no findings, right? I’m not kidding. I’m going to slightly make this up, but I’m dang close. This is close. There is articles that look like, “Does a special test mean you have a rotator cuff tear?” And then you’re like, “Okay.” And then you look at the methodology and there’s 40 studies and subjects range from age 20 to 90. Right? And you’re like jeepers there’s a big difference between a 20 year old and a 90 year old. Right? And then pathology goes from impingement to a full thickness tear with retraction. And then their conclusion is always like, “Well, we didn’t find anything.” Right? Well of course you didn’t find anything. Your subject pool is gigantic. Your pathologies are gigantic. It’s just, it’s a lot. So what can … tell me about this, am I wrong here?
Phil Page: No.
Mike Reinold: That there’s just way too much dilution going on.
Phil Page: I’ve done quite a few of these, actually.
Mike Reinold: The good ones, there’s good ones. I’m not saying they’re all bad because Mike [inaudible 00:15:45] always gets mad at me on Twitter when I say this. Because people feel like I’m talking about them. I’m not talking about them, but kind of if you put a lot of bad studies together, you just have one big, bad study.
Phil Page: I know. Well, the first time I heard this was at ICAS with Mal and Tim. And they were bashing systematic reviews left and right.
Mike Reinold: Oh no way. Really? So you said Mal with you and Tim.
Phil Page: Yeah, this was about three years ago. So I go, “What’s wrong with them?” And it’s because what had happened was systematic reviews became, they didn’t became, they are level one research. Okay. What happens is, and this is level one, which in your Inner Circle presentation I talked about how just because it’s a level one of evidence doesn’t mean it’s a good study. You have to have quality of a study. You have to go beyond the level of that, right?
Mike Reinold: Right.
Phil Page: So yeah, you will have very poor level one studies, meta analyses, all these kinds of things. Because if it’s built out of poorly designed search criteria, poor quality studies, it’s going to give you crap. Crap in is crap out. Right?
Mike Reinold: Right, right.
Phil Page: That’s what’s going to happen with a lot of these. Why are there so many? They’re relatively easy to do, right? So we all know that a clinical trial is not easy to do, right?
Mike Reinold: Yeah, takes a long time.
Phil Page: Systematic reviews are … they’re not easy, but they’re not as hard because you don’t have so many things to control. Right? Second thing is, again … let’s say number one is that they’re level one. Number two is they’re relatively easy. Number three is that they tend to be the most cited types of articles.
Mike Reinold: I know. Yeah, the journals love them.
Phil Page: And that’s why you get a lot of publications because the journal index factors are based on the number of citations. And so is your, sometimes your author index. Right?
Mike Reinold: Right. Yeah. It helps. It helps. I think you left out one too is a lot of times, and this is one of my problems with systemic reviews … systematic review. It’s been a long morning. I can’t wait to read the transcription of that. One of my bees or one of the things I’ve identified here is this is oftentimes a resident that’s doing this. And the senior author, like a well known orthopedic surgeon or something like that might have his residents all doing something and they’re doing a project, but they have no clinical judgment. So they don’t even understand that some of the methodology is poor or the implications they’re making are a little bit off. So I also think this is something where we’re seeing a lot of younger people also doing these types of projects too. And not that they’re not ready for it or they shouldn’t do it or anything like that, but they just don’t have the clinical judgment to say like, “Oh, the way that they measured range of motion may have not been valid.”
Phil Page: Right.
Mike Reinold: For example. So, you know what I mean? So I’m seeing that too, but.
Phil Page: Well no, you’re exactly right. It’s not as easy, and people confuse it with a literature review. Right? It is not a literature review. And actually there are different types of systematic reviews. And the problem is in order for you to do a good systematic review, you really have to have this … a pretty homogenous kind of grouping, right? Because what you’re doing is you’re creating a bucket of evidence. What you’re trying to do is make a conclusion based on similar studies-
Mike Reinold: Right.
Phil Page: -or reviewing these types of things. But to your point, you do have to look at those factors. And if you don’t know what you’re looking for, you’re not extracting the right information. Maybe you’re not even choosing the right articles. So where I’m heading Mike, where I think a lot of people need to be going, are scoping reviews. People don’t know what a scoping review is, but a scoping review is really a broader picture of the literature that really captures everything from our metaanalysis down to our narrative reviews down to our case studies, they include everything. We usually use scoping reviews with a body of knowledge that’s smaller. Okay?
Mike Reinold: Right.
Phil Page: If you don’t have five or ten articles that are worthwhile including in a systematic review, don’t do it. It should be a scoping review. You should expand that out-
Mike Reinold: Right, right.
Phil Page: -because you don’t have enough information to make a good conclusion from a systematic review. A friend of mine is an orthopedic surgeon in Texas, Brad Edwards. He’s done a lot of JSPS and he, I remember a couple years ago, he did a nice little editorial in there that talked exactly about what you’re saying. Is it’s time to stop publishing these systematic reviews that don’t tell you anything. It’s like they should be rejected.
Mike Reinold: Right.
Phil Page: I mentioned that you should not reject an article that’s not significant. A lot of times, sometimes articles are rejected because there’s no difference between the groups, which I think is wrong to reject. But when I have a systematic review that says, “We can’t make conclusions.” Don’t publish it. That’s the way I feel about it.
Mike Reinold: It’s actually, it’s a good point. If you can’t come to a conclusion, I think what happens sometimes is they come to the conclusion and the conclusion is that the body of the evidence is limited and more research is needed. Right? That’s what we see. But again-
Phil Page: Every time.
Mike Reinold: -using that example again, I’m not kidding. “Does manual therapy work for shoulder pain?” Holy smokes that’s a terrible clinical question because it never defines what manual therapy is. You don’t know the experience level of the person applying it. They’re grouping a bunch of interventions as one thing. And then again, you have patients aged 20 to 90 that all have different pathologies. And then they say like, “No.” You know what’s worse? This is what’s worse Phil. We’re going on a tangent here. Here’s what’s worse. Not only is it going to say that, everybody in social media is going to say this study says manual therapy does not work.
Phil Page: Yep. That’s right.
Mike Reinold: And that’s not what it said. That’s not what it said. It just says they couldn’t find anything. They’re going to say manual therapy doesn’t work. Well of course manual therapy, whatever on earth that means in that study over a wide variety of people with a wide variety of pathology, will do a wide variety of things. Right? Is that pretty accurate?
Phil Page: You can’t have it both ways evidence-based people. So evidence-based remember is not based on just a research paper. That’s a systematic review that has a bunch of bias in it. It’s not about articles that confirm your bias, that you think that manual therapy doesn’t work so I’m just going to stick with this one. It doesn’t mean that this article is against you so I don’t talk about it. Right? And it’s about adding the context of the situation of the patient and your experience. So manual therapy might work in certain people. Don’t throw it all out. I love the arguments with ultrasound. I’m a proponent of modalities in certain situations still.
Mike Reinold: Of course, of course.
Phil Page: There is evidence that supports it and then people will turn right around and go, “Well, this one doesn’t.” And I’ll go, “This one does.” Just like you said earlier, you can find something that supports you all day. But what you said earlier, which really hit home for me was the little things that you mentioned Mike about age range, definition of manual therapy. That’s what’s missing from clinicians to be able to look at those things and point that stuff out. Because like you said, a lot of times these articles aren’t written by clinical people. But at the same time, don’t throw it all out. It’s still good information for us to have. We just have to, again, as clinicians be a little bit smarter and learn how to be better critical thinkers. That’s where we need to do a much better job as clinicians.
Mike Reinold: Right. And I think you nailed it right there. That’s why I kind of wanted to hit this podcast episode with this as a little bit of saying like, “Look. Their evidence is amazing, research is amazing. But it’s not all perfect.” Right? There are some flaws in the system. To really be able to get the most clinical implications further we have to understand the good and the bad and stuff like that. So awesome Phil, thanks for joining us on another amazing podcast. Phil is actually a repeat customer now, a repeat guest.
Phil Page: That’s right.
Mike Reinold: Right? You’ve been given in the past. I stole him when we were at one of our society meetings in the past and we did some episodes. So it was good to have you back Phil and thanks again for your presentations for my Inner Circle. I know they’re all going to love it and I really appreciate it. If you have any questions for the podcast, for myself, some of my amazing guests that we have every now and then since we’re Zooming from home during all this stuff, head to mikereinold.com and click on that podcast link and fill out the form to ask us more questions. Please head to iTunes, Spotify. Rate, review, subscribe, all those crazy things you do to podcasts nowadays. And we will see you on the next episode. Thanks so much.
Phil Page: Thanks guys.
