Using the QUADAS Tool to Assess the Quality of Research

Have you been feeling lately that the quality of research reports are not always the same across journals?  Or that some articles you read do not appear to have the best methodology?  With all the emphasis on evidence based medicine, it is more valuable now than ever to assure that research reports are of superior quality to assure that we are conveying accurate information to our colleagues.  me1

Today’s guest post is written by Harrison Vaughan, PT, DPT.  Harrison is a physical  therapy practicing in South Hill, VA at In Touch Therapy.  His professional interests include clinical diagnostic tests and treatment consisting of orthopedic manual therapy, predominantly spinal manipulation.

 

Using the QUADAS Tool to Assess the Quality of Diagnostic Accuracy Research Studies

The research community has fortunately developed a tool to critique studies and aid in clinician’s decisions to choose the correct physical examination tests, called QUADAS.  QUADAS stands for:

  • Q – Quality
  • A – Assessment of
  • D – Diagnostic
  • A – Accuracy
  • S – Studies

Many of you may have never heard of it and from speaking with others, this appears to be the norm.

 

What is QUADAS?

QUADAS is an evidence based tool to be used for the quality assessment of diagnostic accuracy studies.  It consists of 14 items phrased as questions, each of which should be scored a "yes", "no" or "unclear" that examine bias in the study.

 

How do clinicians determine which study is most appropriate based on QUADAS score?

Past studies have shown a score of 7 of 14 or greater of "yes’s" to be of high-quality and scores below 7 to be of low-quality.  However, some authors have recommend articles with 10 or higher "yes’s" as cut-off for a high-quality diagnostic accuracy study.

 

How do I use this tool in my clinical assessment?

Below is an example of comparing two studies using the QUADAS score to clinically diagnose SLAP lesion.  These studies were mentioned in a previous post on the clinical examination of SLAP lesions.

O’Brien’s test: Sensitivity: 100, Specificity: 99, LR+: NA, LR-:NA, QUADAS Score: 3

Biceps Load Test II: Sensitivity: 90, Specificity: 97, LR+: 26.38, LR-: .11, QUADAS Score (0-14): 10

From the data above, it appears from first glance that the O’Brien’s test is superior showing great statistical numbers compared to the latter test (even though both show promising figures). However, if you look at the QUADAS score, you can see the significant differences between the two showing O’Brien’s test has much more bias. The Biceps Load Test II is on the lower end of a high quality score (10/14) but much greater study that O’Brien’s (3/14). I am not specifically picking on O’Brien but this makes a good example of similar diagnostic values but very different QUADAS score. In other words, if you obtained a (+) O’Brien’s test, you shouldn’t be so optimistic of a positive SLAP lesion due to poor study design.

I recommend the book Orthopedic Physical Examination Tests-An Evidence Based Approach by Cook & Hegedus to obtain recent QUADAS scores (as well as sensitivity, specificity, etc) for most, if not all clinical diagnostic tests; both new and old. It will surprise you that most of the "classic" special tests that many have been using for years have poor diagnostic value.

Download the QUADAS tool.

It may be difficult for many to change their evaluation regimen, but I do hope this data changes your outlook on the best special test to choose and strengthen your examination.  I hope this information adds to your realm of knowledge and help you become more objective in clinical diagnostics.  Have you used the QUADAS tool before?  What popular study do you know of that everyone references but has a very low QUADAS score?  Thanks Harrison, great post.

Penny Whiting, Anne WS Rutjes, Johannes B Reitsma, Patrick MM Bossuyt, Jos Kleijnen (2003). The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews BMC Medical Research Methodology, 3 (1) DOI: 10.1186/1471-2288-3-25

10 replies
  1. Anonymous
    Anonymous says:

    Hi Mike,
    I'm currently carrying out a systematic review on the validity of the clinical use of diagnostic ultrasound to measure the multfidus. To rate my studies I'm using the QUADAS tool. I've read the whiting paper and it explains how to score every question in detail, however, it and numerous other studies fail to mention how they interpreted their results. i.e what a score of 6 or 7 or 10 means. I see you've mentioned that scores of 7 and above are to deemed high quality and other have used 10 and above. Can you tell me these studies or do you know an article that gives me a scale for QUADAS scores that I can read? I'd very much appreciate yours or anyone's help in this matter.

  2. Anonymous
    Anonymous says:

    Be it CONSORT, be it QUADAS -I assume nobody can expect clinicians to constantly check every detail of our diagnositic work with the latest advances in EBM (as much as this is desirable). A year ago I made a list of the relevant body regions and checked all the tests I would use against the current evidence and adapted accordingly. I guess I’m quite up to date still, but I’ll run another lit review soon. In my opinion, we don’t owe this to our professions (I’m a chiropractor), we don’t even owe it to our patients. We owe it to ourselfs.

  3. Mike Reinold
    Mike Reinold says:

    selena, no harm intended, didnt mean to offend! I think we all agree with you but unfortunately this just isnt the case in the literature. There are many reasons that papers do and do not get accepted into journals, even high quality journals (who you know, politics, the potential to gain readers, the potential to rank higher in medline, etc). The journal isn’t always going to assure 100% top notch quality, but they try. However, if you as a clinician wants to learn a new technique/test, you need to take it upon yourself to determine if the test is valid/reliable/specific/sensitive enough for you to begin using. Everyone has their own threshold in determining this.

  4. Harrison Vaughan
    Harrison Vaughan says:

    I think what Mike was saying is that QUADAS is another tool that helps clinical decision making and if others (researchers) can do the tedious work of going through each study, then I’m all about it! This is their job and we try to integrate it into our practice. This tool doesn’t undermine our abilities to critique articles and do the work ourselves, but only as a simple, quick addition to the process.
    Amy stated it well in that the tool is able to “Organize those important questions and quantify them in a consistent way so that we may be able to compare them quickly and more easily”.
    I try to make things simple and this tool does it for me. Unfortunately, all research isn’t the same and we will always have poor quality studies. Numbers on this scale put the objective in the subjective aspect of research.
    Harrison

  5. amy castillo
    amy castillo says:

    QUADAS may not be the only tool or perfect, however it does provide an important discussion block. How should the quality of studies be judged? We are still learning as a profession.

    It is not enough to simply understand sensitivity, specificity and liklihood ratios. I wish it was! Until recently I thought it was (Thank you Dr. Brian Pease and Dr Michael Ross and UINDY for opening my eyes). We must ask ourselves, what is the quality of the study.

    I think many of us cover these 14 questions in part or whole when critically appraising. Are the results generizable to my patient population? What about the sample used, what is the reference gold standard? Are there threats to internal and external validity? But perhaps sometimes we may not consider all aspects or the study and can incorrectly adapt our practice.

    I believe the intention of this tool was not to make our jobs as clinicians more difficult but to help us. Organize those important questions and quantify them in a consistent way so that we may be able to compare them quickly and more easily.

    To look again at the Labral test, JOSPT 2008 Vol 38#6, Dessaur and Magarey did a systematic review of all labral tests and used QUADAS to rank and compare them. (Although they used a 26 point scale adaptation I believe) I found this helpful at a quick glance to say… that Bicep Load Test shows some promise…I am going to watch for more emerging literature. Maybe I should stop using OBriens.

  6. Anonymous
    Anonymous says:

    I can check out the book… the issue isn’t that there is another piece or work out there to be of assistance.

    I never said that research was as valid as one would think. The issue is that the journals that we trust should have it a high priority to ensure the level of quality of the published material OR the transparency to report the level of quality. This is not an issue of having someone else “do the work for me,” Mike. I take offense to that comment. The issue is one of trust, responsibility, accountability and quality. It is the responsiblity of not only researchers, but also editors and reviewers for the peer reviewed journals to “do the work.”

    Picture this.. a clinician spends time to understand evidence based practice and does look at the statistics to understand the value of a particular examination test… the clinician takes the time to change practice – habits are difficult to change… and then, the clinician learns the QUADAS thing and apparently no one paid any attention and now either clinicians need to apply that to research OR review a book to see how again their clinical ways need to change? It’s ludicrous. Clincians aren’t yo-yos.

    The process needs to change. That book will be outdated in 3-5 years – it very well may be already. One cannot rely on textbooks in this every changing world.

    Researchers, editors, reviewers… please do your job right the first time. Implement research into your processes. :)

    Selena Horner

  7. Mike Reinold
    Mike Reinold says:

    I would agree with checking out that book Selena. Letting other people do the work for you is fine!

    Unfortunately I do not think we can assume that research is always as valid as one would think. I think Harrison’s example of the O’Brien test is a perfect demonstration of this, yet for some reason this is still the most commonly performed test for SLAPs and some doctors even base their diagnosis off this test exclusively!

  8. Harrison Vaughan
    Harrison Vaughan says:

    Selena,
    I agree with you completely! Clinicians certainly don’t have the time to do this. I think it should be the researchers responsibility to give this information when new diagnostic tests are created to make it easier on clinicians but if it doesn’t qualify as “high-quality”, then of course they don’t want to publish the QUADAS. The authors in the book I linked did a very good job at breaking down all past diagnostic articles and performing the QUADAS themselves. They did all the work and “ran” QUADAS (no telling how much time it took!). If you haven’t seen the book, its worth checking out to see how useful it is.
    -Harrison

  9. Anonymous
    Anonymous says:

    I know I’m going to sound really, really rude, but as a practicing clinician, is it realistic to take every clinical paper on examination tools and run QUADAS? I mean, come on… I would think that 1) researchers would be aware of it and keep those 14 components in mind AND 2) the publishing journals would be somewhat responsible for assisting us clinicians in determining the value of the information.

    I had posted on QUADAS quite a while ago at the Evidence in Motion blog. I know that I have a difficult enough time staying on top of research, reading it, reflecting upon it, determining if the population I treat fall within the research…. to add further responsibility to a practicing clinician does seem somewhat ridiculous when processes could be changed to include something like QUADAS prior to the paper being published. That’s just my opinion though.

    Selena Horner

Comments are closed.