Why This Page Exists
Independent educational resource. This page does not recommend a product or replace medical care.
By MushroomHealthBenefits.com Research & Editorial Team | Reviewed September 12, 2026
A headline says, “A systematic review found that reishi mushroom improves immune function.” Should you trust that sentence? The answer depends on how the review was built, not on the word “systematic.” Two reviews can use the same word and still deserve very different levels of trust. This page shows you what to check before you believe a review’s conclusion, using the methods described in the Cochrane Handbook for Systematic Reviews of Interventions and the PRISMA reporting guideline.
The Reader’s Decision Point
Before you act on a review’s conclusion, you need an answer to one question: did this review look hard enough, look fairly, and combine studies in a way that makes sense? A review that skips any one of those steps can reach a confident-sounding conclusion that the underlying evidence does not support.
A Short Process Map
A well-built systematic review follows a set order. Each stage protects against a specific kind of error.
- Protocol first. The review team writes down its question and methods before searching, so they cannot quietly change the rules once they see the results.
- Broad search. The team searches multiple databases and other sources, not just one, so relevant studies are not missed by accident.
- Screening against fixed criteria. Every record is checked against eligibility rules that were set in advance, not chosen to fit a preferred conclusion.
- Quality and bias check. Each included study is examined for weaknesses that could distort its result.
- Combining the results. If studies are similar enough, their results may be pooled. If they are too different from one another, the review should say so instead of forcing a single average.
How Did They Search? A Question-Led Explainer
Did they search more than one database?
A search limited to a single database misses studies that were indexed elsewhere. The Cochrane Handbook expects reviewers to search multiple bibliographic databases along with trial registers and, often, sources of unpublished or “grey” literature, so that promising results are not overrepresented and less favorable results are not left out.
Can you see the actual search terms?
The PRISMA 2020 statement is a reporting checklist with 27 items and a flow diagram that tracks how studies move from the initial search through screening to final inclusion. A transparent review reports its full search strategy for each source, not just a summary sentence like “a comprehensive search was conducted.”
Does the flow diagram add up?
The PRISMA flow diagram shows, in numbers, how many records were found, how many were removed as duplicates, how many were screened, how many full texts were checked, and how many studies were ultimately included, with reasons given for each exclusion. If a review reports a conclusion but cannot show you this trail, treat the conclusion with more caution.
How Did They Decide What Counts?
Were the eligibility rules set before the search?
Eligibility criteria, such as which mushroom species, preparations, doses, populations, and outcomes qualify, should be fixed in the protocol stage. Criteria written after the results are known can be shaped to include favorable studies and exclude unfavorable ones, whether or not that was the intent.
Did two people screen independently?
Cochrane methodology calls for at least two reviewers to screen studies independently and resolve disagreements, which reduces the chance that one person’s judgment alone decides what evidence gets counted.
How Did They Check Study Quality?
Combining studies without checking their quality can produce a confident-looking number built on weak evidence. For randomized trials, Cochrane’s current tool for this step is called RoB 2. It examines five specific domains: the randomization process, deviations from the planned intervention, missing outcome data, how the outcome was measured, and whether the reported result was selectively chosen. Each domain, and the study overall, is rated as low risk of bias, some concerns, or high risk of bias.
- A study can be well-designed on paper but still earn a “high risk” rating if, for example, a large share of participants dropped out and that dropout was not handled appropriately.
- A review’s overall conclusion is only as strong as its weakest heavily weighted studies. A pooled result built mostly from high-risk-of-bias trials deserves less confidence than one built from a majority of low-risk trials.
Does the review tell you the bias ratings, or only the pooled result?
A trustworthy review reports the risk-of-bias rating for each included study, usually with a summary table or graph, so you can see whether its conclusion rests on strong studies, weak ones, or a mix.
How Did They Combine the Studies? Understanding Heterogeneity
“Heterogeneity” means how much the results of different studies vary from one another beyond what chance alone would explain. This matters because averaging together studies that measured genuinely different things, in different populations, with different doses, can produce a number that does not describe any real-world situation.
What is the I² statistic?
Reviews often report a value called I², which estimates the percentage of variation across study results that comes from real differences between studies rather than random chance. The Cochrane Handbook offers a rough guide to reading it: roughly 0% to 40% might not be important, 30% to 60% may reflect moderate heterogeneity, 50% to 90% may be substantial, and 75% to 100% is considerable. The Handbook is explicit that these bands overlap and should not be read as a strict rule; a review’s own explanation for a high or low value matters more than the number alone.
What should a review do when heterogeneity is high?
- Explore likely reasons, such as differing mushroom species, extract types, doses, or study populations.
- Report results by subgroup instead of forcing one pooled number, when the studies are too different to combine sensibly.
- Use caution language in its conclusion rather than presenting a single average as if it applies broadly.
A review that reports a pooled result without mentioning heterogeneity at all has skipped a step that materially affects how much weight the number deserves.
How Certain Is the Overall Conclusion?
Beyond the quality of individual studies, some review teams rate their overall confidence in a finding using a structured framework that weighs study design, risk of bias, consistency across studies, how directly the evidence answers the question, and precision of the estimate. A review that states its confidence level in plain terms, such as noting that a finding is based on limited or inconsistent evidence, is being more transparent than one that reports a number without any statement of certainty.
Printable Checklist: Reading a Mushroom Systematic Review
- Was a protocol written and registered before the search began?
- Were more than one database and other sources, such as trial registers, searched?
- Is the full search strategy reported, not just summarized?
- Is there a flow diagram showing how many records were found, screened, and included, with reasons for exclusion?
- Were eligibility criteria fixed in advance, and do they match the mushroom species, form, dose, and outcome you care about?
- Did at least two people screen and extract data independently?
- Was each included study assessed for risk of bias, with the ratings shown?
- Is heterogeneity reported and explained, not just a pooled number?
- Does the review state its overall confidence in the finding, including its limits?
If a review you are reading cannot answer most of these questions, treat its headline conclusion as a starting point for more reading, not a settled fact.
What This Page Cannot Tell You
This page explains how to evaluate a review’s methods. It does not grade any specific mushroom, product, or health claim, and it is not a substitute for reading a review’s full methods section. For how this site grades the strength of evidence behind a particular claim, see the human evidence guide. For how to spot when a marketing claim overstates what research actually found, see the myths and evidence guide.
When to Ask a Clinician Instead of a Study
A well-run systematic review can tell you what the current evidence shows across a population. It cannot tell you whether a mushroom product is right for your own body, medications, or health history. If you are pregnant, breastfeeding, managing a health condition, or taking prescription medicine, review the safety and interactions guide and speak with a qualified clinician before making a decision based on any study, review, or article, including this one.
Educational Disclaimer
This article provides general educational information about how systematic reviews and meta-analyses are conducted and evaluated. It is not medical advice, and it does not diagnose any condition or recommend that you start, stop, or change any supplement or medication. MushroomHealthBenefits.com is an independent educational publication and does not sell or endorse mushroom products. Ask a qualified clinician about personal risks, medications, pregnancy, or symptoms.
The information in this article is based on peer-reviewed research, published clinical studies, and guidance from recognized health organizations including the National Institutes of Health (NIH) and the FDA. Individual results and experiences may vary.