Summarizing and Grading the Evidence: The Example of Prevagen Part IV

This is the last of our series using the example of Prevagen, the world’s best-selling memory supplement, to learn how to examine a products’ claims for effectiveness. In prior blogs we examined which sources go to for reliable information, surface-level cues of unreliable manuscripts, and study design. In this last blog, we’ll put things all together to see what the strength of evidence is to support Prevagen’s claims.

From 2018 to 2020 I served on the American Academy of Neurology’s Guideline Subcommittee where I learned how to rigorously review evidence for treatments and diagnostic tests. This group was tasked with overseeing: “the development, dissemination, and implementation of evidence-based systematic reviews, practice advisories, and clinical practice guidelines.” In plain English, we took an important clinical questions (e.g. how to manage stroke in children), did a comprehensive review of the literature, made recommendations based on the evidence, and pointed out areas where there was insufficient evidence to guide care.

This group of doctors and scientists is obsessive (in a good way) about evidence and accuracy. Crafting a single guideline took several years and at least a dozen people. The process included identifying all relevant studies, grading each study on the quality of evidence, grading the overall weight of evidence if there were multiple studies, and finally carefully wording the guideline. There were rules for each step, and every step involved coming to a consensus among the group.

If this group ever took on the clinical question, "Are there supplements that address age-related memory issues in older adults”, they would come across the Prevagen study.(1)

Grading the quality of this study would include:

1. Randomization: Not all ways of randomization are equal. For example, randomizing treatment based on what day of the week a person shows up for the study could introduce bias and be manipulated (e.g. a potential participant can be told to come back tomorrow). The Prevagen study did not describe how they did randomization. This is concerning, particularly as there is no explanation as to why there were 40% more people randomized to Prevagen than placebo.

2. Masking: The Prevagen researchers did a good job with masking participants and evaluators. They did not mention whether statisticians were blinded. As they reported results from analyses that were not pre-planned, one should be concerned that statisticians were not blinded and were searching for results where Prevagen outperformed placebo.

3. Accounting for All Participants: It is important for clinical trials to account for all participants who started the study. If participants drop-out, it is important to know why (e.g. side effects) as this could introduce bias. For example, imagine a 3-month study of drug X to treat headache. Over the course of the study 25% of people taking drug X quit the study because their headaches got worse and another 25% quit because they noticed no benefit. Among the 50% who lasted 3 months many thought their headaches were improved. If we only included the 50% who made it to 3 months, our results would be biased and we would greatly over-estimate the benefits of drug X. The Prevagen study did not provide an account of study participants or even provide a drop-out rate.

4. Statistical Analyses and Presentation of Results: For most people, even researchers, statistics is intimidating. While some aspects of statistics are complex and technical, it is possible for the average citizen to identify many common errors (for more I recommend the classic book How to Lie with Statistics). In this study there are several potentially misleading statistical choices. To focus just on two:

5. All of the results are based on percent change rather than absolute change. Percentage change can be misleading and uninformative. Compare the statements “the cost of lemonade at the neighborhood stand went up by 20%” vs. “the cost of lemonade went from 25 cents to 30 cents.” Importantly, without the actual values we can’t tell whether the changes would be clinically meaningful or even noticeable to the average person.

6. Many differences are reported without making direct comparisons. In several places, the authors note percent changes are less for placebo than Prevagen but don’t emphasize that the difference between the groups are not statistically significant. This means they did not demonstrate that Prevagen outperforms placebo.  They also don’t provide measures of variability (e.g. standard deviations or confidence intervals) that would allow readers to see how much overlap there is between Prevagen and placebo.

CONCLUSION

Putting it all together using the clinical guideline process, we are left at the end of the day not with “clinically proven to improve memory” (or, per the package “healthy brain function”, “sharper mind” and “clearer thinking”), but the following:

For people age 40-90 with normal memory, Prevagen taken daily for 90 days is possibly effective for improving some aspects of memory. This statement is based on a single paper with a moderately high risk of bias.

Clearly, a statement like this would lower sales but shouldn’t people know the facts about what they are buying?

Take Home Points

1. There are methods to grade research evidence based on quality and risk of bias.

2. Applying these methods can give you a more realistic picture of what a product does, how likely it is to work, and how much to trust the data.

Citations

1. Moran DL, Underwood MY, Gabourie TA, Lerner KC. Effects of a Supplement Containing Apoaequorin on Verbal Learning in Older Adults in the Community. Adv Mind Body Med 2016;30:4-11.