New study on quality of health information on the web sheds light on methodological problems

Department of Health Policy, Management and Evaluation,
University of Toronto

Despite frequent research and media reports that health information on the web does not comply to expert guidelines, there is little reason to discourage consumers to obtain health information from the web, according to a study published in May 22/29 issue of the Journal of the American Medical Association (JAMA, 2002; 287: 2691-2700). The study authors criticise previous studies and media reports, suggesting that up to 90 per cent of health information on the web is “inaccurate”. “Obviously, the closer you look, the more flaws you find in health information on the web, but the same is true for health information in other media. I would encourage consumers to obtain as health information from the web, although they should keep a critical eye on quality,” the lead author, Dr. Gunther Eysenbach, Senior Scientist at the Centre for Global eHealth Innovation at Toronto General Hospital and Associate Professor at the University of Toronto. “Medical experts have higher expectations of what a website should provide than consumers do, and our study shows that the conclusion of the studies is depended on the methodology used. The higher the expectation of the expert, the greater the amount of information is found wrong,” explains Eysenbach.

The “quality” of health information has many facets, including that it should be accurate, readable, accessible, contain disclosures of authorship, ownership, target audience, sponsors, intention of the site, date and frequency of updates, and provide references and links to further information.

In order to get a comprehensive overview of the quality of health information on the web, Eysenbach and colleagues performed meta-analysis, systematically reviewing all research studies in which medical experts have critically reviewed health information on the web and pooling the results from 79 of such studies. It was noted that the majority of previous researchers concluded that health information on the web is a problem, warning consumers from obtaining health information from the web. However, Eysenbach and colleagues found that study authors did so mainly on grounds of criticising insufficient completeness of information given on a website, failure to disclose certain information, the difficulty to find relevant information on the web, and that information on the web often written on a higher reading level than printed material, excluding less well educated users.

The authors note that “completeness” of a web page or website—often evaluated as implicit part of “accuracy”—is not an appropriate evaluation criterion, as websites may deliberately and with good reason focus on a single topic in-depth rather than aiming for comprehensiveness, and consumers will usually search across different websites when looking for specific health information.

Unlike traditional media, where the stream of information is filtered “upstream” by editorial control, the web requires the filtering tasks be performed by those “downstream”, at the consumer end. Eysenbach sees the key challenge in developing technological and educational interventions to guide consumers to the best available information on the web. Eysenbach recommends that consumers should learn appropriate search techniques. For example, search engine results can be improved by entering appropriate search terms, which lead to high quality sites rather than quackery sites. Eysenbach recommends that patients should choose Internet savvy physicians and ask them for a “information prescription”, i.e., letting them write down reliable websites and appropriate search terms for information about their condition and treatment.

Eysenbach et al have found little evidence suggesting that information on the web is less accurate than information in other media. For example, an early study about the quality of information on television also concluded that 30 per cent of information was incorrect. Moreover, the risk to encounter “bad” health information also heavily depends on the subject area, and it heavily depends on how consumers search for information. “On the web, you get what you look for. If you enter ‘quick and natural cure for cancer’ into a search engine you shouldn’t be surprised if most of the information is dubious,” says Eysenbach.

Once consumers have encountered a website, Eysenbach recommends search for the imprint and disclosure information, making sure about who is behind the website, what is the motivation and intention of the site owner, who are the authors, what is their qualification, and when the site was last updated. Eysenbach also recommends to look into who is linking to this site and how many other sites are linking to the site in question. Such information can be obtained from search engines such as Google. “If a site claims to be the ‘National Vaccine Information Center’, but no-one links to this site, especially no organisations you trust, this could be an indication for a suspicious site.”

In terms of lack of disclosures Eysenbach concludes that while many websites still have to do a better job in fully disclosing certain information such as when the content was created or what the physician’s credentials are, the meta-analysis showed that only 1 per cent of websites did not disclose the owner of the site, demonstrating that the consumer has at least always the possibility to check who behind the site is. “It is too difficult and time-consuming for the user to gather bits and pieces of this disclosure information from various web pages. We propose a standard for how this information should be provided,” says Eysenbach, who works on various projects proposing such standards. Previous research by Dr Eysenbach published earlier this year in the British Medical Journal (BMJ 2002; 324: 573-577), observed how consumers searched and appraised health information on the web, and demonstrated that consumers still care too little about disclosure statements and who is behind the website. Eysenbach is convinced that client-side software should be developed that support consumers in appraising and filtering information according to their needs.

In terms of readability Eysenbach points out that the methods used to assess readability are all based on evaluating just sentence length and structure using so-called readability formulas. “The web enables the use of graphics, audio and video, to boost the ability of the consumer to understand complex health issues, but multimedia is not taken into account by readability formulae,” says Eysenbach. “We need better methods to measure how consumers comprehend electronic information, and we need new quality criteria for the comprehensibility of multimedia information. This requires actual consumers to be involved in this type of research.”

Eysenbach advises users to shop around on multiple sites. “Our study showed that many medical experts who have evaluated health content critically noted that a single website rarely covers a complete picture of an health issue.” According to Eysenbach, the beauty of the web is that you don’t have to rely on a single information sheet or one opinion, “make sure that you gather information from multiple sites and then discuss the options with your doctor.”

Eysenbach is also working on technological innovations to guide consumers to high quality information on the web. “In an situation where you have jewels and junk laying side by side, innovative computer-based methods to guide consumers to the best available evidence become critical”, says Eysenbach. “After all, the web is an electronic medium, opening more possibilities than traditional media.” His work is in line with the “semantic web” vision of the World-Wide-Web Consortium at MIT. Its director, Tim-Berners Lee, explains the rationale for working towards a semantic web as follows: “The Web can reach its full potential only if it becomes a place where data can be shared and processed by automated tools as well as by people.”

As co-ordinator of the MedCERTAIN and MedCIRCLE semantic web projects, funded by the European Union under its Action Plan for Safer Use of the Internet, Eysenbach intends to make use of the possibilities the semantic web is offering and works on the creation of semi-automatic methods for consumers to assess the credibility of a website. As a first step, in the MedCERTAIN project, Eysenbach and colleagues have developed a machine-readable vocabulary which allows health publishers to make disclosure statements in a standardized, machine readable way. This enables consumers on the web to use intelligent software “agents” which automatically gather information from the web, taking into account 1) preferences entered by the user, 2) what providers say about themselves and 3) what third parties (e.g. portal sites) say about the site.

To support this development, Eysenbach also hopes for more research evidence on which quality criteria predict understanding, behaviour or attitude change on the part of consumers, and eventually lead to improvement of public health. “We need more research evidence to study how different quality criteria are related to each other, and what effect they have on consumers.”

Instead of attempting to estimate the prevalence of imperfect health information on the web and “getting bogged down bogged down by the question of how much information is inaccurate. ” Eysenbach and colleagues recommend that researchers should harness the power of the Internet as the most comprehensive electronic archive of written material representing our world and peoples’ opinions, concerns and desires. Physicians who surf the Internet for the first time are often stunned from what they learn from websites set up by lay people or patient self-support communities. “Material on these venues can be rich source for researchers interested in understanding experiences and views of people and patients,” says Eysenbach. “It presents a unique but largely untapped opportunity for health researchers to analyze where and why gaps exist between evidence-based medicine and consumer behaviour and expectations. Such research may elicit a wealth of valuable data that may inform priorities for research, health communication, and education.“

Source: Empirical studies assessing the quality of health information for consumers on the World Wide Web: A systematic review. Gunther Eysenbach, John Powell, Oliver Kuss, Eunryoung Sa. JAMA, 2002; 287: 2691-2700

Abstract: http://jama.ama-assn.org/issues/v287n20/abs/jrv10005.html