Skip to main content

Clinical vignettes in benchmarking the performance of online symptom checkers

In the USA, over one-third of adults self-diagnose their conditions using the internet, including queries about urgent (ie, chest pain) and non-urgent (ie, headache) symptoms. The main issue with self-diagnosing using websites such as Google and Yahoo is that user may get confusing or inaccurate information, and in the case of urgent symptoms, the user may not be aware of the need to seek emergency care. In recent years, various online symptom checkers (OSCs) based on algorithms or artificial intelligence (AI) have emerged to fill this gap

Online symptom checkers are calculators that ask users to input details about their symptoms of sickness, along with personal information such as gender and age. Using algorithms or AI, the symptom checkers propose a range of conditions that fit the symptoms the user experiences. Developers promote these digital tools as a way of saving time for patients, reducing anxiety and giving patients the opportunity to take control of their own health.

The diagnostic function of online symptom checkers is aimed at educating users on the range of possible conditions that may fit their symptoms. Further to presenting a condition outcome and giving the users a triage recommendation that prioritises their health needs, the triage function of online symptom checkers guides users on whether they should self-care for the condition they are describing or whether they should seek professional healthcare support.3 This added functionality could vastly enhance the usefulness of Online symptom checkers by alerting people about when they need to seek emergency support or seek non-emergency care for common or self-limiting conditions.

In a study published in the journal BMJ Open, we assessed the suitability of vignettes in benchmarking the performance of online symptom checkers. Our approach included providing the vignettes to an independent panel of single-blinded physicians to arrive at an alternative set of diagnostic and triage solutions. The secondary aim was to benchmark the safety of a popular online symptom checkers (Healthily) by measuring the extent that it provided the correct diagnosis and triage solutions to a standardised set of vignettes as defined by a panel of physicians.

We found significant variability of medical opinion depending on which group of GPs considered the vignette script, whereas consolidating the output of two independent GP roundtables (one from RCGP and another panel of panel of independent GPs) resulted in a more refined third iteration (the consolidated standard) which more accurately included the ‘correct’ diagnostic and triage solutions conferred by the vignette script. This was demonstrated by the significant extent that the performance of online symptom checkers improved when benchmarked between the original and final consolidated standards. 

The different qualities of the diagnostic and triage solutions between iterative standards suggest that vignettes are not an ideal tool for benchmarking the accuracy of online symptom checkers, since performance will always be related to the nature and order of the diagnostic and triage solutions which we have shown can differ significantly depending on the approach and levels of input from independent physicians. By extension, it is reasonable to propose that any consolidated standard for any vignette can always be improved by including a wider range of medical opinion until saturation is reached and a final consensus emerges.

The inherent limitations of clinical vignettes render them largely unsuitable for benchmarking the performance of popular online symptom checkers because the diagnosis and triage solutions assigned to each vignette script are amenable to change pending the deliberations of an independent panel of physicians. Although online symptom checkers are already working at a safe level of probable risk, further work is recommended to cross-validate the performance of online symptom checkers against real-world test case scenarios using real patient stories and interactions with GPs as opposed to using artificial vignettes only which will always be the single most important limitation to any cross-validation study.

Comments

Post a Comment

Popular posts from this blog

Protecting Against the "Quad-demic": Influenza, Covid-19, Norovirus and RSV

As the NHS braces for a challenging winter season, it is grappling with a "quad-demic" of health emergencies caused by influenza, Covid-19, norovirus, and respiratory syncytial virus (RSV). This confluence of viral threats poses a significant risk to public health in the UK as well as putting strain on healthcare resources, emphasising the importance of preventive measures to safeguard public health. Public health measures such a vaccination and good personal hygiene are pivotal in reducing the impact of these illnesses, particularly for vulnerable groups. The Four Viruses: What Are They? Influenza: A highly contagious respiratory infection that causes significant illness each winter. It can lead to severe complications, particularly in the elderly, young children, pregnant women, and those with chronic health conditions. Covid-19: Though its most acute phase has passed, Covid-19 remains a concern, especially as new variants of SA...
Post a Comment
Read more

MPH Student Presentations on the NHS Care.Data Programme

As part of a session on primary care data in the Health Informatics module on the Imperial Master of Public Health Programme, I asked students to work in two groups to present arguments for and against the NHS Care.Data programme. Care.Data is an NHS programme that will extract data from the medical records held by general practitioners (GPs) in England. The Care.Data programme takes advantage of the very high level of use of electronic medical records by GPs in England. After extraction, data will be uploaded to the NHS Health and Social Care Information Centre (HSCIC). The data will then be used for functions such as health care planning, monitoring disease patterns and research. The programme has been controversial with proponents arguing that the programme will bring many benefits for the NHS and the population of England; and opponents arguing it is a major breach of privacy. You can view the two presentations to help inform you further about these arguments: Arguments fo...
6 comments
Read more

How can we work successfully across the health and care system to make a success of Pharmacy First?

Pharmacies in England to begin treating patients for seven common conditions. How can we work successfully across the health and care system to make a success of Pharmacy First? 1. The Pharmacy First scheme aims to provide convenient access to healthcare through community pharmacies. Patients with minor ailments or common conditions can seek advice and treatment directly from their local pharmacy instead of visiting a general practice, urgent care centre or emergency department. The conditions covered by the scheme may vary depending on local funding arrangements and participation of pharmacies.  2, A potential problem with Pharmacy First is pharmacists misdiagnosing a patient's condition. It may also lead to delays in patients seeing doctors when medical assessment is needed. To mitigate these risks, appropriate safeguards and referral pathways should be established, ensuring timely medical assessment when necessary. The scheme will also increase the workload of pharmacies, thereb...
Post a Comment
Read more