Research has cautioned against healthcare professionals downloading apps based on their number of downloads or user ratings
Ulster University research has found that user ratings and download rates are an unreliable
indicator of quality in digital health apps.
Highlighting the need for a more nuanced understanding of app quality in the digital health
landscape, the study, titled Don’t Judge a Book or Health App by its Cover was led by
researchers at Ulster University’s School of Computing and delivered in partnership with the
Organisation for the Review of Care and Health Apps (ORCHA).
With over 350,000 healthcare apps available to download at the click of a button, covering
topics from diabetes, to sleep, to mental health, the study analysed data from 1,500 health
apps across various platforms, assessing their quality against user ratings and download
counts.
Implications
In an era where digital health apps are increasingly relied upon for managing various
aspects of health and wellness, and with consumers potentially swayed by popular
perception, the research team explored the disconnect between app popularity metrics and
safe, high-quality digital health apps and the resulting significant implications for healthcare
providers, policy-makers, and app developers.
Prof Raymond Bond, Professor of Human Computer Systems at Ulster University and Chair
of the PhD research project, said: “Our research underscores the importance of looking
beyond metrics like user ratings and the number of app downloads when evaluating digital
health apps. While these metrics may influence user perceptions, our research shows that
they do not necessarily reflect the quality, effectiveness, safety, or reliability of these
healthcare apps.”
Prof Bond stated: “We need to shift the focus from popularity to quality. Consumers deserve
access to high-quality digital health interventions that are evidence-based, user-friendly, and
designed to enhance health and wellbeing outcomes. Relying solely on user ratings and
download counts could lead to misguided decisions and, eventually, undermine public trust
in digital health technologies.”
Limitations
By recognising the limitations of traditional metrics and prioritising rigorous evaluation
processes, stakeholders, including healthcare professionals who recommend apps to
patients, can ensure the applications they are recommending meet the highest standards of
quality and contribute meaningfully to patient care and well-being, said the researchers.
Prof Maurice Mulvenna, Professor of Computer Science at Ulster University, added:
“As the digital health landscape continues to evolve, studies like these serve as a vital
reminder that appearances can be deceptive. In delving deeper into the substance of digital
health apps and by prioritising evidence-based evaluation, we can help consumers to make
informed decisions and harness the full potential of technology in advancing health outcomes.
“We are pleased to see our Ulster University-led research reach a broad audience that plays
a role in safeguarding customers and patients as they make use of the growing digital health
app marketplace.”
Conducted as part of a PhD programme in 2021-2024, the research examined a unique
dataset of over 1,500 digital health apps provided by ORCHA, assessing the level of
evidence behind the app, whether it is user-friendly and accessible, and areas including data
protection and privacy.
Crucial issue
Ms Liz Ashall-Payne, Founding CEO of ORCHA, commented: “Our collaboration with Ulster
University has shed light on a crucial issue in the digital health landscape. User ratings and
download numbers are not reliable indicators of an app’s quality. This research emphasises
the need for comprehensive evaluation to ensure that digital health apps meet the highest
standards of safety, effectiveness, and reliability. By prioritising rigorous assessments, we
can guide healthcare providers and consumers toward better choices, ultimately improving
health outcomes and building trust in digital health solutions.”
Maciej Hyzy, PhD Researcher at Ulster University, added: “Taking part in Ulster University’s
PhD programme and collaborating with ORCHA gave me an opportunity to be part of
important research in the area of digital health. As a result of this collaboration and research
we increased our understanding of digital health apps by analysing their assessment data.
Research like this shows us that we need to rethink metrics by which we judge digital health
apps.”
In the study, the authors observed: “Our results suggest that the cause of this gap may be
that health experts look for evidence of clinical quality, utility, privacy, and security that is
not considered by users when they rate apps on the iOS and Android app stores. Moreover,
users who get their health information, from the internet, rely on search engine results, that
may come from unaccredited sources… This indicates that a trusted objective way to judge
the quality of health apps is needed.”
The study, which was published in the journal PLOS ONE, (https://doi.org/10.1371/journal.pone.0298977), was jointly funded by ORCHA and the Department for the Economy as part of the Co-operative Awards in Science and Technology
(CAST) studentship.
