Methods of usability testing in the development of eHealth applications: A scoping review.

Abstract:

BACKGROUND:The number of eHealth applications has exponentially increased in recent years, with over 325,000 health apps now available on all major app stores. This is in addition to other eHealth applications available on other platforms such as PC software, web sites and even gaming consoles. As with other digital applications, usability is one of the key factors in the successful implementation of eHealth apps. Reviews of the literature on empirical methods of usability testing in eHealth were last published in 2015. In the context of an exponentially increasing rate of App development year on year, an updated review is warranted. OBJECTIVE:To identify, explore, and summarize the current methods used in the usability testing of eHealth applications. METHODS:A scoping review was conducted on literature available from April 2014 up to October 2017. Four databases were searched. Literature was considered for inclusion if it was (1) focused on an eHealth application (which includes websites, PC software, smartphone and tablet applications), (2) provided information about usability of the application, (3) provided empirical results of the usability testing, (4) a full or short paper (not an abstract) published in English after March 2014. We then extracted data pertaining to the usability evaluation processes described in the selected studies. RESULTS:133 articles met the inclusion criteria. The methods used for usability testing, in decreasing order of frequency were: questionnaires (n = 105), task completion (n = 57), 'Think-Aloud' (n = 45), interviews (n = 37), heuristic testing (n = 18) and focus groups (n = 13). Majority of the studies used one (n = 45) or two (n = 46) methods of testing. The rest used a combination of three (n = 30) or four (n = 12) methods of testing usability. None of the studies used automated mechanisms to test usability. The System Usability Scale (SUS) was the most frequently used questionnaire (n = 44). The ten most frequent health conditions or diseases where eHealth apps were being evaluated for usability were the following: mental health (n = 12), cancer (n = 10), nutrition (n = 10), child health (n = 9), diabetes (n = 9), telemedicine (n = 8), cardiovascular disease (n = 6), HIV (n = 4), health information systems (n = 4) and smoking (n = 4). Further iterations of the app were reported in a minority of the studies (n = 41). The use of the 'Think-Aloud' (Pearson Chi-squared test: χ2 = 11.15, p < 0.05) and heuristic walkthrough (Pearson Chi-squared test: χ2 = 4.48, p < 0.05) were significantly associated with at least one further iteration of the app being developed. CONCLUSION:Although there has been an exponential increase in the number of eHealth apps, the number of studies that have been published that report the results of usability testing on these apps has not increased at an equivalent rate. The number of digital health applications that publish their usability evaluation results remains only a small fraction. Questionnaires are the most prevalent method of evaluating usability in eHealth applications, which provide an overall measure of usability but do not pinpoint the problems that need to be addressed. Qualitative methods may be more useful in this regard. The use of multiple evaluation methods has increased. Automated methods such as eye tracking have not gained traction in evaluating health apps. Further research is needed into which methods are best suited for the different types of eHealth applications, according to their target users and the health conditions being addressed.

journal_name

Int J Med Inform

authors

Maramba I,Chatterjee A,Newman C

doi

10.1016/j.ijmedinf.2019.03.018

subject

Has Abstract

pub_date

2019-06-01 00:00:00

pages

95-104

eissn

1386-5056

issn

1872-8243

pii

S1386-5056(18)31318-2

journal_volume

126

pub_type

杂志文章,评审
  • Twenty five years of medical informatics education at Heidelberg/Heilbronn: discussion of a specialized curriculum for medical informatics.

    abstract::The specialized university curriculum for medical informatics (MI) at the University of Heidelberg/School of Technology Heilbronn is one of the oldest educational approaches in the field of MI and has been successful now for 25 years with about 1000 graduates (Diplom-Informatikerin der Medizin or Diplom-Informatiker d...

    journal_title:International journal of medical informatics

    pub_type: 历史文章,杂志文章

    doi:10.1016/s1386-5056(98)00048-3

    authors: Leven FJ,Haux R

    更新日期:1998-06-01 00:00:00

  • Managing a mixed-registration-type appointment system in outpatient clinics.

    abstract:INTRODUCTION:Improving outpatient resource utilization significantly enhances the efficiency of healthcare organizations. Substantial number of walk-in patients (average of 72% in our study) to outpatient services is a universal characteristic of Taiwan's healthcare organizations. Consequently, scheduling becomes extre...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/s1386-5056(03)00008-x

    authors: Su S,Shih CL

    更新日期:2003-04-01 00:00:00

  • Present and future trends with NLP.

    abstract::This paper reflects some desiderata on the role of Natural Language Processing (NLP) in the coming years as foreseen in the medical domain. During the Medical Informatics Europe (MIE) conference in 1997, the NLP track was composed by numerous papers on natural language, knowledge representation, nomenclatures and clas...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/s1386-5056(98)00132-4

    authors: Baud R

    更新日期:1998-10-01 00:00:00

  • Comparative cost-benefit analysis of tele-homecare for community-dwelling elderly in Japan: Non-Government versus Government Supported Funding Models.

    abstract:OBJECTIVE:Tele-homecare is gaining prominence as a viable care alternative, as evidenced by the increase in financial support from international governments to fund initiatives in their respective countries. The primary reason for the funding is to support efforts to reduce lags and increase capacity in access to care ...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2017.04.017

    authors: Akiyama M,Abraham C

    更新日期:2017-08-01 00:00:00

  • Active learning reduces annotation time for clinical concept extraction.

    abstract:OBJECTIVE:To investigate: (1) the annotation time savings by various active learning query strategies compared to supervised learning and a random sampling baseline, and (2) the benefits of active learning-assisted pre-annotations in accelerating the manual annotation process compared to de novo annotation. MATERIALS ...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2017.08.001

    authors: Kholghi M,Sitbon L,Zuccon G,Nguyen A

    更新日期:2017-10-01 00:00:00

  • Communications and EHR: authenticating who's who is vital.

    abstract::It is becoming increasingly essential for individuals to be able uniquely to identify themselves and each other, even where they may never meet. Access to and use of privileges of membership, whether of a professional association, computer network or health insurance policy depends upon the 'members' being able to pro...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/s1386-5056(00)00119-2

    authors: Neame R

    更新日期:2000-11-01 00:00:00

  • A deep learning solution to recommend laboratory reduction strategies in ICU.

    abstract:OBJECTIVE:To build a machine-learning model that predicts laboratory test results and provides a promising lab test reduction strategy, using spatial-temporal correlations. MATERIALS AND METHODS:We developed a global prediction model to treat laboratory testing as a series of decisions by considering contextual inform...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2020.104282

    authors: Yu L,Li L,Bernstam E,Jiang X

    更新日期:2020-12-01 00:00:00

  • Adding insight: a qualitative cross-site study of physician order entry.

    abstract::The research questions, strategies, and results of a 7-year qualitative study of computerized physician order entry implementation (CPOE) at successful sites are reviewed over time. The iterative nature of qualitative inquiry stimulates a consecutive stream of research foci, which, with each iteration, add further ins...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章,多中心研究

    doi:10.1016/j.ijmedinf.2005.05.005

    authors: Ash JS,Sittig DF,Seshadri V,Dykstra RH,Carpenter JD,Stavri PZ

    更新日期:2005-08-01 00:00:00

  • SHARE road map for HealthGrids: methodology.

    abstract::The SHARE(1) project (http://www.eu-share.org) was asked to identify the key developments needed to achieve wide adoption and deployment of HealthGrids throughout Europe. The project was asked to organise these as milestones on a road map, so that all technical advances, social actions, economic investments and ethica...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2008.10.003

    authors: Olive M,Rahmouni H,Solomonides T,Breton V,Legré Y,Blanquer I,Hernandez V

    更新日期:2009-04-01 00:00:00

  • The behavioral side of information technology.

    abstract::Exposure to technologies is becoming more prevalent in healthcare technologies. Few strategies have been developed that work reliably to successfully implement information technologies. Information technology enables, but does not guarantee organizational change. A theoretical model has been developed identifying seve...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/s1386-5056(99)00037-4

    authors: Dixon DR

    更新日期:1999-12-01 00:00:00

  • Distributed health informatics graduate education for working professionals.

    abstract::This paper describes a new distributed Master of Science degree program stream in health informatics (HI) at the University of Victoria, Canada, aimed at healthcare and information technology professionals working in the health setting. This is a part-time graduate program where students are required to complete 10 co...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章,评审

    doi:10.1016/j.ijmedinf.2007.01.008

    authors: Lau F

    更新日期:2007-05-01 00:00:00

  • The impact of clinical leadership on health information technology adoption: systematic review.

    abstract:PURPOSE:To conduct a systematic review to examine evidence of associations between clinical leadership and successful information technology (IT) adoption in healthcare organisations. METHODS:We searched Medline, Embase, Cinahl, and Business Source Premier for articles published between January 2000 to May 2013 with k...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章,评审

    doi:10.1016/j.ijmedinf.2014.02.005

    authors: Ingebrigtsen T,Georgiou A,Clay-Williams R,Magrabi F,Hordern A,Prgomet M,Li J,Westbrook J,Braithwaite J

    更新日期:2014-06-01 00:00:00

  • Smoothing dense spaces for improved relation extraction between drugs and adverse reactions.

    abstract:BACKGROUND AND OBJECTIVE:This work aims at extracting Adverse Drug Reactions (ADRs), i.e. a harm directly caused by a drug at normal doses, from Electronic Health Records (EHRs). The lack of readily available EHRs because of confidentiality issues and their lexical variability make the ADR extraction challenging. Furth...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2019.05.009

    authors: Santiso S,Pérez A,Casillas A

    更新日期:2019-08-01 00:00:00

  • Medical informatics at Heidelberg/Heilbronn: status-evaluation-new challenges in a specialised curriculum for medical informatics after thirty years of evolution.

    abstract::After reporting on characteristics, structure and contents of the specialised informatics-based curriculum for medical informatics (MI) at the University of Heidelberg/University of Applied Sciences Heilbronn, the paper describes the development during the last 5 years, and in particular a complementary health care or...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2003.11.011

    authors: Leven FJ,Knaup P,Schmidt D,Wetter T

    更新日期:2004-03-18 00:00:00

  • Policy for cryptography in healthcare--a view from the NHS.

    abstract::Effective security arrangements, which both protect and assure those information assets of healthcare providers, doctors and patients, are fundamental requirements in a modern electronic healthcare culture. At the heart of healthcare information systems in future will be those infrastructure components and services, w...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/s1386-5056(00)00109-x

    authors: Donaldson A

    更新日期:2000-11-01 00:00:00

  • Feasibility of mobile health game "Fume" in supporting tobacco-related health literacy among early adolescents: A three-armed cluster randomized design.

    abstract:OBJECTIVE:New interventions supporting health literacy and a tobacco-free lifestyle in adolescence are needed to narrow the widening gap in existing health inequalities. Health games offer potential and could be utilized for example in school healthcare, but more research is needed to increase the understanding of the ...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章,随机对照试验

    doi:10.1016/j.ijmedinf.2018.02.013

    authors: Parisod H,Pakarinen A,Axelin A,Löyttyniemi E,Smed J,Salanterä S

    更新日期:2018-05-01 00:00:00

  • Designing web services in health information systems: from process to application level.

    abstract::Service-oriented architectures (SOAs) and web service technologies have been proposed to respond to some central interoperability challenges of heterogeneous health information systems (HIS). We propose a model which we are using to define services and solutions for healthcare applications from the requirements in the...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2006.10.001

    authors: Mykkänen J,Riekkinen A,Sormunen M,Karhunen H,Laitinen P

    更新日期:2007-02-01 00:00:00

  • Nurses' information management across complex health care organizations.

    abstract:UNLABELLED:The purpose of this study was to describe the information management used by hospital and home care nurses for patients in need of continuing care after an episode of hospitalization. METHOD:A prospective descriptive design was used. In total 287 hospital nurses and 220 home care nurses were asked to comple...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2005.07.010

    authors: Hellesø R,Sorensen L,Lorensen M

    更新日期:2005-12-01 00:00:00

  • Commitment of project participants to developing health care services based on the internet technology.

    abstract:BACKGROUND:When developing Information and Communication Technology (ICT), such as services for the decision-making process, skilled health care professionals with their comprehensive knowledge of patients/clients are essential contributors to the project. Careful evaluation is needed to assess the effectiveness of pro...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2005.07.002

    authors: Kouri P,Karjalainen-Jurvelin R,Kinnunen J

    更新日期:2005-12-01 00:00:00

  • H_pe for mHealth: more "y" or "o" on the horizon?

    abstract:OBJECTIVE:Efforts in the domain of mobile health, or mHealth, have been criticized for the unfettered proliferation of pilots and a lack of a rigorous evidence base to support these strategies. In this letter, we present the response of a group of researchers in the mHealth community to the recent calls for evidence is...

    journal_title:International journal of medical informatics

    pub_type: 信件

    doi:10.1016/j.ijmedinf.2012.11.016

    authors: Labrique A,Vasudevan L,Chang LW,Mehl G

    更新日期:2013-05-01 00:00:00

  • Patients' perceptions of a home telecare system.

    abstract:GOAL:To identify any major factors that could affect patients' perceptions of a Home Telecare Management System (HTMS) and use the findings to contribute to development of a theoretical framework for patient acceptance of HTMS. MATERIALS AND METHODS:Ten Focus Group Interviews (FGIs) were conducted with patients suffer...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2007.10.006

    authors: Rahimpour M,Lovell NH,Celler BG,McCormick J

    更新日期:2008-07-01 00:00:00

  • Virtual reality for relearning daily living skills.

    abstract::The explosive increase in the power of computers has enabled the creation of fast, interactive 3D environments, sometimes called virtual reality (VR). This technology, often associated with arcade games, is increasingly being used for more serious applications. This paper describes research showing transfer of skills ...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/s1386-5056(00)00100-3

    authors: Gourlay D,Lun KC,Lee YN,Tay J

    更新日期:2000-12-01 00:00:00

  • Using a sociotechnical framework to understand adaptations in health IT implementation.

    abstract:PURPOSE:When barcode medication administration (BCMA) is implemented nurses are required to integrate not only a new set of procedures or artifacts into everyday work, but also an orientation to medication safety itself that is sometimes at odds with their own. This paper describes how the nurses' orientation (the Prac...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2013.01.009

    authors: Novak LL,Holden RJ,Anders SH,Hong JY,Karsh BT

    更新日期:2013-12-01 00:00:00

  • User-centred design for developing e-Health system for renal patients at home (AppNephro).

    abstract:BACKGROUND:Despite the promising benefits of the e-Health approaches (including provide technology-based healthcare services to anyone, anytime, and anywhere), few solutions are adopted in daily practice. User acceptance is one of the major obstacles that hinder the success of technology approaches. End-users often str...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章,多中心研究

    doi:10.1016/j.ijmedinf.2019.02.007

    authors: Calvillo-Arbizu J,Roa-Romero LM,Estudillo-Valderrama MA,Salgueira-Lazo M,Aresté-Fosalba N,Del-Castillo-Rodríguez NL,González-Cabrera F,Marrero-Robayna S,López-de-la-Manzana V,Román-Martínez I

    更新日期:2019-05-01 00:00:00

  • Heuristic evaluation and simulated use testing of infusion pumps to inform pump selection.

    abstract:BACKGROUND:Poorly designed infusion pumps can lead to user errors and adverse incidents. Therefore, assessments of their usability and performance that can inform managerial decisions about the selection of appropriate medical devices are essential. OBJECTIVE:This study aimed to identify design deficiencies and evalua...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2019.07.011

    authors: Liu K,Chan FY,Or CK,Sun DT,Lai WS,So HY

    更新日期:2019-11-01 00:00:00

  • Integration of a blood pressure controller and an infusion toolbox system using client-server technology.

    abstract::This paper shortly describes an Infusion Toolbox and a blood pressure (BP) control application. It explains how we applied an agent model and client-server technology to integrate them. We show that by using this framework and object oriented technologies the necessary changes to the existing applications are reduced....

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/s1386-5056(98)00093-8

    authors: Meijers RH,Coussaert EJ,Cantraine FR

    更新日期:1998-05-01 00:00:00

  • Standards for detailed clinical models as the basis for medical data exchange and decision support.

    abstract:INTRODUCTION:Detailed clinical models are necessary to exchange medical data between heterogeneous computer systems and to maintain consistency in a longitudinal electronic medical record system. At Intermountain Health Care (IHC), we have a history of designing detailed clinical models. The purpose of this paper is to...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/s1386-5056(02)00103-x

    authors: Coyle JF,Mori AR,Huff SM

    更新日期:2003-03-01 00:00:00

  • Co-authorship network analysis in cardiovascular research utilizing machine learning (2009-2019).

    abstract:BACKGROUND:With the recent advances in computational science, machine-learning methods have been increasingly used in medical research. Because such projects usually require both a clinician and a computational data scientist, there is a need for interdisciplinary research collaboration. However, there has been no publ...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2020.104274

    authors: Higaki A,Uetani T,Ikeda S,Yamaguchi O

    更新日期:2020-11-01 00:00:00

  • Evaluating e-consultation implementations based on use and time-line across various specialties.

    abstract:BACKGROUND:Electronic Consultation (e-consults) can provide improved access, enhance patient and provider satisfaction, and reduce beneficiary travel expenses. We explored how e-consults were implemented across three specialty areas, diabetes (Diab), gastroenterology (GI), and neurosurgery (Neuro), at two Veterans Affa...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2017.09.005

    authors: Parikh PJ,Mowrey C,Gallimore J,Harrell S,Burke B

    更新日期:2017-12-01 00:00:00

  • A novel concept for integrating and delivering health information using a comprehensive digital dashboard: An analysis of healthcare professionals' intention to adopt a new system and the trend of its real usage.

    abstract:OBJECTIVE:To introduce a new concept of medical dashboard system called BESTBoard. Such a system was implemented in all wards in a tertiary academic hospital to explore the development process, core designs, functions, usability and feasibility. METHODS:The task-force team made user interface designs for 6 months base...

    journal_title:International journal of medical informatics

    pub_type: 杂志文章

    doi:10.1016/j.ijmedinf.2016.10.001

    authors: Lee K,Jung SY,Hwang H,Yoo S,Baek HY,Baek RM,Kim S

    更新日期:2017-01-01 00:00:00