Comparison of confidence interval methods for an intra-class correlation coefficient (ICC).

Abstract:

BACKGROUND:The intraclass correlation coefficient (ICC) is widely used in biomedical research to assess the reproducibility of measurements between raters, labs, technicians, or devices. For example, in an inter-rater reliability study, a high ICC value means that noise variability (between-raters and within-raters) is small relative to variability from patient to patient. A confidence interval or Bayesian credible interval for the ICC is a commonly reported summary. Such intervals can be constructed employing either frequentist or Bayesian methodologies. METHODS:This study examines the performance of three different methods for constructing an interval in a two-way, crossed, random effects model without interaction: the Generalized Confidence Interval method (GCI), the Modified Large Sample method (MLS), and a Bayesian method based on a noninformative prior distribution (NIB). Guidance is provided on interval construction method selection based on study design, sample size, and normality of the data. We compare the coverage probabilities and widths of the different interval methods. RESULTS:We show that, for the two-way, crossed, random effects model without interaction, care is needed in interval method selection because the interval estimates do not always have properties that the user expects. While different methods generally perform well when there are a large number of levels of each factor, large differences between the methods emerge when the number of one or more factors is limited. In addition, all methods are shown to lack robustness to certain hard-to-detect violations of normality when the sample size is limited. CONCLUSIONS:Decision rules and software programs for interval construction are provided for practical implementation in the two-way, crossed, random effects model without interaction. All interval methods perform similarly when the data are normal and there are sufficient numbers of levels of each factor. The MLS and GCI methods outperform the NIB when one of the factors has a limited number of levels and the data are normally distributed or nearly normally distributed. None of the methods work well if the number of levels of a factor are limited and data are markedly non-normal. The software programs are implemented in the popular R language.

journal_name

BMC Med Res Methodol

authors

Ionan AC,Polley MY,McShane LM,Dobbin KK

doi

10.1186/1471-2288-14-121

subject

Has Abstract

pub_date

2014-11-22 00:00:00

pages

121

issn

1471-2288

pii

1471-2288-14-121

journal_volume

14

pub_type

杂志文章
  • Likelihood-based random-effects meta-analysis with few studies: empirical and simulation studies.

    abstract:BACKGROUND:Standard random-effects meta-analysis methods perform poorly when applied to few studies only. Such settings however are commonly encountered in practice. It is unclear, whether or to what extent small-sample-size behaviour can be improved by more sophisticated modeling. METHODS:We consider likelihood-based...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-018-0618-3

    authors: Seide SE,Röver C,Friede T

    更新日期:2019-01-11 00:00:00

  • Psychometric properties of an English short version of the Trier Inventory for chronic Stress.

    abstract:BACKGROUND:Although a variety of instruments are available that capture stress experience, the assessment of chronic stress has been hindered by the lack of economical screening instruments. Recently, an English-language version of the Trier Inventory for Chronic Stress (TICS-EN) consisting of 57 items according to a s...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-020-01156-y

    authors: Petrowski K,Braehler E,Schmalbach B,Hinz A,Bastianon C,Ritz T

    更新日期:2020-12-16 00:00:00

  • Pan-Canadian assessment of pandemic immunization data collection: study methodology.

    abstract:BACKGROUND:The collection of individual-level pandemic (H1N1) 2009 influenza immunization data was considered important to facilitate optimal vaccine delivery and accurate assessment of vaccine coverage. These data are also critical for research aimed at evaluating the new vaccine's safety and effectiveness. Systems us...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-10-51

    authors: Pereira JA,Quach S,Heidebrecht C,Foisy J,Quan S,Finkelstein M,Sikora CA,Bettinger JA,Buckeridge DL,McCarthy A,Deeks S,Kwong JC,Public Health Agency of Canada\/Canadian Institutes of Health Research Influenza Research Network Vacc

    更新日期:2010-06-08 00:00:00

  • The thresholds for statistical and clinical significance - a five-step procedure for evaluation of intervention effects in randomised clinical trials.

    abstract:BACKGROUND:Thresholds for statistical significance are insufficiently demonstrated by 95% confidence intervals or P-values when assessing results from randomised clinical trials. First, a P-value only shows the probability of getting a result assuming that the null hypothesis is true and does not reflect the probabilit...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-14-34

    authors: Jakobsen JC,Gluud C,Winkel P,Lange T,Wetterslev J

    更新日期:2014-03-04 00:00:00

  • Heterogeneity and event dependence in the analysis of sickness absence.

    abstract:BACKGROUND:Sickness absence (SA) is an important social, economic and public health issue. Identifying and understanding the determinants, whether biological, regulatory or, health services-related, of variability in SA duration is essential for better management of SA. The conditional frailty model (CFM) is useful whe...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-13-114

    authors: Torá-Rocamora I,Gimeno D,Delclos G,Benavides FG,Manzanera R,Jardí J,Alberti C,Yasui Y,Martínez JM

    更新日期:2013-09-16 00:00:00

  • Randomised controlled comparison of the Health Survey Short Form (SF-12) and the Graded Chronic Pain Scale (GCPS) in telephone interviews versus self-administered questionnaires. Are the results equivalent?

    abstract:BACKGROUND:The most commonly used survey methods are self-administered questionnaires, telephone interviews, and a mixture of both. But until now evidence out of randomised controlled trials as to whether patient responses differ depending on the survey mode is lacking. Therefore this study assessed whether patient res...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章,随机对照试验

    doi:10.1186/1471-2288-7-50

    authors: Lungenhausen M,Lange S,Maier C,Schaub C,Trampisch HJ,Endres HG

    更新日期:2007-11-22 00:00:00

  • Suitability of administrative claims databases for bariatric surgery research - is the glass half-full or half-empty?

    abstract:BACKGROUND:Claims databases are generally considered inadequate for obesity research due to suboptimal capture of body mass index (BMI) measurements. This might not be true for bariatric surgery because of reimbursement requirements and changes in coding systems. We assessed the availability and validity of claims-base...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-020-01106-8

    authors: Li X,Lewis KH,Callaway K,Wharam JF,Toh S

    更新日期:2020-09-07 00:00:00

  • Does updating improve the methodological and reporting quality of systematic reviews?

    abstract:BACKGROUND:Systematic reviews (SRs) must be of high quality. The purpose of our research was to compare the methodological and reporting quality of original versus updated Cochrane SRs to determine whether updating had improved these two quality dimensions. METHODS:We identified updated Cochrane SRs published in issue...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-6-27

    authors: Shea B,Boers M,Grimshaw JM,Hamel C,Bouter LM

    更新日期:2006-06-13 00:00:00

  • Eliciting parental support for the use of newborn blood spots for pediatric research.

    abstract:BACKGROUND:Biomarkers of exposures such as infection or environmental chemicals can be measured in small volumes of blood extracted from newborn dried blood spots (DBS) underscoring their potential utility for population-based research. However, few studies have evaluated the feasibility and utility of this resource; p...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-016-0120-8

    authors: Yeung EH,Louis GB,Lawrence D,Kannan K,McLain AC,Caggana M,Druschel C,Bell E

    更新日期:2016-02-04 00:00:00

  • Positive predictive value of a case definition for diabetes mellitus using automated administrative health data in children and youth exposed to antipsychotic drugs or control medications: a Tennessee Medicaid study.

    abstract:BACKGROUND:We developed and validated an automated database case definition for diabetes in children and youth to facilitate pharmacoepidemiologic investigations of medications and the risk of diabetes. METHODS:The present study was part of an in-progress retrospective cohort study of antipsychotics and diabetes in Te...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-12-128

    authors: Bobo WV,Cooper WO,Stein CM,Olfson M,Mounsey J,Daugherty J,Ray WA

    更新日期:2012-08-24 00:00:00

  • Using a social marketing framework to evaluate recruitment of a prospective study of genetic counseling and testing for the deaf community.

    abstract:BACKGROUND:Recruiting deaf and hard-of-hearing participants, particularly sign language-users, for genetics health service research is challenging due to communication barriers, mistrust toward genetics, and researchers' unfamiliarity with deaf people. Feelings of social exclusion and lack of social cohesion between re...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-13-145

    authors: Kobayashi Y,Boudreault P,Hill K,Sinsheimer JS,Palmer CG

    更新日期:2013-11-25 00:00:00

  • A factorial cluster-randomised controlled trial combining home-environmental and early child development interventions to improve child health and development: rationale, trial design and baseline findings.

    abstract:BACKGROUND:Exposure to unhealthy environments and inadequate child stimulation are main risk factors that affect children's health and wellbeing in low- and middle-income countries. Interventions that simultaneously address several risk factors at the household level have great potential to reduce these negative effect...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-020-00950-y

    authors: Hartinger SM,Nuño N,Hattendorf J,Verastegui H,Karlen W,Ortiz M,Mäusezahl D

    更新日期:2020-04-02 00:00:00

  • A proof of principle for using adaptive testing in Routine Outcome Monitoring: the efficiency of the Mood and Anxiety Symptoms Questionnaire -Anhedonic Depression CAT.

    abstract:BACKGROUND:In Routine Outcome Monitoring (ROM) there is a high demand for short assessments. Computerized Adaptive Testing (CAT) is a promising method for efficient assessment. In this article, the efficiency of a CAT version of the Mood and Anxiety Symptom Questionnaire, - Anhedonic Depression scale (MASQ-AD) for use ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-12-4

    authors: Smits N,Zitman FG,Cuijpers P,den Hollander-Gijsman ME,Carlier IV

    更新日期:2012-01-10 00:00:00

  • Methods to increase response rates to a population-based maternity survey: a comparison of two pilot studies.

    abstract:BACKGROUND:Surveys are established methods for collecting population data that are unavailable from other sources; however, response rates to surveys are declining. A number of methods have been identified to increase survey returns yet response rates remain low. This paper evaluates the impact of five selected methods...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-019-0702-3

    authors: Harrison S,Henderson J,Alderdice F,Quigley MA

    更新日期:2019-03-20 00:00:00

  • On the censored cost-effectiveness analysis using copula information.

    abstract:BACKGROUND:Information and theory beyond copula concepts are essential to understand the dependence relationship between several marginal covariates distributions. In a therapeutic trial data scheme, most of the time, censoring occurs. That could lead to a biased interpretation of the dependence relationship between ma...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-017-0305-9

    authors: Fontaine C,Daurès JP,Landais P

    更新日期:2017-02-15 00:00:00

  • Reliability, validity and administrative burden of the community reintegration of injured service members computer adaptive test (CRIS-CAT)".

    abstract:BACKGROUND:The Computer Adaptive Test version of the Community Reintegration of Injured Service Members measure (CRIS-CAT) consists of three scales measuring Extent of, Perceived Limitations in, and Satisfaction with community integration. The CRIS-CAT was developed using item response theory methods. The purposes of t...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-12-145

    authors: Resnik L,Borgia M,Ni P,Pirraglia PA,Jette A

    更新日期:2012-09-17 00:00:00

  • Evaluating treatments in health care: the instability of a one-legged stool.

    abstract:BACKGROUND:Both scientists and the public routinely refer to randomized controlled trials (RCTs) as being the 'gold standard' of scientific evidence. Although there is no question that placebo-controlled RCTs play a significant role in the evaluation of new pharmaceutical treatments, especially when it is important to ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-11-65

    authors: Kaplan BJ,Giesbrecht G,Shannon S,McLeod K

    更新日期:2011-05-11 00:00:00

  • Use of re-randomized data in meta-analysis.

    abstract:BACKGROUND:Outcomes collected in randomized clinical trials are observations of random variables that should be independent and identically distributed. However, in some trials, the patients are randomized more than once thus violating both of these assumptions. The probability of an event is not always the same when a...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-5-17

    authors: Hozo I,Djulbegovic B,Clark O,Lyman GH

    更新日期:2005-05-10 00:00:00

  • Comparison of sample characteristics in two pregnancy cohorts: community-based versus population-based recruitment methods.

    abstract:BACKGROUND:One of the biggest challenges for population health studies is the recruitment of participants. Questions that investigators have asked are "who volunteers for studies?" and "does recruitment method influence characteristics of the samples?" The purpose of this paper was to compare sample characteristics of ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-13-149

    authors: Leung BM,McDonald SW,Kaplan BJ,Giesbrecht GF,Tough SC

    更新日期:2013-12-06 00:00:00

  • Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines.

    abstract:BACKGROUND:Multiple imputation (MI) provides an effective approach to handle missing covariate data within prognostic modelling studies, as it can properly account for the missing data uncertainty. The multiply imputed datasets are each analysed using standard prognostic modelling techniques to obtain the estimates of ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章,实务指引

    doi:10.1186/1471-2288-9-57

    authors: Marshall A,Altman DG,Holder RL,Royston P

    更新日期:2009-07-28 00:00:00

  • Workplace restructurings in intervention studies - a challenge for design, analysis and interpretation.

    abstract:BACKGROUND:Interventions in occupational health often target worksites rather than individuals. The objective of this paper is to describe the (lack of) stability in units of analysis in occupational health and safety intervention projects directed toward worksites. METHODS:A case study approach is used to describe na...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-8-39

    authors: Olsen O,Albertsen K,Nielsen ML,Poulsen KB,Gron SM,Brunnberg HL

    更新日期:2008-06-13 00:00:00

  • Assessment of reproducibility of cancer survival risk predictions across medical centers.

    abstract:BACKGROUND:Two most important considerations in evaluation of survival prediction models are 1) predictability - ability to predict survival risks accurately and 2) reproducibility - ability to generalize to predict samples generated from different studies. We present approaches for assessment of reproducibility of sur...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章,多中心研究

    doi:10.1186/1471-2288-13-25

    authors: Chen HC,Chen JJ

    更新日期:2013-02-20 00:00:00

  • Prediction intervals for future BMI values of individual children: a non-parametric approach by quantile boosting.

    abstract:BACKGROUND:The construction of prediction intervals (PIs) for future body mass index (BMI) values of individual children based on a recent German birth cohort study with n = 2007 children is problematic for standard parametric approaches, as the BMI distribution in childhood is typically skewed depending on age. METHO...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-12-6

    authors: Mayr A,Hothorn T,Fenske N

    更新日期:2012-01-25 00:00:00

  • Norm values and psychometric properties for the German health regulatory focus scale - results of a representative survey.

    abstract:BACKGROUND:The health regulatory focus is an application of Higgins' regulatory focus theory to a health-specific context. It explains individual differences in health motivation, strategies, and behavior. Previous research found the Health Regulatory Focus Scale (HRFS) to be a reliable and valid measure for the constr...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-020-00927-x

    authors: Schmalbach B,Zenger M,Brähler E,Petrowski K

    更新日期:2020-03-02 00:00:00

  • Impact of preconception enrollment on birth enrollment and timing of exposure assessment in the initial vanguard cohort of the U.S. National Children's Study.

    abstract:BACKGROUND:The initial vanguard cohort of the U.S. National Children's Study was a pregnancy and birth cohort study that sought to enroll some women prior to pregnancy, and to assess exposures early in pregnancy. METHODS:During the recruitment phase (2009-2010), geographically based sampling was used to recruit women ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-015-0067-1

    authors: Stanford JB,Brenner R,Fetterer D,Palmer L,Schoendorf KC,U.S. National Children’s Study.

    更新日期:2015-09-24 00:00:00

  • An international survey of physicians regarding clinical trials: a comparison between Kyoto University Hospital and Seoul National University Hospital.

    abstract:BACKGROUND:International clinical trials are now rapidly expanding into Asia. However, the proportion of global trials is higher in South Korea compared to Japan despite implementation of similar governmental support in both countries. The difference in clinical trial environment might influence the respective physicia...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-13-130

    authors: Ito-Ihara T,Hong JH,Kim OJ,Sumi E,Kim SY,Tanaka S,Narita K,Hatta T,Choi EK,Choi KJ,Miyagawa T,Minami M,Murayama T,Yokode M

    更新日期:2013-10-25 00:00:00

  • The triple variable index combines information generated over time from common monitoring variables to identify patients expressing distinct patterns of intraoperative physiology.

    abstract:BACKGROUND:Mean arterial pressure (MAP), bispectral index (BIS), and minimum alveolar concentration (MAC) represent valuable, yet dynamic intraoperative monitoring variables. They provide information related to poor outcomes when considered together, however their collective behavior across time has not been characteri...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-019-0660-9

    authors: Schnetz MP,Hochheiser HS,Danks DJ,Landsittel DP,Vogt KM,Ibinson JW,Whitehurst SL,McDermott SP,Duque MG,Kaynar AM

    更新日期:2019-01-14 00:00:00

  • Structural issues affecting mixed methods studies in health research: a qualitative study.

    abstract:BACKGROUND:Health researchers undertake studies which combine qualitative and quantitative methods. Little attention has been paid to the structural issues affecting this mixed methods approach. We explored the facilitators and barriers to undertaking mixed methods studies in health research. METHODS:Face-to-face semi...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-9-82

    authors: O'Cathain A,Nicholl J,Murphy E

    更新日期:2009-12-09 00:00:00

  • Recruitment and retention in a multicentre randomised controlled trial in Bell's palsy: a case study.

    abstract:BACKGROUND:It is notoriously difficult to recruit patients to randomised controlled trials in primary care. This is particularly true when the disease process under investigation occurs relatively infrequently and must be investigated during a brief time window. Bell's palsy, an acute unilateral paralysis of the facial...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章,随机对照试验

    doi:10.1186/1471-2288-7-15

    authors: McKinstry B,Hammersley V,Daly F,Sullivan F

    更新日期:2007-03-28 00:00:00

  • Tools used to assess the quality of peer review reports: a methodological systematic review.

    abstract:BACKGROUND:A strong need exists for a validated tool that clearly defines peer review report quality in biomedical research, as it will allow evaluating interventions aimed at improving the peer review process in well-performed trials. We aim to identify and describe existing tools for assessing the quality of peer rev...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-019-0688-x

    authors: Superchi C,González JA,Solà I,Cobo E,Hren D,Boutron I

    更新日期:2019-03-06 00:00:00