当前位置： SCI文献检索 > JOURNAL OF BIOMEDICAL INFORMATICS期刊下所有文献 > Combining glass box and black box evaluations in the identification of heart disease risk factors and their temporal relations from clinical records.

Combining glass box and black box evaluations in the identification of heart disease risk factors and their temporal relations from clinical records.

Abstract：

BACKGROUND:The determination of risk factors and their temporal relations in natural language patient records is a complex task which has been addressed in the i2b2/UTHealth 2014 shared task. In this context, in most systems it was broadly decomposed into two sub-tasks implemented by two components: entity detection, and temporal relation determination. Task-level ("black box") evaluation is relevant for the final clinical application, whereas component-level evaluation ("glass box") is important for system development and progress monitoring. Unfortunately, because of the interaction between entity representation and temporal relation representation, glass box and black box evaluation cannot be managed straightforwardly at the same time in the setting of the i2b2/UTHealth 2014 task, making it difficult to assess reliably the relative performance and contribution of the individual components to the overall task. OBJECTIVE:To identify obstacles and propose methods to cope with this difficulty, and illustrate them through experiments on the i2b2/UTHealth 2014 dataset. METHODS:We outline several solutions to this problem and examine their requirements in terms of adequacy for component-level and task-level evaluation and of changes to the task framework. We select the solution which requires the least modifications to the i2b2 evaluation framework and illustrate it with our system. This system identifies risk factor mentions with a CRF system complemented by hand-designed patterns, identifies and normalizes temporal expressions through a tailored version of the Heideltime tool, and determines temporal relations of each risk factor with a One Rule classifier. RESULTS:Giving a fixed value to the temporal attribute in risk factor identification proved to be the simplest way to evaluate the risk factor detection component independently. This evaluation method enabled us to identify the risk factor detection component as most contributing to the false negatives and false positives of the global system. This led us to redirect further effort to this component, focusing on medication detection, with gains of 7 to 20 recall points and of 3 to 6 F-measure points depending on the corpus and evaluation. CONCLUSION:We proposed a method to achieve a clearer glass box evaluation of risk factor detection and temporal relation detection in clinical texts, which can provide an example to help system development in similar tasks. This glass box evaluation was instrumental in refocusing our efforts and obtaining substantial improvements in risk factor detection.

journal_name

J Biomed Inform

journal_title

Journal of biomedical informatics

authors

Grouin C,Moriceau V,Zweigenbaum P

doi

10.1016/j.jbi.2015.06.014

subject

Has Abstract

pub_date

2015-12-01 00:00:00

pages

S133-42

eissn

1532-0464

issn

1532-0480

pii

S1532-0464(15)00124-0

journal_volume

58 Suppl

pub_type

杂志文章

在线工具

Combining glass box and black box evaluations in the identification of heart disease risk factors and their temporal relations from clinical records.