Information Extraction from Multimodal ECG Documents Fei Wang

Tanveer Syeda-Mahmood David Beymer IBM Almaden Research Center 650 Harry Road, San Jose, CA 95120 {wangfe,stf,beymer}@us.ibm.com

Abstract With the rise of tools for clinical decision support, there is an increased need for automatic processing of electrocardiograms (ECG) documents. In fact, many systems have already been developed to perform signal processing tasks such as 12-lead off-line ECG analysis and real-time patient monitoring. All these applications require an accurate detection of the heart rate of the ECG. In this paper, we present the idea that the image form of ECG is actually a better medium to detect periodicity in ECG. When the ECG trace is scanned or rendered in videos, the peaks of the waveform (R-wave) is often traced thicker due to pixel dithering. We exploit the pixel thickness information, for the first time, as a reliable feature for determining periodicity. Results are presented on a database of 16,613 12-channel ECG waveforms, which demonstrate robustness and accuracy of our image-based period detection method on these ECGs of various cardiovascular diseases. 94.5% of bradycardia and tachycardia patient records are correctly identified using our estimated heart period as the disease criteria.

1

Introduction

An electrocardiogram (ECG) is an important and commonly used diagnostic aid in cardiovascular disease diagnosis. An ECG is an electrical recording of the heart that depicts the cardiac cycle. It is routinely used as a first course of choice in diagnosing many cardiovascular diseases. Often, 12 electrodes are used to record the electrical activity of the heart from different leads. A normal ECG waveform (in lead II) has a characteristic shape indicated in Fig. 1b. Many disturbances in the heart function show as characteristic variations in the sinus rhythm waveform of Fig.1b, and can be served as important cues to diagnose the disease. Physicians routinely make diagnosis by a simple visual examination of these ECG waveforms. With the rise of tools for clinical decision support, there

(a)

(b)

Figure 1. Illustration of the Electrocardiogram (a) The heart cycle. (b) A normal ECG.

has been an increased need for automatic processing of the electrocardiogram (ECG). In fact, many systems have already been developed to perform signal processing tasks such as 12-lead off-line ECG analysis, and real-time patient monitoring. All these applications require an accurate detection of the heart rate so that the disease related features within a heart cycle can be extracted. Furthermore, heart rate estimation is a prerequisite step to identify arrhythmia diseases such as bradycardia and tachycardia. Much of the prior work has focused on determining the heart rate from digital ECG time series signals. There are a large number of ECGs still in paper form. Their digital records are created as scanned images as shown in Fig. 2b. Not much prior art exists in the determination of period from ECG images obtained from scanner paper ECGs. Although most hospitals now have digital ECG recorders, much of the legacy ECG data is still in paper form. A sample paper form ECG is shown in Fig. 2b. Unlocking these ECG records printed on paper and exposing them to digital analysis would be useful. It provides additional data for current ECG analysis techniques, as well as historical data to be used for comparative studies. Digital ECG recordings are sampled very finely (1000 samples/sec) and contain a large amount of noisy data with baseline wandering problems, whereas the paper printouts are actually cleaner in appearance. Further, all 12 lead information is displayed compactly in a standardized (lead

(a)

(b)

echocardiogram videos, as well as digital ECGs, once it is converted into image form. While there is considerable prior art in ECG period detection from digital time series, there has not been much work on detecting periodicity from ECG images directly (rather than detecting periodicity after conversion to a time series). We will briefly review some related methods in literature next.

1.1

(c)

(d)

Figure 2. Illustration of the Electrocardiogram from different sources (a) Digital ECG (b) Paper ECG. (c) ECG embedded in echocardiogram video (d). ECG extracted from the echocardiogram frame from (c)

order is fixed) 3 row by 4 column format. The goal of this paper is to present a computational algorithm as well as medical decision support tool for finding the heart period from ECG images. Specifically, we use the knowledge that when ECG trace is scanned or rendered in videos, the peaks of the waveform (R-wave) is often traced thicker due to pixel dithering. We exploit the pixel thickness information, for the first time, as a reliable feature for determining periodicity. Existing period estimation algorithm, such as estimation of R peaks as largest positive peaks [13] often leads to detection error when R peaks are lower than the P or T peaks. While our method is relatively robust to these cases since our features latched on pixel dithering is more consistent among corresponding R peaks. Furthermore, using our estimated heart period as the disease criteria, 94.5% of bradycardia and tachycardia patient records are correctly identified on a database of 16,613 ECG waveforms. This paper makes several novel contributions. To our knowledge, this is the first period estimation work on image ECG waveforms from scanned ECG printouts or ECGs embedded in echo videos. It is also the first practical application of image-based techniques which utilize the pixel dithering to identify ECG period. Our method can also be used for disease diagnosis validation in healthcare decision support applications, where we can use our technique to identify arrythmia ECG records from patient with bradycardia or tachycardia diseases. Our algorithm can be especially useful for scanned paper ECGs, ECGs embedded in

Previous methods

Most of the existing approaches that deal with paper ECGs actually trace the signals to form a digital time series so that time series correlation methods can be used to detect periodicity. There are considerable algorithms available for single ECG period detection, which have been extensively analyzed in the past using a variety of classical methods including spectral analysis, time-frequency analysis, wavelets, and machine learning methods (see survey at (Koler et al. [4] and Tompkins [9]). One of the most popular approaches is based on detection of the R peak or QRS peak of the ECG waveform (Tompkins [9]). In Zimmerman et al. [13], the R wave is detected as the largest positive peaks in a 3 second sample window, after the ECG signal is first filtered to remove the wandering zero-volt baseline. Similar approaches were also used in Reisman et al. [6] except different kinds of bandpass filters were employed. For example, those filters include LPC filter [4], two-pole recursive filter [9], and bandpass filter in Reisman et al. [6]. These approaches are all based on an assumption that R peaks are the largest positive peaks in a ECG signal, which is not always true for ECG recordings. A modified feature-based auto-correlation function is utilized in Syeda-Mahmood et al. [8] and Wrublewski et al. [11], whereas the peaks in the autocorrelation function correspond to the various periodicity patterns found in the signal. The most common inter-peak duration is representative of a heart beat duration. This approach is however sensitive to the noise, and it tends to produce a segment that is less than a heart cycle due to multiple peaks within a heart beat. The power spectrum of the ECG waveform can provide useful information about the QRS complex. These spectrums can be generated using either Wavelet transformation (Saxena et al. [7] and Ghaffari et al. [3]) or Fourier transformation (Tompkins [9]). The peaks of the frequency spectrum obtained corresponds to the peak energy of the QRS complex. For an overview of these methods, please refer to Chapter 12 of Tompkins’ book [9]. These methods typically require that the frequency (or the scale in wavelet transform) of the QRS complex is known beforehand, so that QRS candidate can be searched in a defined vicinity for higher peaks. More recently, machine learning methods have been an

increased trend towards heart period estimation [1]. For example, in Mehta et al. [5], the Support Vector Machine (SVM) was used as a classifier for detection of QRS complexes in ECG; Vijaya et al. [10] propose to use Neural Networks to estimate ECG periods. A critical difference between these learning based approaches is that these methods require training of the ECG data, while we do not have any such requirement. Another source of ECGs is the synchronization ECG used in echocardiograms which shows as a waveform embedded in images as shown in Fig. 2c. The problem of periodicity estimation is frequently encountered in lots of spatio-temporal analysis of echo videos (Ebadollahi et al. [2]). While it is difficult to estimate the heart cycle directly from the depicted heart region in the video, it is relatively easy to estimate heart rate from the synchronizing ECG. Thus, methods are needed that can estimate the periodicity from embedded ECGs in images.

2

ECG envelope extraction

We extract the ECG waveforms as curves in the respective image segments where each lead position in the image is segmented. Due to noise in recordings as well as the stylus speed in the ECG recorder, there are often gaps which cause problems in curve extraction. Note that these recordings are actually time series or functions of lead-voltage vs time. Thus, a general purpose algorithm such as curve following or skeletonization, may not enforce the constraint that a single y−value occur for each x position in image

Ru

=

[yu (x − 1) − W, yu (x − 1) + W ]

(1)

Rl

=

[yl (x − 1) − W, yl (x − 1) + W ]

(2)

where W is a search window. In practice, W needs to be set large enough to handle the voltage spike at an R wave. Next, we define

Model

In this section, we describe our algorithm in detail. Our method can be divided into two major steps: first we extract ECG envelopes from images, and then estimate the period based on these image-based features. Once the ECG is in the image form, either scanned or extracted from the echocardiographic videos, the next important step is to preprocess the data which will be useful in estimating the period. Our approach of period detection from ECG image is based on a key observation that the difference of the upper envelope and lower envelope of the ECG can signify the local maximum & minimum of the original signal. The upper envelope and (shown in red) and lower envelope of the ECG (shown in blue) are plotted in Fig. 4b. Note that the gap between the two envelopes increase significantly when the signal reaches its peak or valley. This phenomenon is exploited in our algorithm to detect the heart rate, which however cannot be used if we treat ECG as signals since all points would have equal tracing width. We now first discuss the extraction of ECG waveforms from the scanned ECG images.

2.1

coordinates. In places where the axis bifurcates or turns back on itself, a post-processing step would be required for choosing between multiple values of y for a particular value of x. Since the ECG curve may be multiple pixels thick at the scanned resolution, our curve tracing algorithm follows the upper and lower edges of the curve. Fig. 3 depicts these curves, yu (x) and yl (x). In our tracing algorithm, we continue the curve trace from x − 1, thus extending yu (x − 1) and yl (x − 1). We define two search ranges centered around these values:

yu (x)

=

yl (x)

=

min y

such that

T (x, y) = 0

(3)

max y

such that

T (x, y) = 0

(4)

y∈Ru y∈Rl

which keeps yu and yl tracing the upper and lower envelopes of the ECG curve. After tracing the envelopes, the average y(x) = (yu + yl )/2

(5)

is used as the traced value y(x). To provide robustness to noise and missing curve fragments, we use morphological operators and a grouping algorithm across the gaps. A morphological open (erode + dilate) helps fill in holes inside the curve, and a morphological close (dilate + erode) eliminates noise pixels near the curve. To close small gaps, curve tracing is started using a number of seed points along the expected ECG curve location. Gaps are closed between pairs of consecutive fragments if the gap is small enough (4 pixels = 10 msec). For gaps that are larger than the gap threshold, we have found that most larger breaks in the curve occur at the R wave, where the signal spikes up and down. At the R wave, however, the curve is nearly vertical, so the sampling in equations (1)-(2) is nearly tangent to the curve. Thus, we can handle a large fraction of vertical dropout, as shown in Fig. 3. Fig. 4(b) illustrates the upper and lower envelope of the ECG curve. ydif f = yu −yl is the difference between the two envelopes, which is shown in Fig. 4e. It is evident from this plot that the peak of the difference signal is quite consistent among all R peaks, and therefore can serve as a good feature to identify the R peak locations.

2.2

ECG period detection

Differentiation forms the basis of many QRS detection algorithms [12, 11]. The differentiator, in effect, acts as a

waveform is shown in Fig. 4f, in which we noticed that peaks of the product is corresponding to the R wave of the ECG signal. The period of the ECG can therefore be estimated as the interval between these peaks.

3 Figure 3. Illustration of curve tracing across gaps in ECG images. Upper and lower bounding curves of the ECG, yu and yl , are shown in green and red. The R wave spikes show a number of gaps in the original curve that are correctly bridged by our tracing technique.

Figure 4. Illustration of various waveforms from different stages of period detection algorithm. high-pass filter, which characterizes the steep slope of the QRS complex of the ECG signal. Specifically, the first order derivative of the ECG signal amplifies the higher frequencies characteristic of the QRS complex while attenuating the lower frequencies of the P and T waves. For these reasons, we take the product of the ECG derivative with the envelope difference signal to filter out the high frequency noise of the envelope difference signal. Differentiator filters have several different forms [4], we choose to approximate derivative of the input ECG signal by the backward difference. Once the ECG envelopes are extracted from the image, the final product consists of three waveforms, which is expressed as −−→ −−−−−→ −−−→ S(x) = y(x) ∗ ydif f (x) ∗ y 0 (x)

(6)

− where → y is the normalized ECG waveform, in which the −→ wandering baseline is removed; − y− dif f is the normalized en→ − velope difference (shown in Fig. 4e), and y 0 is the normalized first order ECG derivative. The normalized product

EXPERIMENTAL RESULTS

We now present experimental results on the application of our algorithm to both ECGs synthesized from the digital waveforms and those ECGs extracted from echocardiogram videos. First, to demonstrate the robustness of our algorithm, we apply our algorithm to an ECG from an atrial fibrillation patient, in which the ECG waveform is quite noisy and QRS complex are unpronounced. Regular methods such as a derivative-based method all failed to recover the period, which is evident from the derivative waveform in Fig. 5b. The product waveform S(x) produced from our algorithm is shown in Fig. 5c. It is evident that our method is able to pick up the R peak from the noisy waveform, even when the noise level is high. Our method exhibits stronger resistance to noise than the derivative-based and other filterbased approaches. Next, we conduct our experiments on a large collection of digital ECG datasets, which we obtained from a large hospital network. The database contains 16,613 12-channel ECG waveforms. 13043 ECGs come with the ground truth heart rate among the whole data collection. Using our method, we are able to achieve 88% accuracy rate, where the error between ground truth and the estimation is under 5%. In comparison, cross-correlation is only able to achieve a 72% accuracy rate. A closer examination of the those cases in which the relative error from the ground truth is higher than 5% reveals that the majority of those ECGs are either i) arrhythmia cases where each channel gives a distinct period estimate that confuses the algorithms, ii) the ground truth for the ECGs are inaccurate. We believe the accuracy of our algorithm can be further improved if those cases are excluded. Additionally, of the 16613 ECGs, we were able to additionally find a period for 2641 more cases for which these patients had no documented periods. We found 2538 cases of bradycardia, and 1541 cases of tachycardia among those matching with the ground truth, which represents 94.5% of the bradycardia and tachycardia patients when compared with the ground truth disease labels. Sample bradycardia and tachycardia ECGs that are retrieved by our algorithm is shown in Fig. 6. Finally, we apply our algorithm to estimate the heart cycle from ECGs extracted from a large collection echocardiogram videos, which contains 1178 echocardiogram video sequences. For each of these video clips, we first extract the region of interest which contain ECG signal, and feed it to our period detection algorithm. After we estimate the period for all these videos, we randomly pick 100 videos

Figure 5. Robustness in the presence of large noise. a) ECG image; b) first order derivative of the ECG waveform; c) product waveform from our method. (a) Bradycardia

(b) Tachycardia

Figure 6. Sample bradycardia and tachycardia ECGs retrieved by our algorithm.

with ground truth periods, we are able to achieve 84% accuracy rate, where the error between ground truth and the estimation is under 5%. We believe the main cause of error is from the variation of heart beat within single channel ECG waveform embedded in the videos.

4

Conclusions

In this paper, we present a novel computational algorithm as well as medical decision support tool for finding the heart period from multimodal electrocardiogram (ECG) documents. We exploit the pixel thickness information, for the first time, as a reliable feature for determining periodicity. Existing period estimation algorithm, such as estimation of R peaks as largest positive peaks often leads to detection error when R peaks are lower than the P or T peaks. While our method is relatively robust to these cases since our features latched on pixel dithering is more consistent among corresponding R peaks. Furthermore, we are able to correctly identify 94.5% of bradycardia and tachycardia patient records (on a database of 16,613 ECG waveforms) using our estimated heart period as the disease criteria. Future work will focus on building a mathematical model for the dithering effect so that our algorithm can be generalized to documents of other digital signals.

References [1] H. Bunke and A. Sanfeliu, editors. Syntactic and Structural Pattern Recognition Theory and Applications. World Scientific, 1990. [2] S. Ebadollahi, S.-F. Chang, and H. Wu. Automatic view recognition in echocardiogram videos using parts-based representation. In CVPR, pages 2–9, 2004. [3] A. Ghaffari, H. Golbayani, and M. Ghasemi. A new mathematical based qrs detector using continuous wavelet transform. Comput. Electr. Eng., 34(2):81–91, 2008. [4] B.-U. Koler, C. Hennig, and R. Orglmeister. The principles of software QRS detection. IEEE Engineering in Medicine and Biology Magazine, 21:42–57, 2002. [5] J. Mehta and N.S.Lingayat. Comparative study of QRS detection in single lead and 12-lead ecg based on entropy and combined entropy criteria using support vector machine. JATIT, 3:8–18, 2007. [6] S. Reisman and W. Tapp. Variable threshold r wave detector: Use in automated ecg processing. Physiology & Behavior, 35:815–818, 1985. [7] S. C. Saxena, V. Kumar, and S. Hamde. Feature extraction from ECG signals using wavelet transform for disease diagnostics. IJSS, 33:1073– 1085, 2002. [8] T. Syeda-Mahmood, D. Beymer, and F. Wang. Shape-based matching of ECG recordings. IEEE International Conference on Engineering in Medicine and Biology, pages 2012– 2018, 2007. [9] W. J. Tompkins, editor. Biomedical digital signal processing: C-language examples and laboratory experiments for the IBM PC. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1993. [10] G. Vijaya, V. Kumar, and H. Verma. ANN-based QRScomplex analysis of ECG. J Med Eng Technol., pages 160– 167, 1998. [11] T. A. Wrublewski, Y. Sun, and J. Beyer. Real-time early detection of r waves of ECG signals. Engineering in Medicine and Biology Society, pages 38 – 39, 1989. [12] Y.-C. Yeh and W.-J. Wang. QRS complexes detection for ECG signal: The difference operation method. Comput. Methods Prog. Biomed., 91(3):245–254, 2008. [13] T. Zimmerman and T. Syeda-Mahmood. Automatic detection of heart disease from twelve channel electrocardiogram waveforms. In Computers in Cardiology, pages 809–812, 2007.