Medical Image Segmentation for Improved Surgical Navigation Erik Smistad
Norwegian University of Science and Technology
Outline ●
●
●
Introduction –
Image segmentation
–
Image guided surgery
–
Challenges
–
Research goals
Results –
Parallel and GPU acceleration
–
Segmentation of tubular structures
–
Segmentation of ultrasound images
Conclusions and future work
Norwegian University of Science and Technology
2
Introduction
Norwegian University of Science and Technology
3
Image segmentation ●
●
The process of finding which structure or tissue each picture element belongs to Usually, only a few structures are of interest
Norwegian University of Science and Technology
4
Image guided surgery ●
Minimal invasive surgery –
●
●
Reduces the risk of complications, the amount of postoperative pain and shortens recovery time [Darzi and Munz, 2004]
When using small incision, still need to see inside the body The goal of image guided surgery is to enhance MIS with computer technology and images acquired by: –
Ultrasound
–
Computer tomography (CT)
–
Magnetic resonance imaging (MRI)
–
Cameras Norwegian University of Science and Technology
5
Segmentation in image guided surgery ●
●
Medical images contain a lot of information and noise Segmentation is used to extract information from the images –
Visualize only the structures of interest
–
Quantitative measurements
–
Feature-based registration
Norwegian University of Science and Technology
6
Image segmentation challenges ●
Accuracy –
–
Several factors affect the accuracy ●
Partial volume effect
●
Tissue intensity inhomogeneity
●
Low contrast
●
Other image artifacts
Relying on pixel intensity alone is often not sufficient for segmentation → Need models of the anatomy
●
Speed
●
Automation –
Semi-automatic methods
–
User interaction not desired
Norwegian University of Science and Technology
Research goals
Norwegian University of Science and Technology
8
Segmentation speed ●
●
●
●
Speed is important, and not addressed as much as accuracy Many ways to increase the speed One way is to exploit the parallel processing capabilities of modern processors Today, consumer computers contain processors capable of running many operations in parallel at a low cost
Norwegian University of Science and Technology
9
Graphic processing unit (GPU) ●
●
●
●
Created for fast rendering of graphics in computer games and computer aided design (CAD) applications Can process several thousand data elements in parallel The use of GPUs for medical image processing is increasing Researchers are reporting high speedups Research goal 1: Investigate the use of parallel and GPU computing to accelerate image segmentation Norwegian University of Science and Technology
10
Segmentation of tubular structures ●
●
●
Blood vessels and airways are examples of important tubular structures Important in many applications –
Bronchoscopy
–
Neurosurgery
–
Liver tumor resection
In addition to the segmentation, the centerline is also of interest
Norwegian University of Science and Technology
11
Segmentation of tubular structures ●
●
Previous challenges on airway and blood vessel segmentation have shown that Hessian-based tube detection filters perform well (EXACT'09, VESSELS'12) The tube detection filters are computational expensive, but have shown to be suitable for GPU computation [Erdt et al. (2008), Bauer et al. (2009), Helmberger et al. (2013)]
Research goal 2: Create a segmentation method for tubular structures ●
● ●
Generic - Able to extract different tubular structures from various image modalities (CT, MR, Ultrasound). Automatic - No user interaction needed. Fast - Accelerate processing using GPUs. Norwegian University of Science and Technology
12
Segmentation of ultrasound images ●
Ultrasound is an important intraoperative imaging modality –
Real-time imaging
–
Low cost
–
Small footprint
Norwegian University of Science and Technology
13
Segmentation of ultrasound images ●
Challenging both in terms of accuracy and speed –
Several different image artifacts ●
Tissue intensity inhomogeneity
●
Reverberations
●
Acoustic shadowing
–
Model of anatomy is essential
–
Ultrasound delivers several images per second
Norwegian University of Science and Technology
14
Segmentation of ultrasound images ●
State estimation techniques such as Kalman filters can use segmentation from previous image frames to segment the current image –
Have been used previously to segment structures in ultrasound (Orderud et al. (2007), Guerrero et al. (2007))
–
Need an initialization
Research goal 3: Create segmentation methods for ultrasound images which are: ● ● ●
Robust - Using model-based segmentation methods. Automatic - Initialize methods using prior knowledge Real-time - Accelerate processing using GPUs and state estimation methods
Norwegian University of Science and Technology
15
Results
Norwegian University of Science and Technology
16
Parallel and GPU acceleration
Norwegian University of Science and Technology
17
Review on medical image segmentation on the GPU ●
The review discuss the key differences between CPUs and GPUs. –
GPUs have more cores and thread processors
–
GPUs can process many data elements in parallel, but must perform the same instructions (data parallel).
–
The memory cache of GPUs is much smaller than CPUs. Memory access is expensive.
Norwegian University of Science and Technology
18
Review on medical image segmentation on the GPU ●
●
Five key factors influencing the algorithm's suitability for GPU computation were identified –
Data parallelism
–
Thread count
–
Branch divergence
–
Memory usage
–
Synchronization
13 segmentation methods investigated + variations
Norwegian University of Science and Technology
19
Results of review ●
●
●
Amount of data parallelism and thread count is high in most methods. The review concludes that most segmentation methods can benefit from GPU acceleration. However, factors such as synchronization, branch divergence and memory usage may limit the speedup over serial execution.
Norwegian University of Science and Technology
20
Heterogeneous medical image computing ●
●
Parallel and GPU programming is challenging –
Errors in GPU drivers
–
Hard to debug
–
Few libraries
–
Programmer is exposed to many hardware details
–
Need for low level programming and explicit memory handling
Computer systems are becoming increasingly heterogeneous –
●
Multi-core CPUs, GPUs, APUs
As the amount of medical image data increases, it is crucial to exploit the computational power of these processors Norwegian University of Science and Technology
21
Heterogeneous medical image computing ●
●
We see the need for a framework which makes it easier to do efficient medical image computing and visualization on heterogeneous system The insight toolkit (ITK) and the visualization toolkit (VTK) –
●
Have some support for GPU computation, but more as an optional extension
Need to support heterogeneous computing in the core of the framework to get the best performance while being easy to use Norwegian University of Science and Technology
22
Norwegian University of Science and Technology
23
Data management ● ●
High-level data objects Programmer requests data a specific processor –
FAST takes care of all memory transfers and keeping the data up to date on the processors
–
Lazy loading Norwegian University of Science and Technology
24
Execution pipeline ●
Demand driven execution pipeline –
FAST should therefore be easy to learn for experienced ITK/VTK users
Norwegian University of Science and Technology
25
Data streaming ●
Processing pipelines in FAST can handle both static and dynamic/temporal data without any change to the algorithms. –
Ultrasound
Norwegian University of Science and Technology
26
Interoperability ●
Many medical image computing algorithms implemented with ITK and VTK –
● ●
Unrealistic to implement all of these in FAST
Interoperability with ITK and VTK This may ease the integration of FAST into existing applications.
Norwegian University of Science and Technology
27
More features of FAST ●
High performance algorithms
●
Fast concurrent visualization
●
Open-source –
Permissive BSD licence
–
Available on github.com/smistad/FAST/
●
Cross platform
●
Testing –
Has several hundred tests which are executed on all platforms every time the code is changed Norwegian University of Science and Technology
28
Performance results
Norwegian University of Science and Technology
29
Segmentation of tubular structures
Norwegian University of Science and Technology
30
Airways ●
●
Developed a GPU based segmentation method for airways [A2] Integrated into SINTEF's CustusX navigation system and being tested for use in bronchoscopy
National Cancer Institute Norwegian University of Science and Technology
31
Novel cropping method
GPU-based gradient vector flow [A3] Circle fitting [Krissian et al. 2000]
Ridge traversal [Aylward and Bullitt 2002] Region growing [Bauer et al. 2009]
Norwegian University of Science and Technology
32
Cropping method ●
●
●
●
GPU memory limited (at that time 1-3 GB) Created a GPU cropping algorithm for CT thorax images Removes 70-80% of the data, reducing memory usage from 5-8 GB to 1-2 GB Used about 1.5 seconds Norwegian University of Science and Technology
33
Results ●
All steps except centerline extraction on the GPU
●
Used about 20 to 40 seconds –
[Bauer et al. 2009] reported 6 minutes
Norwegian University of Science and Technology
34
Blood vessels and multi-modality ●
●
Extended in article C to also extract blood vessels from CT, MRI and 3D ultrasound. Created a parallel centerline extraction method –
●
Entire method executed on GPU
Optimizations –
Texture memory
–
Normalized 16 bit integers Norwegian University of Science and Technology
35
Blood vessels from MR angio of the brain
Proposed GPU method + Parallel centerline extraction
Proposed GPU method + Ridge traversal
Norwegian University of Science and Technology
Thresholding + skeletonization
36
3D Ultrasound Doppler
Proposed GPU method + Parallel centerline extraction
Proposed GPU method + Ridge traversal
Norwegian University of Science and Technology
Thresholding + skeletonization
37
Liver vessels (CT)
Lung vessels (CT) Liver vessels (MR) Norwegian University of Science and Technology
38
Speed measurements 9 8 7
Runtime (seconds)
6 5 Parallel – 16 bit Parallel - 32 bit Ridge traversal – 16 bit Ridge traversal – 32 bit
4 3 2 1 0 CT Airways
MR Vessels
US Vessels
Dataset
AMD Radeon HD7970 GPU Norwegian University of Science and Technology
39
Accuracy ●
No loss in accuracy with 16 bit
●
Ridge traversal more accurate
●
Proposed parallel CE able to extract more with higher noise levels
Norwegian University of Science and Technology
40
Abdominal aortic aneurysms ●
●
●
Vascular disease resulting in a permanent local dilation of the abdominal aorta. These have non-circular cross section Developed a new TDF in article D to deal with noncircular tubular structures Norwegian University of Science and Technology
41
New tube detection filter ●
Line search in cross-sectional plane
Circle fitting TDF [Krissian et al. 2000] Norwegian University of Science and Technology
Proposed TDF
42
Abdominal aortic aneurysms ●
These aneurysms are also large
●
The traditional Euler GVF method converges slowly –
●
GPU method needed several minutes to converge
In article E, a GPU-based multigrid GVF method was proposed –
Converged in about 1 second
Norwegian University of Science and Technology
43
Region growing
Circle fitting TDF
Norwegian University of Science and Technology
Proposed TDF 44
Region growing
Circle fitting TDF
Norwegian University of Science and Technology
Proposed TDF 45
Segmentation of ultrasound images
Norwegian University of Science and Technology
46
Segmentation of ultrasound images ●
A Kalman filter state estimation pipeline was used for segmentation of ultrasound images in article F and G –
State describes the segmentation of the current image
–
Shape model
–
Appearance model
–
Temporal model
Norwegian University of Science and Technology
47
Left ventricle of the heart ●
●
●
Proposed a method for segmenting the left ventricle in 3D ultrasound [article F] A mesh was used as a shape model A control mesh and mean value coordinates [Ju et al. 2005] was used to deform the shape
Norwegian University of Science and Technology
48
Left ventricle of the heart ●
●
Edge detection was performed along the normal of each vertex on the mesh A step edge detection method was used for the appearance model, requiring the inside to be darker than the border.
Norwegian University of Science and Technology
49
Left ventricle of the heart ●
●
The state of the left ventricle consist of position, rotation, scaling and displacement vectors for each vertex in the control mesh Initialize in the middle of the image
Norwegian University of Science and Technology
50
Evaluation ●
●
●
●
Challenge on Endocardial Three-dimensional Ultrasound Segmentation (CETUS) MICCAI 2014 45 sequences in total The accuracy was compared to 8 other automatic and semi-automatic methods. The proposed method was ranked second in terms of clinical relevance of the fully automatic algorithms –
Mean mesh difference of about 2.72 mm
–
The best algorithm had a mean mesh difference of 2.35 mm Norwegian University of Science and Technology
51
Speed ●
The runtime was 65 ms on average per image frame –
●
The fastest method in the challenge
After the CETUS challenge, GPU acceleration was investigated –
10 ms
–
Parallel edge detection
Norwegian University of Science and Technology
52
Femoral artery ●
●
In article G, methods for tracking the femoral artery in ultrasound images was presented The tracked artery was used to register a model of the surrounding anatomy to the ultrasound images
Norwegian University of Science and Technology
53
Blood vessels ●
●
●
Ellipse used as shape model Appearance model expects a step or ridge edge State consists of position of the ellipse and the minor and major radius (4 values) Norwegian University of Science and Technology
54
Blood vessels ●
●
●
Need the position and radius to initialize the Kalman filter Developed an automatic initialization method which uses the GPU to do a search for black circles in the ultrasound image For every pixel in the image and for different radius calculate
●
Store position and radius with highest S.
●
If S is above a threshold, accept it Norwegian University of Science and Technology
55
Results ●
●
●
12 ultrasound image sequences from 3 subjects Speed –
Initialization: 42 ms
–
Tracking: 5 ms
Accuracy –
Mean absolute distance: 0.42 mm
–
Hausdorff distance: 1.17 mm. Norwegian University of Science and Technology
56
Revision ●
Vessel 3D reconstruction
●
Improved registration
●
Increased number of subjects (6) and acquisitions (48)
Norwegian University of Science and Technology
57
Conclusions ●
●
●
Parallel and GPU computing can significantly accelerate segmentation of medical images. A high-level framework for efficient and concurrent medical image computing and visualization on heterogeneous systems is needed. Good temporal, appearance and shape models are needed for automatic, robust and accurate segmentation of medical images. Norwegian University of Science and Technology
58
Future work ●
Continue to develop FAST
●
Segmentation of tubular structures
●
–
Anisotropic smoothing
–
Global optimization methods for the centerline extraction
Segmentation of ultrasound images –
Better shape and appearance models ● ● ●
–
Create better shape models with training data Block matching / speckle tracking Use both B-mode and Doppler when tracking vessels
Other applications ● ● ●
Right ventricle of the heart Brain ventricle Tracking multiple vessels Norwegian University of Science and Technology
59
References ●
●
●
●
●
●
●
●
●
Aylward, S. R. and Bullitt, E. Initialization, noise, singularities, and scale in height ridge traversal for tubular object centerline extraction. IEEE Transactions on Medical Imaging 21:2. Pages 61-75. 2002. Darzi, Sir A. and Munz, Y. The impact of minimally invasive surgical techniques. Annual review of medicine 55. Pages 232-237. 2004. Erdt, M. Raspe, M. and Suehling, M. Automatic hepatic vessel segmentation using graphics hardware. Proceedings of the 4th international workshop on Medical Imaging and Augmented Reality. Pages 403-412. 2008. Bauer, C. Bischof, H. and Beichel, R. Segmentation of airways based on gradient vector flow. Proceedings of the 2nd International Workshop on Pulmonary Image Analysis. Pages 191-201. 2009. Helmberger, M. Urschler, M. Pienn, M. Bálint, Z. Olschewski, A. and Bischof, H. Pulmonary Vascular Tree Segmentation from Contrast-Enhanced CT Images. Proceedings of the 37th Annual Workshop of the Austrian Association for Pattern Recognition. Pages 1-10. 2013. Orderud, F. Hansgård, J. and Rabben, S. Real-time tracking of the left ventricle in 3D echocardiography using a state estimation approach. Medical Image Computing and Computer-Assisted Intervention (MICCAI). Part I. Pages 858-865. 2007. Guerrero, J. Salcudean, S. E. McEwen, J. A. Masri, B. A. and Nicolaou, S. Real-time vessel segmentation and tracking for ultrasound imaging applications. IEEE Transactions on medical imaging 26:8. Pages 1079-1090. 2007. Krissian, K. Malandain and G. Ayache, N. Model-Based Detection of Tubular Structures in 3D Images. Computer Vision and Image Understanding 80:2. Pages 130-171. 2000. Ju, T. Schaefer, S. and Warren, J. Mean Value Coordinates for Closed Triangular Meshes. ACM Transactions on Graphics 24:3. Pages 561-566. 2005.
Norwegian University of Science and Technology
60