Medical Image Segmentation for Improved Surgical Navigation

Medical Image Segmentation for Improved Surgical Navigation Erik Smistad Norwegian University of Science and Technology Outline ● ● ● Introducti...
12 downloads 0 Views 4MB Size
Medical Image Segmentation for Improved Surgical Navigation Erik Smistad

Norwegian University of Science and Technology

Outline ●





Introduction –

Image segmentation



Image guided surgery



Challenges



Research goals

Results –

Parallel and GPU acceleration



Segmentation of tubular structures



Segmentation of ultrasound images

Conclusions and future work

Norwegian University of Science and Technology

2

Introduction

Norwegian University of Science and Technology

3

Image segmentation ●



The process of finding which structure or tissue each picture element belongs to Usually, only a few structures are of interest

Norwegian University of Science and Technology

4

Image guided surgery ●

Minimal invasive surgery –





Reduces the risk of complications, the amount of postoperative pain and shortens recovery time [Darzi and Munz, 2004]

When using small incision, still need to see inside the body The goal of image guided surgery is to enhance MIS with computer technology and images acquired by: –

Ultrasound



Computer tomography (CT)



Magnetic resonance imaging (MRI)



Cameras Norwegian University of Science and Technology

5

Segmentation in image guided surgery ●



Medical images contain a lot of information and noise Segmentation is used to extract information from the images –

Visualize only the structures of interest



Quantitative measurements



Feature-based registration

Norwegian University of Science and Technology

6

Image segmentation challenges ●

Accuracy –



Several factors affect the accuracy ●

Partial volume effect



Tissue intensity inhomogeneity



Low contrast



Other image artifacts

Relying on pixel intensity alone is often not sufficient for segmentation → Need models of the anatomy



Speed



Automation –

Semi-automatic methods



User interaction not desired

Norwegian University of Science and Technology

Research goals

Norwegian University of Science and Technology

8

Segmentation speed ●







Speed is important, and not addressed as much as accuracy Many ways to increase the speed One way is to exploit the parallel processing capabilities of modern processors Today, consumer computers contain processors capable of running many operations in parallel at a low cost

Norwegian University of Science and Technology

9

Graphic processing unit (GPU) ●







Created for fast rendering of graphics in computer games and computer aided design (CAD) applications Can process several thousand data elements in parallel The use of GPUs for medical image processing is increasing Researchers are reporting high speedups Research goal 1: Investigate the use of parallel and GPU computing to accelerate image segmentation Norwegian University of Science and Technology

10

Segmentation of tubular structures ●





Blood vessels and airways are examples of important tubular structures Important in many applications –

Bronchoscopy



Neurosurgery



Liver tumor resection

In addition to the segmentation, the centerline is also of interest

Norwegian University of Science and Technology

11

Segmentation of tubular structures ●



Previous challenges on airway and blood vessel segmentation have shown that Hessian-based tube detection filters perform well (EXACT'09, VESSELS'12) The tube detection filters are computational expensive, but have shown to be suitable for GPU computation [Erdt et al. (2008), Bauer et al. (2009), Helmberger et al. (2013)]

Research goal 2: Create a segmentation method for tubular structures ●

● ●

Generic - Able to extract different tubular structures from various image modalities (CT, MR, Ultrasound). Automatic - No user interaction needed. Fast - Accelerate processing using GPUs. Norwegian University of Science and Technology

12

Segmentation of ultrasound images ●

Ultrasound is an important intraoperative imaging modality –

Real-time imaging



Low cost



Small footprint

Norwegian University of Science and Technology

13

Segmentation of ultrasound images ●

Challenging both in terms of accuracy and speed –

Several different image artifacts ●

Tissue intensity inhomogeneity



Reverberations



Acoustic shadowing



Model of anatomy is essential



Ultrasound delivers several images per second

Norwegian University of Science and Technology

14

Segmentation of ultrasound images ●

State estimation techniques such as Kalman filters can use segmentation from previous image frames to segment the current image –

Have been used previously to segment structures in ultrasound (Orderud et al. (2007), Guerrero et al. (2007))



Need an initialization

Research goal 3: Create segmentation methods for ultrasound images which are: ● ● ●

Robust - Using model-based segmentation methods. Automatic - Initialize methods using prior knowledge Real-time - Accelerate processing using GPUs and state estimation methods

Norwegian University of Science and Technology

15

Results

Norwegian University of Science and Technology

16

Parallel and GPU acceleration

Norwegian University of Science and Technology

17

Review on medical image segmentation on the GPU ●

The review discuss the key differences between CPUs and GPUs. –

GPUs have more cores and thread processors



GPUs can process many data elements in parallel, but must perform the same instructions (data parallel).



The memory cache of GPUs is much smaller than CPUs. Memory access is expensive.

Norwegian University of Science and Technology

18

Review on medical image segmentation on the GPU ●



Five key factors influencing the algorithm's suitability for GPU computation were identified –

Data parallelism



Thread count



Branch divergence



Memory usage



Synchronization

13 segmentation methods investigated + variations

Norwegian University of Science and Technology

19

Results of review ●





Amount of data parallelism and thread count is high in most methods. The review concludes that most segmentation methods can benefit from GPU acceleration. However, factors such as synchronization, branch divergence and memory usage may limit the speedup over serial execution.

Norwegian University of Science and Technology

20

Heterogeneous medical image computing ●



Parallel and GPU programming is challenging –

Errors in GPU drivers



Hard to debug



Few libraries



Programmer is exposed to many hardware details



Need for low level programming and explicit memory handling

Computer systems are becoming increasingly heterogeneous –



Multi-core CPUs, GPUs, APUs

As the amount of medical image data increases, it is crucial to exploit the computational power of these processors Norwegian University of Science and Technology

21

Heterogeneous medical image computing ●



We see the need for a framework which makes it easier to do efficient medical image computing and visualization on heterogeneous system The insight toolkit (ITK) and the visualization toolkit (VTK) –



Have some support for GPU computation, but more as an optional extension

Need to support heterogeneous computing in the core of the framework to get the best performance while being easy to use Norwegian University of Science and Technology

22

Norwegian University of Science and Technology

23

Data management ● ●

High-level data objects Programmer requests data a specific processor –

FAST takes care of all memory transfers and keeping the data up to date on the processors



Lazy loading Norwegian University of Science and Technology

24

Execution pipeline ●

Demand driven execution pipeline –

FAST should therefore be easy to learn for experienced ITK/VTK users

Norwegian University of Science and Technology

25

Data streaming ●

Processing pipelines in FAST can handle both static and dynamic/temporal data without any change to the algorithms. –

Ultrasound

Norwegian University of Science and Technology

26

Interoperability ●

Many medical image computing algorithms implemented with ITK and VTK –

● ●

Unrealistic to implement all of these in FAST

Interoperability with ITK and VTK This may ease the integration of FAST into existing applications.

Norwegian University of Science and Technology

27

More features of FAST ●

High performance algorithms



Fast concurrent visualization



Open-source –

Permissive BSD licence



Available on github.com/smistad/FAST/



Cross platform



Testing –

Has several hundred tests which are executed on all platforms every time the code is changed Norwegian University of Science and Technology

28

Performance results

Norwegian University of Science and Technology

29

Segmentation of tubular structures

Norwegian University of Science and Technology

30

Airways ●



Developed a GPU based segmentation method for airways [A2] Integrated into SINTEF's CustusX navigation system and being tested for use in bronchoscopy

National Cancer Institute Norwegian University of Science and Technology

31

Novel cropping method

GPU-based gradient vector flow [A3] Circle fitting [Krissian et al. 2000]

Ridge traversal [Aylward and Bullitt 2002] Region growing [Bauer et al. 2009]

Norwegian University of Science and Technology

32

Cropping method ●







GPU memory limited (at that time 1-3 GB) Created a GPU cropping algorithm for CT thorax images Removes 70-80% of the data, reducing memory usage from 5-8 GB to 1-2 GB Used about 1.5 seconds Norwegian University of Science and Technology

33

Results ●

All steps except centerline extraction on the GPU



Used about 20 to 40 seconds –

[Bauer et al. 2009] reported 6 minutes

Norwegian University of Science and Technology

34

Blood vessels and multi-modality ●



Extended in article C to also extract blood vessels from CT, MRI and 3D ultrasound. Created a parallel centerline extraction method –



Entire method executed on GPU

Optimizations –

Texture memory



Normalized 16 bit integers Norwegian University of Science and Technology

35

Blood vessels from MR angio of the brain

Proposed GPU method + Parallel centerline extraction

Proposed GPU method + Ridge traversal

Norwegian University of Science and Technology

Thresholding + skeletonization

36

3D Ultrasound Doppler

Proposed GPU method + Parallel centerline extraction

Proposed GPU method + Ridge traversal

Norwegian University of Science and Technology

Thresholding + skeletonization

37

Liver vessels (CT)

Lung vessels (CT) Liver vessels (MR) Norwegian University of Science and Technology

38

Speed measurements 9 8 7

Runtime (seconds)

6 5 Parallel – 16 bit Parallel - 32 bit Ridge traversal – 16 bit Ridge traversal – 32 bit

4 3 2 1 0 CT Airways

MR Vessels

US Vessels

Dataset

AMD Radeon HD7970 GPU Norwegian University of Science and Technology

39

Accuracy ●

No loss in accuracy with 16 bit



Ridge traversal more accurate



Proposed parallel CE able to extract more with higher noise levels

Norwegian University of Science and Technology

40

Abdominal aortic aneurysms ●





Vascular disease resulting in a permanent local dilation of the abdominal aorta. These have non-circular cross section Developed a new TDF in article D to deal with noncircular tubular structures Norwegian University of Science and Technology

41

New tube detection filter ●

Line search in cross-sectional plane

Circle fitting TDF [Krissian et al. 2000] Norwegian University of Science and Technology

Proposed TDF

42

Abdominal aortic aneurysms ●

These aneurysms are also large



The traditional Euler GVF method converges slowly –



GPU method needed several minutes to converge

In article E, a GPU-based multigrid GVF method was proposed –

Converged in about 1 second

Norwegian University of Science and Technology

43

Region growing

Circle fitting TDF

Norwegian University of Science and Technology

Proposed TDF 44

Region growing

Circle fitting TDF

Norwegian University of Science and Technology

Proposed TDF 45

Segmentation of ultrasound images

Norwegian University of Science and Technology

46

Segmentation of ultrasound images ●

A Kalman filter state estimation pipeline was used for segmentation of ultrasound images in article F and G –

State describes the segmentation of the current image



Shape model



Appearance model



Temporal model

Norwegian University of Science and Technology

47

Left ventricle of the heart ●





Proposed a method for segmenting the left ventricle in 3D ultrasound [article F] A mesh was used as a shape model A control mesh and mean value coordinates [Ju et al. 2005] was used to deform the shape

Norwegian University of Science and Technology

48

Left ventricle of the heart ●



Edge detection was performed along the normal of each vertex on the mesh A step edge detection method was used for the appearance model, requiring the inside to be darker than the border.

Norwegian University of Science and Technology

49

Left ventricle of the heart ●



The state of the left ventricle consist of position, rotation, scaling and displacement vectors for each vertex in the control mesh Initialize in the middle of the image

Norwegian University of Science and Technology

50

Evaluation ●







Challenge on Endocardial Three-dimensional Ultrasound Segmentation (CETUS) MICCAI 2014 45 sequences in total The accuracy was compared to 8 other automatic and semi-automatic methods. The proposed method was ranked second in terms of clinical relevance of the fully automatic algorithms –

Mean mesh difference of about 2.72 mm



The best algorithm had a mean mesh difference of 2.35 mm Norwegian University of Science and Technology

51

Speed ●

The runtime was 65 ms on average per image frame –



The fastest method in the challenge

After the CETUS challenge, GPU acceleration was investigated –

10 ms



Parallel edge detection

Norwegian University of Science and Technology

52

Femoral artery ●



In article G, methods for tracking the femoral artery in ultrasound images was presented The tracked artery was used to register a model of the surrounding anatomy to the ultrasound images

Norwegian University of Science and Technology

53

Blood vessels ●





Ellipse used as shape model Appearance model expects a step or ridge edge State consists of position of the ellipse and the minor and major radius (4 values) Norwegian University of Science and Technology

54

Blood vessels ●





Need the position and radius to initialize the Kalman filter Developed an automatic initialization method which uses the GPU to do a search for black circles in the ultrasound image For every pixel in the image and for different radius calculate



Store position and radius with highest S.



If S is above a threshold, accept it Norwegian University of Science and Technology

55

Results ●





12 ultrasound image sequences from 3 subjects Speed –

Initialization: 42 ms



Tracking: 5 ms

Accuracy –

Mean absolute distance: 0.42 mm



Hausdorff distance: 1.17 mm. Norwegian University of Science and Technology

56

Revision ●

Vessel 3D reconstruction



Improved registration



Increased number of subjects (6) and acquisitions (48)

Norwegian University of Science and Technology

57

Conclusions ●





Parallel and GPU computing can significantly accelerate segmentation of medical images. A high-level framework for efficient and concurrent medical image computing and visualization on heterogeneous systems is needed. Good temporal, appearance and shape models are needed for automatic, robust and accurate segmentation of medical images. Norwegian University of Science and Technology

58

Future work ●

Continue to develop FAST



Segmentation of tubular structures





Anisotropic smoothing



Global optimization methods for the centerline extraction

Segmentation of ultrasound images –

Better shape and appearance models ● ● ●



Create better shape models with training data Block matching / speckle tracking Use both B-mode and Doppler when tracking vessels

Other applications ● ● ●

Right ventricle of the heart Brain ventricle Tracking multiple vessels Norwegian University of Science and Technology

59

References ●

















Aylward, S. R. and Bullitt, E. Initialization, noise, singularities, and scale in height ridge traversal for tubular object centerline extraction. IEEE Transactions on Medical Imaging 21:2. Pages 61-75. 2002. Darzi, Sir A. and Munz, Y. The impact of minimally invasive surgical techniques. Annual review of medicine 55. Pages 232-237. 2004. Erdt, M. Raspe, M. and Suehling, M. Automatic hepatic vessel segmentation using graphics hardware. Proceedings of the 4th international workshop on Medical Imaging and Augmented Reality. Pages 403-412. 2008. Bauer, C. Bischof, H. and Beichel, R. Segmentation of airways based on gradient vector flow. Proceedings of the 2nd International Workshop on Pulmonary Image Analysis. Pages 191-201. 2009. Helmberger, M. Urschler, M. Pienn, M. Bálint, Z. Olschewski, A. and Bischof, H. Pulmonary Vascular Tree Segmentation from Contrast-Enhanced CT Images. Proceedings of the 37th Annual Workshop of the Austrian Association for Pattern Recognition. Pages 1-10. 2013. Orderud, F. Hansgård, J. and Rabben, S. Real-time tracking of the left ventricle in 3D echocardiography using a state estimation approach. Medical Image Computing and Computer-Assisted Intervention (MICCAI). Part I. Pages 858-865. 2007. Guerrero, J. Salcudean, S. E. McEwen, J. A. Masri, B. A. and Nicolaou, S. Real-time vessel segmentation and tracking for ultrasound imaging applications. IEEE Transactions on medical imaging 26:8. Pages 1079-1090. 2007. Krissian, K. Malandain and G. Ayache, N. Model-Based Detection of Tubular Structures in 3D Images. Computer Vision and Image Understanding 80:2. Pages 130-171. 2000. Ju, T. Schaefer, S. and Warren, J. Mean Value Coordinates for Closed Triangular Meshes. ACM Transactions on Graphics 24:3. Pages 561-566. 2005.

Norwegian University of Science and Technology

60

Suggest Documents