Melanoma Detection Using Smartphone and Multimode Hyperspectral Imaging Nicholas MacKinnon a, Fartash Vasefi a, Nicholas Booth a, Daniel L. Farkas a, c,* a

c

Spectral Molecular Imaging Inc., 201 N. Robertson Blvd, Beverly Hills, CA, USA 90211 Department of Biomedical Engineering, University of Southern California, Los Angeles, CA, USA 90089 ABSTRACT

This project’s goal is to determine how to effectively implement a technology continuum from a low cost, remotely deployable imaging device to a more sophisticated multimode imaging system within a standard clinical practice. In this work a smartphone is used in conjunction with an optical attachment to capture cross-polarized and collinear color images of a nevus that are analyzed to quantify chromophore distribution. The nevus is also imaged by a multimode hyperspectral system, our proprietary SkinSpect™ device. Relative accuracy and biological plausibility of the two systems algorithms are compared to assess aspects of feasibility of in-home or primary care practitioner smartphone screening prior to rigorous clinical analysis via the SkinSpect. Keywords: Smartphone, Melanoma, mHealth, Nevus, Hyperspectral Imaging, Polarization Imaging

1. INTRODUCTION There exists a pressing need to pull people into the dermatology continuum of care during the earliest possible stages of melanoma occurrence and development. The most cost effective way to engage the general public is to empower people to manage their own health in a way that can be seamlessly integrated with more advanced clinical interventions. With the development of a range of digital health record systems, the infrastructure to facilitate this is now available. The most ubiquitous digital technology typically available to the individual is their smartphone (or tablet) and many health sensors and applications are already being developed for these devices. In the field of dermatology, the imaging capability of the smartphone is a natural way for dermatologists, general practitioners, or patients to exchange information about skin lesion changes that may be worrisome. Recently the camera systems embedded in smartphones have seen a host of improvements. This has been driven by the adoption of smartphone camera as a standard method of documentation in a variety of industries as well as for purely recreational use. These requirements have resulted in higher spatial resolution, improved sensor sensitivity, image stabilization, better optics allowing macro focusing, autofocusing, etc. capabilities. Consumer desire to share images has resulted in widespread network architectures able to facilitate seamless high quality image transfer and storage. As well as the development of new digital health record systems and technologies, the business practices in health care delivery are changing. Former individual clinical practices have become integrated in medical practice networks to deliver more comprehensive care and to efficiently manage costs and patient loads. 1.1 Current Smartphone Applications in Dermatology Smartphones are used in a variety of ways in dermoscopy and in health care in general. There are apps used for reference purposes such as medical textbooks, journal literature searches and drug compendiums [1]. Apps linked to electronic health records can provide immediate access to diagnostic and treatment protocols as well as record responses to protocol questions or measurements. They can provide patient education materials that can be viewed at the point of care or that can be printed or sent to a patient by email. As well as documentation, apps can provide measurement or diagnostic tools, and can facilitate remote consultations via telehealth systems and online consultations. There are a wide range of optical measurement and imaging technologies being applied to a range of dermatological issues. These often involve complex systems that have been expensive to develop and build. Yet a surprising number of these are Imaging, Manipulation, and Analysis of Biomolecules, Cells, and Tissues IX, edited by Daniel L. Farkas, Dan V. Nicolau, Robert C. Leif, Proc. of SPIE Vol. 9711, 971117 · © 2016 SPIE CCC code: 1605-7422/16/$18 · doi: 10.1117/12.2222415 Proc. of SPIE Vol. 9711 971117-1 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

actively being adapted by researchers to work with smartphones as can be seen in Table 1 below (the checkmark indicates commercial or research apps under development). While not all will become commercially viable we can expect to see many of these devices eventually reaching the market. Table 1. Optical measurement and imaging technologies used in clinical and research dermatology

Dermatology Tool

App for that

Dermoscopy [2][3]

   

Fluorescence imaging [4][5][6] Multi-wavelength imaging [19] Polarization imaging [7] Hyperspectral imaging Optical Coherence tomography[8]



Technology

App for that

Diffuse reflectance spectroscopy Raman spectroscopy [9] Microscopy [10] Confocal microscopy Multimode imaging Modulated imaging

    

1.2 Drivers of mHealth Adoption There are a number of drivers for the adoption of mobile technology. Ryan Beckland, CEO of Validic (mHealth platform company) said in 2014 that there were a number of digital health trends transforming health care and “creating synergies where there was once data silos and skepticism”. These include connecting smart clinical devices, incentivized adoption and data sharing with wearables (through insurance companies and employers), and interoperability and data integration among health systems driven by legislation, billing requirements like new ICD codes, telehealth extending the point of care, and capital investment and funding driven by the opportunities in these markets[18]. At the center of this is the consumer (aka the patient) who wants to use and take personal advantage of digital health technologies. 1.3 Barriers to Mobile Technology In a study published in the British Journal of Dermatology [11] of 39 melanoma apps, five claimed to do risk assessment, while nine referred images for expert review. The remainder fell into the education and documentation categories. This seems to be consistent with other dermatology apps on the market. In a study looking at barriers to adoption of mobile technology in health care [13] the researchers reviewed literature and came up with 12 barriers, detailed in Table 2, which were evaluated and ranked by key opinion leaders in health care. The highest priority issues were “integration & interoperability” and “business case”. Table 2. Barriers to adoption of mHealth technology in order of importance to key opinion leaders

Ranking

Category

Ranking

Category

1

Integration& Interoperability

7

Not adapted for physician

2

Business case

8

Lack of governance

3

Privacy & Security

9

Lack of Evidence

4

Technological obstacles

10

Conservative culture

5

Lack of access for patients

11

Competing payment mechanism

6

Legislation

12

Visionless development

Integration and interoperability issues include difficulty integrating mHealth apps with current health IT systems, lack of integration with other technology solutions, but primarily it is a problem of efficient integration with clinical work flows. These are all solvable problems and as effective solutions are developed apps that meet these needs will be adopted. A more serious problem exists with the small subset of apps that attempt to replace qualified clinical interpretation with lower cost, patient accessible alternatives. In a University of Pittsburgh study [12], Ferris et al tested 4 apps with 188 clinically validated images of skin lesions, of which 60 were melanomas. Three of four apps tested misclassified +30% of melanomas as benign. The fourth app relied on dermatologist interpretation and was much more accurate. This raises

Proc. of SPIE Vol. 9711 971117-2 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

questions about appropriate use of smartphones in the patient journey through diagnosis and treatment and how dermatologists can effectively engage with these tools. During our development of an advanced multimode hyperspectral imaging system for skin analysis (the SkinSpect™) we have looked at a range of devices and technologies, as well as their limitations and sources of error. By understanding these well and comparing the information can collect using smartphones we believe we can devise suitable systems for clinical deployment. A key question is whether apps for smartphones are intrinsically limited or if the problem is just poor implementation of a suitable measurement tool with ineffective or inappropriate software. In this work we begin an investigation into the quality of smartphone hardware and analytical software.

2. METHODS When we consider imaging measurements with smartphones, the first consideration is the image quality. By and large the image quality of current smartphones for ordinary color image capture exceeds that of many color cameras used in dermoscopy and other digital imaging studies over the past 20 years. Improvements in sensor technology and image acquisition software allow the capture of both high resolution, and high dynamic range images. Figure 1 shows an image of a skin blemish captured with a Google Nexus 5 Smartphone. This image was captured using the built in LED flash and has an image size of 8 megapixels. The enlarged detail (magnified without pixel smoothing), shows that pixel resolution and focus are very suitable for image analysis, even without magnification.

Figure 1. Imaging capability of unaugmented smartphone with image of skin blemish and a selected region magnified to show pixel resolution.

The image in Figure 1 is captured with ordinary 8-bit intensity resolution, but there are apps for most smartphones that also high dynamic range image capture via recomposition of sequential images captured with different exposure timing. Lens quality and the ability to focus at different distances via macro imaging, etc. varies among smartphones but is improving constantly. Methods to calibrate images to spatial coordinates and remove lens distortion have been and continue to be developed. [14] SkinSpect is a console based multimodal hyperspectral imaging system [15] able to capture fluorescence, hyperspectral and polarization images of tissue. Figure 2 shows a schematic representation of the system architecture, and Figure 3 shows the data collection capabilities of the system. The acquisition of 50 hyperspectral images, RGB images and fluorescence images in both parallel and cross polarization requires only seconds.

Proc. of SPIE Vol. 9711 971117-3 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

CONSOLE

LED

Skin

Fiber optics

Display (Windows Tablets )

sam.le

Lens Polarize

c Single board

computer (Linux)

LCVR

Emission filter

i

olarizer

Excitation filter

Illumination spectral.. selection device

(OneLight® )

Viewfinder

HANDPIECE

L

Figure 2. Optical and system architecture of SkinSpect.

Reflectance datacube : P- polarized RGB: P- polarized

mNuuuuuuuuuuI

RGB: X- polarized

Reflectance datacube : X- polarized Fluorescence: P and X- polarized

4' auuuuuummui

Figure 3. Data collection capabilities of SkinSpect: wavelength bands have been increased to 50 in the latest version.

The hyperspectral capability of SkinSpect allows the use of image analysis and segmentation algorithms that virtually eliminate the cross-talk between melanin and hemoglobin when trying to measure fundamental tissue constituents [15]. While RGB and polarization imaging have been used to attempt to quantify tissue constituents, they do not effectively manage cross-talk between hemoglobin and melanin. However they are representative of the algorithms used in image processing with this type of data and we have used them here for processing. We used a smartphone (iPhone 5s) to capture ordinary color images of a lesion using the integral flash attachment. We then attached wire grid polarizers (Edmund optics, Barrington, NJ) to the illumination and imaging channels and captured images in parallel and cross polarization modes. We applied the method of Kapsokalyvas et al. [16] in the equations shown below to process RGB color parallel and cross polarized images from cell phone (shown in Figure 5).





. 255 ,

Superficial melanin contrast =





Blood contrast =





,

.

3. RESULTS AND DISCUSSION When we examine the images of skin captured with a cell phone, as shown and described in the methods section above, it is clear that image quality is as good as many dermoscopes even without magnification. With the addition of magnifying

Proc. of SPIE Vol. 9711 971117-4 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

accessories and/or illumination conditioning accessories the image quality is comparable to many conventional dermoscopes. While image capture quality is not in question, algorithms to evaluate smartphone images remain a concern. Also of concern are accessories, both on the market and proposed in research trials to augment smartphones to capture images at multiple wavelengths, using magnifying optics, and by using thermal imaging, polarization imaging and fluorescence imaging. It is a goal of our research to use our multimodal device, SkinSpect to evaluate and compare the relative utility of these approaches After capturing images of a normal nevus with SkinSpect, our multimode hyperspectral imaging system, we processed them using two algorithms, one with melanin correction and one without melanin correction. These are shown in Figure 4 below. RGB (cross)

RGB (Parallel)

Total Melanin

Deep Melanin

Oxy- Hb (corrected)

Deoxy-Hb (Corrected)

Total Hb (corrected)

Oxygenation (corrected)

Total Hb (not corrected)

Oxygenation (not corrected)

Oxy- Hb (not corrected)

Deoxy -Hb (not Corrected)

Figure 4. Hyperspectral imaging of normal nevus showing ability to correct for melanin hemoglobin cross-talk.

The hyperspectral capability of SkinSpect allows the use of image analysis and segmentation algorithms that virtually eliminate the cross-talk between melanin and hemoglobin when trying to measure fundamental tissue constituents [15]. A system comprising of only three color bands, as with cell phone imaging, does not allow this level of effective tissue component decomposition. In Figure 6 below we show images processed using the method discussed above (Kapsokalyvas et al [16]) for linear and cross- polarized RGB images. It shows that processing fails to correct for hemoglobin/melanin cross-talk. If correction was implemented correctly, there would be no apparent blood contrast, since there is no change in total hemoglobin or hemoglobin oxygenation in a normal nevus. Since this problem exists for nonsmartphone RGB dermoscopy images it is not surprising the problem manifests here as well.

Proc. of SPIE Vol. 9711 971117-5 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

RGB (no pol)

RGB (cross)

RGB (parallel)

Superficial Mel

Melanin

Blood contrast

Figure 5. Images of same nevus in Figure 4 captured from a smartphone using conventional RGB imaging as well as using the same smartphone equipped with polarized illumination and parallel and cross polarized image capture.

While RGB images, even with polarization information, may not be suitable for measuring chromophores, are they suitable for other measurements? There have been many attempts to extract ABCD information using automated image segmentation and the development of proxy measurements for border irregularity and color. [20] Asymmetry and diameter measurements are relatively straightforward measurements if there is some calibration reference such as a ruler measurement of the lesion, or a distance positioning reference for the imaging device. The key challenge in all these measurements is accurate segmentation of the lesion from the normal tissue background to determine the lesion border. There have been many dermoscope investigations and a few smartphone apps that attempt to do this but few have had good success. Only a few have had good quality and understandable segmentation algorithms. Most develop some proxy features for ABCD values and then try to use clustering algorithms to associate features with normal or suspicious lesions. Some ignore the proxy features and instead use a batch of image texture, color and calculated features and attempt to develop classifiers using principal component analysis or clustering algorithms. Only some have reasonable sensitivity (Se) and specificity (Sp) results. We believe there is no intrinsic device-specific reason measurements of features used in checklists cannot be done effectively with a smartphone and reported to a clinician. Interpretation of these results is the next challenge. A trained practitioner estimates these features qualitatively and then applies additional context based information when performing an assessment. In one study context information was combined with automatic recognition and improved Se and Sp [17]. They used skin type, age, gender and anatomical location to improve results. With the public availability of the image database made available through the International Skin Imaging Collaboration: Melanoma Project, we may see deep learning projects or other techniques applied to image analysis and classification. As these efforts develop, it is important to remember that classification algorithms that are not tied to some underlying and explainable biological phenomena will be difficult to justify to both clinicians and regulatory bodies, and this could undercut the economic feasibility of implementing these in clinical practice. One interesting capability of automated image capture is the potential to implement multiple diagnostic protocols, such as ABCD-E, seven-point checklist [21], threepoint checklist [22], Menzies [23], and others simultaneously and present scoring results from all of these methods to the clinician for consideration.

Proc. of SPIE Vol. 9711 971117-6 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

In this work we have begun to compare images from smartphone dermoscopy to data from a multimode hyperspectral imaging system and assessed their relative ability to distinguish biological chromophores. Our future work will look at image segmentation accuracy using various wavelengths and polarization conditions to examine the potential and the limitations of smartphone imaging feature recognition related to accepted dermatology checklist criteria. We know that smartphones are already being used to transmit images for tele-dermatology. We know that patients want to engage with the healthcare system using their smartphones just as they do with many other aspects of their lives.

Figure 6. Ways in which the patient can interact with healthcare providers using smartphones in the dermatology setting.

Figure 6 is a graphic representation of possible patient interactions involving smartphone imaging with the dermatology network of care. A patient may be worried about a change in her skin. She could use a mobile application to send a picture to the primary care practitioner or to a teledermatologist for a second opinion. The practitioner could refer her to the specialty clinic for advanced testing with a device like SkinSpect or some other technology. The images she took can be part of her ongoing digital health record. If she is referred to a surgery, her smartphone imaging system can be used to monitor healing and communicate with her follow up care team. If the specialist wants to watch and wait smartphone can facilitate monitoring for change. Tools such as mHealth applications that let the patient interact more efficiently and effectively with the healthcare team are going to provide cost savings and may facilitate early detection.

4. CONCLUSION Smartphones and medical apps are here to stay. They can be integrated in practice as communication tools, health record tools, measurement tools or medical devices. Clinicians and patients need to use them appropriately by understanding their advantages and limitations and how they correlate and integrate with other systems. There are implications for both patients and clinicians regarding the utility of software image analysis with respect to both diagnosis and billing. We have shown an example of how skin with nevus can be measured and how some limitations of smartphones can affect measurement accuracy. We discuss where smartphone imaging maybe implemented effectively and how we can use our more sophisticated measurement system (SkinSpect™) to test and validate the hypothesis of potential smartphone based imaging systems and applications.

Proc. of SPIE Vol. 9711 971117-7 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

5. ACKNOWLEDGEMENT Spectral Molecular Imaging, Inc. (D.L. Farkas, PI) acknowledges support from the US Department of Health and Human Services (under the Qualifying Therapeutic Discovery Program of the Patent Protection and Affordable Care Act of 2010), and by the National Institutes of Health (under -NCI SBIR Grant # 1R44CA183169-01A1). We thank Patricia Vetter for the artwork in Figures 2 and 6.

REFERENCES [1] Mosa, A.S.M., Yoo, I. and Sheets, L., “A systematic review of healthcare applications for smartphones,” BMC medical informatics and decision making, 12(1), p.1 (2012). [2] Börve A, Terstappen K, Sandberg C, Paoli J. “Mobile teledermoscopy—there’s an app for that!,” Dermatology Practical & Conceptual. Apr; 3(2):41(2013). [3] Dermlite App: http://dermlite.com/pages/apps-links (Accessed 03/17/2016) [4] Yu H, Tan Y, Cunningham BT. “Smartphone fluorescence spectroscopy,” Analytical chemistry. Aug 13; 86(17):8805-13 (2014). [5] Zhu H, Mavandadi S, Coskun AF, Yaglidere O, Ozcan A. “Optofluidic fluorescent imaging cytometry on a cell phone,” Analytical chemistry.; 83(17):6641-7 (2011). [6] Lihachev A, Derjabo A, Ferulova I, Lange M, Lihacova I, Spigulis J. “Autofluorescence imaging of basal cell carcinoma by smartphone RGB camera,” Journal of Biomedical Optics. 20(12):120502- (2015). [7] Zouridakis G, Wadhawan T, Situ N, Hu R, Yuan X, Lancaster K, Queen CM. “Melanoma and other skin lesion detection using smart handheld devices,” Mobile Health Technologies: Methods and Protocols.459-96 (2015). [8] Subhash HM, Hogan JN, Leahy MJ. “Multiple-reference optical coherence tomography for smartphone applications,” SPIE Newsroom (2015). [9] Ayas S, Cupallari A, Ekiz OO, Kaya Y, Dana A. “Counting molecules with a mobile phone camera using plasmonic enhancement,” Acs Photonics. 1(1):17-26 (2013). [10] Massone C, Hofmann-Wellenhof R, Ahlgrimm-Siess V, Gabler G, Ebner C, Soyer HP. “Melanoma screening with cellular phones,” PloS one. 2(5):e483 (2007). [11] A.P. Kassianos, J.D. Emery, P. Murchie, and F.M. Walter, “Smartphone applications for melanoma detection by community, patient and generalist clinician users: a review,” British Journal of Dermatology, 172(6), 1507– 1518, (2015). [12] Wolf JA, Moreau J, Akilov O, et al. “Diagnostic inaccuracy of smartphone applications for melanoma detection,” JAMA Dermatol. 149(4):422-426 (2013). [13] Dehzad F, Hilhorst C, de Bie C, Claassen E. “Adopting Health Apps, What’s Hindering Doctors and Patients?,” Health. 2014 (2014). [14] Akhbardeh F, Vasefi F, Tavakolian K, Bradley D, Fazel-Rezai R. “Toward development of mobile application for hand arthritis screening,” In Engineering in Medicine and Biology Society (EMBC), 37th Annual International Conference of the IEEE 2015 Aug 25 (pp. 7075-7078). IEEE (2015). [15] Vasefi F, MacKinnon N, Saager RB, Durkin AJ, Chave R, Lindsley EH, Farkas DL. “Polarization-sensitive hyperspectral imaging in vivo: a multimode dermoscope for skin analysis,” Scientific reports. 4 (2014). [16] Kapsokalyvas, D., et al. “Spectral morphological analysis of skin lesions with a polarization multispectral dermoscope,” Opt Express, 21, 4826-4840 (2013) [17] Alcón JF, Ciuhu C, Ten Kate W, Heinrich A, Uzunbajakava N, Krekels G, Siem D, De Haan G. “Automatic imaging system with decision support for inspection of pigmented skin lesions and melanoma diagnosis,” Selected Topics in Signal Processing, IEEE Journal of. 3(1):14-25 (2009). [18] Health IT outcomes: http://www.healthitoutcomes.com/doc/the-road-ahead-in-digital-health-0001 [19] Verisante Inc. : http://www.verisante.com/newsletters/Verisante_MSI_Newsletter.pdf [20] Rigel DS, Friedman RJ, Kopf AW, Polsky D. “ABCDE—an evolving concept in the early detection of melanoma,” Archives of dermatology.;141(8):1032-4 (2005). [21] Healsmith MF, Bourke JF, Osborne JE, GRAHAM‐BROWN RA. “An evaluation of the revised seven‐point checklist for the early diagnosis of cutaneous malignant melanoma,” British Journal of Dermatology.;130(1):48-50 (1994). [22] Soyer HP, Argenziano G, Zalaudek I, Corona R, Sera F, Talamini R, Barbato F, Baroni A, Cicale L, Di Stefani A, Farro P. “Three-point checklist of dermoscopy. Dermatology,”208(1):27-31 (2004). [23] Menzies SW, Ingvar C, Crotty KA, McCarthy WH. “Frequency and morphologic characteristics of invasive melanomas lacking specific surface microscopic features,” Archives of Dermatology.132(10):1178-82 (1996).

Proc. of SPIE Vol. 9711 971117-8 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/06/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx