An ASAE Meeting Presentation Paper Number: 053017
Citrus Fruit Identification and Size Determination Using Machine Vision and Ultrasonic Sensors Murali Regunathan, Graduate Research Assistant (E-mail:
[email protected]) Won Suk Lee, Assistant Professor (E-mail:
[email protected]) Agricultural and Biological Engineering, University of Florida, Gainesville FL 32611
Written for presentation at the 2005 ASAE Annual International Meeting Sponsored by ASAE Tampa Convention Center Tampa, Florida 17 - 20 July 2005 Abstract Machine vision and image processing have been increasingly used for agricultural applications especially for detection of crop status and quality, and crop harvesting. Estimating the size of the fruit allows growers to get a good idea of the quality of yield. A system using a camera and an ultrasonic sensor is proposed that will enable one to estimate the average size of citrus fruits. Images of the fruit laden trees were obtained using a color camera, and the distance between the fruits and the camera were obtained using ultrasonic sensors. Actual size of a sample fruit in each image was measured manually on the tree. Three different classification techniques, Bayesian, neural network and Fischer’s linear discriminant were implemented to differentiate fruit from the background in the images, using hue and saturation as the separation features. The classified images were then processed to remove noise, to fill the blobs, and to separate the blobs using watershed transformation. These segmented images were then analyzed to find the diameter of the fruit in the images. Using basic trigonometry, the camera’s field of view and the distance information from the ultrasonic sensors, actual size of fruit in an image was estimated. Results from the three classifiers were compared in terms of accuracy with the actual measured fruit size. Keywords. Citrus, Fruit size, Identification, Image processing, Machine vision, Ultrasonic sensor. The authors are solely responsible for the content of this technical presentation. The technical presentation does not necessarily reflect the official position of the American Society of Agricultural Engineers (ASAE), and its printing and distribution does not constitute an endorsement of views which may be expressed. Technical presentations are not subject to the formal peer review process by ASAE editorial committees; therefore, they are not to be presented as refereed publications. Citation of this work should state that it is from an ASAE meeting paper. EXAMPLE: Author's Last Name, Initials. 2005. Title of Presentation. ASAE Paper No. 05xxxx. St. Joseph, Mich.: ASAE. For information about securing permission to reprint or reproduce a technical presentation, please contact ASAE at
[email protected] or 269-429-0300 (2950 Niles Road, St. Joseph, MI 49085-9659 USA).
Introduction Every farmer strives to increase profits and reduce input costs. To achieve the goal, each crop production input has to be managed effectively. Precision agriculture allows farmers to use technology to manage their farms on a site-specific basis and use fertilizers, seeds, and pesticides more effectively. Currently precision agriculture is widely used for corn, wheat, soybean, and other grain crops. Citrus is one of the most important crops in Florida. Florida contributes to more than 74% of the total citrus grown in the United States. The on-tree crop value of citrus was estimated to be $746 million in 2004 (Florida Agricultural Statistics Services, 2005). It is essential to apply precision agriculture techniques to citrus and thereby reduce costs and increase profits. Yield mapping is the first logical step in the precision agriculture cycle, where the main focus of this research lies. Though citrus yield monitors have been developed (Schueller et al., 1999), they only measure the yield once the fruit has been harvested. Machine vision has been used to detect fruit on the tree and selectively harvest them thereby raising the quality of fresh produce, lowering production costs, and reducing the drudgery of manual labor. It was also used effectively for determining the number of fruit (Annamalai et al., 2004). A variety of classifiers and many different features have been used to identify the fruit from the background. Parrish and Goksel (1977) used monochrome images enhanced by color filters to identify apples and to harvest them. A thinness ratio, R = 4πA/P2, where P is the perimeter and A is area of the blobs, was used to distinguish between noise, clusters of fruits and single fruits. Sites and Delwiche (1985) used smoothing and connectivity segmentation of the binarized images. They used a linear discriminant function and a centroid nearest neighbor classifier. Perimeter, area, and elongation were the features used for classifying the blobs into single and multiple fruit. Color images were used for distinguishing the citrus fruit by setting a threshold in the hue value (Slaughter and Harrell, 1987). The threshold in hue was found using spectral reflectance curves of citrus fruit. Grasso and Recce (1996) described classification based on chromatic information mapped into a new two-dimensional space. Pla et al. (1993) used the spherical property of citrus fruits to distinguish them from the leaves. They used the double differential operator on the image to find out the concavity of each pixel. Though many other features have been used to address the problem of citrus fruit identification, color still remains one of the most used features. The feature is simple to use and gives adequately good results. Apart from extracting different features to identify the fruit, different classifiers like neural network based classifier (Molto et al., 1992) and Bayesian (Slaughter and Harrell, 1989) have also been used. Marchant and Onyango (2003) described the performance of a Bayesian classifier as compared to a neural network based classifier in plant/weed/soil discrimination in color images. They used a probability distribution formed out of the training data and reported that given enough memory, the Bayesian classifier performed equally well as the neural network classifier. Variable illumination is another major issue in machine vision applications in natural daylight conditions. An environmentally adaptive algorithm was used to classify weeds from plants and soil (Tian and Slaughter, 1998). To find out the size of the fruit in the images, the distance from the tree to the camera should be known. A laser ranging system was used to find the distance of the end effector of the harvesting robot from the tree (Bulanon et al., 2004; Jimenez et al., 2000). Using stereo cameras, range information could be also found, however it is computationally intensive.
2
The objective of this research was to use machine vision and image processing techniques to identify fruit and estimate the number of fruit from each tree, to compare the use of three different classifiers to identify the citrus fruit from the background, and also to estimate the fruit size using range information gathered by the ultrasonic sensors. Since fruit quality is as important as quantity, fruit size is important to know as a good parameter of fruit quality.
Materials and methods Data acquisition Seventy-four images of the citrus trees, laden with fruits, were obtained using a camera (model FCB-EX780S, Sony, New York, N.Y.). All the images were obtained in a stationary mode with each image having 640 x 480 pixels. The brightness, contrast, shutter speed, and aperture of the camera were kept constant most of the time during imaging. The images were obtained on four different days in an experimental citrus grove in the University of Florida in natural daylight condition during the months of March and April 2005. The fruit variety was Valencia. Four ultrasonic sensors (model Mini-A, Senscomp, Livonia, Mich.) were used to estimate the distance between the camera and the tree. The ultrasonic sensors were calibrated before they were used. The maximum diameter of fruit acquired in images was used as the parameter to represent fruit size. Actual fruit size corresponding to an image was measured with calipers by taking the average of the maximum diameter of the two fruits which were part of the image. Seventy-four images were used for fruit identification (37 for training and 37 for testing) and 32 images were used for size estimation (16 for training and 16 for testing).
Image processing The images were converted from a red, green, and blue (RGB) format to a hue, luminance, and saturation (HLS) format, since the HLS format yielded better results in the preliminary investigation. The hue and saturation values of every pixel were used to classify the pixels into citrus fruit and background (sky, leaves, branches, ground, etc.). The images were segmented using three different classifiers, which are explained later in this section. The segmented images were then processed to remove noise and fill holes using the process of erosion and dilation. Watershed transformation was applied to the processed images to separate any touching blobs. The images were then analyzed to give the count of fruits in the image and maximum diameter (D), area (A), and perimeter (P) of each of the blobs. The representative fruit diameter of each image was chosen on the basis of two features: roundness of the blob and its area. The average (Avg) and standard deviation (SD) of the area (A) of all the blobs were calculated. Roundness of each blob was determined by using the ratio, A/P. For a circle, this ratio should be equal to its diameter divided by four (E = D/4). Hence if the ratio (A/P) was between E – F and E + F, then the blob was assumed to be round. The quantity F was determined from the 16 training images which was found to be about 0.3E. It was the smallest value that gave correct diameter information. The round blobs whose area was not in the range between (Avg – SD) and (Avg + SD) were eliminated. This eliminated all those blobs whose area was too small or too large. The maximum diameter value from the remaining blobs was selected as the representative fruit diameter for that image.
3
Classifier development Three different classification techniques were used to classify each of the pixels into citrus fruit and background: Bayesian classifier, neural network classifier and Fischer’s linear discriminant. Four thousand pixels of fruit and four thousand pixels of background regions from each of the 37 training images were manually selected and their hue and saturation values were stored. Since the probability distribution of the hue and saturation values could take any form, instead of assuming a particular distribution (e.g. Gaussian) and finding out the parameters of the distribution, the 148,000 (4,000 pixels x 37 images) pixel values were used to construct the class conditional probabilities for fruit and background class separately. To simplify the analysis, it was assumed that pixels from one class (fruit or background) did not give any information about the other class. The hue and saturation values of the pixels ranged from 0 to 255. This range was divided into 64 cells for both hue and saturation, resulting in a total of 4,096 cells, each cell having a width of 4 x 4 pixel values. The cell size was chosen in order to maximize the segmentation. Each cell had an associated value, which was increased by one whenever a pixel’s hue and saturation value fell in the cell. All the cell values were then divided by the total number of pixels to give a discrete histogram. After the conditional probabilities had been formed, the following discriminant functions were used for classification: gf(x) = p(x|wf) * P(f) gb(x) = p(x|wb) * P(b)
(1)
where, gf(x) is the discriminant function for the fruit class, and gb(x) for the background class; p(x|wf) and p(x|wb) are the class conditional probabilities found above, and P(f) and P(b) are the a priori probabilities for fruit and background chosen as 0.25 and 0.75, respectively. If gf(x) > gb(x) for a pixel, then the pixel was classified as fruit; otherwise the pixel was classified as background. It was assumed that if the class conditional probabilities for both background and fruit were zero, then the pixel belonged to the background class. A four-layered neural network was chosen as a neural network classifier. This four-layer network was chosen, since the neural network could not converge easily with three layers. The input layer had two nodes, one each for hue and saturation of the pixel, and the output layer had two nodes for each of the two classes (fruit and background). The first hidden layer had ten nodes and the second layer had four nodes both having a sigmoidal activation function. The network was trained using backpropagation with a learning rate of 0.1. The trained network was then used to classify each pixel of the testing images into fruit and background. The number of nodes in the hidden layers could be reduced further with a compromise in performance, however with the advances in memory and processing speed, it would not be essential. The Neural Network Toolbox (MATLAB, Natick, MA) was used for this analysis. For the Fischer’s linear discriminant, the hue and saturation data was projected onto a line thereby converting the data into a one-dimensional space. This is the converse of the neural network approach where the data is transformed into a higher dimensional nonlinear space. The slope of the line on which the data was projected was chosen such that the projected data is readily classified into the two classes. The slope of the line was chosen using the training data set as follows: Slope = w(2)/w(1)
(2)
4
where a two-element matrix, w, is found by equation 3, w(1) is the first element of the matrix w and w(2) is the second element. w = (Sf + Sb)-1(mf - mb)
(3)
where mf is the mean of the data for fruit class, and mb is the mean of the data for the background class. Sf is the scatter for the fruit class, and Sb for the background class calculated as Si = Σj (xj - mi) * (xj - mi)T
(4)
where x is the two dimensional data (hue and saturation) for class i. A threshold point on the line was obtained by visually inspecting the projected points of the two classes. The threshold point was chosen such that most of the points were well separated. Once the slope of the line and a threshold were obtained from the training data, each pixel of the testing images was projected onto the line and classified using the threshold.
Results and discussion Bayesian classifier The probability distribution for the citrus fruit had two sharp peaks and a Gaussian like distribution for low hue values as seen in figure 1. The rest of the cells had mostly zero. The class conditional probability for the background (figure 2) had wider distribution than that of the citrus fruit, though the value of each cell was lower in general. The peaks in figure 2 corresponded to the leaf and the ground pixels.
0.025
0.02
0.015
0.02-0.025 0.015-0.02 0.01-0.015 0.005-0.01 0-0.005
Probability 0.01
0.005 Hue
0 1 6 11 16 21 26 31 36 41 46 51 56 61 Saturation
Figure 1. Class conditional probabilities for citrus fruit class.
5
0.01 0.009 0.008
0.009-0.01 0.008-0.009 0.007-0.008 0.006-0.007 0.005-0.006 0.004-0.005 0.003-0.004 0.002-0.003 0.001-0.002 0-0.001
0.007 0.006 Probability 0.005 0.004 0.003 0.002 0.001 Hue
0 1 6 11 16 21 26 31 36 41 46 51 56 61 Saturation
Figure 2. Class conditional probabilities for background class.
Fischer’s linear discriminant The slope of the line on which the pixels’ hue and saturation values were projected on was found to be -0.66 from the training set. The threshold point obtained was (-20, 13.1). Any projected point that lies below this point on the line (i.e., less than -20 and greater than 13.1) was classified as fruit. Otherwise it was classified as background. Figure 3 shows all the pixels’ hue and saturation values and their respective values projected on the line. All the fruit pixels are shown in orange color, while the background pixels are shown in green color. The purple line is a line perpendicular to the line on which the points are projected and the purple line passes through the threshold value. All the points to the left of the purple line will be classified as citrus fruit and all the points to the right as background.
6
Threshold point
Projected data points
Figure 3. Hue and saturation values of citrus fruit and background pixels and their projected values.
Citrus fruit identification Sample testing images before and after processing are shown in figures 4 and 5. The images have been processed with binarization after classification of the objects, noise removal, gap filling, and watershed segmentation. The difference between the various classifiers can be seen in the processed images.
(a) Original image
(b) Bayesian classifier
7
(c) Neural network classifier
(d) Fischer’s discriminant
Figure 4. Sample test image and fruit identification results by different classifiers.
(a) Original image
(c) Neural network classifier
(b) Bayesian classifier
(d) Fischer’s discriminant
Figure 5. Sample test image and fruit identification results by different classifiers.
As seen from the sample images, the neural network classifier misclassified some fruits as background but didn’t misclassify the background as fruit. This reduced the count of the fruits as given by the algorithm. The Fischer’s discriminant misclassified the background as fruit in certain cases.
8
Count of fruits The three different classifiers were tested on 37 testing images. The number of fruits visible in each image was counted manually and compared with the number of fruits as given by the recognition algorithm. Table 1 shows the fruit count obtained from the algorithm using the various classifiers. Table 1. Comparison of the number of fruits given by algorithm using various classifiers. Image Manual number count Fruit count by algorithm Percent error Neural Fischer's Neural Fischer's Bayesian network discriminant Bayesian network discriminant 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
45.0 38.0 27.0 35.0 51.0 25.0 31.0 48.0 31.0 63.0 35.0 10.0 21.0 52.0 48.0 43.0 57.0 31.0 37.0 37.0 44.0 27.0 30.0 37.0 50.0 54.0 38.0 43.0 27.0 30.0 33.0 42.0 40.0 41.0 20.0
44.0 37.0 45.0 27.0 41.0 31.0 22.0 41.0 40.0 75.0 39.0 17.0 26.0 48.0 55.0 46.0 36.0 126.0 138.0 34.0 67.0 56.0 73.0 25.0 49.0 52.0 47.0 50.0 28.0 28.0 25.0 47.0 41.0 44.0 24.0
48.0 38.0 57.0 61.0 61.0 28.0 50.0 43.0 50.0 79.0 46.0 38.0 41.0 55.0 39.0 43.0 84.0 45.0 59.0 38.0 82.0 54.0 57.0 34.0 37.0 55.0 45.0 44.0 42.0 48.0 34.0 58.0 40.0 46.0 31.0
30.0 70.0 77.0 152.0 194.0 26.0 57.0 52.0 56.0 104.0 36.0 16.0 39.0 48.0 50.0 38.0 47.0 68.0 49.0 31.0 130.0 71.0 73.0 26.0 44.0 62.0 60.0 54.0 60.0 104.0 59.0 88.0 42.0 52.0 50.0
-2.2 -2.6 66.7 -22.9 -19.6 24.0 -29.0 -14.6 29.0 19.0 11.4 70.0 23.8 -7.7 14.6 7.0 -36.8 306.5 273.0 -8.1 52.3 107.4 143.3 -32.4 -2.0 -3.7 23.7 16.3 3.7 -6.7 -24.2 11.9 2.5 7.3 20.0
6.7 0.0 111.1 74.3 19.6 12.0 61.3 -10.4 61.3 25.4 31.4 280.0 95.2 5.8 -18.8 0.0 47.4 45.2 59.5 2.7 86.4 100.0 90.0 -8.1 -26.0 1.9 18.4 2.3 55.6 60.0 3.0 38.1 0.0 12.2 55.0
-33.3 84.2 185.2 334.3 280.4 4.0 83.9 8.3 80.6 65.1 2.9 60.0 85.7 -7.7 4.2 -11.6 -17.5 119.4 32.4 -16.2 195.5 163.0 143.3 -29.7 -12.0 14.8 57.9 25.6 122.2 246.7 78.8 109.5 5.0 26.8 150.0
9
36 37 Average Standard Deviation
49.0 23.0 37.6
40.0 24.0 45.6
49.0 38.0 48.6
84.0 90.0 64.6
-18.4 4.3 27.2
0.0 65.2 39.6
71.4 291.3 81.2
11.4
25.0
13.1
35.9
73.6
54.7
96.0
The root mean square error for the three classifiers was: Bayesian 4.2, neural network 2.6, and Fischer 7.2. The error in the various classifiers for fruit count is slightly high. The average fruit count given by all three classifiers is higher than the average manual count. This could be due to the errors introduced by over-segmentation by the watershed algorithm.
Size of fruits Size information was available for 32 different images. Sixteen of these were used for training and the rest for testing. The size estimates are shown in table 2. The average estimate given by the Bayesian classifier was very close to the actual average diameter. The standard deviation was high in all three classifiers, the neural network classifier performing better than the others. Table 2. Comparison of the actual fruit size and fruit size estimated by the algorithm. Actual Image size Diameter estimate from algorithm Percent error (cm) number (cm) Neural Fischer’s Neural Fischer's Bayesian Network discriminant Bayesian network discriminant 1 7.6 7.5 7.0 7.2 -1.5 -8.4 -5.7 2 8.1 10.1 7.8 9.9 23.8 -4.5 21.6 3 7.5 10.0 7.6 5.9 33.7 1.4 -21.2 4 6.9 9.5 7.4 7.6 39.2 8.4 11.5 5 7.4 6.9 6.0 6.2 -6.6 -18.2 -16.2 6 7.4 4.0 7.1 5.8 -46.4 -3.8 -21.3 7 7.4 3.8 3.8 6.7 -48.5 -48.5 -8.5 8 6.6 11.2 9.0 8.8 70.0 36.2 33.4 9 6.9 9.0 3.0 2.0 31.2 -56.8 -70.7 10 7.1 7.6 9.3 7.0 6.8 30.6 -0.9 11 7.1 9.7 8.5 11.0 35.8 20.1 54.8 12 6.6 2.6 6.4 5.2 -60.6 -3.2 -20.6 13 6.6 5.4 8.7 5.2 -18.5 31.8 -20.8 14 7.4 9.4 7.9 7.8 27.0 7.2 5.8 15 7.6 6.4 5.6 5.3 -16.2 -26.6 -30.9 16 7.1 5.4 4.5 2.5 -24.7 -36.3 -65.0 Average 7.2 7.4 6.9 6.5 2.8 -4.4 -9.7 Standard deviation 0.4 2.6 1.9 2.3 35.8 26.8 31.2
The root mean square error for the three classifiers was: Bayesian 0.5, Fischer 0.6, neural network 0.4. The errors in fruit size estimation could be caused by inaccuracies obtained in ultrasonic range measurements. As the canopy structure of the tree was non uniform, the
10
ultrasonic waves bounced off the leaves in different places giving rise to errors in range measurement, which were in turn translated to errors in fruit size estimation. Partial occlusion of fruits, variable illumination, and clustering of fruits produced errors in both the fruit count and size estimates.
Conclusion A comparison of three different classifiers for citrus fruit identification and fruit size estimation was conducted in this research. The neural network classifier produced better results for fruit size estimation. The root mean square error for the neural network classifier for both fruit size estimation (0.4 cm) and fruit count (2.6) was lesser than the other two classifiers. The errors in fruit size estimation could be caused by inaccuracies obtained in ultrasonic range measurements. Partial occlusion of fruits, variable illumination, and clustering of fruits produced errors in both the fruit count and size estimates.
Acknowledgement The authors would like to thank to Mr. G. H. Pearson and M. Zingaro for their assistance in the project, and the Florida Citrus Production Research Advisory Council for funding this research.
References Annamalai, P., W. S. Lee, and T. Burks. 2004. Color vision system for estimating citrus yield in real-time, ASAE Meeting Paper No. 043054. St. Joseph. Mich.: ASAE. Bulanon, D. M., T. Kataoka, H. Okamoto, and S. Hata. 2004. Determining the 3-D location of the apple fruit during harvest Automation Technology for Off-Road Equipment, In Proceedings of the 7-8 October 2004 Conference (Kyoto, Japan): 701P1004. Florida Agricultural Statistics Service (FASS). 2005. Citrus summary 2003-04. Orlando, FL. Available at http://www.nass.usda.gov/fl/rtoc0.htm. Accessed 20 April 2005. Grasso, G. M., and M. Recce. 1996. Scene analysis for an orange picking robot, Proceedings for 6th International Congress for computer technology in agriculture, (ICCTA'96) C. Lokhorst, A.J. Udink ten Cate, and A.A. Dijkhuizen, VIAS, The Netherlands, pp 275-280. Jimenez, A. R., R. Ceres, J. L. Pons. 2000. A vision system based on a laser range-finder applied to robotic fruit harvesting. Machine Vision and Applications. 11: 321–329. Marchant, J. A., and C. M. Onyango. 2003. Comparison of a Bayesian classifier with a multilayer feed-forward neural network using the example of plant/weed/soil discrimination. Computers and Electronics in Agriculture. 39: 3-22. Molto, E., F. Pla, and F. Juste. 1992. Vision systems for the location of citrus fruit in a tree canopy. Journal of Agriculture Engineering Research. 52: 101-110. Parrish Jr., A. E., and A.K. Goksel. 1977. Pictorial Pattern recognition Applied to fruit harvesting. Transactions of the ASAE. 20(5): 822-827. Pla, F., F. Juste, and F. Ferri. 1993. Feature extraction of spherical objects in image analysis: an application to robotic citrus harvesting. Computers and Electronics in Agriculture. 8: 57-72. Schueller, J. K., J. D. Whitney, T. A. Wheaton, W. M. Miller, and A.E. Turner. 1999. Low-cost automatic yield mapping in hand-harvested citrus. Computers and Electronics in Agriculture. 23(2): 145-154.
11
Sites, P. W., and M. J. Delwiche. 1985. Computer vision to locate fruit on a tree. ASAE Paper No. 853039. St. Joseph, Mich.: ASAE. Slaughter, D. C., and R. C. Harrell. 1987. Color vision in robotic fruit harvesting. Transactions of the ASAE. 30(4): 1144-1148. Slaughter, D. C., and R. C. Harrell. 1989. Discriminating Fruit for Robotic Harvest Using Color in Natural Outdoor scenes. Transactions of the ASAE. 32(2): 757-763. Tian, L. F., and D. C. Slaughter. 1998. Environmentally adaptive segmentation algorithm for outdoor image segmentation. Computers and Electronics in Agriculture. 21: 153-168.
12
