The artificial sensor head: A new approach in assessment of human based quality. Peter Wide Ivan Kalaykov
The Swedish Sensor Center
Applied Autonomous Sensor Systems laboratory
Linköping University
Department of Technology and Science
Sweden
Örebro University
Email:
[email protected]
Fredrik Winquist
Sweden Email:
[email protected] [email protected]
Abstract
phenomenon taken in consideration. The information
The design of a new electronic sensor head using artificial
proceeds usually from two types of sensor models
senses is described. The system involves a chewing process
[1], consolidating data from the same type of
that mimics the human behavior. Before entering the test
information [2], and in the second case, usually
sample in the artificial mouth, the sensor system uses a
named multi-sensor data fusion, merging informa-
video camera to identify the test object. The artificial
tion from different and often complementary sensors
sensor mouth is then measuring the crushing and chewing
to create a environmental based sensor model [3].
process of the samples, mixing it with saliva like liquid. In
We have focused in this approach on a sensor model
parallel it measures the aroma with an electronic nose,
using combination of data information from five
detect the chewing resistance and listens to the crushing
different sensor systems measuring the quality of a
sound. Further, the taste of the mixed solution from the
food product, and more specific an integration of
sample is measured with an electronic tongue sensor.
multiple sensing data in human quality applications. A number of single artificial sensors have been
To the amount of information received, we apply feature
described in different human based quality related
extraction analysis and a fuzzy clustering to assess the
applications, electronic nose [4, 5], electronic tongue
quality. By combining data from different artificial sensor
[6] and in the chewing process [7, 8]. Further, a
systems into a single set of meaningful features, we achieve
combination of the information from artificial smell
information that is of greater benefit than the aggregate of
and taste sensor systems into a merged opinion has
its contributing sensors. The combination of sensor data by
been reported [9].
fuzzy clusters has the aim to perform inferences that may be impossible from the single artificial sensors.
I. Introduction
II. Human analysis of quality estimation The operator in an industrial food process, for example potato chips plant, continuously analyzes the
The combining of data into more meaningful
dynamical process properties, e.g. temperature,
information refers to an essential technology in the
humidity, time, sound, etc. as well as the specific
problem of the information treatment to improve the
product quality like color, size, taste, smell, along the
quality of the sensing data. Data fusion uses various
process line. In the laboratory, tests are regularly
data sources to provide a better understanding of the
made to measure parameters as concentration of salt,
In this paper we propose to combine artificial sensors
color, water content and percentage of fat, a visual
into an electronic sensor head approach containing a
inspection is provided as well.
number of sensor systems that measure essential properties of the tested object, as shown in figure 1.
III. The artificial approach
A. The electronic head
There is a change in the attitude within measurement technology towards the way of and how to collect process information. Instead of measuring single parameters, in many cases it has become more desirable to get information of attributes such as quality, condition or state of a process. Due to different available techniques of extracting human like
features
from
a
huge
information
flow
mimicking the human perception, there is a growing interest in the concept of artificial senses.
artificial sensors feature
vision
extraction chewing test fuzzy clusters
olfaction Environment
object
auditory
A special artificial mouth with hearing and vision capabilities, i.e. an artificial sensor head, is designed and tested in the laboratory. Stationary robot arm feeds the mouth with test samples after the vision cameras has recorded the object. In the mouth, with a temperature of 37 °C kept inside, a crushing process takes place that is similar to human chewing. In parallel the crushing sound and chewing resistance are recorded and the developed aroma pumped from the mouth to the measuring electronic nose. The chewed pieces of the sample object are further mixed with saliva like fluid and the electronic head spits the rest into a cell where the electronic tongue is measuring the taste. After this moment the cell containing the sample test is cleaned up and the system is ready to measure a new sample. The result is presented for visual acceptance on the monitor indicated by the mode of a happy or sad human face. The electronic head system is controlled by a PLC (Programmable Logic Control) pneumatic system and interacting with the measurement PC operating under LabView software.
quality decision
taste
B. The artificial electronic nose The sensor array consists of a number of selective
Figure 1. The model of the artificial sensor head.
semiconductor metal oxide (Taguchi) type sensors, obtained from Figaro Engineering Inc., Japan. The
Although the combination of artificial senses most
measurement interface was built at the laboratory.
likely increases the performance of the measurement,
Gas samples are pumped from the mouth cell by a
articles in this area are lacking. In [7] and [8] an
membrane pump at a flow rate of approximately 500
electronic mouth is described. In [9] and [10] original
ml/min and injected into the sensor chamber, where
sensor fusion methods based on human opinions
the sensors are placed in a row. The injection of gas
about smell and taste and measurement data from
samples is performed at given time intervals by the
artificial nose and tongue sensors is presented.
opening a valve. Thus, samples are injected during
the chewing process.
of 800 mV during 0.5 sec. The voltage is then set to 0 at the instant, when the applied potential is decreased by 100 mV, and the cycle starts again. A
Response from one nose sensor
measurement sequence covers 11 cycles, which
4,9
y = -2E-05x + 4,7842
4,85
results in a final pulse value of -200 mV, see figure 3.
4,8 4,75
Titel: m:\pcsa\research\tongue\docs\tsig.eps Skapad av: MATLAB, The Mathworks, Inc. Förhandsgranska: Den här EPS-bilden sparades inte med en inkluderad förhandlsgranskning. Beskrivning: Den här EPS-bilden kan skrivas ut på en PostScript-skrivare, men inte på andra typer av skrivare.
4,7 4,65 4,6 4,55 4,5 0
2000
4000
6000
8000
10000
12000
Figure 2. A calculated feature from one of the nose sensors. Further, the sensor system collects and preprocesses the data from the sensor array. Each of the foursensor data measurement used in this approach
Figure 3. A series of pulses applied to a tongue electrode during a measurement sequence.
contains 4 variables to be further analyzed where one
A typical recording of a full measurement over all
parameter is the derivative shown in figure 2.
electrodes is shown in figure 4. The sample rate is set to 20 Hz and only the amplitudes which has shown to contain sufficient information, namely from the first,
C. The artificial tongue
second and last samples in each 0.5-second interval, The principle for measurement was based on pulse
are used in this experiment. Each electrode
voltammetry carried out in a standard six-electrode
measurement is characterized by 66 samples; hence,
configuration. In this method, current transients due
a total tongue measurement comprises 396 samples.
to onset of a voltage pulse are measured, giving information concerning both amount and type of charged molecules and of redox active species. The electronic tongue, consisting of a six working
Titel: c:\data\research\tongue\patterns\tomeas.eps Skapad av: MATLAB, The Mathworks, Inc. Förhandsgranska: Den här EPS-bilden sparades inte med en inkluderad förhandlsgranskning. Beskrivning: Den här EPS-bilden kan skrivas ut på en PostScript-skrivare, men inte på andra typer av skrivare.
electrode system also contains an auxiliary electrode, and a reference electrode. The six working electrodes are composed of gold, iridium, palladium, platinum, rhenium and rhodium. The whole configuration is placed in a 150-ml measurement cell. The electrical current transient responses are measured by a potentiostat connected to the measurement PC via an A/D converter.
complete tongue measurement.
The recorded voltammograms are based on large amplitude
pulse
Figure 4. A typical sequence of samples in the
voltammetry
(LAPV).
A
measurement sequence starts by applying a potential
D. The chewing resistance
F. The sound system
A signal from a force sensor connected to the
A microphone is embedded in the mouth construction
pneumatic driving system of the mouth is also
to measure the sound from the chewing process, then
measured. The chewing process behaves similar to
a standard frequency analysis is provided on the
the human; i.e. an initial crushing is applied to the
records. Differences in the spectral power density
test object before the final chewing starts. The shape
(shift of the maximum, level of the horizontal
of this signal reflects the deformation process.
asymptote), the amplitude spectrum (change in the parameters of the envelope curve) and the complex spectrum drawn on the complex plane (varying size
The chewing resistance
and shape of the spot) show they can be used as
4089 ,5 6
characterizing parameters of the quality. Illustrative
5
diagrams are presented in figure 7.
4
3929. 1,262
3
Experiment # 8
2
12000
1
x 10
4
0.8 10000
1
0.6 0.4 8000
0 3700
0.2 6000
3800
3900
4000
4100
0
4200
-0.2 4000 -0.4 -0.6 2000 -0.8
Figure 5. An example curve describing the chewing resistance.
0 0
2
4
6
8
10
12
-1 -1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
x 10 4
12000
1 x 10 4
10
Pxx - X Power Spectral Density
0
10-1 10000 10-2 8000 10
E. The vision system
-3
10-4
6000
10-5 4000 10 2000 10
0
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
10
-6
-7
-8
0
0.1
0.2
0.3
0.4
0.5 Frequency
0.6
0.7
0.8
0.9
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
A color vision camera is used to indicate visual properties of the samples. The picture also directs the
Experiment # 2 12000
computer system to start the measurement procedure
1
x 10
1
4
0.8 10000 0.6 0.4 8000
by opening the mouth. Information about color, shape
0.2 6000
0 -0.2
and size of the sample object is measured.
4000 -0.4 -0.6 2000 -0.8 0 0
2
4
6
8
10
12 x 10
-1 -1
4
12000
1 x 10
10
10
4
Pxx - X Power Spectral Density
0
-1
10000 10-2 10-3
8000
10
-4
6000 10-5
4000
10
10
-6
-7
2000 10
0
10 0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
-8
-9
0
0.1
0.2
0.3
0.4
0.5 Frequency
0.6
0.7
0.8
0.9
1
Figure 7. Crashing sound frequency domain patterns of two samples with different properties IV. Sensor fusion and pattern recognition This section describes an industrial problem and a Figure 6. An image from the vision system.
proposed solution using sensor fusion. The main reasons for this experiment is twofold, first to test the
artificial sensor head in a long time test and to
The nose measurements accommodate a manageable
investigate its usefulness as an industrial on-line
amount of data, which has proven to contain relevant
sensor system. The aim is also to investigate if it is
information [11]. The nose data is obtained from 4
possible to improve the results by combining the
sensors, each one with a unique gas sensitivity
different sensor systems in a real world application.
pattern. Thus, the pattern vector from the artificial nose, xsmell, consists of 16 features.
The sensor fusion process in this approach is defined
Features from the other sensors are constructed in a
as a pattern recognition method, which gathers
similar way giving a unique vector from each
opinions from a number of task specific classifiers.
artificial sensor pattern. The final pattern vector can
Each one of the classifiers is specialized for one
then be formed as
perception specific related property of the artificial
x = [ x Tsmell , x Ttaste , x Tvision , x Tchewing , x Tsound ] T
sensors: smell, taste, vision, hearing and mouth feeling. The fusion method is then combining the features, into a single, more reliable one. The industrial related problem considered here is taken from the food process industry: classification of different qualities of potato chips including classification of the aging processes at room temperature. The recognition task for a given sample of potato chips is to identify the type of chips and classify its quality within four different grades.
.
B. Pattern Recognition We propose here a system that is closely related to the human way of estimating quality parameters. It is based on training of a fuzzy classifier and then using it in estimating how the sample object taken from the conveyor belt fits to the already established classes of production quality. For that purpose we make lots of experiments with potato chips, the quality of which is grouped in 8 classes, depending on the 3 levels salt
A. Feature extraction
content, existence/absence of spices and freshness. Large amount of measurements of the full pattern
Each sensor measurement contains different amount
vector then is stored, preprocessed and used for
of information. Data reduction must be performed to
training of the fuzzy classifier. Other set of
form an efficient classifier. Generally, this task may
measurements is used for test and verification of the
be troublesome due to the problem of modeling the
quality of the classifier. Two fuzzy classification
physical process that generates the measurements.
algorithms, namely fuzzy c-means and Gustafson-
Therefore, our approach is to compute some features
Kessel [12, 13], are applied. The experimental data
of the sensor signal, and by fuzzy cluster analysis,
gave approximately the same results, so any of them
determine its information content. For example, in
can be used.
case of tongue data by using the score- and loadingplots in principal component analysis (PCA) it
V. Conclusions and further work
emerged that the range in each of the first two cycles and the last cycle at each electrode should contain
An artificial human related sensor system evolved
sufficient information. The range is a relative
from human perception measurement is proposed,
measure and should be robust with respect to bias in
with emphasis on issues of complex quality
different measurement setups. The complete tongue
determination and focusing on food measurement
pattern vectors, xtaste, from a complete tongue
based on the human ability to quality estimation. The
measurement then consists of 18 elements.
paper presents basic background in feature extraction
and fusion of artificial sensor systems. In this concept of an artificial perception head we extract feature from the following sensors: - chewing resistance - electronic nose - electronic tongue - vision system - auditory system Applying multivariate analysis methods can show that the sensor units evaluate the properties of experimental samples in different way. However, by combining types of sensors and features from the different sensor data it is possible to reduce the amount of data to be processed in the classification phase. To achieve that, the system has also to be learned to estimate the discriminating abilities of each sensor with respect to the quality assessment of particular product. Then, a proper combination of sensor data can contribute to performing inferences that may not be possible from single sensors. This aspect has to be further developed in more comprehensive and self-contained system, able to include other human based capabilities and enhanced fusion techniques. ACKNOWLEDGMENT This work has been done at the AASS laboratory and at the Swedish Sensor Center at the universities of Örebro and Linköping respectively. We thank OLW, Pär Westergren and Camilia Modig for contributing with industrial knowledge and test samples. We also thank Joakim Arnell and Andreas Fransson for contributing to the development of the demonstrator hardware and software. This work is part of research programs sponsored by KK-foundation in Sweden. References [1] López-Orozco, J.A., de la Cruz, J.M., Sanz, J. and Flores, J., “Multisensor fusion of environment measures using Bayesian Networks”, Proc. of the International Conference on Multisource-
Multisensor Information Fusion, Las Vegas, USA, 1999, pp 487-493. [2] Winquist, F., Holmin, S., Krantz-Rülcker, C., Wide, P. and Lundström, I., “A hybrid electronic tongue”, submitted to Analytica Chimica Acta [3] Wide, P., Saffiotti, A., and Bothe, H., “Environmental Exploration: An Autonomous Sensory Systems Approach”, invited article to IEEE I&M Magazine, September 1999. [4] Gardner, J.W., and Barlett, P. N., ”A brief history of electronic noses, Sensors and Actuators B 1819, 1994, pp. 211-220. [5] Wide, P., Winquist, F. and Driankov, D., "An airquality sensor system with fuzzy classification.", International Journal of Measurement Science Technology, vol. 8, 1997, pp 138-146. [6] Winquist, F., Wide, P. and Lundström, I., "An electronic tongue based on voltammetry”, Analytica Chimica Acta, 357, 1997, pp. 21-31. [7] Andersson, Y., Drake, B., Granquist, A., Halldin, L.,Johansson, B.,Pangborn, R. M. and Åkesson, C., “Fracture force, hardness and brittleness in crisp bread with generalized regression analysis approach to instrument –sensory comparisons”, Journal of Texture Studies, vol 4. 1973, pp 119144. [8] Winquist, F., Wide, P., Eklöv, T., Hjort, C. and Lundström, I., "Chripsbread quality evaluation based on fusion of information from the sensor analogies to the human olfactory, auditory and tactile senses". Submitted to the International Journal of Food Process Engineering. [9] P. Wide, F. Winquist, P. Bergsten, and E.M. Petriu, “The human-based multisensor fusion method for artificial nose and tongue sensor data, IEEE Transaction on Instrumentation and Measurement, vol. 47, No. 5, October 1998. pp 1072-0177 [10] Winquist, F., Wide, P. and Lundström. I., “ The combination of an electronic tongue and an electronic nose”, accepted paper, International Journal - Sensor and Actuators B. [11] Lundström, I., Winquist, F.Hörnsten, E. G. and Sundgren, H., ”From hydrogen sensors to olfactory images-twenty years with catalytic field-effect devices,” Sensors and Actuators B, vol. 13, 1993, pp. 16-23. [12] Gustafson, D. and W.Kessel, "Fuzzy clustering with a fuzzy covariance matrix", Proc. IEEE CDC, San Diego, 1979, pp.761-766. [13] Roger Jang, J.-S. and N. Gulley, "Fuzzy Logic Toolbox", Mathworks, 1995.