320

IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 11, NO. 3, JUNE 2003

Manipulation of Unmodeled Objects Using Intelligent Grasping Schemes David L. Bowers and Ron Lumia

Abstract—We investigate “intelligent” grasping schemes using a fuzzy logic rule base expert system. In this paper, we use a vision system, robot arm and mechanical hand to locate and manipulate unmodeled, randomly placed objects of various sizes and shapes. In the pregrasp stage, we use vision data to provide a nonlinear mapping from object characteristics to hand configuration. In the postgrasp stage, we use hand data to ascertain the security of the grasp. Computational geometry is used to gauge the quality of the grasp and to quantify and validate the choice of hand configurations generated by the fuzzy logic expert system. The system is implemented within a low-cost virtual collaborative environment. Index Terms—Fuzzy systems, grasping, image processing, manipulation, unmodeled objects.

I. INTRODUCTION

R

OBOT grasping is required for many applications, from handling of hazardous material to collecting samples during planetary exploration. In spite of the fact that manipulating objects is easy for humans, grasping remains a challenging task for robots. Research in robotic grasping can be divided into two general approaches: the synthesis approach and the heuristic approach. The synthesis approach is mathematically based and answers the questions: 1) do solutions exist? and 2) is there a computationally efficient algorithm for the solutions? The heuristic approach, on the other hand, is knowledge based and answers the question: do sets of rules exist for achieving solutions? The synthesis approach in robotic grasping seeks to develop computational algorithms for achieving the following four important properties in autonomous multifingered hands. 1) Dexterity: How should the grasping fingers be configured? 2) Equilibrium: How hard to squeeze the grasped object? 3) Stability: How to remain unaffected by external disturbances? 4) Dynamic Behavior: How compliant should a grasp be when doing tasks? Implementation of these properties requires low-level control of the hand. Low-level control includes controlling the current for finger torque, computation of forces on fingertips and joints,

Manuscript received August 28, 2001; revised August 1, 2002 and September 5, 2002. This work was supported in part by the National Aeronautics and Space Administration under Contract NCCW-0087. D. L. Bowers is with Applied Technology Associates, Albuquerque, NM 87131 USA. R. Lumia is with the Mechanical Engineering Department, the University of New Mexico, Albuquerque, NM 87131 USA (e-mail: [email protected]). Digital Object Identifier 10.1109/TFUZZ.2003.812689

and exact finger position. This approach assumes well-defined models, precise sensors and is highly mathematical. Well-defined models are the basis of a synthesis approach. Shimoga [3] provides a good survey and overview of grasp synthesis techniques, including a summary of grasp synthesis algorithms for each of the above properties. At the time (1993), none of the algorithms had been implemented in real time. Shimoga states the following reasons for this: “(i) prohibitive computational complexities of the synthesis algorithms, (ii) lack of precise sensing and control capabilities of the existing robotic hands, and importantly, (iii) the unstructuredness of the real-life objects.” Park and Starr [4] present a simple and efficient algorithm to find an optimum force closure grasp of a planar polygon using a three-fingered robot hand. Their approach first identifies all feasible combinations of edges and/or vertices. Computational geometry is then used to construct a single grasp on each feasible combination. The grasps obtained are then compared heuristically to determine an optimal grasp. Many of the methods for achieving the above mentioned properties involve solving linear and nonlinear programming problems. Coelho, Jr., and Grupen [5] pose online grasp synthesis for achieving statically stable grasp configurations given instantaneous contact information as a robust control problem. Guo and Gruver [6] give a recent example of grasp planning to achieve the stability property. In this paper, they derive the small vibration equation for a multifingered robot hand grasping an object and use nonlinear programming to determine an optimal three-dimensional (3-D) grasp. The heuristic approach to grasping objects is knowledge-based and typically addresses hand pre-shaping. This approach more directly addresses the question: How would a human grasp an object? Generally a combination of the object’s characteristics and the task are the dominant factors in determining hand configuration. Many researchers have studied how humans preshape their hand for various grasping tasks. In [7], Lyons gives a simple set of grasps for a dexterous hand. Cutkosky [8] shows an example of grasp taxonomy for manufacturing tasks in which machinists performing small batch processes were observed and interviewed. This paper, along with developing a taxonomy of grasps, showed that an expert system could predict how a human might grasp an object. Iberall et al. [9] give a task-oriented approach to hand preshaping based on prosthetics and rehabilitation. Both [8] and [9] comment on how task requirement and object geometry combine in influencing grasp choice. Most recently, Kang and Ikeuchi [10] map human grasps to robot grasps in order to reduce planning complexity. Their system first observes a human performing the task, then classifies the human grasp, and finally maps it onto the manipulator.

1063-6706/03$17.00 © 2003 IEEE

BOWERS AND LUMIA: MANIPULATION OF UNMODELED OBJECTS

Some work has been done in which a combination of a synthesis and heuristic approach is used. Caselli et al. [11] consider grasp representation, object representation, and task representation in grasp planning. This work “relies on symbolic and subsymbolic computations to exploit heterogeneous sources of knowledge, such as practical experience learned through experiments with a real device, heuristic rules gained from human observation, geometric reasoning and, when applicable, analytical results.” Our approach to grasping falls into the heuristic camp for several reasons. First, we expect to grasp unknown objects, i.e., objects for which a computer-aided design (CAD) model does not exist. Since all synthesis approaches require a model of the gripper and the object, we would have to use sensors to create the object model, and this has proven to be a challenging research problem. Second, even if a CAD model could be created, the interaction between the gripper and the object model represents significant computational complexity and often cannot be done in real time. Finally, the interaction between the real gripper and object is often slightly different than the interaction between their models. This lack of registration can result in a variety of failure modes. Our approach does not require intensive computation or memory. We use high-level commands for hand control. While this paper uses a specific manipulator, there is every reason to believe the approach would be applicable to other robot end effectors. By using fuzzy logic, we are able to utilize imprecise data from the hand and vision system. This work targets a simple and inexpensive approach to grasping objects. The objective is to provide adequate grasp techniques in an environment where computing power is limited, e.g., planetary exploration. The approach used in our work is mostly a heuristic approach that incorporates fuzzy logic. Stansfield’s work [12] serves as a model of a heuristic approach to robot grasping. In [12], a vision system extracts features from an unmodeled object and an expert system generates valid grasps for the object. The expert system is based on how a human would preshape his hand to grasp an object. Similar to the work in [12] we use a vision system to extract object features and an expert system to generate grasps for the object. Our expert system is implemented using fuzzy logic. In addition to the vision and expert system we add coarse haptic sensing to ascertain the grasp security. We also use computational geometry in an offline mode to determine the quality of the grasp with respect to other possible grasps. The inputs to our system are object attributes (obtained from the vision system) used in the pregrasp stage and finger force (obtained from strain gauges in the hand) used in the postgrasp stage. In addition, a comparison of object size based on finger position to object size based on vision data is used to quantify grasp security. In this paper, “intelligent grasping” refers to the ability of a hand/vision system to reason with vague concepts and imprecise data. No learning occurs in this system. We simply perform a nonlinear mapping from object characteristics to hand configuration. Through this direct mapping we alleviate the need for a model comparison approach, which can be computationally intensive and prone to error due to inaccurate sensing. The system simply performs the fuzzy mappings, preshapes the hand and

321

attempts a grasp. The system then ascertains if the grasp of the object was successful, i.e., the hand has a secure enough grasp of the object to transport it. One of the limitations in grasping is creating a system that can operate in unknown environments. Visual feedback assists in creating a grasping system for these complex environments. Horaud et al. [13] present a visual servoing approach to the problem of object grasping. Horaud shows that by including vision in a grasping technique, prior knowledge of robot-to-world calibration and object types are not needed. There have been many advances in vision systems. Caelli and Bischof [14] discuss various perspectives on interpreting image data with respect to domain knowledge. Specifically, they discuss fuzzy conditional rule generation for the learning and recognition of 3-D objects from two-dimensional images. Wright and Bourne [15] give a list of research directions and open areas in grasping and comments on the limitations of using vision systems for grasping. Among the open problems listed is to “design and carry out experiments to determine whether vision or touch is the dominant sense in size determination, shape determination and feedback for motor control.” In addition to the vision data, we use coarse haptic sensing to address postcontact security after the grasp attempt. Haptic sensing is the ability of a system to acquire information through touch or force feedback. Ohka et al. [16] present a tactile expert system that can recognize the shape and material of the grasped object as well as the stability of the object. They show how using tactile sensors they can infer the material and shape of the grasped object. Caselli [17] presents a novel approach to haptic object recognition based on volumetric representation models. The methodology uses an offline learning phase and a runtime classification phase for matching empirical data with stored models. Work has also been done to incorporate vision and haptic sensing. Allen et al. [18] use custom tactile sensors and real-time vision to estimate finger contacts and applied forces for grasping objects. Vision and haptic sensing provide the main modalities necessary for grasping. The vision system, while providing relevant data, never produces the exact same data even if there is no apparent change in conditions. The hand data provides relevant but approximate information on the finger position and finger force. One method of dealing with uncertainty in complex systems is through the use of fuzzy logic [19]. To help overcome the limitations of our sensors and utilize the high-level hand commands, we chose fuzzy logic to implement our expert system. Fuzzy logic, which allows us to reason with vague or ambiguous data, more accurately reflects the way a human reasons with data and allows the expert to use linguistic terms. Ross [20] provides background and mathematical development of fuzzy logic and gives a simple example of using membership functions to represent geometric shapes. Bezdek [21] describes a method to incorporate human knowledge and reasoning with nonrandom uncertainty in the area of image processing. II. FACILITIES The experimental setup and test bed are shown in Fig. 1. The system consists of the Cognex Vision system [22], the Puma 560

322

IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 11, NO. 3, JUNE 2003

Fig. 2. Barrett three-fingered hand. Fig. 1.

Lab equipment.

robot arm [23], and the Barrett Hand BH8-250 [24]. The vision system consists of a single camera and a controller driven serially from a PC. Serial strings are sent from a host PC to trigger the vision system and obtain vision data. Cimetrix [25] simulation and control operates the six degree-of-freedom Puma 560 robot arm. The camera for the vision system is located on joint 6 of the Puma 560 to allow for maximum flexibility in obtaining vision data. The end effector used on the Puma 560 is the Barrett Hand. The Barrett Hand is a three-fingered robot hand designed for form closure grasping. It, like the vision system, consists of a controller that interprets string commands sent serially from the host PC. Using the C programming language for program development, we access and control all mechanisms from a single host PC. Early program development was done on the LYNX PC in the UNIX environment using the C programming language. Later development was done in the Robotics and Automation Drop and Drive System (RADDS) [26], [27] in the Windows NT environment using the Visual C++ programming language. The robot hand used for this work is the Barrett Hand BH8-250 [24]. The BH8-250 is a three-fingered mechanical hand; the middle finger is stationary and acts as a thumb while the other two fingers can spread synchronously from 0 to 180 degrees about the palm. Each finger has two joints but is driven by one motor (one motor per finger). A fourth motor controls the finger spread. By adjusting the degree of spread of the fingers, the grasp tailors itself for arbitrary object shapes and sizes. A model of the hand developed using Cimetrix software [25] is shown in Fig. 2. When performing grasps, a cable acts as a tendon with the outer link of each finger closing at a faster rate than the inner link of each finger. The BH8-250 is designed to stop if fingertip forces exceed a certain threshold. If the inner link (link closest to the palm; see Fig. 3) of the finger comes in contact with an object first, the hand attempts to wrap around the object producing a form closure grasp. The hand is designed for this type of power grasping and is not meant for fine manipulation. The information available from the hand includes finger position (in the form of motor counts) and finger force (in the form of strain to be the finger spread in degauge counts). If we define

Fig. 3. One finger of barrett hand.

grees and to be the finger spread in motor counts, then the encoder/joint angle relationships are as follows: (1) or (2) to be the angle (measured in degrees) from If we define the palm to the inner finger link and to be the angle from the inner link to the outer link of the finger, as shown in Fig. 3, then (3) or (4) and (5) or (6) to be the torque on fingertip 1 or 2, to be If we define the torque on fingertip 3 and to be the strain gauge value, then the strain gauge/joint torque relationships are (7)

BOWERS AND LUMIA: MANIPULATION OF UNMODELED OBJECTS

323

or (8) for finger 1 or 2 and (9) (a)

or

(b)

(c)

Fig. 4. Membership in the set “circle.” (a) Partial membership. (b) Full membership. (c) Partial membership.

(10) for finger 3. III. VISION ALGORITHMS Before a robot can grasp an object, it needs to know the following three pieces of information about the object: 1) position; 2) orientation; 3) features (shape, size, elongation, weight, etc.). Simple vision algorithms provide information on an object’s position, orientation and size. The algorithms for determining the object’s shape and position are discussed in [2]. We use the Cognex vision system along with these vision algorithms to target, locate, and classify objects. Part of the grasping scheme is to determine the object’s geometry. The vision system provides information about an object using the blob tool. This tool uses a contrast scheme to form blobs of an object. From the blob, information such as pixel area, perimeter, elongation, enclosing rectangle width and height and principal axis angle is returned. We then perform a mapping from vision data to hand configuration. In one instance, a vision algorithm categorizes objects into one of four common shapes, i.e., circle, ellipse, rectangle, or triangle. A fuzzy logic rule base then maps one vision parameter to hand configuration. The object categorization is done using an area fitting process. The other approach used does not implicitly categorize the object; instead, it directly maps vision data to hand configuration. As previously mentioned, the shape categorization algorithm uses an area fitting process. The vision system returns the area and enclosing rectangle width (rw) and rectangle height (rh) of the blob. The program uses the rw and rh to calculate the area of several common shapes (circle, ellipse, rectangle, and triangle). This calculated area is compared against the returned area. The program then categorizes the object into one of these shapes according to the area it fits most closely. Regardless of the object, the program will classify it as a circle, ellipse, rectangle or triangle using this scheme. Once an object is classified, elongation (aspect ratio) is used to drive the hand configuration. This aspect ratio measure gives more detailed knowledge of the object and allows for a better pregrasp hand configuration. For example, the program may classify an object as an ellipse based on the area fitting process described above. The rule base then allows for finer classification by mapping the elongation to finger spread. With an elongation closer to one being more circle like and an increasing elongation being more ellipse like. This is similar to the idea of an object, for example, belonging to the ellipse set with a membership of 0.72 and the circle set with a membership of 0.28, providing a more accurate categorization of the object.

The more accurate classification produces a slightly different mapping between object shape and finger spread, and thus leads to a better pregrasp hand configuration. Fig. 4 illustrates the point. In Fig. 4, the object in the middle would be classified as a circle while the objects on the ends would probably be classified as ellipses using the area fitting classification scheme. By using the fuzzified vision parameter (elongation) described here, the object on the right might be classified as a circle with a membership of 0.59 and as a vertical ellipse with a membership of 0.41. The smaller membership value (i.e., less then unity) causes the spread to be slightly smaller and, thus, leads to a better pregrasp hand configuration. Fig. 5(a) shows the relationship between our four ideal basic object shapes and examples of some specific shapes that are close to these ideals. Fig. 5(b) illustrates a membership function for these shapes. The perfect shape has a membership value of 1.0. In Fig. 5(b), we consider elongation in the direction to show how these classes can be combined into overlapping sets for membership. The sensed elongation will almost never indicate a perfect part, but rather a combination of several parts from the overlapping membership set. The approach we use to determine object location is based on a similar triangle method [28], [29]. The similar triangle method assumes that the camera is looking straight at the object and that the camera’s perspective of the object will not change if the camera is moved straight toward or away from the object. In this method, the camera is centered on the object and a picture is taken. The camera is then moved a known distance toward the object, making the object appear larger. One may then use similar triangles to compute the range to the object. The object’s location relative to the robot’s base may then be computing using homogeneous transformations [30]. The vision system and vision algorithms also provide a size measure of an object using the similar triangle approach. This scheme gives the physical width (height) of an object. The object area and the enclosing rectangle width and height in camera space may also be used as size measures if the corresponding camera height is known. Our general approach to object grasping is to position the center of the palm at the object’s center of mass oriented such that the hand is perpendicular to the principal axis of the object. The vision system provides the angle of the principal axis, which is used to orient the hand. For certain shapes, orientation perpendicular to the principal axis is not adequate and an alternate approach must be taken. For example, in the simple case of orienting a three-fingered hand to grasp an isosceles triangle the thumb may be placed either at the apex of the triangle or along the base using the perpendicular to the principal axis approach.

324

IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 11, NO. 3, JUNE 2003

Fig. 6. Desired thumb orientation.

ratio) the hand is oriented perpendicular to the principal axis. For objects with a small elongation value (less than 2.2), the thumb of the hand is placed along the vector from the enclosing center to the object’s center of mass. rectangle’s During testing it was found that the vision data were not accurate enough to produce the necessary thumb placement by simply dropping a vector from the enclosing rectangle center to the center of mass. However, the data were accurate enough to give the direction of the vector from which accurate thumb placement was achieved. IV. GRASPING

(a)

(b) Fig. 5. (a) Examples of ideal classes and actual shapes. (b) Fuzzy membership based on elongation.

To resolve this ambiguity we drop a vector from the enclosing center location to the object’s center rectangle’s of mass location, as shown in Fig. 6. This procedure orients the thumb along the base of the triangle for isosceles triangles. For objects with a large elongation value (greater than 2.2) or similarly a large rectangle height to rectangle width ratio (aspect

The vision data and vision algorithms provide information about an object’s location, orientation and characteristics (shape, size, elongation, etc.). The procedure for grasping is to use the vision data to provide a nonlinear mapping from this visual information to finger spread or pregrasp hand configuration. By adjusting the degree of spread of the fingers, the grasp tailors itself for arbitrary object shapes and sizes. Once a grasp attempt is made, finger position and finger force determine a measure of grasp security. A block diagram of this procedure is shown in Fig. 7. By adjusting the spread of the fingers, many grasp types are possible. Fuzzy logic rule bases provide the nonlinear mapping from object characteristics to grasp configuration and from finger position to object size. Various fuzzy mappings and rule bases were created and evaluated. We also incorporated computational geometry [31] in order to test the grasp quality and put a more quantitative measure on the grasp selected by the expert system. By computing the convex hull of the wrench space [32], we can do an a priori evaluation of the grasps produced by the fuzzy logic expert system. A. Knowledge Base Data Acquisition In order to develop a knowledge base and investigate mappings from vision data to finger spread and hand data to grasp security, a data collection program was developed. The data collection program is outlined as follows.

BOWERS AND LUMIA: MANIPULATION OF UNMODELED OBJECTS

325

Fig. 7. Procedure for grasping.

1) Go to a configuration and take a picture. 2) Center on the targeted object, move closer and take another picture. 3) Rotate about the camera’s axis by the angle of the principal axis and take another picture. 4) Move to the object and orient the hand. 5) Prompt the user for a spread value and make a grasp attempt. 6) Obtain hand data and prompt the user for the grasp quality. 7) Write the data to a file. All of the data are written to a comma-delimited ASCII file, creating a database of vision data, corresponding hand data, hand configuration, description of the grasp attempt, and object number. These data are then used to create the fuzzy logic rule bases. During the data collection phase, good grasps were noted visually and by applying external forces to the object, as well as by recording the strain gauge values in each of the fingers. For “good” grasps of various objects, the strain gauge values are similar. With this in mind, one should be able to ascertain the security of a grasp based on the gauge data returned from the hand after a grasp attempt is made. A simple rule for doing this was derived based on the empirical data. An average strain gauge value for each object grasped was computed. The average and standard deviation of these values were then computed. By checking whether the strain gauge values are within the average strain gauge values plus or minus the standard deviation we

are able to ascertain the grasp security. This is defined by RULE 1. RULE 1) If current gauge is between average gauge standard deviation, then grasp is secure. In observing the grasps, we used the intuitive notion that a “good” grasp is one in which the fingers surround the object and there is as much contact area between the hand and object as possible. As described in Section IV-B, many of the rules are designed to get as much contact area on the object as possible both in fingertip grasping scenarios and form closure grasping scenarios. B. Fuzzy Mapping Rule Bases Several methods for transforming vision data to hand configurations were investigated. In the first approach the object was classified as a basic shape and then a single-input–single-output (SISO) fuzzy rule base system (FRBS) was designed to map one vision input parameter to one hand output parameter. The second approach directly mapped the vision-input parameters to one hand output parameter with a multiple-input–single-output (MISO) FRBS. In this method, there was no shape categorization stage. 1) SISO Mappings: In this approach, the object was classified as a rectangle, triangle, circle, or ellipse using the area fitting process previously described. Based on the classification, a SISO fuzzy mapping was performed. Three different fuzzy rule bases were designed for grasping objects, one for each shape except for the rectangle, which was the default case. If the object is

326

IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 11, NO. 3, JUNE 2003

classified as a triangle, then a fuzzy rule base maps elongation (vision input) to finger spread (hand output). Similarly, if the object is classified as an ellipse, a fuzzy rule base maps elongation to spread. If the object is classified as a circle, the fuzzy rule base maps area (vision input) to spread (hand output). Finally if the object is classified as a rectangle a default grasp configuration is produced. A universe of discourse was established for each of the fuzzy input variables and output. For example, we only considered objects of a certain size and elongation and achievable hand configurations. The fuzzy rule bases are generated by taking data on a small set of objects, logging the vision data (used as input to a fuzzy rule base), hand data (used as output to a fuzzy rule base) and noting the grasp quality (semantically). The data collection procedure was outlined in Section IV-B. The sample sets consist of triangle-like objects, ellipse-like objects and circle-like objects of various sizes and aspect ratios. There are ten objects per set in this experiment. In order to find a “good” spread value for a given object, various grasp attempts were taken. For each grasp attempt, the vision data, hand data, object and grasp quality were recorded. These data were fed into the Fuzzy Logic Development Kit (FULDEK) [33], which was used to create the membership functions and rules for the fuzzy rule bases from FULDEK’s AutoRule function. The input for each fuzzy rule base was made up of five triangular membership functions and the output membership function was a singleton, or single spread value. For the fuzzy inference, multiplication was used for the conjunction (AND) operations and the maximum was used for the disjunction (OR) operations. Defuzzification produces the crisp value for the finger spread. The procedure is as follows. First, the object is classified to be a member of one of the primitive shapes (circle, rectangle, ellipse, triangle) using the are of the object (AO) and the area of the enclosing rectangle around the object (AER). Classify

where circle, triangle, square, rectangle (11)

Once the object is classified to be a member of primitive object , then the spread is determined by Spread

(12)

Rules for (12) were generated from the data gathered in the learning phase. FULDEK created a set of rules for each of the primitive shapes based on triangular membership functions from the training data. Table I describes the input variables, number of membership functions (MFs) and rules generated for each primitive shape. The goal of this experiment was to see if we could roughly classify objects based on a simple area fitting process and then find one vision parameter to map to an adequate hand configuration for grasping the object. 2) MISO Mappings: The second approach is to map the vision data directly to hand configuration eliminating the area

TABLE I FULDEK DEFUZZIFICATION RULES FOR EACH PRIMITIVE SHAPE

based categorization stage. Again, FULDEK’s AutoRule function generated rules from the training data. Various vision-input parameters were tested. We also produced our own rules based on studying the object and the vision data. We incorporated “knowledge tidbits” [21] from our own experience in grasping. These “knowledge tidbits” include being able to estimate an object’s center of mass, size, and aspect ratio and then, using these parameters, configure a hand in such a way as to produce a stable grasp. In developing the rule base, it was found that area and elongation were good vision features. These vision parameters give a measure of size and aspect ratio and can be combined to grasp objects. In general, larger elongation values lead to smaller spread values (and vice versa) and larger areas lead to larger spread values (and vice versa). When both parameters are moderate there is interplay between the two. If either parameter becomes excessive, then it dominates the spread. For example, if the elongation is very large the spread will be very small, regardless of the area. C. Grasp Quality We use a fuzzy logic expert system to map the vision data of an object to a hand configuration for grasping the object. We also use hand data to ascertain the security of the grasp. In these two steps, nothing was known a priori about the object. We now analyze the grasps generated by our rules to give an analytical, quantitative measure of the grasp quality. That is, given the contact points, position of the gripper and the object, we want to say a posteriori, “this was a good grasp.” One idea of a “good” grasp is formally represented by the definition of a force closure grasp [32]. Definition 1: A grasp is said to be force closure if it is possible to apply force and torque at the contact points such that any external force and torque can be balanced. This is a measure of the stability of the grasp to external forces. Certain grasp configurations are better than others because they can achieve force closure without applying excessive contact forces. Large contact force implies high finger torque, which implies large driving current for the finger motors, causing high power consumption and inefficient grasps. Further, deformation to the object and/or the gripper may occur. We use the grasp quality measure developed by Ferrari and Canny [34] and further refined by Miller and Allen [35]. For simplicity, we only analyze a few relatively planar objects and assume only point and line contacts. A line contact may be made up of two point contacts. This leads to the analysis of fingertip grasps produced on semiflat (planar) objects and excludes the analysis of form closure power grasps. This is a major restriction to the capability of our system because the

BOWERS AND LUMIA: MANIPULATION OF UNMODELED OBJECTS

Fig. 8. object.

327

Depiction of three-point contacts and friction cone for a triangular

Barrett Hand is designed mainly for form closure grasping, not fingertip grasps. We make the conservative assumption that the static coefficient of friction, , between the rubber-like material on our hand and the rock objects is equal to 0.25. For comparison, the static coefficient of friction between grooved rubber and dry pavement is 0.55 [39]. The boundaries of the friction cone [3], or in the planar case the friction triangle are computed (see Fig. 8). In order for the point contact not to as slip, it must lie within this friction cone. For point contacts, “generating forces” in the planar case (the edges of there are generating forces is broken the friction triangle). Each of the into their respective and components and the torque for each force is computed. For computing the torque we take the moment arm from the object’s center of mass to each point contact. The combination of a generating force’s and components and associated torque is called a wrench, . For the th contact . The convex combination of point we have is called the wrench space . all wrenches Wrenches are used to describe the system of forces and moments exerted on the object [32]. The second method described by Ferrari and Canny [34] makes the sum magnitude of the con, where is tact normal forces 1. A torque multiplier, the largest moment arm for which the grasp yields a consistent system, i.e., a system whose units are all force units. Thus, we have each wrench given by

Fig. 9.

Objects for experiment 1.

(13) Fig. 10.

Objects for experiment 1.

for the th contact point and the th wrench. An efficient method of finding the wrench space uses computational geometry and may be found as

V. EXPERIMENTS A. Experiment 1

ConvexHull

(14)

software We use the algorithm outlined in [36] and the [37] for this computation. The minimum of the perpendicular distance from each facet of the hull to the origin of the wrench space gives a measure of the efficiency of the grasp and is used as the grasp quality measure. This distance corresponds to the weakest measure of the grasp’s ability to efficiently resist an external wrench [38]. The closer this distance is to 1.0 the more efficient the grasp. In order for the grasp to be stable, the hull must enclose the origin. The grasps generated by the fuzzy logic expert system are evaluated on a few objects using this method to give a measure of how well the expert system selects a hand configuration for a given object.

Experiment 1 consisted of grasping various objects and performing a strain gauge threshold test. This experiment exercises and tests the rule bases and the strain gauge rule. For this experiment, the objects shown in Figs. 9 and 10 were placed in random locations and random orientations. The SISO FRB performed mappings and the strain gauge rule was used to check grasp security. In this experiment, all of the objects were successfully picked up. A grasp security warning was issued on four of the objects, i.e., the returned strain gauge values fell outside the range of the strain gauge rule. The stability of each grasp was checked by visual inspection and by applying external forces manually to the object. In three of the four warnings issued, the grasp was either visually awkward and/or not stable to external forces. On one of the warnings issued, the grasp was both visually secure and secure to external disturbances indicating a false

328

IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 11, NO. 3, JUNE 2003

TABLE II SUMMARY OF EXPERIMENT 1

warning. The strain gauge rule is most sensitive to ellipsoidal objects with small elongation values. Results for experiment 1 are summarized in Table II. B. Experiment 2 Experiment 2 is a “clear the table” demonstration. It consists of randomly scattering objects of various size and shape through out the robot’s workspace and then having the system clear the table. Fig. 11 shows an initial setup of the workspace with randomly placed objects of different shapes, elongation and sizes. The program follows the flow chart of Fig. 7. A)

B)

C)

D)

E)

Vision Sensor Data 1) Take a top-level picture in one region of the workspace. 2) Target the object with the largest area, center and move closer to the object. 3) Take a second picture and rotate around the camera’s -axis so that the principal axis of the object is aligned with the horizontal axis of the camera. 4) Take a third picture; classify the object as a circle, triangle, rectangle, or ellipse. Nonlinear Mapping Using Fuzzy Logic 5) Based on the vision data a fuzzy rule base maps area to finger spread for objects classified as circles and elongation to spread if the object is classified as a triangle or an ellipse. A default finger spread of zero degrees is used for objects classified as rectangles Pregrasp Hand Configuration 6) The hand orientation is configured based on the ratio of the enclosing rectangle width to rectangle height (aspect ratio). For aspect ratios greater than 2.2, the hand is oriented perpendicular to the principal axis. For aspect ratios less than 2.2, the thumb of the hand is oriented along a vector from the enclosing rectangles center to the enclosed blob’s center of mass. 7) The palm of the hand is placed over the object’s center of mass and a spread is generated based on step 5). Grasp Attempt 8) The hand is configured and a grasp attempt is made. Hand Sensor 9) The security of the grasp is ascertained by reading the strain gauge values. If a secure grasp is obtained go to step 10), if not, issue a warning and repeat the process outlined in steps 1)–8). 10) The object is placed in bin.

Fig. 11.

Setup for “clear the table.”

The process is repeated until the workspace is cleared. This experiment was run with various layouts and objects. If the robot could not reach the object there was no grasp attempt. The grasp attempt failed if an object was close enough to the targeted object to interfere with finger closure on a grasp attempt. Sometimes the object(s) involved are moved by the grasp attempt in such a way that upon the next attempt a successful grasp was obtained. In situations where the object was not successfully picked up, the strain gauge thresholding rule would be issued and the grasp would be reattempted. C. Experiment 3 Experiment 3 shows the results of the grasp quality measure. Following [34], we define quality as the minimum distance of the wrench to the origin from each facet of the convex hull of the wrench space. Quality

such that

(15)

In Table III, we compare the grasp quality of different hand configurations used to grasp the same object. We list the spreads (hand configurations measured in degrees) for the SISO FRB mappings, the MISO mapping, and a default mapping. The default mapping is used as a comparative measure. The default mapping is a one to one mapping of a specific object shape and size to a specific spread value. Mainly due to the object’s size, the default spread was 525 motor counts (30 from the spread open position or 150 from the spread close position; see (1) and (2). The default mappings provide five one to one mappings from specific size and shape to a finger spread and use five rules. The fuzzy logic mappings provide an unlimited number of mappings from vision data to a finger spread and in the SISO case use five rules (in the MISO case, we use 13 rules). Recall that our grasp quality measure is conservative and uses only point and line contacts of planar objects with assumed coefficient of . As a reference, the optimal point contact sofriction, (for lution for a circle produces a quality measure, three point contacts). The results show that the FRB mappings have a higher grasp quality than the default mappings. Further outlined is the importance and interplay of the relationship between object geometry

BOWERS AND LUMIA: MANIPULATION OF UNMODELED OBJECTS

TABLE III QUALITY OF GRASPS WITH DIFFERENT MAPPINGS

and size to gripper geometry and size. When designing our rules we used the intuitive, common sense notion that for a “good” grasp the fingers should enclose the object if possible and there should be as much contact area between the fingers and the object as possible. The idea of a force closure grasp and the quality measure used here formalize this notion. If we look strictly at object size and geometry and ignore the gripper size and geometry then the default mappings seem more appropriate. We start with the example of the flat circle-like object, of diameter roughly 4 in. The default mapping provides a spread that generates a 120 separation between each of the three fingers, the optimal solution for three point contacts on a circle. Because of the hand size and geometry when executing this grasp only a small portion of the two spreading fingers come in contact with the object modeled as point contacts in the grasp quality analysis. By issuing smaller spread values, as in the SISO and MISO mappings, we are able to place a more complete portion of the finger width on the object that is modeled as line contacts in the grasp quality analysis. This leads to a more stable grasp and therefore a higher grasp quality. As a second example, consider the triangle-like object used in Table III. This is a nearly isosceles triangle with a base of roughly 6 in and a height of roughly 3 in. In this case, the default-spread value and the MISO spread value produce more point-like contacts (smaller line contacts) by the two spreading fingers. By increasing the spread value, we are able to place full fingertip contact by the two spreading fingers on the sides of the triangle-like object. This leads to a slightly higher grasp quality for the SISO mapping. Based on this experiment, it is clear that gripper size and geometry as well as object size and geometry play a roll in determining a good hand configuration. Further, the intuitive notion of a good grasp used to build the FRB has been validated and quantified by this analysis. The system is capable of generating grasps very quickly and provides adequate grasp quality. As shown in Table III, the MISO and SISO mappings performed similarly as measured by the grasp quality. The SISO mappings have the advantage of simplicity, in most cases, mapping the single invariant parameter of elongation to finger spread. The MISO mappings, although more complicated, avoid the categorization stage used with the SISO mappings. The MISO mappings out performed the SISO mappings for the circle-like grasp because it provided a more accurate categorization of the circle-like object by using both area and elongation. The SISO mappings outperformed the MISO mappings when the objects were classified as something other than a circle with elongation not area being the dominant parameter.

329

VI. CONCLUSION This work has shown how fuzzy rule bases provide adequate information for a robotic system to locate and manipulate unknown objects. The grasps determined by the fuzzy rule based system were analyzed using computational geometry. The results showed that the grasps were reasonable, i.e., nearly the optimal with respect to the QHULL measure. A future direction of this work will be to incorporate learning into the process so that the rules are modified as a result of experience.

REFERENCES [1] D. Bowers, L. Montes, A. Ramos, B. Joyce, and R. Lumia, “Demonstration of the low-cost virtual collaborative environment (VCE),” presented at the NASA URC Technical Conf., Albuquerque, NM, Feb. 1997. [2] L. Montes, D. Bowers, and R. Lumia, “Vision algorithms to determine shape and distance for manipulation of unmodeled objects,” presented at the NASA URC Technical Conf., Huntsville, AL, Feb. 1998. [3] K. B. Shimoga, “Robot grasp synthesis algorithms: A survey,” Int. J. Robot. Res., vol. 15, pp. 230–266, Aug. 1996. [4] Y. C. Park and G. P. Starr, “Grasp synthesis of polygonal objects using a three-fingered robot hand,” Int. J. Robot. Res., vol. 11, no. 3, pp. 163–184, June 1992. [5] J. A. Coelho Jr and R. A. Grupen, “Online grasp synthesis,” presented at the 1996 IEEE Conf. Robotics Automation, Minneapolis, MN. [6] G. Guo and W. Gruver, “Dynamic grasp planning of multifingered robot hands based on asymptotic stability,” IEEE Trans. Syst., Man, Cybern. B , vol. 26, pp. 764–768, Oct. 1996. [7] D. M. Lyons, “A simple set of grasps for a dexterous hand,” in IEEE Int. Conf. Robotics Automation, St. Louis, MO, 1985. [8] M. R. Cutkosky, “On grasp choice, grasp models, and the design of hands for manufacturing tasks,” IEEE Trans. Robot. Automat., vol. 5, pp. 269–279, June 1989. [9] T. Iberall, G. Sukhatme, D. Beattie, and G. A. Bekey, “Control philosophy and simulation of a robotic hand as a model for prosthetic hands,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots Systems, vol. 2, 1993, pp. 824–838. [10] S. B. Kang and K. Ikeuchi, “Toward automatic robot instruction from perception—mapping human grasps to manipulator grasps,” IEEE Trans. Robot. Automat., vol. 13, pp. 81–95, Feb. 1997. [11] S. Caselli, E. Faldella, B. Fringuelli, and F. Zanichelli, “A hybrid system for knowledge-based synthesis of robot grasps,” presented at the IEEE/RSJ Int. Conf. Intelligent Robots Systems, Yokohama, Japan, July 26–30, 1993. [12] S. A. Stansfield, “Robotic grasping of unknown objects: A knowledgebased approach,” Robot. Research, vol. 10, no. 4, Aug. 1991. [13] R. Horaud, F. Dornaika, and B. Espiau, “Visually guided object grasping,” IEEE Trans. Robot. Automat., vol. 14, pp. 525–532, Aug. 1998. [14] T. Caelli and W. F. Bischof, Machine Learning and Image Interpretation. New York: Plenum, 1997. [15] P. K. Wright and D. A. Bourne, Manufacturing Intelligence. Reading, MA: Addison-Wesley, 1998. [16] M. Ohka, M. Kobayashi, T. Shinokura, and S. Sagisawa, “Tactile expert system using a parallel-fingered hand fitted with three-axis tactile sensors,” JSME Int. J. Dyna., Control, Robot., Design, Manufact., vol. 37, no. 1, Mar. 1994. [17] S. Caselli, C. Magnanini, and F. Zanichelli, “Haptic object recognition with a dexterous hand based on volumetric shape representations,” presented at the IEEE Int. Conf. Multisensor Fusion Integration Intelligent Systems, Las Vegas, NV, Oct. 1994. [18] P. Allen, A. T. Miller, P. Y. Oh, and B. S. Leibowitz, “Using tactile and visual sensing with a robotic hand,” presented at the IEEE Int. Conf. Robotics Automation, Albuquerque, NM, Apr. 1997. [19] L. A. Zadeh, “Fuzzy sets,” Inform. Control, vol. 8, no. 3, pp. 338–353, 1965. [20] T. J. Ross, Fuzzy Logic With Engineering Applications. New York: McGraw-Hill, 1995. [21] J. C. Bezdek, “Computational intelligence and edge detection,” presented at the Artificial Intelligence, Fuzzy-Neuro-System, Soest, Germany, 1997. [22] Checkpoint Reference Manual, Cognex Corp., Natick, MA.