A tactile sensor includes a photosensing structure, a volume of elastomer capable of transmitting an image, and a reflective skin covering the volume of elastomer. The reflective skin is illuminated through the volume of elastomer by one or more light sources, and has particles that reflect light incident on the reflective skin from within the volume of elastomer. The reflective skin is geometrically altered in response to pressure applied by an entity touching the reflective skin, the geometrical alteration causing localized changes in the surface normal of the skin and associated localized changes in the amount of light reflected from the reflective skin in the direction of the photosensing structure. The photosensing structure receives a portion of the reflected light in the form of an image, the image indicating one or more features of the entity producing the pressure.
|
0. 40. A sensor comprising:
a volume of elastomer capable of transmitting an image, the volume of elastomer having a reflective surface that reflects light incident on the reflective surface from within the volume of elastomer, the reflective surface being deformable in response to pressure applied by an object touching the reflective surface, the deformations causing localized changes in the surface normal of the reflective surface and associated localized changes in the amount of light reflected from the reflective surface in the direction of a photosensing structure, the photosensing structure being disposed to receive a portion of the reflected light in the form of an image, the image indicating one or more features of the object producing said pressure;
at least a first light source and a second light source, each light source disposed to illuminate the reflective surface from within the volume of the elastomer, and each light source disposed to illuminate the reflective surface from substantially different azimuths relative to each other, the first light source producing a first image and the second light source producing a second image; and
a photometric stereo analyzer for receiving information about the one or more features indicated in the first and the second images, the analyzer estimating at least one of a surface normal and a surface height of the object at a plurality of positions on the object based on the information.
0. 27. A method of reconstructing a 3-dimensional shape of an object, the method comprising:
providing a volume of elastomer capable of transmitting an image, the volume of elastomer having a reflective surface that reflects light incident on the reflective surface from within the volume of elastomer;
illuminating the reflective surface through the volume of elastomer with at least a first light source and a second light source, each light source disposed to illuminate the reflective surface from substantially different azimuths relative to each other;
contacting the reflective surface with an object to cause the reflective surface to deform in response to the contact by the object, the deformation causing localized changes in the surface normal of the reflective surface and associated localized changes in the amount of light from each of the light sources reflected from reflective surface;
receiving in a photosensing structure at least a portion of the light from the first light source reflected from the reflective surface in the form of a first image indicating one or more features of the object contacting the reflective surface;
receiving in the photosensing structure at least a portion of the light from the second light source reflected from the reflective surface in the form of a second image indicating one or more features of the object contacting the reflective surface, the first image differing from the second image; and
analyzing information about the one or more features indicated in the first and second images to estimate at least one of a surface normal and a surface height of the object at a plurality of positions on the object.
0. 1. A tactile sensor comprising:
a photosensing structure;
a volume of elastomer capable of transmitting an image; and
a reflective skin covering said volume of elastomer, said reflective skin being illuminated through said volume of elastomer by one or more light sources, said reflective skin having particles that non-directionally reflect light incident on the reflective skin from within the volume of elastomer, said reflective skin being geometrically altered in response to pressure applied by an entity touching said reflective skin, said geometrical alteration causing localized changes in the surface normal of said skin and associated localized changes in the amount of light reflected from said reflective skin in the direction of said photosensing structure; wherein
said photosensing structure is positioned to receive a portion of said reflected light in the form of an image, said image indicating one or more features of the entity producing said pressure.
0. 2. The tactile sensor of
0. 3. The tactile sensor of
0. 4. The tactile sensor of
0. 5. The tactile sensor of
0. 6. The tactile sensor of
0. 7. The tactile sensor of
0. 8. The tactile sensor of
0. 9. The tactile sensor of
0. 10. The tactile sensor of
0. 11. The tactile sensor of
0. 12. The tactile sensor of
0. 13. The tactile sensor of
0. 14. A method of performing tactile sensing, comprising:
(a) providing a volume of elastomer capable of transmitting an image;
(b) covering the elastomer with a reflective skin having an inner surface facing the elastomer and an outer surface, wherein the reflective skin comprises particles that non-directionally reflect light incident on the inner surface from within the volume of elastomer;
(c) illuminating the reflective skin through the volume of elastomer with one or more light sources, wherein at least a portion of the light is reflected by the inner surface of the reflective skin;
(d) contacting the outer surface of the reflective skin with an entity, wherein the contact produces pressure that geometrically alters the reflective skin, wherein the alteration causes localized changes in the inner surface of the reflective skin, and wherein the localized changes in the inner surface of the reflective skin cause associated localized changes in the light reflected from the inner surface of the reflective skin;
(e) positioning a photosensing structure to receive a portion of the light reflected from the inner surface of the reflective skin in the form of an image indicating one or more features of the entity contacting the outer surface of the reflective skin.
0. 15. The method of
0. 16. The method of
0. 17. The method of
0. 18. The method of
0. 19. The method of
0. 20. The method of
0. 21. The method of
0. 22. The method of
0. 23. The method of
0. 24. The method of
0. 25. The method of
0. 26. The method of
0. 28. The method of claim 27, wherein the reflective surface comprises reflective particles bound to the elastomer.
0. 29. The method of claim 27, wherein the reflective surface comprises reflective particles included in a paint, the paint being applied to the elastomer.
0. 30. The method of claim 27, wherein the reflective surface comprises a reflective skin in contact with the elastomer.
0. 31. The method of claim 30, wherein the reflective skin is bound to the elastomer.
0. 32. The method of claim 27, the photosensing structure comprising a camera.
0. 33. The method of claim 27, the reflective surface comprising reflective particles that reflect light substantially uniformly in all directions, the reflective particles causing the reflective surface to reflect light substantially non-directionally.
0. 34. The method of claim 27, the reflective surface comprising flakes that exhibit at least one of surface roughness, irregular shape, and random alignment relative to each other, the reflective particles causing the reflective surface to reflect light moderately directionally.
0. 35. The method of claim 27, the reflective surface comprising flakes that are substantially flat, have a mirror-like surface, and are substantially well aligned relative to each other, the reflective particles causing the reflective surface to reflect light highly directionally.
0. 36. The method of claim 27, the photosensing structure comprising a first camera and a second camera, the first image based on light reflected from the reflective surface being received in the first camera and the second image based on light reflected from the reflective surface being received in the second camera.
0. 37. The method of claim 27, the illuminating the reflective surface comprising alternating between:
illuminating the reflective surface with the first light source and receiving in the photosensing structure at least a portion of the light from the first light source reflected from the reflective surface; and
illuminating the reflective surface with the second light source and receiving in the photosensing structure at least a portion of the light from the second light source reflected from the reflective surface.
0. 38. The method of claim 27, wherein:
the illuminating the reflective surface comprises illuminating the reflective surface simultaneously with each of the first and the second light sources;
the first light source provides light of a first color and the second light source provides light of a second color;
the photosensing structure comprises a camera responsive to light of the first color and to light of the second color; and
the analyzing information about the one or more features indicated in the first and second images comprises distinguishing between the first and second images based on the color of the light reflected from the reflective surface.
0. 39. The method of claim 38, the photosensing structure comprising a first and a second camera, the first camera being responsive to light of the first color and the second camera being sensitive to light of the second color.
0. 41. The tactile sensor of claim 40, wherein the reflective surface comprises reflective particles bound to the elastomer.
0. 42. The tactile sensor of claim 40, wherein the reflective surface comprises reflective particles included in a paint, the paint being applied to the elastomer.
0. 43. The tactile sensor of claim 40, wherein the reflective surface comprises a reflective skin in contact with the elastomer.
0. 44. The tactile sensor of claim 43, wherein the reflective skin is bound to the elastomer.
0. 45. The sensor of claim 40, the photosensing structure comprising a camera.
0. 46. The sensor of claim 40, the reflective surface comprising reflective particles that reflect light substantially uniformly in all directions, the reflective particles causing the reflective surface to reflect light substantially non-directionally.
0. 47. The sensor of claim 40, the reflective surface comprising flakes that exhibit at least one of surface roughness, irregular shape, and random alignment relative to each other, the reflective particles causing the reflective surface to reflect light moderately directionally.
0. 48. The sensor of claim 40, the reflective surface comprising flakes that are substantially flat, have a mirror-like surface, and are substantially well aligned relative to each other, the reflective particles causing the reflective surface to reflect light highly directionally.
0. 49. The sensor of claim 40, the photosensing structure comprising a plurality of cameras, each camera of the plurality being disposed at an orientation that differs from any other camera relative to the reflective surface.
0. 50. The sensor of claim 40, further comprising an illumination controller for alternating between illuminating the reflective surface with the first light source and illuminating the reflective surface with the second light sources.
0. 51. The sensor of claim 40, wherein:
the first light source provides light of a first color and the second light source provides light of a second color;
the photosensing structure comprises a camera responsive to light of the first color and to light of the second color; and
the photometric stereo analyzer configured to estimate at least one of surface normal and the surface height of the object at a plurality of positions on the object based on the information by distinguishing between the first and the second images based on the color of the light reflected from the reflective surface.
0. 52. The sensor of claim 51, the photosensing structure comprising a first and a second camera, the first camera being responsive to the light of the first color and the second camera being responsive to the light of the second color.
|
FIG. 13 is a schematic diagram illustrating the elements of a rolling scanner.The sensor skin can be made by adding reflective particles to the elastomer when it is in a liquid state, via solvent or heat, or before curing. This makes a reflective paint that can be attached to the surface by standard coating techniques such as spraying or dipping. The skin may be coated directly on the surface of the bulk elastomer, or it may be first painted on a smooth medium such as glass and then transferred to the surface of the bulk material and bound there. Also, the particles (without binder) can be rubbed into the surface of the bulk elastomer, and then bound to the elastomer by heat or with a thin coat of material overlaid on the surface. Also, it may be possible to evaporate, precipitate, sputter, other otherwise attach thin films to the surface.
The reflective particles in the skin may reflect light directionally or non-directionally. If the particles reflect light uniformly in all directions regardless of the light's angle of incidence, the resulting skin will behave like a Lambertian surface, which is entirely non-directional. Titanium dioxide powder, as is used in white paint, leads to a largely Lambertian reflectance. If the reflective particles are comprised of fine metal flakes, and if these flakes tend to be aligned with each other, then the skin will reflect light directionally, meaning that, for a given angle of incident light, there will be a non-uniform distribution of reflected light. If the metal flakes are flat and mirror-like, and if they are well aligned with each other, the distribution of reflected light will be highly directional. If the metal flakes are rough or irregular, or if there is randomness in their alignment, then the distribution of reflected light will be moderately directional, with an appearance resembling sandblasted metal. Directional reflectance can also be obtained with flakes of other materials such as mica. In addition there are pigments comprising flakes covered with multilayer interference coatings that can have different directionality for different wavelengths of light. Further still, metal flake powder aluminum can be used.
Skin with highly directional reflectance illuminated by a highly directional light source yields a device that is sensitive to small variations in pressure. This sensitivity can be increased by recording the skin's image in its resting state, and using this as a baseline image that is subtracted from images recorded when pressure is applied to the skin. Softer elastomers lead to devices that are more sensitive to low amplitude pressure patterns.
Another example is shown in
The image pixel values do not directly encode pressure. If spatially uniform pressure is applied to the entire skin surface, there will be no change in surface normal and thus no observable variation in the image. The image pixel values depend on surface normal, which in turn depends on the spatial derivative of pressure. Thus, it is the pattern of pressure variation across the surface that is encoded in the image.
Pressure can be applied to the skin by a rigid object or a non-rigid object. In the case of a non-rigid object, such as a fingertip, both the object and the skin will deform, and the skin's shape will depend on the balance of pressures that the skin and the object exert on each other. Pressure can also be applied by a liquid or gas. For example, a stream of water striking the skin causes it to deform, and the pattern of deformation is visible in the image. If the skin and the elastomer are made of very soft gel-like materials, and if a froth of soap bubbles is placed in contact with the skin, one can visualize the forces exerted by the soap bubble walls.
To make the structure compact, it may be preferable to introduce the light at the edge of the support.
In some applications it is desirable that the light source and the camera be placed at optical infinity so that the angle of incidence and reflectance are parallel when the device is in its resting state. This causes the devices optical properties to be spatially uniform across the recorded image.
In accordance with another exemplary embodiment of the invention, it is desirable to reconstruct the 3-D shape of the deformed surface. In
The use of multiple lights to get multiple images is useful even when 3D reconstruction is not being performed. Each light brings out surface normal variation along one axis, but not along the orthogonal axis. By using two or more lights, the lights can be arranged so that one light reveals the relief that is missed by another light. This makes it possible to distinguish a wide range of surface normals in different directions. The preferred method of using two or more lights is to have them be different colors, so that a color camera will separate the information about the different lights into different color channels.
It is also possible to use standard stereoscopic techniques as well, in which multiple cameras are placed in different azimuths relative to the skin of the sensor. The techniques set forth herein of sensing changes in luminance are believed to yield better sensitivity and resolution in many applications relative to known techniques. Furthermore, stereopsis can be used in combination with the techniques disclosed herein. The microtexture of the skin and/or the deformation image can be used to establish correspondence.
In some applications it is desirable to make a sensor surface that covers a large area. For example one may require a touchpad that covers an entire desktop. If the device is simply scaled up, then the camera must be placed at a large distance from the surface, making the device undesirably large. One way to ameliorate this problem is to use the methods of folded optics that are used, for example, in many rear projection televisions. Another way is to use a tiled array of cameras, as shown in
In another embodiment, a sensor is formed into a cylinder, which can be rolled over the surface of the object. In one illustrative configuration, the sensor would look like a brayer or paint roller. As the roller rolls over the surface, a video camera inside the roller is aimed continuously at the portion that is in contact with the surface of the object. The series of images so obtained can be combined into a single image by the same methods that are used to obtain panoramic photographs from a series of smaller photographs.
Alternatively, the roller could be in a fixed position while the surface of interest was pulled over it. For example, as shown in FIG. 13, an illustrative embodiment of a rolling scanner is a counterfeit banknote detector 300 which has a clear elastomer 310 formed into a cylinder, and a reflective skin 320. The detector 300 also has a slot 330 for feeding a banknote 340. A roller 350, driven by a motor, pulls in the banknote 340 and forms an image of the banknote's surface shape based on an embossed surface of the banknote's paper and/or the raised level of printed ink. A genuine banknote will have a known profile, and a mismatch would indicate a counterfeit banknote.
In some applications it is advantageous for the skin to have a texture rather than being smooth. In some situations one wishes to study the distribution of pressures across a region of human skin. For example, when a skin care product is applied to the skin, the application process produces a certain distribution of pressure on the skin which changes over time. In order to estimate this changing distribution, a sensor can be made that mimics the texture, elasticity, and other properties of human skin. When a skin care product is, for example, wiped across the artificial skin, it causes the skin to distort in a manner similar to that of human skin. The pattern of distortion can be assessed by making a tactile sensor with mechanical properties emulating human skin. This means that the reflective skin has texture and elasticity like the upper layer of human skin, and the clear elastomer beneath the skin has mechanical properties like the deeper layers of human skin. Multiple layers of clear elastomer with different mechanical properties are required to mimic the complex properties of human skin. When a skin care product is applied to this device, the reflective skin distorts in response to the mechanical forces applied to it, and this distortion is viewed by a camera looking through the clear elastomer layers.
In some applications it may be desired to study the distribution of pressure over the surface of a specific object, such as a tire or the sole of a shoe. It is possible to form the tactile sensor into the same shape as this specific object, and with the same hardness or other mechanical properties as this specific object.
In other embodiments, it is not necessary that the image be formed by a camera. Many flatbed scanners use a Contact Image Sensor, or CIS, which is a linear array of lenses and photosensors placed in close contact with the object being scanned. No image forming lens is required.
In another embodiment, a multitouch touchscreen device is made in conjunction with a flat panel LED display. A thin sheet of clear elastomer, covered with a semi-reflective skin, covers the front surface of the display. Most of the light that is emitted by the LED's passes through the skin and is seen by a viewer. A portion of the light is reflected by the skin back toward the LEDs. LEDs have the ability to act as photosensors, and thus can be used to measure the amount of reflected light. Pressure variation on the skin causes local changes in the surface normal, which changes the amount of light reflected toward any given LED in the array. The LED photosensing responses comprise an image that is indicative of the pattern of pressure on the skin. This image indicates where the user is touching the screen. In addition, because this is an inherently high resolution image, it is possible to detect the fingerprint of the user. This allows each finger of each user to be distinguished.
In another exemplary embodiment, the device is used to measure fluid flow.
There are applications for which high resolution is not needed and not desirable. An extra layer of elastomer on top of the skin acts as a mechanical lowpass filter. For example, a 1 mm thick layer reduces the resolution to be on the order of 1 mm.
A fluorescent pigment can be used in the skin, illuminated by Ultraviolet (UV) light or blacklight. If the blacklight comes at a grazing angle, it can readily reveal variations in surface normal. The material will be fairly close to Lambertian. To reduce interreflections, one would select a surface that appears dark to emitted wavelengths. This principle is true with ordinary light as well. If one is using a Lambertian pigment in the skin, it is better for it to be gray than white, to reduce interreflections.
Blacklight or UV can be used to illuminate a fluorescent surface, which would then serve as a diffuse source. In some cases, it would be useful to use a single short flash (for instance, recording the instantaneous deformation of an object against the surface) or multiple periodic (strobed) flashes (to capture rapid periodic events or to modulate one frequency down to another frequency.)
Although the present invention has been shown and described with respect to several preferred embodiments thereof, various changes, omissions and additions to the form and detail thereof, may be made therein, without departing from the spirit and scope of the invention.
Patent | Priority | Assignee | Title |
10837916, | Mar 09 2017 | Spirit AeroSystems, Inc. | Optical measurement device for inspection of discontinuities in aerostructures |
9870101, | Aug 22 2014 | Toyota Jidosha Kabushiki Kaisha | Operating device for vehicle having a screen with an operating surface and a projecting unit configured to project an image on the screen |
Patent | Priority | Assignee | Title |
4120585, | Nov 19 1976 | FINGERMATRIX, INC , A NY CORP | Fingerprint identification system using a pliable optical prism |
4340300, | Aug 11 1980 | Siemens Corporation | Input sensor unit for a fingerprint identification system |
4481815, | Dec 23 1982 | Tactile sensor | |
4549093, | Sep 29 1982 | Emi Limited | Tactile array sensor |
4584625, | Sep 11 1984 | Capacitive tactile sensor | |
4588348, | May 27 1983 | AT&T Bell Laboratories | Robotic system utilizing a tactile sensor array |
4775961, | Jun 24 1985 | VYSOKA SKOLA DOPRAVY A SPOJOV V ZILINE, A CORP OF 20 MOYZESOVA ZILINA, CZECHOSLOVAKIA | Tactile sensor |
5357799, | Jul 14 1992 | Regents of the University of California, The | Method and apparatus for determining abrasion potential of tire treads |
5448649, | May 24 1994 | Apparatus for imaging fingerprint or topographic relief pattern on the surface of an object | |
5616839, | Jan 29 1996 | Ford Motor Company | Apparatus and method for measuring contact force distribution of a tire |
5737071, | Aug 16 1996 | Safariland, LLC | Method and apparatus for enhancing live-scan fingerprint reader images |
5967940, | Sep 17 1997 | Toyota Jidosha Kabushiki Kaisha | Method and apparatus for reducing backlash sound in gear mechanism |
5967990, | Aug 13 1998 | President and Fellows of Harvard College | Surgical probe comprising visible markings on an elastic membrane |
6144757, | Feb 28 1997 | NEC Corporation | System and method for identifying an organism |
6234031, | Nov 27 1997 | NEC Corporaion | Fingerprint detection apparatus |
6909084, | Aug 31 2000 | Toudai TLO, Ltd | Optical tactile sensor having a transparent elastic tactile portion |
7707001, | Mar 09 2004 | Nagoya Industrial Science Research Institute | Control of object operating force, object gripping force and robot hands |
20030178556, | |||
20040237669, | |||
20040252867, | |||
20040255128, | |||
20060119837, | |||
20070146536, | |||
20070288186, | |||
20080106258, | |||
20080219521, | |||
20080284925, | |||
20100284565, | |||
20130070074, | |||
JP2005121858, | |||
JP2007520040, | |||
JP2008008746, | |||
WO2005085766, | |||
WO2006098719, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Oct 03 2013 | Massachusetts Institute of Technology | (assignment on the face of the patent) | / | |||
Jul 11 2018 | Massachusetts Institute of Technology | NATIONAL SCIENCE FOUNDATION | CONFIRMATORY LICENSE SEE DOCUMENT FOR DETAILS | 047287 | /0422 |
Date | Maintenance Fee Events |
Jul 30 2014 | ASPN: Payor Number Assigned. |
Sep 19 2016 | STOL: Pat Hldr no Longer Claims Small Ent Stat |
Oct 02 2020 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Date | Maintenance Schedule |
Oct 07 2017 | 4 years fee payment window open |
Apr 07 2018 | 6 months grace period start (w surcharge) |
Oct 07 2018 | patent expiry (for year 4) |
Oct 07 2020 | 2 years to revive unintentionally abandoned end. (for year 4) |
Oct 07 2021 | 8 years fee payment window open |
Apr 07 2022 | 6 months grace period start (w surcharge) |
Oct 07 2022 | patent expiry (for year 8) |
Oct 07 2024 | 2 years to revive unintentionally abandoned end. (for year 8) |
Oct 07 2025 | 12 years fee payment window open |
Apr 07 2026 | 6 months grace period start (w surcharge) |
Oct 07 2026 | patent expiry (for year 12) |
Oct 07 2028 | 2 years to revive unintentionally abandoned end. (for year 12) |