An augmented reality environment allows interaction between virtual and real objects and enhances an unstructured real-world environment. An object datastore comprising attributes of an object within the environment may be built and/or maintained from sources including manufacturers, retailers, shippers, and users. This object datastore may be local, cloud based, or a combination thereof. Applications may interrogate the object datastore to provide user functionality.
|
1. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause one or more processors to perform acts comprising:
detecting an object in an environment from an image of the environment;
determining one or more attributes about the object via one or more sensors within the environment;
generating a preliminary identification of the object based at least in part on a comparison of the one or more attributes to attributes maintained in an object datastore;
calculating a confidence value for the preliminary identification;
comparing the confidence value for the preliminary identification to a confidence threshold;
in response to the confidence value for the preliminary identification reaching or exceeding the confidence threshold, identifying the object using the preliminary identification; and
in response to the confidence value for the preliminary identification being below the confidence threshold:
providing a visual indicator to identify the object, wherein the visual indicator is displayed proximate to the object or overlapping with the object;
querying a user for identification information associated with the object identified by the visual indicator; and
storing the identification information associated with the object in the object datastore.
11. A system comprising:
one or more processors; and
memory storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:
detecting an object in an environment from an image of the environment;
determining one or more attributes about the object via one or more sensors within the environment;
generating a preliminary identification of the object based at least in part on a comparison of the one or more attributes to attributes maintained in an object datastore;
calculating a confidence value for the preliminary identification;
comparing the confidence value for the preliminary identification to a confidence threshold;
in response to the confidence value for the preliminary identification reaching or exceeding the confidence threshold, identifying the object using the preliminary identification; and
in response to the confidence value for the preliminary identification being below the confidence threshold:
providing a visual indicator to identify the object, wherein the visual indicator is displayed proximate to the object or overlapping with the object;
querying a user for identification information associated with the object identified by the visual indicator; and
storing the identification information associated with the object in the object datastore.
6. A computer-implemented method comprising:
detecting an object in an environment from an image of the environment;
determining, via at least one computing device, one or more attributes about the object via one or more sensors within the environment;
generating, via the one or more computing devices, a preliminary identification of the object based at least in part on a comparison of the one or more attributes to attributes maintained in an object datastore;
calculating, via the at least one computing device, a confidence value for the preliminary identification;
comparing, via the at least one computing device, the confidence value for the preliminary identification to a confidence threshold;
in response to the confidence value for the preliminary identification reaching or exceeding the confidence threshold, identifying, via the at least one computing device, the object using the preliminary identification; and
in response to the confidence value for the preliminary identification being below the confidence threshold and via the at least one computing device:
providing a visual indicator to identify the object, wherein the visual indicator is displayed proximate to the object or overlapping with the object;
querying a user for identification information associated with the object identified by the visual indicator; and
storing the identification information associated with the object in the object datastore.
2. The one or more non-transitory computer-readable storage media of
3. The one or more non-transitory computer-readable storage media of
recognizing speech of the user at least partly in response to the querying; and
wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the speech.
4. The one or more non-transitory computer-readable storage media of
recognizing a gesture made by the user at least partly in response to the querying; and
wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the gesture.
5. The one or more non-transitory computer-readable storage media of
7. The computer-implemented method of
8. The computer-implemented method of
recognizing speech of the user at least partly in response to the querying; and
wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the speech.
9. The computer-implemented method of
recognizing a gesture made by the user at least partly in response to the querying; and
wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the gesture.
10. The computer-implemented method of
12. The system of
13. The system of
recognizing speech of the user at least partly in response to the querying; and
wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the speech.
14. The system of
recognizing a gesture made by the user at least partly in response to the querying; and
wherein the storing of the identification information in the object datastore occurs at least partly in response to the recognizing of the gesture.
15. The system of
|
|||||||||||||||||||||||||
Augmented reality environments allow interaction among users and real-world objects and virtual or computer-generated objects and information. This merger between the real and virtual worlds paves the way for new interaction opportunities.
The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.
An augmented reality system may be configured to interact with objects within a scene and generate an augmented reality environment. The augmented reality environment allows for virtual objects and information to merge and interact with tangible real-world objects, and vice versa. Disclosed herein are techniques and devices suitable for storing attributes about those objects in an object datastore and accessing that datastore within the augmented reality environment.
The augmented reality system includes a computing device configured to support the augmented reality environment. Objects and attributes associated with those objects may be stored in an object datastore. The object datastore may be queried to allow for functionality within the augmented reality environment. For example, a user may request an inventory of items within a room for insurance purposes or query the object datastore to look for a misplaced pair of sunglasses.
The object datastore may be local to the computing device, accessible to the computing device via a local area network, a cloud or remote resource accessible via a wide area network such as the Internet, or a combination thereof. For example, a local datastore may contain objects which are currently or likely to be in the user's environment while the cloud object datastore may contain aggregated data for many more objects.
The object datastore may be populated by data transfers or by input of information from sensors in the augmented reality environment. Data transfer for attributes of an object may be provided to the object datastore by entities handling goods. For example, upon purchase of an object the merchant may provide a model number and device serial number for inclusion in the object datastore. A shipping company transporting the object may provide a date of acquisition, mass, and overall volume of the object for inclusion in the object datastore. In another example, a manufacturer may provide data about functional attributes.
The object datastore may also be populated by detecting the object in the augmented reality environment. Using sensors accessible to the computing device which maintains the augmented reality environment, attributes of the detected objects are determined. These determined attributes may be used to query the object datastore in an attempt to identify the object. When a confidence value of the identification is below a pre-determined confidence threshold, the environment may query a user for assistance in identifying the object or for refined identification. For example, the environment may project an image of an illuminated circle around a detected but unidentified object and ask the user to identify this thing, such as via a speaker and speech synthesis module. The user may provide a refined identification such as “a can of diet Dr. Salt flavored soda” or at least a partial identification or classification such as “a can of soda” to assist the system in further identification. Once the user has made the identification, either refined or partial, the identification may be stored within the object datastore for later use. This later use may include further processing to improved recognition of other objects within the environment, as well as for disambiguation of objects such as those objects which have a partial identification.
The object datastore may include several categories of attributes including, but not limited to, specific object attributes, physical attributes, functional attributes, ownership attributes, and location attributes. Each of these categories, in turn, may store several attributes and sub-attributes. The specific object attributes comprise attributes unique to a particular object. For example, a unique serial number. The physical attributes comprise physical attributes inherent in and presented by the object. For example, the size and shape of the object. The functional attributes comprise functional and operational constraints associated with the object, such as those which affect use of the object. For example, intended uses and safety parameters. The ownership attributes comprise property rights associated with the object. For example, user Alice owns the object and may freely sell or lease it. The location attributes comprise location data of the object. For example, the current location of the object in the augmented reality environment. The object datastore and details about the various attributes are discussed in detail below.
As described herein for illustration and not by way of limitation, the augmented reality environment may be provided at least in part by a combination of a structured light source, such as an image projector or other light source configured to generate structured light patterns and a camera to image those patterns. The projector and camera may further be incorporated into a single unit and designated as an augmented reality functional node (ARFN). In other implementations, other combinations of elements such as projectors, video displays, computer screens, cameras, microphones, ultrasound transducers, depth-sensing devices, weight sensors, touch sensors, tactile output devices, and so forth may be also be present within or coupled to the ARFN. For convenience, and not by way of limitation, the examples in this disclosure refer to the use of structured light for the characterization of the physical environment of the scene including objects therein. However, in addition to or in place of structured light, other techniques may be used such as light detection and ranging (LIDAR), optical time-of-flight, ultrasonic ranging, stereoscopic imaging, radar, infrared scanning, and so forth either alone or in combination with one another.
Within the augmented reality environment, the ARFN includes an augmented reality module. This module is configured to identify and track objects within the scene, maintain a user interface, and facilitate queries of the object datastore, as discussed in detail below.
Illustrative Environment
Each of the ARFNs 102(1)-(N) couples to or includes a computing device 104. This computing device 104 may be within the ARFN 102, or disposed at another location and connected to the ARFN 102. The computing device 104 comprises a processor 106, an input/output interface 108, and a memory 110. The processor 106 may comprise one or more processors configured to execute instructions. The instructions may be stored in memory 110, or in other memory accessible to the processor 106.
The input/output interface 108 may be configured to couple the computing device 104 to other components such as projector, cameras, microphones, other ARFNs 102, other computing devices, other devices within the augmented reality environment, and so forth. For example, the input/output interface 108 may be configured to exchange data with computing devices, cleaning robots, home automation devices, and so forth in the environment. The coupling between the computing device 104 and the devices may be via wire, fiber optic cable, or wireless connection including but not limited to radio frequency, optical, or acoustic signals.
The memory 110 may include computer-readable storage media (“CRSM”). The CRSM may be any available physical media accessible by a computing device to implement the instructions stored thereon. CRSM may include, but is not limited to, random access memory (“RAM”), read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technology, compact disk read-only memory (“CD-ROM”), digital versatile disks (“DVD”) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computing device.
Several modules such as instructions, datastores, and so forth may be stored within the memory 110 and configured to execute on a processor, such as the processor 106. An operating system module 112 is configured to manage hardware and services within and coupled to the computing device 104 for the benefit of other modules. Modules may be stored in the memory of the ARFN 102, storage devices accessible on the local network, cloud storage accessible via a wide area network, or distributed across a combination thereof.
An object datastore 114 is configured to maintain information about objects within or accessible to the augmented reality environment or users. These objects may be tangible real world objects or virtual objects. Tangible objects include items such as tables, chairs, animals, plants, food containers, printed books, and so forth. Virtual objects include media content such as songs, movies, electronic books, computer generated scenes, media files, and so forth. Virtual objects may include stored copies of those objects or access rights thereto. The object datastore 114 may include a library of pre-loaded reference objects, as well as objects which are temporally persistent within a particular environment, such as a wall, a specific table, a user and so forth. The object datastore 114 or the other datastores described below may be stored on one or more of the memory of the ARFN 102, storage devices accessible on the local network, cloud storage accessible via a wide area network, or distributed across one or more of these. Data within the object datastore 114 may be stored in hierarchical, non-hierarchical, or hybrid arrangements. In a hierarchical structure objects may be classified in a structure which may be taxonomic in nature. For example, an object may be defined in the datastore as inanimate, metal exterior, food, ready-to-eat, beverage, soda, grape flavor. Non-hierarchical systems may comprise systems where apparently arbitrary or variable connections are established between objects. These arbitrary associations may be between physical and virtual objects, such as associating a grouping of digital photos with a small statute of the Eiffel tower. The object datastore 114 is discussed in more detail below in regards to
A user interaction datastore 116 is configured to maintain information about interactions between one or more users and one or more objects. The user interaction data within the datastore 116 may be used in some implementations to facilitate additional functions such as to receive indications of user selections of advertisements, providing usage data, and so forth. The collection and use of the user interaction data may be limited based upon the preferences of the user. Changes in objects attributes as a result of interactions may be stored in the object datastore 114. For example, when the user moves a pair of sunglasses from one part of the augmented reality environment to another, the previous and current location attributes may be stored in the object datastore 114.
A transaction datastore 118 is configured to maintain information about changes in property rights associated with objects. For example, the datastore 118 may be configured to track the purchase, rental, sale, lease, and so forth of objects. Changes to the ownership rights may update the ownership attributes in the object datastore 114.
An advertisement datastore 120 is configured to maintain information about advertisements available to the augmented reality environment. These advertisements may be designated for presentation based on one or more of the attributes maintained within the object datastore 114. For example, a user bringing an apple into the augmented reality environment may be presented with an advertisement for a prepackaged apple pie the user has previously purchased.
An augmented reality module 122 is configured to generate augmented reality output in concert with the physical environment. The module 122 may access one or more of the datastores described herein. The augmented reality module 122 may include a tracking and control module 124 is configured to identify objects including users. This identification may include the use of a camera, structured light, radio frequency identification equipment, communication with the object, and so forth within the ARFN 102. Objects may be identified as described below by comparing attributes in the object datastore 114 such as shape, text thereon, universal product code (UPC), object behaviors (including an ability to move independently), optical barcode, radio frequency identification tag (RFID), and so forth. Where the object is capable of communication, it may exchange information with the ARFN 102, and such information may be stored within the object datastore 114.
The term object encompasses several categories including unitary, package, group, container, and so forth. As described above, objects may be hierarchically or arbitrarily associated with one another. Some objects may be unitary in that they are complete in and of themselves, such as a baseball bat or wrench. Some objects may be packages containing a plurality of items which may or may not be identical, such as a twelve-can case of cola or set of different sized wrenches. An object may also be a group of closely interrelated sub-objects, which operate together to form a whole, such as a jigsaw puzzle. An object may also be a container holding other objects, such as a filled tote with the significance being the contents of the tote and not necessarily the container itself. Different attributes may be stored for different categories of objects.
The category of an object may be determined at least in part by factors including context, prior use, pre-determined data, and so forth. For example in the context of home use and purchase, wine may be considered a unitary object and designated by the bottle not by the case. Similarly, pre-determined data may indicate that totes are used for storage, and as such a user pointing to a tote filled with objects may be determined as a default to be designating the objects within the tote, and not the tote itself.
A user interface module 126 which may be in the augmented reality module 122 is configured to accept and interpret input and generate output for the user. The ARFN 102 may use a camera, structured light, stereoscopic vision, and so forth to read the input from the user. The user interface module 126 may also include an object datastore query module configured to allow applications to access information in the object datastore 114. For example, a user may query the object datastore 114 to locate his misplaced sunglasses. The augmented reality module 122 may also include a transaction module 128 is configured to associate objects with advertisements, facilitate transactions involving third parties such as buying and selling objects, and perform other functions.
A chassis 204 holds the components of the ARFN 102. Within the chassis 204 may be disposed a projector 206 that generates and projects images into the scene 202. These images may be visible light images perceptible to the user, visible light images imperceptible to the user, images with non-visible light, or a combination thereof. This projector 206 may be implemented with any number of technologies capable of generating an image and projecting that image onto a surface within the environment. Suitable technologies include a digital micromirror device (DMD), liquid crystal on silicon display (LCOS), liquid crystal display, 3LCD, and so forth. The projector 206 has a projector field of view 208 which describes a particular solid angle. The projector field of view 208 may vary according to changes in the configuration of the projector. For example, the projector field of view 208 may narrow upon application of an optical zoom to the projector. In some implementations, a plurality of projectors 206 or other displays such as televisions, monitors, and so forth may be used.
A camera 210 may also be disposed within the chassis 204. The camera 210 is configured to image the scene in visible light wavelengths, non-visible light wavelengths, or both. For example, in one implementation the camera 210 may be configured to generate a thermal image as well as a visible light image. The camera 210 has a camera field of view 212 which describes a particular solid angle. The camera field of view 212 may vary according to changes in the configuration of the camera 210. For example, an optical zoom of the camera may narrow the camera field of view 212. In some implementations, a plurality of cameras 210 may be used.
The chassis 204 may be mounted with a fixed orientation, or be coupled via an actuator to a fixture such that the chassis 204 may move. Actuators may include piezoelectric actuators, motors, linear actuators, and other devices configured to displace or move the chassis 204 or components therein such as the projector 206 and/or the camera 210. For example, in one implementation the actuator may comprise a pan motor 214, tilt motor 216, and so forth. The pan motor 214 is configured to rotate the chassis 204 in a yawing motion. The tilt motor 216 is configured to change the pitch of the chassis 204. By panning and/or tilting the chassis 204, different views of the scene may be acquired. The spatial analysis module 114 may use the different views to monitor objects within the environment.
One or more microphones 218 may be disposed within the chassis 204, or elsewhere within the scene. These microphones 218 may be used to acquire input from the user, for echolocation, location determination of a sound, or to otherwise aid in the characterization of and receipt of input from the scene. For example, the user may make a particular noise, such as a tap on a wall or snap of the fingers, which are pre-designated as attention command inputs. The user may alternatively use voice commands. Such audio inputs may be located within the scene using time-of-arrival or other techniques among the microphones.
One or more speakers 220 may also be present to provide for audible output. For example, the speakers 220 may be used to provide output from a text-to-speech module or to playback pre-recorded audio.
A transducer 222 may be present within the ARFN 102, or elsewhere within the environment, and configured to detect and/or generate inaudible signals, such as infrasound or ultrasound. For example, the transducer 222 may be configured to detect a characteristic ultrasonic sound signature produced by keys on a keyring. Inaudible signals may also be used to provide for signaling between accessory devices and the ARFN 102.
A ranging system 224 may also be provided in the ARFN 102. The ranging system 224 is configured to provide distance information from the ARFN 102 to a scanned object or set of objects. The ranging system 224 may comprise radar, light detection and ranging (LIDAR), ultrasonic ranging, stereoscopic ranging, and so forth. In some implementations the transducer 222, the microphones 218, the speaker 220, or a combination thereof may be configured to use echolocation or echo-ranging to determine distance and spatial characteristics.
In this illustration, the computing device 104 is shown within the chassis 204. However, in other implementations all or a portion of the computing device 104 may be disposed in another location and coupled to the ARFN 102. This coupling may occur via wire, fiber optic cable, wirelessly, or a combination thereof. Furthermore, additional resources external to the ARFN 102 may be accessed, such as resources in another ARFN 102 accessible via a local area network, cloud resources accessible via a wide area network connection, or a combination thereof.
Also shown in this illustration is a projector/camera linear offset designated “O”. This is a linear distance between the projector 206 and the camera 210. Placement of the projector 206 and the camera 210 at distance “O” from one another aids in the recovery of structured light data from the scene. The known projector/camera linear offset “O” may also be used to calculate distances, dimensioning, and otherwise aid in the characterization of objects within the scene 202. In other implementations the relative angle and size of the projector field of view 208 and camera field of view 212 may vary. Also, the angle of the projector 206 and the camera 210 relative to the chassis 204 may vary.
In other implementations, the components of the ARFN 102 may be distributed in one or more locations within the environment 100. As mentioned above, microphones 218 and speakers 220 may be distributed throughout the scene. The projector 206 and the camera 210 may also be located in separate chassis 204. The ARFN 102 may also include discrete portable signaling devices used by users to issue command attention inputs. For example, these may be acoustic clickers (audible or ultrasonic), electronic signaling devices such as infrared emitters, radio transmitters, and so forth.
Resources available via the network 304 may include cloud object datastore servers 306. The cloud or remote object datastore may comprise an extended object datastore. Transfers of data such as received from other entities as shipping manifests, inventory data, manufacturing data, and so forth may build this object datastore as described below. The cloud object datastore may also be populated by aggregating at least a subset of the local object datastores from augmented reality environments. Such aggregation may take place after user approval and may include providing data comprising one or more object attributes in the local object datastore to the cloud object datastore. The cloud object datastore servers 306 receive the data and may integrate that data into the datastore. This integration may involve de-duplication, confirmation of identification, analysis of attributes to determine a range of possible values for a given object or class of object, and so forth. For example, the range of possible shapes and colors of apples may be determined from a sample of the attributes of thousands of individual apples across many augmented reality environments.
The cloud object datastore servers 306 are configured to provide data in the cloud object datastore to applications. In some implementations, the providing may comprise maintaining an application programming interface or other service configured to respond to queries against the datastore. For example, an application executing on a given computing device 104 may attempt to identify a detected object and query the cloud object datastore servers 306 to search the larger datastore contained therein
Object attributes 308(1), 308(2), . . . , 308(A) may be transferred via the network 304 between devices. For example, the computing device 104 may transfer object attributes 308(1) for the detected but unidentified object to the cloud object datastore servers 306 for refined identification. In other implementations the cloud object datastore servers 306 or other servers may exchange object attributes 308. For example, merchant servers 310 may provide object attributes 308(2) associated with purchased objects to the object datastore 114 of the computing device 104, cloud object datastore servers 306, and so forth. The merchant servers 310 may include bricks-and-mortar as well as online stores.
Likewise, the merchant servers 310 may request data from the cloud object datastore servers 306 via the network 304 to determine actual usage statistics for an object. For example, the merchant servers 310 may wish to determine how long a toy dump truck is played with before being discarded or having ownership transferred to another user. Such usage information may be useful in product development, marketing, and so forth.
Shipper servers 312 may be configured to couple to the network 304 and exchange object attributes 308 or other data with other devices. A common carrier, freight company, and so forth may maintain or provide logistics-related data to the shipper servers 312 relating to objects. For example, the shipper servers 312 may contain a manifest which lists objects in a given shipment, delivery information, weight, volumetric data, and so forth. The computing device 104 may populate the object datastore 114 by receiving object attributes 308(4) including this manifest data, such that when the user is unpacking the box, the augmented reality environment is able to identify the objects from the set of objects which were packed in the box per the manifest, rather than all objects existing in the aggregated cloud object datastore.
Marketing servers 314 may also access object attributes 308 via the network 304. The marketing servers 314 may be configured to query the object datastore to determine usage patterns, purchasing patterns, and so forth. Such information may then be used to develop marketing strategies. For example, where the augmented reality identifies that the user has a collection of porcelain figurines in his augmented reality environment, advertisements specific to that user's apparent interest in figurines may be provided.
The object datastore 114 may store several categories of attributes including, but not limited to, specific object attributes 402, physical attributes 404, functional attributes 406, ownership attributes 408, and location attributes 410. Each of these categories, in turn, may include several attributes and sub-attributes. These categories are provided by way of example, and in some implementations attributes or sub-attributes may be present in a plurality of categories. The categories are discussed next with regards to
A unique identifier 502 may be stored. This unique identifier may include a serial number, global unique identifier, and so forth. The unique identifier 502 may be considered to distinguish this object from all other similar or otherwise identical objects. The unique identifier 502 may be embedded within the device, affixed thereto with a label or tag, or otherwise associated therewith. For example, a portable game console may have a unique serial number embedded within its firmware. The unique identifier 502 may be stored on a tag such as an optical barcode or radio frequency identification chip.
An expiration date 504 may be associated with the object. The expiration date 504 is a calendar date or interval after which the object is considered to be degraded or at which time the object becomes unusable. For example, a printer cartridge may be configured to have an operational lifetime and thus expiration date of 120 days after installation in a printer.
A unique flag 506 indicates whether this specific object is considered unique. For example, the unique flag 506 for a custom-made diamond necklace may be “yes” while the unique flag 506 for a pencil is set to “no.”
A preliminary/refined identification (flag) 508 may also be present in the object datastore 114. This flag 508 may be used to indicate when the specific object has received a preliminary identification which may be subject to change, or a refined identification which is considered to be absolute.
An identification confidence threshold 510 may also be maintained. The identification confidence threshold 510 indicates a level at which a calculated confidence value for an object is deemed sufficient to warrant the identification. The confidence value may be based on the comparison of one or more attributes of the object with data in the object datastore 114 for other objects, reliability of a user providing the identification, and so forth. An object may also have different confidence thresholds 510. For example, an identification provided by an authorized adult user may be given a 95% confidence threshold while an identification by an unauthorized child user may be given a 20% confidence threshold. The environment may be configured such that objects having a confidence value above a pre-determined threshold confidence threshold are deemed to have a refined identification. Likewise, objects with a calculated confidence value below a pre-determined confidence threshold may be assigned a preliminary identification which may be subject to change as additional data becomes available to the augmented reality environment.
Specific object attributes 402 may also include a hardware address 512 or other data networking parameter. This hardware address 512 may comprise a hardware or media-access control address, and so forth.
Size of the object 602 may be stored in the object datastore 114. For example, the size may include a volumetric measurement such as the object fits within a cuboid of dimensions 32 cm by 16 cm by 51 cm.
Mass 604 of the object may be stored. In one implementation a pressure sensitive touch sensor or scale in the augmented reality environment may provide an indication of the object's mass. In another implementation, the mass may be acquired from a third-party, such as from a shipper or merchant who is transporting or selling the object.
Topology 606 is a two- or three-dimensional map or representation of a configuration or shape of at least a portion of the object's exterior. For example, the overall topology of a beverage can is that of a cylinder, while the overall topology of a dog is more complex.
Color 608 of the object may also be stored in the object datastore 114. The color 608 may be determined from ambient light or from the interaction of different spectral bands of light as generated at particular times by a wavelength modulated light source. The color 608 may also include sub-attributes such as reflectance values or spectral responses for different spectral bands. In some implementations, the spectral data may be used to characterize the composition of the object.
Texture 610 of the object may acquired and stored as well. The texture of the object is the disposition or arrangement of relatively small scale surface features. For example, the texture of a tennis ball is fibrous while texture of a golf ball is smooth with dimples. Texture may be determined by imaging, a scan such as with structured light or LIDAR, tactile sensor, and so forth. As with any of the attributes in the object datastore 114, many textures 610 may be stored and associated with a given object or set of objects. For example, the texture of a user's face may change as his beard grows.
Baseline sound signatures 612 may also be stored within an object datastore 114. The sound signatures described herein may include sounds audible to the human ear, as well as inaudible sounds such as infrasound or ultrasound. Some objects produce a baseline of specific waveforms. Objects which are electrical or mechanical may produce particular sounds while operating. For example, a particular laptop may produce a distinctive sound due to cooling fans and the particular airflow of that device. Objects which are organic may produce particular sounds such as breathing.
Comparison of the baseline sound signature 612 or waveform may aid in identification a detected object. Returning to the laptop example, a portable computer device which is detected and has an operating fan could not be a portable device with no fan. As a result, the augmented reality environment may limit the search in the object datastore 114 for objects having cooling fans or the baseline sound signature 612 associated therewith.
An interacting sound signature 614 of the object interacting with the environment or users may also be stored. This sound signature 614 results during the object interacting with another object in the augmented reality environment rather than passively sitting there. For example, a key ring full of keys sitting on a table may have no operating sound signature 612. However, a user picking up those keys may cause them to “jingle” and emit a particular sound signature 614. This sound signature 614 may include audible and ultrasonic aspects. Similarly, the user placing the keys on a table may also generate a particular interacting sound signature 614.
Physical attributes 404 may also include a thermal signature 616. This thermal signature 616 may include data which indicates the particular range of temperatures the object is typically expected to have. For example, the thermal signature 616 for a human would be between about 96° Fahrenheit (F) and 105° F. The thermal signature 616 may also be used to distinguish and in some implementations aid in the identification of an object. For example, the augmented reality environment may be able to distinguish a bowl of frozen yogurt from a bowl of pudding based at least in part upon the temperature difference.
The thermal signature 616 attribute may comprise coarse temperature or “spot” measurements such as determined by an infrared photodetector or thermal image data such as acquired from an infrared camera. Where the thermal signature 616 comprises thermal image data, this image data may further be used to characterize an object. For example, thermal images of faces may be used to distinguish one user from another. Or the thermal image of a laptop may serve to distinguish one model from another based on the different relative position of relatively hot internal components such as the processor.
The physical attributes 404 may also include power requirements 618 for the object. Objects which consume electrical power may have their power requirements 618 stored in the object datastore 114. These power requirements may include the operating power requirements, actual power consumption, low power or sleep mode power requirements, and so forth. In some implementations electrical supplies such as a residential or commercial power system may provide for the exchange of data with the augmented reality environment. For example, a “smart grid” electric meter may be configured to provide data to the augmented reality environment. Where an individual object is so equipped, such as a personal computer or laptop, the object may provide data regarding electrical power consumption, requirements, and so forth to the augmented reality environment. In some implementations, a sub-attribute may include thermal power requirements, such as thermal power output during operation.
User contact points 620 may also be maintained. Many objects have a particular shape which is defined by how a user is intended to interact with the object. The user contact points 620 are where a user typically touches or otherwise interacts with the device. By monitoring the interaction of the user with the object in the environment, the user contact points 620 may aid in the identification of an object. For example, a crochet hook may be differentiated from a pencil based on the different contact points 620 resulting from differences in how the user grasps each.
Other physical attributes 404 may be stored as well in the augmented reality environment. For example, smells such as determined by machine olfaction devices may be stored. In another implementation chemical composition may be stored. Or characteristics of magnetic fields generated or distorted by the object as determined by magnetic field sensors may be stored.
One or more intended uses 702 may be stored. These are operations for which the object is intended during normal use to be used for. For example, scissors have as intended uses cutting sheets of material including paper, fabric, and so forth.
One or more alternate uses 704 are those operations for which the object may be used, but is not intended to be used for. For example, the scissors may be used to puncture rather than cut.
One or more intended users 706 may be specified. These intended users may be specific individuals or groups of users. For example, the scissors may have as intended users the group of users over age three.
Objects in the augmented reality environment may be used in conjunction with other objects. Objects used in conjunction with 708 are items which may, but need not always, be used together. For example, the scissors may have objects used in conjunction 708 with them of paper, glue, and glitter.
Operating or use cost 710 may be stored in the object datastore 114. This cost may be measured in terms of time, money, resource utilization, and so forth. For example, the operating cost 710 of an electronic game may be 350 watts of electrical power per hour.
A designated operating environment 712 may also be stored. The designated operating environment 712 indicates how and where the object typically is used. The object may be considered more likely to be present in the designated operating environment 712 than in other environments. For example, a laptop object has a designated operating environment 712 which differs from a scrubbing brush object 712 which resides in the kitchen sink.
Safety parameters 714 designate boundaries of a safe operating regime associated with the object. These safety parameters 714 may include safety to users, safety to non-users, safe operation to prevent damage to the object, and so forth. For example, the safety parameters 714 for a particular blender object may indicate that the blender should not operate continuously for more than fifteen minutes to minimize the risk of electrical fire from overheating.
Other 716 functional attributes 406 may also be stored in the object datastore 114. For example, details about consumable components or inputs for the object may be maintained, object behavior, object operation, and so forth. An object category may also be maintained to indicate whether the object is unitary, a package, a group, or a container for other objects.
The ownership attributes 408 may include a date acquired 802. The date acquired 802 may be determined by the sensors in the augmented reality environment, transfer of data from merchants or shippers, and so forth.
One or more ownership categories 804 may be specified. These categories may be used to designate if the object was received as a gift, purchased, leased, rented, and so forth.
Ownership rights 806 indicate what property rights are held by a particular user or group of users. These may include particular rights of alienability, such as whether the objects may be sold, rented, leased, performed publicly, and so forth. For example, the ownership rights 806 may include possession but not ownership if the object is being leased or rented from another.
An ownership rights expiration date 808 may be stored which indicates a point in time where the ownership rights will change. For example, access rights to a digital object such as a movie may be temporary and expire after seven days. Or the date when a rental period for a piece of furniture concludes. In some implementations an ownership rights expiration event attribute may also be stored. For example, ownership rights continue until a rental payment is missed.
A purchase price 810 of the object may be maintained. The purchase price 810 may be input manually or received from a shipper, merchant, or other party providing or interacting with the delivery of the object.
A current value 812 of the object may also be stored in the object datastore 114. The current value 812 may be configured to reflect a current market value of the object, the owner's current valuation, and so forth. For example, the market value of the object may be determined by searching for current used sale prices of the same or a similar object in one or more marketplaces.
Additional ownership attributes 408 may also be stored. For example, a list of authorized transferees may be stored which designate to whom an object may be transferred.
A current location 902 may be stored. For example, the sunglasses at coordinates 3400 mm, −975 mm, 65 mm or on the “kitchen counter” in the augmented reality environment.
A default location 904 may be stored which indicates where the object is typically found or should be stowed. This default location 904 may be pre-loaded, determined via statistical analysis from data in other object datastores 114, defined by a manufacturer, based at least in part on a location history of the object, and so forth. For example, the default location 904 for a medical thermometer is in a medicine cabinet in a master bathroom based on where it is typically stored in that household. In another household, the default location 904 for the medical thermometer may be a medicine cabinet in a bathroom proximate to a child's room.
One or more associated locations 906 may be stored. The associated locations 906 are places where the object is likely to reside. For example, the associated locations 906 for the sunglasses may be in the car, on a table next to the patio, and so forth.
A quantity at a location 908 may be maintained. For example, where the objects are paper towels in a pantry, the quantity of paper towels at that location may be tracked.
A quantity threshold 910 may be designated for a given location, or overall throughout the environment. The quantity threshold 910 indicates what pre-determined quantity has been set to be maintained at that location. For example, a pre-determined minimum of two rolls of paper towels may be set for the pantry.
A location history 912 indicating the location of the object over a period of time may be maintained in the object datastore 114. This location history 912 may have variable levels of resolution in time, space, or both. For example, the augmented reality environment may be configured to maintain the history in ten minute intervals of what room the sunglasses were in, but not necessarily the exact location within the room. The location history 912 data may be used to determine usage patterns of objects.
In some implementations, the gesture may occur at least in part free from contact with other objects. For example, the gesture may comprise the user extending a pointer finger and touching an object, or vice versa. The gesture need not, but may, call for contact. For example, the gesture may include the user waving a hand, pointing at an object, arranging their fingers in a particular configuration, and so forth.
Some gestures may also include audible elements, such as a user raising a hand and generating a finger snap from the raised hand, or rapping on a wall. Input may also include speech accepted by a speech recognition module 1004. Other forms of input may also be accepted from input devices including keyboards, buttons, pointers, touch sensors, and so forth.
A user prompt module 1006 in the user interface module 126 is configured to generate user prompts such as highlights, pointers, menus, icons, and so forth. For example, the user prompt module 1006 may be configured to present a menu of options to the user. The user prompt module 1006 may generate audible, visual, tactile, or other outputs.
An object datastore query module 1008 is configured to accept user inputs and generate a query of the object datastore 114 including local, cloud, or both. For example, the user may make a hand gesture indicating a query. These queries may be used to determine presence or absence of an object in the environment, in another augmented reality environment, request data about an object, and so forth. For example, a user may wish to know a location of his sunglasses in the environment.
Illustrative Processes
The processes described in this disclosure may be implemented by the architectures described herein, or by other architectures. These processes are illustrated as a collection of blocks in a logical flow graph. Some of the blocks represent operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order or in parallel to implement the processes. It is understood that the following processes may be implemented on other architectures as well.
At 1104, one or more attributes are determined about the object via the one or more sensors within the augmented reality environment. For example, the texture 610 of the object may be determined via data obtained by the structured light.
At 1106, a local object datastore 114 coupled to the augmented reality environment is accessed to generate a preliminary identification of the object based at least in part on the one or more attributes. The local object datastore 114 may be stored within the memory 110 in the computing device 104, or accessible via a local area network. The preliminary identification may be made by comparing one or more of the attributes of the detected object with previously stored attributes in the object datastore 114.
At 1108, a cloud object datastore 114 coupled to the augmented reality environment may be accessed to generate a refined identification. This refined identification may be based at least in part on the preliminary identification, the one or more attributes, or both. For example, the preliminary identification may indicate the object is a beverage can, but of an unidentified variety. This preliminary identification may be used to narrow the search in the cloud object datastore 114 to search only beverage cans, reducing the computational requirements for the search.
The cloud object datastore 114 may be stored within the cloud object datastore servers 306. Because of the increased storage and processing capability available within the cloud resource, the cloud object datastore may contain many more attributes for many more items, with those attributes at one or more different resolutions.
As mentioned above, the local and cloud object datastores may be populated with data from several sources. Individual users may identify objects, object attributes may be received from merchants, shippers, manufacturers, and so forth, or the augmented reality environment may be configured to try and determine a preliminary or refined identity. As described below with regards to
At 1202, an object is detected in an augmented reality environment, such as by one or more of the sensors in the ARFN 102. At 1204, one or more attributes are determined about the object via the one or more sensors within the augmented reality environment.
At 1206, an object datastore 114 coupled to the augmented reality environment is accessed to generate a preliminary identification of the object based at least in part on the one or more attributes. As above, the preliminary identification may be made by comparing one or more of the attributes of the detected object with previously stored attributes in the object datastore 114.
At 1208, when a confidence value of the preliminary identification is below a pre-determined identification confidence threshold 510 the user is queried for identification data. The identification confidence threshold may be calculated by using one or more weights associated with object characteristics. For example, when the input is ambiguous as to the object characteristics, the associated weights with that data would indicate a level of ambiguity as to the measurement. The interrogation may comprise an audible prompt, visual prompt, tactile prompt, and so forth. For example, a selection indicator designating the detected but unidentified object may be generated. This selection indicator may comprise a visual indicator proximate to or overlapping the unidentified object. For example, a ring, arrow, overlay, audible prompt and so forth may be used to indicate the unidentified object.
In other implementations, audible prompts may also be used in conjunction with the visible prompts. These audible indicators may be configured to have an apparent source proximate to the associated object. For example, an audible prompt may be perceived by the user to come from (or nearby) the object. Or the audible prompt may comprise speech asking about the object such as “what is the object on the table?” A combination of prompts may be used. For example, the selection indicator may highlight the unidentified object while synthesized speech provides audio output asking the user “what is this?”
At 1210, the identification data is accepted from the user resulting from the interrogation for identification of the object. For example, the user may respond “that is a garlic press”. In some implementations the environment may further query the user to establish or confirm other attributes such as functional attributes 406. For example, the environment may ask “what do you use a garlic press for?” and the user may respond “when cooking”.
The augmented reality environment may be configured to accept the identification data in a variety of ways. In one implementation the identification data may comprise human speech. In another implementation the identification data may comprise a recognized human gesture. For example, the gesture may comprise a letter, word, or phrase in American Sign Language. Or the user may be presented with a list of potential identifications and may be asked to point to the correct identification.
At 1212, the accepted identification of the object is stored in the object datastore 114. This accepted identification as well as object attributes may in turn be provided to other object datastores 114. For example, once that particular garlic press has been identified in an augmented reality user environment, data about that object's attributes may be shared to the cloud object datastore and thus accessible to other users in other augmented reality environments or portions thereof.
At 1302, a query of the object datastore 114 is received in an augmented reality environment. The query may include one or more attributes of an object. For example, the user may query the augmented reality environment to provide an inventory of the items located in a small closet.
At 1304, one or more conditions of the augmented reality environment are applied to the query. These conditions may include limitations based on sensor capabilities, available data, environmental attributes, and so forth. In some implementations, these conditions may be considered to filter the query such that objects which conform to physical realities of the environment are returned by the query, while those which do not are omitted. Continuing the example above, the one or more conditions may include the dimensions of the small closet, object attributes such as size 602, and sensor limitations such as resolution of image data acquired by the camera 210 for an ARFN in view of the small closet.
At 1306, the object datastore 114 is accessed to generate query results based at least in part upon the query and the one or more associated conditions. Continuing the example, the object datastore 114 may be queried to inventory and identify objects detected within the closet. Due to the size limitations of the closet, large pieces of furniture such as sofas, tables, chairs, and so forth may be excluded from the search. The search for identifiable objects may be configured to be limited to objects with an overall volume less than that of the closet and dimensionality such that the object would fit within the closet. By comparing these conditions with the attributes in the object datastore 114 the identification of the objects may be simplified.
In some implementations a tiered or weighted set of conditions may also be applied. For example, the functional attributes 406 of items in the small closet may apply a weighting factor to the query results such that household objects such as linens, towels are searched first and results involving these objects are more likely to be accurate than objects associated with outdoor use such as gardening tools.
At 1308, output based at least in part on the query results may be generated. This generation may comprise compiling, exporting, presenting, and so forth. For example, the inventory of the closet may be projected via the projector 206 onto a wall in the hallway or presented on a handheld computing device.
At 1402, an event of interest associated with an object is received in an augmented reality environment. For example, the transducer 222 and microphone 218 may detect the interacting sound signature 614 of keys on a key ring jingling. This interacting sound signature 614 may be configured as an event of interest.
At 1404, at least partly in response to the event of interest, a scan is initiated within the augmented reality environment. This scan involves the acquisition of data by one or more sensors, configured to survey or view the augmented reality environment. For example, in response to the jingling, the structured light scan may be initiated to track the motion of the keys in the environment.
In some implementations the scan may comprise an increase in the frequency, resolution or both of a scan previously in progress. For example, the structured light scan may increase from a scan interval of five scans per second to sixty scans per second.
During times in the augmented reality environment where little or no activity is taking place, the set of sensors and their associated scan rates, resolution, and so forth may be reduced or discontinued. This may be done to reduce power consumption, minimize computational load, and so forth. For example, in an empty room the ARFN 102 may discontinue use of structured light and rely on audio signals received by the microphones 218 to determine an event of interest, such as a user's footsteps, breathing, opening of a door, and so forth.
At 1406, a change in one or more of the attributes associated with the event of interest is determined. For example, a change in the current location 902 of the keys is determined as a result of the user picking the keys up, causing them to jingle.
At 1408, the object datastore is updated to reflect the changes in the one or more attributes. For example, the current location 902 may now reflect that the keys are now in the garage with the user, and the location history 912 may be updated to show the path of the keys through the house.
As described above with regards to
At 1502, user approval for aggregation of one or more object attributes of an object in an augmented reality environment is accepted. The user approval may comprise an “opt in” to allow anonymous or anonymized data to be collected into the cloud object datastore. In some implementations this user approval may be restricted to users with particular user permissions, such as a designated administrator of the augmented reality environment. In some implementations the collected object attributes relating to particular users, objects, or other criteria may be omitted or prevented from participating in aggregation. For example, a user may choose to not provide identified data about a rare collection of slide rules.
At 1504, data comprising at least a subset of a local object datastore from the augmented reality environment is received. For example, the cloud object datastore servers 306 may receive a set of attributes describing an object which was identified by the user 302 in that user's augmented reality environment, or that user's portion of the augmented reality environment. In some implementations the entire local object datastore may be received.
At 1506, the received data is aggregated into the cloud object datastore. This aggregation may include filtering, checking, verifying, weighing, or otherwise modifying the data. For example, data may be preliminarily aggregated into the cloud object datastore pending confirmation by a trusted user or by cross checking with object attributes of similar identified objects.
At 1508, the cloud object datastore is provided for query. In one implementation, an application programming interface may be maintained allowing applications to access the datastore. As described above, the cloud object datastore may be accessed by a variety of users or entities to address questions relating to the objects and interactions of users at least partly within an augmented reality environment.
Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.
Worley, III, William Spencer, Crump, Edward Dietz
| Patent | Priority | Assignee | Title |
| 10019845, | Aug 30 2012 | WEST TEXAS TECHNOLOGY PARTNERS, LLC | Method and apparatus for content association and history tracking in virtual and augmented reality |
| 10366522, | Sep 27 2017 | Microsoft Technology Licensing, LLC | Augmented and virtual reality bot infrastructure |
| 10382287, | Feb 23 2012 | Persistent node framework | |
| 10438261, | Dec 30 2014 | Ebay Inc.; eBay Inc | Marketplace listing generation using message metadata |
| 10628017, | Jun 28 2013 | Nokia Technologies Oy | Hovering field |
| 10733779, | Sep 27 2017 | Microsoft Technology Licensing, LLC | Augmented and virtual reality bot infrastructure |
| 10895868, | Apr 17 2015 | Tulip Interfaces, Inc. | Augmented interface authoring |
| 10948975, | Jan 29 2016 | Rovi Guides, Inc. | Methods and systems for associating input schemes with physical world objects |
| 10983753, | Jun 09 2017 | International Business Machines Corporation | Cognitive and interactive sensor based smart home solution |
| 10996660, | Apr 17 2015 | TULIP INTERFACES, INC | Augmented manufacturing system |
| 11120627, | Aug 30 2012 | WEST TEXAS TECHNOLOGY PARTNERS, LLC | Content association and history tracking in virtual and augmented realities |
| 11397503, | Jun 28 2019 | SNAP INC | Association of user identifiers to augmented-reality content |
| 11423909, | Aug 12 2016 | MAGIC LEAP, INC | Word flow annotation |
| 11455671, | Dec 30 2014 | Ebay Inc. | Marketplace listing generation using message metadata |
| 11507180, | Jan 29 2016 | Rovi Guides, Inc. | Methods and systems for associating input schemes with physical world objects |
| 11574423, | Jan 29 2021 | BOOMANITY CORP. A DELAWARE CORPORATION; BOOMANITY CORP A DELAWARE CORPORATION | Augmented reality (AR) object communication and interaction system and method |
| 11763530, | Aug 30 2012 | WEST TEXAS TECHNOLOGY PARTNERS, LLC | Content association and history tracking in virtual and augmented realities |
| 11853648, | Jun 09 2017 | International Business Machines Corporation | Cognitive and interactive sensor based smart home solution |
| 11868518, | Jan 29 2016 | Rovi Guides, Inc. | Methods and systems for associating input schemes with physical world objects |
| 9589000, | Aug 30 2012 | WEST TEXAS TECHNOLOGY PARTNERS, LLC | Method and apparatus for content association and history tracking in virtual and augmented reality |
| 9767720, | Jun 25 2012 | Microsoft Technology Licensing, LLC | Object-centric mixed reality space |
| 9978182, | Jun 30 2015 | CITIBANK, N A | Technique for more efficiently displaying text in virtual image generation system |
| Patent | Priority | Assignee | Title |
| 6611196, | Mar 20 1998 | Xerox Corporation | System and method for providing audio augmentation of a physical environment |
| 7298931, | Oct 14 2002 | Samsung Electronics Co., Ltd. | Image retrieval method and apparatus using iterative matching |
| 7418392, | Sep 25 2003 | Sensory, Inc. | System and method for controlling the operation of a device by voice commands |
| 7720683, | Jun 13 2003 | Sensory, Inc | Method and apparatus of specifying and performing speech recognition operations |
| 7774204, | Sep 25 2003 | Sensory, Inc. | System and method for controlling the operation of a device by voice commands |
| 8244069, | Feb 28 2005 | Adobe Inc | Facilitating computer-assisted tagging of object instances in digital images |
| 8369492, | Nov 26 2003 | Microsoft Technology Licensing, LLC | Directory dialer name recognition |
| 20020138481, | |||
| 20050169503, | |||
| 20050207622, | |||
| 20060126905, | |||
| 20060282649, | |||
| 20090245577, | |||
| 20090282025, | |||
| 20100086192, | |||
| 20100104134, | |||
| 20100208038, | |||
| 20110038512, | |||
| 20110077942, | |||
| 20110103691, | |||
| 20110170769, | |||
| 20110243449, | |||
| 20110289455, | |||
| 20120027290, | |||
| 20120183221, | |||
| 20120223885, | |||
| 20120236172, | |||
| 20120257797, | |||
| 20120321130, | |||
| 20130054622, | |||
| 20130060786, | |||
| 20130063550, | |||
| WO2011088053, |
| Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
| Sep 14 2011 | Rawles LLC | (assignment on the face of the patent) | / | |||
| Nov 28 2011 | CRUMP, EDWARD DIETZ | Rawles LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 027452 | /0717 | |
| Nov 30 2011 | WORLEY III, WILLIAM SPENCER | Rawles LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 027452 | /0717 | |
| Nov 06 2015 | Rawles LLC | Amazon Technologies, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 037103 | /0084 |
| Date | Maintenance Fee Events |
| Aug 10 2018 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
| Oct 03 2022 | REM: Maintenance Fee Reminder Mailed. |
| Mar 20 2023 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
| Date | Maintenance Schedule |
| Feb 10 2018 | 4 years fee payment window open |
| Aug 10 2018 | 6 months grace period start (w surcharge) |
| Feb 10 2019 | patent expiry (for year 4) |
| Feb 10 2021 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Feb 10 2022 | 8 years fee payment window open |
| Aug 10 2022 | 6 months grace period start (w surcharge) |
| Feb 10 2023 | patent expiry (for year 8) |
| Feb 10 2025 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Feb 10 2026 | 12 years fee payment window open |
| Aug 10 2026 | 6 months grace period start (w surcharge) |
| Feb 10 2027 | patent expiry (for year 12) |
| Feb 10 2029 | 2 years to revive unintentionally abandoned end. (for year 12) |