method and systems for modeling user possession of a mobile device for a user authentication framework are provided. The method includes analyzing sensor data representing information captured from sensor(s) associated with at least one of a plurality of devices, the plurality including the user's mobile device. The method allows for determining, based on the analyzed sensor data, a probability that the user maintains possession of the user's mobile device; and configuring the probability for use in determining whether to require authentication. The determination may include configuring a probabilistic model that includes a first state indicating that the user possesses the mobile device and a second state indicating that the user does not; classifying motions of the mobile device by types, the motions being determined based on the sensor data; and updating probabilities of the two states in response to determining that at least one of the motions has occurred.
|
1. A method for modeling user possession of a mobile device for a user authentication framework, the method comprising:
analyzing sensor data, the sensor data representing information captured from sensors associated with at least two of a plurality of devices;
determining, based on the analyzed sensor data, a probability that the user is in possession of the mobile device, the mobile device being one of the plurality of the devices; and configuring the probability for use in determining whether to require authentication,
wherein the determination of the probability that the user is in possession of the mobile device is further based on an analysis of a difference between a first action of the user represented by the sensor data associated with a first one of the two devices and a second action of the user represented by the sensor data associated with a second one of the two devices;
allowing the user to access a first set of functionality of the mobile device without requiring authentication based on the determined probability being higher than a first threshold; and
requiring the user to perform authentication in order to access a second set of functionality of the mobile device based on the determined probability being lower than a second threshold.
18. A non-transitory computer-readable storage medium having embodied thereon instructions, which when executed by one or more processors, perform a method modeling user possession of a mobile device for a user authentication framework, the method comprising:
analyzing sensor data, the sensor data representing information captured from one or more sensors associated with at least two of a plurality of devices;
determining, based on the analyzed sensor data, a probability that the user is in possession of the mobile device, the mobile device being one of the plurality of the devices; and configuring the probability for use in determining whether to require authentication,
wherein the determination of the probability that the user is in possession of the mobile device is further based on an analysis of a difference between a first action of the user represented by the sensor data associated with a first one of the two devices and a second action of the user represented by the sensor data associated with a second one of the two devices;
allowing the user to access a first set of functionality of the mobile device without requiring authentication based on the determined probability being higher than a first threshold; and
requiring the user to perform authentication in order to access a second set of functionality of the mobile device based on the determined probability being lower than a second threshold.
12. A system for modeling user possession of a mobile device for a user authentication framework, the system comprising a plurality of devices including the mobile device, the mobile device comprising: a processor; and
a memory communicatively coupled to the processor, the memory storing instructions which when executed by the processor perform a method comprising:
analyzing sensor data, the sensor data representing information captured from sensors associated with at least two of a the plurality of devices;
determining, based on the analyzed sensor data, a probability that the user is in possession of the mobile device; and
configuring the probability for use in determining whether to require authentication,
wherein the determination of the probability that the user is in possession of the mobile device is further based on an analysis of a difference between a first action of the user represented by the sensor data associated with a first one of the two devices and a second action of the user represented by the sensor data associated with a second one of the two devices;
allowing the user to access a first set of functionality of the mobile device without requiring authentication based on the determined probability being higher than a first threshold; and
requiring the user to perform authentication in order to access a second set of functionality of the mobile device based on the determined probability being lower than a second threshold.
8. A method, for modeling user possession of a mobile device for a user authentication framework, the method comprising:
analyzing sensor data, the sensor data representing information captured from one or more sensors associated with at least one of a plurality of devices;
determining, based on the analyzed sensor data, a probability that the user is in possession of the mobile device, the mobile device being one of the plurality of the devices; and configuring the probability for use in determining whether to require authentication, wherein the determining probability that user possesses the mobile device comprises: configuring a first probabilistic model, the first probabilistic model including at least two states, the at least two states including a first state indicating that the user possesses the mobile device and a second state indicating that the user does not possess the mobile device;
classifying motions of the mobile device by types, the motions being determined based on the sensor data received from one or more devices of the plurality of the devices; and
updating probabilities of the at least two states in response to determining that at least one of the motions has occurred, and
wherein probabilities of the at least two states are reset to predetermined initial values using a user authentication mechanism of the mobile device provided by an original equipment manufacturer;
allowing the user to access a first set of functionality of the mobile device without requiring authentication based on the determined probability being higher than a first threshold; and
requiring the user to perform authentication in order to access a second set of functionality of the mobile device based on the determined probability being lower than a second threshold.
2. The method of
3. The method of
4. The method of
5. The method of
configuring a first probabilistic model, the first probabilistic model including at least two states, the at least two states including a first state indicating that the user possesses the mobile device and a second state indicating that the user does not possess the mobile device;
classifying motions of the mobile device by types, the motions being determined based on the sensor data received from one or more devices of the plurality of the devices; and
updating probabilities of the at least two states in response to determining that at least one of the motions has occurred.
6. The method of
7. The method of
determining a location of the mobile device relative to the user, the location of the mobile device being determined based on the sensor data using a second probabilistic model; and
adjusting the probability of the at least two states and probabilities of determining the motions based on the determined location of the mobile device.
9. The method of
10. The method of
11. The method of
based upon the determining, based on the analyzed sensor data, of the probability that the user is in possession of the mobile device, causing a reduction of power consumption of the mobile device.
13. The system of
14. The system of
motion sensors, inertial sensors, visual sensors, location sensors, bio sensors, audio sensors, proximity sensors, and a feed from one or more social networks.
15. The system of
configuring a first probabilistic model, the first probabilistic model including at least two states, the at least two states including a first indicative of the user possessing the mobile device and a second state indicative of the user not possessing the mobile device;
classifying motions of the mobile device by types, the motions being determined based on the sensor data received from one or more devices of the plurality of the devices; and
updating probabilities of the at least two states in response to determining that at least one of the motions has occurred.
16. The system of
17. The system of
determining a location of the mobile device relative to the user, the location of the mobile device being determined based on the sensor data using a second probabilistic model; and
adjusting the probability of the at least of two states and probabilities of determining the motions based on the determined location of the mobile device.
|
The present application claims the benefit of U.S. Provisional Application No. 61/906,840, filed on Nov. 20, 2013. The subject matter of the aforementioned application is incorporated herein by reference for all purposes.
The present application relates generally to authentication, and more specifically to systems and methods regarding authentication of users based on sensor data.
Mobile devices are widely used for various applications. Some mobile applications, for example, those designed for reading and writing email messages, messages for social networks, banking, and payment processing, may require a high level of confidence that the mobile device is being used by its owner. In other applications, for example, in those designed to take pictures or browse public internet pages, there may be no need for such a high level of confidence.
One of the conventional solutions to ensure the mobile device is in possession of its owner is to lock the mobile device if it has not been used for a pre-determined period of time. Then, the owner of the mobile device can be asked to provide login credentials each time he or she wants to regain access to the mobile device. For example, the owner can be asked to enter the user password or asked for a fingerprint to be taken. This conventional approach may be very inconvenient in some situations, for example, when the user needs to take a photo to capture some fleeting moment that would be otherwise lost if time had to be taken first to provide user credentials or have a fingerprint taken. There is also the inconvenience associated with needing to reenter credentials or have a fingerprint taken even though the mobile device has remained in the user's possession for longer than the pre-determined period for locking of the mobile device.
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
The systems and methods for modeling user possession of a mobile device for a user authentication framework are provided. According to an example embodiment, the method can include analyzing sensor data. The sensor data can represent information captured from one or more sensors associated with at least one of a plurality of devices. The method can further include determining, based on the analyzed sensor data, a probability that the user is in possession of a mobile device, the mobile device being one of the plurality of the devices. Various embodiments of the method also include configuring the probability for use in determining whether to require authentication. The authentication may be reauthentication and may be either passive or require action by a user.
In some embodiments, the analyzing of the sensor data includes monitoring the sensor data.
The sensors can include at least one of the following: motion sensors, inertial sensors, visual sensors, location sensors, bio sensors, audio sensors, and proximity sensors. In certain embodiments, devices of the plurality of the devices are communicatively coupled via a network and are operable to exchange the sensor data with each other. In some embodiments, the sensor data can include a representation of at least one feed from at least one social network.
In some embodiments, the determination of a probability that the user is in possession of the mobile device is based on an analysis of coherence of the sensor data received from at least two devices of the plurality of the devices. The determined probability may be further configured for use in granting access to a number and a type of applications.
In some embodiments, determining probability of the user possessing the mobile device includes configuring a first probabilistic model. The first probabilistic model can include at least two states: “the user is in possession of the mobile device” and “the user is not in possession of the mobile device”. The method can further include classifying motions of the mobile device by type. The motions can be determined based on sensor data received from one or more devices of the plurality of the devices. The method can allow updating probabilities of the two states in response to determining that at least one of the motions has occurred.
In some embodiments, the probabilities of the two states are updated based on the type of the motions, probabilities of determining the motions, and a number of motions occurring in a pre-determined time period.
In some embodiments, the method can further include determining a location of the mobile device with respect to the user. The location of the mobile device can be determined based on the sensor data using a second probabilistic model. The method can include adjusting the probability of the two states and probabilities of determining the motions based on the determined location of the mobile device.
In some embodiments, the method includes providing a location of the mobile device with respect to the user and the probabilities of the two states to the mobile device to facilitate determining whether to require user interaction for active authentication.
In some embodiments, the probabilities of the two states are reset to predetermined initial values using a user authentication mechanism provided by an original equipment manufacturer of the mobile device.
According to another example embodiment of the present disclosure, the steps of the method for authentication of a user based on sensor data are stored on a machine-readable medium comprising instructions, which when implemented by one or more processors perform the recited steps.
Other example embodiments of the disclosure and aspects will become apparent from the following description taken in conjunction with the following drawings.
Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like references indicate similar elements and in which:
The present disclosure provides example systems and methods for facilitating authentication of users based on sensor data. As used herein, “sensor data” may refer variously to raw data, processed data, and/or a representation of raw or processed data from one or more sensors. Embodiments of the present disclosure can be practiced, but not limited to, on various mobile devices, for example, a smart phone, a mobile phone, a tablet computer, an all-in-one, a laptop computer, a wearable (e.g., smart watch, smart glasses, etc.), a car communication system, an infotainment system, a teleconference system, and so forth. The mobile devices can be used in stationary and portable environments. Stationary environments can include residential and commercial buildings or structures, and the like. Portable environments can include moving vehicles, moving persons, other transportation means, and the like.
According to an example embodiment, a method for modeling user possession of a mobile device for a user authentication framework is provided. The method includes analyzing sensor data received from sensors. In various embodiments, the sensor data represents information captured from one or more sensors associated with at least one of a plurality of devices. The representation may involve multiple sensors. The analysis involving information captured from multiple sensors may be referred to variously herein as “sensor fusion”.
The method can proceed with determining, based on the analyzed sensor data, a probability that the user is in possession of a mobile device, the mobile device being one of the plurality of the devices. The method can continue with configuring the probability for use in determining whether to require user authentication. In some embodiments, the probability is used to determine a passive authentication technique is warranted wherein the probability that the user is possession of the mobile device is determined to exceed a predetermined threshold based on the analyzed sensor data without requiring the user to provide some predetermined authentication such as credentials, a fingerprint, or other suitable identification. In various embodiments, the probability is used to determining whether to require active authentication that requires the user to (actively) provide the predetermined authentication such as credentials, a fingerprint, or other suitable identification.
Referring now to
The sensors 120, in various embodiments, include motion sensors, inertial sensors, proximity sensors, visual sensors, audio sensors, location sensors, and the like. For example, the sensors 120 can include an accelerometer, a magnetometer, a gyroscope, an Inertial Measurement Unit (IMU), a temperature sensor, an altitude sensor, a proximity sensor, a barometer, a humidity sensor, a color sensor, a light sensor, a pressure sensor, a Global Positioning System (GPS) module, a beacon, a (video) camera, a WiFi sensor, an ultrasound sensor, an infrared sensor, a touch sensor, and an audio sensor. The video camera is configured to capture still or motion images of an environment. The images captured by the video camera may include pictures taken within the visible light spectrum or within a non-visible light spectrum such as the infrared light spectrum (“thermal vision” images). The video camera can generate a video signal of the environment, which may include one or more sound sources (for example, talkers) and optionally one or more noise sources (for example, other talkers and operating machines). In some embodiments, the audio sensor can include one or more microphones configured to capture acoustic sounds. The acoustic sounds can include sounds in the human hearing range and/or out of the human hearing range.
In further embodiments, the sensors 120 include a bio sensor, photoplethys-mogram (PPG), a Galvanic skin response (GSR) sensor, an ion gas analyzer, an electroencephalogram (EEG), or an electrocardiogram (EKG). In some embodiments where the mobile device is a wearable, the sensors 120 include a top microphone and a bottom microphone.
In some embodiments, the sensors 120 include a “social sensor”. The social sensor includes feeds, or representations of feeds, from social networks (such as, for example, Twitter, Instagram, Facebook, and so forth) that the mobile device 110 is operable to receive. The mobile device can be operable to share the social sensor data (e.g., a social media feed or representation thereof) with other mobile devices and/or process the social sensor data in combination with other sensor data and/or audio data. For example, a feed from social sensors can indicate that there is a parade or a demonstration congregating within an area. The method may use this information (e.g., representation of the feed as a “social sensor”) to glean information associated with the location that the user of the mobile device is heading to, or to which the user is planning to head.
The sensors 120 may include the “Internet of things”. The Internet of things can provide for the ability of the mobile device 110 to share information with other mobile devices nearby. The information may include inputs from sensors and microphones belonging to the mobile device 110 and information obtained as a result of processing a combination of sensor data (a “sensor fusion” or a “sensor mesh”). The mobile device 110 may pass the information which was obtained from other mobile device to yet another mobile device. The mobile device 110 may selectively receive information from other mobile devices in order to ascertain context associated with an environment in which the mobile device is being operated and process the combination of local sensor data and information from other mobile devices.
Some of the mobile devices 110 are associated with a user 140. For example, mobile devices 110 associated with the user 140 can include a smart phone, a tablet computer, an all-in-one computer, a smart watch, a smart glass, and so forth. In various embodiments, some of the mobile devices 110 are not associated with the user 140. For example, some of the mobile device 110 can include mobile devices associated with users other than user 140. In some embodiments, the mobile device 110 can include a car communication system, a teleconference system, a radio, an ultrasonic beacon, and the like.
In some embodiments, the mobile devices 110 are operable to exchange data between each other, for example, sensor data. In certain embodiments, the data is exchanged using a wireless or wired network(s) 130 using, for example, an Internet protocol. The mobile devices 110 can be connected to network(s) 130 via Wi-Fi, Bluetooth, NFC, and the like.
In some embodiments, the mobile devices 110 are communicatively connected to a cloud-based computing resource(s) 150 (also referred to as “computing cloud 150”). The cloud-based computing resources 150 can include computing resources (hardware and software) available at a remote location and accessible over a network (for example, the Internet). The cloud-based computing resources 150 can be shared by multiple users and can be dynamically re-allocated based on demand. The cloud-based computing resources 150 may include one or more server farms/clusters including a collection of computer servers which can be co-located with network switches and/or routers. In various embodiments, the mobile devices 110 can be connected to the computing cloud 150 via one or more wired or wireless network(s) 130. The mobile devices 110 can be operable to send data to computing cloud 150, request computational operations to be performed in the computing cloud, and receive back the results of the computational operations.
The processor 220 can include hardware and/or software, which is operable to execute computer programs stored in a memory 230. The processor 220 can use floating point operations, complex operations, and other operations, including steps of the method for authentication of a user based on sensor data. In some embodiments, the processor 220 of the mobile device, for example, can include at least one of a digital signal processor, an image processor, an audio processor, a general-purpose processor, and the like.
The audio processing system 250 may be configured to receive acoustic signals representing sounds captured from acoustic sources via microphones 240 and process the acoustic signals components. In certain embodiments, the microphones 240 are spaced a distance apart such that acoustic waves impinging on the device from certain directions exhibit different energy levels at the two or more microphones. After reception by the microphones 240, the acoustic signals may be converted into electric signals. These electric signals may themselves be converted by an analog-to-digital converter (not shown) into digital signals for processing in accordance with some embodiments. The processed audio signal can be transmitted for processing to the processor 220, stored in memory storage 230, and/or transmitted to a computing cloud for further processing.
In various embodiments, where the microphones 240 are omni-directional microphones that are closely spaced (e.g., 1-2 cm apart), a beamforming technique can be used to simulate a forward-facing and a backward-facing directional microphone response. A level difference can be obtained using the simulated forward-facing and the backward-facing directional microphone. The level difference can be used to discriminate speech and noise in, for example, the time-frequency domain, which can be used in noise and/or echo reduction. In some embodiments, some microphones are used for the most part to detect speech and other microphones are used for the most part to detect noise. In various embodiments, some microphones are used to detect both noise and speech. Noise can include unwanted sounds such as, for example, street noise, ambient noise, speech from entities other than an intended speaker(s), distracting sounds due to operations of machines, cars, and the like.
In some embodiments, noise reduction can be carried out by the audio processing system 250 noise based on inter-microphone level difference, level salience, pitch salience, signal type classification, speaker identification, and so forth. An example audio processing system suitable for performing noise reduction is discussed in more detail in U.S. Pat. No. 8,473,287, issued on Jun. 25, 2013, the disclosure of which is incorporated herein by reference for all purposes.
In some embodiments, the noise reduction includes noise cancellation and/or noise suppression. By way of example and not limitation, noise reduction methods are described in U.S. patent application Ser. No. 12/215,980, filed Jun. 30, 2008, and in U.S. Pat. No. 8,194,880, issued on Jun. 5, 2012, which are incorporated herein by reference in their entireties.
Communication device 260 can include modules operable to communicate data between mobile devices 110 and computing cloud(s) 150 via a network. In various embodiments, the communication devices can include a Bluetooth module, an Infrared module, a Wi-Fi module, an NFC module, beacon module, and the like.
In some embodiments, the sensor data of a mobile device 110 (as shown in example in
In an example embodiment, a combination of sensor data obtained from several mobile devices, for example, smart phones belonging to several people, can be used to detect an occurrence of an event. For example, an occurrence of an earthquake can be detected. The mobile device can read motion sensor data (e.g., from an accelerometer, a gyroscope, an IMU, and the like) and determined that the mobile device is shaking. The mobile device may request and obtain motion data from surrounding mobile devices to determine whether the surrounding mobile devices are shaking as well. By analyzing the sensor data received from the surrounding mobile devices, or receiving an indication of whether other mobile devices are shaking, the mobile device may determine whether the earthquake has occurred.
In another example embodiment, a combination of sensor data obtained from several mobile devices belonging to a group of several people can be used to determine whether one or more members of the group provide data different from the data provided by other members of the group. For example, a mobile device is operable to monitor a body temperature of a person using mobile device that includes a bio sensor, a skin sensor, and the like. The mobile device may detect that one person in a group of persons is sweating while other people in the group are not. The mobile device of that person can obtain an indication of a difference in body temperature and associate the indication of the difference in body temperature with a potential sickness or other health problem.
The sensor data analysis module 310 can be configured to receive sensor data representing information captured from one or more sensors of the mobile device as well as sensor data of other mobile devices. The received sensor data can be analyzed to determine any changes in sensor data. For example, sensor data from or representing information captured from proximity, motion and inertial sensors can be used to determine a trajectory of a motion of the mobile device relative to a user of the mobile device, and/or relative to the surrounding mobile devices, for example, a wearable of the user or another mobile device. The type of trajectory and/or sensor data can be passed to state determination module 320.
In some embodiments, the state determination module 320 is operable to determine, based on sensors data, a probability that the mobile device 110 remains under control of the owner (“on Owner” state) or is not under control of the owner (“off Owner” state). In certain embodiments, the probability may be a percentage. In other embodiments, the probability may be used to determine a level of confidence that the mobile device 110 is in “on Owner” state. In some embodiments, the state determination module 320 and sensor data analysis module 310 are included in a combined module 340.
In various embodiments, the level of confidence or probability is configured (e.g., in combined module 340) for use in determining whether to require authentication. For example, the probability (or level of confidence) can be configured for use by the authentication and access control module 330 to which the probability may be passed.
In various embodiments, the authentication and access control module 330 is operable to receive the configured probability provided by the method of the present technology and use the configured probability in order to make a determination of the level of access to allow, the additional authentication to require for certain access, and the like for the mobile device. In this exemplary embodiment, the method does not provide the functionality of the authentication and access control module 330 but rather provides the probability or level of confidence the authentication and access control module 330 in some configured form. The authentication and access control module 330 may be operable to determine, based on the configured probability or level of confidence (e.g., received from the combined module 340 or from just the state determination module 320), and possibly the sensor data, whether the one in possession of the mobile device is required to pass an active authentication in order to obtain a full or partial access to the mobile device's functionalities and applications. Active authentication may include prompting the user to provide a text password, a voice password, a fingerprint, and the like. In some embodiments of the method of the present technology, the functionality of the authentication and access control module 330 can be included.
In certain embodiments, the authentication of the user can be performed passively without requiring the user to enter a password (e.g., passive authentication.) Passive authentication provides that, based on the level of confidence or probability (exceeding some threshold, for example), the user can obtain full or partial access to the mobile device's functionalities and applications. In situations where the level of confidence is not high enough, the user access may be given limited access to the mobile device, e.g., to only certain of the mobile device's applications and functionalities. For example, the user might be allowed some capability to browse the Internet with the mobile device (e.g., to search), but not allowed to order or pay for services and goods using the mobile device. To regain the full control or full access, the user may be required to provide active authentication.
The example probabilistic model 400 includes a Markov model including states “on Owner” X1 and “off Owner” X2 and four transitions T0, T1, T2, and T3. Probabilities of transitions P(T0), P(T1), P(T2), and P(T3) are determined based on features, for example, type of motions of the mobile device. The type of motions can be determined based on sensor data and audio received by the mobile device and sensor data and audio received by the mobile device from surrounding mobile devices.
In some embodiments, while operating in state X1 (“On Owner”) or in state X2 (“off Owner”) the mobile device is configured to detect a motion, or other sensor data, of the mobile device and classify the motion, or other sensor data. Initially, to switch the mobile device from state X2 to state X1, an active authentication of the user (the owner) can be required. The initial authentication can be performed by asking the user to type a password, provide a fingerprint, a voice command, or other identification requiring a user action. An initial confidence level that the mobile device is in “on Owner” state may be determined based on a type of the initial identification. For example, upon entering the state X1 by providing a fingerprint, the confidence level can be set high, for example at 95-99%.
While in state X1, a passive authentication of the user can be carried out based on types of motions and other sensor data. The confidence level that mobile device is in “On Owner” state can be reduced with time based on a type of a motion of the mobile device or other sensor data. In some embodiments, the confidence level can be scaled based on a number of motions, or other sensor data, occurring within a pre-determined period of time. In certain embodiments, the confidence level is scaled based on how many motions, or other sensor data, have occurred and how many times the motions, or other sensor data, are classified correctly. If no motions or other sensor data have occurred within a predetermined period of time, the confidence is not decreased. In some embodiments, the confidence level is adjusted using information from other sensors, for example, a GPS, a Wi-Fi sensor, a beacon, and the like.
The actions associated with transition T2 when the mobile device has not changed the state X1 include, for example, putting the mobile device in a pocket, getting the mobile device out of pocket to answer a phone call, taking a picture by the mobile device, a GPS reading, and the like. The current confidence level associated with the state X1 (“On Owner”) can be used to restrict or grant to user access to applications and functionalities of the mobile device.
In some embodiments, a probabilistic model similar to model 400 can be used to determine a “carry context” associated with the mobile device. The carry context may be used by model 400 (or incorporated therein). By way of example and not limitation, exemplary systems and methods for determining a “carry context” of the mobile device using a probabilistic model are described in U.S. application Ser. No. 14/321,707, filed on Jul. 1, 2014, which is incorporated herein by reference in their entireties.
Determining the carry context includes understanding how the mobile device is being held or carried by the user; whether the mobile device is in the user's hand, whether the user is looking at the mobile device, whether the mobile device is in a pocket, whether the mobile device is on worn on a wrist as a wearable, and so on. The carry context can be determined by analyzing motions of the mobile device and surrounding mobile devices, data from a proximity sensor, a skin sensor, a temperature sensor and the like. In various embodiments, the mobile device is capable of learning or remembering various actions associated with a user. The mobile device is capable of making probabilistic state determinations based upon these learned or remember actions and comparing them to sensor data.
In some embodiments, the carry context includes determining where the mobile device is located at the moment: on a user, where on the user, or off the user. When the probability that the mobile device is “off owner” is high, the mobile device can be switched from state X1 to state X2.
As an example, a mobile device is in state X1 and it is known that the mobile device is in the user pocket. The mobile device can detect a motion corresponding being taken “out of pocket” and determine whether the mobile device is being held by a hand of the user based on changes in the data from inertial sensors, proximity sensors, and/or temperature sensors. Knowing that the mobile device is located on the owner at a certain place, for example in his/her pocket, increases the confidence of classifying possible motions.
In some embodiments, a probabilistic state of the mobile device is determined based on a mesh of sensor data from two or more mobile devices associated with the user. The mobile device can monitor coherence of data obtained from sensors of several mobile devices. The mobile device can determine that probability of transitioning from state X1 to state X2 is high if sensor data is obtained from the mobile device is not in agreement with sensor data obtained from other mobile devices.
If, for example, a user has a mobile device (e.g., a mobile phone) and also has another mobile device in the form of a wearable (e.g., on the user's wrist, ankle, etc.), the mobile phone can determine, based on mobile devices' sensor data, that the mobile phone remains stationary while the wearable indicates that the user is moving. If this is the case, the mobile phone may switch to “Off Owner” state since the motion of the wearable does not match the motion of the mobile phone. In certain embodiments, a distance between the mobile phone and the wearable can be traced using, for example, inertial sensors. The mobile phone can be switched to the “Off Owner” state when the distance between the mobile phone and the wearable becomes larger than a pre-determined maximum distance. Although this example is described regarding a mobile phone, various embodiments may be practiced for other suitable mobile devices.
In some embodiments, the decision tree 500 is implemented in system 100 of
In some embodiments, cost of memory can be very high at lower levels. Additionally, memory can cause power consumption, especially, for example, leakage power. Hence, the memory allocation can be kept to a minimum in the lower levels. For example, memory such as Static random-access memory (SRAM) at lower levels needs to be programmable since it may not be retained across power cycles. In some embodiments, at higher level storage (persistent memory, for example, Flash or hard disk) may be less expensive. Hence, a “context aware block caching” may be provided whereby higher levels, upon receiving triggers from lower levels, may take the following exemplary steps:
1) evaluate the validity of the trigger and established context, for example, determine whether the user is walking, web surfing, making phone calls, and so on. At higher levels, the context information is determined for a per user or per network level.
2) extract relevant context information and refresh (may swap old information out and push down new information) the block cached information at the lower levels. By way of example and not limitation, the context information may include key words, motion thresholds, acceleration profiles, user trained phonemes, security and authentication certificates, user dependent vocal recognition databases, and so on. For example, once level 2 decides that the current usage context is photography, the level 2 sends down a database which includes commands for camera control. Level 0 devices can swap out old commands and replace them with camera control commands. The recited steps may be replicated between every set of levels in the hierarchy.
In another example, there are two subsystems. The first subsystem is configured to receive and process audio and the second subsystem is configured to read and process data from sensors. The audio subsystem can provide processed audio streams (for example, a speech stream and a background stream) to a server. The server can use the audio speech stream to communicate with the far end mobile device (e.g., mobile phone) and use the background stream to analyze the type of background (noise). For example, the analysis can determine that the noise includes automobile noise with music playing from the stereo. This information can be passed back down to the sensor subsystem and allow the sensor subsystem to resolve disambiguation between motion analyzed by this subsystem. For example, the subsystem may use this information about the noise in order to distinguish whether the motion is caused by a car driving on a rough road with bumps or by a person composing a text message while driving. Similarly, the sensor subsystem can decide to shut down, for some time, one or more sensors (for example, the atmospheric pressure sensor) because these sensors are not useful in this context. In addition, the talker can be identified by the voice extracted from the audio speech stream and this information can be passed to the sensor subsystem to upload a user profile.
In another example, the sensor data is collected from sensors and fused, at some level, into the sensory subsystem. The result of fusion can be passed to the application processor (AP) to be fused with other information (for example, GPS or WiFi). The AP, at this point, can determine the location of the mobile device with respect to the user. For example, a direct WiFi link can be used to recognize that the user is not wearing the mobile phone at the moment he or she speaks because the motion of a wearable watch mobile device that he or she is wearing does not correspond to the motion of the mobile phone being used for the conversation. Based on the determined location of the mobile phone (e.g., further away than the watch from the user), the AP can decide to change the configuration of the voice communication system by loading the Far Far Field (more than 1 m up to 8 m) noise suppression configuration to improve the quality of the communication.
Other examples may include audio and video subsystems (for security enhancement) or two different sensor subsystems or two audio subsystems or the combination of all the above.
The components shown in
Mass data storage 730, which can be implemented with a magnetic disk drive, solid state drive, or an optical disk drive, is a non-volatile storage device for storing data and instructions for use by processor unit 710. Mass data storage 730 stores the system software for implementing embodiments of the present disclosure for purposes of loading that software into main memory 720.
Portable storage device 740 operates in conjunction with a portable non-volatile storage medium, such as a flash drive, floppy disk, compact disk, digital video disc, or Universal Serial Bus (USB) storage device, to input and output data and code to and from the computer system 700 of
User input devices 760 can provide a portion of a user interface. User input devices 760 may include one or more microphones, an alphanumeric keypad, such as a keyboard, for inputting alphanumeric and other information, or a pointing device, such as a mouse, a trackball, stylus, or cursor direction keys. User input devices 760 can also include a touchscreen. Additionally, the computer system 700 as shown in
Graphics display system 770 include a liquid crystal display (LCD) or other suitable display device. Graphics display system 770 is configurable to receive textual and graphical information and processes the information for output to the display device.
Peripheral devices 780 may include any type of computer support device to add additional functionality to the computer system.
The components provided in the computer system 700 of
The processing for various embodiments may be implemented in software that is cloud-based. In some embodiments, the computer system 700 is implemented as a cloud-based computing environment, such as a virtual machine operating within a computing cloud. In other embodiments, the computer system 700 may itself include a cloud-based computing environment, where the functionalities of the computer system 700 are executed in a distributed fashion. Thus, the computer system 700, when configured as a computing cloud, may include pluralities of computing devices in various forms, as will be described in greater detail below.
In general, a cloud-based computing environment is a resource that typically combines the computational power of a large grouping of processors (such as within web servers) and/or that combines the storage capacity of a large grouping of computer memories or storage devices. Systems that provide cloud-based resources may be utilized exclusively by their owners or such systems may be accessible to outside users who deploy applications within the computing infrastructure to obtain the benefit of large computational or storage resources.
The cloud may be formed, for example, by a network of web servers that comprise a plurality of computing devices, such as the computer system 700, with each server (or at least a plurality thereof) providing processor and/or storage resources. These servers may manage workloads provided by multiple users (e.g., cloud resource customers or other users). Typically, each user places workload demands upon the cloud that vary in real-time, sometimes dramatically. The nature and extent of these variations typically depends on the type of business associated with the user.
The present technology is described above with reference to example embodiments. Therefore, other variations upon the example embodiments are intended to be covered by the present disclosure.
Steele, James, Murgia, Carlo, Caldwell, Frederic, Vitus, Deborah
Patent | Priority | Assignee | Title |
10197592, | Feb 05 2016 | LOGITECH EUROPE S A | Method and system for calibrating a pedometer |
10311219, | Jun 07 2016 | VOCALZOOM SYSTEMS LTD | Device, system, and method of user authentication utilizing an optical microphone |
10429454, | Feb 05 2016 | Logitech Europe S.A. | Method and system for calibrating a pedometer |
10490051, | Feb 05 2016 | Logitech Europe S.A. | Method and system for detecting fatigue in an athlete |
10527452, | Feb 05 2016 | Logitech Europe S.A. | Method and system for updating a calibration table for a wearable device with speed and stride data |
10630681, | Apr 07 2014 | Jumio Corporation | Bio leash for user authentication |
10766458, | Jul 13 2018 | NXP B.V. | Sensor fusion for passive keyless entry systems |
10896673, | Sep 21 2017 | WELLS FARGO BANK, N A | Authentication of impaired voices |
11195354, | Apr 27 2018 | Honeywell International Inc | Gesture access control system including a mobile device disposed in a containment carried by a user |
11226672, | Oct 03 2014 | GOOGLE LLC | Intelligent device wakeup |
11282351, | Oct 24 2012 | Apple Inc. | Devices and methods for locating accessories of an electronic device |
11381903, | Feb 14 2014 | Sonic Blocks Inc. | Modular quick-connect A/V system and methods thereof |
11449124, | Oct 03 2014 | GOOGLE LLC | Intelligent device wakeup |
11601806, | Sep 28 2016 | Sony Corporation | Device, computer program and method |
11606669, | Sep 28 2018 | Apple Inc. | System and method for locating wireless accessories |
11641563, | Sep 28 2018 | Apple Inc | System and method for locating wireless accessories |
11687164, | Apr 27 2018 | Honeywell International Inc | Modeling of preprogrammed scenario data of a gesture-based, access control system |
11736938, | Aug 28 2020 | Apple Inc | Maintenance of wireless devices |
11809632, | Apr 27 2018 | Honeywell International Inc | Gesture access control system and method of predicting mobile device location relative to user |
11863671, | Apr 17 2019 | Apple Inc | Accessory assisted account recovery |
11889302, | Aug 28 2020 | Apple Inc | Maintenance of wireless devices |
11935524, | Sep 21 2017 | Wells Fargo Bank, N.A. | Authentication of impaired voices |
12073705, | May 07 2021 | Apple Inc. | Separation alerts for notification while traveling |
12075313, | Sep 28 2018 | Apple Inc. | System and method for locating wireless accessories |
12093353, | Sep 04 2020 | Shopify Inc | Systems and methods for user authentication |
12106641, | Oct 24 2012 | Apple Inc. | Devices and methods for locating accessories of an electronic device |
12143895, | Jun 04 2021 | Apple Inc. | Pairing groups of accessories |
12170892, | Aug 28 2020 | Apple Inc. | Maintenance of wireless devices by electronic devices |
ER3470, |
Patent | Priority | Assignee | Title |
5809471, | Mar 07 1996 | MEDIATEK INC | Retrieval of additional information not found in interactive TV or telephony signal by application using dynamically extracted vocabulary |
6243476, | Jun 18 1997 | Massachusetts Institute of Technology | Method and apparatus for producing binaural audio for a moving listener |
6593956, | May 15 1998 | Polycom, Inc | Locating an audio source |
7131136, | Jul 10 2002 | TELESIS GROUP, INC , THE; E-WATCH, INC | Comprehensive multi-media surveillance and response system for aircraft, operations centers, airports and other commercial transports, centers and terminals |
8194880, | Jan 30 2006 | SAMSUNG ELECTRONICS CO , LTD | System and method for utilizing omni-directional microphones for speech enhancement |
8326625, | Nov 10 2009 | Malikie Innovations Limited | System and method for low overhead time domain voice authentication |
8473287, | Apr 19 2010 | SAMSUNG ELECTRONICS CO , LTD | Method for jointly optimizing noise reduction and voice quality in a mono or multi-microphone system |
8577677, | Jul 21 2008 | Samsung Electronics Co., Ltd.; Korea University Research and Business Foundation | Sound source separation method and system using beamforming technique |
8712069, | Apr 19 2010 | Knowles Electronics, LLC | Selection of system parameters based on non-acoustic sensor information |
8787587, | Apr 19 2010 | SAMSUNG ELECTRONICS CO , LTD | Selection of system parameters based on non-acoustic sensor information |
8880396, | Apr 28 2010 | SAMSUNG ELECTRONICS CO , LTD | Spectrum reconstruction for automatic speech recognition |
9185487, | Jun 30 2008 | Knowles Electronics, LLC | System and method for providing noise suppression utilizing null processing noise subtraction |
9195994, | Apr 25 2012 | Wells Fargo Bank, N.A. | System and method for a mobile wallet |
20030016835, | |||
20030236604, | |||
20040044516, | |||
20040052391, | |||
20050008169, | |||
20060217977, | |||
20060247927, | |||
20070096979, | |||
20080019548, | |||
20080173717, | |||
20090055170, | |||
20090143972, | |||
20090158423, | |||
20090323982, | |||
20100033424, | |||
20100128881, | |||
20100128894, | |||
20100174506, | |||
20100312547, | |||
20100315905, | |||
20100318257, | |||
20110172918, | |||
20110239026, | |||
20110257967, | |||
20120058803, | |||
20120071151, | |||
20120252411, | |||
20130045759, | |||
20130106894, | |||
20130253880, | |||
20130268792, | |||
20130297926, | |||
20130332156, | |||
20140068692, | |||
20140137191, | |||
20140226837, | |||
20140244273, | |||
20140316783, | |||
20140342758, | |||
20140351560, | |||
20150012248, | |||
20150081296, | |||
20150247729, | |||
20160061934, | |||
20160134609, | |||
20160164577, | |||
WO2013148588, | |||
WO2014039552, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
May 20 2013 | CALDWELL, FREDERIC | AUDIENCE, INC | EMPLOYMENT, CONFIDENTIAL INFORMATION AND INVENTION ASSIGNMENT AGREEMENT | 035785 | /0962 | |
Nov 19 2014 | Knowles Electronics, LLC | (assignment on the face of the patent) | / | |||
Dec 22 2014 | MURGIA, CARLO | AUDIENCE, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 034851 | /0495 | |
Jan 29 2015 | VITUS, DEBORAH KATHLEEN | AUDIENCE, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 035725 | /0932 | |
Jan 29 2015 | STEELE, JAMES V | AUDIENCE, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 035725 | /0932 | |
Dec 17 2015 | AUDIENCE, INC | AUDIENCE LLC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 037927 | /0424 | |
Dec 21 2015 | AUDIENCE LLC | Knowles Electronics, LLC | MERGER SEE DOCUMENT FOR DETAILS | 037927 | /0435 | |
Dec 19 2023 | Knowles Electronics, LLC | SAMSUNG ELECTRONICS CO , LTD | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 066216 | /0464 |
Date | Maintenance Fee Events |
Aug 29 2017 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Mar 23 2021 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Oct 03 2020 | 4 years fee payment window open |
Apr 03 2021 | 6 months grace period start (w surcharge) |
Oct 03 2021 | patent expiry (for year 4) |
Oct 03 2023 | 2 years to revive unintentionally abandoned end. (for year 4) |
Oct 03 2024 | 8 years fee payment window open |
Apr 03 2025 | 6 months grace period start (w surcharge) |
Oct 03 2025 | patent expiry (for year 8) |
Oct 03 2027 | 2 years to revive unintentionally abandoned end. (for year 8) |
Oct 03 2028 | 12 years fee payment window open |
Apr 03 2029 | 6 months grace period start (w surcharge) |
Oct 03 2029 | patent expiry (for year 12) |
Oct 03 2031 | 2 years to revive unintentionally abandoned end. (for year 12) |