A system is disclosed for recognizing typing from typing transducers that provide the typist with only limited tactile feedback of key position. The system includes a typing decoder sensitive to the geometric pattern of a keystroke sequence as well as the distance between individual finger touches and nearby keys. The typing decoder hypothesizes plausible key sequences and compares their geometric pattern to the geometric pattern of corresponding finger touches. It may also hypothesize home row key locations for touches caused by hands resting on or near home row. The resulting pattern match metrics may be combined with character sequence transition probabilities from a spelling model. The typing decoder then chooses the hypothesis sequence with the best cumulative match metric and sends it as key codes or commands to a host computing device.
|
0. 22. A method for recognizing typing, the method comprising:
receiving a touch location and time sequence for a plurality of keystrokes;
generating a set of key hypothesis sequences for the plurality of keystrokes;
computing a geometry match metric for each key hypothesis sequence; and
choosing a best hypothesized key sequence based on the geometry match metrics.
0. 19. A typing recognition apparatus comprising:
a typing surface;
at least one touch sensor configured to provide surface coordinates of each touch by a typist to the typing surface;
a hypothesis tree generator configured to generate key hypothesis sequences from the surface coordinates of each touch; and
a pattern geometry evaluator configured to compute a geometry match metric for each of the key hypothesis sequences.
0. 35. A method for recognizing typing, the method comprising:
receiving a touch location and time sequence for a plurality of keystrokes;
generating a set of key hypothesis sequences for the plurality of keystrokes;
computing a geometry match metric for each key hypothesis sequence;
computing a character transition cost for each key hypothesis sequence based on whether the key hypothesis sequence is building a dictionary word; and
selecting a best hypothesized key sequence from the hypothesized key sequences, the best hypothesized key sequence having a best cumulative match metric formulated from the geometry match metric and the character transition cost.
0. 31. A typing recognition apparatus comprising:
a typing surface;
at least one touch sensor integrated with the typing surface and configured to provide surface coordinates of each touch on the typing surface;
a hypothesis tree generator configured to generate key hypothesis sequences from the surface coordinates of each touch;
a pattern geometry evaluator configured to compute a geometry match metric for each of the key hypothesis sequences;
a dictionary selector configured to compute a character transition cost for each of the key hypothesis sequences based on whether the hypothesized key sequence is building a dictionary word; and
a decoder configured for selecting a best hypothesized key sequence from the hypothesized key sequences, the best hypothesized key sequence having a best cumulative match metric formulated from the geometry match metric and the character transition cost.
0. 23. A typing recognition apparatus that compensates for finger and hand drift during typing on a touch-sensitive surface, the apparatus comprising:
sensor scanning hardware configured for providing surface coordinates of each touch received on the touch-sensitive surface; and
a processor programmed for
extending existing key hypothesis sequences with hypotheses for keys in a neighborhood of each new touch,
computing geometry match metrics for the hypothesized key sequences by comparing touch separation vectors between successive touch locations with key separation vectors between successively hypothesized key locations and measuring zero-order key/touch alignment error,
computing a character transition cost for each of the hypothesized key sequences based on whether the hypothesized key sequence is building a dictionary word,
selecting a best hypothesized key sequence from the hypothesized key sequences, the best hypothesized key sequence having a best cumulative match metric formulated from the geometry match metric and the character transition cost, and
communicating symbols and commands represented by the best hypothesized key sequence to a host computer application.
1. A typing recognition apparatus for touch typing on surfaces with limited tactile feedback that compensates for finger and hand drift during typing and discourages any integrated spelling model from choosing dictionary words over unusual but carefully typed strings, the apparatus comprising:
a typing surface means that displays symbols indicating the locations of touchable keys;
touch sensor means that provides the surface coordinates of each touch by a typist attempting to strike said key symbols on said surface;
hypothesis tree generator means that extends existing key hypothesis sequences with hypotheses for keys in the neighborhood of each new touch;
pattern geometry evaluation means that computes geometry match metrics for the hypothesized key sequences by comparing separation vectors between the successive touch locations with separation vectors between the successively hypothesized key locations as well as by measuring the zero-order key/touch alignment error;
decoding means that finds the hypothesized key sequence with the best cumulative match metric; and,
transmission means for communicating the symbols and commands represented by the best hypothesized key sequence to host computer applications.
0. 27. A method for compensating for finger and hand drift during typing on a touch-sensitive surface, comprising:
obtaining a touch location and time sequence for each detected touch in a touch sequence;
computing a set of touch separation vectors of increasing orders between the detected touches in the touch sequence;
generating a set of key hypothesis sequences for each touch in the touch sequence, each key hypothesis sequence associated with a key near the location of the touch;
for each key hypothesis sequence, computing a set of key separation vectors of increasing orders between the keys in the hypothesized key sequence;
for each key hypothesis sequence, computing a geometry match metric as a function of a magnitude of a zero-order touch/key alignment error and the magnitudes of each order's touch and key separation vector difference;
computing a character transition cost for each of the hypothesized key sequences based on whether the hypothesized key sequence is building a dictionary word;
selecting a best hypothesized key sequence from the hypothesized key sequences, the best hypothesized key sequence having a best cumulative match metric formulated from the geometry match metric and the character transition cost, and
transmitting symbols and commands represented by the best hypothesized key sequence to a host computer for further action.
13. A method for recognizing typing from typing devices that sense lateral finger position but provide limited tactile feedback of key location, the method advantageously compensating for finger and hand drift during typing and discouraging any integrated spelling model from choosing dictionary words over unusual but carefully typed strings, wherein the method comprises the following steps:
forming a touch location and time sequence from the fingertip position at the end of each keystroke as measured by typing sensors;
generating a set of key hypothesis sequences for the given touch sequence, each hypothesis in a sequence being for a key near the location of the touch causing the hypothesis;
for each key hypothesis, computing a key/touch alignment error vector as the difference between the location of the hypothesized key and the location of its causing touch;
for each key hypothesis, computing a geometry match metric as a function of the magnitude of the hypothesis' key/touch alignment error as well as of the magnitude of differences between the hypothesis' key/touch alignment error vector and that of preceding hypotheses in its sequence;
combining the geometry match metrics from each hypothesis in a key hypothesis sequence into a cumulative match metric for the hypothesis sequence;
choosing the hypothesized key sequence with the best cumulative metric as the best hypothesized key sequence; and,
transmitting the symbols and commands represented by the best hypothesized key sequence to a host computer for further action.
7. A method for recognizing typing from typing devices that sense lateral finger position but provide limited tactile feedback of key location, the method advantageously compensating for finger and hand drift during typing and discouraging any integrated spelling model from choosing dictionary words over unusual but carefully typed strings, wherein the method comprises the following steps:
forming a touch location and time sequence from the fingertip position at the end of each keystroke as measured by typing sensors;
computing a set of touch separation vectors of increasing orders from the location difference between the newest touch and previous touch in said touch location sequence;
generating a set of key hypothesis sequences for the given touch sequence, each hypothesis in a sequence being for a key near the location of the touch causing the hypothesis;
for each key hypothesis, computing a set of key separation vectors of increasing orders from differences between the position of the newest key and previous keys in the hypothesized sequence;
for each key hypothesis, computing a geometry match metric as a function of the magnitude of the zero-order touch/key alignment error as well as of the magnitudes of each order's touch and key separation vector difference;
combining the geometry match metrics from each hypothesis in a key hypothesis sequence into a cumulative match metric for the hypothesis sequence;
choosing the hypothesized key sequence with the best cumulative metric as the best hypothesized key sequence; and,
transmitting the symbols and commands represented by the best hypothesized key sequence to a host computer for further action.
2. The apparatus of
4. The apparatus of
5. The apparatus of
6. The apparatus of
8. The method of
9. The method of
10. The method of
11. The method of
12. The method of
14. The method of
15. The method of
16. The method of
17. The method of
18. The method of
0. 20. The typing recognition apparatus of
a decoder configured to select as a best hypothesized key sequence from among the key hypothesis sequences based on the computed geometry match metrics.
0. 21. The typing recognition apparatus of
a transmitter configured to send at least one symbol or command represented by the best hypothesized key sequence.
0. 24. The typing recognition apparatus of
0. 25. The typing recognition apparatus of
0. 26. The typing recognition apparatus of
0. 28. The method of
0. 29. The method of
0. 30. The method of
0. 32. The typing recognition apparatus of
0. 33. The typing recognition apparatus of
0. 34. The typing recognition apparatus of
0. 36. The method of
0. 37. The method of
0. 38. The method of
transition cost is set to neutral or zero when a hypothesized key location is a command or editing key.
|
Ser. No. 09/236,513 Jan. 1, 1999 U.S. Pat. No. 5,463,388 Jan. 29, 1993 U.S. Pat. No. 5,812,698 Jul. 14, 1997 U.S. Pat. No. 5,818,437 Jul. 26, 1995 U.S. Pat. No. 6,137,908 Jun. 29, 1994 U.S. Pat. No. 6,107,997 Jun. 27, 1996.
1. Field of the Invention
The present invention pertains to typing recognition systems and methods, and more particularly to recognition of typing in air or on a relatively smooth surface that provides less tactile feedback than conventional mechanical keyboards.
2. The Related Art
Typists generally employ various combinations of two typing techniques: hunt and peck and touch typing. When hunting and pecking, the typist visually searches for the key center and strikes the key with the index or middle finger. When touch typing, the fingers initially rest on home row keys, each finger is responsible for striking a certain column of keys and the typist is discouraged from looking down at the keys. The contours and depression of mechanical keys provide strong tactile feedback that helps typists keep their fingers aligned with the key layout. The finger motions of touch typists are ballistic rather than guided by a slow visual search, making touch typing faster than hunt and peck. However, even skilled touch typists occasionally fall back on hunt and peck to find rarely-used punctuation or command keys at the periphery of the key layout.
Many touchscreen devices display pop-up or soft keyboards meant to be activated by lightly tapping a displayed button or key symbol with a finger or stylus. Touch typing is considered impractical on such devices for several reasons: a shrunken key layout may have a key spacing too small for each finger to be aligned with its own key column, the smooth screen surface provides no tactile feedback of finger/key alignment as keys are struck, and most touchscreens cannot accurately report finger positions when touched by more than one finger at a time. Such temporal touch overlap often occurs when typing a quick burst of keys with both hands, holding the finger on modifier keys while striking normal keys, or attempting to rest the hands. Thus users of touchscreen key layouts have had to fall back on a slow, visual search for one key at a time.
Since touchscreen and touch keyboard users are expected to visually aim for the center of each key, typing recognition software for touch surfaces can use one of two simple, nearly equivalent methods to decide which key is being touched. Like the present invention, these methods apply to devices that report touch coordinates interpolated over a fine grid of sensors rather than devices that place a single large sensor under the center of each key. In the first method, described in U.S. patent application Ser. No. 09/236,513 by Westerman and Elias, the system computes for each key the distance from key center to the sensed touch location. The software then selects the key nearest the finger touch. In the second method, described in U.S. Pat. No. 5,463,388 to Boie et al., the software establishes a rectangle or bounding box around each key and decides which, if any, bounding box the reported touch coordinates lie within. The former method requires less computation, and the latter method allows simpler control over individual key shape and guard bands between keys, but both methods essentially report the key nearest to the finger touch, independent of past touches. Hence we refer to them as ‘nearest key’ recognizers.
Unlike touchscreens, the multi-touch surface (MTS) described by Westerman and Elias in Ser. No. 09/236,513 can handle resting hands and temporal finger overlap during quick typing bursts. Since the MTS sensing technology is fully scalable, an MTS can easily be built large enough for a full-size QWERTY key layout. The only remaining barrier to fast touch typing on an MTS is the lack of tactile feedback. While it is possible to add either textures or compressibility to an MTS to enhance tactile feedback, there are two good reasons to keep the surface firm and smooth. First, any textures added to the surface to indicate key centers can potentially interfere with smooth sliding across the surface during multi-finger pointing and dragging operations. Second, the MTS proximity sensors actually allow zero-force typing by sensing the presence of a fingertip on the surface whether or not the finger applies noticeable downward pressure to the surface. Zero-force typing reduces the strain on finger muscles and tendons as each key is touched.
Without rich tactile feedback, the hands and individual fingers of an MTS touch typist tend to drift out of perfect alignment with the keys. Typists can limit the hand drift by anchoring their palms in home position on the surface, but many keystrokes will still be slightly off center due to drift and reach errors by individual fingers. Such hand drift and erroneous finger placements wreak havoc with the simple ‘nearest key’ recognizers disclosed in the related touchscreen and touch keyboard art. For example, if the hand alignment with respect to the key layout drifts by half a key-spacing (˜9 mm or ⅜″), all keystrokes may land half-way between adjacent keys. A ‘nearest key’ recognizer is left to choose one of the two adjacent keys essentially at random, recognizing only 50% of the keystrokes correctly. A spelling model integrated into the recognizer can help assuming the typist intended to enter a dictionary word, but then actually hinders entry of other strings. Thus there exists a need in the touchscreen and touch keyboard art for typing recognition methods that are less sensitive to the hand drift and finger placement errors that occur without strong tactile feedback from key centers.
For many years, speech, handwriting, and optical character recognition systems have employed spelling or language models to help guess users' intended words when speech, handwriting, or other input is ambiguous. For example, in U.S. Pat. No. 5,812,698 Platt et al. teach a handwriting recognizer that analyzes pen strokes to create a list of probable character strings and then invokes a Markov language model and spelling dictionary to pick the most common English word from that list of potential strings. However, such systems have a major weakness. They assume all user input will be a word contained in their spelling or language model, actually impeding entry of words not anticipated by the model. Even if the user intentionally and unambiguously enters a random character string or foreign word not found in the system vocabulary, the system tries to interpret that input as one of its vocabulary words. The typical solution is to provide the user an alternative (often comparatively clumsy) process with which to enter or select strings outside the system vocabulary. For example, U.S. Pat. No. 5,818,437 to Grover et al. teaches use of a dictionary and vocabulary models to disambiguate text entered on a ‘reduced’ keyboard such as a telephone keypad that assigns multiple characters to each physical key. In cases that the most common dictionary word matching an input key sequence is not the desired word, users must select from a list of alternate strings. Likewise, users of speech recognition system typically fall back on a keyboard to enter words missing from the system's vocabulary.
Unfortunately, heavy reliance on spelling models and alternative entry processes is simply impractical for a general-purpose typing recognizer. Typing, after all, is the fallback entry process for many handwriting and speech recognition systems, and the only fallback conceivable for typing is a slower, clumsier typing mode. Likewise, personal computer users have to type into a wide variety of applications requiring strange character strings like passwords, filenames, abbreviated commands, and programming variable names. To avoid annoying the user with frequent corrections or dictionary additions, spelling model influence must be weak enough that strings missing from it will always be accepted when typed at moderate speed with reasonable care. Thus a general-purpose typing recognizer should only rely on spelling models as a last resort, when all possible measurements of the actual typing are ambiguous.
Since a typing recognizer cannot depend too much on spelling models, there still exists a need in the touchscreen and touch keyboard art for spelling-independent methods to improve recognition accuracy. The main aspect of the present invention is to search for the geometric pattern of keys that best matches the geometric pattern of a touch sequence, rather than just searching for the key closest to each touch. This method improves recognition accuracy without any assumptions about the character content being typed.
According to this aspect of the invention, touch or finger stroke coordinates reported by a sensing device and key coordinates from a key layout feed into a typing recognizer module. The typing recognizer then hypothesizes plausible sequences of keys by extending existing sequences with keys that are within the immediate neighborhood of the newest finger touch. It can also hypothesize home row key locations for touches caused by hands resting on or near the home row keys. For each hypothesized sequence, the typing recognizer computes separation vectors between the layout position of successive keys in the sequence. The typing recognizer also computes separation vectors between successive touch positions in the touch sequence. Each key sequence is evaluated according to a pattern match metric that includes not only the distance between each finger touch and the corresponding key but also how closely the separation vectors between successive touches match the separation vectors between successive keys. The hypothesized sequence with the best cumulative match metric is transmitted to the host computer, possibly replacing an older, higher cost partial sequence that was transmitted previously.
It is therefore an objective of this invention to provide typing recognition methods that overcome the shortcomings of the related touchscreen and touch keyboard art.
A primary object of the present invention is to recognize typing accurately even when lack of tactile key position feedback leads to significant hand and finger drift.
Yet another objective of this invention is to improve typing recognition accuracy without excessive dependence on spelling models.
A further objective of this invention is to disambiguate typing as much as possible with measurements of its geometric pattern before falling back on a spelling model to resolve any remaining recognition ambiguities.
A secondary objective of this invention is to beneficially incorporate key/hand alignment measurements from resting hands into recognition decisions without explicitly shifting the key layout into alignment with the resting hands.
In the preferred embodiment, the typing recognition methods of this invention are utilized within a multi-touch system like that shown in FIG. 1. The sensor scanning hardware 6 detects touches by fingers 2 on the surface 4. The proximity image formation 8 and contact tracking 10 modules determine the touch timing and surface coordinates and report these to the typing recognizer 12. The typing recognizer decides which keys the user intended to press and tells the host communications interface 16 to send those keys to the host computer 18. The system may also include a chord motion recognizer module 14 that interprets lateral sliding of multiple fingers as pointing or gesture input and effectively disables the typing recognizer for such touches. The synchronization detector 13 searches for simultaneous presses or releases of multiple fingers, thereby aiding in detection of chord slides, chord taps, and resting hands. All modules besides the typing recognizer are fully described in related U.S. patent application Ser. No. 09/236,513 by Westerman and Elias. That application is incorporated herein by reference in its entirety. The present invention constitutes improvements to the rudimentary ‘nearest key’ typing recognizer described in that application.
Those skilled in the art will recognize that the typing recognizer disclosed herein could be utilized with any sensing device that accurately reports the lateral position of fingertips as they near the end of their stroke, whether or not the fingers actually touch a surface of depress physical keys. Examples of such alternative finger position sensing systems include micro radar, data gloves, and pressure-sensitive surface materials. The term touch location will be used hereafter for the lateral position or x and y coordinates detected for fingertips within a plane roughly normal to the fingertips at the end of their stroke, even for sensing devices that require no physical contact with a surface at the end of the stroke. Likewise, the typing recognition software need not reside within a microprocessor packaged with the sensing device. It could just as easily execute within the host computer system, or the host computer system and sensing device might be combined such that the same microprocessor executes finger tracking, typing recognition, and user application software.
Related art ‘nearest key’ typing recognizers typically assume that touch location errors are independent from keystroke to keystroke. But for typing devices that don't provide strong tactile feedback of key position, the hand sometimes drifts slightly out of alignment with the key layout. This causes the absolute location errors for most touches to be biased in the drift direction and statistically dependent. However, if the typist still reaches the proper amount (a whole number of key spacings) relative to recent touches, the lateral separations between finger touches will closely match the separations between the keys the typist intended to strike, regardless of the overall hand drift.
A related type of bias occurs when individual fingers drift relative to the rest of the hand. This causes the absolute location errors to be biased the same way for all keys typed by the drifting finger(s). However, keys typed by adjacent fingers may not share this bias.
An important discovery of the present invention is that when trying to recognize a sequence of touches located ambiguously between keys, searching for key sequences whose relative geometric pattern matches the touch pattern greatly narrows the list of plausible key sequences. This is illustrated intuitively in FIG. 2.
Since typists expect the symbol of each touched key to appear on the host computer screen immediately after each corresponding finger stroke, a typing recognizer cannot wait for an entire touch sequence to complete before choosing the best key sequence. In a preferred embodiment of this invention, the recognizer module decodes the touch sequence incrementally, extending key hypothesis sequences by one key each time a new touch is detected. This process will form a hypothesis tree whose nodes are individual key hypotheses. It is important to note that related art ‘nearest key’ recognizes need not construct a hypothesis tree since they assume that finger placement errors from each keystroke are statistically independent.
If a key is within the radius Ract of the new touch, step 114 creates a new hypothesis hj[n] (using data structure 85) descended from the current parent hp[n−1]. The new hypothesis' parent hypothesis reference 86 is set accordingly. Block 116 evaluates how well the new key hypothesis hj[n] and its parent sequence matches the touch sequence T[0] . . . T[n].
Once hypotheses descended from parent hp[n−1] have been generated for all keys near the touch T[n], decision diamond 120 decides whether the previous touch T[n−1]'s stack 76 contains additional parent hypotheses that need to be extended. If so, the parent hypothesis index p is incremented in step 122, and steps 106-122 repeat for the next parent. Once all parent hypotheses have been extended, block 124 actually outputs the best hypothesis sequence as described further in FIG. 8. Step 126 prunes from the tree those hypotheses whose cumulative match metric is already so poor that they are very unlikely to spawn best hypotheses in the future. This prevents exponential growth of the hypothesis tree by discarding clearly had hypotheses but preserving competitive hypotheses that might become parents of the best hypothesis for a future touch. The most efficient pruning method is to start at the bottom of T[n]'s stack 76 and discard all hypotheses whose cumulative metric is not within a future cost margin of the top (best) hypothesis's cumulative match metric. When all of a parent's child hypotheses have been discarded the parent is discarded as well. The pruning step 126 completes all processing of touch T[n], leaving step 128 to increment the touch index n so decision diamond 102 can resume waiting for the next touch.
Working together, steps 118, 124, and 126 constitute a stack decoder. They sort all of the new hypotheses for the current touch T[n] according to their cumulative match metric, choose the lowest cost sequence that winds up at the top of the stack as the best hypothesis sequence to output, and prune the implausible sequences at the bottom of the stack whose costs are much greater than the current best sequence. The stack decoder is a well-known method in the speech recognition, handwriting recognition, and digital communications arts for finding the optimal path through a hypothesis tree. For example, see F. Jelinek, Statistical Methods for Speech Recognition (published by The MIT Press, pages 93-110, 1997). Those skilled in the art will recognize that a basic Viterbi decoder would only be appropriate in place of the stack decoder if the touch geometry metric only included first order separation vectors. Including higher order separation vectors as is necessary to get a wholesome hand drift estimate makes the touch cost dependent on more than the previous touch and thus violates the first-order Markov condition for basic Viterbi decoders.
In case the previous touch's output had been some key other than ‘D’, say ‘F’, the preliminary ‘F’ output would need to be undone and replaced with ‘D’ by sending a Backspace or Erase key followed by ‘DA’ to the host. The hypothesis tree extensions and output of best sequence would continue similarly for the t2 and t3 touches, except that the match metrics for these touches would include second and third-order separation vectors, respectively. Pruning of hypothesis chains 160 that accumulate relatively high total costs prevents the tree from growing exponentially as more touches occur.
The flowchart in
Copy hj[n] and its parent hypothesis sequence into hseq[n]. . .hseq[0]
for (m=0; m < 10 && n−m >= 0; m++) {
if (m == 0) {//zero-order key/touch alignment error
hseq[n].geomcost = d0(T[n].x − hseq[n].x,T[n].y − hseq[n].y)
continue;
} else if (′T′[n].hand_identity != T[n−m].hand_identity)
continue;
else if (T[n−m] not keystroke or resting finger)
break;
τ[m].x = T[n].x − T[n−m].x //touch separation vectors
τ[m].y = T[n].y − T[n−m].y
λ[m].x = hseq[n].x − hseq[n−m].x //key separation vectors
λ[m].y = hseq[n].y − hseq[n−m].y
wt[m] = ft(T[n].tpress−T[n−m].trelease)
wa[m] = fa(τ[m].x,τ[m].y)
hseq[n].geomcost +=wt[m]*wa[m]*
dM(τ[m].x−λ[m].x,τ[m].y−λ[m].y)
}
hseq[n].cumulcost = hseq[n].geomcost + hseq[n−1].cumulcost
For notational and computational convenience, step 200 copies the particular key hypothesis sequence to be evaluated into the array hseq[ ], starting at hj[n], the new leaf of the hypothesis tree, traversing back through its parent hypothesis references, and stopping at the root. Step 202 computes the zero-order, key/touch misalignment error and stores it as the hypothesis' geometry match metric 96, hseq[n].geomcost. The distance metric d0 determines how the hseq[n].geomcost scales with misalignment in the x and y dimensions. Those skilled in the art will realize that any of a Manhattan metric, Euclidean distance, squared Euclidean distance metric or other metrics would be suitable here. Related art ‘nearest key’ typing recognizers essentially stop with this zero-order alignment error as the final geometry metric, but the current invention includes higher order separation vector mismatches in the geometry metric via the following steps.
Step 204 initializes the order index m to 1. Since each hand's drift is presumed to be independent of the other's drift, only separation vectors for touches and keys typed within the same hand should be considered. Decision diamond 206 tests whether the m th previous hypothesized key hseq[n−m]is normally typed by the same hand as the currently hypothesized key hseq[n]. If not, hseq[n−m] presumably contains no information about the drift of the current hand, so the evaluation process skips m th-order separation vector computations and advances to step 218.
If both touches come from the same hand, decision diamond 207 decides whether the m th previous was actually typing related and thus a possible predictor of hand drift. Decision diamond 207 is particularly important for multi-touch systems that support non-typing synchronous touches such as chord taps, lateral chord slides, and hand resting. For instance, finger location at the beginning or end of pointing motions has nothing to do with subsequent typing drift, so decision diamond 207 should break the loop and skip to the final cost accumulation step 222 when it encounters a touch involved in pointing or any other sliding gesture. However, when typing on a surface, resting a hand (all fingers simultaneously) on home row in between words is quite convenient. Any slight misalignments between the home row keys and finger locations within the resting chord are a good predictor of hand/key misalignment during subsequent typing. Such resting finger locations can be incorporated into separation vector evaluation by having the synchronization detector 13 insert a chain of five special resting finger hypotheses into the hypothesis tree for any five nearly simultaneous touches deemed to be part of a hand resting on or near its home row keys. Each resting finger hypothesis is given key coordinates 92 from the home row key that its finger normally rests on. The hypothesis can look up its finger and hand identity 71 through its causing touch reference 88, and the identities can then index into a table of home row key center coordinates. Resting finger hypotheses are given a null key code 94 so that they produce no output signals to the host computer. For the purpose of key and touch separation vector matching, however, decision diamond 207 and steps 208-216 of
For typing-related touches from the same hand, step 208 creates the m th-order touch separation vector I.[m] by subtracting the spatial and temporal coordinates of the m th previous touch T[n−m] from the current touch T[n]. Likewise, step 210 creates the m th-order key separation vector {umlaut over (l)}>>[m] by subtracting the layout coordinates of hseq[n−m]'s key from the currently hypothesized key hseq [n].
Step 212 computes the temporal confidence weighting wr[m] that should decrease monotonically toward 0 with the time elapsed between the press 72 of the current touch, T[n].tpress and release 73 of the m th previous touch, T[n−m].trelease. The release time is used in case the preceding touch was caused by a hand that began resting near home row many seconds ago but lifted off quite recently. This temporal confidence weighting is meant to reflect the fact that old touches are poorer predictors of the current hand drift than newer touches. Those skilled in the art will realize that the exact downward slope for this weighting function can be empirically optimized by computing old and new touch drift correlations from actual typing samples. For instance, if the typing samples showed that the hand/layout alignment error remained fairly consistent over ten second periods, then the weighting function should be designed to stay well above 0 for touches less than ten seconds old.
Step 214 computes a touch adjacency weighting wa[m] that should decrease monotonically toward 0 as the separation between the current and m th previous touch increases. The touch adjacency weighting is meant to reflect the fact that the separation between touches by the same finger or an adjacent finger, especially if the fingers have not reached far between the touches, is a better predictor of finger drift and overall hand drift than separation vectors for touches by non-adjacent fingers. Thus the second-order separation vector between t2 and t0 in
Step 216 adds to the geometry metric a cost for any mismatch between the m th-order touch separation vector {umlaut over (l)}.[m] and the m th-order key separation vector {umlaut over (l)}>>[m]. This incremental cost should generally increase with the magnitude of the difference between the two vectors. In the preferred embodiment, the square of the magnitude of the vector difference is weighted by the temporal confidence wr[m] and adjacency confidence wa[m] to obtain the m th-order cost increment. The squared Euclidean metric is preferred for dM because it favors sequences with uniformly small vector differences.
Step 218 increments the order index m so that decision diamond 220 can decide whether to continue evaluating higher order separation vectors. Ideally, the evaluation process would continue with previous touches all the way back to the tree root, where m reaches n, but in practice it is usually sufficient to include separation vectors from the ten or so most recent typing-related touches. Once decision diamond 220 decides m has reached its useful limit, flow falls through to the final step 222. Step 222 sets the sequence cumulative match metric hj[m].cumulcost to the sum of the new touch cost hseq[n]. geomcost and the parent's cumulative metric hseq[n−1].cumulcost.
It is also instructive to examine an alternative embodiment of geometry match metric evaluation that, mathematically, is the exact equivalent of and produces the same result as the process in FIG. 7. However, a different factoring of the computations lends this alternative embodiment a differently intuitive interpretation. For the convenience of those of ordinary skill in the art, this alternative embodiment is shown below as pseudocode:
Copy hj[n] and its parent hypothesis sequence into hseq[n]. . .hseq[0]
Allocate key/touch error array e[ ] for different orders
for (m=0; m < 10 && n−m >= 0; m++) {
e[m].x = T[n−m].x − hseq[n−m].x //alignment errors
e[m].y = T[n−m].y − hseq[n−m].y
if (m == 0) {//zero-order key/touch alignment error
hseq[n].geomcost = d0(e[0].x,e[0].y)
continue;
} else if (T[n].hand_identity != T[n−m].hand_identity)
continue;
else if(T[n−m] not keystroke or resting finger)
break;
wt[m] = ft(T[n].tpress − T[n−m].trelease)
τ[m].x = T[n].x − T[n−m].x //touch separation vectors
τ[m].y = T[n],y − T[n−m].y
wa[m] = fa(τ[m].x,τ[m].y)
hseq[n].geomcost +=wt[m]*wa[m]*
dM(e[0].x−e[m].x,e[0].y−e[m].y)
}
hseq[n].cumulcost = hseq[n].geomcost + hseq[n−1].cumulcost
Both embodiments compute the zero-order alignment error component the same, but this alternative embodiment restates the comparison between the m th-order key and touch separate vectors as a comparison between the new touch T[n]'s key/touch alignment error vector, c[0], and the m th previous touch T[n−m]'s key/touch alignment error vector, e[m]. This suggests that the stack decoder in either embodiment will tend to pick as the best sequence a key hypothesis sequence whose individual key/touch alignment error vectors at small yet consistent with one another. Clearly this alternative, equivalent embodiment falls well within the scope of this invention.
The output module in
The preferred embodiment of the output module adopts a compromise. It will only replace characters within the current word (i.e. it will not go back past any space characters and change any completed words), and it will only replace these characters it they have only been typed within the last couple seconds, before the typist has had a chance to notice and correct the probably erroneous old characters himself. The output module starts with the current best hypothesis 350 hbest[n]from the stack decoder. Step 352 sets the previous output index m to 1. Decision diamond 354 checks whether the hypothesis 77 whose key was output for touch T[n−m] was hbest[n]'s parent hypothesis hbest[n−m]. If not, decision diamond 356 checks whether the old key was a word-breaking space or was output more than a few seconds ago. It not, step 358 sends an Erase or Backspace key to the host to undo the old character, and step 360 increments m to continue checking for a parent hypothesis that both the best sequence and previously sent sequence share. Once that parent is found or the search is aborted at a word boundary, step 362 begins sending the replacement key codes 94 from the hbest[ ] sequence, looping through step 363 to increment m until decision diamond finds that m has reached 0, and hseq[n]'s key code 94 has been transmitted.
Now that the preferred embodiment of the typing recognizer has been described, it is instructive to consider additional consequences of its design. One important consequence is that the key activated may not always be the key nearest the fingertip. Generating a neighboring key when the finger actually lands right on top of another key would be startling to the user. However, if the adjacency weightings are kept sufficiently low, the separation vectors cannot override a zero-order, key/touch position error near zero. Proper tuning of the adjacency weighting function ensures that separation vectors can only be decisive when the finger lies in a zone between keys, at least 2-4 mm (⅛″-¼″) from the center of any key.
To further improve recognition accuracy when typing plain English or another predictable language, alternative embodiments of the typing recognizer may incorporate a spelling model. Such integration of spelling models into character recognizers is clearly taught in the handwriting recognition art (see, for example, the post-processing with Markov model and Dictionary in U.S. Pat. No. 5,812,698 to Platt et al. and the use of trigrams in U.S. Pat. No. 6,137,908), and will only be summarized here briefly. Basicly, the spelling model computes for each hypothesis a character transition cost that indicates whether the hypothesized key/character is building a dictionary word out of its parent hypothesis sequence. Costs will be high for character transitions that cannot be found in the dictionary. Command or editing keys can be given a neutral or zero spelling cost. Step 222 of
The case of a finger repetitively striking the same location halfway between keys is a good example of the advantages of considering touch sequence geometry in addition to zero-order alignment error, especially for typing recognizers that include a spelling model. Typists find it disconcerting if they strike the same location repeatedly yet the decoder outputs different neighboring characters. This can happen, say, if the user intended to type ‘DDD’ but the three consecutive finger strikes occur roughly between the ‘S’, ‘E’, and ‘W’ and ‘D’ keys. For a ‘nearest key’ recognizer with spelling model, the zero-order alignment errors for all four keys would be roughly equal, leaving the character transition costs to dominate and encourage the stack decoder to output common spelling sequences like ‘WES’, ‘SEW’, and ‘DES.’ But for a typing recognizer improved with touch geometry matching, only the key sequences ‘SSS’, ‘EEE’, ‘DDD’ and ‘WWW’ have small key separation vectors matching the small touch separations, so these sequences' relatively low geometry match costs would override the spelling model, causing one of them to be output. Even though the ‘SSS’ or ‘EEE’ sequences may not be what the typist intended, they are less disconcerting than a mixed output sequence like ‘SEW’ when the typist knows her finger was not hopping between keys. Thus separation vector matching can overcome misleading character transition costs to ensure the typist sees a consistent, homogeneous output sequence when a finger strikes approximately the same location repeatedly.
Though embodiments and applications of this invention have been shown and described, it will be apparent to those skilled in the art that numerous further embodiments and modifications than mentioned above are possible without departing from the inventive concepts disclosed herein. The invention, therefore, is not to be restricted except in the true spirit and scope of the appended claims.
Patent | Priority | Assignee | Title |
10216279, | Jun 19 2008 | Tactile Display, LLC | Interactive display with tactile feedback |
10254890, | Jan 03 2007 | Apple Inc. | Front-end signal compensation |
10338789, | Jul 30 2004 | Apple Inc. | Operation of a computer with touch screen interface |
10437459, | Jan 07 2007 | Apple Inc. | Multitouch data fusion |
10459523, | Apr 13 2010 | Tactile Displays, LLC | Interactive display with tactile feedback |
10705692, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Continuous and dynamic scene decomposition for user interface |
10719131, | Apr 13 2010 | Tactile Displays, LLC | Interactive display with tactile feedback |
10725587, | Jan 03 2007 | Apple Inc. | Front-end signal compensation |
10908729, | May 06 2004 | Apple Inc. | Multipoint touchscreen |
10915207, | May 02 2006 | Apple Inc. | Multipoint touch surface controller |
10921941, | Mar 04 2005 | Apple Inc. | Electronic device having display and surrounding touch sensitive surfaces for user interface and control |
10990183, | Apr 13 2010 | Tactile Displays, LLC | Interactive display with tactile feedback |
10990184, | Apr 13 2010 | Tactile Displays, LLC | Energy efficient interactive display with energy regenerative keyboard |
10996762, | Apr 13 2010 | Tactile Displays, LLC | Interactive display with tactile feedback |
11195118, | Nov 20 2017 | International Business Machines Corporation | Detecting human input activity in a cognitive environment using wearable inertia and audio sensors |
11353989, | Jan 03 2007 | Apple Inc. | Front-end signal compensation |
11360509, | Mar 04 2005 | Apple Inc. | Electronic device having display and surrounding touch sensitive surfaces for user interface and control |
11481109, | Jan 07 2007 | Apple Inc. | Multitouch data fusion |
11604547, | May 06 2004 | Apple Inc. | Multipoint touchscreen |
11816329, | Jan 07 2007 | Apple Inc. | Multitouch data fusion |
11853518, | May 02 2006 | Apple Inc. | Multipoint touch surface controller |
7848825, | Jan 03 2007 | Apple Inc | Master/slave mode for sensor processing devices |
8049732, | Jan 03 2007 | Apple Inc | Front-end signal compensation |
8068093, | Oct 10 2006 | Promethean House | Duplicate objects |
8352884, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Dynamic reconfiguration of GUI display decomposition based on predictive model |
8375295, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Customization of GUI layout based on history of use |
8405617, | Jan 03 2007 | Apple Inc | Gated power management over a system bus |
8413075, | Jan 04 2008 | Apple Inc | Gesture movies |
8434003, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Touch control with dynamically determined buffer region and active perimeter |
8553004, | Jan 03 2007 | Apple Inc. | Front-end signal compensation |
8583421, | Mar 06 2009 | Google Technology Holdings LLC | Method and apparatus for psychomotor and psycholinguistic prediction on touch based device |
8639494, | Dec 28 2010 | INTUIT INC. | Technique for correcting user-interface shift errors |
8711129, | Jan 03 2007 | Apple Inc | Minimizing mismatch during compensation |
8797283, | Nov 22 2010 | Sony Interactive Entertainment LLC | Method and apparatus for performing user-defined macros |
8907903, | Jan 13 2011 | Sony Interactive Entertainment LLC | Handing control of an object from one touch input to another touch input |
8971572, | Aug 12 2011 | The Research Foundation of State University of New York | Hand pointing estimation for human computer interaction |
9009588, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Customization of GUI layout based on history of use |
9305229, | Jul 30 2012 | Method and system for vision based interfacing with a computer | |
9311528, | Jan 03 2007 | Apple Inc. | Gesture learning |
9323405, | Jan 03 2007 | Apple Inc. | Front-end signal compensation |
9367216, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Hand-held device with two-finger touch triggered selection and transformation of active elements |
9372546, | Aug 12 2011 | The Research Foundation for The State University of New York | Hand pointing estimation for human computer interaction |
9448701, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Customization of GUI layout based on history of use |
9513705, | Jun 19 2008 | Tactile Displays, LLC | Interactive display with tactile feedback |
9524085, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Hand-held device with ancillary touch activated transformation of active element |
9778807, | Jan 03 2007 | Apple Inc. | Multi-touch input discrimination |
9927964, | May 21 2009 | SONY INTERACTIVE ENTERTAINMENT INC | Customization of GUI layout based on history of use |
Patent | Priority | Assignee | Title |
3333160, | |||
3541541, | |||
3662105, | |||
3798370, | |||
4246452, | Jan 05 1979 | Mattel, Inc. | Switch apparatus |
4550221, | Oct 07 1983 | VOLKS COMMUNICATION, INC | Touch sensitive control device |
4672364, | Jun 18 1984 | Carroll Touch Inc | Touch input device having power profiling |
4672558, | Sep 25 1984 | CANBERRA ALBUQUERQUE, INC | Touch-sensitive data input device |
4692809, | Nov 20 1984 | HE HOLDINGS, INC , A DELAWARE CORP ; Raytheon Company | Integrated touch paint system for displays |
4695827, | Nov 20 1984 | HE HOLDINGS, INC , A DELAWARE CORP ; Raytheon Company | Electromagnetic energy interference seal for light beam touch panels |
4733222, | Dec 27 1983 | Integrated Touch Arrays, Inc.; INTEGRATED TOUCH ARRAYS, INC A CORP OF DE | Capacitance-variation-sensitive touch sensing array system |
4734685, | Jul 28 1983 | Canon Kabushiki Kaisha | Position control apparatus |
4746770, | Feb 17 1987 | Sensor Frame Incorporated; SENSOR FRAME INCORPORATION | Method and apparatus for isolating and manipulating graphic objects on computer video monitor |
4771276, | Apr 15 1985 | International Business Machines Corporation | Electromagnetic touch sensor input system in a cathode ray tube display device |
4788384, | Dec 18 1986 | Centre National de la Recherche Scientifique | Device for two-dimensional localization of events that generate current on a resistive surface |
4806846, | Jul 06 1987 | High accuracy direct reading capacitance-to-voltage converter | |
4898555, | Mar 23 1989 | PROQUEST BUSINESS SOLUTIONS INC | Display screen bezel and assembly method |
4968877, | Sep 14 1988 | Sensor Frame Corporation | VideoHarp |
5003519, | May 26 1988 | ETA S.A. Fabriques d'Ebauches | Alarm arrangement for a timepiece |
5017030, | Jul 07 1986 | Ergonomically designed keyboard | |
5178477, | Jun 06 1991 | MOTIONLESS KEYBOARD COMPANY | Ergonomic keyboard input device |
5189403, | Sep 26 1989 | HANGER SOLUTIONS, LLC | Integrated keyboard and pointing device system with automatic mode change |
5194862, | Jun 29 1990 | U.S. Philips Corporation | Touch sensor array systems and display systems incorporating such |
5224861, | Sep 17 1990 | L-3 Communications Corporation | Training device onboard instruction station |
5241308, | Feb 22 1990 | AVI SYSTEMS, INC ; TMY, INC | Force sensitive touch panel |
5252951, | Apr 28 1989 | INTERNATIONAL BUSINESS MACHINES CORPORATION A CORP OF NEW YORK | Graphical user interface with gesture recognition in a multiapplication environment |
5281966, | Jan 31 1992 | Method of encoding alphabetic characters for a chord keyboard | |
5305017, | Sep 04 1991 | Cirque Corporation | Methods and apparatus for data input |
5345543, | Nov 16 1992 | Apple Inc | Method for manipulating objects on a computer display |
5376948, | Mar 25 1992 | 3M Innovative Properties Company | Method of and apparatus for touch-input computer and related display employing touch force location external to the display |
5398310, | Apr 13 1992 | Apple Inc | Pointing gesture based computer note pad paging and scrolling interface |
5442742, | Dec 21 1990 | Apple Inc | Method and apparatus for the manipulation of text on a computer display screen |
5463388, | Jan 29 1993 | THE CHASE MANHATTAN BANK, AS COLLATERAL AGENT | Computer mouse or keyboard input device utilizing capacitive sensors |
5463696, | May 27 1992 | Apple Inc | Recognition system and method for user inputs to a computer system |
5483261, | Feb 14 1992 | ORGPRO NEXUS INC | Graphical input controller and method with rear screen image detection |
5488204, | Jun 08 1992 | Synaptics Incorporated; Synaptics, Incorporated | Paintbrush stylus for capacitive touch sensor pad |
5495077, | Jun 08 1992 | Synaptics, Inc. | Object position and proximity detector |
5513309, | Jan 05 1993 | Apple Computer, Inc. | Graphic editor user interface for a pointer-based computer system |
5523775, | May 26 1992 | Apple Inc | Method for selecting objects on a computer display |
5530455, | Aug 10 1994 | KYE SYSTEMS AMERICA CORPORATION | Roller mouse for implementing scrolling in windows applications |
5543590, | Jun 08 1992 | SYNAPTICS, INC ; Synaptics, Incorporated | Object position detector with edge motion feature |
5543591, | Jun 08 1992 | SYNAPTICS, INC | Object position detector with edge motion feature and gesture recognition |
5563632, | Apr 30 1993 | 3M Innovative Properties Company | Method of and apparatus for the elimination of the effects of internal interference in force measurement systems, including touch - input computer and related displays employing touch force location measurement techniques |
5563996, | Apr 13 1992 | Apple Inc | Computer note pad including gesture based note division tools and method |
5565658, | Jul 13 1992 | Cirque Corporation | Capacitance-based proximity with interference rejection apparatus and methods |
5579036, | Apr 28 1994 | NCR Corporation | Touch screen device and shielding bracket therefor |
5581681, | Dec 14 1994 | Apple Inc | Pointing gesture based computer note pad paging and scrolling interface |
5583946, | Sep 30 1993 | Apple Inc | Method and apparatus for recognizing gestures on a computer system |
5590219, | Sep 30 1993 | Apple Inc | Method and apparatus for recognizing gestures on a computer system |
5592566, | Jan 05 1993 | Apple Inc | Method and apparatus for computerized recognition |
5594810, | Sep 19 1993 | Apple Inc | Method and apparatus for recognizing gestures on a computer system |
5596694, | May 27 1992 | Apple Inc | Method and apparatus for indicating a change in status of an object and its disposition using animation |
5612719, | Dec 03 1992 | Apple Inc | Gesture sensitive buttons for graphical user interfaces |
5631805, | Sep 27 1995 | 3M Innovative Properties Company | Touch screen enclosure having an insertable graphic sheet |
5633955, | May 27 1992 | Apple Computer, Inc. | Method of connecting shapes on a display of a computer system |
5634102, | Aug 07 1995 | Apple Inc | Methods and apparatus for a selectable backdrop |
5636101, | Sep 27 1995 | 3M Innovative Properties Company | Touch screen enclosure system having touch screen pan and hinged rear enclosure section for ease of serviceability |
5642108, | Jun 28 1991 | Infogrip, Inc. | Chordic keyboard system for generating a signal in response to a chord that is assigned using a correlation based on a composite chord-difficulty index |
5644657, | Jan 05 1993 | Apple Computer, Inc. | Method for locating and displaying information in a pointer-based computer system |
5666113, | Jul 31 1991 | 3M Innovative Properties Company | System for using a touchpad input device for cursor control and keyboard emulation |
5666502, | Aug 07 1995 | Apple Inc | Graphical user interface using historical lists with field classes |
5666552, | Dec 21 1990 | Apple Inc | Method and apparatus for the manipulation of text on a computer display screen |
5675361, | Aug 23 1995 | Computer keyboard pointing device | |
5677710, | May 10 1993 | Apple Inc | Recognition keypad |
5689253, | Apr 10 1991 | Kinesis Corporation | Ergonomic keyboard apparatus |
5710844, | May 27 1992 | Apple Inc | Method for searching and displaying results in a pen-based computer system |
5729250, | May 08 1995 | Toshiba Global Commerce Solutions Holdings Corporation | Front cover assembly for a touch sensitive device |
5730165, | Dec 26 1995 | Atmel Corporation | Time domain capacitive field detector |
5736976, | Feb 13 1995 | Computer data entry apparatus with hand motion sensing and monitoring | |
5741990, | Feb 17 1989 | Notepool, Ltd. | Method of and means for producing musical note relationships |
5745116, | Sep 09 1996 | Google Technology Holdings LLC | Intuitive gesture-based graphical user interface |
5745716, | Aug 07 1995 | Apple Inc | Method and apparatus for tab access and tab cycling in a pen-based computer system |
5746818, | Aug 31 1995 | Seiko Epson Corporation | Pigment ink composition capable of forming image having no significant bleeding or feathering |
5748269, | Nov 21 1996 | Westinghouse Air Brake Company | Environmentally-sealed, convectively-cooled active matrix liquid crystal display (LCD) |
5764222, | May 28 1996 | International Business Machines Corporation | Virtual pointing device for touchscreens |
5767457, | Nov 13 1995 | Cirque Corporation | Apparatus and method for audible feedback from input device |
5767842, | Feb 07 1992 | International Business Machines Corporation | Method and device for optical input of commands or data |
5790104, | Jun 25 1996 | International Business Machines Corporation | Multiple, moveable, customizable virtual pointing devices |
5790107, | Jun 07 1995 | ELAN MICROELECTRONICS CORP | Touch sensing method and apparatus |
5802516, | Nov 03 1993 | Apple Inc | Method of controlling an electronic book for a computer system |
5808567, | May 17 1993 | DSI DATOTECH SYSTEMS INC | Apparatus and method of communicating using three digits of a hand |
5809267, | Dec 30 1993 | Xerox Corporation | Apparatus and method for executing multiple-concatenated command gestures in a gesture based input system |
5812698, | May 12 1995 | Synaptics, Inc. | Handwriting recognition system and method |
5821690, | Aug 26 1993 | Cambridge Display Technology Limited | Electroluminescent devices having a light-emitting layer |
5821930, | Aug 23 1992 | Qwest Communications International Inc | Method and system for generating a working window in a computer system |
5823782, | Dec 29 1995 | Tinkers & Chance | Character recognition educational system |
5825351, | May 12 1994 | Apple Computer, Inc. | Method and apparatus for noise filtering for an input device |
5825352, | Jan 04 1996 | ELAN MICROELECTRONICS CORP | Multiple fingers contact sensing method for emulating mouse buttons and mouse operations on a touch sensor pad |
5854625, | Nov 06 1996 | Synaptics Incorporated | Force sensing touchpad |
5880411, | Jun 08 1992 | Synaptics Incorporated | Object position detector with edge motion feature and gesture recognition |
5898434, | May 15 1991 | Apple Inc | User interface system having programmable user interface elements |
5920309, | Jan 04 1996 | ELAN MICROELECTRONICS CORP | Touch sensing method and apparatus |
5923319, | May 08 1995 | Toshiba Global Commerce Solutions Holdings Corporation | Front cover assembly for touch sensitive device |
5933134, | Jun 25 1996 | LENOVO SINGAPORE PTE LTD | Touch screen virtual pointing device which goes into a translucent hibernation state when not in use |
5943044, | Aug 05 1996 | INTERLINK ELECTRONIC | Force sensing semiconductive touchpad |
6002389, | Apr 24 1996 | ELAN MICROELECTRONICS CORP | Touch and pressure sensing method and apparatus |
6002808, | Jul 26 1996 | Mitsubishi Electric Research Laboratories, Inc | Hand gesture control system |
6020881, | May 24 1993 | Sun Microsystems | Graphical user interface with method and apparatus for interfacing to remote devices |
6031524, | Jun 07 1995 | Intermec IP CORP | Hand-held portable data terminal having removably interchangeable, washable, user-replaceable components with liquid-impervious seal |
6037882, | Sep 30 1997 | Apple Inc | Method and apparatus for inputting data to an electronic system |
6050825, | May 08 1998 | Speedskin LLC | Opaque, one-size-fits-all computer keyboard cover which covers only the three or four alpha-numeric rows |
6052339, | Jun 11 1997 | Asulab S.A. | Watch with touch reading and setting of time functions |
6072494, | Oct 15 1997 | Microsoft Technology Licensing, LLC | Method and apparatus for real-time gesture recognition |
6084576, | Sep 27 1997 | User friendly keyboard | |
6107997, | Jun 27 1996 | Touch-sensitive keyboard/mouse and computing device using the same | |
6128003, | Dec 20 1996 | Hitachi, Ltd. | Hand gesture recognition system and method |
6131299, | Jul 01 1998 | FARO TECHNOLOGIES, INC | Display device for a coordinate measurement machine |
6135958, | Aug 06 1998 | Siemens Medical Solutions USA, Inc | Ultrasound imaging system with touch-pad pointing device |
6137908, | Jun 29 1994 | Microsoft Technology Licensing, LLC | Handwriting recognition system simultaneously considering shape and context information |
6144380, | Nov 03 1993 | Apple Inc | Method of entering and using handwriting to identify locations within an electronic book |
6188391, | Jul 09 1998 | Synaptics, Incorporated | Two-layer capacitive touchpad and method of making same |
6198515, | Mar 16 1998 | Apparatus and method for controlled sealing between bezel and CRT | |
6208329, | Aug 13 1996 | AVAGO TECHNOLOGIES GENERAL IP SINGAPORE PTE LTD | Supplemental mouse button emulation system, method and apparatus for a coordinate based data input device |
6222465, | Dec 09 1998 | Lucent Technologies Inc. | Gesture-based computer interface |
6239790, | Aug 05 1996 | Interlink Electronics | Force sensing semiconductive touchpad |
6243071, | Nov 03 1993 | Apple Inc | Tool set for navigating through an electronic book |
6246862, | Feb 03 1999 | Google Technology Holdings LLC | Sensor controlled user interface for portable communication device |
6249606, | Feb 19 1998 | CREATIVE TECHNOLOGY LTD | Method and system for gesture category recognition and training using a feature vector |
6288707, | Jul 29 1996 | NEODRÓN LIMITED | Capacitive position sensor |
6289326, | Jun 04 1997 | Portable interactive kiosk | |
6292178, | Oct 19 1998 | JOHNSON & JOHNSON SURGICAL VISION, INC | Screen navigation control apparatus for ophthalmic surgical instruments |
6323846, | Jan 26 1998 | Apple Inc | Method and apparatus for integrating manual input |
6347290, | Jun 24 1998 | HEWLETT-PACKARD DEVELOPMENT COMPANY, L P | Apparatus and method for detecting and executing positional and gesture commands corresponding to movement of handheld computing device |
6377009, | Sep 08 1999 | UUSI, LLC | Capacitive closure obstruction sensor |
6378234, | Apr 09 1999 | Sequential stroke keyboard | |
6380931, | Jun 08 1992 | Synaptics Incorporated | Object position detector with edge motion feature and gesture recognition |
6411287, | Sep 08 1999 | Tyco Electronics Corporation | Stress seal for acoustic wave touchscreens |
6414671, | Jun 08 1992 | Synaptics Incorporated | Object position detector with edge motion feature and gesture recognition |
6421234, | Oct 10 2000 | JUNIPER SYSTEMS INC | Handheld electronics device having ergonomic features |
6452514, | Jan 26 1999 | Atmel Corporation | Capacitive sensor and array |
6457355, | Aug 27 1999 | Level sensing | |
6466036, | Nov 25 1998 | NEODRÓN LIMITED | Charge transfer capacitance measurement circuit |
6515669, | Oct 23 1998 | Olympus Optical Co., Ltd. | Operation input device applied to three-dimensional input device |
6525749, | Dec 30 1993 | Xerox Corporation | Apparatus and method for supporting the implicit structure of freeform lists, outlines, text, tables and diagrams in a gesture-based input system and editing system |
6535200, | Jul 29 1996 | NEODRÓN LIMITED | Capacitive position sensor |
6543684, | Mar 28 2000 | NCR Voyix Corporation | Transaction terminal with privacy shield for touch-screen pin entry |
6543947, | Mar 14 2001 | Keyboard having keys arranged in a pan configuration | |
6570557, | Feb 10 2001 | Apple Inc | Multi-touch system and method for emulating modifier keys via fingertip chords |
6593916, | Nov 03 2000 | ELO TOUCH SOLUTIONS, INC | Touchscreen having multiple parallel connections to each electrode in a series resistor chain on the periphery of the touch area |
6610936, | Jun 08 1992 | Synaptics, Inc. | Object position detector with edge motion feature and gesture recognition |
6624833, | Apr 17 2000 | WSOU Investments, LLC | Gesture-based input interface system with shadow detection |
6639577, | Mar 04 1998 | Rovi Technologies Corporation | Portable information display device with ergonomic bezel |
6650319, | Oct 29 1996 | ELO TOUCH SOLUTIONS, INC | Touch screen based topological mapping with resistance framing design |
6658994, | Apr 10 2002 | Antares Capital LP; ANTARES CAPITAL LP, AS SUCCESSOR AGENT | Modular assembly for a holding cabinet controller |
6670894, | Feb 05 2001 | System and method for keyboard independent touch typing | |
6677932, | Jan 28 2001 | Apple Inc | System and method for recognizing touch typing under limited tactile feedback conditions |
6677934, | Jul 30 1999 | L-3 Communications Corporation | Infrared touch panel with improved sunlight rejection |
6724366, | Apr 03 2001 | PINEAPPLE34, LLC | Thumb actuated x-y input device |
6757002, | Nov 04 1999 | Hewlett-Packard Company | Track pad pointing device with areas of specialized function |
6803906, | Jul 05 2000 | SMART Technologies ULC | Passive touch system and method of detecting user input |
6842672, | Feb 28 2002 | Garmin International, Inc. | Cockpit instrument panel systems and methods with redundant flight data display |
6856259, | Feb 06 2004 | ELO TOUCH SOLUTIONS, INC | Touch sensor system to detect multiple touch events |
6888536, | Jan 26 1998 | Apple Inc | Method and apparatus for integrating manual input |
6900795, | Feb 27 2002 | WORD MACHINERY, INC | Unitary molded lens filter for touch screen interface |
6927761, | Mar 29 2002 | 3M Innovative Properties Company | Moisture deflector for capacitive NFI touch screens for use with bezels of conductive material |
6942571, | Oct 16 2000 | SG GAMING, INC | Gaming device with directional and speed control of mechanical reels using touch screen |
6965375, | Apr 27 2001 | Qualcomm Incorporated | Compact integrated touch panel display for a handheld device |
6972401, | Jan 30 2003 | SMART Technologies ULC | Illuminated bezel and touch system incorporating the same |
6977666, | Sep 04 1998 | INNOVATIVE SOLUTIONS AND SUPPORT INC | Flat panel display using dual CPU's for an aircraft cockpit |
6985801, | Feb 28 2002 | Garmin International, Inc. | Cockpit instrument panel systems and methods with redundant flight data display |
6992659, | May 22 2001 | Qualcomm Incorporated | High transparency integrated enclosure touch screen assembly for a portable hand held device |
7031228, | Aug 30 2002 | Asulab S.A. | Timepiece with touch-type reading and control of time data |
20020118848, | |||
20030006974, | |||
20030076301, | |||
20030076303, | |||
20030076306, | |||
20030095095, | |||
20030095096, | |||
20030098858, | |||
20030206202, | |||
20030234768, | |||
20040263484, | |||
20050012723, | |||
20050052425, | |||
20050104867, | |||
20050110768, | |||
20060022955, | |||
20060022956, | |||
20060026521, | |||
20060026535, | |||
20060026536, | |||
20060032680, | |||
20060033724, | |||
20060053387, | |||
20060066582, | |||
20060085757, | |||
20060097991, | |||
20060197753, | |||
CA1243096, | |||
DE10251296, | |||
EP288692, | |||
EP464908, | |||
EP664504, | |||
EP1014295, | |||
WO2003088176, | |||
WO2006023569, | |||
WO97018547, | |||
WO97023738, | |||
WO9814863, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jan 13 2006 | Apple Inc. | (assignment on the face of the patent) | / | |||
Aug 31 2007 | FINGERWORKS, INC | Apple Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 019856 | /0474 |
Date | Maintenance Fee Events |
Nov 04 2009 | ASPN: Payor Number Assigned. |
Jun 15 2011 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Jul 01 2015 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
Nov 24 2012 | 4 years fee payment window open |
May 24 2013 | 6 months grace period start (w surcharge) |
Nov 24 2013 | patent expiry (for year 4) |
Nov 24 2015 | 2 years to revive unintentionally abandoned end. (for year 4) |
Nov 24 2016 | 8 years fee payment window open |
May 24 2017 | 6 months grace period start (w surcharge) |
Nov 24 2017 | patent expiry (for year 8) |
Nov 24 2019 | 2 years to revive unintentionally abandoned end. (for year 8) |
Nov 24 2020 | 12 years fee payment window open |
May 24 2021 | 6 months grace period start (w surcharge) |
Nov 24 2021 | patent expiry (for year 12) |
Nov 24 2023 | 2 years to revive unintentionally abandoned end. (for year 12) |