A virtual computer monitor is described which enables instantaneous and intuitive visual access to large amounts of visual data by providing the user with a large display projected virtually in front of the user. The user wears a head-mounted display or holds a portable display containing a head-tracker or other motion tracker, which together allow the user to position an instantaneous viewport provided by the display at any position within the large virtual display by turning to look in the desired direction. The instantaneous viewport further includes a mouse pointer, which may be positioned by turning the user's head or moving the portable display, and which may be further positioned using a mouse or analogous control device. A particular advantage of the virtual computer monitor is intuitive access to enlarged computer output for visually-impaired individuals.
|
1. A computer implemented method for assisting a user in the control and operation of a computer system having a display screen coupled with the computer system, the computer implemented method comprising the steps of:
equipping the user with a portable display device coupled to the computer system;
mapping visual feedback generated by the computer system and intended for display upon a display screen into a virtual desktop suitable for display by the portable display device;
continually displaying a certain portion of the virtual desktop within the portable display device such that the user can view the certain portion of the virtual desktop;
tracking motion of the portable display device including discrete motion gestures initiated by the user; and
when the tracked motion corresponds to a request for a special discrete command, performing the special discrete command;
wherein the virtual desktop represents a text document; and
wherein the tracked motion is a shaking motion of the portable device and the special discrete command is an erase command.
24. A computer implemented method for assisting a user in the control and operation of a computer system having a display screen coupled with the computer system, the computer implemented method comprising the steps of:
equipping the user with a portable display device coupled to the computer system;
equipping the user with a wearable control device operable to monitor movement and gestures made by the user with the portable control device, the computer system storing a map corresponding certain gestures and movements to certain computer operating instructions;
mapping visual feedback generated by the computer system and intended for display upon a display screen into a virtual desktop suitable for display by the portable display device;
continually displaying a certain portion of the virtual desktop within the portable display device such that the user can view the certain portion of the virtual desktop;
tracking motion of the portable control device;
determining that a user implemented gesture with the portable control device corresponds to a specific computer operating instruction; and
executing the specific computer operating instruction.
11. A computer implemented method for assisting a vision-impaired user in the control and operation of a computer system, the computer system being display driven in that a primary source of feedback for interacting with the computer system is visual feedback generated by the computer system, the visual feedback intended for display upon a display screen coupled with the computer system, the computer implemented method comprising the steps of:
equipping the vision-impaired user with a portable display device coupled to the computer system;
mapping the visual feedback generated by the computer system and intended for display upon a display screen into a virtual desktop suitable for display by the portable display device;
continually displaying a certain portion of the virtual desktop within the portable display device such that the vision-impaired user can view the certain portion of the virtual desktop, the certain portion of the virtual desktop corresponding to that portion of the virtual desktop which is located within the line of sight of the vision-impaired user;
adjusting a magnification of the virtual desktop in accordance with input from the visually-impaired user, whereby the certain portion is correspondingly adjusted such that the certain portion displayed within the portable device is a larger or a smaller percentage of the virtual desktop depending upon the vision-impaired user's magnification adjustment, the adjust magnification input being a function of the portable display;
tracking changes in the vision-impaired user's line of sight; and
updating the portable display device such that the certain portion displayed maintains the correspondence with that portion of the virtual desktop which is located within the line of sight of the vision-impaired user,
whereby the visually-impaired user can adjust the magnification of the visual feedback provided by the computer system to a magnification level improving comprehension of the visual feedback, and, even when the magnification level precludes simultaneous display of all the visual feedback within the portable display device, the visually-impaired user may still access all the visual feedback by adjusting his or her line of sight.
0. 2. A computer implemented method as recited in
0. 3. A computer implemented method as recited in
0. 4. A computer implemented method as recited in
0. 5. A computer implemented method as recited in
0. 6. A computer implemented method as recited in
0. 7. A computer implemented method as recited in
0. 8. A computer implemented method as recited in
0. 9. A computer implemented method as recited in
10. A computer implemented method as recited in
12. A computer implemented method as recited in
13. A computer implemented method as recited in
receiving a request to adjust the fixed relation between the pointer and the vision-impaired user's line of sight; and
adjusting the fixed relation in accordance with the received request.
14. A computer implemented method as recited in
15. A computer implemented method as recited in
receiving a create selection box command via the pointer control input device;
anchoring a first comer corner of a selection box at a first point where the pointer is disposed within the certain portion displayed upon receipt of the create selection box command;
sizing the selection box in conjunction with movement of the vision-impaired user's line of sight;
receiving a close selection box command via the pointer control input device;
defining the selection box by anchoring a second comer corner of the selection box, the second comer corner opposing the first comer corner, the second comer corner anchored at a second point where the pointer is disposed within the certain portion displayed upon receipt of the close selection box command; and
marking as selected that portion of the virtual desktop, which is bounded by the selection box.
16. A computer implemented method as recited in
17. A computer implemented method as recited in
18. A computer implemented method as recited in
19. A computer implemented method as recited in
20. A computer implemented method as recited in
21. A computer implemented method as recited in
22. A computer implemented method as recited in
monitoring a real scene in real space and time;
capturing images of the real scene; and
displaying within a first window of the multiple application windows the captured images of the real scene.
23. A computer implemented method as recited in
25. A computer implemented method as recited in
26. A computer implemented method as recited in
27. A computer implemented method as recited in
28. A computer implemented method as recited in
29. A computer implemented method as recited in
|
This application claims can notheadtenns terms of several preferred embodiments, there are many alterations, permutations, and equivalents which may fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and apparatuses of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.
Fateh, Sina, Flack, James F., Zwern, Arthur L.
Patent | Priority | Assignee | Title |
10067559, | Nov 30 2011 | GOOGLE LLC | Graphical interface having adjustable borders |
10180572, | Feb 28 2010 | Microsoft Technology Licensing, LLC | AR glasses with event and user action control of external applications |
10268888, | Feb 28 2010 | Microsoft Technology Licensing, LLC | Method and apparatus for biometric data capture |
10300383, | Dec 22 2010 | Nintendo Co., Ltd. | Game system, game device, storage medium storing game program, and game process method |
10539787, | Feb 28 2010 | Microsoft Technology Licensing, LLC | Head-worn adaptive display |
10860100, | Feb 28 2010 | Microsoft Technology Licensing, LLC | AR glasses with predictive control of external device based on event input |
11703945, | Nov 19 2020 | Beijing Boe Optoelectronics Technology Co., Ltd.; BOE TECHNOLOGY GROUP CO., LTD. | Augmented reality information prompting system, display control method, equipment and medium |
8606316, | Oct 21 2009 | Xerox Corporation | Portable blind aid device |
8643951, | Mar 15 2012 | GOOGLE LLC | Graphical menu and interaction therewith through a viewing window |
8957910, | Dec 22 2010 | Nintendo Co., Ltd. | Game system, game device, storage medium storing game program, and game process method |
8970692, | Sep 01 2011 | Industrial Technology Research Institute | Head mount personal computer and interactive system using the same |
9013264, | Mar 12 2011 | Perceptive Devices, LLC | Multipurpose controller for electronic devices, facial expressions management and drowsiness detection |
9035878, | Feb 29 2012 | GOOGLE LLC | Input system |
9091851, | Feb 28 2010 | Microsoft Technology Licensing, LLC | Light control in head mounted displays |
9097890, | Feb 28 2010 | Microsoft Technology Licensing, LLC | Grating in a light transmissive illumination system for see-through near-eye display glasses |
9097891, | Feb 28 2010 | Microsoft Technology Licensing, LLC | See-through near-eye display glasses including an auto-brightness control for the display brightness based on the brightness in the environment |
9128281, | Sep 14 2010 | Microsoft Technology Licensing, LLC | Eyepiece with uniformly illuminated reflective display |
9129295, | Feb 28 2010 | Microsoft Technology Licensing, LLC | See-through near-eye display glasses with a fast response photochromic film system for quick transition from dark to clear |
9134534, | Feb 28 2010 | Microsoft Technology Licensing, LLC | See-through near-eye display glasses including a modular image source |
9182596, | Feb 28 2010 | Microsoft Technology Licensing, LLC | See-through near-eye display glasses with the optical assembly including absorptive polarizers or anti-reflective coatings to reduce stray light |
9223134, | Feb 28 2010 | Microsoft Technology Licensing, LLC | Optical imperfections in a light transmissive illumination system for see-through near-eye display glasses |
9229227, | Feb 28 2010 | Microsoft Technology Licensing, LLC | See-through near-eye display glasses with a light transmissive wedge shaped illumination system |
9245364, | Jan 16 2012 | Lenovo (Beijing) Co., Ltd.; Beijing Lenovo Software Ltd.; LENOVO BEIJING CO , LTD ; Beijing Lenovo Software Ltd | Portable device and display processing method for adjustment of images |
9268136, | Sep 28 2012 | GOOGLE LLC | Use of comparative sensor data to determine orientation of head relative to body |
9285589, | Feb 28 2010 | Microsoft Technology Licensing, LLC | AR glasses with event and sensor triggered control of AR eyepiece applications |
9329689, | Feb 28 2010 | Microsoft Technology Licensing, LLC | Method and apparatus for biometric data capture |
9341843, | Dec 30 2011 | Microsoft Technology Licensing, LLC | See-through near-eye display glasses with a small scale image source |
9366862, | Feb 28 2010 | Microsoft Technology Licensing, LLC | System and method for delivering content to a group of see-through near eye display eyepieces |
9448687, | Feb 05 2014 | GOOGLE LLC | Zoomable/translatable browser interface for a head mounted device |
9547406, | Oct 31 2011 | GOOGLE LLC | Velocity-based triggering |
9557152, | Sep 28 2012 | GOOGLE LLC | Use of comparative sensor data to determine orientation of head relative to body |
9759917, | Feb 28 2010 | Microsoft Technology Licensing, LLC | AR glasses with event and sensor triggered AR eyepiece interface to external devices |
9808717, | Dec 22 2010 | Nintendo Co., Ltd. | Game system, game device, storage medium storing game program, and game process method |
9875406, | Feb 28 2010 | Microsoft Technology Licensing, LLC | Adjustable extension for temple arm |
9996149, | Feb 22 2016 | IMMERSACAD CORPORATION | Method for one-touch translational navigation of immersive, virtual reality environments |
Patent | Priority | Assignee | Title |
4209255, | Mar 30 1979 | WESTINGHOUSE NORDEN SYSTEMS INCORPORATED | Single source aiming point locator |
4227209, | Aug 09 1978 | The Charles Stark Draper Laboratory, Inc. | Sensory aid for visually handicapped people |
4548485, | Sep 01 1983 | Reading device for the visually handicapped | |
4565999, | Apr 01 1983 | Bankers Trust Company | Light pencil |
4567479, | Dec 23 1982 | Directional controller apparatus for a video or computer input | |
4682159, | Jun 20 1984 | PRENTKE ROMICH COMPANY | Apparatus and method for controlling a cursor on a computer display |
4790028, | Sep 12 1986 | Westinghouse Electric Corp. | Method and apparatus for generating variably scaled displays |
5003300, | Jul 27 1987 | Microvision, Inc | Head mounted display for miniature video display system |
5109282, | Jun 20 1990 | SCHEPENS EYE RESEARCH INSTITUTE, INC , THE | Halftone imaging method and apparatus utilizing pyramidol error convergence |
5125046, | Jul 26 1990 | Digitally enhanced imager for the visually impaired | |
5151722, | Nov 05 1990 | The Johns Hopkins University | Video display on spectacle-like frame |
5195180, | Jun 23 1988 | Sharp Kabushiki Kaisha | Method for displaying an image including characters and a background |
5267331, | Jul 26 1990 | Digitally enhanced imager for the visually impaired | |
5281957, | Nov 14 1984 | Schoolman Scientific Corp. | Portable computer and head mounted display |
5283560, | Jun 25 1991 | HEWLETT-PACKARD DEVELOPMENT COMPANY, L P | Computer system and method for displaying images with superimposed partially transparent menus |
5320538, | Sep 23 1992 | L-3 Communications Corporation | Interactive aircraft training system and method |
5322441, | Oct 05 1990 | Texas Instruments Incorporated | Method and apparatus for providing a portable visual display |
5325123, | Apr 16 1992 | Method and apparatus for variable video magnification | |
5359675, | Jul 26 1990 | Video spectacles | |
5367315, | Nov 15 1990 | Eyetech Corporation | Method and apparatus for controlling cursor movement |
5367614, | Apr 01 1992 | Grumman Aerospace Corporation | Three-dimensional computer image variable perspective display system |
5373857, | Jun 18 1993 | TDG Acquisition Company, LLC | Head tracking apparatus |
5422653, | Jan 07 1993 | SIMULATED PERCEPTS, LLC | Passive virtual reality |
5442734, | Mar 06 1991 | Fujitsu Limited | Image processing unit and method for executing image processing of a virtual environment |
5450596, | Jul 18 1991 | REDWEAR INTERACTIVE INC | CD-ROM data retrieval system using a hands-free command controller and headwear monitor |
5526481, | Jul 26 1993 | Dell USA L.P. | Display scrolling system for personal digital assistant |
5526812, | Jun 21 1993 | General Electric Company | Display system for enhancing visualization of body structures during medical procedures |
5579026, | May 14 1993 | Olympus Optical Co., Ltd. | Image display apparatus of head mounted type |
5581271, | Dec 05 1994 | Raytheon Company; HE HOLDINGS, INC , A DELAWARE CORP | Head mounted visual display |
5581670, | Jul 21 1993 | JB PARTNERS, LLC | User interface having movable sheet with click-through tools |
5587936, | Nov 30 1990 | Sun Microsystems, Inc | Method and apparatus for creating sounds in a virtual world by simulating sound in specific locations in space and generating sounds as touch feedback |
5590062, | Jul 02 1993 | MATSUSHITA ELECTRIC INDUSTRIAL CO , LTD | Simulator for producing various living environments mainly for visual perception |
5602566, | Aug 24 1993 | Hitachi Maxell, Ltd | Small-sized information processor capable of scrolling screen in accordance with tilt, and scrolling method therefor |
5617114, | Jul 21 1993 | Xerox Corporation | User interface having click-through tools that can be composed with other tools |
5645077, | Jun 16 1994 | Massachusetts Institute of Technology | Inertial orientation tracker apparatus having automatic drift compensation for tracking human head and other similarly sized body |
5661632, | Jan 04 1994 | Dell USA, L.P. | Hand held computer with dual display screen orientation capability controlled by toggle switches having first and second non-momentary positions |
5666499, | Aug 04 1995 | AUTODESK CANADA CO | Clickaround tool-based graphical interface with two cursors |
5675746, | Sep 30 1992 | MARSHALL, CHARLES C, MR ; MAXUS STRATEGIC SYSTEMS INC | Virtual reality generator for use with financial information |
5683297, | Dec 16 1994 | Head mounted modular electronic game system | |
5686940, | Dec 24 1993 | Rohm Co., Ltd. | Display apparatus |
5689287, | Oct 27 1993 | Xerox Corporation | Context-preserving display system using a perspective sheet |
5689619, | Aug 09 1996 | The United States of America as represented by the Secretary of the Army | Eyetracker control of heads-up displays |
5689667, | Jun 06 1995 | AUTODESK CANADA CO | Methods and system of controlling menus with radial and linear portions |
5734421, | May 30 1995 | MAGUIRE, SUSAN C | Apparatus for inducing attitudinal head movements for passive virtual reality |
5742264, | Jan 24 1995 | Matsushita Electric Industrial Co., Ltd. | Head-mounted display |
5777715, | Jan 21 1997 | ALLEN VISION SYSTEMS, INC , A CORPORATION OF PENNSYLVANIA | Low vision rehabilitation system |
5790769, | Aug 04 1995 | AUTODESK CANADA CO | System for editing time-based temporal digital media including a pointing device toggling between temporal and translation-rotation modes |
5835077, | Jan 13 1995 | Rosemount Aerospace Inc | Computer control device |
5844544, | Jun 17 1994 | H K EYECAN LTD | Visual communications apparatus employing eye-position monitoring |
5844824, | Oct 02 1995 | RPX Corporation | Hands-free, portable computer and system |
5923307, | Jan 27 1997 | Microsoft Technology Licensing, LLC | Logical monitor configuration in a multiple monitor environment |
5926178, | Jun 06 1995 | AUTODESK CANADA CO | Display and control of menus with radial and linear portions |
5959605, | Nov 22 1995 | Picker International, Inc. | Video magnifier |
5973669, | Aug 22 1996 | AUTODESK, Inc | Temporal data control system |
5977935, | Aug 12 1993 | Seiko Epson Corporation | Head-mounted image display device and data processing apparatus including the same |
5991085, | Apr 21 1995 | i-O Display Systems LLC | Head-mounted personal visual display apparatus with image generator and holder |
6005482, | Sep 17 1998 | Xerox Corporation | Surface mounted information collage |
6061064, | Aug 31 1993 | Sun Microsystems, Inc. | System and method for providing and using a computer user interface with a view space having discrete portions |
6084556, | Jan 21 1999 | Meta Platforms, Inc | Virtual computer monitor |
6115025, | Sep 30 1997 | AUTODESK, Inc | System for maintaining orientation of a user interface as a display changes orientation |
6115028, | Aug 22 1996 | AUTODESK, Inc | Three dimensional input system using tilt |
6118427, | Apr 18 1996 | AUTODESK CANADA CO | Graphical user interface with optimal transparency thresholds for maximizing user performance and system efficiency |
6127990, | Nov 28 1995 | Rembrandt Portable Display Technologies, LP | Wearable display and methods for controlling same |
6148271, | Jan 14 1998 | Under Armour, Inc | Speed, spin rate, and curve measuring device |
6151563, | Jul 01 1999 | Under Armour, Inc | Speed, spin rate, and curve measuring device using magnetic field sensors |
6184847, | Sep 22 1998 | Facebook, Inc | Intuitive control of portable data displays |
6184859, | Apr 21 1995 | Sony Corporation | Picture display apparatus |
6292158, | May 08 1997 | Shimadzu Corporation | Display system |
6353436, | Aug 31 1993 | Sun Microsystems, Inc. | Graphical user interface |
6359603, | Nov 18 1995 | Meta Platforms, Inc | Portable display and methods of controlling same |
6361507, | Jun 16 1994 | Massachusetts Institute of Technology | Inertial orientation tracker having gradual automatic drift compensation for tracking human head and other similarly sized body |
6445364, | Nov 28 1995 | Meta Platforms, Inc | Portable game display and method for controlling same |
6457024, | Jul 18 1991 | Wearable hypermedium system | |
6590583, | May 14 1996 | Intellectual Ventures I LLC | Method for context-preserving magnification of digital image regions |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jan 18 2000 | FATEH, SINA | VEGA VISTA, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 030876 | /0458 | |
Jan 18 2000 | FLACK, JAMES F | VEGA VISTA, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 030876 | /0458 | |
Sep 05 2000 | ZWERN, ARTHUR L | VEGA VISTA, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 030876 | /0458 | |
Oct 18 2007 | VEGA VISTA, INC | Rembrandt Technologies, LP | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 030879 | /0322 | |
Oct 06 2009 | Rembrandt Portable Display Technologies, LP | (assignment on the face of the patent) | / | |||
Nov 09 2010 | Rembrandt Technologies, LP | Rembrandt Portable Display Technologies, LP | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 025348 | /0041 | |
Mar 29 2012 | Rembrandt Portable Display Technologies, LP | VEGA VISTA, INC | PATENT ACQUISITION AGREEMENT | 028466 | /0229 | |
Mar 29 2012 | Rembrandt Portable Display Technologies, LP | VEGA VISTA, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 030469 | /0811 | |
Jul 24 2014 | VEGA VISTA, INC | Facebook, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 042803 | /0921 | |
Oct 28 2021 | Facebook, Inc | Meta Platforms, Inc | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 058897 | /0824 |
Date | Maintenance Fee Events |
Sep 17 2012 | REM: Maintenance Fee Reminder Mailed. |
Feb 03 2013 | EXPX: Patent Reinstated After Maintenance Fee Payment Confirmed. |
Feb 27 2013 | PMFP: Petition Related to Maintenance Fees Filed. |
Jun 04 2013 | M2553: Payment of Maintenance Fee, 12th Yr, Small Entity. |
Jun 06 2013 | PMFG: Petition Related to Maintenance Fees Granted. |
Aug 04 2017 | ASPN: Payor Number Assigned. |
Date | Maintenance Schedule |
May 10 2014 | 4 years fee payment window open |
Nov 10 2014 | 6 months grace period start (w surcharge) |
May 10 2015 | patent expiry (for year 4) |
May 10 2017 | 2 years to revive unintentionally abandoned end. (for year 4) |
May 10 2018 | 8 years fee payment window open |
Nov 10 2018 | 6 months grace period start (w surcharge) |
May 10 2019 | patent expiry (for year 8) |
May 10 2021 | 2 years to revive unintentionally abandoned end. (for year 8) |
May 10 2022 | 12 years fee payment window open |
Nov 10 2022 | 6 months grace period start (w surcharge) |
May 10 2023 | patent expiry (for year 12) |
May 10 2025 | 2 years to revive unintentionally abandoned end. (for year 12) |