Method and apparatus for providing translucent images on a computer display

Abstract

A method and apparatus is described for producing a translucent image over a base image created on the display screen of a computer system by a selected first application program, and conducting image operations either on the base image created by the selected application program with reference to the translucent image produced, or conducting image operations on the translucent image with reference to the base image of the first application program. The first application program runs on a central processing unit (CPU) of a computer system to produce a base image, and another application program referred to as the overlay program is run to produce the translucent image such that portions of the base image which are overlapped by the overlay image are at least partially visible through the translucent image. There is also a mechanism for blending the first video data and the second video data to produce a blended image on the screen assembly.

Description

CROSS-REFERENCE TO FIG. 18 is a view of a Macintosh computer screen showing a desktop, a window produced by an application program called “AppleShare” and a utility program known as “PenBoard”;

FIG. 19 illustrates a non-transparent overlay which mostly obscures the desktop and window of the AppleShare application program;

FIG. 20 illustrates the overlay keyboard after it has been made transparent by the method and apparatus of the present invention;

FIGS. 21a-21c illustrate the entry of data to the active window of the AppleShare program;

FIG. 22 is a diagram illustrating the “Display an Overlay Image” step 138 of FIG. 6B;

FIG. 23 illustrates an alternate embodiment of the “Display an Overlay Image” step 138 of FIG. 6B;

FIG. 24 illustrates the operation of the “Blending Engine” 1190 of FIG. 23;

FIG. 25 illustrates a video driver circuitry of a prior art Macintosh computer system produced by Apple Computer, Inc. of Cupertino, Calif.; and

FIG. 26 illustrates video driver circuitry in accordance with the present invention which provides overlay VRAM and blending capabilities. In FIG. 18, a screen 1040 of a Macintosh computer system made by Apple Computer, Inc., of Cupertino. Calif., includes a desktop image 1042 produced by a Macintosh operating system, a window 1044 produced by a “AppleShare” application program made by Apple Computer, Inc., and a palette 1046 produced by a small application program or “utility” known as “PenBoard” made by Apple Computer, Inc. The desktop 1042, which includes a menu bar 1048 and a desk area 1050, often displays a number of icons 1052, 1054 and 1056, which represent different objects or functions. For example, the icon 1052 represents a hard disk drive; icon 1054 represents the “trash can” in which files can be deleted; and icon 1056 represents a folder which can contain applications and documents of various types. The menu bar 1048 preferably includes a number of labels 1058, 1060, and 1062 for pull-down menus, as is well known to Macintosh users.

As mentioned previously, the desktop 1042 is created by the operating system (sometimes referred to as the “Finder”). The Finder can be considered to be a specialized form of application program which displays an image on the entirety of the screen 1040. In other words, the “window” size of the desktop 1042 is the same size as the screen 1040. The application program AppleShare which creates the window 1044 typically does not take over the entire screen 1040. Similarly, the palette 1046(which is just a specialized form of window) is produced by the PenBoard application, and does not occupy the entire space of the screen 1040.

As is apparent by studying FIG. 18, the screen 1040 can quickly become occupied with icons, windows and palettes. This is not a major problem in traditional computer systems wherein the primary forms of input comprise keyboards and pointer devices, such a mice. However, in the pen computer systems where these more traditional forms of input devices are not always available, the limitations of screen size becomes readily apparent.

In FIG. 19, a keyboard image 1064 has been provided on screen 1040 to aid in the input of data to the AppleShare application program described previously. Preferably, this keyboard image 1064 is provided by dragging a keyboard icon 1066 off of the PenBoard palette 1046 in a fashion more fully described in copending U.S. patent application Ser. No. 08/060,458, filed May 10, 1993, on behalf of Gough et al., entitled “Method and Apparatus for Interfacing With a Computer System”, and assigned to the assignee of the present application, the disclosure of which is hereby incorporated herein by reference in its entirety. As can be seen in this FIG. 19, the keyboard image 1064 completely obscures the icons 1052, 1054 and 1056 of FIG. 18, and almost totally obscures the window 1044 of the AppleShare application program. Information can be entered into the window 1044 of the application program from the keyboard image 1064 by “tapping” on a “key” with the stylus 38. For example, arrow 1068 on the keyboard image 1064 represents the “tapping” on the key “R” with the stylus 38. This tapping action will send a “R” to be displayed in the window 1044 of the AppleShare application just as if a “R” had been typed on a physical keyboard. Again, the functioning of the keyboard image 1064 is discussed in the aforementioned copending U.S. patent application of Gough et al.

While the keyboard image 1064 can be used to input data into a currently active application program (such as AppleShare), the keyboard image prevents any user feedback of the information being entered into application windows obscured by the keyboard image. Therefore, it is difficult for the user to determine whether data has been properly entered into the application program. This, in turn, slows down the data entry process, and greatly increases the chances for errors.

The present invention solves this problem, as illustrated in FIG. 20. A user taps on a “transparency” icon 1069 on the keyboard image 1064 of FIG. 19 with the stylus 38 to cause the keyboard 1064 to become partially transparent or “translucent.” By “translucent” it is meant herein that the overlay image can be seen, but it can also be seen through. Tapping on the transparency icon 1069 of the keyboard image 1064′ of FIG. 20 would cause the “solid” keyboard image 1064 of FIG. 19 to reappear.

As can be seen, the translucent keyboard image 1064′ allows the window 1044 and icons 1052, 1054, and 1056, to be seen through the translucent keyboard image 1064′. In other words, portions of base images which are overlapped by the keyboard image 1064′, can still be seen (with some loss in resolution) through the translucent keyboard image 1064′.

The functioning of the keyboard image 1064′ will be explained in greater detail with reference to FIGS. 21a-21c. In FIG. 21a, the stylus 38 is used to “tap” on the “r” key as indicated by the arrow 1068 and the shading of the “r” key. The keyboard image 1064′ “intercepts” the tap 1068 which would otherwise fall on the window 1044, and, instead causes a “r” to be sent to the AppleShare program and be displayed in a password field of the window 1044. (Actually, AppleShare would display a “bullet” instead of the “r” to maintain the security of the password, but it will be assumed in this example that the typed password will remain visible). The “r” within the password field of window 1044 can be seen through the translucent window 1064′ in this figure. In FIG. 21b, second tap 1068 on the “i” key will cause the keyboard image 1064′ to “intercept” the tap which would otherwise fall on the window 1044, and to send a “i” character to the AppleShare application program which then displays an “i” after the “r” in the password field of window 1044. Next, as seen in FIG. 21c, the “p” key is tapped at 1068, causing the keyboard 1064′ to intercept the tap which would otherwise fall on the window 1044 and to send the “p” character to the AppleShare program which displays the character in the password field after the character “r” and “i.” Other characters and control characters (such as the “return” button 1070) can be sent to the application program controlling window 1044 in a similar fashion.

It will be apparent with a study of FIGS. 20 and 21a-21c that the translucent keyboard image 1064′ is a distinctly superior user interface for situations in which screen area is at a premium. Since images “beneath” the translucent keyboard image 1064′ can be seen through the keyboard image, the user has immediate feedback as to the accuracy of his or her input to the active application program. For example, if a key were “tapped” in error, the backspace key 1072 can be tapped on the translucent keyboard 1064′ so that the correct character can be reentered. The translucent keyboard 1064′ therefore effectively expands the useful area of screen 1040 by providing multiple, usable, overlapped images.

A preferred method in accordance with the present invention for implementing the process 133 on a Macintosh computer system is illustrated with reference to FIG. 22. The illustrated method of FIG. 22 is fairly specific to the Macintosh computer system. It will therefore be apparent to those skilled in the art that when the process 133 is implemented on other computer systems, such as MS-DOS compatible computer systems and UNIX computer systems, that the methodology of FIG. 22 will have to be modified. However, such modifications will become readily apparent to those skilled in the art after studying the following descriptions of how the process 133 is implemented on the Macintosh computer system.

In FIG. 22, the operating system, application program, overlay utility, system routines, etc., are shown in a somewhat hierarchical fashion. At the highest level is the operating system 1096 of the computer system 10 of FIG. 1. Running under the operating system 1096 is an application program 1098, such as the aforementioned AppleShare application program. Application program 1098, when it wants to open a window such as window 1044 of FIG. 18, calls a set of routines 1100 provided by the operating system 1096. More specifically, in the Macintosh operating system, application program 1098 calls a “New Window” routine 1102 which, in turn, calls a “Frame Rect” routine 1104. The Frame Rect routine uses a pointer table 1106 to call a “Shield Cursor” routine 1107 and a “Show Cursor” routine 1108. If the application program 1098 were running on system 1096 without the process 133 of the present invention, this would be the entirety of the calls to open up the window 1044 of FIG. 18. This process is extensively documented in the multi-volume reference set, Inside Macintosh, by C. Rose et al., Addison-Wesley Publishing Company, Inc., July 1988 and are well known to those skilled in the art of programming on the Macintosh operating system.

The implementation of computer implemented process 133 modifies this normal flow of routine calls in the following way. When the application program 1098 calls the New Window routine 1102 which calls the Frame Rect routine 1104, which attempts to call the Shield Cursor Routine, the Frame Rect routine 1104 instead calls a portion of the process of step 138 of FIG. 6B known as the Overlay Shield Cursor Patch 1110. This is accomplished by having the process 138 modify the pointer table 1106 such that when the Frame Rect routine 1104 is trying to call the Shield Cursor Routine 1107 it, instead, calls the Overlay Shield Cursor Patch 1110. After the Overlay Shield Cursor Patch 1110 completes its process, the Shield Cursor Routine 1107 is then called. As far as the Frame Rect routine 1104 is concerned, it does not know of the diversion of process control to the Overlay Shield Cursor Patch process 1110, and instead believes that it directly called the Shield Cursor Routine 1107.

The process step 138 of FIG. 6B similarly “tricks” the Frame Rect routine 1104 when it attempts to call the Show Cursor Routine 1108. In that instance, when the Frame Rect routine 1104 goes to the pointer table 1106 in an attempt to call the Show Cursor Routine 1108, process control is instead diverted to a process 1112 known as “Overlay Show Cursor Patch”. The Overlay Show Cursor Patch process 1112 interacts with a Blending Engine process 1114 to blend a first screen image 1116 generated by the Macintosh operating system and the application program, with a second image 1118 (in this case, the keyboard image) to form the blended image 1120. The operation of the Blending Engine will be discussed in greater detail subsequently. After the completion of the blending process of 1114, the Overlay Show Cursor Patch process 1112 turns over process control to the “Show Cursor Routine” process 1108. Again, as far as the Frame Rect routine 1104 is concerned, it made a direct call to the “Show Cursor Routine” 1108 and was ignorant of the diversion of the process control to the Overlay Show Cursor Patch 1112 and the Blending Engine 1114.

FIG. 23 illustrates an alternate embodiment of the present invention which has been optimized for screen-writing speed. While the process of FIG. 22 works very well, it requires that the entirety of the base screen 1116 be rewritten whenever the blended image 1120 is to be refreshed. The alternative process of FIG. 23 only refreshes the portions of the blended image that needs to be refreshed, thereby greatly increasing the writing speed to the screen 1040.

Much of the operation of the process illustrated in FIG. 23 is similar to that described in FIG. 22. An operating system 1172 supports an application program 1174 which, when it wants to open a window, calls a set of routines 1176 including a “New Window routine” 1178 and Frame Rect routine 1180. The Frame Rect routine 1180 then, as before, attempts to first call the Shield Cursor Routine 1182 first and then the Show Cursor Routine 1184. Again, as before, the pointer table is modified such that when the Frame Rect routine tries to call the Shield Cursor Routine 1182, it instead calls the Overlay Shield Cursor Patch 1186 of the present invention, and when the Frame Rect routine 1180 attempts to call the Show Cursor Routine 1184 it instead calls the Overlay Show Cursor Patch 1188. The Overlay Show Cursor Patch calls a Blending Engine 1190 which blends a partial base image 1192 with an overlay image 1194 to create a blended image 1196.

The system 1172, as part of its functioning, will make periodic calls to various system task processes 1198. The system task 1198 performs such functions as execute “Device Driver Code” and “Desk Accessory Code.” The process of the present invention opportunistically takes advantage of these periodic system task calls by modifying a pointer table 1200 to turn over process control to an Overlay System Task Patch 1202. This Overlay System Task Patch, along with the Overlay Shield Cursor Patch 1186, the Overlay Show Cursor Patch 1188, and the Blending Engine 1190 comprise the overlay utility 133 of FIGS. 6A and 6B in this second preferred embodiment.

FIG. 24 is used to illustrate the operation of the Blending Engine 1190 of FIG. 23 in greater detail. The process 138 of FIG. 6B remaps certain pages of VRAM to the RAM screen buffer when an overlay image contains objects that overlap these pages. The RAM overlay screen buffer 1194 is then merged with the RAM screen buffer 1192 in the Blending Engine 1190 by a process similar to that previously described and inserts the blended image into a “hole” 1204 of the VRAM screen buffer 1196. The portions 1206 and 1208 of the VRAM screen buffer remain the VRAM since the overlay image of the present invention does not overlap pages comprising these portions of the screen.

Since portions 1206 and 1208 are pages of VRAM screen buffer memory which are not overlapped, at least in part, by an overlay image of the present invention, these portions 1206 and 1208 can remain in VRAM screen buffer. VRAM screen buffer is much faster memory for video purposes than the RAM screen buffer 1192. Also, changes made to the RAM screen buffer 1192 or to the RAM overlay screen buffer 1194 that do not cause a change in portions 1206 and 1208 do not require that the system blend the portions 1206 and 1208. The combination of these factors substantially increase the blending speed of the VRAM screen buffer and therefore of the display on screen 1040.

FIGS. 25 and 26 are used to illustrate an alternate embodiment of the present invention wherein the blending of the base image and the overlay image are performed in the video driver hardware rather than within a computer implemented process on the CPU. In FIG. 25, a prior art video driver system of a Macintosh computer system is illustrated. In this prior art example, the video driver circuit 1302 is coupled to an address bus 1304 and a data bus 1306 connected to a Motorola 68030 microprocessor. The video driver circuit 1302 includes a color screen controller CSC 1307, and two banks of VRAM 1308 and 1310. The CSC 1307 produces LCD control and data on a bus 1312 which control a black and white or color liquid crystal display (LCD). For example, the video driver circuit 1302 can drive an Esher LCD circuit for a 640 by 400 bit display, with eight bits of information per pixel.

In FIG. 26, a modified video driver circuit 1302′ is coupled to the same Motorola 68030 address bus 1304 and data bus 1306, and includes the same CSC 1307, VRAM 1308, and VRAM 1310. However, the data and address connections have been modified as indicated. In this implementation, data from the screen buffer and the overlay screen buffer are input into the VRAM of modified video driver circuit 1302′, and combined therein to provide LCD control and blended data on the bus 1312. Again, the video driver circuit 1302′ can control a black and white or color LCD, except this time instead of having eight bits per pixel, there are four bits allocated to the base image and four bits allocated to the overlay image. A color look-up table (CLUT)—not shown—of CSC 1307 is loaded with 256 entries which detail each possible combination of bits from the 4 bit screen and the 4 bit overlay, and what the resultant blended value is. The color capability of the CSC 1307 is therefore no longer used for color look-up, and is instead used for the blending values. This technique makes it possible to use off-the-shelf integrated circuits, such as the CSC 1307 which is available from Chips & Technologies, Inc. of San Jose, Calif., to perform an entirely new operation.

In summary, the method of the invention includes establishing translucent images on a display screen including displaying a translucent images and conducting image operations enabled by the translucent image. Image operations can be any kind of operation conducted on an image or window. Drawing an image, placing an image, or for that matter modifying, moving, expanding, or changing an image or a window, are considered to be image operations. A reference image could be provided by a selected first application program. The translucent image could be produced by a selected second application program. The user is thus enabled to make sketches on the translucent image or window based upon what he or she sees on the base image produced by the first application program. This is made possible without any direct intervention in the operations of the first application program. In short, the features of the first application program are advantageously employed, without any modification of the first application program itself. The technical enablement of this cooperative screen is found in a feature of the invention according to which the second application program intercepts certain screen inputs of the first application program and uses them to supply the screen input needed as to the second application program.

The image operations enabled by the concurrent interoperability of the two applications can be implemented by user selected intervention at any of a number of screen operational levels. The base image or window is considered to operate at a lower level, or below the level of the translucent image or window. Thus, the translucent image or window is known as the “overlay” image or window. Typically, the cursor is active at the particular level at which the user can operate. In any case, according to the invention, it may be useful to operate at either the base level, i.e., the level of the base image or window, or at the translucent overlay level. In other words, user input is permitted at either the base image or the translucent image. By a particular user input with respect to an image, the user implements a selected computer implemented process and the process receives screen inputs which contact or are otherwise associated with a particular window as the computer implemented process is effective for processing the screen inputs. These various inputs are controllable selectively by the user, in that users can take specific actions to determine which of the levels will be active for them. This can, for example, be accomplished by action of clicking or activating a pen or stylus or by another well known action users are considered capable of actuating. A particular window just opened is automatically active, as the newest window created or activated. Another window or image can be activated merely by user selection in positioning the cursor over the window or image and clicking on the mouse, trackball or another applicable interface device.

While this invention has been described in terms of several preferred embodiments, it is contemplated that many alterations, permutations, and equivalents will be apparent to those skilled in the art. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.

Claims

1. A method for establishing a translucent window having a translucent window background and a translucent window frame on a display screen of a computer system, comprising the steps of:

displaying a translucent window on a display screen such that a base window can be seen through said translucent window, and

conducting image operations on at least one of said translucent window and said base window.

2. A method as recited in claim 1 wherein said base window is produced on said display screen by a first computer implemented process running on said computer system, and said translucent window is produced by a second computer implemented process running on said computer system.

3. A method as recited in claim 2 wherein said second computer implemented process receives screen inputs which are associated with said translucent window and processes said screen inputs.

4. A method as recited in claim 2, wherein said second computer implemented process receives screen inputs which are physically applied in connection with said translucent window and processes said screen inputs.

5. A method as recited in claim 2, wherein said second computer implemented process receives screen inputs which make image contact with said translucent window and processes said screen inputs.

6. A method as recited in claim 1 wherein image operations are conducted with respect to said translucent window.

7. A method as recited in claim 1 wherein image operations are conducted below said translucent window.

8. A method as recited in claim 1 wherein image operations are conducted in connection with windows referenced by a cursor indication.

9. A method for displaying images on a display screen of a computer system, comprising the steps of:

displaying a base image on a display screen of a computer system; and

displaying a translucent image on said screen such that portions of said base image which are covered by said translucent image are at least partially visible through said translucent image.

10. A method as recited in claim 9 wherein said base image and said translucent image are produced by independent computer implemented processes.

11. A method as recited in claim 9 wherein said base image is active to receive user inputs.

12. A method as recited in claim 9 wherein said translucent image is active to receive user inputs.

13. A method as recited in claim 12 wherein said translucent image is made active by user action.

14. A method as recited in claim 12 wherein said translucent image is made active by positioning the cursor at a portion of the translucent image and conducting a select action.

15. A method as recited in claim 12 wherein said translucent image is made active by clicking a mouse when the cursor is over a portion of the translucent image.

16. A method as recited in claim 9 wherein said translucent image and said base image are selectably active to receive user inputs.

17. A method for displaying images on a display screen of a computer system comprising the steps of:

running an application program on the central processing unit (CPU) of a computer system to produce a base image on a display screen coupled to said CPU; and

running an overlay program on said CPU to produce a translucent image on said display screen such that portions of said base image are overlapped by said translucent image and are at least partially visible through said translucent image.

18. A method as recited in claim 17 wherein said step of running an overlay program comprises the steps of:

displaying a translucent image on said display screen;

intercepting screen inputs which contact the translucent image;

processing said intercepted screen inputs in said CPU; and

updating said application program based upon said process screen inputs.

19. A method as recited in claim 18 wherein said step of displaying a translucent image comprises the step of blending a translucent image with said base image.

20. A method as recited in claim 19 wherein said step of blending is accomplished in said CPU.

21. A method as recited in claim 19 wherein said step of blending is accomplished externally to said CPU.

22. A method as recited in claim 18 wherein said step of processing said intercepted screen inputs includes redirecting at least one page of memory within the memory management means of said computer system.

23. A method of performing image operations in a computer system having a display screen, including the steps of:

presenting a first selected image with respect to which image operations are desired, and

producing a translucent image effective for overlapping at least a portion of said first selected image.

24. A method according to claim 23 wherein said first selected image contains features of interest, and image operations are conducted on said translucent image with respect to said features of interest.

25. A method according to claim 23 wherein said translucent image contains features of interest, and image operations are conducted with respect to said first selected image based upon said features of interest.

26. A computer system comprising:

a central processing unit (CPU);

screen means for displaying images, said screen means being coupled to said CPU;

display means coupled to said screen means for displaying a translucent image on said screen means; and

means for conducting image operations on a region including the level of a translucent image produced by said display means and the level beneath the translucent image.

27. A computer system according to claim 26 wherein said means for conducting image operations performs image operations with reference to a translucent image on said screen means.

28. A computer system according to claim 26 wherein said means for conducting image operations performs image operations selectably with reference to a translucent image on said screen means and below the level of a translucent image on said screen means.

29. A method for displaying images on a display screen of an electronic device, comprising the steps of:

displaying a base image on a display screen of the electronic device; and

displaying a translucent image on said screen such that portions of said base image which are covered by said translucent image are at least partially visible through said translucent image, wherein said translucent image and said base image are selectably active to receive user input and the base image remains at least partially covered by said translucent image even when selected.

30. A method as recited in claim 29, wherein the electronic device is a handheld device.

31. A method of performing image operations in an electronic device, including the steps of:

presenting a first selected image with respect to which image operations are desired,

producing a translucent image effective for overlapping at least a portion of said first selected image, wherein said translucent image contains at least one feature of interest, and

conducting an image operation on said first selected image using said feature of interest while the translucent image overlaps at least a portion of the first selected image.

32. A method as recited in claim 31, wherein the electronic device is a handheld device.

33. A method for displaying images on a display screen of an electronic device, comprising the steps of:

displaying a base image on a display screen of said electronic device;

displaying a translucent image on said screen such that portions of said base image which are covered by said translucent image are at least partially visible through said translucent image; and

receiving input in said displayed base image while said base image remains at least partially covered by said translucent image.

34. A method as recited in claim 33, wherein said base image is active to receive user inputs.

35. A method as recited in claim 33, wherein the electronic device is a handheld device.