Methods, computers, and computer readable media are disclosed for recording user-driven events within an application running on a computer that utilizes a graphical user interface. changes of focus of the application are detected and entries for the resulting focus are logged. While the changes of focus are being detected, the application is monitored for a window message that signifies user activity. When a window message occurs, the activity of the window message is used to specify the user-driven event. A focus of an entry previously logged may also be obtained and used to specify the user-driven event, such as a mouse click on a specific tool bar button of the graphical user interface. A series of user-driven events can be recorded up to the occurrence of an application malfunction, and then the recorded events can be followed by a user to reproduce the malfunction and begin debugging the application.
|
1. A method of recording user-driven events occurring for a computer running an application within a window of a graphical user interface, comprising:
detecting each change of focus of the application within the graphical user interface;
logging an entry for a focus that results from each change of focus;
while detecting each change of focus, monitoring for a window message signifying user activity;
when a window message occurs, obtaining the focus of an entry previously logged; and
recording the obtained focus in association with the user activity of the window message to specify the user-driven event.
9. A computer that records user-driven events occurring for an application within a window of a graphical user interface, comprising:
a memory;
a processor that executes the application and the graphical user interface, the processor being configured to:
detect changes of focus of the application and record a focus resulting from each change as an entry to memory,
monitor for a window message specifying user activity while detecting the changes of focus,
when a window message occurs, specify the user-driven event by recording the user activity of the message and also recording a focus of an entry stored in memory.
15. A computer readable medium that contains computer executable instructions for performing the steps of:
detecting a change of focus of an application within a graphical user interface;
logging a focus resulting from each change of focus that occurs as an entry to memory;
while detecting the changes of focus, monitoring for window messages signifying user activity;
when a window message occurs, determining whether the window message indicates a mouse click or a keyboard type;
when a mouse click is indicated, obtaining a focus of an entry previously logged and recording the obtained focus in association with the user activity of the window message to specify the user-driven event.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
8. The method of
10. The computer of
11. The computer of
12. The computer of
13. The computer of
14. The computer of
16. The computer readable medium of
17. The computer readable medium of
18. The computer readable medium of
19. The computer readable medium of
20. The computer readable medium of
|
The present invention is related to recording user-driven events in a computer application. More particularly, the present invention is related to recording user-driven events occurring within a graphical user interface of the computer application.
Computer application programs, as they are being developed and tested, often suffer from malfunctions known as bugs. A bug in an application program may cause the application to behave incorrectly or to stop responding altogether. The efficiency of resolving the bug is improved if the bug can be reproduced. Reproducing the bug requires that the user know the sequence of user-driven events that occurred leading up to the occurrence of the bug. Sometimes, a bug results from the culmination of many events happening in a particular sequence and can be nearly impossible to remember.
Furthermore, the person experiencing the bug may not be the same individual who will attempt to debug the application. Therefore, the person experiencing the bug must be able to convey the sequence of events to the person who will attempt to debug the application. Conveying the sequence of events requires not only that the person experiencing the bug recall each of the events, but each event must be conveyed accurately. If the person forgets a particular user-driven event, forgets the particular order of events, or misunderstands a user-driven event that occurred when experiencing the bug, then the user-driven events will not be adequately reproducible. As a result, the bug may not be easily detected and corrected. Thus, automatically recording the user- driven events of an application is necessary to properly reproduce the bug.
However, for applications that operate within a graphical user interface (“GUI”), many of the user-driven events occur through use of a mouse in addition to use of a keyboard. User-driven events include mouse clicks on particular objects of the GUI. Merely recording the X and Y coordinates of a mouse pointer when the click occurs is not helpful because one cannot determine from the X and Y coordinates the object that was actually clicked due to variations in window size, placement, object arrangement, and other factors. Therefore, user-driven events within GUIs are not adequately recorded.
The present invention records user-driven events within an application of a computer by detecting changes in a focus of the application while monitoring for window messages from the application window to the operating system of the computer. The focus of the application includes where a mouse pointer is located on a graphical user interface of the application, such as a focus on a particular button of a toolbar that causes the button to have a raised appearance. The window message specifies user activity such as a click of a mouse button and pressing of keys on the keyboard. The focus that results from each change of focus is logged. When a window message occurs, a focus that has been logged and the user activity of the window message can be recorded to specify the user-driven event. The user-driven event, such as typing of particular keys or a mouse click at a particular focus can be displayed on the screen and/or saved to a file and later used as an instruction for recreating the bug.
A computer that records user-driven events includes a memory and a processor. The processor executes the application and the graphical user interface provided to the user through a display device. The processor detects the changes of focus of the application and records a focus resulting from each change as an entry to memory. The processor also monitors for a window message specifying user activity while detecting the changes of focus. When a window message occurs, the processor specifies the user-driven event by recording the user activity of the message and also recording a focus of an entry stored in the memory.
A computer readable medium specifies steps to be executed by a computer to record the user-driven events. The steps include detecting a change of focus of an application within a graphical user interface and logging each change of focus that occurs as an entry to memory. While detecting the changes of focus, window messages signifying user activity are monitored, and when a window message occurs, it is determined whether the window message indicates a mouse click or a keyboard type. When a mouse click is indicated, a focus of an entry previously logged is obtained and recorded in association with the user activity of the window message to specify the user-driven event.
The present invention may be more clearly understood and appreciated from a review of the following detailed description and by reference to the drawings and claims.
Embodiments of the present invention provide a method, computer system, and computer-readable medium for recording user-driven events occurring within an application. In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration of specific embodiments or examples. These embodiments may be combined, other embodiments may be utilized, and structural changes may be made without departing from the spirit and scope of the present invention. The following detailed description is, therefore, not be taken in a limiting sense, and the scope of the present invention is defined by the appended claims and their equivalents.
Referring now to the drawings, in which like numerals represent like elements through the several figures, aspects of the present invention and the exemplary operating environment will be described.
Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention as applied to the personal computer of
The mass storage device 114 is connected to the CPU 104 through a mass storage controller (not shown) connected to the bus 112. The mass storage device 114 and its associated computer-readable media, provide non-volatile storage for the personal computer 102. Although the description of computer-readable media contained herein refers to a mass storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available media that can be accessed by the personal computer 102.
By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the computer.
According to various embodiments of the invention, the personal computer 102 may operate in a networked environment using logical connections to remote computers through a TCP/IP network 118, such as the Internet. The personal computer 102 may connect to the TCP/IP network 118 through a network interface unit 120 connected to the bus 112. It should be appreciated that the network interface unit 120 may also be utilized to connect to other types of networks and remote computer systems. The personal computer 102 may also include an input/output controller 122 for receiving and processing input from a number of devices, including a keyboard or mouse (not shown). Similarly, an input/output controller 122 may provide output to a display screen 124, a printer, or other type of output device.
As mentioned briefly above, a number of program modules and data files may be stored in the mass storage device 114 and RAM 108 of the personal computer 102, including an operating system 116 suitable for controlling the operation of a stand-alone or networked personal computer. The mass storage device 114 and RAM 108 may also store one or more application programs such as the application 126 and recorder application 128.
Embodiments of the present invention provide program modules for use in conjunction with the recorder application program 128. The program modules implement logical operations to record the user-driven events within the application program 126 being used. Various embodiments of the logical operations of the program modules for the recorder application program 126 are discussed below with reference to
The logical operations of the executable begin at signal operation 202 where the CPU 104 awaits a user activity signal that is provided from the logical operations of
When a keyboard type is detected instead of a mouse click, then record operation 208 records the keyboard type as a user-driven event. As discussed, recording the user-driven event may involve one or more techniques, such as displaying a textual description of the keyboard type on the display screen, saving a description to an electronic file, and/or printing the textual description via a printer. Each character that is typed may be placed on the same line of the textual description as shown in the screenshots until the typed key is an <Enter> key or unless there is a control key such as <Backspace> or <Ctrl>+“A”. After each keystroke is recorded, operational flow returns to signal operation 202.
When a mouse click is detected instead of a keyboard type, then operational flow transitions to stack operation 210. At stack operation 210, the CPU 104 references the stack 130 in storage 114 to obtain a time stamp for the last entry in the stack and a time stamp for the previous entry. As discussed below, the time stamp indicates the time when the focus of an entry was recorded to the stack upon the change of focus that occurs in the application program 126. The spacing in time of the last entry from the previous entry is determined by comparison of the time stamp of the last entry to the time stamp of the previous entry.
Query operation 212 detects whether the spacing in time of the last entry in the stack from the previous entry is greater than a pre-defined interval. It has been found that in some applications utilizing a GUI interface, there are focus entries that are recorded to the stack after the correct focus by the time the user performs the mouse click. It has also been found that a spacing of 0.3 seconds is a good indication that the last entry most likely contains the correct focus that should be recorded. If the last entry is spaced in time less than 0.3 seconds from the previous entry, then a previous entry may contain the correct focus instead. Therefore, for a particular application, either the focus of the last entry or a focus of a previous entry may be the correct focus to be recorded when the spacing is less than the pre-defined interval, such as less than 0.3 seconds.
When query operation 212 detects that the spacing is less than the pre-defined interval, then entry operation 216 determines which focus, either from the last entry or a preceding entry in the stack, has a lesser NULL setting or other indicative criteria specific to the application that signals which is most likely the correct focus. Once the entry with the correct focus is determined, record operation 218 records the user-driven event as a mouse click on that focus. When query operation 212 detects that the spacing is greater than or equal to the pre-defined interval, then record operation 214 records the user-driven event as a mouse click on the focus of the last entry in the stack. As discussed above, recording the user-driven event may involve displaying text describing the user-driven event on a display or printout and/or saving the description to an electronic file. After recording a mouse click event at record operation 214 or 218, a new stack is delineated at stack operation 220 and operational flow returns to signal operation 202.
Query operation 404 detects whether a change in focus has occurred. If not, the monitor operation 402 continues monitoring to detect a change. Once a change in focus is detected, log operation 406 logs the focus information into an entry in the stack 130 along with a time stamp from the system clock. The focus information received from the application program may include one or more types of data to describe the focus. The name of the control object where the mouse pointer is located may be provided, such as “Format”. Also, the type of control object may be provided such as [Menu Item]. Other pieces of information may also be included, such as an identification of the parent application, an indication of the coordinates of the mouse pointer, and an identification of the window having focus. Once the one or more types of data describing the focus are logged as an entry to the stack 130 with a time stamp, operational flow returns to monitor operation 402.
As shown in the screenshot 700 of
After typing this phrase but before moving the mouse pointer, the user types=rand( ) and presses the <Enter> key. The application 126 responds by inserting the text 706. The user-driven event is recorded as Keyboard type: =rand( )<Enter> as displayed at line 616 in screenshot 800 of
The user now wishes to select all of the text 706 so the user presses the combination of the <Ctrl> key and the <A> key. The user-driven event is recorded as Keyboard type: <Ctrl>+A as displayed at line 620. The user then moves the mouse pointer over a “Format” menu item 708 shown in screenshot 900 of
After clicking the “Format” menu item, the user moves the mouse pointer to a “Font” menu item provided in response to the mouse click. Therefore, the focus of the new stack 510 becomes the “Font” menu item at time 1:57.2. The user then clicks the mouse button to select the “Font” menu item. The user-driven event is recorded as Click[Menu item] “Font . . . ” at line 624, which specifies the activity of clicking along with the focus which is the “Font” menu item. The “Font” focus at this point is the last entry in the stack 510 and there is no other entry with a time stamp to compare to so it as the last entry can be used as the correct focus of the user-driven event. The window switches to the “Font” child window 902.
After clicking the “Font” menu item, the user moves the mouse pointer to a “Font Color” menu button 904 of the child window 902 as shown in screenshot 1000 of
The user-driven event is recorded as Click[Menu button] “Font Color” as displayed at line 626, rather than incorrectly specifying the X button as the focus, because the time stamp for the last entry occurred less than the pre-defined interval from the previous time stamp. In this example, the pre-defined interval is 0.3 seconds and the last entry occurred only 0.1 seconds after the previous entry, and the X menu button has a NULL setting as specified for the application that is greater than the NULL setting for the “Font Color” menu button. Therefore, the “Font Color” menu button is obtained from the stack 512 as the correct focus for the mouse click and is recorded as the user-driven event. Thereafter, the recording of user-driven events continues with a new stack 514 and so on until the user stops using the recording application 128 or until the application 126 malfunctions.
If the application 126 malfunctions, the user or another who wishes to reproduce the malfunction to debug the application 126 can follow the user-driven events specified in the window 606. The recorded user-driven events of the window 606 may be printed or electronically transferred as a file to a remote location, such as a database that maintains the user-driven events necessary to reproduce bugs for an application 126.
Thus, it can be seen from the discussion of the foregoing embodiments that user- driven events ocurring within an application may be recorded. These embodiments provide recording without access to source code of the applications so that any user can record the events. Furthermore, the user need not specially set up a computer, such as by utilizing implementation, debugging, and compiling software and settings for such recording but merely run the recorder application program 128. From these embodiments, it can also be seen that the recorder application 128 may be provided such that its operation does not overly tax system resources and does not slow system response to user actions.
Although the present invention has been described in connection with various exemplary embodiments, those of ordinary skill in the art will understand that many modifications can be made thereto within the scope of the claims that follow. Accordingly, it is not intended that the scope of the invention in any way be limited by the above description, but instead be determined entirely by reference to the claims that follow.
Patent | Priority | Assignee | Title |
10146662, | Dec 17 2013 | AIRBNB, INC | Recording GUI data |
10180980, | Mar 31 2004 | GOOGLE LLC | Methods and systems for eliminating duplicate events |
10423679, | Dec 31 2003 | GOOGLE LLC | Methods and systems for improving a search ranking using article information |
7490299, | Jun 30 2004 | GOOGLE LLC | System and method for handling unexpected focus change messages in a computing device |
7644314, | Mar 28 2006 | Microsoft Technology Licensing, LLC | Retroactive verbose logging |
7653721, | Oct 29 2004 | Oracle America, Inc | Mechanism for capturing high level events on user interface components |
7680809, | Mar 31 2004 | GOOGLE LLC | Profile based capture component |
7680888, | Mar 31 2004 | GOOGLE LLC | Methods and systems for processing instant messenger messages |
7711104, | Mar 31 2004 | AVAYA LLC | Multi-tasking tracking agent |
7725508, | Jun 30 2004 | GOOGLE LLC | Methods and systems for information capture and retrieval |
7770175, | Sep 26 2003 | AVAYA LLC | Method and apparatus for load balancing work on a network of servers based on the probability of being serviced within a service time goal |
7941439, | Mar 31 2004 | GOOGLE LLC | Methods and systems for information capture |
7949121, | Sep 27 2004 | AVAYA LLC | Method and apparatus for the simultaneous delivery of multiple contacts to an agent |
8005806, | Nov 15 2006 | Verizon Patent and Licensing Inc | System and method for information retrieval using context information |
8005943, | Oct 12 2005 | CA, INC | Performance monitoring of network applications |
8037046, | Jun 29 2007 | Microsoft Technology Licensing, LLC | Collecting and presenting temporal-based action information |
8056007, | Nov 15 2006 | R2 SOLUTIONS LLC | System and method for recognizing and storing information and associated context |
8094804, | Sep 26 2003 | AVAYA LLC | Method and apparatus for assessing the status of work waiting for service |
8099407, | Mar 31 2004 | GOOGLE LLC | Methods and systems for processing media files |
8161053, | Mar 31 2004 | GOOGLE LLC | Methods and systems for eliminating duplicate events |
8185524, | Dec 22 2006 | SALESFORCE, INC | Method and system for locating events in-context |
8196200, | Sep 28 2006 | CA, INC | Piggybacking malicious code blocker |
8234141, | Sep 27 2004 | AVAYA LLC | Dynamic work assignment strategies based on multiple aspects of agent proficiency |
8239528, | Oct 12 2005 | CA, INC | Performance monitoring of network applications |
8275839, | Mar 31 2004 | GOOGLE LLC | Methods and systems for processing email messages |
8306212, | Feb 19 2010 | AVAYA LLC | Time-based work assignments in automated contact distribution |
8346777, | Mar 31 2004 | GOOGLE LLC | Systems and methods for selectively storing event data |
8386728, | Mar 31 2004 | GOOGLE LLC | Methods and systems for prioritizing a crawl |
8448138, | Jan 15 2008 | Microsoft Technology Licensing, LLC | Recording user-driven events within a computing system |
8458215, | Nov 24 2003 | International Business Machines Corporation | Dynamic functional module availability |
8522257, | Nov 15 2006 | Verizon Patent and Licensing Inc | System and method for context information retrieval |
8631076, | Mar 31 2004 | GOOGLE LLC | Methods and systems for associating instant messenger events |
8667409, | Oct 31 2005 | SAP SE | Systems and methods for handling focus of application user interface elements |
8688491, | Sep 29 2005 | The MathWorks, Inc. | Testing and error reporting for on-demand software based marketing and sales |
8701002, | Apr 19 2011 | Autodesk, Inc. | Hierarchical display and navigation of document revision histories |
8701090, | Dec 31 2010 | Verizon Patent and Licensing Inc.; VERIZON PATENT AND LICENSING, INC | Graphical user interface testing systems and methods |
8738412, | Jul 13 2004 | AVAYA LLC | Method and apparatus for supporting individualized selection rules for resource allocation |
8751274, | Sep 26 2003 | AVAYA LLC | Method and apparatus for assessing the status of work waiting for service |
8812515, | Mar 31 2004 | GOOGLE LLC | Processing contact information |
8874525, | Apr 19 2011 | Autodesk, Inc. | Hierarchical display and navigation of document revision histories |
8891747, | Sep 26 2003 | AVAYA LLC | Method and apparatus for assessing the status of work waiting for service |
8954420, | Dec 31 2003 | GOOGLE LLC | Methods and systems for improving a search ranking using article information |
8984429, | Nov 14 2005 | Red Hat, Inc. | Searching desktop objects in view of time based statistics |
9025761, | Sep 26 2003 | AVAYA LLC | Method and apparatus for assessing the status of work waiting for service |
9189553, | Mar 31 2004 | GOOGLE LLC | Methods and systems for prioritizing a crawl |
9262446, | Dec 29 2005 | GOOGLE LLC | Dynamically ranking entries in a personal data book |
9311408, | Mar 31 2004 | GOOGLE LLC | Methods and systems for processing media files |
9542297, | Dec 17 2013 | AIRBNB, INC | Recording GUI data |
9836544, | Mar 31 2004 | GOOGLE LLC | Methods and systems for prioritizing a crawl |
9858173, | Dec 01 2011 | Microsoft Technology Licensing, LLC | Recording user-driven events within a computing system including vicinity searching |
D553626, | Oct 13 2005 | Arg Soft Intellectual Property Holdings, Limited | Computer network monitoring web interface for a computer screen |
D553628, | Oct 13 2005 | Arg Soft Intellectual Property Holdings, Limited | Computer network monitoring catalog master interface for a computer screen |
Patent | Priority | Assignee | Title |
5634019, | Apr 15 1994 | Microsoft Technology Licensing, LLC | Method and system for in-place interaction with contained objects |
6353452, | Oct 20 1997 | International Business Machines Corporation | Data item display method and device, and recording medium storing a program for controlling display of data item |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
May 24 2002 | CHANG, MICHAEL J | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 012996 | /0045 | |
May 24 2002 | Wen, Li | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 012996 | /0045 | |
Jun 05 2002 | Microsoft Corporation | (assignment on the face of the patent) | / | |||
Oct 14 2014 | Microsoft Corporation | Microsoft Technology Licensing, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 034766 | /0001 |
Date | Maintenance Fee Events |
Apr 22 2009 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Jul 05 2013 | REM: Maintenance Fee Reminder Mailed. |
Nov 22 2013 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Nov 22 2008 | 4 years fee payment window open |
May 22 2009 | 6 months grace period start (w surcharge) |
Nov 22 2009 | patent expiry (for year 4) |
Nov 22 2011 | 2 years to revive unintentionally abandoned end. (for year 4) |
Nov 22 2012 | 8 years fee payment window open |
May 22 2013 | 6 months grace period start (w surcharge) |
Nov 22 2013 | patent expiry (for year 8) |
Nov 22 2015 | 2 years to revive unintentionally abandoned end. (for year 8) |
Nov 22 2016 | 12 years fee payment window open |
May 22 2017 | 6 months grace period start (w surcharge) |
Nov 22 2017 | patent expiry (for year 12) |
Nov 22 2019 | 2 years to revive unintentionally abandoned end. (for year 12) |