A technique for synchronizing a visual browser and a voice browser. A visual browser is used to navigate through visual content, such as WML pages. During the navigation, the visual browser creates a historical record of events that have occurred during the navigation. The voice browser uses this historical record to navigate the content in the same manner as occurred on the visual browser, thereby synchronizing to a state equivalent to that of the visual browser. The creation of the historical record may be performed by using a script to trap events, where the script contains code that records the trapped events. The synchronization technique may be used with a multi-modal application that permits the mode of input/output (I/O) to be changed between visual and voice browsers. When the mode is changed from visual to voice, the record of events captured by the visual browser is provided to the voice browser, thereby allowing the I/O mode to change seamlessly from visual to voice. Likewise, the voice browser captures events which may be provided to the visual browser when the I/O mode is changed from voice to visual.
|
0. 32. A wireless computing device comprising a first memory and a first processor, the wireless computing device configured to:
execute a visual browser interacting with a content page, the visual browser stored in the memory and operating in the wireless computing device for interacting with the content page using one or more of user data and a first data set, wherein the visual browser displays the interaction through a visual display; and
execute, in a script engine, a script operable to record events generated during the interacting according to a user-specified one of a plurality of granularity levels, wherein each of said granularity levels represents a precision of browser inputs and outputs from a set of available precisions including a coarse level of precision and at least one finer level of precision, the script comprising a plurality of sets of instructions, the sets of instructions operable to create a record of a particular one of the events based on an event indicated by the visual browser, wherein the recorded events are operable to allow a second wireless computing device executing a voice browser to recreate a state of the visual browser, wherein the second wireless computing device comprises a second memory and a second processor, and wherein the voice browser is storable in the second memory and executable on the second processor for enabling interaction with the content page using an audio speaker and a microphone, wherein a state manager receives a record of events created by the script engine and provides the record of events to the voice browser, and wherein the state of the voice browser is synchronized with the state of the visual browser from the information received from the state manager.
0. 33. A wireless computing device comprising a first memory and a first processor, the wireless computing device configured to:
execute a voice browser interacting with a content page, wherein the voice browser is storable in the first memory and executable on the first processor for enabling interaction with the content page using an audio speaker and a microphone; and
execute, in a script engine, a script comprising a plurality of sets of instructions executable by the script engine, the script engine being configured to invoke a particular one of the sets of instructions based on an event indicated by the voice browser, wherein the particular one of the sets of instructions is invoked to create a record of a particular one of the events according to a user-specified one of a plurality of granularity levels, wherein each of said granularity levels represents a precision of browser inputs and outputs from a set of available precisions including a coarse level of precision and at least one finer level of precision, wherein the recorded event is transmitted via a state manager to a second wireless computing device executing a visual browser and recreating a state of the voice browser based on the recorded event, wherein the second wireless computing device comprises a second memory and a second processor, and wherein the visual browser is storable in the second memory and executable on the second processor for enabling interaction with the content page using one or more of user data and a first data set, wherein the visual browser displays the interaction through a visual display, and wherein the state of the voice browser is synchronized with the state of the visual browser from the information received from the state manager.
0. 26. A state manager for synchronizing browsers, the state manager comprising a processor and memory and configured to:
receive a voice browser state from a first wireless computing device executing a voice browser interacting with a content page, the voice browser storable in a memory of the first wireless computing device and executable on a processor of the first wireless computing device, wherein the voice browser enables interaction with the content page using an audio speaker and a microphone, and wherein the voice browser sends the voice browser state indicating a state of interaction with the content page to the state manager;
in response to receipt of the voice browser state, synchronize the voice browser state with a visual browser state to recreate the voice browser state at one of a plurality of granularity levels, the voice browser state being indicative of a state of interaction with the content page; and
transmit the visual browser state to a second wireless computing device executing a visual browser stored in a memory of the second wireless computing device for interacting with the content page, wherein the granularity levels are representative of a precision of browser inputs and outputs from a set of available precisions including a coarse level of precision and at least one finer level of precision with which the voice browser state is recreated, and wherein the granularity levels comprise a page level granularity associated with the coarse level of precision and at least one finer level of granularity associated with the at least one finer level of precision, the at least one finer level of granularity comprising at least one of a card level granularity, a field level granularity, and a cursor level granularity.
0. 19. A state manager for synchronizing browsers, the state manager comprising a processor and memory and configured to:
receive a visual browser state from a first wireless device executing a visual browser operable to interact with a content page, the visual browser state indicative of a state of interaction with the content page, the visual browser stored and operating in the first wireless device for interacting with the content page using one or more of user data and a first data set, wherein the visual browser stores the visual browser state indicating the state of interaction with the content page, and wherein the visual browser displays the interaction through a visual display; and
transmit the received visual browser state to a second wireless device including an audio speaker and microphone and executing a voice browser stored in a memory of the second wireless device, the voice browser configured to allow interaction of a user with the content page using an audio speaker and microphone, the second wireless device configured to receive the visual browser state and recreate the visual browser state at the voice browser at one of a plurality of granularity levels, wherein each of the granularity levels represents a precision of browser inputs and outputs from a set of available precisions including a coarse level of precision and at least one finer level of precision for recreating the visual browser state, wherein the granularity levels comprise a page level granularity associated with the coarse level of precision and at least one finer level of granularity associated with the at least one finer level of precision, the at least one finer level of granularity comprising at least one of a card level granularity, a field level granularity, and a cursor level granularity, and wherein the first wireless device is communicatively coupled to the second wireless device and a state of the voice browser is synchronized with the visual browser state.
17. A system of for using a visual browser for navigation through a collection of data, resulting in the generation of events, and the synchronization of the state of a voice browser with the state of said visual browser, comprising:
a first device further comprising:
a first memory to store a first set of data and a first set of instructions;
a first processor to execute said first instruction set and manipulate said first data set;
a visual display for displaying visual output;
an input device for accepting user data from a user;
an audio speaker to render audio output;
a microphone to accept audio input;
said visual browser stored in said first memory and operating in said first device for interacting with a content page using one or more of said user data and said first data set, wherein said visual browser displays said interaction through said visual display;
a script engine for recording the generated events according to a user-specified one of a plurality of granularity levels by executing instructions in a script, wherein each of said granularity levels represents a precision of browser inputs and outputs from a set of available precisions including a coarse level of precision and at least one finer level of precision, wherein said script comprises a plurality of sets of instructions interpretable by the script engine, and wherein each of said sets of instructions being adapted to create a record of a particular one of the events, said script engine being adapted to invoke a particular one of said sets of instructions according to which said events is signaled by said visual browser; and
a second device further comprising:
a second memory to store second set of data and second set of instructions;
a second processor to execute said second instruction set and manipulate said second data set;
said voice browser storable in said second memory and executable on said second processor for enabling interaction with said content page using said audio speaker and said microphone;
wherein said first wireless device is in communication with said second device and a state manager via a network; and
wherein said state manager receives the record of events created by said script engine and provides the record of events to the voice browser;
whereby the voice browser recreates the state of the visual browser.
18. A system of for using a voice browser for navigation through a collection of data, resulting in the generation of events, and the synchronization of the state of a visual browser with the state of said voice browser, comprising:
a first device further comprising:
a first memory to store a first set of data and a first set of instructions;
a first processor to execute said first instruction set and manipulate said first data set;
a visual display for displaying visual output;
an input device for accepting user data from a user;
an audio speaker to render audio output;
a microphone to accept audio input;
said visual browser storable in said first memory and operating in said first device for interacting with a content page using one or more of said user data and said first data set, wherein said visual browser displays said interaction through said visual display;
a script engine for recording the generated events according to a user-specified one of a plurality of granularity levels by executing instructions in a script, wherein each of said granularity levels represents a precision of browser inputs and outputs from a set of available precisions including a coarse level of precision and at least one finer level of precision, wherein said script comprises a plurality of sets of instructions interpretable by the script engine, and wherein each of said sets of instructions being adapted to create a record of a particular one of the events, said script engine being adapted to invoke particular one of said sets of instructions according to which of said events is signaled by said voice browser; and
a second device further comprising:
a second memory to store a second set of data and a second set of instructions;
a second processor to execute said second instruction set and manipulate said second data set;
said voice browser storable in said second memory and executable on said second processor for enabling interaction with said content page using said audio speaker and said microphone;
wherein said first wireless device is in communication with said second device and a state manager via a network;
and wherein said state manager receives the record of events created by said script engine, and provides the record of events to the visual browser;
whereby the visual browser recreates the state of the voice browser.
9. A system for using a voice browser for navigation through a collection of data and synchronization of the state of a visual browser with state of said voice browser, said system comprising:
a first wireless device further comprising:
a first memory to store a first set of data and a first set of instructions;
a first processor to execute said first instruction set and manipulate said first data set;
a visual display for displaying visual output;
an input device for accepting user data from a user;
an audio speaker to render audio output;
a microphone to accept audio input;
said visual browser stored in said first memory and operating in said first wireless device;
a second device further comprising:
a second memory to store a second set of data and a second set of instructions;
a second processor to execute said second instruction set and manipulate said second data set;
said voice browser storable in said second memory and executable on said second processor, wherein the voice browser enables interaction with a content page using said audio speaker and said microphone, and wherein said voice browser sends said voice browser state indicating a state of interaction with said content page to a state manager;
wherein said first wireless device is in communication with said second device and said state manager via a network; and
said state manager for receiving said voice browser state and providing said received voice browser state to said visual browser for allowing said visual browser to recreate the voice browser state at one of a plurality of granularity levels specified by said user, wherein each of said granularity levels represent the represents a precision of browser inputs and outputs from a set of available precisions including a coarse level of precision and at least one finer level of precision with which the visual browser recreates the voice browser state, and wherein said granularity levels comprise one of a page level granularity associated with the coarse level of precision and at least one finer level of granularity associated with the at least one finer level of precision, the at least one finer level of granularity comprising at least one of a card level granularity, a field level granularity, and a cursor level granularity; and
whereby the voice browser state is synchronized with the visual browser state from the information received from said state manager.
1. A system for using a visual browser for navigation through a collection of data and synchronization of the state of a voice browser with the state of said visual browser, said system comprising:
a first wireless device further comprising:
a first memory to store a first set of data and a first set of instructions;
a first processor to execute said first instruction set and manipulate said first data set;
a visual display for displaying visual output;
an input device for accepting user data from a user;
an audio speaker to render audio output;
a microphone to accept audio input; and
said visual browser stored in said first memory and operating in said first wireless device for interacting with a content page using one or more of said user data and said first data set, wherein said visual browser stores said visual browser state indicating a state of interaction with said content page, and wherein said visual browser displays said interaction through said visual display;
a state manager for receiving said visual browser state and providing said received visual browser state to said voice browser to recreate the visual browser state at said voice browser at one of a plurality of granularity levels specified by said user, wherein each of said granularity levels represent the represents a precision of browser inputs and outputs from a set of available precisions including a coarse level of precision and at least one finer level of precision with which the voice browser recreates the visual browser state, and wherein said granularity levels comprise one of a page level granularity associated with the coarse level of precision and at least one finer level of granularity associated with the at least one finer level of precision, the at least one finer level of granularity comprising at least one of a card level granularity, a field level granularity, and a cursor level granularity; and
a second device further comprising:
a second memory to store a second set of data and a second set of instructions;
a second processor to execute said second instruction set and manipulate said second data set;
said voice browser storable in said second memory and executable on said second processor, wherein the voice browser enables interaction of said user with said content page using said audio speaker and said microphone, and receives said visual browser state from said state manager;
wherein said first wireless device is in communication with said second device and said state manager through a network;
whereby the state of the voice browser is synchronized with the state of the visual browser from the information received from said state manager.
2. The system of
3. The system of
4. The system of
5. The system of
6. The system of
7. The system of
8. The system of
10. The system of
11. The system of
12. The system of
13. The system of
14. The system of
15. The system of
16. The system of
0. 20. The state manager of claim 19, wherein the visual browser state comprises a historical record of events associated during navigation with the visual browser.
0. 21. The state manager of claim 19, wherein the second wireless device is configured to use the state of the voice browser, which is synchronized with the visual browser state, to navigate to the content page.
0. 22. The state manager of claim 19, wherein the second wireless device is configured to use the card level granularity to point to a card within the content page.
0. 23. The state manager of claim 19, wherein the second wireless device is configured to use the field level granularity to point to a field within a card in the content page.
0. 24. The state manager of claim 19, wherein the second wireless device is configured to use the cursor level granularity to point to a cursor position.
0. 25. The state manager of claim 19, wherein the voice browser is configured to allow interaction of a user with the content page using an audio speaker and microphone.
0. 27. The state manager of claim 26, wherein the voice browser state comprises a historical record of events occurring during the interaction.
0. 28. The state manager of claim 26, wherein the second wireless computing device is configured to use the visual browser state, which is synchronized with the voice browser state, to navigate to the content page.
0. 29. The state manager of claim 26, wherein the second wireless computing device is configured to use the card level granularity to point to a card within the content page.
0. 30. The state manager of claim 26, wherein the second wireless computing device is configured to use the field level granularity to point to a field within a card in the content page.
0. 31. The state manager of claim 26, wherein the second wireless computing device is configured to use the cursor level granularity to point to a cursor position.
|
This application is a reissue of U.S. application Ser. No. 11/973,467, filed Oct. 9, 2007, now U.S. Pat. No. 7,584,249 B2, issued Sep. 1, 2009, which is a divisional application of application Ser. No. 11/252,250 filed 17 Oct. 2005 now abandoned, which is a division of Ser. No. 10/187,486 U.S. Pat. No. 6,983,307, titled “Synchronization Among Plural Browsers” filed on Jul. 1, 2002, which claims the benefit of U.S. provisional application Ser. No. 60/304,499 titled “Synchronization Among Plural Browsers” filed on Jul. 11, 2001.
The present invention relates generally to the field of networked computing. More particularly, the invention provides a technique for synchronizing the state of plural browsers at various levels of granularity. The technique may be used to synchronize between visual and voice browsers in wireless multi-modal applications.
Increasingly, wireless communications devices such as wireless telephones are becoming adapted for use with the Internet. It is estimated that, by the year 2003, half of all Internet traffic will come from wireless devices. Many present-day wireless telephones have at least some capability not only to capture and render audio information, but also to allow users to interact with data using a visual display and some form of data input device. Many wireless carriers are marketing access to the so-called “wireless web” as part of wireless telephone service.
While wireless data access is clearly a boon to electronic device users, there is a notable deficiency in the current wireless data access technology: a user must generally choose to interact with the data either using the voice components of the wireless device (i.e., microphone and speaker), or using the visual components (i.e., screen and keypad), but cannot use both at the same time. Some aspects of communication work best with a visual interface, and others work best with a voice interface. For example, suppose that an application provides directions to a specified location—e.g., for use while driving. It is convenient to speak the name of the desired location as input to the application, but it is cumbersome to receive the directions themselves in the form of speech. A visual map combined with written directions such as “turn right on Elm” is a very convenient format in which to receive the directions, but it is may be less convenient to input the desired location using a keypad or stylus than it is merely to speak the location into a microphone. Thus, the ideal interface for certain applications is, in many cases, not visual or voice alone, but rather a combination of visual and voice. Present wireless application often allow one or the other but not both.
One problem that impedes the integrated and combined use of voice and visual interfaces to data is that each mode of communication generally requires its own browser. Typically, a particular piece of content (e.g., a web page) may be represented in both a visual markup language (such as Wireless Markup Language or “WML”), and in a voice markup language (such as Voice eXtensible Markup Language or “VXML”). A visual browser permits the user to navigate through the WML content using the screen and keypad. Similarly, a voice browser, which is generally a software component separate from the visual browser, permits the user to navigate through the VXML content using the microphone and speaker. Not only are the visual and voice browsers separate software components; they often execute on separate, and distantly located, devices. A visual browser typically executes on a wireless handset (such as a wireless-web-enabled telephone). However, the handset is generally “dumb” with respect to voice—i.e., it can capture and render audio signals, but does not have the means to browse and navigate content based on the content of the received audio, or to generate audio signals based on VXML data. Thus, a voice browser typically executes on a voice server and communicates with the user through the microphone and speaker of the wireless device by transmitting and receiving digital signals to the device through an ordinary voice circuit within a wireless network.
Because the voice and visual browsers are separate, it is difficult to switch seamlessly back and forth between a visual and voice modes of interacting with wireless data, because the voice browser may be unaware of what the visual browser is doing and vice versa. That is, the voice and visual browsers are not normally “synchronized,” in the sense that neither knows the other's state with respect to the underlying content that the voice and visual browsers are manipulating. For example, suppose that a wireless handset user uses a visual browser to navigate through a series of web pages, eventually ending up at a particular URL. If the user then decides to switch to the voice interface, the voice browser does not know where the user has navigated to because it is unaware of what the visual browser has been doing. Upon switching to a voice interface, the voice browser can simply re-start the user at a “home” URL, but this is inconvenient for the user because the user loses the benefit of all of the navigation that has already been performed. Similarly, within the page located at a particular URL, the user may have navigated through several cards, and may have positioned the cursor at a particular field on a particular card using the visual browser, but the voice browser will be unaware of all of this activity. The problem, in this example, is that the voice and visual browsers are not “synchronized.”
In view of the foregoing, there is a need for a browser synchronization technique that overcomes the drawbacks of the prior art.
The present invention provides a technique for the synchronization of plural browsers—e.g., the synchronization of a visual browser with a voice browser. In accordance with the invention, a first browser generates events relating to navigational activity being performed by a user. For example, in WML where a page comprises “deck” of “cards,” an action wherein the user moves from one card to another may be an event. Similarly, where there may be plural fields within a single card, the act of moving a cursor from one field to another may be an event. A representation of the events that have occurred in a navigational session are recorded and are subsequently used to set a second browser to a state equivalent to that which the second browser would be in if the user had performed, on the second browser, a navigational activity equivalent to that which the user performed on the first browser.
The “state” of a browser may be defined at various levels of granularity. For example, in WML, a page consists of a deck of cards. Within a card, there may be a plurality of fields (e.g., a card that asks the user to enter his or her name and address may have two input fields, one for the user's name and one for the user's address). Within a given field, the cursor may be at a particular position. Thus, depending on the precision with which synchronization between the first and second browsers is desired, the “state” of the first browser may be defined as the currently displayed page, the currently displayed card within the page, the field to which the cursor is currently pointing, or the position of a cursor within the field. How much precision is required depends on context in which synchronization is required. In one application, it may be sufficient to say that two browsers are in the same “state” so long as they are on the same page—even if the first and second browsers are displaying different cards from that page. In another context, synchronization may require that the two browsers are presently accepting input at the same field of the same card. The present invention provides for the synchronization of two browsers at any granularity.
In one embodiment of the invention, the events generated by the first browser are captured by a script engine, which executes instructions contained in a script, such as a JAVA script. A script may be structured such that a given set of instructions that correspond to a particular event and are invoked by that event. In accordance with an aspect of the invention, the script contains instructions that, upon the occurrence of a given event, make a record documenting the fact that the event occurred. Thus, a function of the script is to produce a record documenting what events occurred during a navigational session on the first browser, and in what order those events occurred. The record of the events is sent to a state manager, which, at the time of synchronization, forwards those events to the second browser. By stepping through the same series of events that occurred on the first browser, the second browser may be put in the same state as the first browser.
In another embodiment of the invention, the first browser may be adapted to generate state information without a script. For example, a modified browser may create a record of every event that occurs directly (without using a script), or it may otherwise record information that indicates the state that it is currently in. This information may be sent to a state manager for use in synchronization the second browser with the first.
Two browser that interact with information in different formats or “modes” may be synchronized. For example, the first browser may be a visual browser that renders WML information, while the second browser may be a voice browser that renders Voice eXtensible Markup Language (VXML) information. A visual browser that is rendering a WML page may by synchronized with a voice browser that is rendering a VXML page, even if the WML and VXML pages are not identical. An equivalence relationship may be defined such that two browsers can be synchronized to equivalent states, even if complete identity of state is not possible due to the differing nature of their respective interfaces (e.g., visual vs. voice), and their different markup languages (e.g., WML vs. VXML).
The present invention's technique of synchronizing browsers may, for example, be deployed in an architecture that supports wireless multi-modal applications. For example, a wireless handset may have a browser that renders WML, plus “dumb” audio-rendering capability that produces audio from a digital signal but does not generate an audio signal from higher-level information. The generation of audio signals may be performed by a voice browser that generates audio signals from VXML input, and which is located remotely from the handset. The technique of the present invention may be used to synchronize the voice and visual browsers, such that the user can change between visual I/O and audio I/O.
Other features of the invention are described below.
The foregoing summary, as well as the following detailed description of preferred embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the invention, there is shown in the drawings exemplary constructions of the invention; however, the invention is not limited to the specific methods and instrumentalities disclosed. In the drawings:
Overview
Wireless devices have traditionally been able to operate in only one input/output (“I/O”) mode at a time—i.e., either in an audio mode or a visual mode. For example, a traditional wireless telephone sends and receives audio (voice). Some new wireless telephones have a small display through which the user can view the “wireless web.” However, a user can use such a telephone only in one mode at a time, as the voice and visual features cannot generally be used in concert.
One way to support the use of voice and visual I/O modes in concert is for the content with which the user interacts to be provided in two similar forms—a visual markup language (such as Wireless Markup Language (WML)), and a voice markup language (such as Voice extensible Markup Language (VXML)). Supporting the concurrent use of voice and visual I/O modes in this manner generally requires that two browsers be running at the same time—one browser that generates visual images from version of the content that is in the visual markup language, and another browser that renders audio based on the version of the content that is in the voice markup language. In order to support relatively seamless switching between visual and voice I/O modes, it may be necessary to synchronize the visual and voice browsers so that both browsers are at the same navigational point, regardless of which browser the user has been using to interact with the content. The present invention provides a technique for performing this synchronization.
Exemplary Architecture for Browser Synchronization
Page 104 is provided to computing device 108. Computing device 108 may be any type of device that is capable of performing computation. As is known in the art, such a device typically has a memory that stores data and instructions; a processor adapted to execute the instructions and manipulate the data stored in the memory; means for input (e.g., keypad, touch screen, microphone, etc.), and means for output (liquid crystal display (LCD), cathode ray tube (CRT), audio speaker, etc.). A computing device may also have means for communicating with other computing devices over a network—e.g., an Ethernet port, a modem, a wireless transmitter/receiver for communicating in a wireless communications network. Such a device may take the form of a personal computer (PC), laptop computer, or palm-sized computer. It will also be appreciated that many devices that are not traditionally labeled “computers” do, in fact, have computing capability. Wireless telephones, pagers, and wireless e-mail devices are examples such devices, and thus the generic term “computing device” applies to any such device, whether or not such device is traditionally described as a computer. In a preferred embodiment of the invention, computing device 108 is a wireless handset adapted to communicate in a wireless telephone network, although such an embodiment of computing device 108 is not limiting of the invention.
Visual browser 110 is a software application which is stored on computing device 108 and which executes thereon. Visual browser 110 is adapted to receive content in the form of a visual markup language page and to render that content on a visual display 116 associated with computing device 108. As one example, visual browser 110 may be a WML browser that renders WML content on the LCD display of a wireless telephone that is adapted to allow its user to interact with the “wireless web.” Visual browser 110 may also be adapted to receive user data input from input device 120 associated with computing device 108. For example, input device 120 may be the keypad of a wireless telephone, and the user may use the keypad to enter data in order to interact with content that is being rendered on visual display 116 by visual browser 110. (E.g., the user may use the keypad to enter his or her name into the “name” field of page 104.)
Page 106 is also provided to computing device 112. Like computing device 108, computing device 112 may be any type of computing device. Voice browser 114 is a software application which is stored on computing device 108 and which executes thereon. Voice browser 114 is adapted to receive content in the form of a voice markup language page and to render that content on audio speaker 118. Voice browser 114 may also be adapted to receive audio user input from microphone 122 or other audio input device. For example, the user may use microphone 122 to enter data into an audio “form” that is being rendered by voice browser 114. (E.g., the user may speak his name in response to the “enter name” voice prompt that voice browser 114 renders based on voice markup language page 106.)
While computing device 112 may be any type of computing device, in a preferred embodiment computing device 112 is a relatively powerful server machine that renders voice markup pages for a large network. As discussed more particularly in connection with
Visual browser 110 may be at some “state” with respect to the user's navigation through visual markup page 104. Likewise, voice browser 114 may be at some “state” with respect to the user's navigation through voice markup page 106. Since pages 104 and 106 represent the same underlying content 102, albeit in slightly different formats (e.g., WML vs. VXML), it is possible to “synchronize” the respective states of visual browser 110 and voice browser 114 with respect to the navigation. For example, using visual browser 110, the user may point a cursor to the “address” field of page 104. Thus, a description of the state of navigation through page 104 is that the cursor is presently pointed at the “address” field of page 104 and the browser is waiting for input in that field. An equivalent state of navigation through page 106 may be voice browser 114's rendering of an “enter address” audio prompt and waiting for audio input. Thus, in this example, if voice browser 114 is “synchronized” with visual browser 110, the appropriate action for voice browser 114 may be to render the “enter address” audio prompt.
In accordance with the present invention, visual browser 110 and voice browser 114 may be synchronized by exchanging information as to their state. When the user is navigating through content 102 using visual browser 110, visual browser 110 may provide state information to state manager 124, which may store this state information in state database 126. At an appropriate time, state manager 124 may provide this state information to voice browser 114, whereby voice browser may re-create the state of visual browser 110. This process may also happen in the other direction. That is, while the user is navigating through content 102 using voice browser 114, voice browser 114 may provide state information to state manager 124 for storage in state database 126. At an appropriate time, state manager 124 may provide this state information to visual browser 110, whereby visual browser 110 may recreate the state of voice browser 114. What constitutes an “appropriate time” to transfer this state information depends on the application in which browsers are being synchronized. For example, an “appropriate time” to transfer state information may mean continuously, periodically, or every time the I/O mode in which the user is performing the navigation switches between visual and voice. The manner and format in which state information is recorded, stored, and transmitted is more particularly discussed below in connection with
Exemplary page 104 comprises a plurality of cards 202-210. The relationship among cards 202-210 is shown by arrows. For example, card 202 displays a question to be answered by the user; the user navigates either to card 204 or card 206, depending upon which of the answer choices he or she selects at card 202. Similarly, cards 204 and 206 lead the user to different places depending upon the user's answer to a question. Navigation paths may converge; cards 204 and 206 may both lead to card 210.
The state of navigation may be defined as the place at which the user is currently performing I/O, as identified from among the entire universe of content available to the user. The location of this I/O may be identified at varying levels of precision, and this precision may be referred to as the “granularity” of the state. For example, at the coarse end of the granularity scale, the state of the user's navigation may be defined as the particular page that the user is viewing. Thus, in the example of
As an example of a slightly finer granularity, the state may be defined by the particular card the user is viewing. For example, the state may be defined as card 208 of page 104. At an even finer granularity, the state may be defined as the particular field of a card in which the user in entering input—e.g., the address field of card 208, as indicated by box 212. At an even finer granularity, the state may be defined as the position of the cursor in on the card, as indicated by box 214.
The effect of using the various granularities is readily apparent when one envisions performing a synchronization between two browsers at the various granularities. Suppose the user is using a first browser, and the user's cursor is positioned at box 214. At the page level of granularity, the relevant state information is that the user is navigating somewhere in page 104, and thus an attempt to synchronize the first browser with a second browser will result in the second browser being pointed to some arbitrary point on page 204 (e.g., at the beginning of the first card). At the card level of granularity, it is known not only that the user is on page 104, but also that the user is somewhere within card 208. Thus, upon synchronization, the second browser will be pointed to an arbitrary point in card 208 (e.g., the beginning of the card), but not necessarily to the place where the user's cursor was pointed in the first browser. At the field level of granularity, it is known that the user is in the “address” field of card 208, and thus synchronization results in the second browser being pointed to the “address” field, but not necessarily to any point within the address field. At the cursor level of granularity, however, it is known that the user is not only in the “address” field but is in the middle of entering data in the field. Thus, the second browser can be synchronized to a state in which a partially filled-out address is placed in the “address” field displayed on the second browser, and the cursor is in such a position that the user continue where he or she left off.
As noted above, a particularly useful application for browser synchronization is where one browser is a visual browser and the other is a voice browser. While voice browsers do not have “cursors” per se, the notion what it means for a cursor to be located at a particular “position” in a voice dialogue can be given meaning. For example, if the user had begun to enter the address “123 Elm Street,” but has only entered as far as “123 El . . . ” in the visual browser prior to switching to voice, the voice browser could emulate the position of a cursor by prompting the user: “You have entered: 123 El. Please continue speaking from that point.”
Events
State Capturing and State Synchronization
Turning to
Browser 110 is adapted to signal events in such a way that specific actions can be taken in response to the occurrence of events. As one example, browser 110 may be coupled to script engine 402. Script engine 402 interprets scripts written in a scripting language such as JAVA, and causes computing device 108 to perform actions based on such scripts. (While script engine 402 is shown as being external to, and communicatively coupled with, browser 110, it should be noted that this structure is merely exemplary; in the alternative, browser 110 may include script engine 402.) An example of such a script that may be executed by script engine is event-recording script 404. Event recording script 404 contains interpretable code that is invoked upon each event generated in browser 110, where this code performs the function of recording the generated event and memorializing the event in event record 406. For example, one of the events generated by browser 110 may be a navigation from card 202 to card 204 (e.g., event 302, shown in
Event record 406 may be used to synchronize browser 114 (shown in
Moreover, although browser 110 may send event record 406, or the information derived therefrom, directly to browser 114, in an alternative embodiment browser 110 sends event record 406 or the derived information to state manager 124 (shown in
Moreover, it should be understood that, while
It should be noted that, while pages 104 and 106 represent the same content 102, their representations of that content is not necessarily identical. Returning briefly to
A process of recording state information, and of using the recorded information to synchronize a second browser to the state of a first browser, is shown in
At step 704, it is determined whether a triggering event has occurred. The “triggering event” detected at step 704 is an event that causes state information to be transmitted by the device on which the first browser executes to another device. The following is a non-exhaustive list of triggering events: expiration of a timer, a demand for synchronization to take place, or a “mode change” that results in browsing being switched from the first browser to the second browser (e.g., from a visual browser to a voice browser). However, it should be understood that the foregoing list is non-exhaustive, and that any triggering event may be detected at step 704 without departing from the spirit and scope of the invention. If it is determined at step 704 that no triggering event has occurred, then the process returns to step 702 to capture more state information.
If it is determined at step 704 that a triggering event has occurred, then the captured state information is sent from the first browser to the second browser. The sending of state information from the first browser to the second browser may, optionally, include sending the state information to a state manager 124 (step 706), whereby state manager 124 stores the state information in data store 126 (step 708) for forwarding to the second browser at an appropriate time. However, the use of state manger 124 is not limiting of the invention, and the first browser may forward captured state information directly to the second browser.
At step 710, the second browser receives state information that was captured by the first browser. As noted above, this state information may be received directly from the first browser, or, alternatively, may be received indirectly through a state manager that performs functions including the collecting of state information to be forwarded at an appropriate time.
At step 712, it is determined whether an event occurs that triggers the second browser to synchronize its state to that of the first browser. Events that may trigger synchronization are non-exhaustively listed above in connection with step 704. If no triggering event has occurred, the process returns to step 710, wherein the second browser continues to receive state information captured by the first browser, and waits for a triggering event to occur. On the other hand, if a triggering event is detected at step 712, then the second browser adjusts its state to reflect the received state information (step 714). As discussed above, one way that this state adjustment can take place is if the state information includes a historical record of events that have occurred on the first browser, in which case the second browser may step through that same sequence of events (where the events may have, optionally, undergone a “transformation” to account for the fact that the first and second browser may be rendering the same content in slightly different formats (e.g., in different markup languages)).
Following the adjustment of the second browser to reflect the state of the first, the two browsers continue with the process of capturing state data, and each browser's adjusting its state to reflect the state data captured by the other. It should be noted that this process of capturing and adjusting is a mutual process that proceeds in both directions. That is, each browser is capable of capturing state data (which is generally done when the browser is being used by a user to perform navigation), and is also capable of synchronizing to a given state based on the state data provided by the other browser. Thus, while
Exemplary Environment for Synchronization of Visual and Voice Browsers
With reference to
The exemplary computing device 108 shown in
Because visual browser 110 executes on computing device 108, application server 910 provides content in the form of visual markup language directly to computing device 108. That is, when the application 912 is operating in visual mode, application server 912 provides visual markup language content to switch 908, so that such content can be sent out to computing device 108. Computing device 108 then uses visual browser 110 to interact with a user of computing device 108 on the basis of the visual markup content. However, computing device 108, in this example, does not run a voice browser; rather, computing device 108 merely accepts audio input for voice browser 114 and renders audio output generated by voice browser 114. Voice browser 114 runs on computing device 112 which, in the example of
Because visual browser 110 and voice browser 114 are located separately from each other in the example of
It is noted that the foregoing examples have been provided merely for the purpose of explanation and are in no way to be construed as limiting of the present invention. While the invention has been described with reference to various embodiments, it is understood that the words which have been used herein are words of description and illustration, rather than words of limitations. Further, although the invention has been described herein with reference to particular means, materials and embodiments, the invention is not intended to be limited to the particulars disclosed herein; rather, the invention extends to all functionally equivalent structures, methods and uses, such as are within the scope of the appended claims. Those skilled in the art, having the benefit of the teachings of this specification, may effect numerous modifications thereto and changes may be made without departing from the scope and spirit of the invention in its aspects.
Mumick, Inderpal Singh, Sibal, Sandeep
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
5133011, | Dec 26 1990 | International Business Machines Corporation | Method and apparatus for linear vocal control of cursor position |
5809415, | Dec 11 1995 | UNWIRED PLANET IP MANAGER, LLC; Unwired Planet, LLC | Method and architecture for an interactive two-way data communication network |
5835077, | Jan 13 1995 | Rosemount Aerospace Inc | Computer control device |
5944791, | Oct 04 1996 | Open Invention Network, LLC | Collaborative web browser |
5954798, | Oct 06 1997 | NCR Voyix Corporation | Mechanism for dependably managing web synchronization and tracking operations among multiple browsers |
6018710, | Dec 13 1996 | Siemens Corporation | Web-based interactive radio environment: WIRE |
6101510, | Jan 29 1997 | Microsoft Technology Licensing, LLC | Web browser control for incorporating web browser functionality into application programs |
6125376, | Apr 10 1997 | Nuance Communications, Inc | Method and apparatus for voice interaction over a network using parameterized interaction definitions |
6151622, | Feb 02 1998 | International Business Machines Corp.; IBM Corporation | Method and system for portably enabling view synchronization over the world-wide web using frame hierarchies |
6195357, | Sep 24 1996 | InterVoice Limited Partnership | Interactive information transaction processing system with universal telephony gateway capabilities |
6349132, | Dec 16 1999 | Intellisync Corporation | Voice interface for electronic documents |
6418199, | Dec 05 1997 | MORGAN STANLEY SERVICES GROUP INC | Voice control of a server |
6418471, | Oct 06 1997 | NCR Voyix Corporation | Method for recording and reproducing the browsing activities of an individual web browser |
6430599, | Jun 15 1999 | Oracle America, Inc | Just-in-time services for small footprint devices |
6453342, | Dec 03 1998 | International Business Machines Corporation | Method and apparatus for selective caching and cleaning of history pages for web browsers |
6494762, | Mar 31 2000 | Matsushita Electrical Industrial Co., Ltd. | Portable electronic subscription device and service |
6594348, | Feb 24 1999 | HEWLETT-PACKARD DEVELOPMENT COMPANY, L P | Voice browser and a method at a voice browser |
6601170, | Dec 30 1999 | SECURE AXCESS LLC | Secure internet user state creation method and system with user supplied key and seeding |
6604075, | May 20 1999 | Alcatel Lucent | Web-based voice dialog interface |
6606611, | Feb 27 1999 | Nuance Communications, Inc | System and method for audio-only internet browsing using a standard telephone |
6609005, | Mar 28 2000 | Intel Corporation | System and method for displaying the location of a wireless communications device wiring a universal resource locator |
6668055, | Oct 08 1999 | Grape Technology Group, Inc. | Personalized assistance system and method |
6738803, | Sep 03 1999 | Cisco Technology, Inc | Proxy browser providing voice enabled web application audio control for telephony devices |
6771743, | Sep 07 1996 | Nuance Communications, Inc | Voice processing system, method and computer program product having common source for internet world wide web pages and voice applications |
6807254, | Nov 06 1998 | ONMOBILE LIVE, INC | Method and system for interactive messaging |
6807529, | Feb 27 2002 | Google Technology Holdings LLC | System and method for concurrent multimodal communication |
6816880, | Mar 26 1997 | Wilmington Trust, National Association, as Administrative Agent | Browser user inter face for client workstation |
6820116, | Dec 21 2001 | NOKIA SOLUTIONS AND NETWORKS OY | Mobile browsing booster system |
6865608, | Mar 31 2000 | NM, LLC | Method and system for simplified access to internet content on a wireless device |
6922733, | Jun 30 1999 | International Business Machines Corporation | Method for coordinating visual and speech web browsers |
6976077, | Jul 06 1999 | Microsoft Technology Licensing, LLC | Automatic and transparent synchronization of server-side state information with a client application |
6976094, | May 09 2000 | RPX CLEARINGHOUSE LLC | Automated web browser synchronization by using session initiation protocol during a real-time session |
6983273, | Jun 27 2002 | International Business Machines Corporation | Iconic representation of linked site characteristics |
6999607, | Feb 20 2001 | CYTOKINETICS, INC | Method and apparatus for automated cellular bioinformatics |
7080315, | Jun 28 2000 | Nuance Communications, Inc | Method and apparatus for coupling a visual browser to a voice browser |
7210098, | Feb 18 2002 | KIRUSA, INC | Technique for synchronizing visual and voice browsers to enable multi-modal browsing |
7225225, | Mar 30 2001 | Sharp Kabushiki Kaisha | Selective synchronization of web browsers |
7657828, | Jun 28 2000 | Nuance Communications, Inc | Method and apparatus for coupling a visual browser to a voice browser |
7685252, | Oct 12 1999 | TWITTER, INC | Methods and systems for multi-modal browsing and implementation of a conversational markup language |
8448059, | Sep 03 1999 | Cisco Technology, Inc | Apparatus and method for providing browser audio control for voice enabled web applications |
20020052747, | |||
20020065875, | |||
20050228883, | |||
EP848373, | |||
EP1100013, | |||
EP2280357, | |||
WO9955049, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Feb 01 2011 | KIRUSA INC | WALOOMBA TECH LTD , L L C | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 027192 | /0026 | |
Sep 01 2011 | Gula Consulting Limited Liability Company | (assignment on the face of the patent) | / | |||
Aug 28 2015 | WALOOMBA TECH LTD , L L C | Gula Consulting Limited Liability Company | MERGER SEE DOCUMENT FOR DETAILS | 037028 | /0475 |
Date | Maintenance Fee Events |
Dec 02 2019 | PTGR: Petition Related to Maintenance Fees Granted. |
Feb 11 2021 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
Jul 28 2023 | 4 years fee payment window open |
Jan 28 2024 | 6 months grace period start (w surcharge) |
Jul 28 2024 | patent expiry (for year 4) |
Jul 28 2026 | 2 years to revive unintentionally abandoned end. (for year 4) |
Jul 28 2027 | 8 years fee payment window open |
Jan 28 2028 | 6 months grace period start (w surcharge) |
Jul 28 2028 | patent expiry (for year 8) |
Jul 28 2030 | 2 years to revive unintentionally abandoned end. (for year 8) |
Jul 28 2031 | 12 years fee payment window open |
Jan 28 2032 | 6 months grace period start (w surcharge) |
Jul 28 2032 | patent expiry (for year 12) |
Jul 28 2034 | 2 years to revive unintentionally abandoned end. (for year 12) |