System and method for linking streams of multimedia data to reference material for display

Abstract

A system for indexing displayed elements that is useful for accessing and understanding new or difficult materials, in which a user highlights unknown words or characters or other displayed elements encountered while viewing displayed materials. In a language learning application, the system displays the meaning of a word in context; and the user may include the word in a personal vocabulary to build a database of words and phrases. In a Japanese language application, one or more Japanese language books are read on an electronic display. Readings (‘yomi’) for all words are readily viewable for any selected word or phrase, as well as an English reference to the selected word or phrase. Extensive notes are provided for difficult phrases and words not normally found in a dictionary. A unique indexing scheme allows word-by-word access to any of several external multi-media references.

Description

or. The Main Control Buttons are located just below the Word field. The arrow keys display the next or previous words based on the sort key indicated by the Sort Button in the bottom left corner. The Show Notes button displays the Note information about the Word. This button toggles to Hide Notes when the field is displayed and Show Notes when hidden. Additional notes and annotations can be entered directly. The Quick Search button displays the word in a pop-window for quick search of a single character. After the pop-up is displayed, the user can click on the desired character to search. The Flash Words button displays the words in the personal dictionary in slide show fashion. Sort order or random order are selectable: sort order uses the current sort order.

The Find button displays the search dialogue window. Words are searchable by the following keys: Word, Yomi, English Reference, Category, Source, Priority, or Date. The personal dictionary supports logical “AND” searching for each of the above keys. The following features are supported:

• 1. Jump to Text—this button jumps control and display from the personal dictionary to the source of the word in the original text;
• 2. Show Info—this button displays the Word Information Buttons, as well as the Date Indicator; this button toggles to Hide Info when displayed, and Show Info when hidden; and
• 3. Word Information—this button appears on the bottom of the screen and has the following functions:
  • a. Current Sort—sets the sort order for the Dictionary to either Category, Source, Priority, or Date;
  • b. Category—provides for a set of predefined Categories for words as well as the ability to add new Categories;
  • c. Source—indicates the source of the Word: user entered words are indicated by the user name or if not available, by the default User;
  • d. Priority—allows the user to assign a priority to a word from 1 to 5; and
  • e. Date Display—the date is displayed in the bottom right hand corner; the date is automatically updated each time the word is displayed.
    Searching

Both the electronic viewer module and the personal dictionary module provide search features accessible via the Word Menu. After selecting Find from the menu, the search dialogue appears.

The electronic viewer module includes a simple search feature that allows the user to search for a string of text anywhere in the book. The user enters the desired text and clicks Find to execute the Search. Find Next searches for the next occurrence of the word in the text.

In the personal dictionary, a slightly more complex search feature is provided. The search dialogue allows the user to enter multiple search terms. For example, a user can search for a certain term in the ‘Economics’ category or the user can look for a Kanji that has a certain reading. More search terms result in increased search time. The search term for Word, Yomi, Reference, Note, and Source are indexed based on the first seven characters contained in the field. Characters appearing after the seventh character in any of these fields are not found with the ‘Starts With’ selection. Selecting ‘Contains’ searches the entire text in the field.

To search, the user enters the desired word or character and then selects ‘Starts With’ or ‘Contains’ from the menu. A ‘Starts With’ search is the fastest. The ‘Category’ search terms are based on the category list. The integers 1 to 5 can be entered for ‘Priority.’ Date searching can be performed as ‘is’, ‘is before’, or ‘is after.’ After entering the desired search information, the user clicks ‘Find’ to execute the Search. Find Next searches for the next occurrence in the personal dictionary.

Importing/Exporting Word Lists

Text files can be read into the personal dictionary to make data exchange with other programs and colleagues feasible. The following format should be followed to allow accurate importing. One may use a spreadsheet program to build the word table and export the information as a tab delimited file. If a word processor is used, the user must add an extra tab for blank fields and follow the format listed below. In the exemplary embodiment of the invention, Export and Import uses the following format:

Word          [TAB];
Pronunciation [TAB};
Meaning       [TAB];
Notes         [TAB];
Category      [TAB];
Source        [TAB];
Priority      [TAB]; and
Date          [Hard Return].

Setting up the Word field as column A in a spreadsheet and then exporting as a text file results in this format. If a word processor is used, one should also save as a text file. One should not include any hard returns (user entered returns) within the string of text for the given word. If given the option, the user should elect to have soft returns (automatically entered returns) deleted. To import, the user selects Import Words from the File Menu, and then chooses the file for import. To export, the user selects Export Words from the File Menu, and then enters a name for the given file.

Although the invention is described herein with reference to the preferred embodiment, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the present invention. For example, the invention may be used to index images such that elements of the image are linked to an external reference. Thus, an illustration of the human body may include descriptive external resources for each of the body's internal organs, and would thereby aid in the study of anatomy. Likewise, a video or other moving picture display, for example animated displays, could be indexed such that the picture could be stopped and various elements within a frame of the picture could be examined through links to external references. The invention allows such application because it does not embed information within the source material as is the case with prior art hyperlink technology. Rather, the invention creates a physical counterpart to the image in which a selected image position defines an offset that locates a desired external reference. Accordingly, the invention should only be limited by the claims included below.

Claims

1. A system for linking source material to reference material for display comprising:

a source material image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information, said source material image stored in an electronic database;

means for determining an address on said electronic database for the beginning position of said source material image;

means for cutting said source material image into said discrete pieces;

means for determining an address on said electronic database for a start point and an end point of said discrete pieces of said image based upon said beginning position of said source material image;

means for recording said start and said end point addresses in a look-up table;

means for selecting a discrete portion of said source material image;

means for determining the address on said electronic database of said selected discrete portion;

means for converting said address of said selected discrete portion to an offset value from said beginning position address of said source material image;

means for comparing said offset value with said recorded start and end point addresses of said discrete pieces in said look-up table;

means for selecting an external reference that corresponds to said look-up table start and end point addresses; and

means for reproducing said external reference.

2. The system of claim 1, further comprising:

a linking engine for linking said source material to said reference information on any of a word-by-word and phrase-by-phrase basis.

3. The system of claim 2, said linking engine further comprising:

word cut means for dividing said source material into discrete pieces;

linking means for establishing at least one link between each of said discrete pieces and said reference information;

compiler means for assembling an integrated source image from said discrete pieces;

indexing means for linking said assembled discrete pieces to said reference information.

4. The system of claim 3, said linking engine further comprising:

means for building an index to link each of said source material pieces to said reference information.

5. The system of claim 4, wherein said index links said source material pieces to said reference information based upon the value of the offset of the starting and ending position addresses of said source material pieces from the beginning position address of said integrated source image.

6. The system of claim 5, wherein said offset locates said reference information to a corresponding source material piece based upon offset occurrence within a range defined by the value of the offsets of the starting and ending point addresses of said source material pieces from said beginning position address of said integrated source image.

7. The system of claim 1, further comprising:

means for manipulating said stored source material and reference information with at least two user keys.

8. A method for linking source material to reference material for display, comprising:

determining the beginning position address of a source material image stored in an electronic database, said source material image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information;

cutting said source material image into said discrete pieces;

determining a starting point address and an ending point address of said discrete pieces of said image based upon said beginning position address of said source material image;

recording said starting and said ending addresses in a look-up table;

selecting a discrete portion of said source material image;

determining the address of said selected discrete portion;

converting said address of said selected discrete portion to an offset value from said beginning position address of said source material image;

comparing said offset value with said recorded start and end point addresses of said discrete pieces in said look-up table;

selecting an external reference that corresponds to said look-up table start and end point addresses; and

reproducing said external reference.

9. In a language learning method, a method for linking source material to reference material for display, comprising the steps of:

reading a foreign language source material image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information with an electronic viewer;

accessing reference materials on selected portions of said source material image;

determining the beginning position address of said source material image;

cutting said source material image into said discrete pieces;

determining a start point address and an end point address of said discrete pieces of said image based upon said beginning position address of said source material image;

recording said start and said end point addresses in a look-up table;

selecting a discrete portion of said source material image;

determining the address of said selected discrete portion;

converting said address of said selected discrete portion to an offset value from said beginning position address of said source material image;

comparing said offset value with said recorded start and end point addresses of said discrete pieces in said look-up table;

selecting an external reference that corresponds to said look-up table start and end point addresses; and

reproducing said external reference.

10. The method of claim 9, further comprising the step of:

linking said source material to said reference information with a linking engine on any of a word-by-word and phrase-by-phrase basis.

11. The method of claim 10, said linking step further comprising the steps of:

dividing said source material into discrete pieces;

establishing at least one link between each of said discrete pieces and said reference information;

assembling an integrated source image from said discrete pieces; and

linking said assembled discrete pieces to said reference information.

12. The method of claim 11, said linking step further comprising the step of:

building an index to link each of said source material pieces to said reference information.

13. The method of claim 12, wherein said index links said source material pieces to said reference information based upon the offset between the starting position address for said source material pieces and the beginning position address of said integrated source image.

14. The method of claim 13, wherein said offset locates said reference information to a corresponding source material piece based upon offset occurrence within a range defined by the value of the offsets of the starting and ending position addresses of said source material pieces from said beginning position address of said integrated source image.

15. In a language learning system, a system for linking source material to reference material for display, comprising:

a text image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information;

means for determining the beginning position address of said text image;

means for cutting said text image into said discrete pieces;

means for determining a starting point address and an ending point address of said discrete pieces of said image based upon said beginning position address of said source material image;

means for recording said starting and said ending point addresses in a look-up table;

means for selecting a discrete portion of said text image;

means for determining the address of said selected discrete portion;

means for converting said address of said selected discrete portion to an offset value from said beginning position address of said source material image;

means for comparing said offset value with said recorded start and end point addresses of said discrete pieces in said look-up table;

means for selecting an external reference that corresponds to said look-up table start and end point addresses; and

means for displaying said external reference.

16. In a language learning method, a method for linking source material to reference material for display, comprising the steps of:

determining the beginning position address of a text image, said text image including a plurality of discrete pieces having links to external reference materials comprising any of textual, audio, video, and picture information;

cutting said source material image into said discrete pieces;

determining a starting point address and an ending point address of said discrete pieces of said image based upon said beginning position address of said text image;

recording said starting and said ending point addresses in a look-up table;

selecting a discrete portion of said text image;

determining the address of said selected discrete portion;

converting said address of said selected discrete portion to an offset value from said beginning position of said text image;

comparing said offset value with said recorded start and end point addresses of said discrete pieces in said look-up table;

selecting an external reference that corresponds to said look-up table start and end point address; and

displaying said external reference.

17. A method for linking source material to external reference materials for display, the method comprising the steps of:

determining a beginning position address of a source material stored in an electronic database;

cutting the source material into a plurality of discrete pieces;

determining a starting point address and an ending point address of at least one of the plurality of discrete pieces based upon the beginning position address;

recording in a look up table the starting and ending point addresses;

linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;

selecting a discrete portion of an image of the source material;

determining a display address of the selected discrete portion of the source material image;

converting the display address of the selected discrete portion to an offset value from the beginning position address;

comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces;

selecting one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces; and

displaying the selected one of the plurality of external reference materials.

18. The method of claim 17, wherein cutting the source material into a plurality of discrete pieces is done manually.

19. The method of claim 17, wherein cutting the source material into a plurality of discrete pieces is done automatically.

20. The method of claim 19, wherein automatically cutting the source material into a plurality of discrete pieces is done using a grammar parser.

21. The method of claim 19, wherein automatically cutting the source material into a plurality of discrete pieces is done without using tags.

22. The method of claim 19, wherein automatically cutting the source material into a plurality of discrete pieces is done without reference to any tags which may be located in the source material.

23. The method of claim 17, wherein the link is a hyperlink.

24. The method of claim 17, wherein the link is an address of the selected one of the plurality of external reference materials.

25. The method of claim 17, wherein the link is reference information for retrieving the selected one of the plurality of external reference materials.

26. The method of claim 17, wherein the source material is text based source material.

27. The method of claim 17, wherein the source material is image based source material.

28. The method of claim 17, wherein the source material is graphic based source material.

29. The method of claim 17, wherein the source material is audio based source material.

30. The method of claim 17, wherein the source material is video based source material.

31. The method of claim 17, wherein the source material is a combination of two or more of text based source material, image based source material, graphic based source material, audio based source material, and video based source material.

32. The method of claim 17, wherein linking at least one of the plurality of discrete pieces is done manually.

33. The method of claim 17, wherein linking at least one of the plurality of discrete pieces is done automatically.

34. A system for linking source material to external reference materials for display, the system comprising:

means for determining a beginning position address of a source material stored in an electronic database;

means for cutting the source material into a plurality of discrete pieces;

means for determining a starting point address and an ending point address of at least one of the plurality of discrete pieces based upon the beginning position address;

means for recording in a look up table the starting and ending point addresses;

means for linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;

means for selecting a discrete portion of an image of the source material;

means for determining a display address of the selected discrete portion of the source material image;

means for converting the display address of the selected discrete portion to an offset value from the beginning position address;

means for comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces;

means for selecting one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces; and

means for displaying the selected one of the plurality of external reference materials.

35. A method for linking textual source material to external reference materials for display, the method comprising the steps of:

determining a beginning position address of textual source material stored in an electronic database;

cutting the textual source material into a plurality of discrete pieces;

determining a starting point address and an ending point address of at least one of the plurality of discrete pieces based upon the beginning position address;

recording in a look up table the starting and ending point addresses;

linking at least one of the plurality of discrete pieces to at least one of a plurality of external reference materials, the plurality of external reference materials comprising any of textual, audio, video, and picture information;

selecting a discrete portion of an image of the textual source material;

determining a display address of the selected discrete portion of the textual source material image;

converting the display address of the selected discrete portion to an offset value from the beginning position address;

comparing the offset value with the starting and ending point addresses recorded in the look-up table to identify one of the plurality of discrete pieces;

selecting one of the plurality of external reference materials corresponding to the identified one of the plurality of discrete pieces; and

displaying the selected one of the plurality of external reference materials.