US20060174211A1 - Methods, apparatus and data structures for providing a user interface which facilitates decision making - Google Patents

Abstract

A user interface for facilitating a decision making process, such as planning a trip. A unified view of various types of information related to an event may be provided. The unified view may be presented in a simulated three-dimensional environment having different types of information depicted on different windows. Different types of information related to a common event may be visually linked. A window showing a articular type of information may be brought into a focus view for closer inspection by a user. Alternative decisions may be depicted to permit vagueness or uncertainty, particularly at early iterations in the decision making process.

Description

§ 1. BACKGROUND OF THE INVENTION
• § 1.1 Field of the Invention
• The present invention concerns a user interface which facilitates a decision making process, for example, for planning a trip. In particular, the present invention concerns a user interface that visually links related information rendered in multiple windows. Alternative choices may be visually depicted.
• § 1.2 Related Art
• § 1.2.1 The Problem of Decision Making and Planning
• Over the past few decades, two trends have permitted computer users to access a wealth of data and information. The first trend was the explosion in the amount of relatively inexpensive data storage means. The second trend was the increasing interconnection of computers, first by local area networks (or “LANs”), and more recently by wide area networks (or “WANs”) which may be private, such as intranets for example, or public, such as the Internet for example. Both of these trends are expected to continue.
• Although computer users now can access a wealth of data and information, challenges in interpreting data and using data and information to complete tasks remain. Applications for statistically processing data and for visually depicting (e.g., with pie charts, bar charts, graphs) data are widely available. Similarly, using data and information to complete relatively simple tasks is straight forward (e.g., determining the least expensive three or four bedroom home in Virginia using a relational data base).
• However, many tasks involve making decisions based on a number of factors (or choices) and are subject to initial uncertainty or vagueness. Such decisions typically require fixing or limiting the value of one or more factors (or choosing), interpreting the result, and iterating towards a desirable result. For example, in the context of planning a trip, factors may include (i) what one wants to do, (ii) when one wants to do it, (iii) where one wants to go, (iv) how much one wants to spend, etc. Choices in a group of factors may limit choices in that group or another group. That is, options are narrowed as choices are made or factors are fixed. For example, if a person defines a budget of $1000.00 for a vacation, and decides to spend $400.00 on airfare and $400.00 on hotel accommodations, and decides that they will need $150.00 for meals, that person will not have the option of a $200.00 helicopter tour unless they increase their budget.
• Again, in the context of planning a trip, such as a personal vacation for example, a person in Virginia might want to see a Washington Redskins football game, visit an aquarium, visit Aunt Betty in Delaware, see the Statue of Liberty and see autumn foliage. Time constraints should be accounted for. For example, when will a user's work schedule allow them to take a vacation, on what dates do the Washington Redskins play at home, when are aquariums and the Statue of Liberty open, when will Aunt Betty be home. Places may be determined. For example, do I want to see autumn foliage in Virginia or Vermont and do I want to visit the public aquarium in Baltimore, Md. or Camden, N.J. Budgetary limits may also be checked. For example, are those Redskins tickets going to cost too much? As choices are made, other choices are affected. For example, the peak time for autumn foliage in Vermont is late September, but the peak time for autumn foliage in Virginia is mid October. Further, data and information related to each of the factors may be best represented to a user in different ways. For example, time or date factors may be best represented on a calendar, cost factors may be best represented as a summed number, geographic factors may be best represented on a map, etc.
• §1.2.2 Known Decision Making User Interfaces and their Perceived Limitations
• The foregoing trip planning scenario illustrates the challenges involved in performing or planning complex tasks having a number of variables. The articles: L. Tweedie, R. Spence, D. Williams, and R. Bhoghal, “The Attribute Explorer”,Video Proceedings, CHI '94, Boston, Mass., ACM Press (1994), downloaded from http://www.ee.ic.ac.uk/resarch/information/www/LisaDir/at tv.html on Jul. 29, 1998 (hereafter referred to as “the Attribute Explorer article”) and H. Dawkes, L. Tweedie, and R. Spence, “VICKI—The VIsualisation Construction KIt”,Advanced Visual Interfaces Conference,Gubbio, Italy (May 27-29), downloaded from http://www.ee.ic.ac.uk/research/information/www/LisaDir/V ICKI/VICKI.html on Jul. 29, 1998 (hereafter referred to as “the VICKI article”) discuss tools for exploring “problem space” using a two dimensional graphical user interface (or “GUI”). The GUIs discussed in the Attribute Explorer and VICKI articles use sliders to permit parameters to be adjusted. Results are then fed back to the user via color coded histograms. More specifically, the Attribute Explorer and VICKI articles discuss manipulating factors (e.g., type, price, number of bedrooms, and garden size) in a house purchase task or decision. The various attributes are related via color.
• Unfortunately, the GUIs discussed in the Explorer and VICKI articles are two dimensional and depict data related to various factors in the same way. Therefore, it is believed that these GUIs are not as useful when various types of data or information are best depicted in different ways.
• The article, L. Tweedie, R. Spence, H. Dawkes, and H. Su, “The Influence Explorer—A Tool for Design”, downloaded from http://www1.acm.org:81/sigchi . . . ceedings/videos/Tweedie/I t2txt.htm on Jul. 29, 1998 (hereafter referred to as “the Influence Explorer article”) discusses a similar GUI in which the relationships between performance results and design parameters are depicted. Unfortunately, as with the GUIs discussed in the Attribute Explorer and VICKI articles, the GUI discussed in the Influence Explorer article is one dimensional and depicts data related to various factors in the same way. Therefore, it is believed that this GUI is not as useful when various types of data or information are best depicted in different ways.
• The VISAGE product, from MAYA Design Group, Inc. of Pittsburgh, Pa. permits information objects to be depicted in various frames (e.g., a hierarchical outline, a bar chart, and a map). Using a hierarchical outline, a user can navigate through the information by “drilling down” (or segmenting aggregated data) and “rolling up” (or aggregating segmented data). Each of the frames has a script which governs the appearance of objects that the frame contains. Color is used to coordinate the same object shown in various frames. Objects can be dragged and dropped within and between various frames. Although the VISAGE product depicts data in various ways, the same data (not merely related information) is depicted in each of the frames. Furthermore, the VISAGE product depicts objects in frames in only one or two dimensions. The VISAGE product is discussed in the articles: S. Roth, M. Chuah, S. Keredjiev, J. Kolojujchick and P. Lucas, “Towards and Information Visualization Workspace: Combining Multiple Means of Expression”, downloaded from http:www.cs.cmu,edu/˜sage/HCI-journal-96/HCI-journal.html on Jul. 29, 1998; J. Kolojujchick, S. Roth, and P. Lucas, “Information Appliances and Tools in Visage”,IEEE Computer Graphics and Applications,pp. 32-41 (July/August 1997); P. Lucas and S. Roth, “Exploring Information with Visage”, downloaded from http://www.maya.com/visage/base/papers/mayaVid.htm on Jul. 29, 1998; and P. Lucas, C. Gomberg, and S. Roth, “Visage: Dynamic Information Exploration”, downloaded from http://www.maya.com/visage/base/papers/mayaDemo.htm on Jul. 29, 1998.
• None of the “decision making” user interfaces introduced above exploit three-dimensional graphics. As discussed in § 1.2.2.1 below, some known three dimensional user interfaces are concerned with accessing information, but not with using information when making decisions.
• § 1.2.2.1 Known Three Dimensional User Interfaces to Information
• User interfaces providing a three dimensional depiction of related information have been discussed. For example, the article: Robertson, et al., “The Next Generation GUIS: Information Visualization Using 3D Interactive Animation,”Communications of the ACM, Vol. 35, No. 4, pages 57-71 (April 1993) (hereinafter referred to as “the Information Visualizer article”) discusses various proposals for an “Information Workspace”. The Information Visualizer article discusses a perspective wall which permits information, having some sort of linear relationship or thread, to be presented in the relatively narrow aspect ratio of a typical video monitor. When an item is selected, the wall moves the item to a center panel as if it were a sheet in a player piano reel. Its intuitive three dimensional metaphor allows smooth transitions among views, thereby helping a user to perceive object constancy. Files may be classified by their modification date. Although the perspective wall technique lends itself to information having a linear (e.g., timeline) thread, it is less useful for other types of information, or for information in which a linear thread is unimportant. Furthermore, the perspective wall is not specifically designed for facilitating an iterative planning or decision making process.
• Similarly, in the video, R. Kullberg, “Dynamic Timelines: Visualizing the History of Photography”,CHI '96Video Program,first 4:19, also discussed in the thesis of the same title, downloaded from http://robin.www.media.mit.ed/people/robin/thesis on Aug. 5, 1998 (hereafter referred to as “the Dynamic Timelines thesis”), a three dimensional user interface to photographs is discussed. Unfortunately, the photographs have a linear, and more specifically, a temporal, thread-. Also, like the perspective wall, the dynamic timeline is not specifically designed for facilitating an iterative planning or decision making process.
• § 1.2.3 Unmet Needs
• In view of the foregoing, there is an unmet need for a user interface which facilitates planning or making decisions based on a number of factors (or choices). Since factors may be subject to initial uncertainty or vagueness, such a user interface should facilitate an iterative planning or decision making process, allowing the user to make partial and/or vague decisions during the process. The results of alternative decisions should be visualized. Relationships between information should be depicted. These depictions are not limited to relating the same information shown in different ways. The user interface should permit a single display to include different windows of different types of related information such that a unified view of the task is presented to a user. However, the user interface should permit easy user interaction with any one of the different windows. Finally, intelligent help and access to information that may be needed or useful to complete the task should be provided to the user.
§ 2. SUMMARY OF THE INVENTION
• The present invention provides a graphical user interface in which a number of windows or planes depict different types of related information. The consequences of changes to one type of information on one window are depicted in the other types of information on the other windows.
• The present invention may further provide a graphical user interface in which the relationship between information in various windows is visually depicted, for example using color and colored rays. The colored rays may be translucent so that they do not occlude any information “behind” them.
• The present invention may simulate a three dimensional environment including different windows or planes having the different types of information. The windows or planes may be related to one another as surfaces of a folded paper, or as facets of a geometric object that are connected with one another. The present invention may permit any plane or window to be brought into a normal, head on, view (referred to as “focus view”) and depicted at a centered position on the display. In this regard, the present invention may permit the user to navigate between various windows, bringing any one of the windows into a focus view at any given time. The present invention may represent the windows as surfaces or facets on a geometric object, such as a cube for example, and may employ animation when a user navigates from one window to another. The present invention may use “world-in-miniature” (or “WIM”) techniques in addition to, or instead of the representation of the windows on a geometric object, to facilitate navigation.
• Finally, the present invention may facilitate the generation of queries, to be launched via an information browser, from words or phrases of free form notes. The present invention may recognize relevant facts (e.g., in the context of a trip planner, recognize dates, places, etc.) in one information type to generate related information in another information type.
§ 3. BRIEF DESCRIPTION OF THE DRAWINGS
• The file of this patent contains at least one drawing executed in color. Copies of this patent with color drawing(s) will be provided by the Patent and Trademark Office upon request and payment of the necessary fee.
• FIG. 1A is a personal computer on which the user interface of the present invention may be effected.FIG. 1B is a machine on which the user interface of the present invention may be effected.
• FIGS. 2A and 2B are displays of default views of a user interface in accordance with an exemplary embodiment of the present invention.FIG. 2C is a display of folders, such as that ofFIGS. 2A and 2B, arranged in a box.FIG. 2D is a display of the box of folders ofFIG. 2C on a desktop.
• FIG. 3 is a display of a calendar window, in a focus view, of a user interface in accordance with an exemplary embodiment of the present invention.
• FIG. 4A is a high level diagram of a display, andFIG. 4B is an actual display, of a map window, in a focus view, of a user interface in accordance with an exemplary embodiment of the present invention.
• FIG. 5A is a high level diagram of a display, andFIG. 5B is an actual display, of a browser window, in a focus view, of a user interface in accordance with an exemplary embodiment of the present invention.
• FIG. 6A is a high level diagram of a display, andFIG. 6B is an actual display, of a bulletin board window, in a focus view, of a user interface in accordance with an exemplary embodiment of the present invention.FIG. 6C is a display of a note, in a preferred viewing location, the contents of which may be edited.FIG. 6D is a display of a browser query launched based on the contents of a note.FIG. 6E is a display of a location on a map based on contents of a note.
• FIG. 7 is a diagram of states and state transitions in an exemplary embodiment of the present invention.
• FIG. 8 is a diagram of processes which may be used to effect certain aspects of an exemplary embodiment of the present invention.
• FIGS. 9 through 12 depict data structures which may be used in an exemplary embodiment of the present invention.
• FIG. 13 is a flow diagram of an exemplary main routine process in accordance with an exemplary methodology of the present invention.
• FIG. 14 is a flow diagram of an exemplary process to get related information, which process may be called by the main routine process ofFIG. 13.
• FIG. 15 is a flow diagram of an exemplary process to update viewing plane states, which process may be called by the main routine process ofFIG. 13.
• FIG. 16 is a flow diagram of an exemplary process to manage a bulletin board, which process may be called by the process ofFIG. 15.
• FIG. 17 is a flow diagram of an exemplary process to manage a map, which process may be called by the process ofFIG. 15.
• FIG. 18 is a flow diagram of an exemplary process to manage a calendar, which process may be called by the process ofFIG. 15.
• FIG. 19 is a flow diagram of an exemplary process to generate visual links, which process may be called by the main routine process ofFIG. 13.
• FIG. 20 is a flow diagram of an exemplary process to generate a focus view, which process may be called by the main routine process ofFIG. 13.
• FIGS. 21A through 21C are displays depicting alternative map and calendar windows.
• FIG. 22 is an alternative default display of a user interface in accordance with the present invention.
• FIG. 23A is a display depicting an alternative calendar window.FIG. 23B is a table of graphical representations which may be used in the alternative calendar window ofFIG. 23A.
• FIG. 24 is a display depicting an alternative map window.
• FIG. 25 is a display depicting alternative combined calendar and map windows.
• FIGS. 26 and 27 are displays depicting alternative combined calendar, map, and event views.
• FIG. 28 is a display which shows an alternative arrangement of windows on a geometric object.
• FIG. 29 is an alternative display of a trip plan.
• FIG. 30 is a display depicting alternative calendar and map views.
• FIGS. 31A and 31B are views of a display depicting alternative event and map views.
• FIG. 32 is a display depicting alternative map and calendar views.
• FIGS. 33A and 33B are views of a display having alternative map, calendar, and event views.
§ 4. DETAILED DESCRIPTION
• The present invention concerns novel methods, apparatus and data structures for providing a user interface. The following description is presented to enable one skilled in the art to make and use the invention, and is provided in the context of a particular application (i.e., travel planning) and its requirements. Various modifications to the disclosed embodiment will be apparent to those skilled in the art, and the general principles set forth below may be applied to other embodiments and applications. Thus, the present invention is not intended to be limited to the embodiments shown.
• Features of, and functions which may be performed by, the present invention will first be described in § 4.1 below. Then, structures, methodologies, data structures and displays of exemplary embodiments of the present invention will be described in § 4.2 below.
• § 4.1 Functions which may be Performed by the Present Invention
• Recall from § 1.2.3 above, that there exists a need for a user interface, and in particular a graphical user interface, for facilitating planning or making decisions based on a number of factors (or choices). Recall also that since factors may be subject to initial uncertainty or vagueness, such a user interface should facilitate an iterative planning or decision making process and should visually depict the results of alternative decisions. The present invention meets this need by providing a graphical user interface in which a number of windows or planes depict different types of related information. The consequences of changes to one type of information on one window are depicted in the other types of information on the other windows.
• Recall also from § 1.2.3 above, that relationships between information, not necessarily the same information depicted in different ways, should be depicted. The present invention meets this need by providing a graphical user interface in which the relationship between information in various windows is visually depicted, for example using color and colored rays. The colored rays may be translucent so that they do not occlude any information “behind” them.
• Recall also from § 1.2.3 above, that the user interface should permit a single display to have different windows of different types of related information such that a unified view of the task is presented to a user. The present invention meets this need by simulating a three dimensional environment including different windows or planes having the different types of information. The windows or planes may be related to one another as surfaces of a folded paper, or as facets of a geometric object that are connected with one another. However, recall from § 1.2.3 that the user interface should permit easy interaction with any one of the different displays. One way the present invention meets this need is by permitting any plane or window to be brought into a normal, head on, view (referred to as “focus view”) and depicted at a centered position on the display. However, this is not the only way a user can interact with the displays—in an exemplary user interface, all of the (rendered) windows are always active. In such an exemplary user interface, the focus view is merely provided to aid interaction for those users that prefer a larger, normal view. In this regard, the present invention may permit the user to navigate between various windows, bringing any one of the windows into a focus view at any given time. The present invention may represent the windows as surfaces or facets on a geometric object, such as a cube for example, and may employ animation when a user navigates from one window to another. The present invention may use “world-in-miniature” (or “WIM”) techniques in addition to, or instead of the representation of the windows on a geometric object, to facilitate navigation.
• Finally, recall from § 1.2.3 above, that intelligent help and access to information that may be needed in completing the task should be provided to the user. The present invention meets this need in two (2) ways. First, the present invention may generate queries, to be launched via an information browser, from words or phrases of free form notes. Second, the present invention may recognize relevant facts (e.g., in the context of a trip planner, recognize dates, places, etc.) in one information type to generate related information in another information type.
• Having described functions which may be performed by the present invention, structures, methodologies, data structures and displays which may be used by the present invention are now described in § 4.2.
• § 4.2 Structures, Methodologies, Data Structures, and Displays which may be Used by the Present Invention
• In the following, exemplary systems on which the present invention may operate are described in § 4.2.1, exemplary displays which may be generated by the present invention are described in § 4.2.2, a state diagram which illustrates exemplary interactions between the user interface and a user is described in § 4.2.3, exemplary processes and data structures which may be used to effect certain aspects of the present invention are described in § 4.2.4, flow diagrams showing an operation of an exemplary methodology of the present invention are described in § 4.2.5, and alternative displays which may be generated by the present invention are described in § 4.2.6.
• § 4.2.1 Exemplary Systems
• FIG. 1A and the following discussion provide a brief, general description of an exemplary apparatus in which at least some aspects of the present invention may be implemented. The present invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a personal computer. However, the methods of the present invention may be effected by other apparatus. Program modules may include routines, programs, objects, components, data structures, etc. that perform a task(s) or implement particular abstract data types. Moreover, those skilled in the art will appreciate that at least some aspects of the present invention may be practiced with other configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network computers, minicomputers, set top boxes, mainframe computers, and the like. At least some aspects of the present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote memory storage devices.
• Referring toFIG. 1A, anexemplary apparatus100 for implementing at least some aspects of the present invention includes a general purpose computing device in the form of a conventionalpersonal computer120. Thepersonal computer120 may include aprocessing unit121, asystem memory122, and asystem bus123 that couples various system components, including thesystem memory122, to theprocessing unit121. Thesystem bus123 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory may include read only memory (ROM)124 and/or random access memory (RAM)125. A basic input/output system126 (BIOS), containing basic routines that help to transfer information between elements within thepersonal computer120, such as during start-up, may be stored inROM124. Thepersonal computer120 may also include ahard disk drive127 for reading from and writing to a hard disk, (not shown), amagnetic disk drive128 for reading from or writing to a (e.g., removable)magnetic disk129, and anoptical disk drive130 for reading from or writing to a removable (magneto)optical disk131 such as a compact disk or other (magneto) optical media. Thehard disk drive127,magnetic disk drive128, and (magneto)optical disk drive130 may be coupled with thesystem bus123 by a harddisk drive interface132, a magneticdisk drive interface133, and a (magneto)optical drive interface134, respectively. The drives and their associated storage media provide nonvolatile storage of machine readable instructions, data structures, program modules and other data for thepersonal computer120. Although the exemplary environment described herein employs a hard disk, a removablemagnetic disk129 and a removableoptical disk131, those skilled in the art will appreciate that other types of storage media, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROM), and the like, may be used instead of, or in addition to, the storage devices introduced above.
• A number of program modules may be stored on thehard disk123,magnetic disk129, (magneto)optical disk131,ROM124 orRAM125, such as an operating system135 (for example, Windows NT® 4.0, sold by Microsoft Corporation of Redmond, Wash.), one ormore application programs136, other program modules137 (such as ReActor infrastructure and Microsoft Interactive Sound System, both from Microsoft Corporation of Redmond, Wash., for example), and/orprogram data138 for example. A user may enter commands and information into thepersonal computer120 through input devices, such as akeyboard140 andpointing device142 for example. Other input devices (not shown) such as a microphone, joystick, game pad, satellite dish, scanner, or the like may also be included. These and other input devices are often connected to theprocessing unit121 through aserial port interface146 coupled to the system bus. However, input devices may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB). The video monitor147 or other type of display device may also be connected to thesystem bus123 via an interface, such as avideo adapter148 for example. Thevideo adapter148 may include a graphics accelerator (e.g.,Intense 3D Pro 1000 or Intense 3D Pro 2200 from Intergraph Corporation of Huntsville, Ala.). One ormore speakers162 may be connected to thesystem bus123 via a sound card161 (e.g., a wave table synthesizer such as product number AWE64 Gold Card from Creative Labs of Milpitas, Calif.). In addition to themonitor147 and speaker(s)162, thepersonal computer120 may include other peripheral output devices (not shown), such as a printer for example.
• Thepersonal computer120 may operate in a networked environment which defines logical connections to one or more remote computers, such as aremote computer149. Theremote computer149 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and may include many or all of the elements described above relative to thepersonal computer120, although only a memory storage device150 has been illustrated inFIG. 1A. The logical connections depicted inFIG. 1A include a local area network (LAN)151 and a wide area network (WAN)152, an intranet and the Internet.
• When used in a LAN, thepersonal computer120 may be connected to theLAN151 through a network interface adapter (or “NIC”)153. When used in a WAN, such as the Internet, thepersonal computer120 may include amodem154 or other means for establishing communications over thewide area network152. Themodem154, which may be internal or external, may be connected to thesystem bus123 via theserial port interface146. In a networked environment, at least some of the program modules depicted relative to thepersonal computer120 may be stored in the remote memory storage device. The network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
• FIG. 1B is a moregeneral machine100′ which may effect one or more of the processes discussed above. Themachine100′ basically includes a processor(s)102, an input/output interface unit(s)104, a storage device(s)106, and a system bus ornetwork108 for facilitating data and control communications among the coupled elements. The processor(s)102 may execute machine-executable instructions to effect one or more aspects of the present invention. At least a portion of the machine executable instructions and data structures may be stored (temporarily or more permanently) on thestorage devices106 and/or may be received from an external source via aninput interface unit104.
• § 4.2.2 Exemplary Displays
• As discussed in § 4.1 above, present invention may provide a graphical user interface in which a number of windows or planes depict different types of related information. Thedisplay200 of an exemplary graphical user interface, for use in the context of planning a trip, is depicted inFIG. 2A. Thedisplay200 includes a number ofwindows210,220,230, and240. Afirst window210 represents a bulletin board upon which notes (none shown in this Figure)' containing things that a user wants to do (e.g., see a Redskins game, visit Aunt Bettie, see autumn foliage, see an aquarium, and see the Statue of Liberty) are posted. Asecond window220 is a map for depicting places (e.g., Wash., D.C., Aunt Betty's home town in Delaware, Vermont or Virginia, Baltimore or Camden, N.Y. harbor, etc.) to be visited during the vacation. Athird window230 is an information browser, such as an Internet browser for example, for finding information related to things that the user wants to do. Finally, afourth window240 is a calendar for depicting when various places will be visited during the vacation. In this example, the calendar permits three (3) alternative trips to be compared and the trip will span two (2) weeks.
• FIG. 2B is adisplay200′ in which thewindows210,220,230 and240 depict relevant information. For example, thebulletin board window210 includes a number ofnotes212 on which the user may jot down things that they want to do on their trip. The contents of thenotes212 are unconstrained—they are free form textual inputs. This facilitates initial user vagueness or uncertainty. Themap window220 depicts a map, in which places222 to be visited are highlighted, with a colored marker icon for example. The colors of the highlighting may match various colors ofrelated notes212. TheInternet browser window230 includes information which may be related to thenotes212 on thebulletin board window220, theplaces222 on themap window220, and/or to dates241 on thecalendar window240. For example, as described in more detail below, information entered on anote212 may be used to generate a query which is launched via theInternet browser230. Similarly, a place entered on a map and/or a date entered on a calendar may also be used to generate a query. Any geographic information in a web page rendered on theInternet browser230 may be visually linked with related information on themap window220, with translucent colored ray223. The information on the map may be visually linked with related information on thecalendar240, with translucentcolored ray224. Finally, thecalendar window240 includes three (3)alternative trip scenarios242. Each of the scenarios may include one ormore segments246. The segments typically have acorresponding note212 on thebulletin board window210 which has a corresponding color.
• Note that each of thewindows210,220,230, and240 can represent a sheet of a folder. Referring toFIG. 2C,various folders201 corresponding to various trips (or other objects of a decision making process) may be stored in abox representation203. As shown inFIGS. 2C and 2D, thebox203 may be arranged on the top surface of adesk representation205. Thedesk205,box203, andfolder201 metaphors may be rendered when starting and exiting the user interface of the present invention. Alternatively, a new or existing trip file may be opened in the user interface of the present invention and managed via a known file management utility, such as File Manager of Windows NT® for example.
• Recall that user interaction with any given window should be easy. By clicking the maximizebutton214,224,234, or244 of any one of the windows, that window is presented in a focus view. A user may then return to thedefault display200′ by clicking the minimize button314,414,514, or614 of any of the windows then in the focus view, which is described below. Users may act on windows in both the default and the focus view; the focus view serves to provide extra screen space to a specific window.
• FIG. 3 is adisplay300 which depicts thecalendar window240 in focus view. The displayedcalendar window240 allows three (3) alternative plans242 to be entered. Each of the three (3) alternative plans242 has a first week (among the top group of the alternatives) and a second week (among the bottom group of the alternatives).Trip intervals246 may be automatically generated from temporal information within anote212 on thebulletin board window210 and/or from temporal information extracted from a file rendered on thebrowser window230, or they may be manually added by a user. Calendar information may also be provided from a network-based groupware application such as Microsoft Exchange™. When acursor299 is positioned “on”, or relatively close to, aninterval246, “handles” for manipulating the interval may be presented. In this case, afirst handle243 may be used to drag the interval, in its entirety, forward (right) or backward (left) in time. Further handles244aand244bmay be used to move the start or end of the interval, respectively, forward (right) or backward (left) in time.
• In general, when aninterval246 is moved to an area (time or date) occupied by another interval, it will “push” that other interval. However, an interval may be fixed in time, either automatically or in response to a user command, such that when another interval is moved into its area (time), that other interval will be divided over the fixed interval, or jump, in its entirety, to the other side of the fixed interval. As shown inFIG. 3, afixed interval247 may be rendered differently fromnormal intervals246.
• If a minimizebutton310 is activated by the user, thetrip folder201 is folded up and returned to the box oftrips203. If the maximize/minimizetoggle button244 is activated,default display200′ is reentered. Finally, as shown inFIG. 3, in addition to thecalendar window240, aWIM tool 3456 is displayed. In response to a user activating (e.g., clicking) any of the faces of theWIM tool 3456, a corresponding window is displayed. TheWIM tool 3456, even if inactive, serves as a map to aid inter-window navigation by the user. Thus, as can be appreciated, when thecalendar view300 is displayed in focus view, an downward flicking gesture will cause themap window220 to be displayed in the focus view.
• FIG. 4A is a high level diagram of adisplay400, andFIG. 4B is anactual display400′, of amap window220 in a focus view. Locations of events are depicted with markers such as circles, as shown inFIGS. 4A and 4B. These circles may be colored to match colors ofcorresponding notes212 on thebulletin board window210 andintervals246 or247 on thecalendar window240. These markers may be moved by the user, for example using traditional drag-and-drop mouse gestures. Further, the user may add or delete events from the map. In one implementation, when an event is added to the map, anote212 with the name of the location is added to thebulletin board210 and a query may be launched. Finally, the user can navigate the map via zoom-in, zoom-out, and scrolling commands or command tools (not shown).
• If a minimizebutton410 is activated by the user,trip folder201 is folded up and returned to the box oftrips203. If the maximize/minimizetoggle button424 is activated, thedefault display200′ is reentered. In addition to themap window220, aWIM tool 3456 is displayed. In response to a user activating (e.g., clicking) any of the faces of theWIM tool 3456, a corresponding window is displayed. To reiterate, the WIM tool, even if inactive, aids inter-window navigation. Thus as can be appreciated, when themap window220 is displayed infocus view400/400′, a right, left, or upward flicking gesture will cause thebrowser window230, thebulletin board window210, or thecalendar window240, respectively, to be displayed in the focus view.
• FIG. 5A is a high level diagram of adisplay500, andFIG. 5B is anactual display500′, which depict thebrowser window230 displayed in focus view. Thebrowser window230 renders a retrieved web page, such as an HTML web page for example. Places (e.g., addresses)511 and/or times or dates512 may be extracted from the rendered web page and displayed on themap window220 and/or thecalendar window240, respectively. If a minimizebutton510 is activated by the user,trip folder201 is folded up and returned to the box oftrips203. If the maximize/minimizetoggle button234 is activated, thedefault display200′ is reentered. As was the case with thecalendar window240 and themap window220, in addition to thebrowser window230, aWIM tool 3456 is displayed. In response to a user activity (e.g., clicking) any of the faces of theWIM tool 3456, a corresponding window is displayed. When in thebrowser view500/500′, a left flicking gesture will cause themap window220 to be displayed in the focus view.
• FIG. 6A is a high level diagram of adisplay600, andFIG. 6B is anactual display600′, which depict thebulletin board window210 in focus view. Anote212 can be opened in response to a user command, such as clicking on an empty portion of the bulletin board for example. Ablank note212 will then be brought to a preferred viewing position (e.g., to the right foreground) as shown inFIG. 6C. Text may then be entered onto thenote212′. Similarly, an existing note may be edited by selecting (e.g., via a mouse click) the note to bring it to the preferred viewing position. From the preferred viewing location, thedefault view200′ can be reentered by deselecting (e.g., by double-clicking or a flicking gesture) thenote212′. When the note is returned to thebulletin board210, the text may be used by the system to launch a query using the browser (See, e.g.,FIG. 6D.), any dates may be indicated on thecalendar window240, and/or any places or addresses may be indicted on the map window220 (See, e.g.,FIG. 6E.).
• Thenotes212 can be repositioned on thebulletin board210 using, for example, traditional drag and-drop mouse operations.
• Referring back toFIGS. 6A and 6B, if a minimizebutton610 is activated by the user,trip folder201 is folded up and returned to the box oftrips203. If the maximize/minimizetoggle button214 is activated, thedefault display200′ is reentered. In addition to thebulletin board window210, aWIM tool 3456 is displayed. In response to a user activating (e.g., clicking) any of the faces of theWIM tool 3456, a corresponding window is displayed. As can be appreciated, when in thebulletin board view600/600′, a right flicking gesture will cause themap window220 to be displayed in the focus view.
• Recall that in each of the focus views shown indisplays300,400/400′,500/500′, and600/600′ ofFIGS. 3, 4A,4B,5A,5B,6A, and6B, respectively, a world-in-miniature (or “WIM”)tool 3456 is provided. Recall further that theWIM tool 3456 includes a miniature representation of thedisplay200′ and includes a miniature bulletinboard window representation210″, a miniaturemap window representation220″, a miniaturebrowser window representation230″, and a miniaturecalendar window representation240″. Clicking any of the miniature windows representations will bring the corresponding window into the focus view. World-in-Miniature tools are discussed in the article, R. Stoakley, M. J. Conway, R. Pausch, “Virtual Reality on a WIM: Interactive Worlds in Miniature”, downloaded from http://www.cs.cmu.edu/-stage3/publications/95/conferences/chi/paper.html.) on Jul/ 30, 1998 (referred to as “the WIM article”).
• § 4.2.3 Exemplary User-GUI Interaction (State Diagram)
• FIG. 7 is diagram illustrating states and state transitions that may be used by the exemplary trip planner of the present invention. When in a stand-by ordefault state710, the present invention may renderdisplay200′. If a cursor, the position of which may be controlled by a user input to a pointing device which may provide a two-dimensional input, is positioned over an empty spot on thebulletin board window210 and a note creation command, such as a mouse click for example, is entered, or if the cursor is positioned over anote212 on thebulletin board window210 and a note edit command, such as a mouse click for example, is entered, the bulletinboard update state720 is entered. Thisstate720 is discussed in more detail in § 4.2.5 below with reference toFIG. 16. In this state, the user may (a) create a new note or (b) edit an existing note. In either case, thenote212 being created or edited is brought to the foreground of thedisplay200′. (RecallFIG. 6C.) The note may correspond to events or themes of the vacation (e.g., visit Aunt Betty, see autumn foliage, see Redskins game, visit an aquarium, see the Statue of Liberty). The contents of thenote212 may be used to generate a query to be launched via an information browser, such as an Internet browser or a address list (or “contacts” found in the Outlook™ program sold by Microsoft Corporation of Redmond, Wash.). A query based on the contents of thenote212 may be launched when anote212 is commanded, using a mouse flickering gesture for example, to be placed back (or “posted”) on thebulletin board210. The movement of thenote212 from the foreground of thedisplay200 to thebulletin board window210 may be accompanied by an animation. For example, a note “See the Statue of Liberty” may uncover a home page of the Statue of Liberty or a note “Visit Aunt Betty” may uncover a contact file for Aunt Betty. The query may also include other information, such as a user profile for example. The user profile may be based on explicitly entered information and/or information inferred from past use of the application, or other applications, by the user. For example, if the user profile indicates that a user is an architecture buff, the note “Statue of Liberty” may also uncover architecturally significant buildings in downtown Manhattan.
• The information returned may include information relevant to one of the windows. In the context of the trip planning application, such information (which may be referred to as “factoids”) may include dates, places, addresses, proper names, etc. Thus, if, as shown inFIG. 7, the information uncovered in the query includes time or date factoids, the calendar generation or update state740 is entered. This state740 is discussed in more detail in § 4.2.5 below with reference toFIG. 18. For example, if a contact file for Aunt Betty includes her birthday, that date will be reflected in thecalendar window240 or if the Statue of Liberty home page has a date of an upcoming special event, the date of that event will be reflected in thecalendar window240.
• Similarly, if the information uncovered in the query includes geographic or address factoids, the map generation or updatestate730 is entered. Thisstate730 is discussed in more detail in § 4.2.5 below with reference toFIG. 17. For example, if a contact file for Aunt Betty includes her home address, the address, at an appropriate level (street, city, county, state) of detail will be reflected in themap window220 or if the Statue of Liberty home page has its location, that information will be reflected, at an appropriate level of detail, in themap window220.
• Naturally, if the geographic scope of the trip is defined in themap window220 or the temporal scope of the trip is defined in thecalendar window240, place or time factoids that fall within the defined geographic scope or temporal scope, respectively, will be considered particularly relevant.
• Referring to the bulletinboard update state720, if the cursor is removed from the bulletin board, thestandby state710 is reentered. Also, in the map generation or updatestate730 and the calendar generation or update state740, after the factoid inducedmap220 orcalendar240 is generated, thestandby state710 is reentered710.
• The map generation or updatestate730 may also be entered from thestandby state710 when the cursor is positioned on themap window220. Thestandby state710 is reentered when the cursor is taken off of themap window220. Similarly, the calendar generation or update state740 may also be entered from thestandby state710 when the cursor is positioned on thecalendar window240. Thestandby state710 is reentered when the cursor is taken off thecalendar window240.
• Also, from the standby state, if themap window220,browser window230,calendar window240, orbulletin board window210 is maximized, then the map focus view state750 (RecallFIGS. 4A and 4B.), the browser focus view state760 (RecallFIGS. 5A and 5B.) the calendar focus view state770 (RecallFIG. 3.), or the bulletin board focus view state780 (RecallFIGS. 6A and 6B.), respectively, is entered into. As mentioned above, the standby state is reentered from any of the focus view states when the window then in the focus view state is minimized (Recall buttons310,410,510, and610.).
• As discussed above, each of the windows may be thought of as a surface of a geometric object, in this case, as four (4) sides of a cube for example. Therefore, by rotating the cube, the window in a focus view can be changed. For example, referring toFIGS. 2A, 2B, and7, from the mapfocus view state750, a right “flicking” (e.g., holding a mouse button briefly while moving the mouse to the right) gesture causes entry into the bulletin boardfocus view state780, a left flicking gesture causes entry into the information browserfocus view state760, and a downward flicking gesture causes entry into the calendarfocus view state770. Similarly, from the information browserfocus view state760, a right flicking gesture causes entry into the mapfocus view state750. From the bulletin boardfocus view state780, a left flicking gesture causes entry into the mapfocus view state750. Finally, from the calendarfocus view state770, an upward flicking gesture causes entry into the mapfocus view state750. Transitions from one focus view state to another may be accompanied by a short rotation animation and associated audio to reinforce the cube metaphor. As discussed above, though not shown inFIG. 7 for purposes of clarity, various focus view states may be entered using a world-in-miniature (or “WIM”)tool 3456. Further, as discussed above, theWIM tool 3456, even if inactive, can serve as an inter-window navigation aide.
• Having described exemplary displays and states which may be used by the present invention, exemplary processes and data structures are now described in § 4.2.4 below with reference toFIGS. 8 through 12.
• § 4.2.4 Exemplary Processes and Data Structures
• § 4.2.4.1 Exemplary Processes
• data which may be used to effect, at least certain aspects of, the user interface of the present invention. Basically, the processing by the present invention may be thought of as a sequence of cycles. In each cycle, user inputs are accepted, states are determined, windows are updated, if necessary, based on such user inputs, and a display is rendered (and audio is output) based on the state and the contents of the windows. Referring toFIG. 8, user inputs are managed by an input management process (or more generally, “an input manager”)820. Referring back toFIG. 1A, this process may be carried out by thepointer142, theserial port interface146, andprogram modules137. Any user inputs may be provided to aninput command queue822 for buffering. The queued commands are then read by aninterface management process810.
• Theinterface management process810 will maintain a present location of a cursor. The information browser view management process (or more generally, “an information browser manager”)852, the bulletin board view management process (or more generally, “a bulletin board manager”)854, the map view management process (or more generally, “a map manager”)856, and the calendar view management process (or more generally, “a calendar manager”)858 manage thebrowser window230, thebulletin board window210, themap window220, and thecalendar window240, respectively. Each of these processes maintains status information of its associated window. If any one of these windows is maximized, or if a window representation in a world-in-miniature (or “WIM”)tool3456 is selected, the focus view generation process (or more generally, “a focus view generator”)870 is invoked. Consequently, the output generation process (or more generally, “an output generator”)830 will generate a display with the appropriate window in the focus view.
• If the window in the focus view is changed, for example by a flicking gesture or by selecting another window in theWIM tool 3456, then the focus view update process (or more generally, “a focus view update facility”)880 is invoked. The change from one window in the focus view to another may be accompanied by a storedanimation882 and asound cue884. Theoutput generation process830 generates a video display of this animation.
• If a user enters information into any of the windows, such as the entry of a note of thebulletin board window210 for example, the implicit query process (or more generally, “a implicit query facility”)840 is invoked. The query may use the information browser, such as an Internet browser or address file for example, to facilitate the query. (An exemplary implicit query method is disclosed in U.S. patent application Ser. No. 09/152,713, entitled “Methods, Apparatus and Data Structures for Providing a User Interface which Exploits Spatial Memory in Three Dimensions, to Objects, and which Visually Groups Matching Objects,” filed on Sep. 14, 1998, and incorporated herein by reference.) The results of the query are then submitted to a factoids extraction process (or more generally, “a factoids extractor”)845. If any times, dates, or places are returned, thecalendar window240 ormap window220, respectively, are updated based on such factoid information.
• Having described processes which may be used by the present invention, exemplary data structures which may be used by the present invention are described in §4.2.4.2 below.
• §4.2.4.2 Exemplary Data Structures
• FIGS. 9 through 12 illustrate exemplary data structures which may be used by the present invention. Referring first toFIG. 9, recall that notes212 can be entered or edited and posted on thebulletin board window210. Thedata structure900 includesrecords910. Each of therecords910 corresponds to anote212 and may include afield912 containing a value for uniquely identifying thenote212, afield914 containing a location of thenote212, afield916 for indicating a state (e.g., active or inactive) of thenote212, and afield918 for storing content (e.g., text) of thenote212.
• Referring toFIG. 10, recall that thecalendar window240 may include a number of alternative trip plans. Each of the trip plans includes one or more events (e.g., see a Redskins game, visit Aunt Betty, visit an aquarium, see autumn foliage, see the Statue of Liberty). Thedata structure1000 includesrecords1010. Each of therecords1010 corresponds to a trip plan and may include afield1012 for containing a unique identifier for identifying each of the alternative trip plans, fields1014 for containing the start time of each of the events, and fields1016 for containing the end time of each of the events.
• Referring toFIG. 11, recall that entering information, such as flicking anote212 onto thebulletin board window210 for example, in one window may be used to generate a query for processing by the information browser. The returnedinformation1100 includesrecords1110 corresponding to each piece (e.g., a web page or a contact card) of returned information. Eachrecord1110 may include afield1112 for containing an identifier for the related information, afield1114 for containing a related information rank, and afield1116 for containing a related information state (e.g., active or top-of-stack, inactive, next-in-stack, etc.). Therefore, if the contents of anote212 entered on thebulletin board window210 are used to generate a query launched via an Internet browser, a number of web pages may be returned. The web pages may be identified by their URL (“Uniform Resource Locator”), be rank ordered based on their perceived relevance, and be active and shown on thebrowser window230 or inactive. Thus, a number of web pages (or other content being browsed) may be cycled through by the user.
• Referring toFIGS. 2B and 12, recall that related information in the various windows are visually linked, for example, with translucent colored rays. An event defines a group of related information. That is, a Statue of Liberty event may include anote212 to visit the Statue of Liberty on thebulletin board window210, a marker at the location of the Statue of Liberty on themap window220, a home page of the Statute of Liberty on theinformation browser window230, and a date of a special event at the Statue of Liberty on thecalendar window240. As shown inFIG. 12, each event has a record1210. Each record1210 may include afield1212 for containing a temporal location of the event on thecalendar window240, afield1214 for containing a location of the event on themap window220, afield1216 for containing a location of the event on thebrowser window230, and afield1218 for containing a unique color associated with the event. Actually, since thecalendar window240 may include a number of alternative trip plans, it may show a given event at different (or the same) dates in different trip alternatives. Therefore, thefield1212 may include a location of an event for each of the alternative trip plans.
• § 4.2.5 Exemplary Methodologies
• Having described various displays, processes, and data structures above, methodologies for effecting at least some of the processes described in § 4.2.4.1 above are now described with reference toFIGS. 13 through 20.
• FIG. 13 is a flow diagram of a main routine1300 which may be run or managed by theinterface management process810. First, as shown instep1302, a user input is accepted. (Recalluserinput management process820 andinput command queue822.) Next, as shown indecision step1304, it is determined whether or not the user has entered an “exit” (or “quit”) command. If so, the main routine1300 is terminated viareturn node1306. If, on the other hand, no exit command was entered, processing continues atstep1308 where a state of an appropriate window is updated, if necessary. (Recall states720,730, and740 ofFIG. 7.) Next, related information is gathered instep1310. (Recall theimplicit query process840 and thefactoids extraction process845 ofFIG. 8.) Then, the states of the other windows are updated, if necessary, instep1312. (Recall transitions fromstate720 tostates730 and740 inFIG. 7.)
• Indecision step1316, it is determined whether a particular window is in the focus view. (Recall states750,760,770, and780 ofFIG. 7, andFIGS. 3, 4,5, and6.) If so, as shown instep1318, the particular window is displayed in the focus view. (Recall process870 ofFIG. 8 and states750,760,770, and780 ofFIG. 7.) Processing then returns to step1302. If, on the other hand, no particular window is in the focus view, as shown instep1320, then all of the windows are displayed. (RecallFIGS. 2A and 2B andstate710 ofFIG. 7.) Since all of the windows have been updated and may include related information, such related information is visually associated. Instep1324, visual links between related information in various windows are generated. (Recall process860 ofFIG. 8.) Also recall fromFIG. 12 that related information are associated with an event and that the locations of the related information in a given event are stored in records1210. Processing then returns to step1302.
• Recall fromFIG. 13, that related information is obtained instep1310.FIG. 14 is a flow diagram of anexemplary process1310′ for obtaining related information. First, an implicit query process is invoked as shown instep1410. (Recall process840 ofFIG. 8.) The query may be formed, at least in part, based on words entered on a note. Though not described in detail here, the query may also be formed, at least in part, based on a place entered on themap window220 or a date entered on thecalendar window240. Recall that information from a user profile may also be used when generating the query. Then, as shown instep1420, a factoids extraction process is invoked. (Recall process845 ofFIG. 8.) That is, if the information returned from the query includes a time or date, or a place such information is depicted on thecalendar window240 or themap window220, respectively. Theprocess1310′ is then terminated viareturn node1430.
• Recall fromFIG. 13 that the state of an appropriate window may be updated atstep1308.FIG. 15 is a flow diagram of anexemplary process1308′ for updating an appropriate one of the windows. First, atdecision step1510, it is determined whether or not the cursor is on thebulletin board window210. If so, a bulletin board view manager is invoked (Recall process854 ofFIG. 8.) as shown instep1515 and processing continues atdecision step1520. If not, processing simply continues atdecision step1520. Atdecision step1520, it is determined whether or not the cursor is on themap window220. If so, a map view manager is invoked (Recall process856 ofFIG. 8.) as shown instep1525 and processing continues atdecision step1530. If not, processing simply continues atdecision step1530. Atdecision step1530, it is determined whether or not the cursor is on thecalendar window240. If so, a calendar view manager is invoked (Recall process858 ofFIG. 8.) as shown instep1535 and processing continues atdecision step1540. If not, processing simply continues atdecision step1540. Finally, atdecision step1540, it is determined whether or not the cursor is on theinformation browser window230. If so, an information browser view manager is invoked (Recall process852 ofFIG. 8.) as shown instep1545 and theprocess1308′ is terminated viareturn node1550. If not, theprocess1308′ is simply terminated viareturn node1550.
• Recall fromFIG. 15 that the bulletin board view manager (Recall process854 ofFIG. 8) may be invoked atstep1515.FIG. 16 is a flow diagram of anexemplary process1515′ for managing the bulletin board. Basically, the bulletinboard management process1515′ permits (a) notes to be created or edited, (b) queries to be generated and launched, and (c) notes to be repositioned.
• First, regarding creating (or editing) notes, as shown indecision step1610 if the user generates a note creation input, such as a mouse click when the cursor is located on a blank part of thebulletin board window210 for example, a note is created as shown instep1612 and entered text is accepted as shown instep1614. Processing then continues atdecision step1615. Returning todecision step1610, if the user does not generate a note creation input, processing branches todecision step1615. Atdecision step1615, if an edit command, such as a mouse click when the cursor is located on anote212 for example, is entered, thenote212 is edited based on (mouse and/or keyboard) entries as shown instep1616, and processing continues atdecision step1620. If a double click is entered when the cursor is on anote212, that note will be “pulled” from thebulletin board210 and moved to a preferred viewing location as shown inFIG. 6C. If a double click is entered when the cursor is on a note in the preferred viewing location, it will be returned to thebulletin board210 at the position that it formerly occupied. Otherwise, processing simply proceeds todecision step1620. Anote212 being created or edited may be rendered at the foreground of thedisplay200′. (RecallFIG. 6C.)
• Now, regarding the generation and launching of queries, if a user launches a note-based query, for example by flicking thenote212 towards thebulletin board window210, a query, based, at least in part, on the contents of thenote212, is generated and launched. More specifically, in the context of an Internet browser, a query is run for home pages on the Internet based on the contents of the note as shown instep1630. Then, the rank ordered web pages (RecallFIG. 12) are retrieved as shown instep1640. Next, any factoids, such as times or dates and places, are extracted from the retrieved web pages as shown instep1650. Next, as shown indecision steps1660 and1670, if there are no dates and places, theprocess1515′ is terminated viareturn node1680. If, on the other hand, there are any places extracted, a relevant map is retrieved as shown instep1662. If any dates are extracted, a calendar is generated on the calendar window based on such dates, or a calendar on the calendar window is updated, as shown instep1672.
• Returning to step1620, if a query is not launched, processing continues atdecision step1690. Atdecision step1690, it is determined whether the user is commanding to move the note, such as with a mouse drag operation for example. If not, theprocess1515′ is terminated viareturn node1680. If so, the location of the note on the bulletin board is updated instep1692. A note drag may be accompanied by a scraping sound, the pitch, volume, and timbre of which may be varied based on the velocity and direction of the note drag.
• Recall fromFIG. 15 that the map view manager (Recall process856 ofFIG. 8) may be invoked atstep1525.FIG. 17 is a flow diagram of anexemplary process1525′ for managing the map. Basically, themap management process1525′ permits (a) the map to be navigated and (b) events to be relocated or deleted. First, as shown instep1710, a map is generated based on the query results. For example, if thenote212 that generated and launched a query read “visit a public aquarium”, the Baltimore and Camden aquarium home pages may have been returned. The locations of these aquariums would be extracted and amap window220, showing both Baltimore, Maryland and Camden, N.J. would be generated. As shown inFIG. 4A, the map generated may include markers, such as translucent colored circles for example, matching the color of the note, at or around Baltimore, Maryland and Camden, N.J. If the markers are rendered as translucent colored circles, the radius of these circles may be changed by user input.
• Map navigation is now described. Atdecision step1720, it is determined whether or not the user is commanding a zoom in or zoom out. If not, processing continues atstep1730. If so, as shown instep1725, such zoom in or zoom out commands are effected. Processing continues atdecision step1730. Atdecision step1730, it is determined whether or not the user is commanding a scroll east, west, north or south. If not, processing continues atdecision step1740. If so, as shown instep1735, such scroll east, west, north, or south commands are effected. Processing then continues atdecision step1740.
• Event manipulation is now described. Atdecision step1740 it is determined whether a location is deleted from the map or not. If so, the deletion is effected as shown instep1745 and processing continues atdecision step1750. Otherwise processing simply continues atdecision step1750. For example, if a map displaying and highlighting both Baltimore, Md. and Camden, N.J. is displayed, and the user decides that they want to visit the Baltimore aquarium but not the Camden aquarium, the user can delete the marker at Camden, N.J. on the map. Atdecision step1750, it is determined whether not the user is commanding an event to be moved, such as with a mouse drag for example, on the map or not. If so, such a move is effected instep1755 and theprocess1525′ is terminated via thereturn node1760. Otherwise, theprocess1525′ is simply terminated via thereturn node1760. For example, if the user has deleted Camden but later decides to visit the Camden aquarium instead of the Baltimore aquarium, they can drag the marker from Baltimore to Camden. In an alternative embodiment, rather than having a single map in which the user can zoom and pan, a fixed set of maps may be used by themap window220.
• Recall fromFIG. 15 that the calendar view manager (Recall process858 ofFIG. 8) may be invoked atstep1535.FIG. 18 is a flow diagram of anexemplary process1535′ for managing the calendar. Basically, thecalendar management process1535′ permits (a) the events to be moved on the calendar, (b) events to be deleted from the calendar, and (c) the duration of events to be lengthened or shortened on the calendar. First, as shown indecision step1810, it is determined whether a given interval on the calendar is active or not, An interval may become active when the cursor is located, or hovers, over it. If not, theprocess1535′ is terminated viareturn node1870. If so, processing continues todecision step1820.
• Atdecision step1820, it is determined whether or not the user is commanding to lengthen the active interval, for example by an outward mouse drag when the cursor is at an edge of the interface. More specifically, when an interval is active, “handles” may be rendered at its left and right edges. (Recall handles244aand244bofFIG. 3.) If the user lengthens the duration of the interval, using the edge handles for example, the interface is lengthened as shown instep1825 and processing continues atdecision step1830. Otherwise, processing simply continues atdecision step1830. Atdecision step1830, it is determined whether or not the user is commanding to shorten the active interval, for example by an inward mouse drag when the cursor is at an edge of the interface. If so, the interface is shortened as shown instep1835 and processing continues atdecision step1840. Otherwise, processing simply continues atdecision step1840.
• Atdecision step1840, it is determined whether or not the user is commanding the active interval to be deleted, for example by pressing a delete button or by dragging the interval to a trash icon. If so, the interval is removed from the calendar as shown instep1845 and processing continues atstep1850. Otherwise processing simply continues atstep1850.
• Atdecision step1850, it is determined whether or not the user is commanding to move the active interval, by a mouse drag when the cursor is at the center of the interval for example. More specifically, when an interval is active, a drag “handle” may be rendered on it. (Recall handle243 ofFIG. 3.) If so, the interval is moved as shown instep1855 and processing continues atdecision step1860. Otherwise, processing simply continues atdecision step1860.
• Atdecision step1860 it is determined whether or not the user enters a date. If so, the calendar is generated with the entered date depicted as shown instep1862 and a query, based on the entered date, may be generated instep1864, before theprocess1535′ is terminated viareturn node1870. Alternatively, or in addition, if a note (or other piece of information of an event) is active, a mouse drag input on an unoccupied dates of the calendar will generate an interval related to the event. Otherwise, theprocess1535′ is simply terminated via thereturn node1870.
• Recall thecalendar window240 may have a number of alternative trip plans242. Therefore intervals may be moved or copied from one alternative to one or more other alternatives, and an interval deleted, lengthened or shortened in one alternative will not affect an interval to the same event in one or more other alternatives. Note, however, that when an interval is automatically generated, such as when a date factoid is found in information returned in response to a query, all of the calendar alternatives may be updated to include the event. If an interval is manually entered in one calendar alternative, copies may be provided to the other calendar alternatives if the dates in question are otherwise unoccupied.
• Recall fromFIG. 13 that visual links (Recall process860 ofFIG. 8) may be generated between windows atstep1314.FIG. 19 is a flow diagram of anexemplary process1314′ for generating such visual links. Basically, for each event, a visual link is generated between information in each window related to the event. Recall fromFIG. 12 that each event has a corresponding record1210. Beginning atstep1910, since all events are to processed, an event count (E_CT) is initialized (set to 1). Next, as shown instep1920, a visual link, such as a colored (which may match the color stored in field1218) translucent ray for example, is generated from the event (marker) on the map window220 (the location of which is stored in field1214) to the event on thecalendar window240′ (the location of which is stored in field1212). Since thecalendar window240 may includemultiple alternatives242 which include intervals related to the event, the visual link is terminated at the relevant interval in an active or selected one of the calendar alternatives. Next, as shown instep1930, a visual link, such as a colored translucent ray for example, is generated from the event (marker) on themap window220 to the event on the information browser window (the location of which is stored in field1216). Thereafter, the event count (E_CT) is incremented atstep1940, Next, asstep1950 it is determined whether all of the events have been processed (E_CT>MAX). If so, theprocess1314′ exits viareturn node1960. If not, processing continues atstep1920 at which the next event is processed.
• Recall fromFIG. 13 that if a particular window is in the focus view state, then the window is displayed in focus view (Recall process870 ofFIG. 8) atstep1318.FIG. 20 is a flow diagram of anexemplary process1318′ for generating such focus views. Recall that a particular window can be brought into focus view in one of three ways; (i) if a window is maximized, (ii) if a WIM representation of the window is clicked, or (iii) if a flicking gesture is input when another window is in focus view. Accordingly, atdecision step2010 it is determined whether or not a window was maximized, for example by the user clicking its maximize (214,224,234, or244) button. If so, that window is shown head on and enlarged (i.e., in the focus view) as shown in step2015 (RecallFIGS. 3, 4A,4B,5A,5B,6A, and6B.) and processing continues todecision step2020. Otherwise, processing simply proceeds directly todecision step2020.
• Atdecision step2020, it is determined whether the user selected a representation of a window in theWIM tool3456. If so, that window is shown in the focus view as shown instep2025 and processing continues todecision step2030. Otherwise, processing simply continues directly todecision step2030.
• Finally, atdecision step2030, it is determined whether a flicking (up, down, left or right) gesture was input the by user. If so, the window above, below, to the left of, or to the right of the previous window in focus view is brought into focus view and theprocess1318′ is terminated viareturn node2040. (Recall focusview update process880 ofFIG. 8.) The transition may be accompanied by a transition animation. Otherwise, theprocess1318′ is simply terminated directly viareturn node2040.
• Having described the methodologies that may be used by the present invention, alternative displays are now introduced in § 4.2.6 below.
• § 4.2.6 Alternative Displays and Alternative Windows
• FIG. 21A is portion of analternative display2100 in which the locations2122 of events on amap window2120 are associated with the intervals related to the events on acalendar window2140 by means oftranslucent arcs2150. Note that thecalendar window2140 may use a “perspective wall” metaphor as discussed § 1.2.2.1 above. Athumb wheel tool2162 is used to select various alternative trip plans. As shown, bar meters enable a user to quickly discern a trip alternative'scost2164, fun factor2166 (for example, how many desired events will actually be accomplished by a given trip plan alternative) andtime2168.
• FIG. 21B is a similar display, but further conveys travel periods and relates such travel periods to routes on a map. More specifically, a time period T_SAcorresponds to a travel route from Seattle to location A, a time period T_ABcorresponds to a travel route from location A to location B, and a time period T_BCcorresponds to a travel route from location B to location C. Note that the travel periods may be indicted with a car (or plane or train or boat)icon2170. Note further that the travel periods on the calendar are black since each of these times will typically correspond to time between “events”, rather than an event itself. The locations of events on the map window are associated with the intervals related to the events on a calendar window by means of colored,perspective view ladders2150. Finally.,buttons2184 and2182 may be used to progress or backtrack, respectively, through a trip.FIG. 21C is adisplay2100″ similar to that2100′ ofFIG. 21B but which associates the locations of events on the map window with the intervals related to the events of the calendar window by means of colored,translucent rays2150′.
• FIG. 22 is another alternative display having amap window2220, acalendar time line2240, andinformation2230 in the form of web pages. The information may represent the desired events. The pages of information are arranged on thecalendar time line2240 and are visually linked, vialines2150, to markers2222 on themap window2220. The depth of each page may be used to represent the duration of the event, depicted on the web page, on the timeline.
• FIG. 23A is analternative calendar window2340 in whichmultiple trip alternatives2341 are depicted. In thisalternative calendar window2340, ambiguity or uncertainty with respect to time is indicated by “washed out” color on an event interval.Elements2343 and2345 depict events with certain durations.Elements2342 and2345 depict travel “legs” (which may be temporally between events or destinations for example) of known durations.Element2346 depicts a discrete event to occur at an uncertain time, whileelement2347 depicts a discrete event to occur at a known time. Finally,element2348 depicts an event having a known start time but an unknown end time, whileelement2349 depicts an event having unknown start and end times.
• FIG. 23B is a table2380 of graphical representations that may be used to depict common combinations of time-slot status and event, destination, or travel-leg status in the calendar ofFIG. 23A. Generally, the upper visual representation represents the duration of an event, destination, or travel leg, while the lower bar represents the available slot of time in the trip, within which an event, destination, or travel-leg may take place. More specifically, the columns of the table2380 represent a time (e.g., a date and time), and the rows of the table2380 represent duration.
• Rows2391 and2394 include representations that depict events, destinations, or travel legs having fixed durations.Rows2392 and2395 include representations that depict events, destinations, or travel legs having uncertain durations.Row2393 includes representations that depict events, destinations, or travel legs that occur almost instantaneously, that is, having a duration of about zero (0).
• Column2381 includes representations that depict events, destinations, or travel legs having known start and end times.Column2382 includes representations that depict events, destinations, or travel legs having known start times but uncertain end times.Column2383 includes representations that depict events, destinations, or travel legs having uncertain start times but known end times.Column2384 includes representations that depict events, destinations, or travel legs having uncertain start and end times.Column2385 includes representations, similar to those ofcolumn2382, but in which the known start time has been fixed.Column2386 includes representations, similar to those ofcolumn2383, but in which the known end time has been fixed.
• FIG. 24 is an alternative display having amap window2420 on whichroutes2424 of alternative trip plans are depicted. Thelines2422 at the markers are used to distinguish one alternative route from another. The duration of segments of the trip are indicated by the spacing of thelines2422—the denser the display of thelines2422, the longer the duration of the segment.
• FIG. 25 is another alternative display having a facetted cylindrical calendar (each of the facets of the facetted cylinder, not shown, contains one of various alternative trip plans)2540 which may be rotated to bring various trip plan alternatives to the fore. Themap window2520 includeslines2522 which indicate travel between events.
• FIG. 26 is yet another alternative display having amap window2620, acalendar time line2640, and event representations2610 (spherical markers)Lines2650 may visually link some of theevents2610 to thecalendar time line2640 and themap window2620. Someevents2610 are not associated with the time-line2640 or themap2620. Note thatevent2610′ has three alternative2660 dates—(i) Saturday, (ii) Sunday and Monday, or (iii) Monday and Tuesday.
• FIG. 27 is still another alternative display having amap window2720,events2710, and acalendar window2730. Again,lines2750 visually link anevent2710 with related date and location information. The projection from eachevent2710 to the time-line2730 indicates when the event occurs.
• FIG. 28 is an another alternative display. Rather than faces of a cube, thewindows2812 are shown as facets of ageometric object2810. A world-in-miniature (or “WIM”)tool2820 is provided to assist in navigating the facets of thegeometric object2810, particularly when a window is in focus view.
• FIG. 29 is yet anotheralternative display2900 which depicts a first person view of a trip.Events2910 are marked in the display. A user can progress forward inspace2942 or back inspace2944 from one location to another. Indications of event locations that are not immediately viewable in the visible portion of the map may be provided. Finally, indications of event locations that are “behind” the virtual location and orientation of the user may be provided. Such indications allow the user to easily shift their view to event locations that are not currently in the field-of-view. Acalendar time line2940 is also provided.
• FIG. 30 is still another alternative display in which events are shown as disks3012 having map views on a face. Awindow3040 may include a calendar time line. The thickness of the disks indicates the duration of the event. Aroute3050, which may be flattened in a focus view, indicates a route from a destination of a first event to a destination of a temporally adjacent event. In this alternative display, the route is represented as aflexible ribbon3050. The start of theflexible ribbon3050 is attached to a start disk3012 corresponding to a location at the start of the trip. Similarly, the end of theflexible ribbon3050 is attached to an end disk3012 corresponding to a location at the end of the trip. Thus, if theribbon3050 and disks3012 were laid out on a flat surface, the alignment of each piece would correspond to its location on a traditional flat projection map.
• FIGS. 31A and 31B are views of adisplay3100 depicting alternative event and map views. More specifically, in this case, events are depicted pictorially, rather than textually. Themap3110 includes markers3120 related to locations of the events. A pictorial representation3130 of the event is displayed adjacent to an associated marker3120. Aroute3140 between events is also depicted. A user can navigate through their trip, traveling along theroute3140.
• FIG. 32 is adisplay3200 depicting alternative map and calendar views. More specifically, areas3220 centered on the locations of events on amap3210 are brightened. A translucent rod3240 connects the map location of each event to a corresponding time length box3250. The time length boxes3250 are projected onto aplan3257 as shadows3255. The shadows3255 may be labeled with corresponding days of the week as shown.
• Finally,FIGS. 33A and 33B are views of adisplay3300 having alternative map, calendar, and event views. As was the case with thedisplay3200 ofFIG. 32 in thedisplay3300,areas3320 centered on the locations of events on amap3310 are brightened. (FIGS. 33A and 33B are alternatives that superimpose route indications on a map.) Atranslucent wall3360 has a base which follows a route between the locations of events on themap3310. A height of thewall3360 corresponds to the length of time into the trip. This height value may be continuous as shown or may be quantized to days, hours, minutes or seconds, for example. The exact time is indicated by the segment's projection onto the time-line surface3257. The translucent wall is projected onto a longitude-calendar (or alternatively latitude-calendar)plane3367 to formprojection3365. The longitude-calendar (or latitude-calendar)plane3367 may have horizontal lines corresponding to days. Further, vertical lines, corresponding the longitude (or latitude) of the location of the events may be provided on the longitude-calendar (or latitude-calendar)plane3367. As can be appreciated, predominately east-west travel is best projected onto a longitude-calendar plane3367, while predominately north-south travel is best projected onto a latitude-calendar plane (now shown).
• In this example, the events are represented pictorially rather than textually. Thepictorial representations3330 of the events may be arranged on parallel projections from the longitudinal locations of the events on the map.
• § 4.3 Conclusions
• In view of the foregoing, the present invention provides a user interface for facilitating a decision making process, such as planning a trip. The present invention provides a unified view of various types of information related to an event. The unified view may be presented in a simulated three-dimensional environment. A window showing a particular type of information may be brought into a focus view for closer inspection by a user. Alternative decisions may be depicted to permit vagueness or uncertainty, particularly at early iterations in the decision making process.
• Although the described embodiments concern a user interface for planning a trip, the present invention can be applied to other decision making processes. Although the information related to an event were described as free form textual notes, a map, a calendar, and an Internet browser, other types of information related to an event may be depicted. For example, pictorial representations of events may be used instead of, or in addition to the free form textual notes. (RecallFIGS. 31A, 31B,33A and33B.) Also, a list of contacts, “to do” notes, scheduled appointments, etc., associated with an Outlook™ application or documents associated with a Word application (both from Microsoft Corporation of Redmond, Wash.) can be browsed or searched rather than, or in addition to, web pages on the Internet. Further, although events were entered in free form textual notes and queries for related information were launched based on the contents of such notes, information can be entered and queries launched based on information from any of the windows. For example, a user can enter a marker on the map, and a query can be launched based on the location of the marker on the map. Thus, the present invention is not intended to be limited to the embodiments described above

Claims (20)

1. A user interface that facilitates decision making, comprising:

a reception component that receives one or more expectational event; and

a rendering component that displays the one or more expectational event on a first facet of a polyhedral display object.

2. The user interface ofclaim 1, further comprising one or more information sources that include disparate information regarding the one or more expectational event.

3. The user interface ofclaim 2, the one or more information sources employed to gather, based at least in part on the one or more expectational event, a first datum of information and a second datum of information.

4. The user interface ofclaim 3, the first datum of information projected by the rendering component on a second facet of the polyhedral display object

5. The user interface ofclaim 4, the rendering component projects the second datum of information on a third facet of the polyhedral display object.

6. The user interface ofclaim 5, further including a mapping component that generates at least one non-occlusional colored ray to associate the first datum of information, the second datum of information and the one or more expectational event.

7. The user interface ofclaim 6, the rendering component projects the at least one non-occlusional colored ray that associates the first datum, the second datum and the one or more expectational event into three-dimensional space to connect the first, second and third facets of the polyhedral display object.

8. The user interface ofclaim 5, the first facet, the second facet and the third facet of the polyhedral display object focusable based at least in part on a user action.

9. The user interface ofclaim 8, the user action comprises holding a mouse button while moving a mouse in a cardinal direction.

10. The user interface ofclaim 3, the first datum of information and the second datum or information dynamically adjustable based on changes in the one or more expectational event.

11. The user interface ofclaim 10, the first datum of information dynamically changeable based on changes in the second datum of information.

12. The user interface ofclaim 10, the second datum of information dynamically changeable based at least in part on changes in the first datum of information.

13. The user interface ofclaim 2, the one or more expectational result employed to form a query to search the one or more information sources.

14. A method that facilitates decision making, comprising:

receiving an inclinational event; and

displaying the inclination event on a first plane of a three-dimensional display object rendered on a visual display.

15. The method ofclaim 14, further comprising utilizing a plurality of data sources to retrieve a first item of information and a second item of information relating to the inclination event.

16. The method ofclaim 15, further including projecting the first item of information and the second item of information onto a second plane of the three-dimensional display object, and a third plane of the three-dimensional display object respectively.

17. The method ofclaim 15, further comprising associating a non-occluding colored ray with the first item of information, the second item of information, and the inclinational event.

18. The method ofclaim 17, further comprising projecting the non-occluding colored ray into one or more three-dimensional space within the three-dimensional display object to connect the first plane, the second plane and third plane of the three-dimensional object.

19. The method ofclaim 15, the inclinational event employed to generate a query to search the plurality of data sources.

20. A user interface that facilitates decision making, comprising:

means for receiving an aspirational event;

means for rendering a multi-dimensional object;

means for obtaining one or more information associated with the aspirational event;

means for projecting the aspirational event and the one or more information associated with the aspirational event onto a plurality of planes that comprise the multi-dimensional object; and

means for associating a non-occluding visible ray with the aspirational event and the one or more information projected onto the plurality of planes that comprises the multi-dimensional object.

Images