US20120272171A1

Movatterモバイル変換

Info

Publication number: US20120272171A1
Application number: US13/091,620
Authority: US
Inventors: Keiji Icho; Ryouichi Kawanishi
Original assignee: Panasonic Corp
Current assignee: Panasonic Corp
Priority date: 2011-04-21
Filing date: 2011-04-21
Publication date: 2012-10-25
Also published as: CN102959549B; US9348500B2; CN102959549A; US20130097542A1; WO2012144225A1; JPWO2012144225A1; JP5982363B2

Abstract

The information appliance displays content in a visually perceptible grid from which the user selects a target content and upon selection of the processor automatically identifies related content each with an associated relatedness score. Movement by the user of the target content causes the related content to move and follow the target content as if attracted by an invisible spring force or tensile force. The system thus presents the user with a graphical representation of moving items of content which are attracted to the related content based on the degree of relatedness. In this way the user quickly learns how to control selection and organization of related content by mimicking movement of physical objects acting under kinematic forces that mimic natural objects.

Description

FIELD

The present disclosure relates generally to organization, categorization and extraction of computerized content, including computerized images, data and icons. More particularly the present disclosure relates to computer-implemented technology to assist a user in extracting and reorganizing desired content using a graphical user interface that models content as computer-generated physical objects in display space having kinematic properties that map to content relatedness properties. The system associates relatedness between targeted content and other content with a computer-generated physical parameter.

BACKGROUND

Computerized content can take many forms. In photographic applications the content is typically stored as raw image data files or as compressed image files (e.g., jpg format). In video applications the content is typically stored as a collection of image frames encoded using a suitable CODEC (e.g., mpeg format). Text applications may store content as generic text files, as application-specific files (e.g., Microsoft Word doc or docx format), or as printable files (e.g., pdf). Some applications store content comprising both text and image data. Examples include presentation software applications (e.g. Microsoft PowerPoint). Database applications typically store text and numeric data, and sometimes also image data according to a predefined data structure that assigns meaning to the stored data. Icon organizing and editing applications store icons as image data, in some cases with additional metadata.

When a user wishes to organize, categorize and extract content from a software system, such as those identified above or others, the process has heretofore been tedious and far from intuitive. Typically the software system requires the user to interact with a complex system of menus, dialog boxes or commands to achieve a desired content selection. Where the content includes a lot of non-text content, such as photographs, images, movies and the like, interaction becomes even more difficult because text searching techniques are not highly effective and may not even be available

Where the content data store is large, such as with a large collection of stored photographic images, the task of organizing, categorizing and extracting desired content can be quite daunting. There are some automated tools that can be used to categorize image content, based on image characteristic extraction, face/object recognition, and the like. However, these tools often retrieve too many hits, many of which the user must then manually reject.

SUMMARY

The disclosed system associates relatedness between targeted content and other content with a physical parameter. In this way, the disclosed system provides a user-friendly, natural way for a user to organize, categorize and extract content from a data store, such as a data store of digitized images or other visual content.

The system maps content relatedness (degree of relationship) onto computer-generated physical object properties. In the computer-generated display space, the items of content are depicted as moveable objects to which the physical object properties are associated. Using a suitable touch gesture, or pointing device selection operation, the user can select and move a desired item of content. In so doing, other related items of content move as if attracted to the selected content by an invisible attractive force (e.g., invisible spring force, invisible gravitational force, or other kind of force). Thus by dragging a selected content item, the items of related content will follow, exhibiting kinematic motion as if they were physical objects acted upon by the invisible attractive force, where the degree of relatedness defines the strength of that force. Thus strongly related contents are attracted by a stronger force than less related content. Thus by simply watching the content movement the user can tell how closely the content items relate to the selected content.

Although rendered in computer-generated display space, the selected item of content, and those related to it, move as if mimicking the behavior physical objects. The user thus learns quite quickly and naturally how to organize, categorize and extract content, simply by touch and drag or (click and drag) movements.

A general concept is to use some kind of physical parameter to represent the relatedness between targeted content and other contents. Examples of such physical parameters include but are not limited to: (a) force acting between related content (or icons) and targeted content (or icons), such as tensile force and/or attractive force; (b) speed with which related content (or icons) come close to targeted content (or icons); (c) final relative position of related content (or icons) to final position of targeted content (or icons); and combinations thereof.

Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

DRAWINGS

The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

FIG. 1 is a plan view of an exemplary information appliance, illustrating how the user organizes, categorizes and extracts content;

FIG. 2 is a schematic block diagram of a computer hardware implementation of the information appliance ofFIG. 1;

FIG. 3 is a user interface diagram, illustrating in greater detail the user interface components of the information appliance ofFIG. 1;

FIG. 4ais a detailed user interface diagram, showing the grid component of the user interface ofFIG. 3 and illustrating how items of content are related prior to movement of a selected content by the user;

FIG. 4bis a detailed user interface diagram, showing the grid component of the user interface ofFIG. 3 and illustrating how items of content move according to a user-designated trajectory and further illustrating how items of content are rearranged during such movement according to their respective degrees of relatedness;

FIG. 5 illustrates an alternate embodiment that features a control mechanism that permits a user to adjust a relatedness threshold or correlation metric threshold which regulates how many items of content are attracted during movement along the user-designated trajectory;

FIG. 6 is a software block diagram, illustrating the manner of programming the computer hardware ofFIG. 2, it being understood that the depicted software is stored in the computer memory and operated upon by the CPU;

FIG. 7 is a flowchart diagram depicting one preferred embodiment whereby selected content is analyzed and motion of that content is generated by the suitably programmed computer hardware;

FIG. 8 is a graphical representation of one embodiment of a computer-implemented model for controlling how motion of content is generated by the suitably programmed computer hardware, the model featuring a display space reflecting how items of content are positioned and move within the display space of the computer screen;

FIG. 9 is a flow chart diagram explaining the operation of the information appliance according to the model ofFIG. 8;

FIGS. 10a-10dillustrate the information appliance in use, performing a basic screen transition according to user operation;

FIGS. 11a-11dillustrate the information appliance in use, creating a personal category for an individual user or the user's family;

FIGS. 12a-12dillustrate the information appliance in use, creating a different category from the same content;

FIGS. 13a-13dillustrate the information appliance in use, creating a compound cluster from plural previously created clusters;

FIG. 14ais a detailed user interface diagram, showing an alternate embodiment of how items of content are related prior to movement of a selected content by the user according to a tree structure;

FIG. 14bis a detailed user interface diagram, illustrating how items of content ofFIG. 14amove according to a user-designated trajectory and further illustrating how items of content are rearranged during such movement according to their respective degrees of relatedness and based on the tree structure;

FIG. 15 is a graphical representation of the alternate embodiment ofFIGS. 14aand14b, namely a computer-implemented model for controlling how motion of content is generated by the suitably programmed computer hardware, the model featuring a display space reflecting how items of content are positioned and move within the display space of the computer screen according to a tree structure; and

FIG. 16 illustrates a variation of the embodiment ofFIGS. 14aand14bthat features a control mechanism that permits a user to adjust a relatedness threshold or correlation metric threshold which regulates how many items of content are attracted during movement along the user-designated trajectory.

Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.

DETAILED DESCRIPTION

Example embodiments will now be described more fully with reference to the accompanying drawings.

As noted above, the computer-implemented apparatus and method for organization, categorization and extraction of computerized content, defines a relationship between relatedness between targeted content and other content using a predefined physical parameter. The present description will explain in detail how to implement different examples of such apparatus and methods, using different examples of predefined physical parameters. By way of non-limiting example, the predefined physical parameter can be a kinematic-related parameter, such as be a force, a speed, a relative position, or the like.

In this regard, an exemplary physical parameter can be the force acting between related content (or icons) and targeted content (or icons), such as tensile force and attractive force. Such force is generated by computer according to the following relationships:

{right arrow over (F)}=k_i({right arrow over (x)}_i−{right arrow over (x)}_T)

- {right arrow over (F)}: Force acting between related content/icon i and targeted content/icon T
- k_i: Parameter depending on relatedness between related content/icon i and targeted content/icon T (k_i>0)
- {right arrow over (x)}_i: Position of related content/icon i
- {right arrow over (x)}_T: Position of targeted content/icon T

Alternatively, the physical parameter can be a speed parameter, representing, for example, the speed with which related content (or icons) are attracted to the target content (or target icon). Such speed is generated by computer according to the following relationships:

{\vec{x}}_{i} (t) = {\vec{x}}_{i} (t - Δ t) + \frac{{\vec{x}}_{i} (t - Δ t) - {\vec{x}}_{T} (t - Δ t)}{l_{i}}

- {right arrow over (x)}_i(t): Position of related content/icon i at time t
- {right arrow over (x)}_T(t): Position of targeted content/icon T at time t
- l_i: Parameter depending on relatedness between related content/icon i and targeted content/icon T (l_i>1)

Alternatively, the physical parameter can be a position parameter, representing, for example, the final position of related content (or icon) relative to the targeted content (or icon). Such relative position is generated by computer according to the following relationships:

Final relative position of related content/icon i, {right arrow over (r)}_i={right arrow over (x)}_i,FINAL−{right arrow over (x)}_T,FINAL, is set depending on relatedness between related content/icon i and targeted content/icon T. For example, related contents are assigned at the final time in decreasing order of relatedness, most closely-related first, as below:

Here, in terms of speed with which related content/icon comes close to the position of {right arrow over (x)}_T+{right arrow over (r)}_i, it becomes like the following;
${\vec{x}}_{i} (t) = {\vec{x}}_{i} (t - Δ t) + \frac{{\vec{x}}_{i} (t - Δ t) - {{\vec{x}}_{T} (t - Δ t) + {\vec{r}}_{i}}}{l}$

If desired, the physical parameter can be comprised of combinations of parameters, such as combinations of a speed parameter (as described above) and a relative position parameter (as described above). In this regard, the following considerations would be applicable:
Related content/icon i comes close to the position of {right arrow over (x)}_T+{right arrow over (r)}_i({right arrow over (r)}_iis set depending on relatedness between related content/icon i and targeted content/icon T) and the force acting between the related content/icon i and the position of {right arrow over (x)}_T+{right arrow over (r)}_ibecomes like the following;
{right arrow over (F)}=k_i({right arrow over (x)}_i−({right arrow over (x)}_T+{right arrow over (r)}_i))

- k_i: Parameter depending on relatedness between related content/icon i and targeted content/icon T (k_i>0)

Alternatively, if desired, the physical parameter can be comprised of other combinations, such as combinations of speed and relative position. In this regard, the following considerations would be applicable:
Related content/icon i comes close to the position of {right arrow over (x)}_T+{right arrow over (r)}_i({right arrow over (r)}_iis set depending on relatedness between related content/icon i and targeted content/icon T) and the speede with which related content/icon comes close to the position of {right arrow over (x)}_T+{right arrow over (r)}_ibecomes like the following;
${\vec{x}}_{i} (t) = {\vec{x}}_{i} (t - Δ t) + \frac{{\vec{x}}_{i} (t - Δ t) - {{\vec{x}}_{T} (t - Δ t) + {\vec{r}}_{i}}}{l_{i}}$

It will of course be understood that other implementations, using other physical parameters may also be used. Thus the above examples of physical parameters are not intended as limiting examples.
To understand how the apparatus and methods for associating relatedness with physical parameter(s) described herein may be used, an information appliance will be featured. Again, it will be understood that this information appliance is merely an example of a device that may use the teachings herein.
Referring toFIG. 1, an exemplary information appliance has been illustrated at20. The information appliance has a touch-enableddisplay screen22 upon which the touch-enabled user interface is displayed. In the illustrated example ofFIG. 1, the displayed user interface comprises a plurality of different regions that the user can interact with using touch gestures. While a touch-enabled information appliance is shown here to illustrate the principles of the invention, it will be understood that other types of devices and other types of user interfaces, supporting other types of user interaction are possible. Thus, for example a computer device having a mouse or stylus driven interface could also be used.
For purposes of illustrating some of the principles of the invention, the depicted information appliance is adapted to manage image content, such as photo library content. It will be understood, however, that the principles of the invention can be applied to other types of content, such as video content, textual content, hypertext content, database content and the like. Thus where photographic images and thumbnail images or icons are described below, the reader will understand that the displayed images could represent a different type of content, such as video content, textual content, hypertext content, database content and the like.
As illustrated inFIG. 2, the information appliance is preferably implemented using a computer architecture that includes a central processing unit orCPU26 coupled to abus28, to whichrandom access memory30 andstorage memory32 are also attached. The computer architecture may also include an input/output (I/O) module attached tobus28 to facilitate communication with external devices via any suitable means such as wired connection or wireless connection. Adisplay driver36 is coupled to thebus28 to support thetouch display22. To simplify the illustration, thedisplay driver36 ofFIG. 2 includes the necessary circuitry to drive the visual display and to receive the touch input commands produced when the user performs a touch gesture upon the touch display. In this regard, as illustrated inFIG. 1, the user performs a touch and drag operation to effect selection of related content, as will be more fully discussed herein.
Referring toFIG. 3, the user interface displayed inFIG. 1 is now shown in greater detail. In the exemplary application illustrated inFIG. 1, the information appliance organizes and displays image content, such as photographs within a user's personal photo collection. The managed content is preferably organized into different classes or groups using automatic categorization technology. In the user interface ofFIG. 3, the category groups are displayed graphically as thumbnail depictions or icons within the predefined region of the display screen at40. The user can select one of the applicable categories by suitable touch gesture. InFIG. 3, the category designated at42 has been so selected.
Once the user selects a category, the user interface then displays individual thumbnail or icon representations of individual pieces of content belonging to that category. These are displayed in a grid as at44. The user can then select from that grid one or more individual pieces of content by suitable touch selection. By way of example, inFIG. 3 the user has selected the content at46. Once selected, the user interface displays an enlarged view of the selected contents withinwindow48.
In some instances, the displayed content may, itself, comprise identifiable sub-components. For example, the displayed content may include several individually identifiable objects, such as buildings, geographic features, animals, human faces and the like. InFIG. 3, the displayed content withinwindow48 comprises a photograph featuring three persons' faces. If desired, these identifiable sub-components can be used to define a query by which the system searches for additional related content.
Thus, for example, by selecting one of the displayed persons' faces via touch gesture, the system uses the selected person to initiate a query to retrieve other related content (e.g., other images in which that person appears). The system will performs the query against all associated content, such as all content within the selected category group, to generate similarity scores associated with each element of the category group. Based on the results of the recognition algorithms, the images are each given their own similarity score. Images in which the selected person appears are given a high similarity score, whereas images lacking that person are given a low similarity score.
Of course, it will be understood that the specific similarity matching algorithms used will depend on the type of content being managed. In the case of image content such as photographic content and video content, face and object recognition algorithms as well as image characteristic extraction techniques would be used. In other applications, such as database applications, other database query techniques would be used.
The user interface further includes afamily space region50 into which the user can drag selected content that he or she desires to be associated into a subset or family of the category. As will be more fully explained below, one aspect of the present technology is to provide an easy to use and intuitive way of extracting related content for placement into the family.
Referring toFIGS. 4aand4b, the user extracts content for inclusion in the family by a dragging operation whereby the user selects a target content from grid44 (e.g., target46a) and drags that content tolocation46bwhich lies outside the confines ofgrid44.
More specifically, when the user selects a target content, such astarget content46ainFIG. 4a, those additional pieces of content that are related tocontent46a(by virtue of the automatic categorization technique being used) are highlighted as illustrated. Related content which have a high similarity score or high relationship score are preferably depicted in a more prominent fashion, such as by highlighting those pieces of content in a visually perceptible manner and also by displaying connecting lines that also connote a strong connection or relationship. InFIG. 4a, the strongly related content are shown at52a,54aand56a. Additional content with a lower degree of relatedness are graphically depicted in a different way to connote the lower degree of relatedness. This may include shading or highlighting the related content in a more subdued fashion and also by generating connecting lines that are less prominent that those used to convey strongly related content. InFIG. 4a, these lesser related content are shown at58a,60aand62a.
The display of related information in this fashion can support multiple levels of relatedness. Thus inFIG. 4a, additional content at64aand66aare illustrated with light shading, to convey a degree of relationship withtarget46athat is less than any of the other related pieces of content. In addition to using light or light highlighting the connecting lines may also be rendered using a lighter shade to convey a less prominent or less bold relationship.
As illustrated inFIG. 4b, when the user drags the target content away from its resting place as depicted inFIG. 4a, the related content follows themotion trajectory70 of the target content. Thus, inFIG. 4b, the target content is shown beyond the confines ofgrid44 as at46b. Note how the related content have followed the target content.
When the target content is moved via the dragging gesture, the associated content generally follows thesame trajectory70 as the target content. In one preferred embodiment, the associated content become spatially reorganized while following thetrajectory70, so that the associated content with a higher degree of relatedness becomes arranged closer to thetarget content46bthan the related content having a weaker relationship. This has been illustrated inFIG. 4b.
In one preferred embodiment, each related piece of content “follows” the target content as if it were attached by an invisible spring having a spring force that is proportional to the degree of the relationship. Thus, closely related content, such ascontent items52b,54band56bare pulled toward thetarget content46bby an invisible spring force that is stronger than the spring force that pulls less related content, such ascontent items58b,60b,62band so forth. Thus, whereas the initial positions of the pieces of content are distributed according to thegrid44, as illustrated inFIG. 4a, the individual pieces of content are reordered according to the degree of relationship (strength of relationship) as the target content is moved by the user as illustrated inFIG. 4b.
To enhance the visual effect, the attractive force (invisible spring force) may be buffered or tempered by introducing a velocity-sensitive component that resists the attractive spring force. This velocity-sensitive component may be modeled as if each interconnecting link between target and related component includes a velocity-sensitive “dashpot.” Employing both a spring force and a retarding velocity-sensitive force, the force acting upon each item of related content may be expressed as F=k dx/dt−c dv/dt.
The effect of the velocity-sensitive component is to make the movement of individual content items somewhat sluggish, so that the motion and response to the invisible spring force is not instantaneous. An alternate way of expressing the relationship would be to think of the items of content as moving through a viscous medium, so that changes in the position of thetarget content46bare not instantaneously mimicked by a comparable instantaneous change in the position of all related content. Rather, the related content will continue to coast to their new positions for a short time after the target content has already stopped.
The visual effect produced by the velocity-sensitive component is to slow down the motion of the content following the target content, so that the user is able to see the strongly related content outpace the less related content as each moves to its final clustered position. Because the invisible spring forces attracting each piece of content to the target content depend on the individual relationship strength, the more strongly related items are attracted more quickly and thus tend to situate themselves most closely to the target content when the target content finally comes to rest.
In another embodiment, the related content items follow the target content in a more complex kinematic relationship whereby the overall number of related items attracted during motion of the target content can be controlled by how quickly the user moves the target content. In this embodiment, if the user moves thetarget content46bslowly, then even weakly related content will follow thetrajectory70. On the other hand, if the user moves thetarget content46bquickly, then only target content above a certain threshold will follow. The effect is as if the weaker interconnecting links (carrying the invisible spring force) can be broken if the speed of movement of the target content exceeds a certain threshold. As will be more fully explained, the threshold may be velocity dependent, so that the user can actually control how many items of related content are pulled away from thegrid44 by simply controlling how quickly he or she moves the target content.
In yet another embodiment, depicted inFIG. 5, a gesture-operatedcontrol72 is provided to set the threshold and thus control to what extent the degrees of relationship or layers of linkages with the target content are attracted as the target is moved. The user rotates thecontrol72 in a clockwise direction to increase the number of related content items and counterclockwise to decrease the related number of items.
Referring now toFIG. 6, the computer programming used to implement embodiments of the disclosed system and method will now be discussed. Specifically,FIG. 6 shows the software components and manner of programming (CPU26,FIG. 2) to effect the content categorizing, selecting and graphical display to effect the following motion trajectory as discussed above. The software components may be loaded into memory30 (FIG. 2) and are then acted upon byCPU26 to produce the above-described behaviors when the computer program is run. If desired, these components can be incorporated into or associated with the operating system of the information appliance20 ofFIG. 1.
For purposes of illustrating the principles of the invention, certain ones of the provided software modules are specifically adapted for handling visual image processing, such as face recognition and object recognition. Other components are more general in nature and are adapted for extracting features of any creative find description, which can include not only features extracted from visual content (photographs, motion pictures, and the like) but also other data types as may be applicable to more general purpose data mining applications.
As diagrammatically depicted at100, the computer program and software modules used to implement the functionality described above may comprise afunctional block100 performs the content categorization and presentation through the graphically user interface of the information appliance. Thisfunctional block100 comprises a categoryreorganization user interface102 that in turn employs several software components. As illustrated, one of the basic functions of the category reorganization user interface is categorization of the content. Thus, one of the illustrated functions of the category reorganization interface is the function ofgeneral categorizing104. Depending on the application involved, this general categorizing can involve certain additional sub-categorizing aspects. Illustrated here are four such aspects, namely face recognizing106, object recognizing108, feature extracting110 and content tagging112. These categorizing software modules work as follows.
When the content is a photograph, for example, theface recognizing module106 identifies regions within the image that represent faces using a suitable face recognition algorithm that analyzes the image to detect features corresponding to a subject's face, such as eyes, nose, cheekbones, jaw, etc.
Theobject recognizing module108, like theface recognizing module106, performs feature identification. However, whereas the face recognizing module is specifically designed to recognize features found in the human face, the object recognizing module is more general and is thus able to recognize objects such as buildings, geographic features, furniture and the like. Both the face recognizing module and the object recognizing module may be implemented using a trained system that is capable of learning by extracting features from known faces and known objects. Both face recognizingmodule106 and object recognizingmodule108 thus rely upon the general feature extracting capabilities of thefeature extracting module110.
In some instances, the user may have applied tags to certain content and also to certain features found within that content. Thecontent tagging module112 administers this functionality. Thus, for example, if the person identifies a certain face as belonging to his or her daughter, the portions of the image corresponding to the daughter's face may be tagged with her name. Whereas, the feature extracting techniques operate upon elements that are inherent to the image itself, content tagging involves additional metadata that is added by the user or is added as a result of some query or grouping having been performed.
In use, when the user interacts with the information appliance through the touch display22 (FIG. 2), thegeneral categorizing module104 and its associatedsub-modules106,108,110 and112, are called into action when needed to organize the content into different categories. With reference toFIG. 3, such categories may be displayed in category groups as in40.
The category reorganizationuser interface module102 further includes software modules that handle the user interaction of selecting a target content within grid44 (FIG. 3), defining relationships between the target content and other content as well as handling all of the connecting wire visualization and following motion processing as was described in connection withFIGS. 4a,4band5.
Thus, the category reorganization user interface includes a selected contentposition determining module114, which functions to interpret which target content has been selected by the user when the user touches one of the content items within grid44 (FIG. 3). The contentrelationship analyzing module116 works in conjunction withmodule114, as well as the general categorizing modules, to determine which additional pieces of content are related to the one the user has selected. This determination includes associating a relatedness score (or correlation metric) to each piece of content that is related to the target content selected. In this regard, a numerical score may be assigned to the relationship. For example, a relatedness score of 0-100% may be assigned. A 100% relationship would denote a very strong relationship to the target content, whereas a 0% score would denote the absence of a relationship. Thus, relationships between the target content and the remaining content can vary over a suitable range as required by the data being analyzed.
Once the contentrelationship analyzing module116 has performed its function, the connectingwire visualizing module118 generates connecting wires or lines between the target content and the related content. As discussed and illustrated in connection withFIGS. 4a,4band5, the connecting wires may be visually depicted using different boldness or intensity values to denote different degrees of relatedness. For example, content items having a relatedness score of 75%-100% would be given a strong bold appearance, scores between 50% and 74% would be given a less bold appearance, scores between 25% and 49% would be depicted using a light line or dotted line, and so forth. Depending on the application, content items having a similarity score below a certain threshold, such as below 25%, may be given no connecting wire visualization and would thus be considered “not related”. As an alternative to controlling the boldness or intensity of the connecting wires, different colors may be used to indicate different levels of relatedness.
As discussed in connection withFIGS. 4aand4b, the category reorganization user interface produces a user-friendly visualization whereby related content follow the trajectory of the target content as the user moves the target content from thegrid region44 to the family space region50 (FIG. 3). As discussed, the individual items of related content are treated as if they are connected by an invisible spring which produces a pulling force causing related content to follow the target content as the user moves it. This pulling force or tensile force is calculated inmodule120. Further details of this calculation will be discussed below.
The tensile force or spring force is used by the followingmotion processing module122, which associates a motion trajectory with each of the pieces of related content. To give the visual display a user-friendly, natural presentation, the followingmotion processing module122 causes each of the related content to follow a trajectory generally in the direction of the target content whereby the pulling force acting on each item of content is equal to the tensile force associated with that piece of content.
If desired, a velocity-sensitive motion-resisting counterforce or dashpot may be associated with each piece of related content to give the effect that the related content moves toward the target content through a viscous medium so that the related content items reach their final destination after the target content has ceased to move. The produced visual effect makes it appear that the related content are being pulled by elastic strings that stretch when the target content is moved and that continue to pull the associated content towards the target content, through a viscous medium, after the target content has come to rest.
Because closely related content is attracted more strongly (stronger tensile force) to the target content, such associated content will naturally cluster more closely to the target content than less strongly related content. The category reorganizationuser interface module102 must therefore organize the relocated content items after motion is effected; otherwise, the related content may overlap and be difficult to visualize. To handle this, the related contentposition determining module124 defines a boundary about each piece of content and applies a rule dictating that the related content will be positioned radially adjacent the target content based on relatedness score, with the further provision that the related content items shall be repositioned so that the individual content items do not overlap on another.
Finally, having assembled a cluster of related content, the category reorganization user interface, through its madecategory reorganizing module126, associates recognition information with the newly formed cluster. This allows the cluster to be tagged and saved for recall at a later time and also to be used as a starting point for performing further content recognition. For example, the user might select a first category group, such as group42 (FIG. 3) and then perform the above-described selection operation to assemble a cluster of related content items. The user could then save that assembled cluster and then use it as a basis for searching through a different category group selected from the category groups forFIG. 3.
For a better understanding of the software modules, refer toFIG. 7, where the process flow implemented bymodules118,120,122 and124 has been illustrated. This process flow represents one presently preferred method for remapping the target content and associated content to a new category, as described above. Thus, atstep150 the process determines the selected content position sequentially. This step is performed by module114 (FIG. 6) when the user selects a target content. Upon selection of a target,module124 identifies the related content atstep152 and then further ascertains the current position of the related content within the grid44 (see, for example,FIG. 4a). In this presently preferred embodiment, the individual pieces of selected content are processed sequentially. That is, the process depicted inFIG. 7 is implemented as a loop whereby each item of content is sequentially processed. However, due to the speed of the CPU, the user perceives the individual content items as moving simultaneously as the user drags the target content towards thefamily space region50.
Atstep154, the system calculates the tensile force (invisible spring force) sequentially for each item of content. In this presently preferred embodiment, the tensile force can be modeled as a spring force according to the formula F=kx, where k is proportional to the degree of relationship between that content and the target content. While a linear relationship is presently preferred, non-linear relationships may be used instead to achieve a different attractive force profile between the target content and the related content.
In accordance with the linear relationship F=kx, when the displacement (x) between the target content and the related content changes upon movement of the target content by the user, the tensile force becomes non-zero and may be calculated by the stated formula. As noted, in this preferred embodiment, each item of content is treated individually and each item may have its own tensile force value, depending on the particular degree of relatedness.
Having calculated the tensile force for the given related content, at step156 a motion calculation is performed to determine how the related content will move as the target content is moved by the user. Using a physical object analogy, motion of the related content can be calculated using the equation F=ma, where m is a standardized mass (which can be the same value for all pieces of content) and a is the acceleration produced by the force F. Because the mass of all content may be treated as equal, it is seen that the applied force (the tensile force for that piece of content) is proportional to the acceleration produced.
Thus, the motion process determines an acceleration value for each piece of related content. This acceleration value is then used to calculate the motion that the related content will exhibit. Such motion is, of course, a vector quantity. That is, motion of the related content proceeds in a certain direction as dictated by the following motion model implemented by themodule122. In this presently preferred embodiment, the motion model is based on an analogy whereby each item of related content is attracted to the target content by an invisible spring force (the tensile force) between them. Thus, the vector direction of motion of the related content is towards the center of the target content. Accordingly, as the user moves the target content, each item of related content will be attracted to and thus follow the trajectory of the target content, corresponding to step158.
If desired, in order to give the visual appearance a more realistic “real world” feel, the following motion calculation can include a velocity-sensitive, dashpot, term that tends to resist instantaneous changes in motion, thereby making the related content appear to move as if they were immersed in a viscous medium. While not required, this additional velocity-sensitive term makes movement of the related content lag behind movement of the target content. Thus, when the user stops moving the target content, the related content will continue to coast toward their final destinations, the final destinations being determined at the points where the tensile force returns to zero or until further movement of the related content is blocked because another piece of content already occupies the space.
In addition to computing the motion of each piece of related content, the process also generates the connecting wires or lines between each item of related content and the target content. This is performed atstep160. Specifically, this step defines a line segment between the centers of the respective thumbnail images. As discussed above, the boldness or color of these line segments can be adjusted based on the degree of relatedness.
After all of the related content have been selected and moved, the collected set of content are then remapped into a new category atstep164. If desired, this step may include prompting the user to provide a category label that is then associated with the items of content. The system permits more than one category tag or label to be associated with each item of content. Thus individual items of content can belong to more than one category, if desired.
In one embodiment, the system includes a mechanism to allow the user to control how many items of related content are “attracted” to the target content during the selection and moving process. As described in connection withFIG. 5, this can be accomplished by providing a touch-enabled control wheel that the user can manipulate to adjust the threshold of which content will be captured and which will not. The control ofFIG. 5 works as follows. Thecontrol72 produces a numerical threshold value that changes over a range of values from a low value to a high value as the user rotates the control clockwise or counterclockwise. The value produced bycontrol72 is then used to set the threshold by which the system determines whether an item of content will be included or not. For example, if thecontrol72 is manipulated to a high threshold, then only content having a relatedness score of above 75% will be captured. Conversely, if the control is manipulated to a low value, then content having a relatedness score of above 25% will be captured.
In yet another embodiment, the user is able to control how much content is captured by the speed at which the user moves the target content. The embodiment models an object being pulled across a frictional surface so that the frictional force acts to oppose movement in the pulling direction. The line or wire representing the tensile force is fragile and can stretch and break if the pulling force becomes too great. Weakly associated content is mapped using a more fragile connection, resulting in weakly associated content not being selected when its connection breaks. TheFIG. 8 illustrates how this may be accomplished.
Referring toFIG. 8, the system defines an artificial affinity space (not shown) whereby all items of content are related based on how strongly a target content (or icon) is related to the other content (or icons). The system then establishes a relationship between the affinity value and a physical parameter. For purposes of illustration, the relationship can be a kinematic parameter such as object weight, where content having a low affinity value (e.g., unrelated) is assigned a relatively heavier weight, whereas content having a high affinity value (e.g., highly related) is assigned a light weight. This object weight is then mapped to thedisplay space202, where displayed objects appear to move as the target content T_Dis pulled in a certain direction by the user. Objects having a mapped heavy weight will more slowly, or not at all, based on a predetermined threshold friction assigned to the “surface” upon which the displayed objects sit in display space. Conversely, objects with lighter weight will move more freely, following the general trajectory of the target content as it moves.
Alternatively, the affinity space can map the affinity value to a tensile force parameter. Weakly related (e.g., unrelated) content objects are assigned a weak tensile force, whereas strongly related content objects are assigned a stronger tensile force. This tensile force is then mapped to thedisplay space202 so that the more strongly related the content, the more strongly it is attracted to the target content as it is pulled by the user.
As yet another alternative, the affinity space can map the affinity value to a fragility value corresponding to how strong the affinity relationship is. When the fragility value is mapped to display space, objects are connected to the target object through a link (shown as a connecting line in display space) having a strength based on the fragility value. Links with low fragility value break as the target object is pulled. In this way, unrelated or weakly related content severs its relationship with the target content and does not follow the target content as the target is pulled by the user. For example, the link represented by force F₄may represent a comparatively fragile link, depicted by a thin connecting line. This fragile link will break based on how the target content is moved.
It should be understood that the embodiments depicted inFIG. 8 are intended to illustrate a few examples of how kinematic motion behavior can be mapped onto the otherwise unrelated problem of how to select and display related content. Other models can be used instead.
To further explain the relationship inFIG. 8, refer now to the flow diagram inFIG. 9. Beginning indisplay space200, the user selects an object representing a target content at206. Selection of this object causes the system inaffinity space202 to identify related objects at208. The user then moves the selected object at210 in display space. The speed at which the user moves the selected object is captured and used in affinity space atstep212 where the system determines which objects will follow for a given movement speed. Step212 thus corresponds to the setting of the escape velocity threshold inFIG. 8. The system then instep214 assigns a tensile force for each object, based on the mapped parameter (e.g., object weight, tensile force, link material, etc.). The assigned tensile forces F_nare then supplied back to display space where they are used to cause related objects to move using their respective assigned tensile forces at216.
In the embodiment illustrated inFIGS. 4a,4band8, each of the content elements were directly attracted to the target content. Variations of this basic concept are possible. Thus, as shown inFIGS. 14a,14band15, content elements may be organized according to a tree structure. Thus certain elements are directly attracted to the target content, whereas other elements are attracted, as children, grandchildren, etc., of the directly attracted content.
Referring toFIGS. 14aand14b, the user has selected element46 (46a,46b) as the target, moving it along thetrajectory70 as illustrated inFIG. 14b. With reference toFIG. 14a, note thatelements52a,54aand56aare directly linked as having a strong affinity withtarget content46. These linked elements, in turn, have affinities for other elements thereby defining a parent-child-grandparent tree structure relationship. For example,element52ahas an affinity withelement58a, which, in turn, has an affinity withelement58a. Thus, when thetarget content46bis moved alongtrajectory70, as shown inFIG. 14b, thechild content58bandgrandchild content64bofelement52bare attracted as well.
FIG. 15 illustrates how the embodiment ofFIGS. 14aand14boperate. As shown in thedisplay space202 inFIG. 15, the items of content are capable of being attracted to one another (or pulled by one another) and thus captured by invisible tensile forces between one another. Thus, content element A_mis attracted to element A_oby force F₄; and element A_nis attracted to element A_oby force F5. In other words the attractive force is between elements that are closest in proximity to one another in display space and not necessarily directly attracted to the target content Td.
Computation of each individual force may be performed in this embodiment using essentially the same computational process as used to calculate the force for the embodiment ofFIG. 8, with the exception that inFIG. 8 all forces are attracted to the same target T_D, whereas in the present case ofFIG. 15, the target for each content element is the parent of that element. Thus the force calculation can be performed recursively, following the tree structure.
If desired, acontrol mechanism72 may be included with the embodiment ofFIGS. 14aand14b. This is illustrated inFIG. 16.

Example Use Cases

Having now explained the basic principles of the technology, some examples of the technology in use will be presented with reference toFIGS. 10a-10d,11a-11d,12a-12dand13a-13d. Referring first toFIGS. 10a-10d, there is shown a basic screen transition example whereby the user selects content (FIG. 10a), drags the selected content thereby attracting related content (FIG. 10b), maps the related content into the Family Space region (FIG. 10c) and then associates a label [Fido] to the gathered content (FIG. 10d). The associated content having been labeled, is now available for display as a new category group.

FIGS. 11a-11dshow a representative use case, similar to that ofFIGS. 10a-10d, but where the user specifically selects the portion of the photograph inFIG. 11bdepicting a dog. In other words, the user selects a portion of the image, such as the dog, and that portion is used as a basis for a relatedness query to find other images depicting that dog.

FIGS. 12a-12dillustrate a further use case, similar to that ofFIGS. 11a-11d, but illustrating that the user can create new categories based on different selected content within a given image. Thus inFIG. 12athe user selects Mt. Fuji in the image and then pulls related images containing Mt. Fuji into the Family Space atFIG. 12b. Similarly, inFIG. 12c, the user starts with the same photograph asFIG. 12a, but this time selects the blooming cherry blossoms and uses that selected content to pull related images containing blooming cherry blossoms.

FIGS. 13a-13dillustrate how it is possible to create more complex categories, based on previously created categories. Thus as illustrated inFIG. 13b, the user selects one of the images within a previously defined category. The selected image is displayed as an enlarged image in the window to the left. Then, as depicted inFIG. 13c, the user selects one of the persons in the photograph and pulls a new category containing the selected person into a family space. Note that the family space now contains both the original category cluster and the newly created one. These two category clusters may then be joined to define a composite cluster if desired.

The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.

Example embodiments are provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail.