TABLE 1

	Recommendation rule
User	(reason for recommendation)	Feedback

1	X → A	Positive
1	X → B	(none)
2	Y → A	Negative
3	Y → A	(none)
4	X → A	Negative
5	X → A	Negative

Applying the vote-tallying approach described above produces the vote totals shown in Table 2. No entry appears in Table 2 for X→B because this recommendation rule did not receive any feedback.

TABLE 2

Recommendation	Positive	Negative
Rule	Votes	Votes

X → A	1	2
Y → A	0	1

In this example, it is assumed that a total vote count of three is required before the system relies on the vote/feedback data for a given rule. Because the total vote count for Y→A does not meet this threshold, the system does not use the rule's vote data to adjust the system's reliance on the rule.

For X→A, the total vote count meets the minimum threshold of three. Because the ratio of positive votes to total votes for this rule is relatively low (0.333), the recommendation system may automatically decrease its reliance on this rule to generate recommendations. For example, the system may filter out, or lower the display rank of, recommendations that are based on this rule.

The feedback-basedadjuster54 may filter out recommendations, or lower the display ranks, using a probabilistic or other algorithm that ensures that particular recommendation rules will not be permanently “blacklisted.” For example, suppose a particular recommendation rule has a low positive-to-total vote ratio, such as 2/10. If the system merely filters out all recommendations that are based on this rule (or otherwise discontinues use of the rule to recommend items), no additional feedback will be collected for the rule; thus, the recommendation rule will remain blacklisted, even if the low vote ratio is the result of aberrational user activity. To avoid this scenario, the adjuster may only filter out the recommendations some percentage of the time (e.g., 90%), such that the rule's vote ratio has an opportunity to recover. A similar approach may be used for lowering the display ranks of recommended items.

Theadjuster54 may also occasionally increase the display ranks of items in the recommendation set using an algorithm that seeks to collect a statistically significant sample of feedback data for each recommendation rule. For example, in the example of Tables 1 and 2, the system may temporarily boost the display rank of recommendations based on Y→A to increase the likelihood of obtaining a statistically significant quantity of vote data for this rule.

Numerous variations to the particular implementation shown inFIG. 1 are possible. For instance, rather than counting all votes equally, more weight can be given to votes from certain types users, such as frequent users or “top 100 reviewers.” More weight can also be given, for example, to a five-star rating than to a four-star rating. As another example, when a given recommendation is based on multiple recommendation rules, the feedback vote may be apportioned between these rules; for instance, if negative feedback is given on a recommendation that is based on two recommendation rules, one half of a negative vote may be counted for each recommendation rule. As yet another example, therule assessor50 may periodically generate a score for each recommendation rule, and these scores may be used in place of the vote counts to determine whether/how to adjust the recommendations.

Another variation is to move the task of taking the aggregated feedback into consideration to therecommendation engine34 itself. For instance, if a given recommendation rule has a significantly lower than average positive-to-total vote count ratio or score, therecommendation engine34 may give proportionally less weight to that recommendation rule when generating personalized recommendations. This tends to reduce the frequency with which the rule is used to generate recommendations that are seen by users. With this approach, theadjuster54 shown inFIG. 1 may be eliminated. As depicted by the dashed line inFIG. 1, this approach may be implemented by periodically updating therules database32, or some other data repository accessed by therecommendation engine34, with rule-specific vote statistics and/or scores.

II. Generation and Use of Recommendation Rules

The recommendation system may use any of a variety of different types of recommendation rules to generate the personalized recommendations. Typically, each recommendation rule partially or fully specifies a condition to be checked, and specifies an item (or possibly a group of items) to be recommended or nominated for recommendation to users who satisfy the condition. The condition may, for example, be that a particular item, set of items, event, or set of events must be present in an event history or item collection of the target user. A recommendation rule may also include an optional weight value that specifies a strength of the association between the condition and the item.

As one example, the recommendation rules can include or consist of rules of the following form: item A→item B (0.8). This rule represents an association between items A and B with a strength or weight of 0.8 on a scale of zero to one. (Rules of this type are sometimes referred to as “item-to-item mappings,” “item similarities,” or “item-to-item similarity mappings.”) This rule may, for example, be used to select or nominate item B to recommend to a user who has purchased, indicated ownership of, or otherwise indicated an affinity for, item B. Methods for generating and using this type of recommendation rule are described in U.S. Pat. Nos. 6,266,649 and 6,853,982.

As depicted inFIG. 1, the recommendation rules may be generated automatically by a recommendationrule mining component60 that periodically analyzes aggregatedevent history data62 of a large population of users (typically millions of users). These event histories may, for example, include purchase histories, item viewing histories, rental histories, item download histories, search histories, item tagging activities, or some combination thereof. For instance, if purchase histories are used, the rule item A→item B (0.8) may be generated based on an observation that a relatively large number of users who purchase item A also purchase item B. If item viewing histories are used, the rule may be generated based on an observation that a relatively large number of users who select item A for viewing also select item B for viewing during the same browsing session. Although asingle mining component60 is shown, different software modules and algorithms may be used to detect different types of associations and generate different types of rules.

The particular type of user behavior used to generate a recommendation rule may govern how that rule is used to generate recommendations. For example, a rule that specifies a purchase-based association between two items may be used to generate recommendations that are based on items the target user has purchased or owns. On the other hand, a rule that specifies an item-viewing based association between two items may be used to generate session-specific recommendations that are based on the items viewed by the user during the current browsing session. U.S. Pat. No. 6,853,982, referenced above, includes examples of algorithms that may be used to generate and use purchase-based and item-viewing-based recommendation rules in the form of item-to-item mappings.

The following are examples of other types of recommendation rules that may be mined from collected behavioral data and used to generate recommendations.

- {A and B}→C. This type of rule associates a pair of items with a target item. If the rule is derived from user purchase histories (meaning that users who purchase both A and B also tend to purchase C), it may be used to nominate C for recommendation to a user who has purchased both A and B. Recommendation rules of this type may be generated using various known methods, including but not limited to those described in U.S. Pat. Pub. 2006/0106665.
- Time-ordered purchases with delay. This type of rule specifies that users who purchase a first item tend to purchase a second item after a designated amount of time. For example, therule mining component60 may detect that a relatively large number of users who purchase a particular printer also purchase a particular replacement ink cartridge three to five months later. Based on this observation, a recommendation rule may be generated that causes the ink cartridge to be recommended (or nominated for recommendation) to any user who both (a) has purchased the printer three to five months ago and (b) has not yet purchased the ink cartridge. This type of rule is also useful for recommending books and videos that are sequels. Examples of data mining methods that may be used to generate this type of recommendation rule are described in application Ser. No. 10/945,547, filed Sep. 20, 2004, the disclosure of which is hereby incorporated by reference.
- Search-based rules. This type of recommendation rule associates a particular search query with a particular item. Such a rule may be generated based on an observed tendency of users who submit the search query to select the item for viewing, purchase, rental, or some other purpose. This type of rule may be used to generate personalized recommendations that are based on the target user's search history. Examples of data mining methods that may be used to generate associations between particular search queries and particular items are described in U.S. application Ser. No. 10/966,827, filed Oct. 15, 2004, the disclosure of which is hereby incorporated by reference.
- Source event type, item A→target event type, item B. This type of rule associates two items based on two different types of item-related events. As one example, the rule “view item A→purchase item B” may be generated to reflect that a large percentage of those who view item A thereafter purchase item B. Examples of event types that may be used as source and target events include, but are not limited to, the following: purchase, view, rent, review, rate, rate favorably, add to cart, add to wish list, add to rental queue, save to library, tag, forward, print. This type of rule may be used to recommend item B for purposes of event type B to a user who has performed event type A on item A. Examples of data mining methods that may be used to generate this type of recommendation rule are described in U.S. application Ser. No. 10/864,288, filed Jun. 9, 2004, the disclosure of which is hereby incorporated by reference.
- Email domain based rule. This type of recommendation rule associates a particular email address domain with a particular item or set of items. These rules may be generated by analyzing user purchase histories, or another type of item-related event history, to identify items that are unusually or uniquely popular among users having particular domain names in their email addresses. For example, if a particular book is significantly more popular among users with “microsoft.com” in their email addresses than in a more general user population, a recommendation rule may be created that links the “microsoft.com” domain to this particular book. (In this example, the association reveals that the book is likely unusually popular among Microsoft employees.) The rule may then be used to recommend the book to users who have this domain name in their email addresses. Examples of statistical methods that may be used to create these types of rules are described in U.S. Pat. No. 6,963,850, the disclosure of which is hereby incorporated by reference.
- Geography-based rules. This type of recommendation rule is similar to an email domain based recommendation rule, but associates a particular zip code, city, or other geographic identifier with a particular item. This type of rule may be generated by analyzing user event histories (e.g., purchase histories) in conjunction with users' shipping addresses to identify items that are unusually popular in specific geographic regions. Such a rule may be used to select items to recommend to users having mailing addresses falling in these geographic regions. Examples of statistical methods that may be used to create these types of rules are described in U.S. Pat. No. 6,963,850, referenced above.

Recommendation rules may also be generated based partly or wholly on content-based associations between items. For example, a content analysis component may compare the text, attributes, and/or other types of content of particular items, and create rules that associate items having similar or related content. In addition, recommendation rules can be generated that are based on a combination of content-based and behavior-based associations, as described in U.S. application Ser. No. 11/424,730, filed Jun. 16, 2006, the disclosure of which is hereby incorporated by reference.

The foregoing are merely examples. Numerous other types of recommendation rules can be used, and can be assessed using the feedback methods described above.

As depicted inFIG. 1, therecommendation engine34 may, in some embodiments, generate recommendations based on the event history of the target user, or based on some collection of items identified from the user's event history. For example, if the recommendation rules are of the form item A→item B (w) and are derived from user purchase histories, therecommendation engine34 may generate the recommendations based on the following collection of items, or a selected subset of these items: (a) items the target user has purchased, (b) items for which the user has indicated ownership, (c) items the user has rated, or has rated favorably. The recommendations may be generated and displayed in real time based on some action of the target user, such as selection of a “view your recommendations” link.

As mentioned above, multiple recommendation rules may be used in combination to generate a particular recommendation of an item. For instance, suppose the following purchase-based recommendation rules exist:

- item 1234→item 5678 (0.3)
- item 2345→item 5678 (0.6)
  If the target user has purchased both item 1234 and item 2345, both of these rules may be invoked in combination to nominate or select item 5678 for recommendation. In this situation, the weights associated with these two rules may be appropriately combined (e.g., added) to generate a recommendation score for the recommendation of item 5678. (The weights themselves may be used as the recommendation scores where only one rule is invoked.) The recommendation scores may be used to rank all of the nominated items relative to one another for purposes of display, and/or to filter out (inhibit recommendations of) items having low recommendation scores.

Where recommendation scores are generated as described above, theadjuster54 may adjust these scores based on the user feedback data (e.g., positive-to-total vote ratio) for the corresponding recommendation rules. Alternatively, the weight values of particular recommendation rules may be adjusted upward or downward based on the received feedback on those rules, so that the recommendation scores will reflect the user feedback.

III. Assessment of Recommendation Rules (FIG.3)

FIG. 3 illustrates one example of a process that may be used by therule assessor50 to compile statistical data or scores for particular recommendation rules. This process may be executed periodically (e.g., hourly, daily or weekly) to analyze the recommendation feedback data collected since the most recent iteration. The statistical data may additionally or alternatively be updated in real time as feedback events occur.

Instep80, the logged recommendation and feedback events for the most recent time period are retrieved. The length of the time period, and thus the frequency with which new feedback data is analyzed, may depend on the nature of the items involved. For instance, for products such as books and DVDs, the collected feedback data may be analyzed on a relatively infrequent basis, such as daily or weekly. For items such as news articles that are popular for much shorter periods of time, the feedback data may be analyzed more frequently, such as hourly or in real time. The recommendation rules may, but need not, be re-assessed more frequently than they are regenerated by therule mining component60.

Instep82 ofFIG. 3, a join operation is performed to identify those recommendation events that resulted in feedback. This step enables each feedback event to be associated with a particular recommendation rule or set of recommendation rules. As explained above, the identities of the invoked recommendation rules may alternatively be captured when the feedback events are recorded.

Instep84, the process cycles through each joined recommendation/feedback event pair and counts the total number of votes, and the total number of positive votes, for each recommendation rule. As explained above, if the recommendation interface ofFIG. 2 is used, all rating events (including “not interested” ratings) may be treated as votes, and ratings of three to five stars may be treated as positive votes. As will be apparent, numerous other options are available for compiling the recorded feedback events into meaningful statistical information or scores.

As further explained above, a given feedback event need not be counted as a single vote for each of the recommendation rules involved. For example, the vote may be apportioned among the invoked recommendation rules. As another example, the vote amount may be varied based on the type of the feedback event and/or based on information known about the user. As mentioned above, another option is to disregard feedback on recommendations that resulted from multiple recommendation rules; with this approach, a feedback event is counted only if it can be uniquely tied to a single recommendation rule.

Instep86, the results (vote counts for the current time period) for each recommendation rule are persistently stored. In addition, these results are combined with the results from the last N time periods to generate cumulative vote counts that are used to adjust the recommendation process. For instance, if the time period is one day, the vote totals for the most recent day's worth of feedback data may be combined with the results from the immediately preceding nineteen days to generate cumulative vote statistics for a 20-day “moving window.” In combining the results for multiple consecutive time periods, more weight may optionally be given to more recent results. For instance, a linear or non-linear time-based decay function may be used to weight the constituent sets of results. Rather than disregarding old data, an exponential time-based decay function may be applied to collected data from all time periods, with this approach, all of the collected feedback data is considered, but progressively older feedback data is given progressively less weight.

The cumulative vote totals are represented inFIG. 1 by the vote statistics table52. If time periods of one day are used, this table52 may be re-generated on a daily based.

IV. Example System Architecture (FIGS.1 and4)

The various components shown inFIG. 1 can be implemented using software modules executed by one or more general purpose computers (physical machines) or servers. The recommendationrule mining component60 and therule assessor50 may be implemented as respective programs that are executed periodically in an off-line processing mode. Therecommendation service30 may be implemented using service code that generates recommendations in real time in response to requests from other entities. Therecommendation user interface36 may be implemented in a combination of executable code and web page templates. The executable code of the foregoing

components

30,36,50 and60 may be stored on any type of computer storage device or medium.

The

data repositories

32,40,42 and62 shown inFIG. 1 may be implemented using any type or types of computer storage, and may be implemented using databases, flat files, or any other type of computer storage architecture. The statistics table52 may be cached in the volatile RAM of one or more machines of therecommendation service30 to provide a high degree of real time performance, or may be stored on a separate machine or service.

FIG. 4 illustrates one particular example of how the system ofFIG. 1 may be incorporated into a web-basedelectronic catalog system100 that provides functionality for users to browse and make purchases from an electronic catalog of items. The system shown inFIG. 4 includes one or more web server machines104 (collectively “web server”) that generate and serve pages of a host web site in response to page requests fromuser computing devices102. Theweb server104 dynamically generates the requested pages using a repository oftemplates110. The templates directly or indirectly specify the service calls that are made to various services to, e.g., request data needed to generate the requested page.

Theweb server104 provides user access to a catalog of items represented in adatabase108 or collection of databases. The items may include or consist of items that may be purchased via the web site (e.g., book, music, and video titles in physical or downloadable form; consumer electronics products; household appliances; apparel items, magazine and other subscriptions, etc.). Thedatabase108 may also store data regarding how the items are arranged within a hierarchical browse structure. Data regarding the catalog items and the browse structure is accessible via acatalog service106, which may be implemented as a web service.

As users browse and make purchases from the electronic catalog, the system records various types of user-generated events, such as detail page views, shopping card adds, wish list adds, item rating events, tagging events, purchases, and/or search query submissions. As illustrated inFIG. 4, the event histories, and data derived from such event histories, are stored in a data repository62 (e.g., one or more databases or data services). For example, thedata repository62 may store users' purchase histories, item viewing histories, search histories, item ratings, item tags, wish lists, etc. The purchase histories and item viewing histories may be stored as lists of item identifiers together with associated event timestamps. The various types of user data may be accessible to other components of the system via a data service (not shown), which may be implemented as a web service.

With further reference toFIG. 4, therecommendation service30 responds to requests from theweb server104 by generating and returning ranked lists of recommended items together. Although therecommendation service30 may identify many hundreds or thousands recommended items in response to a single request, the ranked list it returns in one embodiment contains only a selected subset of these items (i.e., the top 15 items, or the next 15 items when the user selects “more results”). The web server uses these lists, together with item descriptions obtained from thecatalog service106, to build recommendation pages of the type shown inFIG. 2. In some embodiments, the recommendation service may also return one or more reasons for each recommendation, and theweb server104 may incorporate these reasons into the web page (as inFIG. 2).

Each time a recommendation page is generated and returned, the system logs a set of recommendation events identifying the recommended items listed on the page and the recommendation rule(s) that led to each such recommendation. This task may be performed by therecommendation service30, or, as depicted inFIG. 4, by theweb servers104. In addition, each time a user provides feedback on a recommendation, theweb server104 logs a feedback event. The recommendation events and feedback events may be recorded in a log file, in a relational database, or in any other type of data repository.

In embodiments that take implicit feedback into consideration, the system may detect some types of implicit feedback events by analyzing user clickstreams or event histories. For example, the event history of a particular user session may be analyzed to determine whether the user purchased a recommended item at some point after viewing a recommendation of that item. This may be accomplished using well known log analysis techniques.

As illustrated inFIG. 4, the recorded recommendation events and feedback events are analyzed by arule assessment component50. In some embodiments, therule assessment component50 may also take other types of data into consideration in assessing a recommendation rule, such as how frequently the rule is invoked, how frequently it is invoked in combination with another rule to make a recommendation, how long the rule has been in existence, etc.

Therule assessment component50 passes rule assessment data (e.g., statistics and/or scores) to therecommendation service30. Therecommendation service30 uses the rule assessment data to refine its reliance on particular recommendation rules using one or more of the methods described above. In one embodiment, therecommendation service30 only uses the rule assessment data to filter out, or to lower the display positions of, particular recommendations. In another embodiment, therecommendation service30 also uses the data to boost the rankings of items, and/or to cause items that would otherwise be filtered out (due to their low recommendation scores) to be presented to the user.

V. Conclusion

Each of the processes and algorithms described in the preceding sections may be embodied in, and fully automated by, code modules executed by one or more computers or computer processors. The code modules may be stored on any type of computer-readable medium or computer storage device. The processes and algorithms may also be implemented partially or wholly in application-specific circuitry. The results of the disclosed processes and process steps may be stored, persistently or otherwise, in any type of computer storage.

The foregoing description is intended to illustrate specific embodiments of the invention, and not to define or limit the invention. Thus, nothing in the foregoing description should be construed to imply that any particular feature or component is essential to the invention. The invention is defined by the claims, which are intended to be construed without reference to any definitions that may be explicitly or implicitly included in any incorporated-by-reference materials.