USRE49562E1 - Methods, systems, and articles of manufacture for determining optimal parameter settings for business initiative testing models - Google Patents

Abstract

A system, method, and article of manufacture is disclosed for determining optimal parameter settings for a business initiative testing model used for testing initiatives for business locations included in a business network. In one aspect, a method is disclosed that includes defining a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting may include a set of one or more parameter setting options. The method may also include performing virtual tests on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the method may include determining a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Moreover, the method may include configuring the business initiative testing model using the optimal parameter settings from the set for the first test type to test a business initiative to apply to the business network.

Description

CROSS-REFERENCE TO RELATED APPLICATIONAPPLICATIONS

This application is a Reissue of U.S. Pat. No. 8,571,916 (previously U.S. patent application Ser. No. 11/364,197, filed Mar. 1, 2006), which is a continuation-in-part of U.S. patent application Ser. No. 10/767,191, entitled “Methods, Systems, and Articles of Manufacture for Analyzing Initiatives for a Business Network,” and filed Jan. 30, 2004, now U.S. Pat. No. 8,010,399, which is are hereby incorporated by reference in its their entirety.

DESCRIPTION OF THE INVENTION

1. Field of the Invention

This invention relates to business initiative analysis systems, and more particularly, to methods, systems, and articles of manufacture for performing a segmented initiative analysis for initiatives implemented at selected business locations in order to identify in which other locations to implement the initiative. Further, this invention relates to methods and systems for determining optimal parameter settings for business initiative testing models.

2. Background of the Invention

For as long as retailers have been selling products and services, they have been seeking ways to increase profits. Accordingly, many retailers create new initiatives that they believe will have a positive impact on profits. These initiatives usually cover various aspects of business operations that drive profits. For example, retailers may change product prices, move products to different locations of a sales floor, change the amount of space allocated to each product, test new products, add or reduce staff, introduce new concepts (e.g., in-store kiosks), remodel older stores, and test new marketing campaigns. Retailers may test these new initiatives in selected test locations (i.e., certain stores) and subsequently determine whether to introduce the initiatives to remaining business locations based on the success of the initiatives at the selected test locations. Historically, retail management has used business instinct and/or anecdotal evidence to assess the success of the initiatives in order to make a decision whether to implement the initiatives at the rest of its business locations.

In recent years, however, some retailers have become more structured and analytical in their set up and analysis of tests. These retailers collect performance metrics, such as sales and gross profit data from the test locations and analyze the data using conventional software products, such as spreadsheet or statistical software packages. Retailers may measure the change in the performance metrics at the locations that introduced the new initiatives relative to a change in the same metrics at a control group of locations that did not implement the initiatives. In doing so, the retailers attempt to identify the impact of the initiatives on the performance metrics in order to determine whether they provide a positive return on investment. This allows these retailers can make an informed decision whether to extend the concept to remaining locations in a business network. However, the way in which the analyst at the retailer sets up this analysis will have an effect on the perceived results. For example, the analyst has to decide which of the stores that did not implement the initiative should serve as the control group and the analyst has to decide what time periods to use in measuring impact. Retailers often define one or more parameters for their analysis in a manner that seems intuitive, but lack empirical evidence that the parameters are in fact optimized for isolating the impact of an initiative. As a result, retailers may not recognize and filter inconsistent data, and therefore be less able to measure the impact of their initiatives.

Accordingly, there is a need for a system and method that automatically identifies one or more analytical parameters that filter out the most inconsistent data to maximize a retailer's ability to analyze the results of an initiative test. Using more consistent data allows retailers to better identify those initiatives to extend to certain locations that will provide the most anticipated profit gains.

SUMMARY OF THE INVENTION

Methods and systems consistent with certain aspects of the present invention identify optimum parameter settings for initiative testing models.

In one aspect of the invention, a method is provided for determining parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network. In one aspect, the method includes defining a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting may include a set of one or more parameter setting options. The method may also include performing a virtual test on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the method may include determining a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Moreover, the method may include configuring the business initiative testing model using the optimal parameter settings from the set for the first test type to test a business initiative for application in the business network.

In another aspect of the invention, a system is provided for determining parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network. In one aspect, the system may include a computer configured to define a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting including a set of one or more parameter setting options. Further, the computer is configured to execute a virtual test on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the computer may determine a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Also, the computer may configure the business initiative testing model using the optimal parameter settings from the set for the first test type to a test on a business initiative to apply to the business network.

In another aspect, a computer-readable medium is disclosed that includes instructions for, when executed by a processor, performing a method for determining parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network. In one aspect, the method includes defining a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting may include a set of one or more parameter setting options. The method may also include performing a virtual test on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the method may include determining a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Moreover, the method may include configuring the business initiative testing model using the optimal parameter settings from the set for the first test type to test a business initiative to apply to the business network.

Additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of methods, systems, and articles of manufacture consistent with features of the present invention. The aspects of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several aspects of the invention and together with the description, serve to explain the principles of the invention. In the drawings,

FIG.1 illustrates an exemplary system environment in which methods, systems, and articles of manufacture, consistent with features and principles of the present invention may be implemented;

FIG.2 illustrates a flowchart of an exemplary initiative segmentation process, consistent with certain aspects of the present invention;

FIGS.3A and3B illustrate flowcharts of an exemplary initiative analysis process, consistent with certain aspects of the present invention;

FIG.4 shows a screen shot of an exemplary test site selection web page, consistent with certain aspects of the present invention;

FIG.5 shows a screen shot of an exemplary analysis set up web page, consistent with certain aspects of the present invention;

FIG.6 shows a screen shot of an exemplary initial analysis results web page, consistent with certain aspects of the present invention;

FIG.7 illustrates a flowchart of an exemplary segmentation process, consistent with certain aspects of the present invention;

FIG.8 shows an exemplary test site attribute table, consistent with certain aspects of the present invention;

FIG.9 shows an exemplary unsegmented attribute quartile result graph, consistent with certain aspects of the present invention;

FIG.10 shows a block diagram of a segmented attribute quartile comparison process, consistent with certain aspects of the present invention;

FIG.11 shows a screen shot of an exemplary attribute table web page, consistent with certain aspects of the present invention;

FIG.12 shows a screen shot of an exemplary attribute details web page, consistent with certain aspects of the present invention;

FIG.13 shows a screen shot of an exemplary model selection web page, consistent with certain aspects of the present invention;

FIG.14 shows a screen shot of an exemplary model results web page, consistent with certain aspects of the present invention;

FIG.15 shows a screen shot of an exemplary model results graph web page, consistent with certain aspects of the present invention;

FIG.16 shows an exemplary business location analysis table, consistent with certain aspects of the present invention;

FIG.17 shows an exemplary business region analysis table, consistent with certain aspects of the present invention;

FIG.18 shows a flow chart of an exemplary parameter setting process, consistent with certain aspects of the present invention;

FIG.19 shows a flow chart of an exemplary virtual test process, consistent with certain aspects of the present invention;

FIG.20 shows a flow chart of an exemplary noise determination process, consistent with certain aspects of the present invention;

FIG.21 shows a block diagram of exemplary performance data associated with a virtual test, consistent with certain aspects of the present invention;

FIG.22 shows a flow chart of an exemplary parameter setting determination process, consistent with certain aspects of the present invention;

FIG.23 shows a block diagram of an exemplary data structure with selected optimal parameter setting options, consistent with certain aspects of the present invention;

FIG.24 shows a block diagram of another exemplary data structure, consistent with certain aspects of the present invention;

FIG.25 shows a block diagram of another exemplary data structure with selected optimal parameter setting options, consistent with certain aspects of the present invention; and

FIG.26 shows a flow chart of an exemplary initiative testing process, consistent with certain aspects of the present invention.

DETAILED DESCRIPTION

Methods, systems and articles of manufacture consistent with certain aspects of the present invention identify business locations that respond most positively to a tested initiative and build a model based on the common attributes of those identified test sites to predict the impact of the tested initiative on other locations in a business network.

In one aspect of the invention, a server is disclosed for performing one or more processes for analyzing a business initiative for a business network including business locations having a set of test sites that have implemented the business initiative for a predetermined test period and a set of control group sites that did not have the initiative implemented. Each of the sites have an associated set of attributes reflecting various characteristics corresponding to the respective site, such as geographical location, size of business location, number of employees, etc. In one aspect of the invention, the server collects from a client a performance value for each of the test and control group sites reflecting a level of performance of each site during the test period. The server then calculates the change in performance in the test sites during the test period, relative to the change in performance in the control group sites over the same time period. The server may segment the performance values for each test site attribute to identify those attributes that have a greater impact on the performance values of the test sites than other attributes. Further, the system configures a model for predicting the performance values of the test sites using the identified attributes and determines whether the model accurately predicts these performance values. If so, the server applies the model to all non-tested sites (i.e., all sites inbusiness network110 excluding the test sites) to predict the performance values of each site. Based on the predicted performance values, a user operating the client or associated with the business network may select one or more of those sites to implement the business initiative.

Reference will now be made in detail to the exemplary aspects of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

The above-noted features and other aspects and principles of the present invention may be implemented in various environments. Such environments and related applications may be specially constructed for performing the various processes and operations of the invention or they may include a general purpose computer or computing platform selectively activated or reconfigured by program code to provide the necessary functionality. The processes disclosed herein are not inherently related to any particular computer or other apparatus, and may be implemented by a suitable combination of hardware, software, and/or firmware. For example, various general purpose machines may be used with programs written in accordance with teachings of the invention, or it may be more convenient to construct a specialized apparatus or system to perform the required methods and techniques.

The present invention also relates to computer readable media that include program instructions or program code for performing various computer-implemented operations based on the methods and processes of the invention. The program instructions may be those specially designed and constructed for the purposes of the invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of program instructions include for example machine code, such as produced by a compiler, and files containing a high level code that can be executed by the computer using an interpreter.

I. SYSTEM OVERVIEW

FIG.1 illustrates anexemplary system environment100 in which certain aspects related to the present invention may be implemented. As shown,system environment100 may include abusiness network110,network120, and aserver130.

Business network110 represents a network of businesses that provide similar products and/or services to customers. The businesses may be geographically dispersed.Business network110 may include one or more business regions112-1 to112-N that represent different geographical locations, such as cities, counties, states, country regions, virtual boundaries having a predetermined bounded distance (e.g., 100 mile radius), and any other type of defined region within a particular location. Each region112-1 to112-N may include one ormore business locations114 that represent retailers that sell products and/or services associated withbusiness network110. For example,business network110 may represent a grocery store chain that includes a plurality of stores geographically dispersed among a state of the United States. Each region112-1 to112-N may represent locations that include grocery stores in the chain (e.g., business locations114) that are within a predetermined distance of one another (e.g.,100 mile radius).

Also included inbusiness network110 isbusiness management unit116, which represents a business entity that managesbusiness network110. For example,business management unit116 may be an organization headquarters or a marketing division forbusiness network110.Unit116 may include one ormore clients118 that are operated by one or more users associated withbusiness network110 that perform various management and/or marketing duties forbusiness network110. Following the above example,business management unit110 may be a marketing division of the exemplary grocery store chain that determines which products and/or services each retailer location (e.g.,114) should provide.

In one aspect of the invention,client118 may be one or more computer systems such as a desktop computer, workstation, laptop, personal digital assistant or any other similar computing system operated by a user.Client118 may include a processor, associated memory, and other elements and functionalities available in computer systems (all not shown). These elements may include input/output devices, such as a keyboard, mouse, and display, although these input means may take other forms. Also,client118 may include a network interface for communicating with external resources, such asnetwork120 andserver130. Further,client118 may execute browser software that enables a user to request, receive, and render information fromnetwork120.

AlthoughFIG.1 showsbusiness network110 including a single business management unit116 (and accompanying client118), eachbusiness location114 may include aclient118 that is capable of accessingnetwork120 andserver130 to access and perform processes consistent with certain aspects related to the present invention. Further, althoughbusiness network110 is shown including a plurality of regions112-1 to112-N,network110 may includebusiness locations114 that are all included within a single region.

Network120 may be a shared, public, or private network and encompasses a wide area or local area.Network120 may be implemented through any suitable combination of wired and/or wireless communication networks. For example,network120 may be implemented through a Wide Area Network (“WAN”), Local Area Network (“LAN”), an intranet and/or the Internet. Further,network120 may represent multiple networks, such as a wireless carrier network connected to the Internet.

Server130 may be a computer system such as a desktop computer, workstation, or any other similar server side computing system that performs one or more server-side processes. In one aspect of the invention,server130 may include, among other things,interface unit132, processing unit,134,memory136, anddatabase140. Althoughserver130 is shown including elements132-140, it may include additional computing elements, such as an input/output device (e.g., display, mouse, keyboard) that enables a user to access the data and software inserver130. In addition,server130 may be a group of servers.

Interface unit132 may be one or more hardware and/or software components that are configured to receive and send information from/tonetwork120.Interface unit132 may include, or leverage, different types of software to control access to information maintained byserver130. For example,interface unit132 may execute firewall software that controls access to information and processes provided byserver130 using one or more security protocols (e.g., password processing, encryption technologies, etc. The above listed functionalities are not intended to be limiting andinterface unit132 may perform other types of interface processes to facilitate the exchange of information betweenserver130 andnetwork120.

Processing unit134 may be one or more processing devices that execute software processes stored in one or more memory devices, such asmemory136. Further, processingunit134 may execute software processes stored internally, such as internal flash memory configured withinprocessing unit134.

Memory136 may be one or more memory devices that store data and software programs that, when executed by processingunit134, perform processes consistent with certain aspects related to the present invention. These programs may includesegmentation engine137, andmodel processing engine138. Further,memory136 may include server-side software applications that enableserver130 to create and provide content including information produced by other programs executed by processingunit134, such as web server software. Alternatively,server130 may leverage web server software that is located in another memory device (not shown) internal or external toserver130. The above software programs and functionalities are not intended to be limiting andmemory136 may be configured to store additional or fewer types of data and/or software programs that perform other processes consistent with server-side computing systems.

Database140 may be one or more memory devices that store data and/or executable software that is used byserver130 to perform processes consistent with certain aspects related to the present invention. For example,database140 may include one or more data structures (e.g., tables, maps, arrays, directories, etc.) that include data that is used by one or more of the software programs stored inmemory136 and executed by processingunit134. Althoughserver130 showsdatabase140 andmemory136 as separate elements, the information and software stored in these elements may be combined in a single memory device. Also,memory136 and/ordatabase140 may be located external toserver130 and accessible throughnetwork120 or a dedicated back-end communication path, such as a Local Area Network (LAN).

II. INITIATIVE ANALYSIS PROCESS

Methods, systems, and articles of manufacturer consistent with aspects of the present invention enable one or more users ofclient118 to access functionalities offered byserver130 to determine retail strategies associated withbusiness network110. For example, a user inbusiness management unit116 may execute browser software inclient118 to request analysis information regarding one or more initiatives implemented at one ormore business locations114.FIG.2 shows an exemplary initiative segmentation process that may be performed by methods, systems, and/or articles of manufacture consistent with aspects of the present invention.

Initially, a user or users atbusiness management unit116 may create and select an initiative that is to be applied to a predetermined number of test sites reflecting a set of business locations114 (e.g., thirty stores) included in business network110 (Step210). The proposed initiative may be a change in store layouts, offering different products and/or services, different marketing campaigns (e.g., advertising, etc.), and any other type of idea that may be implemented at the selected test sites.

The user ofbusiness management unit116 may then direct the selected test sites to implement the proposed initiative (Step220). The initiative may be implemented for a predetermined amount of time (e.g., three months, six months, etc.), which is also known as a test period. Alternatively, the initiatives may be implemented at the test sites indefinitely. During the test period,business management unit116 may collect feedback results from each test site based on one or more performance metrics selected by unit116 (Step230). For example,business management unit116 may request sales information, foot traffic (i.e., number of persons entering a business location), etc. from each test site during the test period and may request additional performance data for each test site for a period before the test period.

Once the appropriate feedback results are collected, a user atbusiness management unit116 may useclient118 to perform an initiative analysis process based on the results (Step240). The initiative analysis process provides information regarding the effectiveness of the initiative at the test sites. Further, the initiative process provides information regarding a proposed effectiveness of the initiative if implemented in other non-tested business locations in business network110 (i.e., retailer locations that did not implement the initiative). These other business locations may be referred to as non-test group sites. In one aspect of the invention, the initiative analysis process includes the exchange of information between client118 (under the operation of a user) andserver130. For example, auser operating client118 may use browser software to access a web site that is provided or leveraged byserver130 to perform the initiative analysis process. Accordingly, server103 may provide content toclient118 that includes queries, data, and analysis information associated with the initiative analysis process.

A user, such as a manager atbusiness management unit116, or software processes may analyze the information provided byserver130 and selects one or more applicable non-test group sites to implement the initiative (Step250). Accordingly,business management unit116 determines and implements one or more strategies associated with the tested initiative in a selected number ofbusiness locations114, and/or groups of locations, such as regions112-1 to112-N.

In certain aspects,client118 may download information toserver130 associated with the results of the test initiatives at the test sites. Further,server130 may receive, or retrieve, attributes associated with eachbusiness location114 fromclient118, such as store size, locations, sale data, types of products and/or service offered, location and distances of competing merchant sites, age of the business location, amount of a lease (if any), weather during the test period, staff information, etc. Alternatively, or in addition to the information provided byclient118,server130 may execute software that determines, collects, and/or retrieves attribute information associated withbusiness network110, regions112-1 to112-N (e.g., population and demographic data, etc.), andbusiness locations114. The attribute information may be stored indatabase140 for subsequent use byserver130 and/orclient118.

FIGS.3A and3B show a flowchart of an exemplary initiative analysis process, consistent with certain aspects of the present invention. As explained above in connection withFIG.2, auser operating client118 may accessserver130 through browser software. In one aspect of the invention,server130 may leverage web server software that creates and renders web page content that includes information associated with the initiative analysis process. For example, the initiative analysis process may begin with a user ofclient118 receiving a web page including a list of target sites that implemented the initiative and provided feedback results toclient118.FIG.4 shows an exemplary screen shot of a test site selection web page including a list oftest sites410, the location of eachtest site420, the type of each test site (not shown), and a query section for proposed dates the user may select to perform ananalysis440. From this information, the user may select the test sites from whichserver130 is perform an initiative analysis (Step305).

Once the test sites are selected,server130 may provideclient118 with information that enable the user to set up the initiative analysis (Step310).FIG.5 shows an exemplary screen shot of an analysis setup page that allows the user to define one or more items associated with the initiative analysis process. For example, in setting up the analysis, the user may select one ormore performance metrics510 from which the user wishes to use in its analysis. A performance metric may include sales data in dollars (change in average sales during the predetermined test time period), gross profit margins, number of products sold, and any other type of metric associated with the performance of thebusiness locations114. Further,server130 may allow the user to select a product/service category520 for the analysis, such as a type of product, all products, etc.

In addition to performance metrics and categories, the user may selecttime periods540 from which the initiatives were implemented by the test sites. Also, the user may request thatserver130 eliminates any initiative feedback data that appear to be anomalous by removing any outlier sites (selection550). An outlier site has performance values that are too different than the rest of its respective test or control group. The user determines a threshold performance value beyond which a site is considered to be too different than its peers and thus deemed an outlier byserver130. By removing outlier test sites from consideration, the initiative analysis process does not consider the information for this test site during its analysis operations.

Additionally, or alternatively, the user may select acontrol group560 of business locations in thenetwork110 that have not implemented the initiative. For example, the user may select to have all or a portion of the remaining business locations considered in the analysis process, or request that the server select a group of sites that are most similar to the test group of sites in which the initiative was implemented. This group of stores selected by the user orserver130 is the control group.

Once the user has set the appropriate performance metrics, categories, etc. for the initiative analysis process,server130 may perform an initial analysis on the test site initiative result data (Step315). In one aspect,server130 may determine a performance value for each test site as well as the control group sites for each performance metric and category selected by the user. The performance value of a test site may be measured as a change in a performance metric/category combination for each test site relative to the change in the control group sites over a common time period. Onceserver130 determines the performance values for the test sites, it may provide the results of the initial analysis to, user viaclient118.FIG.6 shows an exemplary screen shot of an initial analysis results web page including data reflecting the results of the initial analysis process.

As shown inFIG.6,server130 may provide information associated with the performance (e.g., performance values) of the test sites relative to the control group sites for the selected performance metric. In the exemplary screen shot ofFIG.6,server130 provides the user with performance metric value changes in absolute values (e.g., actual dollars in sales) (e.g., graph610), and in percentage values (e.g., graph620). Further,server130 may determine the average change in the selected performance metric for the test sites relative to the change in the same performance metric for the control group sites. This may be displayed in graph form as well. Additionally,server130 may determine the likelihood that the performance of the test group relative to the control group of sites is not random chance.Server130 may provide such confidence values to the user as a 1-p value (e.g., graph640). The confidence value may be determined based on information reflecting a difference between test site performance metric data and control group site performance metric data.

Onceserver130 completes the initial analysis process and provides the results of this process to the user, it may execute a segmentation process consistent with certain aspects related to the present invention (Step320). The segmentation process may directprocessor unit134 to access the attributes associated withbusiness locations114 stored indatabase140. Once accessed, the segmentation process sorts through the attributes to identify those that most strongly relate to the different test site's performance values during implementation of the initiative. In one aspect,server130 may determine the strength of the relationship between each attribute and differential performance by test site (e.g., the attributes that are most in common among the top performing test sites and most different than the worst performing test sites).

III. SEGMENTATION PROCESS

FIG.7 shows a flowchart of an exemplary segmentation process consistent with certain aspects of the present invention. To segment the test site results,server130 may executesegmentation engine137 that initially sorts the test sites based on their corresponding performance metric results (Step705). For example, if the selected performance metric is an average percentage of sales difference over a predetermined test period, thensegmentation engine137 may sort the test sites from highest to lowest average sales difference.FIG.8 shows an exemplary test site attribute table including a sorted list of selectedtest sites801 based on their corresponding performance metric performance values802. Each test site is associated with one ormore attributes803 that correspond to some characteristic related to the test sites. Accordingly,segmentation engine137 may determine a measurement of the strength of the relationship between eachattribute803 and the performancemetric values802 of test sites801 (Step710). In one aspect,segmentation engine137 will use statistical processes to determine the strength of the relationship between theattributes803 andperformance metrics802. For example,segmentation engine137 may analyze the correlation and R-squared of each attribute to the performance of the test group sites. In such an example, theserver130 would create a scatter plot graph for each attribute. Data points in the scatter plot are associated with individual test sites.Server130 plots the site's value for a selectedattribute803 on the x-axis and plots the value for the selectedperformance metric802 on the y-axis.Server130 then determines and places a regression line through the scatter plot to calculate the correlation and r-squared values of that regression line. These values (i.e., correlation and r-squared values) indicate the strength of the relationship between the selectedattribute803 and theperformance metric802.Server130 repeats this process for every attribute. In one aspect,server130 determines positive and/or negative correlations. A positive correlation means that as the value of an attribute increases, the value of the selected performance metric also increases. A negative correlation means that as the attribute value increases, the value of the selected performance metric decreases.

Once determined,segmentation engine137 may sort the attributes based on the strength relationship values (Step715). The sorting allowssegmentation engine137 to have a prioritized list of attributes that are ranked based on their relationship with the performance of the test sites. Thus, referring toFIG.8, ifsegmentation process137 determines thatattribute 1 has the strongest relationship with the average sales difference oftest sites801 when compared to other attributes, that attribute may be ranked first in the list.

In certain aspects,segmentation engine137 may perform an unsegmented or segmented process on the attributes and test sites. Unsegmented processing involves analyzing the performance of each test site against the performance of all control group sites. Segmented processing involves analyzing the performance of each test site against the performance of control group test sites having similar attribute relationships. Steps725-750 and760-785 demonstrate the differences between the two processes.

Ifsegmentation engine137 is directed or configured to perform unsegmented processing (Step720; UNSEGMENTED), an initial attribute may be selected for processing (Step725).Segmentation engine137 may analyze the performance metric values associated with the selected attribute and splits the test sites into fragments that represent sets of test sites that are divided based on the statistical rank of their performance metric values (Step730). In one aspect of the invention,segmentation engine137 separates the test sites into quartiles, quintiles, or any number of fragments.Segmentation engine137 may then analyze the performance metric values of each test site attribute fragment against the performance of all control group test sites (Step735). Based on the analysis,segmentation engine137 may generate unsegmented attribute fragment results, which may include a resultant map of fragments representing the difference between the performance metrics of each test site fragment and all of the control group sites (Step740). This information may be stored inmemory136 ordatabase140.

In one aspect of the invention,segmentation engine137 may also determine a value representing a relationship between the performance of the test sites (i.e., performance metric) and the selected attribute. For example,segmentation engine137 may determine the 1-p value of the relationship between the performance of the test sites (i.e., performance metric) and the selected attribute. The 1-p value represents the probability that the relationship between the attribute and the performance metric (either positive or negative) is not random chance.

FIG.9 shows agraph900 reflecting a relationship between the performance of the test sites (i.e., performance metric) and a selected attribute. As shown,graph900 includes a number of fragments901-905 reflecting the unsegmented attribute fragment results computed bysegmentation engine137. For this exemplary attribute,line910 shows a positive correlation between the performance metric values and the attribute values.

If there aremore attributes803 to process (Step745; YES), thensegmentation engine137 may select another attribute (Step750) and the unsegmented process continues atStep730. On the other hand, if there are nomore attributes803 to process (Step745; NO),server130 may end the segmentation process and proceed to Step325 of the initiative analysis process ofFIG.3 (Step790).

Ifsegmentation engine137 is directed or configured to perform segmented processing on the attributes (Step720; SEGMENTED), aninitial attribute803 is selected for processing (Step760).Segmentation engine137 may split the test site performance metric results into fragments that represent sets of test sites that are divided based on the statistical rank of their values for the selected attribute (Step765). In addition,segmentation engine137 may split the control group sites into fragments based on their values for the same selected attribute (Step770).Segmentation engine137 may then analyze each test site fragment against a corresponding control group fragment (Step775). The results of the analysis are processed into segmented attribute fragment results that reflect the differences in the performance metric values for the test sites and corresponding control group sites for the selected attribute (Step777). To better understand this segmented analysis,FIG.10 shows a block diagram of an exemplary segmented attribute fragment comparison process consistent with certain aspects related to the present invention.

As shown,memory137 may store a set of control group fragment information1010-1040 and a set of test site fragments1050-1080 for a particular attribute under analysis. During a segmented process operation,segmentation engine137 may compare corresponding fragments from each set to generate segmented attribute fragment information1090-1096. For example,control group fragment1010 may be analyzed againsttest site fragment1050 to producesegmented attribute fragment1090, whilecontrol group fragment1020 is analyzed againsttest fragment1060 to producesegmented attribute fragment1092. The segmented attribute fragment information1090-1096 is stored inmemory137 for subsequent processing. Accordingly, atStep775 ofFIG.7,segmentation engine137 generates and stores the segmented attribute fragment results in a memory device, such asmemory136 ordatabase140.

Also,segmentation engine137 may determine a correlation value and a 1-p value in a manner similar to that described above in connection withStep740 andFIG.9. That is,segmentation engine137 may determine a correlation and 1-p values that represent the strength of the relationship between the performance of the test sites (i.e., performance metric) and the selected attribute. The correlation values, 1-p values, and associated information may be stored inmemory136 ordatabase140.

If there aremore attributes803 to process (Step780; YES), thensegmentation engine137 may select another attribute (Step785) and the segmented process continues atStep765. On the other hand, if there are nomore attributes803 to process (Step780; NO),server130 may end the segmentation process and proceed to Step325 of the initiative analysis process ofFIG.3 (Step790).

In addition to the processes performed during the segmentation process,server130 may also generate pointers, software links, etc. between each attribute and its corresponding unsegmented and/or segmented attribute fragment information. Using these links,server130 may generate content and data that allows the user to receive, view, and analyze the results of the segmentation processes (Step325). For example,FIG.11 shows an exemplary screen shot of an attribute table1100 generated byserver130 consistent with aspects of the present invention. Table1100 may be produced byserver130 using the attribute information generated during the segmentation process performed inStep320 ofFIG.3. A user may request to view the attribute information by instructingserver130, viaclient118, using queries, hyperlinks, and any other form of request technique through the browser software executing atclient118.

As shown, attribute table1100 may include a list of all of theattributes1110 associated with the test sites that have implemented the proposed initiative. These attributes may correspond toattributes803 described above in connection withFIG.8. Further, table1100 may include astrength value1120 representing a strength of the relationship between each attribute and the performance metric value on the test sites. As shown, attributes1110 are ranked based on their corresponding strength values (e.g., R-squared values). Additionally, table1100 may includecorrelation values1130, and 1-p values for each attribute. These values may reflect the correlation values, 1-p values and information determined inSteps730 and/or775 ofFIG.7. As shown inFIG.11,server130 may determine and present correlation values for the test sites and control group sites.

In another aspect of the invention, theattributes1110 listed in table1100 may be active in that the user may select an attribute (e.g., click) using an input device to view additional information regarding the attribute selected.FIG.12 shows a screen shot of an exemplary attribute details page associated with segmented attribute quartile results for a selectedattribute1210. In one aspect, the details page may include information representing the fragments generated bysegmentation engine137. For example, the details page may include a graph having segmentedfragments1220 forattribute1210. On the X-axis of the graph may be the attribute information1240 and the Y-axis is the performancemetric values1230, determined in this example in segmented form. Further, the details page may include achart1250 having information associated with the segmented attribute fragment data reflected in the graph. This information may include data reflecting the number oftest sites1260 andcontrol group sites1270 in each fragment, the computed performancemetric data1280, and a 1-p value1290 reflecting a measure of likelihood that the relationship between the attribute and the performance metric values is not random.Segmentation engine137 may calculate this value using statistical algorithms and/or processes based on the information provided by the segmented attribute fragment results.

Additionally, or alternatively,server130 may generate and render a scatter plot graph in which each test site is a data point in the graph, the X-axis is the value for the selected attribute and the Y-axis is the performance metric values.Server130 may be configured to allow the user to select more than one attribute from attribute table1100. In response to this selection,server130 may process and generate attribute graphs (e.g., fragment-based, scatter plot-based, etc.) including information showing the relationship between the multiple attributes and their corresponding performance metric values for the test sites.

Referring back toFIG.3A, once the user has analyzed the segmentation process output data (Step325),server130 allows the user to select a model type and provide associated parameters for the model to be used for analyzing the initiative performed in the test sites (Step330).Server130 is configured to execute the selected model using information related to the test sites to identify the attributes that are deemed most relevant to the success of certain test sites that implemented the initiative.Server130 uses the test site information to train the selected model. Once the user, or a software program, determines that the model is predicting accurate performance metric data based on the actual initiative feedback data provided by the test sites,server130 may apply the model to the control group sites.

In one aspect of the invention,server130 may offer different types of modeling methodologies that are used to predict performance values of the test and/or control group sites, such as linear regression, staged linear regression, neural network basic train, neural network stepwise regression, decision tree, K-means similar site modeling, and any other type of software model that may be used to analyze the information produced and/or collected bysegmentation engine137 andserver130 during the initiative analysis process.Server130 may generate a web page that includes a list of modeling methodologies from which the user may select. Also,server130 may allow the user to program a custom modeling methodology that may be used to analyze the initiative. Once the model is selected,server130 may query the user for parameters and other type of information used by the selected model.FIG.13 shows a screen shot of an exemplary modelselection web page1300 that includes menus for allowing the user to select a model and provide one or more parameters for the selected model.

As shown,web page1300 allows the user to select the type of performance metric1310 to run in the selectedmodel1320. Further,server130 may allow the user to select which attributes1330 are available for consideration by the selected model when executed. As shown inFIG.13, theexemplary web page1300 may allow the user to select/deselect anattribute1330 usingcorresponding selection boxes1340 and1350 to indicate that the particular attribute is to be used by the selected model. Also, the user may provide parameter information, such as selecting the maximum number ofattributes1380 the model should use when completing its analysis of the test sites and the implemented initiative. For example, the user may instructserver130 to consider only a certain number of the attributes with the strongest relationship to performance (e.g., the ten most important attributes) from among the entire list ofattributes1330. The user may force the model to include a particular attribute by selectingparameter box1350 for that particular attribute. In this instance,server130 always considers and uses the attributes selected viaparameter1350 when predicting performance values of the test sites.

Further, the user may also directserver130 to considercross correlations1360 between attributes when executing the selected model. Cross correlation is a parameter that reflects the similarity between attributes and their affect on the selected model's results. For example, two or more of the selected attributes1330 may be closely related, such as an attribute for the number of competitive business locations within a five mile radius and another attribute for the number of competitive business locations within a ten mile radius. These two exemplary attributes may affect the model in a similar fashion. The user may not want such similar attributes considered in the selected model. Accordingly, by setting thecross correlation parameter1360, the user directsserver130 to remove one of the similar attributes (leaving one attribute) from consideration by the selected model. This enablesserver130 to consider independent attributes when executing the selected model. Also, the user may select the number ofiterations1370 the model should run when training itself for producing its results.

Once the model and associated parameters are selected,server130 may execute the selected model (Step335). In one aspect,server130 may leveragemodel processing engine138 to process the model. In such an instance,server130 provides the user's selections and parameters toengine138 for subsequent processing. Further,model processing engine138 may re-execute the model based on the number ofiterations1370 selected by user inStep330. In this regard,engine138 tests and trains the model to calibrate its results. For example,model processing engine138 may build a model using a subset of the test sites (e.g., 20 out of 30 test sites) and tests the results of the model on the remaining test sites (e.g., the remaining 10 test sites). The model may compute predicted performance metrics for each of the remaining test sites based on the attributes selected by the model and compares them to the actual performance metrics of the remaining test sites.Server130 may repeat this process for the number of iterations set by the user inStep330.

Once the model is executed,server130 may generate and render the results to the user, viaclient110. The user may then analyze the results of the executed model (Step340). In one aspect,server130 may generate a model results web page that includes information pertaining to the results of the executed model.FIG.14 shows a screen shot of an exemplary model resultsweb page1400 consistent with certain aspects of the present invention. As shown, model resultsweb page1400 includes details of the attributes selected by the model. For example,web page1400 may include a list ofattributes1410 used by the model. For each of these attributes, and based on the type of model selected by the user,server130 may also provide details regarding the variables, constants, and other data used by the selected model. For example, resultsweb page1400 shows the results for a selected linear regression model. In this example,server130 may provide the values for a linear least squares fitting algorithm used to find the best fitting line through a set of points. This algorithm may include a constant and co-efficient that are assigned to each attribute. Accordingly,server130 may provide theco-efficient values1420 assigned to each attribute used by the model. Further, the constant1430 used by the model for this type of model algorithm may be included inresults web page1400. Additionally,server130 may provide information showing theaverage impact1440 of each attribute on the model results.

Moreover,server130 may provide information showing how accurate the model predicts performance values of the test sites based on the selected parameters and the initiative. In one aspect,server130 may provide quantitative measures reflecting the fitness of the model for predicting and analyzing the performance values of the test sites. For example,server130 may provide an R-squaredvalue1450 reflecting a quantitative value of the model's fitness for predicting the success of the initiative in the control group sites. Alternatively,server130 may express the fitness value as an error rate percentage value reflecting whether the predicted performance values are within a certain range of actual performance values associated with the test sites.

Additionally,server130 may provideconfidence measurements1460 reflecting quantitative values of how well each attribute is actually contributing to producing a model that will successfully predict the performance metrics of the control group sites when implementing the initiative. Also,server130 may determine and provide a percentpopulated value1470 for eachattribute1410 that represents the percentage of test sites thatserver130 has information for that particular attribute.

In another aspect of the invention,model processing engine138 may generate a graph showing the results of the executed model.FIG.15 shows an exemplarymodel results graph1500 page showing the results of the linear regression model explained above with respect toFIG.14. The Y-axis ofgraph1500 represents the actual performance metric values for each of the test sites and the X-axis represents the predicted performance metric values for the same test sites.Line1510 represents a match between predicted performance metric values and actual performance metric values. Themodel processing engine138 may determine the number of test sites are within certain ranges of a predetermined benchmark value. This information may be reflected as table1520 that includes information showing the percentage and number of test sites within various percentages of the benchmark.

The user ofbusiness management unit116 uses information reflected ingraph1500 and/or modelresults web page1400 to determine whether the selected model and/or attributes are acceptable (Step345). Alternatively, a software program executed byserver130 may analyze the information produced by the executed model to determine whether the model and/or attributes considered by the model are acceptable based on predetermined rules. Various factors may be considered by the user, or software program, when determining whether the model and/or attributes are acceptable. For example, a user may determine that one of theattributes1410 considered by the selected model was not an appropriate attribute based on the percentpopulated value1440 for that attribute. Alternatively, or additionally, theconfidence measurement1460 for an attribute may cause the user or the software program to determine that that attribute is not acceptable for consideration by the model. On a broader scale, the user or software program may determine that the selected model is not appropriate for predicting performance metrics for the non-test sites based on the confidence measurement for that model (e.g., r-squared value1450). The above exemplary factors are not intended to be limiting and the user and/or software program may use other result information to determine the appropriateness of the selected model and/or attributes.

If the results are not acceptable (Step345; NO), the user and/or, software program may re-select a model, attributes, and/or parameters for analyzing the initiative (e.g., return to Step330). For example, the user may remove one or more attributes from consideration prior to re-execution of the selected model. If, however, the results are acceptable (Step345; YES), the user and/or software program may directserver130 to apply the model to the non-test group sites to determine a predicted impact of the initiative on these sites (Step350). Theserver130 may do this by applyingattributes1410 and theircorresponding coefficients1420 selected by the model instep335 to each non-test site's values for thoseattributes1410. In doing so,server130 calculates the predicted performance of the initiative on each non-test site based on that site's relevant attributes1410.

Onceserver130 applies the tested model to the non-test group sites and the test sites, a determination may be made as to those sites having the highest or more acceptable predicted performance metric values.Server130 may rank the sites accordingly and provide this information to the user viaclient118.FIG.16 shows a business location analysis table that includes exemplary results from applying the model to the control group sites. As shown,FIG.16 shows a list of business locations (e.g., control group and test sites)1610 that are ranked based on each site's predicted performance metric values1620. In other words, using the sales lift performance metric as an example,server130 may place those sites that have an estimated larger sales lift when implementing the initiative higher in the list ofsites1610 than those who have lower estimated sale values. Further,server130 may provide information showing the actual performancemetric values1630 for those test sites that implemented the initiative and provided actual feedback information toserver130. Based on the exemplary results depicted inFIG.16, the user may select one or more of the non-test sites to implement the initiative. Accordingly, the user analyzes the information provided bymodel processing engine138 andserver130 to make an informed decision on which sites to implement the initiative tested at the test sites. Along the same lines, the user may also select those sites that should not implement the proposed initiative.

In another aspect of the invention,server130 may provide results of the processed model based on any grouping of the sites. One such example of a group of sites are regions112-1 to112-N explained above in connection withFIG.1.FIG.17 shows a table that includes exemplary results from applying the model to the non-test group sites in a region format For example,FIG.17 shows an exemplary table listing theregions1710 having those business locations (e.g., control group sites and/or test sites)1720 and their associated predicted performancemetric values1730 and actual performance metric values1740 (if applicable). As with the table depicted inFIG.16,server130 may rank theregions1710 accordingly to their predicted performance metric values. Thus, those regions having business locations with corresponding high estimated performance metric values are ranked above those regions with lower or less attractive performance metric values. Accordingly, instead of selecting individual business locations for implementing the initiatives, the user may determine which regions to apply the proposed initiative based on the information provided byserver130 in this format.

IV. ADDITIONAL EMBODIMENTS

As described, methods and systems consistent with certain aspects of the present invention utilize a software model that is used for analyzing initiatives to be applied in test sites (e.g., business locations). In one aspect,server130 may configure the model based one on or more selected parameters that control the manner in which the model processes data and measures the performance values of test sites. In one aspect, performance may be measured as a change in relevant performance values for the test sites from before the start of an initiative to during or after the initiative starts, relative to the change in performance in a control group over the same time period. For example, the parameters may include data reflecting whether the model should consider changes in absolute sales or in comparable (“comp”) (e.g., year over year) sales, the manner by which the model selects a control site group (e.g., sites not included in a selected set of test sites, for example, balance of market, comparable market, etc.), whether or not the model should filter outliers, the process used by the model to filter outliers, identifying the test measurement period, etc.

Certain aspects of the present invention enableserver130 to identify and set those model parameters that will best filter out noise associated with date related to the stores where the initiative was applied in order to produce more accurate results regarding the impact of the initiative. Noise may be a quantified measurement of inconsistent performance data for sites used in the analysis performed by the model. Aspects of the present invention create a simulation environment where the model performs a number of virtual initiative tests (e.g., initiative tests that have not actually been implemented in a business location) using different parameters. Based on the results of the virtual tests,server130 may identify parameter settings that best filter noise from the results. Those parameter settings are then automatically selected for a particular test type as default settings that are subsequently used in performing actual initiative tests for predicting the performance of selected business locations based on the proposed initiative.

a. Overview of Determining Parameter Settings

FIG.18 shows a flowchart of an exemplary parameter setting process consistent with certain aspects related to the present invention. In one aspect,server130 may execute software that performs the parameter setting process. Initially,server130 may collect historical data forbusiness locations114 within business network110 (Step1810). The historical data may include performance data associated with performance metrics, such as sales information, foot traffic, etc. The historical data may have been previously retrieved byserver130 frombusiness locations114 viaclient118 and stored indatabase140. Accordingly,server130 may collect the historical data fromdatabase140 duringstep1810, although other methods of collecting the historical data may be implemented by aspects of the present invention.

Server,130 may also define a test type that is associated with the business initiative testing model having a set of parameter settings (Step1820). A test type may be a type of business initiative testing model thatserver130 executes when running actual initiative tests to predict the performance of certain test sites, in a manner similar to the above disclosed aspects of the present invention. For example, a test type may be defined based on a particular characteristic of the initiative test, such as the length of a test period for the initiative test, etc. For purposes of illustrating aspects of the present invention,server130 may define the test type based on a selected test period. It should be noted that any other attribute of a test may be used to define a test type.Server130 may receive input from a user defining the test type or, alternatively, may automatically define the test type.

Once the test type if defined,server130 may perform a virtual test on the defined test type (Step1830). A virtual test reflects a mock initiative test that is performed on a selected number of test sites based on defined configuration settings for the model executed byserver130. The virtual test is considered a mock initiative test because no actual initiative test is planned or has been implemented in any virtual test sites (e.g., business locations identified for purposes of the virtual test). In this regard,server130 creates a simulation environment for running virtual initiative tests for collecting information used for identifying the optimal default model parameters settings for the defined test type. Performing a virtual test may include configuring the virtual test such thatserver130 iteratively performs a business initiative test for selected virtual test sites for each of a certain number of model parameter settings. Each model parameter setting may include one or more parameter setting options. Thus, for each iteration,server130 may determine a noise value for each combination of available parameter settings and options and stores the values in a memory device for subsequent processing.

Server130 may compare the noise values for each parameter setting combination to determine the parameter settings for the defined test type (Step1840). In one aspect,server130 may perform a statistical process on the noise values for each parameter setting combination, such as determining an average noise value for each combination over the iterations of collected noise values. In another aspect,server130 may determine the parameter settings by identifying a parameter setting combination that produced the least amount of noise during the virtual test.Server130 may identify this optimal parameter setting combination as a set of parameters settings that are to be used to configure the defined test type when executed for performing actual initiative tests to predict the performance of selected test sites. As such,server130 may store the parameter settings for the defined test type in memory for subsequent use by the model (Step1850). These and other aspects of the parameter setting process are described in further detail below in connection withFIGS.19-26.

b. Performing a Virtual Test

FIG.19 shows a flowchart of an exemplary virtual test process that may be performed byserver130 consistent with certain aspects of the present invention. When performing a virtual test,server130 may first select a parameter setting set (Step1910). To do so, in one aspect,server130 may execute a process that first determines the number of parameter settings and parameter setting options for the type of model that will perform the initiative tests. As describe above,server130 may execute a number of different models that are used for performing an initiative test. Each model may have a certain number of parameter settings and each parameter setting may include one or more parameter setting options. For example, a model may include a control group selection parameter setting that may define different methods for selecting a control group. One method may include selecting all business locations (i.e., sites) not included in a virtual test site set to be included in the control group for the virtual test performed by server130 (e.g., balance of chain process). Alternatively, the control group may be selected based on a balance of market or other geographic region-related data, or based on a determined similarity to the virtual test sites (e.g., size, customer base profiles, sales, etc.), etc. Each of these exemplary control group selection methods may be a parameter setting option for the control group selection parameter setting.

Other types of parameter settings include a parameter that controls whether the model will use algorithms to filter outliers (e.g., performance data for a virtual test site that is too inconsistent with other related performance data). Thus, one option of this type of parameter setting may be to use outlier data, and another option may be to disregard outlier data. Other outlier parameter setting options may include the type of algorithm implemented when disregarding outlier data. For example, one option may be to remove the determined performance data entirely from the set of data used by the model. Another option may be to use a capped algorithm that changes the data of an outlier result to make the result no longer an outlier. For instance, if an outlier is defined as an observation (e.g., data result) that is more than three standard deviations from a mean, the capped algorithm may overwrite a outlier that is four standard deviations from the mean with a value that represents three standard deviations from the mean.

Another type of parameter setting may include whether the model will view changes in absolute performance data (e.g., actual sales data) or comp performance data (e.g., year to year sales data). For example, in measuring actual sales, a test site may have been selling $300 per day before an initiative was implemented. Following implementation, the test site may be selling $330 per day. Accordingly, the test site experienced a 10% lift in absolute sales. Comp sales, on the other hand, measures change in year on year sales. For example, a test site may have been selling 5% higher than in the prior year before an initiative was implemented. Following implementation, the test site may be selling 8% higher than in the prior year. Accordingly, the test site experienced a roughly 3% increase in comp sales.

A model may also have a parameter setting that defines how long of a pre-period to implement in the initiative tests run by the model. A pre-period may be a length of time prior to a selected test start date for a virtual test for which the model will measure performance before the initiative was implemented. In the above example, a test site was selling $300 per day before the initiative was implemented and $330 per day after the initiative was implemented. The $300 in sales per day represents performance in the pre-period. This exemplary parameter setting determines what length of time to use in measuring pre-period performance. For example, if a model is configured to compare performance data for a selected test period following a test start date (e.g., Sep. 1, 2004), a one month selected pre-period may be reflect the time period including all of August, 2004. Similarly, a one year pre-period may include the time period including the beginning of September 2003 to the end of August 2004. Any length of pre-period may be selected in accordance with certain aspects related to the present invention.

The above exemplary parameter settings and parameter setting options are not intended to be limiting.Server130 may define any type of parameter setting and parameter setting options for any type of model, based on the characteristics of the model.

Once the number parameter settings and parameter setting options are selected,server130 may determine the maximum number of parameter setting combinations that may be tested using the model. For example, for an exemplary set of parameters including four parameter settings, and each parameter setting having four respective parameter setting options,server130 may determine there are 256 possible parameter setting combinations available for the model to test (e.g., {P1(1), P2(1), P3(1), P4(1)} . . . {P1(4), P2(4), P3(4), P4(4)}).

Thus, in selecting a parameter setting set,server130 may set a value, PS(Max), to 256 representing the maximum number of combinations for the test type. Further,server130 may then select a first parameter setting set reflecting a selected combination of parameter settings including one option from each available parameter setting. For instance,server130 may select the first option from each of the four exemplary parameter settings mentioned in the above example (e.g., Parameter Setting Set (PS)={PS1(1), PS2(1), PS3(1), PS4(1)}).Server130 may select any combination duringstep1910. As explained later, however,server130 may be configured to select a different parameter setting set for each iteration of the virtual test.

Server130 may also select a number of virtual test sites for the virtual test (Step1920). In one aspect,server130 may randomly select a certain number of virtual test sites from the available sites reflectingbusiness locations114 inbusiness network110. For example,server130 may select fifty (50) virtual test sites from an available pool of one thousand (1000) sites inbusiness network110.

Based on the selected test period associated with the defined test type (e.g., one month),server130 may select a test start date (Step1940). The test start date may be a previous date that is already included in the monitoring periods associated with the historical data collected for the business locations114 (i.e., sites) inbusiness network110. For example, ifdatabase140 maintains historical data up to Dec. 31, 2004,server130 may randomly select a start date that is before that date. The test start date may be further bounded based on the selected test period. Thus, for a one month test period,server130 may be configured to select a test start date that is no later than Nov. 30, 2004 to allow for the test period to include the historical data for Dec. 31, 2004. Similarly, for a six month test period,server130 may be configured to select the test start date no later than Jun. 30, 2004. Within these boundaries,server130 may randomly select the test start date. Alternatively,server130 may receive user input identifying the test start date.

In another aspect of the invention,server130 may select a pre-period associated with the test period. A pre-period is a selected period of time extending back from the test start date. For example,server130 may select a pre-period of one month. Thus, if Sep. 1, 2004 is selected as the test start date, the one-month pre-period would include Aug. 1, 2004 through Aug. 31, 2004. The pre-period may be equal to, or different from, the selected test period for the defined test type. Also, the pre-period may be selected based on the type of parameter settings and corresponding parameter setting options selected instep1910. That is, if the test type includes a parameter setting option for determining how to select the pre-period,server130 selects the pre-period according to that selected parameter setting option.

Also, based on the selected parameter settings,server130 may determine a control site group for the virtual test, reflecting a selected set of sites (i.e., business locations114) withinbusiness network110. For example, a control group selection parameter setting option may directserver130 to select the control group from the remaining sites inbusiness network110 that are not included in the set of selected virtual test sites. Thus,server130 may select the remaining 950 sites in the exemplary1000 sites ofbusiness network110 that were not chosen for the 50 virtual test sites. As noted above, a control group parameter setting may configureserver130 to select the control group based on different criteria, and thus the above examples are not intended to be limiting.

Once the parameter setting set, virtual test sites, and test start date are determined,server130 may execute the model to run a virtual test. The model may execute an initiative test based on the configuration information and produce a result for a selected performance metric (Step1950). For instance, if the model is configured to monitor sales data,server130 may execute a process to collect sales data for the selected virtual test sites for the selected pre-period and test period. Thus, following the above examples,server130 may collect and analyze the historical data to determine the sales data for the fifty virtual test sites during August 2004 and September 2004. Additionally,server130 may collect and analyze the historical data for the control group to determine the sales data for the sites within that group (e.g., 950 sites). As noted above,server130 may produce different types of result data based on the type of performance metric being monitored byserver130 when performing the virtual test.

Once the performance data (e.g., sales data) is determined,server130 may determine the noise for the parameter setting set selected in step1910 (e.g., (PS)={PS1(1), PS2(1), PS3(1), PS4(1)}) (Step1960). As explained, noise may be represented as a value reflecting an inconsistency in the performance data for the virtual test sites in relation to the control site group. Thus, when the performance data for the virtual test sites vary from the performance data for the control group sites,server130 may calculate a value reflecting this difference. Because theserver130 has created a simulation environment by performing the virtual test on the virtual test sites, no actual initiative was performed in these test sites. Accordingly, the performance data for the virtual test sites determined during the virtual test should be similar, if not identical to, the performance data for the control group. For instance,server130 may determine during the virtual test that the control group experienced a 2.5% sales lift from August 2004 to September 2004 (e.g., performance metric result data for the pre-period vs. the performance metric result data for the test period).Server130 may also determine that the virtual test sites experienced a 3% sales lift during the same time frame. This difference (i.e., 0.5% lift) may be the result of a difference in the sales trends between the test set and control group. Accordingly,server130 may identify the difference as a quantified value of noise.Server130 may then store the determined noise for the selected parameter setting set in a data structure maintained in a memory device, such as database140 (Step1970).

According to certain aspects of the invention,server130 may be configured to iteratively run a virtual test for the selected parameter setting for a predetermined number of times. Accordingly,server130 may define a iteration threshold that controls how many times the selected parameter setting set is tested (e.g., 10 times, 100 times, etc.). The iteration threshold value may be determined at any time before checking the threshold, such as when the test type is defined (e.g., step1820), when selecting the parameter setting set (step1910), etc. Therefore, asserver130 performs a virtual test for a selected parameter setting set, it may track the number of times it has run a virtual test on the set. As such, instep1980,server130 determines whether the iteration threshold has been met. If so (Step1980; Yes), the virtual test process continues to Step1990. However, if the iteration threshold is not met (Step1980; No),server130 may return to step1920 to select a new set of virtual test sites and new test date to run another virtual test for determining a new noise value for the selected parameter setting set. The new noise value for the current iteration is also stored in the data structure. This process continues until the iteration threshold is met (e.g., the selected parameter setting is tested 100 times).

Once the iteration threshold is met,server130 may determine whether all combinations of the parameter settings have been iteratively tested. In one aspect,server130 may determine whether a parameter setting set threshold is met, which may be set equal to PS(Max) previously determined by server130 (Step1990). If the threshold is met (Step1990; Yes), the virtual test process ends (Step1995). On the other hand, if the parameter setting set threshold is not met (Step1990; No),server130 selects a new parameter setting set from the available combinations previously determined, and repeats the iterative testing processes described above. In this regard,server130 ensures each combination of parameter settings based on their corresponding options have been iteratively tested, and a noise value determined and stored in the data structure.

c. Determining Noise

As explained,server130 may execute a software process that determines noise for each iteration of a virtual test of a selected parameter setting set. Noise reflects a quantified measurement of inconsistent performance data for sites used in the analysis performed by a business initiative testing model.FIG.20 shows a flowchart of an exemplary noise determination process consistent with certain aspects related to the present invention.

Instep2010,server130 may determine the performance data for the virtual test during the pre-period and test period. In one aspect,server130 may accessdatabase140 to collect the historical data for each virtual test site to determine, based on the selected performance metric, the performance data during each time period.Server130 may average the determined performance data for the virtual test sites, or determine other statistical values associated with the determined performance data. Similarly,server130 also determines the performance data for the control group sites during the pre-period and test period (Step2020).Server130 may then determine the noise for the selected parameter setting set based on a comparison of the performance data for the virtual test sites and control group sites (Step2030).

To better understand these aspects of the present invention,FIG.21 shows a block diagram of performance data associated with a virtual test having atest period2160 and pre-period2150 of one month, a test start date2710 of Sep. 1, 2004, and a performance metric of total sales. As shown,server130 may determine for the virtual test sites the performance data for a pre-period2110 (e.g., August 2004) and a test period2120 (e.g., September 2004).Server130 may also determine, for the control group sites, the performance data for the pre-period2130 andtest period2140.Server130 may also determine a statistical value associated with the performance data of each period for both control sites and virtual test sites. For instance, based on a parameter setting controlling howserver130 looks at performance data during the virtual test,server130 may determine that the absolute sales performance data for the virtual test sites shows a 3% lift from August 2004 to September 2004. On the other hand,server130 may determine that the absolute sales performance data for the control group sites shows a 2.5% lift in the same time frame.Server130 may quantify this difference as noise. Thus, based on the determined performance data,server130 determines the noise value for the selected parameter setting set as roughly 0.5 (i.e., (1.03/1.025)−1). This noise value is stored in the data structure in accordance with the disclosed aspects of the present invention.

Although as described above,server130 determines noise based on a difference of performance data, methods and systems consistent with certain aspects of the present invention are not limited to the above examples and may quantify noise based on other relationships between performance data of the test and control sites.

d. Determining Parameter Settings

FIG.22 shows a flowchart of an exemplary parameter setting process consistent with certain aspects of the present invention. In one aspect, once the noise values for each iterative test of each parameter setting set are determined and stored,server130 may determine the average noise for each parameter setting set and store the average noise value in the data structure. (Step2210). Thus,server130 may maintain indatabase140, for example, a data structure of noise values corresponding to each parameter setting set.FIG.23 shows a block diagram of anexemplary data structure2300 consistent with certain aspects of the present invention. In one example,data structure2300 includes a field for each parameter setting set2310 included in the available combinations of parameter setting sets (e.g., up to PS(Max)). As shown, each parameter setting set includes particular parameter setting set options for each parameter setting.Data structure2300 shows exemplary parameter setting options for four parameter settings, each having four parameter setting options. Thus, parameter setting set1 includes the set, {PS1(1), PS2(1), PS3(1), PS4(1)} and parameter setting set PS(MAX) includes the set {PS1(4), PS2(4), PS3(4), PS4(4)}. Further, each parameter setting setfield2310 includes anoise value2320 for each virtual test iteration performed byserver130. In this example,data structure2300 includes 100 noise values representing the determined noise for each of the 100 virtual test iterations performed byserver130 for a given parameter setting set. Additionally,server130 may determine and store theaverage noise value2330 for a given parameter setting set based on the iteration noise values2320.

In another aspect of the present invention,server130 may store noise values in a data structure based on each parameter setting option.FIG.24 shows a block diagram of anexemplary data structure2400 consistent with these aspects. As shown,exemplary data structure2400 includes data associated with a four parameter setting combination, each parameter setting including four options. Data structure includes fields including data identifying eachparameter setting option2450 for a given parameter setting2410-2440. Eachparameter setting option2450 is associated with fields includingnoise values2460 determined byserver130 during different iterations of the virtual tests. In this example,data structure2400 include 100 noise values reflecting the 100times server130 determined a noise value when the given parameter setting option was used during the virtual test (e.g., 1, 2, 3, or 4). For example,parameter setting option 1 of the first parameter setting2410 includes noise values for 100 iterations of the virtual test when that option was used to configure the model performing the virtual test. Thus, when performing the virtual test process,server130 may store a noise value in a successive field forparameter setting option 1, each time that option was tested.Server130 may also determine and store theaverage noise2470 for eachparameter setting option2450 of each parameter setting2410-2440 indata structure2400.

Aspects of the present invention are not limited to the configurations shown inFIGS.23 and24.Server130 may configure and store noise values in any type of data structure and memory device. Further, aspects of the present invention are not limited toserver130 storing average noise values and/or parameter setting set noise values in the same data structure as the noise values. For example,data structures2300 and/or2400 may each represent a data structure formed from multiple data structures distributed across multiple memory devices.

Once the average noise value for each parameter setting is determined,server130 may determine the optimal combination of parameter settings for the parameter settings (Step2220). In one aspect,server130 may identify the parameter setting set having the lowest average noise as the optimal parameter setting set. For example,server130 may search the data structure storing the average noise values to identify the lowest values. Thus, for example, if a parameter setting set3 ofFIG.23 includes the lowestaverage noise2360,server130 may designate parameter setting {PS1(3), PS2(1), PS3(1), PS4(1)} as the optimal parameter setting. In other words,server130 will define the parameter settings for the given test type to include the third parameter setting option of parameter setting one, and the first parameter setting option of parameter settings two through four.

In another example corresponding todata structure2400,server130 may identify the lowest average noise value for each parameter setting option for each parameter setting, and then combine these lowest average noise values to form the optimal parameter settings to define the default settings. For example,FIG.25 showsdata structure2400 identifying exemplary lowest average noise values2515,2525,2535, and2545 for respectiveparameter setting options2510,2520,2530, and2540. Accordingly, in this example,server130 may form an optimal parameter setting set including, {PS1(4), PS2(4), PS3(3), PS4(1)}.

It should be noted,server130 may be configured to execute programs that identify the lowest noise values for the parameter settings using different methods than that described above. As such, the above described processes are exemplary and not intended to be limiting.

e. Performing an Initiative Test Using Parameters

In accordance with certain aspects of the present invention, onceserver130 identifies the parameter settings for a given test type, the setting is stored in a memory device (e.g., database140) for subsequent use byserver130 when running an initiative test for selected test sites.FIG.26 shows a flowchart of an exemplary initiative test process consistent with these aspects of the invention. Initially, at some time,server130 may receive a request to perform an initiative test for a selected set of test sites in a manner consistent with that described above and in parent U.S. patent application Ser. No. 10/767,191.Server130 may retrieve parameter settings for the requested initiative test based on the type of test to be performed (Step2620). In one aspect,server130 may accessdatabase140 to retrieve the parameter settings previously determined by running the virtual tests for that test type in accordance with the processes described above. Once retrieved,server130 configures the initiative test in accordance with the parameter settings, and performs the test (Step2630). Results may then be collected and processed byserver130 in a manner consistent with that described above and in parent U.S. patent application Ser. No. 10/767,191 (Step2640).

VI. CONCLUSION

Variations of the methods and systems consistent with features of the present invention previously described may be implemented without departing from the scope of the invention. For example,server130 may be configured with software that automatically performs some or all of the analysis and decisions performed by auser operating client118. For instance,server130 may include software stored inmemory136 that, when executed by processingunit134, analyzes the list of attributes ranked according to their impact on the performance metric values of the test sites (e.g., correlation and/or r-squared values).Server130 may also execute software that selects the model and associated parameters (e.g., attributes, iterations, etc.) for predicting the performance levels of the test and non-test group sites. Additional,server130 may execute software that selects one or more of the non-test group sites to implement the tested initiative based on the predicted performance values produced by the executed model. The above processes are not intended to be limiting and other procedures may be performed by software processes executed byserver130 and/orclient118 to supplement and/or compliment those decisions, data input, and analysis performed by a user.

Also, although the initiative analysis process is described as using web server and browser software for exchanging information withclient118 andserver120, methods and systems consistent with aspects of the invention may use any type of technology to allow a user to send and receive information fromserver120. Further, although the above described aspects of the invention include communications between aclient118 and aserver120, methods and systems consistent with aspects of the invention are not limited to client-server network configurations. That is, a user associated withbusiness network110 may leverage a computing environment that is local (i.e., same computing system) to a system operated by the user.

Moreover, the virtual test processes described herein are not limited to the sequences, steps, and systems shown inFIGS.18-26. Additional, fewer, and different processes and elements may be implemented to configure and run the virtual test processes to identify noise values for given parameter settings.

Further, methods, systems, and articles of manufacture, consistent with features of the present invention may be implemented using various network models, and is not limited to a particular computer and/or network environment. Furthermore, methods, systems, and articles of manufacture, consistent with features of the present invention are not limited to the implementation of systems and processes compliant any particular type of programming language. Any number of programming languages may be utilized without departing from the scope of the present invention.

Additionally, although aspects of the present invention are described as being associated with data stored in memory and other storage mediums, one skilled in the art will appreciate that these aspects can also be stored on or read from other types of computer-readable media, such as secondary storage devices, like hard disks, floppy disks, or CD-ROM; a carrier wave from the Internet; or other forms of RAM or ROM. Accordingly, the invention is not limited to the above described aspects of the invention, but instead is defined by the appended claims in light of their full scope of equivalents.

Claims (69)

What is claimed is:

1. A method for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, comprising:

identifying, by a computer, a business initiative testing model having a set of parameter settings;

selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;

performing, by a computer, a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test performed on test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is performed on the virtual test sites using a variation of each parameter setting;

determining, by a computer, actual performance data associated with the set of virtual test sites;

determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining, by a computer, a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and

configuring, by a computer, the business initiative testing model using the optimal parameter settings to test a business initiative for application in the business network.

2. The method ofclaim 1, wherein performing the virtual test includes:

performing a series of virtual tests for each possible combination of the parameter settings for the business initiative testing model.

3. The method ofclaim 2, wherein iteratively performing the series of virtual tests includes:

determining a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.

4. The method ofclaim 3, further including:

for each iteration of the series of virtual tests for a given parameter settings combination,

storing the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.

5. The method ofclaim 3, further including:

for each given parameter settings combination,

determining an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.

6. The method ofclaim 2, wherein performing the series of virtual tests for each possible combination of the parameter settings includes:

for each iteration of the series of virtual tests,

randomly selecting the set of virtual test sites, and

randomly selecting a test start date for collecting historical data associated with each of the randomly selected virtual test sites.

7. The method ofclaim 1, further including:

iteratively performing the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.

8. The method ofclaim 1, wherein iteratively performing the virtual test includes:

storing a noise value for each iteration and first parameter setting in a data structure.

9. The method ofclaim 1, further including:

selecting a second parameter setting set when the iteration threshold is met; and

iteratively performing the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.

10. The method ofclaim 1, wherein iteratively performing the virtual test includes:

storing a noise value for each iteration and second parameter setting in the data structure.

11. The method ofclaim 1, wherein determining optimal parameter settings for the first test type includes:

analyzing the noise values associated with the first and second parameter settings sets; and

selecting either the first or second parameter settings set as the optimal parameter settings based on whether the first or second parameter settings sets have a lower average noise value over the iterations of the virtual test.

12. The method according toclaim 1, further comprising defining a test type associated with the business initiative testing model, wherein the test type is defined based on an attribute of the business initiative test.

13. The method ofclaim 12, wherein determining the set of optimal parameter settings includes:

identifying the parameter settings combination having the lowest average noise value as the optimal parameter settings for the first test type of the business initiative testing model.

14. The method ofclaim 12, wherein configuring the business initiative testing model further includes:

receiving a request identifying the first test type to test the business initiative to apply to the business network.

15. The method ofclaim 12, wherein defining the first test type includes:

defining a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.

16. The method ofclaim 12, further including:

defining a second test type of the business initiative model;

performing the virtual test based on the defined second test type to determine second optimal parameter settings for the second test type; and

performing a business initiative test using the second test type of the business initiative testing model that is automatically configured using the second optimal parameter settings.

17. The method according toclaim 1, wherein each parameter setting comprises a set of one or more parameter setting options, wherein a parameter setting option further defines a parameter setting.

18. A system for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, the system including:

a computer configured to: identify a business initiative testing model having a set of parameter settings;

selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;

execute a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test for-test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is performed on the virtual test sites using a variation of each parameter setting;

determine actual performance data associated with the set of virtual test sites;

determine actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determine a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and

configure the business initiative testing model using the optimal parameter settings to a test on a business initiative to apply to the business network.

19. The system ofclaim 18, wherein the computer is further configured to:

perform a series of virtual tests for each possible combination of the parameter settings for the business initiative testing model.

20. The system ofclaim 19, wherein the computer is further configured to perform the series of virtual tests by:

determining a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.

21. The system ofclaim 20, wherein the computer is further configured to:

for each iteration of the series of virtual tests for a given parameter settings combination,

store the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.

22. The system ofclaim 20, wherein the computer is further configured to:

for each given parameter settings combination,

determine an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.

23. The system ofclaim 22, wherein the computer is further configured to determine the set of optimal parameter settings by identifying the parameter settings combination having the lowest average noise value as the optimal parameter settings of the business initiative testing model.

24. The system ofclaim 19, wherein the computer is further configured to perform the series of virtual tests for each possible combination of the parameter settings by:

for each iteration of the series of virtual tests, randomly selecting the set of virtual test sites, and

randomly selecting a test start date for collecting historical' data associated with each of the randomly selected virtual test sites.

25. The system ofclaim 24, wherein the computer is further configured to:

defining a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.

26. The system ofclaim 18, wherein the computer is further configured to:

receive a request identifying the first test type to test the business initiative to apply to the business network, and

define the first test type of the business initiative testing model based on the request.

27. The system ofclaim 18, wherein the computer is further configured to:

iteratively perform the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.

28. The system ofclaim 18, wherein the computer is further configured to iteratively perform the virtual test by storing a noise value for each iteration and first parameter setting in a data structure.

29. The system ofclaim 18, wherein the computer is further configured to:

select a second parameter setting set when the iteration threshold is met; and

iteratively perform the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.

30. The system ofclaim 18, wherein the computer is further configured to iteratively perform the virtual test by storing a noise value for each iteration and second parameter setting in the data structure.

31. The system ofclaim 18, wherein the computer is further configured to determine optimal parameter settings by:

analyzing the noise values associated with the first and second parameter settings sets; and

selecting either the first or second parameter settings set as the optimal parameter settings based on whether the first or second parameter settings sets have a lower average noise value over the iterations of the virtual test.

32. The system according toclaim 18, wherein a test type is associated with the business initiative testing model and is defined based on an attribute of the business initiative test.

33. The system ofclaim 32, wherein the computer is further configured to:

define a second test type of the business initiative model;

perform the virtual test based on the defined second test type to determine second optimal parameter settings for the second test type; and

perform a business initiative test using the second test type of the business initiative testing model that is configured using the second optimal parameter settings.

34. A non-transitory computer-readable medium including instructions for, when executed by a processor, performing a method for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, the method comprising:

identifying, by a computer, a business initiative testing model having a set of parameter settings;

selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;

performing, by a computer, a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test using a variation of each parameter setting and is performed on test sites where no actual initiative test is performed on the virtual test sites;

determining, by a computer, actual performance data associated with the set of virtual test sites;

determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining, by a computer, a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and

configuring, by a computer, the business initiative testing model using the optimal parameter settings to test a business initiative to apply to the business network.

35. A method for testing business initiatives comprising:

receiving, by a computer, a request to test a business initiative to be applied to a set of business locations in a business network;

determining, by a computer, actual performance data associated with a set of virtual test sites;

determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

configuring, by a computer, a business initiative testing model based on the request and a default set of parameter settings for the model, each default parameter setting having a set of parameter setting options and the default set of parameter settings reflecting optimal parameter settings for the model that are determined by identifying a combination of parameter setting options that produce the least amount of noise during iterations of a set of virtual business initiative tests performed for selected virtual test sites, wherein each virtual test is a simulated business initiative test performed on test sites where no actual initiative test has been implemented at those test sites and wherein noise is a measurement of inconsistency between performance data associated with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings; and

testing, by a computer, the business initiative using the configured business initiative testing model to produce results reflecting a predicted performance of the business initiative if applied to the set of business locations.

36. A method for determining optimal parameter settings for a predictive machine-learning model in business initiative testing software used to filter inconsistent data for testing initiatives for business locations included in a business network, comprising:

identifying, by a computer, a predictive machine-learning business initiative testing model having a set of parameter settings;

selecting, by the computer, a first parameter setting set for performing a virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the predictive machine-learning business initiative testing model, wherein at least one parameter setting includes a time period;

simulating performance of the first parameter setting set by performing, by the computer, the virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test performed on test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is iteratively performed on the virtual test sites using a variation of a parameter setting option for each parameter setting;

determining, by the computer, actual performance data associated with the set of virtual test sites;

determining, by the computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using tested parameter settings;

determining, by the computer, a noise value for the first parameter setting set, the noise value reflecting an inconsistency between actual performance data associated with the set of virtual test sites and actual performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining, by the computer, a set of optimal parameter settings for the predictive machine-learning business initiative testing model based on results from performing the virtual test whereby the optimal parameter settings best minimize the noise value from the results;

training, by the computer, the predictive machine-learning business initiative testing model by configuring, by the computer, the predictive machine-learning business initiative testing model using the optimal parameter settings based on the noise value for each parameter setting option of each parameter setting of the parameter setting set to test a business initiative for application in the business network; and

executing, by the computer, the predictive machine-learning model using the set of optimal parameter settings.

37. The method of claim 36, wherein performing the virtual test includes:

performing, by the computer, a series of virtual tests for each possible combination of the parameter settings for the predictive machine-learning business initiative testing model.

38. The method of claim 37, wherein iteratively performing the series of virtual tests includes:

determining, by the computer, a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.

39. The method of claim 38, further including:

for each iteration of the series of virtual tests for a given parameter settings combination, storing, by the computer, the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.

40. The method of claim 38, further including:

for each given parameter settings combination, determining, by the computer, an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.

41. The method of claim 37, wherein performing the series of virtual tests for each possible combination of the parameter settings includes:

for each iteration of the series of virtual tests, randomly selecting, by the computer, the set of virtual test sites, and

randomly selecting, by the computer, a test start date for collecting historical data associated with each of the randomly selected virtual test sites.

42. The method of claim 36, further including:

iteratively performing, by the computer, the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.

43. The method of claim 36, wherein iteratively performing the virtual test includes:

storing, by the computer, a noise value for each iteration and first parameter setting in a data structure.

44. The method of claim 42, further including:

selecting, by the computer, a second parameter setting set when the iteration threshold is met; and

iteratively performing, by the computer, the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.

45. The method of claim 44, wherein iteratively performing the virtual test includes:

storing, by the computer, a noise value for each iteration and second parameter setting in a data structure.

46. The method of claim 44, wherein determining optimal parameter settings for a first test type includes:

analyzing, by the computer, the noise values associated with the first and second parameter settings sets; and

selecting, by the computer, either the first or second parameter settings set as the optimal parameter settings based on whether the first and second parameter settings sets have a lower average noise value over the iterations of the virtual test,

wherein the first test type is defined based on an attribute of the simulated business initiative test.

47. The method according to claim 36, further comprising defining, by the computer, a test type associated with the predictive machine-learning business initiative testing model, wherein the test type is defined based on an attribute of the simulated business initiative test.

48. The method of claim 46, wherein determining the set of optimal parameter settings includes:

identifying, by the computer, the parameter settings combination having the lowest average noise value as the optimal parameter settings for the first test type of the predictive machine-learning business initiative testing model.

49. The method of claim 46, wherein configuring the predictive machine-learning business initiative testing model further includes:

receiving, by the computer, a request identifying the first test type to test the business initiative to apply to the business network.

50. The method of claim 46, wherein defining the first test type includes:

defining, by the computer, a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.

51. The method of claim 46, further including:

defining, by the computer, a second test type of the predictive machine-learning business initiative testing model;

performing, by the computer, the virtual test based on the defined second test type to determine second optimal parameter settings for the second test type; and

performing, by the computer, a simulated business initiative test using the second test type of the business initiative testing model that is automatically configured using the second optimal parameter settings.

52. The method according to claim 36, wherein each parameter setting comprises a set of one or more parameter setting options, wherein a parameter setting option further defines a parameter setting.

53. A system for determining optimal parameter settings for a predictive machine-learning model in business initiative testing software used to filter inconsistent data for testing initiatives for business locations included in a business network, the system including:

a computer comprising a processor and a non-transitory computer-readable medium including instructions that when executed by the processor are configured to:

identify a predictive machine-learning business initiative testing model having a set of parameter settings;

selecting a first parameter setting set for performing a virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the predictive machine-learning business initiative testing model, wherein at least one parameter setting includes a time period;

execute the virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test for test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is iteratively performed on the virtual test sites using a variation of a parameter setting option for each parameter setting;

determine actual performance data associated with the set of virtual test sites; determine actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using tested parameter settings;

determine a noise value for the first parameter setting set, the noise value reflecting an inconsistency between actual performance data associated with the set of virtual test sites and actual performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

train the predictive machine-learning business initiative testing model by automatically configuring the predictive machine-learning business initiative testing model when the processor determines a set of optimal parameter settings for the predictive machine-learning business initiative testing model based on results from performing the virtual test whereby the optimal parameter settings best minimize the noise value from the results based on the noise value for each parameter setting option of each parameter setting;

configure the predictive machine-learning business initiative testing model using the optimal parameter settings to test a business initiative to apply to the business network; and

execute the predictive machine-learning business initiative testing model using the set of optimal parameter settings.

54. The system of claim 53, wherein the computer is further configured to:

perform a series of virtual tests for each possible combination of the parameter settings for the predictive machine-learning business initiative testing model.

55. The system of claim 54, wherein the computer is further configured to perform the series of virtual tests by:

determining a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.

56. The system of claim 55, wherein the computer is further configured to:

for each iteration of the series of virtual tests for a given parameter settings combination, store the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.

57. The system of claim 55, wherein the computer is further configured to:

for each given parameter settings combination, determine an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.

58. The system of claim 57, wherein the computer is further configured to determine the set of optimal parameter settings by identifying the parameter settings combination having the lowest average noise value as the optimal parameter settings of the predictive machine-learning business initiative testing model.

59. The system of claim 54, wherein the computer is further configured to perform the series of virtual tests for each possible combination of the parameter settings by:

for each iteration of the series of virtual tests, randomly selecting the set of virtual test sites, and randomly selecting a test start date for collecting historical data associated with each of the randomly selected virtual test sites.

60. The system of claim 59, wherein the computer is further configured to:

define a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.

61. The system of claim 53, wherein the computer is further configured to:

receive a request, identifying a first test type, to test the business initiative to apply to the business network, and

define the first test type of the predictive machine-learning business initiative testing model based on the request.

62. The system of claim 53, wherein the computer is further configured to:

iteratively perform the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.

63. The system of claim 53, wherein the computer is further configured to iteratively perform the virtual test by storing a noise value for each iteration and first parameter setting in a data structure.

64. The system of claim 62, wherein the computer is further configured to:

select a second parameter setting set when the iteration threshold is met; and

iteratively perform the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.

65. The system of claim 64, wherein the computer is further configured to iteratively perform the virtual test by storing a noise value for each iteration and second parameter setting set in a data structure.

66. The system of claim 64, wherein the computer is further configured to determine optimal parameter settings by:

analyzing the noise values associated with the first and second parameter settings sets; and

selecting either the first or second parameter settings sets set as the optimal parameter settings based on whether the first or second parameter settings sets have a lower average noise value over the iterations of the virtual test.

67. The system according to claim 53, wherein the first test type is associated with the predictive machine-learning business initiative testing model and is defined based on an attribute of the simulated business initiative test.

68. The system of claim 61, wherein the computer is further configured to:

define a second test type of the predictive machine learning business initiative model;

perform the virtual test based on the defined second test type to determine a second optimal parameter settings set for the second test type; and

perform a business initiative test using the second test type of the predictive machine-learning business initiative testing model that is configured using the second optimal parameter settings set.

69. A non-transitory computer-readable medium including instructions for, when executed by a processor, performing a method for determining optimal parameter settings for a predictive machine-learning model in business initiative testing software used to filter inconsistent data for testing initiatives for business locations included in a business network, the method comprising:

identifying, by a computer, a predictive machine-learning model business initiative testing model having a set of parameter settings;

selecting, by the computer, a first parameter setting set for performing a virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the predictive machine-learning model business initiative testing model, wherein at least one parameter setting includes a time period;

performing, by the computer, the virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test iteratively performed using a variation of a parameter setting option for each parameter setting and is performed on test sites where no actual initiative test is performed on the virtual test sites;

determining, by the computer, actual performance data associated with the set of virtual test sites;

determining, by the computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using tested parameter settings;

determining, by the computer, a noise value for the first parameter setting set, the noise value reflecting an inconsistency between actual performance data associated, with the set of virtual test sites and actual performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;

determining, by the computer, a set of optimal parameter settings for the predictive machine-learning model business initiative testing model based on results from performing the virtual test whereby the optimal parameter settings best minimize the noise value from the results;

training, by the computer, the predictive machine-learning business initiative testing model by automatically configuring, by the computer, the predictive machine-learning model business initiative testing model using the optimal parameter settings to best minimize noise based on the noise value for each parameter setting option of each parameter setting of the parameter setting set to test a business initiative to apply to the business network; and

executing, by the computer, the predictive machine-learning business initiative testing model using the set of optimal parameter settings.

Images