A system, method, and article of manufacture is disclosed for determining optimal parameter settings for a business initiative testing model used for testing initiatives for business locations included in a business network. In one aspect, a method is disclosed that includes defining a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting may include a set of one or more parameter setting options. The method may also include performing virtual tests on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the method may include determining a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Moreover, the method may include configuring the business initiative testing model using the optimal parameter settings from the set for the first test type to test a business initiative to apply to the business network.
|
0. 36. A method for determining optimal parameter settings for a predictive machine-learning model in business initiative testing software used to filter inconsistent data for testing initiatives for business locations included in a business network, comprising:
identifying, by a computer, a predictive machine-learning business initiative testing model having a set of parameter settings;
selecting, by the computer, a first parameter setting set for performing a virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the predictive machine-learning business initiative testing model, wherein at least one parameter setting includes a time period;
simulating performance of the first parameter setting set by performing, by the computer, the virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test performed on test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is iteratively performed on the virtual test sites using a variation of a parameter setting option for each parameter setting;
determining, by the computer, actual performance data associated with the set of virtual test sites;
determining, by the computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using tested parameter settings;
determining, by the computer, a noise value for the first parameter setting set, the noise value reflecting an inconsistency between actual performance data associated with the set of virtual test sites and actual performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
determining, by the computer, a set of optimal parameter settings for the predictive machine-learning business initiative testing model based on results from performing the virtual test whereby the optimal parameter settings best minimize the noise value from the results;
training, by the computer, the predictive machine-learning business initiative testing model by configuring, by the computer, the predictive machine-learning business initiative testing model using the optimal parameter settings based on the noise value for each parameter setting option of each parameter setting of the parameter setting set to test a business initiative for application in the business network; and
executing, by the computer, the predictive machine-learning model using the set of optimal parameter settings.
0. 53. A system for determining optimal parameter settings for a predictive machine-learning model in business initiative testing software used to filter inconsistent data for testing initiatives for business locations included in a business network, the system including:
a computer comprising a processor and a non-transitory computer-readable medium including instructions that when executed by the processor are configured to:
identify a predictive machine-learning business initiative testing model having a set of parameter settings;
selecting a first parameter setting set for performing a virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the predictive machine-learning business initiative testing model, wherein at least one parameter setting includes a time period;
execute the virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test for test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is iteratively performed on the virtual test sites using a variation of a parameter setting option for each parameter setting;
determine actual performance data associated with the set of virtual test sites; determine actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using tested parameter settings;
determine a noise value for the first parameter setting set, the noise value reflecting an inconsistency between actual performance data associated with the set of virtual test sites and actual performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
train the predictive machine-learning business initiative testing model by automatically configuring the predictive machine-learning business initiative testing model when the processor determines a set of optimal parameter settings for the predictive machine-learning business initiative testing model based on results from performing the virtual test whereby the optimal parameter settings best minimize the noise value from the results based on the noise value for each parameter setting option of each parameter setting;
configure the predictive machine-learning business initiative testing model using the optimal parameter settings to test a business initiative to apply to the business network; and
execute the predictive machine-learning business initiative testing model using the set of optimal parameter settings.
0. 69. A non-transitory computer-readable medium including instructions for, when executed by a processor, performing a method for determining optimal parameter settings for a predictive machine-learning model in business initiative testing software used to filter inconsistent data for testing initiatives for business locations included in a business network, the method comprising:
identifying, by a computer, a predictive machine-learning model business initiative testing model having a set of parameter settings;
selecting, by the computer, a first parameter setting set for performing a virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the predictive machine-learning model business initiative testing model, wherein at least one parameter setting includes a time period;
performing, by the computer, the virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test iteratively performed using a variation of a parameter setting option for each parameter setting and is performed on test sites where no actual initiative test is performed on the virtual test sites;
determining, by the computer, actual performance data associated with the set of virtual test sites;
determining, by the computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using tested parameter settings;
determining, by the computer, a noise value for the first parameter setting set, the noise value reflecting an inconsistency between actual performance data associated, with the set of virtual test sites and actual performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
determining, by the computer, a set of optimal parameter settings for the predictive machine-learning model business initiative testing model based on results from performing the virtual test whereby the optimal parameter settings best minimize the noise value from the results;
training, by the computer, the predictive machine-learning business initiative testing model by automatically configuring, by the computer, the predictive machine-learning model business initiative testing model using the optimal parameter settings to best minimize noise based on the noise value for each parameter setting option of each parameter setting of the parameter setting set to test a business initiative to apply to the business network; and
executing, by the computer, the predictive machine-learning business initiative testing model using the set of optimal parameter settings.
0. 1. A method for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, comprising:
identifying, by a computer, a business initiative testing model having a set of parameter settings;
selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;
performing, by a computer, a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test performed on test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is performed on the virtual test sites using a variation of each parameter setting;
determining, by a computer, actual performance data associated with the set of virtual test sites;
determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
determining, by a computer, a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and
configuring, by a computer, the business initiative testing model using the optimal parameter settings to test a business initiative for application in the business network.
0. 2. The method of
performing a series of virtual tests for each possible combination of the parameter settings for the business initiative testing model.
0. 3. The method of
determining a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.
0. 4. The method of
for each iteration of the series of virtual tests for a given parameter settings combination,
storing the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.
0. 5. The method of
for each given parameter settings combination,
determining an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.
0. 6. The method of
for each iteration of the series of virtual tests,
randomly selecting the set of virtual test sites, and
randomly selecting a test start date for collecting historical data associated with each of the randomly selected virtual test sites.
0. 7. The method of
iteratively performing the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.
0. 8. The method of
storing a noise value for each iteration and first parameter setting in a data structure.
0. 9. The method of
selecting a second parameter setting set when the iteration threshold is met; and
iteratively performing the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.
0. 10. The method of
storing a noise value for each iteration and second parameter setting in the data structure.
0. 11. The method of
analyzing the noise values associated with the first and second parameter settings sets; and
selecting either the first or second parameter settings set as the optimal parameter settings based on whether the first or second parameter settings sets have a lower average noise value over the iterations of the virtual test.
0. 12. The method according to
0. 13. The method of
identifying the parameter settings combination having the lowest average noise value as the optimal parameter settings for the first test type of the business initiative testing model.
0. 14. The method of
receiving a request identifying the first test type to test the business initiative to apply to the business network.
0. 15. The method of
defining a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.
0. 16. The method of
defining a second test type of the business initiative model;
performing the virtual test based on the defined second test type to determine second optimal parameter settings for the second test type; and
performing a business initiative test using the second test type of the business initiative testing model that is automatically configured using the second optimal parameter settings.
0. 17. The method according to
0. 18. A system for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, the system including:
a computer configured to: identify a business initiative testing model having a set of parameter settings;
selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;
execute a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test for-test sites where no actual initiative test has been implemented at those test sites, and wherein the virtual test is performed on the virtual test sites using a variation of each parameter setting;
determine actual performance data associated with the set of virtual test sites;
determine actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
determine a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and
configure the business initiative testing model using the optimal parameter settings to a test on a business initiative to apply to the business network.
0. 19. The system of
perform a series of virtual tests for each possible combination of the parameter settings for the business initiative testing model.
0. 20. The system of
determining a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.
0. 21. The system of
for each iteration of the series of virtual tests for a given parameter settings combination,
store the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.
0. 22. The system of
for each given parameter settings combination,
determine an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.
0. 23. The system of
0. 24. The system of
for each iteration of the series of virtual tests, randomly selecting the set of virtual test sites, and
randomly selecting a test start date for collecting historical' data associated with each of the randomly selected virtual test sites.
0. 25. The system of
defining a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.
0. 26. The system of
receive a request identifying the first test type to test the business initiative to apply to the business network, and
define the first test type of the business initiative testing model based on the request.
0. 27. The system of
iteratively perform the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.
0. 28. The system of
0. 29. The system of
select a second parameter setting set when the iteration threshold is met; and
iteratively perform the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.
0. 30. The system of
0. 31. The system of
analyzing the noise values associated with the first and second parameter settings sets; and
selecting either the first or second parameter settings set as the optimal parameter settings based on whether the first or second parameter settings sets have a lower average noise value over the iterations of the virtual test.
0. 32. The system according to
0. 33. The system of
define a second test type of the business initiative model;
perform the virtual test based on the defined second test type to determine second optimal parameter settings for the second test type; and
perform a business initiative test using the second test type of the business initiative testing model that is configured using the second optimal parameter settings.
0. 34. A non-transitory computer-readable medium including instructions for, when executed by a processor, performing a method for determining optimal parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network, the method comprising:
identifying, by a computer, a business initiative testing model having a set of parameter settings;
selecting a first parameter setting set for performing the virtual test, the first parameter setting set including a set of selected parameter setting options each respectively corresponding to one of the parameter settings for the business initiative testing model;
performing, by a computer, a virtual test on a set of virtual test sites, each virtual test site reflecting a selected business location in the business network, wherein each virtual test is a simulated business initiative test using a variation of each parameter setting and is performed on test sites where no actual initiative test is performed on the virtual test sites;
determining, by a computer, actual performance data associated with the set of virtual test sites;
determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
determining a noise value for the first parameter setting set, the noise value reflecting an inconsistency between performance data associated, with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
determining, by a computer, a set of optimal parameter settings for the business initiative testing model based on results from the virtual test whereby the optimal parameter settings best minimize noise from the results; and
configuring, by a computer, the business initiative testing model using the optimal parameter settings to test a business initiative to apply to the business network.
0. 35. A method for testing business initiatives comprising:
receiving, by a computer, a request to test a business initiative to be applied to a set of business locations in a business network;
determining, by a computer, actual performance data associated with a set of virtual test sites;
determining, by a computer, actual performance data associated with a set of control group sites reflecting second selected business locations in the business network using the tested parameter settings;
configuring, by a computer, a business initiative testing model based on the request and a default set of parameter settings for the model, each default parameter setting having a set of parameter setting options and the default set of parameter settings reflecting optimal parameter settings for the model that are determined by identifying a combination of parameter setting options that produce the least amount of noise during iterations of a set of virtual business initiative tests performed for selected virtual test sites, wherein each virtual test is a simulated business initiative test performed on test sites where no actual initiative test has been implemented at those test sites and wherein noise is a measurement of inconsistency between performance data associated with the set of virtual test sites and performance data associated with the set of control group sites reflecting second selected business locations in the business network using the tested parameter settings; and
testing, by a computer, the business initiative using the configured business initiative testing model to produce results reflecting a predicted performance of the business initiative if applied to the set of business locations.
0. 37. The method of claim 36, wherein performing the virtual test includes:
performing, by the computer, a series of virtual tests for each possible combination of the parameter settings for the predictive machine-learning business initiative testing model.
0. 38. The method of claim 37, wherein iteratively performing the series of virtual tests includes:
determining, by the computer, a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.
0. 39. The method of claim 38, further including:
for each iteration of the series of virtual tests for a given parameter settings combination, storing, by the computer, the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.
0. 40. The method of claim 38, further including:
for each given parameter settings combination, determining, by the computer, an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.
0. 41. The method of claim 37, wherein performing the series of virtual tests for each possible combination of the parameter settings includes:
for each iteration of the series of virtual tests, randomly selecting, by the computer, the set of virtual test sites, and
randomly selecting, by the computer, a test start date for collecting historical data associated with each of the randomly selected virtual test sites.
0. 42. The method of claim 36, further including:
iteratively performing, by the computer, the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.
0. 43. The method of claim 36, wherein iteratively performing the virtual test includes:
storing, by the computer, a noise value for each iteration and first parameter setting in a data structure.
0. 44. The method of claim 42, further including:
selecting, by the computer, a second parameter setting set when the iteration threshold is met; and
iteratively performing, by the computer, the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.
0. 45. The method of claim 44, wherein iteratively performing the virtual test includes:
storing, by the computer, a noise value for each iteration and second parameter setting in a data structure.
0. 46. The method of claim 44, wherein determining optimal parameter settings for a first test type includes:
analyzing, by the computer, the noise values associated with the first and second parameter settings sets; and
selecting, by the computer, either the first or second parameter settings set as the optimal parameter settings based on whether the first and second parameter settings sets have a lower average noise value over the iterations of the virtual test,
wherein the first test type is defined based on an attribute of the simulated business initiative test.
0. 47. The method according to claim 36, further comprising defining, by the computer, a test type associated with the predictive machine-learning business initiative testing model, wherein the test type is defined based on an attribute of the simulated business initiative test.
0. 48. The method of claim 46, wherein determining the set of optimal parameter settings includes:
identifying, by the computer, the parameter settings combination having the lowest average noise value as the optimal parameter settings for the first test type of the predictive machine-learning business initiative testing model.
0. 49. The method of claim 46, wherein configuring the predictive machine-learning business initiative testing model further includes:
receiving, by the computer, a request identifying the first test type to test the business initiative to apply to the business network.
0. 50. The method of claim 46, wherein defining the first test type includes:
defining, by the computer, a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.
0. 51. The method of claim 46, further including:
defining, by the computer, a second test type of the predictive machine-learning business initiative testing model;
performing, by the computer, the virtual test based on the defined second test type to determine second optimal parameter settings for the second test type; and
performing, by the computer, a simulated business initiative test using the second test type of the business initiative testing model that is automatically configured using the second optimal parameter settings.
0. 52. The method according to claim 36, wherein each parameter setting comprises a set of one or more parameter setting options, wherein a parameter setting option further defines a parameter setting.
0. 54. The system of claim 53, wherein the computer is further configured to:
perform a series of virtual tests for each possible combination of the parameter settings for the predictive machine-learning business initiative testing model.
0. 55. The system of claim 54, wherein the computer is further configured to perform the series of virtual tests by:
determining a noise value for the respective parameters settings combination for each iteration of the series of virtual tests for a given parameter settings combination.
0. 56. The system of claim 55, wherein the computer is further configured to:
for each iteration of the series of virtual tests for a given parameter settings combination, store the determined noise value in a data structure including data reflecting noise values for each iteration of the virtual test performed on the virtual test sites for each of the parameter settings combinations.
0. 57. The system of claim 55, wherein the computer is further configured to:
for each given parameter settings combination, determine an average noise value for the given parameter settings combination based on stored noise values for each iteration of the virtual test for the given parameter settings combination.
0. 58. The system of claim 57, wherein the computer is further configured to determine the set of optimal parameter settings by identifying the parameter settings combination having the lowest average noise value as the optimal parameter settings of the predictive machine-learning business initiative testing model.
0. 59. The system of claim 54, wherein the computer is further configured to perform the series of virtual tests for each possible combination of the parameter settings by:
for each iteration of the series of virtual tests, randomly selecting the set of virtual test sites, and randomly selecting a test start date for collecting historical data associated with each of the randomly selected virtual test sites.
0. 60. The system of claim 59, wherein the computer is further configured to:
define a test period and a test pre-period, each reflecting a period of time that historical data was collected for the business locations included in the business network.
0. 61. The system of claim 53, wherein the computer is further configured to:
receive a request, identifying a first test type, to test the business initiative to apply to the business network, and
define the first test type of the predictive machine-learning business initiative testing model based on the request.
0. 62. The system of claim 53, wherein the computer is further configured to:
iteratively perform the virtual test on the set of virtual test sites using the first parameter setting set until an iteration threshold is met.
0. 63. The system of claim 53, wherein the computer is further configured to iteratively perform the virtual test by storing a noise value for each iteration and first parameter setting in a data structure.
0. 64. The system of claim 62, wherein the computer is further configured to:
select a second parameter setting set when the iteration threshold is met; and
iteratively perform the virtual test on the set of virtual test sites using the second parameter setting set until the iteration threshold is met.
0. 65. The system of claim 64, wherein the computer is further configured to iteratively perform the virtual test by storing a noise value for each iteration and second parameter setting set in a data structure.
0. 66. The system of claim 64, wherein the computer is further configured to determine optimal parameter settings by:
analyzing the noise values associated with the first and second parameter settings sets; and
selecting either the first or second parameter settings sets set as the optimal parameter settings based on whether the first or second parameter settings sets have a lower average noise value over the iterations of the virtual test.
0. 67. The system according to claim 53, wherein the first test type is associated with the predictive machine-learning business initiative testing model and is defined based on an attribute of the simulated business initiative test.
0. 68. The system of claim 61, wherein the computer is further configured to:
define a second test type of the predictive machine learning business initiative model;
perform the virtual test based on the defined second test type to determine a second optimal parameter settings set for the second test type; and
perform a business initiative test using the second test type of the predictive machine-learning business initiative testing model that is configured using the second optimal parameter settings set.
|
|||||||||||||||||||||||||||||||||||||
This application is a Reissue of U.S. Pat. No. 8,571,916 (previously U.S. patent application Ser. No. 11/364,197, filed Mar. 1, 2006), which is a continuation-in-part of U.S. patent application Ser. No. 10/767,191, entitled “Methods, Systems, and Articles of Manufacture for Analyzing Initiatives for a Business Network,” and filed Jan. 30, 2004, now U.S. Pat. No. 8,010,399, which is are hereby incorporated by reference in its their entirety.
1. Field of the Invention
This invention relates to business initiative analysis systems, and more particularly, to methods, systems, and articles of manufacture for performing a segmented initiative analysis for initiatives implemented at selected business locations in order to identify in which other locations to implement the initiative. Further, this invention relates to methods and systems for determining optimal parameter settings for business initiative testing models.
2. Background of the Invention
For as long as retailers have been selling products and services, they have been seeking ways to increase profits. Accordingly, many retailers create new initiatives that they believe will have a positive impact on profits. These initiatives usually cover various aspects of business operations that drive profits. For example, retailers may change product prices, move products to different locations of a sales floor, change the amount of space allocated to each product, test new products, add or reduce staff, introduce new concepts (e.g., in-store kiosks), remodel older stores, and test new marketing campaigns. Retailers may test these new initiatives in selected test locations (i.e., certain stores) and subsequently determine whether to introduce the initiatives to remaining business locations based on the success of the initiatives at the selected test locations. Historically, retail management has used business instinct and/or anecdotal evidence to assess the success of the initiatives in order to make a decision whether to implement the initiatives at the rest of its business locations.
In recent years, however, some retailers have become more structured and analytical in their set up and analysis of tests. These retailers collect performance metrics, such as sales and gross profit data from the test locations and analyze the data using conventional software products, such as spreadsheet or statistical software packages. Retailers may measure the change in the performance metrics at the locations that introduced the new initiatives relative to a change in the same metrics at a control group of locations that did not implement the initiatives. In doing so, the retailers attempt to identify the impact of the initiatives on the performance metrics in order to determine whether they provide a positive return on investment. This allows these retailers can make an informed decision whether to extend the concept to remaining locations in a business network. However, the way in which the analyst at the retailer sets up this analysis will have an effect on the perceived results. For example, the analyst has to decide which of the stores that did not implement the initiative should serve as the control group and the analyst has to decide what time periods to use in measuring impact. Retailers often define one or more parameters for their analysis in a manner that seems intuitive, but lack empirical evidence that the parameters are in fact optimized for isolating the impact of an initiative. As a result, retailers may not recognize and filter inconsistent data, and therefore be less able to measure the impact of their initiatives.
Accordingly, there is a need for a system and method that automatically identifies one or more analytical parameters that filter out the most inconsistent data to maximize a retailer's ability to analyze the results of an initiative test. Using more consistent data allows retailers to better identify those initiatives to extend to certain locations that will provide the most anticipated profit gains.
Methods and systems consistent with certain aspects of the present invention identify optimum parameter settings for initiative testing models.
In one aspect of the invention, a method is provided for determining parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network. In one aspect, the method includes defining a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting may include a set of one or more parameter setting options. The method may also include performing a virtual test on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the method may include determining a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Moreover, the method may include configuring the business initiative testing model using the optimal parameter settings from the set for the first test type to test a business initiative for application in the business network.
In another aspect of the invention, a system is provided for determining parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network. In one aspect, the system may include a computer configured to define a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting including a set of one or more parameter setting options. Further, the computer is configured to execute a virtual test on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the computer may determine a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Also, the computer may configure the business initiative testing model using the optimal parameter settings from the set for the first test type to a test on a business initiative to apply to the business network.
In another aspect, a computer-readable medium is disclosed that includes instructions for, when executed by a processor, performing a method for determining parameter settings for business initiative testing software used for testing initiatives for business locations included in a business network. In one aspect, the method includes defining a first test type of a business initiative testing model having a set of parameter settings. Each parameter setting may include a set of one or more parameter setting options. The method may also include performing a virtual test on a set of virtual test sites based on the defined test type. Each virtual test site may reflect a selected business location in the business network. Also, the method may include determining a set of optimal parameter settings for the first test type of the business initiative testing model based on results from the virtual test. Moreover, the method may include configuring the business initiative testing model using the optimal parameter settings from the set for the first test type to test a business initiative to apply to the business network.
Additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of methods, systems, and articles of manufacture consistent with features of the present invention. The aspects of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several aspects of the invention and together with the description, serve to explain the principles of the invention. In the drawings,
Methods, systems and articles of manufacture consistent with certain aspects of the present invention identify business locations that respond most positively to a tested initiative and build a model based on the common attributes of those identified test sites to predict the impact of the tested initiative on other locations in a business network.
In one aspect of the invention, a server is disclosed for performing one or more processes for analyzing a business initiative for a business network including business locations having a set of test sites that have implemented the business initiative for a predetermined test period and a set of control group sites that did not have the initiative implemented. Each of the sites have an associated set of attributes reflecting various characteristics corresponding to the respective site, such as geographical location, size of business location, number of employees, etc. In one aspect of the invention, the server collects from a client a performance value for each of the test and control group sites reflecting a level of performance of each site during the test period. The server then calculates the change in performance in the test sites during the test period, relative to the change in performance in the control group sites over the same time period. The server may segment the performance values for each test site attribute to identify those attributes that have a greater impact on the performance values of the test sites than other attributes. Further, the system configures a model for predicting the performance values of the test sites using the identified attributes and determines whether the model accurately predicts these performance values. If so, the server applies the model to all non-tested sites (i.e., all sites in business network 110 excluding the test sites) to predict the performance values of each site. Based on the predicted performance values, a user operating the client or associated with the business network may select one or more of those sites to implement the business initiative.
Reference will now be made in detail to the exemplary aspects of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.
The above-noted features and other aspects and principles of the present invention may be implemented in various environments. Such environments and related applications may be specially constructed for performing the various processes and operations of the invention or they may include a general purpose computer or computing platform selectively activated or reconfigured by program code to provide the necessary functionality. The processes disclosed herein are not inherently related to any particular computer or other apparatus, and may be implemented by a suitable combination of hardware, software, and/or firmware. For example, various general purpose machines may be used with programs written in accordance with teachings of the invention, or it may be more convenient to construct a specialized apparatus or system to perform the required methods and techniques.
The present invention also relates to computer readable media that include program instructions or program code for performing various computer-implemented operations based on the methods and processes of the invention. The program instructions may be those specially designed and constructed for the purposes of the invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of program instructions include for example machine code, such as produced by a compiler, and files containing a high level code that can be executed by the computer using an interpreter.
Business network 110 represents a network of businesses that provide similar products and/or services to customers. The businesses may be geographically dispersed. Business network 110 may include one or more business regions 112-1 to 112-N that represent different geographical locations, such as cities, counties, states, country regions, virtual boundaries having a predetermined bounded distance (e.g., 100 mile radius), and any other type of defined region within a particular location. Each region 112-1 to 112-N may include one or more business locations 114 that represent retailers that sell products and/or services associated with business network 110. For example, business network 110 may represent a grocery store chain that includes a plurality of stores geographically dispersed among a state of the United States. Each region 112-1 to 112-N may represent locations that include grocery stores in the chain (e.g., business locations 114) that are within a predetermined distance of one another (e.g., 100 mile radius).
Also included in business network 110 is business management unit 116, which represents a business entity that manages business network 110. For example, business management unit 116 may be an organization headquarters or a marketing division for business network 110. Unit 116 may include one or more clients 118 that are operated by one or more users associated with business network 110 that perform various management and/or marketing duties for business network 110. Following the above example, business management unit 110 may be a marketing division of the exemplary grocery store chain that determines which products and/or services each retailer location (e.g., 114) should provide.
In one aspect of the invention, client 118 may be one or more computer systems such as a desktop computer, workstation, laptop, personal digital assistant or any other similar computing system operated by a user. Client 118 may include a processor, associated memory, and other elements and functionalities available in computer systems (all not shown). These elements may include input/output devices, such as a keyboard, mouse, and display, although these input means may take other forms. Also, client 118 may include a network interface for communicating with external resources, such as network 120 and server 130. Further, client 118 may execute browser software that enables a user to request, receive, and render information from network 120.
Although
Network 120 may be a shared, public, or private network and encompasses a wide area or local area. Network 120 may be implemented through any suitable combination of wired and/or wireless communication networks. For example, network 120 may be implemented through a Wide Area Network (“WAN”), Local Area Network (“LAN”), an intranet and/or the Internet. Further, network 120 may represent multiple networks, such as a wireless carrier network connected to the Internet.
Server 130 may be a computer system such as a desktop computer, workstation, or any other similar server side computing system that performs one or more server-side processes. In one aspect of the invention, server 130 may include, among other things, interface unit 132, processing unit, 134, memory 136, and database 140. Although server 130 is shown including elements 132-140, it may include additional computing elements, such as an input/output device (e.g., display, mouse, keyboard) that enables a user to access the data and software in server 130. In addition, server 130 may be a group of servers.
Interface unit 132 may be one or more hardware and/or software components that are configured to receive and send information from/to network 120. Interface unit 132 may include, or leverage, different types of software to control access to information maintained by server 130. For example, interface unit 132 may execute firewall software that controls access to information and processes provided by server 130 using one or more security protocols (e.g., password processing, encryption technologies, etc. The above listed functionalities are not intended to be limiting and interface unit 132 may perform other types of interface processes to facilitate the exchange of information between server 130 and network 120.
Processing unit 134 may be one or more processing devices that execute software processes stored in one or more memory devices, such as memory 136. Further, processing unit 134 may execute software processes stored internally, such as internal flash memory configured within processing unit 134.
Memory 136 may be one or more memory devices that store data and software programs that, when executed by processing unit 134, perform processes consistent with certain aspects related to the present invention. These programs may include segmentation engine 137, and model processing engine 138. Further, memory 136 may include server-side software applications that enable server 130 to create and provide content including information produced by other programs executed by processing unit 134, such as web server software. Alternatively, server 130 may leverage web server software that is located in another memory device (not shown) internal or external to server 130. The above software programs and functionalities are not intended to be limiting and memory 136 may be configured to store additional or fewer types of data and/or software programs that perform other processes consistent with server-side computing systems.
Database 140 may be one or more memory devices that store data and/or executable software that is used by server 130 to perform processes consistent with certain aspects related to the present invention. For example, database 140 may include one or more data structures (e.g., tables, maps, arrays, directories, etc.) that include data that is used by one or more of the software programs stored in memory 136 and executed by processing unit 134. Although server 130 shows database 140 and memory 136 as separate elements, the information and software stored in these elements may be combined in a single memory device. Also, memory 136 and/or database 140 may be located external to server 130 and accessible through network 120 or a dedicated back-end communication path, such as a Local Area Network (LAN).
Methods, systems, and articles of manufacturer consistent with aspects of the present invention enable one or more users of client 118 to access functionalities offered by server 130 to determine retail strategies associated with business network 110. For example, a user in business management unit 116 may execute browser software in client 118 to request analysis information regarding one or more initiatives implemented at one or more business locations 114.
Initially, a user or users at business management unit 116 may create and select an initiative that is to be applied to a predetermined number of test sites reflecting a set of business locations 114 (e.g., thirty stores) included in business network 110 (Step 210). The proposed initiative may be a change in store layouts, offering different products and/or services, different marketing campaigns (e.g., advertising, etc.), and any other type of idea that may be implemented at the selected test sites.
The user of business management unit 116 may then direct the selected test sites to implement the proposed initiative (Step 220). The initiative may be implemented for a predetermined amount of time (e.g., three months, six months, etc.), which is also known as a test period. Alternatively, the initiatives may be implemented at the test sites indefinitely. During the test period, business management unit 116 may collect feedback results from each test site based on one or more performance metrics selected by unit 116 (Step 230). For example, business management unit 116 may request sales information, foot traffic (i.e., number of persons entering a business location), etc. from each test site during the test period and may request additional performance data for each test site for a period before the test period.
Once the appropriate feedback results are collected, a user at business management unit 116 may use client 118 to perform an initiative analysis process based on the results (Step 240). The initiative analysis process provides information regarding the effectiveness of the initiative at the test sites. Further, the initiative process provides information regarding a proposed effectiveness of the initiative if implemented in other non-tested business locations in business network 110 (i.e., retailer locations that did not implement the initiative). These other business locations may be referred to as non-test group sites. In one aspect of the invention, the initiative analysis process includes the exchange of information between client 118 (under the operation of a user) and server 130. For example, a user operating client 118 may use browser software to access a web site that is provided or leveraged by server 130 to perform the initiative analysis process. Accordingly, server 103 may provide content to client 118 that includes queries, data, and analysis information associated with the initiative analysis process.
A user, such as a manager at business management unit 116, or software processes may analyze the information provided by server 130 and selects one or more applicable non-test group sites to implement the initiative (Step 250). Accordingly, business management unit 116 determines and implements one or more strategies associated with the tested initiative in a selected number of business locations 114, and/or groups of locations, such as regions 112-1 to 112-N.
In certain aspects, client 118 may download information to server 130 associated with the results of the test initiatives at the test sites. Further, server 130 may receive, or retrieve, attributes associated with each business location 114 from client 118, such as store size, locations, sale data, types of products and/or service offered, location and distances of competing merchant sites, age of the business location, amount of a lease (if any), weather during the test period, staff information, etc. Alternatively, or in addition to the information provided by client 118, server 130 may execute software that determines, collects, and/or retrieves attribute information associated with business network 110, regions 112-1 to 112-N (e.g., population and demographic data, etc.), and business locations 114. The attribute information may be stored in database 140 for subsequent use by server 130 and/or client 118.
Once the test sites are selected, server 130 may provide client 118 with information that enable the user to set up the initiative analysis (Step 310).
In addition to performance metrics and categories, the user may select time periods 540 from which the initiatives were implemented by the test sites. Also, the user may request that server 130 eliminates any initiative feedback data that appear to be anomalous by removing any outlier sites (selection 550). An outlier site has performance values that are too different than the rest of its respective test or control group. The user determines a threshold performance value beyond which a site is considered to be too different than its peers and thus deemed an outlier by server 130. By removing outlier test sites from consideration, the initiative analysis process does not consider the information for this test site during its analysis operations.
Additionally, or alternatively, the user may select a control group 560 of business locations in the network 110 that have not implemented the initiative. For example, the user may select to have all or a portion of the remaining business locations considered in the analysis process, or request that the server select a group of sites that are most similar to the test group of sites in which the initiative was implemented. This group of stores selected by the user or server 130 is the control group.
Once the user has set the appropriate performance metrics, categories, etc. for the initiative analysis process, server 130 may perform an initial analysis on the test site initiative result data (Step 315). In one aspect, server 130 may determine a performance value for each test site as well as the control group sites for each performance metric and category selected by the user. The performance value of a test site may be measured as a change in a performance metric/category combination for each test site relative to the change in the control group sites over a common time period. Once server 130 determines the performance values for the test sites, it may provide the results of the initial analysis to, user via client 118.
As shown in
Once server 130 completes the initial analysis process and provides the results of this process to the user, it may execute a segmentation process consistent with certain aspects related to the present invention (Step 320). The segmentation process may direct processor unit 134 to access the attributes associated with business locations 114 stored in database 140. Once accessed, the segmentation process sorts through the attributes to identify those that most strongly relate to the different test site's performance values during implementation of the initiative. In one aspect, server 130 may determine the strength of the relationship between each attribute and differential performance by test site (e.g., the attributes that are most in common among the top performing test sites and most different than the worst performing test sites).
Once determined, segmentation engine 137 may sort the attributes based on the strength relationship values (Step 715). The sorting allows segmentation engine 137 to have a prioritized list of attributes that are ranked based on their relationship with the performance of the test sites. Thus, referring to
In certain aspects, segmentation engine 137 may perform an unsegmented or segmented process on the attributes and test sites. Unsegmented processing involves analyzing the performance of each test site against the performance of all control group sites. Segmented processing involves analyzing the performance of each test site against the performance of control group test sites having similar attribute relationships. Steps 725-750 and 760-785 demonstrate the differences between the two processes.
If segmentation engine 137 is directed or configured to perform unsegmented processing (Step 720; UNSEGMENTED), an initial attribute may be selected for processing (Step 725). Segmentation engine 137 may analyze the performance metric values associated with the selected attribute and splits the test sites into fragments that represent sets of test sites that are divided based on the statistical rank of their performance metric values (Step 730). In one aspect of the invention, segmentation engine 137 separates the test sites into quartiles, quintiles, or any number of fragments. Segmentation engine 137 may then analyze the performance metric values of each test site attribute fragment against the performance of all control group test sites (Step 735). Based on the analysis, segmentation engine 137 may generate unsegmented attribute fragment results, which may include a resultant map of fragments representing the difference between the performance metrics of each test site fragment and all of the control group sites (Step 740). This information may be stored in memory 136 or database 140.
In one aspect of the invention, segmentation engine 137 may also determine a value representing a relationship between the performance of the test sites (i.e., performance metric) and the selected attribute. For example, segmentation engine 137 may determine the 1-p value of the relationship between the performance of the test sites (i.e., performance metric) and the selected attribute. The 1-p value represents the probability that the relationship between the attribute and the performance metric (either positive or negative) is not random chance.
If there are more attributes 803 to process (Step 745; YES), then segmentation engine 137 may select another attribute (Step 750) and the unsegmented process continues at Step 730. On the other hand, if there are no more attributes 803 to process (Step 745; NO), server 130 may end the segmentation process and proceed to Step 325 of the initiative analysis process of
If segmentation engine 137 is directed or configured to perform segmented processing on the attributes (Step 720; SEGMENTED), an initial attribute 803 is selected for processing (Step 760). Segmentation engine 137 may split the test site performance metric results into fragments that represent sets of test sites that are divided based on the statistical rank of their values for the selected attribute (Step 765). In addition, segmentation engine 137 may split the control group sites into fragments based on their values for the same selected attribute (Step 770). Segmentation engine 137 may then analyze each test site fragment against a corresponding control group fragment (Step 775). The results of the analysis are processed into segmented attribute fragment results that reflect the differences in the performance metric values for the test sites and corresponding control group sites for the selected attribute (Step 777). To better understand this segmented analysis,
As shown, memory 137 may store a set of control group fragment information 1010-1040 and a set of test site fragments 1050-1080 for a particular attribute under analysis. During a segmented process operation, segmentation engine 137 may compare corresponding fragments from each set to generate segmented attribute fragment information 1090-1096. For example, control group fragment 1010 may be analyzed against test site fragment 1050 to produce segmented attribute fragment 1090, while control group fragment 1020 is analyzed against test fragment 1060 to produce segmented attribute fragment 1092. The segmented attribute fragment information 1090-1096 is stored in memory 137 for subsequent processing. Accordingly, at Step 775 of
Also, segmentation engine 137 may determine a correlation value and a 1-p value in a manner similar to that described above in connection with Step 740 and
If there are more attributes 803 to process (Step 780; YES), then segmentation engine 137 may select another attribute (Step 785) and the segmented process continues at Step 765. On the other hand, if there are no more attributes 803 to process (Step 780; NO), server 130 may end the segmentation process and proceed to Step 325 of the initiative analysis process of
In addition to the processes performed during the segmentation process, server 130 may also generate pointers, software links, etc. between each attribute and its corresponding unsegmented and/or segmented attribute fragment information. Using these links, server 130 may generate content and data that allows the user to receive, view, and analyze the results of the segmentation processes (Step 325). For example,
As shown, attribute table 1100 may include a list of all of the attributes 1110 associated with the test sites that have implemented the proposed initiative. These attributes may correspond to attributes 803 described above in connection with
In another aspect of the invention, the attributes 1110 listed in table 1100 may be active in that the user may select an attribute (e.g., click) using an input device to view additional information regarding the attribute selected.
Additionally, or alternatively, server 130 may generate and render a scatter plot graph in which each test site is a data point in the graph, the X-axis is the value for the selected attribute and the Y-axis is the performance metric values. Server 130 may be configured to allow the user to select more than one attribute from attribute table 1100. In response to this selection, server 130 may process and generate attribute graphs (e.g., fragment-based, scatter plot-based, etc.) including information showing the relationship between the multiple attributes and their corresponding performance metric values for the test sites.
Referring back to
In one aspect of the invention, server 130 may offer different types of modeling methodologies that are used to predict performance values of the test and/or control group sites, such as linear regression, staged linear regression, neural network basic train, neural network stepwise regression, decision tree, K-means similar site modeling, and any other type of software model that may be used to analyze the information produced and/or collected by segmentation engine 137 and server 130 during the initiative analysis process. Server 130 may generate a web page that includes a list of modeling methodologies from which the user may select. Also, server 130 may allow the user to program a custom modeling methodology that may be used to analyze the initiative. Once the model is selected, server 130 may query the user for parameters and other type of information used by the selected model.
As shown, web page 1300 allows the user to select the type of performance metric 1310 to run in the selected model 1320. Further, server 130 may allow the user to select which attributes 1330 are available for consideration by the selected model when executed. As shown in
Further, the user may also direct server 130 to consider cross correlations 1360 between attributes when executing the selected model. Cross correlation is a parameter that reflects the similarity between attributes and their affect on the selected model's results. For example, two or more of the selected attributes 1330 may be closely related, such as an attribute for the number of competitive business locations within a five mile radius and another attribute for the number of competitive business locations within a ten mile radius. These two exemplary attributes may affect the model in a similar fashion. The user may not want such similar attributes considered in the selected model. Accordingly, by setting the cross correlation parameter 1360, the user directs server 130 to remove one of the similar attributes (leaving one attribute) from consideration by the selected model. This enables server 130 to consider independent attributes when executing the selected model. Also, the user may select the number of iterations 1370 the model should run when training itself for producing its results.
Once the model and associated parameters are selected, server 130 may execute the selected model (Step 335). In one aspect, server 130 may leverage model processing engine 138 to process the model. In such an instance, server 130 provides the user's selections and parameters to engine 138 for subsequent processing. Further, model processing engine 138 may re-execute the model based on the number of iterations 1370 selected by user in Step 330. In this regard, engine 138 tests and trains the model to calibrate its results. For example, model processing engine 138 may build a model using a subset of the test sites (e.g., 20 out of 30 test sites) and tests the results of the model on the remaining test sites (e.g., the remaining 10 test sites). The model may compute predicted performance metrics for each of the remaining test sites based on the attributes selected by the model and compares them to the actual performance metrics of the remaining test sites. Server 130 may repeat this process for the number of iterations set by the user in Step 330.
Once the model is executed, server 130 may generate and render the results to the user, via client 110. The user may then analyze the results of the executed model (Step 340). In one aspect, server 130 may generate a model results web page that includes information pertaining to the results of the executed model.
Moreover, server 130 may provide information showing how accurate the model predicts performance values of the test sites based on the selected parameters and the initiative. In one aspect, server 130 may provide quantitative measures reflecting the fitness of the model for predicting and analyzing the performance values of the test sites. For example, server 130 may provide an R-squared value 1450 reflecting a quantitative value of the model's fitness for predicting the success of the initiative in the control group sites. Alternatively, server 130 may express the fitness value as an error rate percentage value reflecting whether the predicted performance values are within a certain range of actual performance values associated with the test sites.
Additionally, server 130 may provide confidence measurements 1460 reflecting quantitative values of how well each attribute is actually contributing to producing a model that will successfully predict the performance metrics of the control group sites when implementing the initiative. Also, server 130 may determine and provide a percent populated value 1470 for each attribute 1410 that represents the percentage of test sites that server 130 has information for that particular attribute.
In another aspect of the invention, model processing engine 138 may generate a graph showing the results of the executed model.
The user of business management unit 116 uses information reflected in graph 1500 and/or model results web page 1400 to determine whether the selected model and/or attributes are acceptable (Step 345). Alternatively, a software program executed by server 130 may analyze the information produced by the executed model to determine whether the model and/or attributes considered by the model are acceptable based on predetermined rules. Various factors may be considered by the user, or software program, when determining whether the model and/or attributes are acceptable. For example, a user may determine that one of the attributes 1410 considered by the selected model was not an appropriate attribute based on the percent populated value 1440 for that attribute. Alternatively, or additionally, the confidence measurement 1460 for an attribute may cause the user or the software program to determine that that attribute is not acceptable for consideration by the model. On a broader scale, the user or software program may determine that the selected model is not appropriate for predicting performance metrics for the non-test sites based on the confidence measurement for that model (e.g., r-squared value 1450). The above exemplary factors are not intended to be limiting and the user and/or software program may use other result information to determine the appropriateness of the selected model and/or attributes.
If the results are not acceptable (Step 345; NO), the user and/or, software program may re-select a model, attributes, and/or parameters for analyzing the initiative (e.g., return to Step 330). For example, the user may remove one or more attributes from consideration prior to re-execution of the selected model. If, however, the results are acceptable (Step 345; YES), the user and/or software program may direct server 130 to apply the model to the non-test group sites to determine a predicted impact of the initiative on these sites (Step 350). The server 130 may do this by applying attributes 1410 and their corresponding coefficients 1420 selected by the model in step 335 to each non-test site's values for those attributes 1410. In doing so, server 130 calculates the predicted performance of the initiative on each non-test site based on that site's relevant attributes 1410.
Once server 130 applies the tested model to the non-test group sites and the test sites, a determination may be made as to those sites having the highest or more acceptable predicted performance metric values. Server 130 may rank the sites accordingly and provide this information to the user via client 118.
In another aspect of the invention, server 130 may provide results of the processed model based on any grouping of the sites. One such example of a group of sites are regions 112-1 to 112-N explained above in connection with
As described, methods and systems consistent with certain aspects of the present invention utilize a software model that is used for analyzing initiatives to be applied in test sites (e.g., business locations). In one aspect, server 130 may configure the model based one on or more selected parameters that control the manner in which the model processes data and measures the performance values of test sites. In one aspect, performance may be measured as a change in relevant performance values for the test sites from before the start of an initiative to during or after the initiative starts, relative to the change in performance in a control group over the same time period. For example, the parameters may include data reflecting whether the model should consider changes in absolute sales or in comparable (“comp”) (e.g., year over year) sales, the manner by which the model selects a control site group (e.g., sites not included in a selected set of test sites, for example, balance of market, comparable market, etc.), whether or not the model should filter outliers, the process used by the model to filter outliers, identifying the test measurement period, etc.
Certain aspects of the present invention enable server 130 to identify and set those model parameters that will best filter out noise associated with date related to the stores where the initiative was applied in order to produce more accurate results regarding the impact of the initiative. Noise may be a quantified measurement of inconsistent performance data for sites used in the analysis performed by the model. Aspects of the present invention create a simulation environment where the model performs a number of virtual initiative tests (e.g., initiative tests that have not actually been implemented in a business location) using different parameters. Based on the results of the virtual tests, server 130 may identify parameter settings that best filter noise from the results. Those parameter settings are then automatically selected for a particular test type as default settings that are subsequently used in performing actual initiative tests for predicting the performance of selected business locations based on the proposed initiative.
a. Overview of Determining Parameter Settings
Server, 130 may also define a test type that is associated with the business initiative testing model having a set of parameter settings (Step 1820). A test type may be a type of business initiative testing model that server 130 executes when running actual initiative tests to predict the performance of certain test sites, in a manner similar to the above disclosed aspects of the present invention. For example, a test type may be defined based on a particular characteristic of the initiative test, such as the length of a test period for the initiative test, etc. For purposes of illustrating aspects of the present invention, server 130 may define the test type based on a selected test period. It should be noted that any other attribute of a test may be used to define a test type. Server 130 may receive input from a user defining the test type or, alternatively, may automatically define the test type.
Once the test type if defined, server 130 may perform a virtual test on the defined test type (Step 1830). A virtual test reflects a mock initiative test that is performed on a selected number of test sites based on defined configuration settings for the model executed by server 130. The virtual test is considered a mock initiative test because no actual initiative test is planned or has been implemented in any virtual test sites (e.g., business locations identified for purposes of the virtual test). In this regard, server 130 creates a simulation environment for running virtual initiative tests for collecting information used for identifying the optimal default model parameters settings for the defined test type. Performing a virtual test may include configuring the virtual test such that server 130 iteratively performs a business initiative test for selected virtual test sites for each of a certain number of model parameter settings. Each model parameter setting may include one or more parameter setting options. Thus, for each iteration, server 130 may determine a noise value for each combination of available parameter settings and options and stores the values in a memory device for subsequent processing.
Server 130 may compare the noise values for each parameter setting combination to determine the parameter settings for the defined test type (Step 1840). In one aspect, server 130 may perform a statistical process on the noise values for each parameter setting combination, such as determining an average noise value for each combination over the iterations of collected noise values. In another aspect, server 130 may determine the parameter settings by identifying a parameter setting combination that produced the least amount of noise during the virtual test. Server 130 may identify this optimal parameter setting combination as a set of parameters settings that are to be used to configure the defined test type when executed for performing actual initiative tests to predict the performance of selected test sites. As such, server 130 may store the parameter settings for the defined test type in memory for subsequent use by the model (Step 1850). These and other aspects of the parameter setting process are described in further detail below in connection with
b. Performing a Virtual Test
Other types of parameter settings include a parameter that controls whether the model will use algorithms to filter outliers (e.g., performance data for a virtual test site that is too inconsistent with other related performance data). Thus, one option of this type of parameter setting may be to use outlier data, and another option may be to disregard outlier data. Other outlier parameter setting options may include the type of algorithm implemented when disregarding outlier data. For example, one option may be to remove the determined performance data entirely from the set of data used by the model. Another option may be to use a capped algorithm that changes the data of an outlier result to make the result no longer an outlier. For instance, if an outlier is defined as an observation (e.g., data result) that is more than three standard deviations from a mean, the capped algorithm may overwrite a outlier that is four standard deviations from the mean with a value that represents three standard deviations from the mean.
Another type of parameter setting may include whether the model will view changes in absolute performance data (e.g., actual sales data) or comp performance data (e.g., year to year sales data). For example, in measuring actual sales, a test site may have been selling $300 per day before an initiative was implemented. Following implementation, the test site may be selling $330 per day. Accordingly, the test site experienced a 10% lift in absolute sales. Comp sales, on the other hand, measures change in year on year sales. For example, a test site may have been selling 5% higher than in the prior year before an initiative was implemented. Following implementation, the test site may be selling 8% higher than in the prior year. Accordingly, the test site experienced a roughly 3% increase in comp sales.
A model may also have a parameter setting that defines how long of a pre-period to implement in the initiative tests run by the model. A pre-period may be a length of time prior to a selected test start date for a virtual test for which the model will measure performance before the initiative was implemented. In the above example, a test site was selling $300 per day before the initiative was implemented and $330 per day after the initiative was implemented. The $300 in sales per day represents performance in the pre-period. This exemplary parameter setting determines what length of time to use in measuring pre-period performance. For example, if a model is configured to compare performance data for a selected test period following a test start date (e.g., Sep. 1, 2004), a one month selected pre-period may be reflect the time period including all of August, 2004. Similarly, a one year pre-period may include the time period including the beginning of September 2003 to the end of August 2004. Any length of pre-period may be selected in accordance with certain aspects related to the present invention.
The above exemplary parameter settings and parameter setting options are not intended to be limiting. Server 130 may define any type of parameter setting and parameter setting options for any type of model, based on the characteristics of the model.
Once the number parameter settings and parameter setting options are selected, server 130 may determine the maximum number of parameter setting combinations that may be tested using the model. For example, for an exemplary set of parameters including four parameter settings, and each parameter setting having four respective parameter setting options, server 130 may determine there are 256 possible parameter setting combinations available for the model to test (e.g., {P1(1), P2(1), P3(1), P4(1)} . . . {P1(4), P2(4), P3(4), P4(4)}).
Thus, in selecting a parameter setting set, server 130 may set a value, PS(Max), to 256 representing the maximum number of combinations for the test type. Further, server 130 may then select a first parameter setting set reflecting a selected combination of parameter settings including one option from each available parameter setting. For instance, server 130 may select the first option from each of the four exemplary parameter settings mentioned in the above example (e.g., Parameter Setting Set (PS)={PS1(1), PS2(1), PS3(1), PS4(1)}). Server 130 may select any combination during step 1910. As explained later, however, server 130 may be configured to select a different parameter setting set for each iteration of the virtual test.
Server 130 may also select a number of virtual test sites for the virtual test (Step 1920). In one aspect, server 130 may randomly select a certain number of virtual test sites from the available sites reflecting business locations 114 in business network 110. For example, server 130 may select fifty (50) virtual test sites from an available pool of one thousand (1000) sites in business network 110.
Based on the selected test period associated with the defined test type (e.g., one month), server 130 may select a test start date (Step 1940). The test start date may be a previous date that is already included in the monitoring periods associated with the historical data collected for the business locations 114 (i.e., sites) in business network 110. For example, if database 140 maintains historical data up to Dec. 31, 2004, server 130 may randomly select a start date that is before that date. The test start date may be further bounded based on the selected test period. Thus, for a one month test period, server 130 may be configured to select a test start date that is no later than Nov. 30, 2004 to allow for the test period to include the historical data for Dec. 31, 2004. Similarly, for a six month test period, server 130 may be configured to select the test start date no later than Jun. 30, 2004. Within these boundaries, server 130 may randomly select the test start date. Alternatively, server 130 may receive user input identifying the test start date.
In another aspect of the invention, server 130 may select a pre-period associated with the test period. A pre-period is a selected period of time extending back from the test start date. For example, server 130 may select a pre-period of one month. Thus, if Sep. 1, 2004 is selected as the test start date, the one-month pre-period would include Aug. 1, 2004 through Aug. 31, 2004. The pre-period may be equal to, or different from, the selected test period for the defined test type. Also, the pre-period may be selected based on the type of parameter settings and corresponding parameter setting options selected in step 1910. That is, if the test type includes a parameter setting option for determining how to select the pre-period, server 130 selects the pre-period according to that selected parameter setting option.
Also, based on the selected parameter settings, server 130 may determine a control site group for the virtual test, reflecting a selected set of sites (i.e., business locations 114) within business network 110. For example, a control group selection parameter setting option may direct server 130 to select the control group from the remaining sites in business network 110 that are not included in the set of selected virtual test sites. Thus, server 130 may select the remaining 950 sites in the exemplary 1000 sites of business network 110 that were not chosen for the 50 virtual test sites. As noted above, a control group parameter setting may configure server 130 to select the control group based on different criteria, and thus the above examples are not intended to be limiting.
Once the parameter setting set, virtual test sites, and test start date are determined, server 130 may execute the model to run a virtual test. The model may execute an initiative test based on the configuration information and produce a result for a selected performance metric (Step 1950). For instance, if the model is configured to monitor sales data, server 130 may execute a process to collect sales data for the selected virtual test sites for the selected pre-period and test period. Thus, following the above examples, server 130 may collect and analyze the historical data to determine the sales data for the fifty virtual test sites during August 2004 and September 2004. Additionally, server 130 may collect and analyze the historical data for the control group to determine the sales data for the sites within that group (e.g., 950 sites). As noted above, server 130 may produce different types of result data based on the type of performance metric being monitored by server 130 when performing the virtual test.
Once the performance data (e.g., sales data) is determined, server 130 may determine the noise for the parameter setting set selected in step 1910 (e.g., (PS)={PS1(1), PS2(1), PS3(1), PS4(1)}) (Step 1960). As explained, noise may be represented as a value reflecting an inconsistency in the performance data for the virtual test sites in relation to the control site group. Thus, when the performance data for the virtual test sites vary from the performance data for the control group sites, server 130 may calculate a value reflecting this difference. Because the server 130 has created a simulation environment by performing the virtual test on the virtual test sites, no actual initiative was performed in these test sites. Accordingly, the performance data for the virtual test sites determined during the virtual test should be similar, if not identical to, the performance data for the control group. For instance, server 130 may determine during the virtual test that the control group experienced a 2.5% sales lift from August 2004 to September 2004 (e.g., performance metric result data for the pre-period vs. the performance metric result data for the test period). Server 130 may also determine that the virtual test sites experienced a 3% sales lift during the same time frame. This difference (i.e., 0.5% lift) may be the result of a difference in the sales trends between the test set and control group. Accordingly, server 130 may identify the difference as a quantified value of noise. Server 130 may then store the determined noise for the selected parameter setting set in a data structure maintained in a memory device, such as database 140 (Step 1970).
According to certain aspects of the invention, server 130 may be configured to iteratively run a virtual test for the selected parameter setting for a predetermined number of times. Accordingly, server 130 may define a iteration threshold that controls how many times the selected parameter setting set is tested (e.g., 10 times, 100 times, etc.). The iteration threshold value may be determined at any time before checking the threshold, such as when the test type is defined (e.g., step 1820), when selecting the parameter setting set (step 1910), etc. Therefore, as server 130 performs a virtual test for a selected parameter setting set, it may track the number of times it has run a virtual test on the set. As such, in step 1980, server 130 determines whether the iteration threshold has been met. If so (Step 1980; Yes), the virtual test process continues to Step 1990. However, if the iteration threshold is not met (Step 1980; No), server 130 may return to step 1920 to select a new set of virtual test sites and new test date to run another virtual test for determining a new noise value for the selected parameter setting set. The new noise value for the current iteration is also stored in the data structure. This process continues until the iteration threshold is met (e.g., the selected parameter setting is tested 100 times).
Once the iteration threshold is met, server 130 may determine whether all combinations of the parameter settings have been iteratively tested. In one aspect, server 130 may determine whether a parameter setting set threshold is met, which may be set equal to PS(Max) previously determined by server 130 (Step 1990). If the threshold is met (Step 1990; Yes), the virtual test process ends (Step 1995). On the other hand, if the parameter setting set threshold is not met (Step 1990; No), server 130 selects a new parameter setting set from the available combinations previously determined, and repeats the iterative testing processes described above. In this regard, server 130 ensures each combination of parameter settings based on their corresponding options have been iteratively tested, and a noise value determined and stored in the data structure.
c. Determining Noise
As explained, server 130 may execute a software process that determines noise for each iteration of a virtual test of a selected parameter setting set. Noise reflects a quantified measurement of inconsistent performance data for sites used in the analysis performed by a business initiative testing model.
In step 2010, server 130 may determine the performance data for the virtual test during the pre-period and test period. In one aspect, server 130 may access database 140 to collect the historical data for each virtual test site to determine, based on the selected performance metric, the performance data during each time period. Server 130 may average the determined performance data for the virtual test sites, or determine other statistical values associated with the determined performance data. Similarly, server 130 also determines the performance data for the control group sites during the pre-period and test period (Step 2020). Server 130 may then determine the noise for the selected parameter setting set based on a comparison of the performance data for the virtual test sites and control group sites (Step 2030).
To better understand these aspects of the present invention,
Although as described above, server 130 determines noise based on a difference of performance data, methods and systems consistent with certain aspects of the present invention are not limited to the above examples and may quantify noise based on other relationships between performance data of the test and control sites.
d. Determining Parameter Settings
In another aspect of the present invention, server 130 may store noise values in a data structure based on each parameter setting option.
Aspects of the present invention are not limited to the configurations shown in
Once the average noise value for each parameter setting is determined, server 130 may determine the optimal combination of parameter settings for the parameter settings (Step 2220). In one aspect, server 130 may identify the parameter setting set having the lowest average noise as the optimal parameter setting set. For example, server 130 may search the data structure storing the average noise values to identify the lowest values. Thus, for example, if a parameter setting set 3 of
In another example corresponding to data structure 2400, server 130 may identify the lowest average noise value for each parameter setting option for each parameter setting, and then combine these lowest average noise values to form the optimal parameter settings to define the default settings. For example,
It should be noted, server 130 may be configured to execute programs that identify the lowest noise values for the parameter settings using different methods than that described above. As such, the above described processes are exemplary and not intended to be limiting.
e. Performing an Initiative Test Using Parameters
In accordance with certain aspects of the present invention, once server 130 identifies the parameter settings for a given test type, the setting is stored in a memory device (e.g., database 140) for subsequent use by server 130 when running an initiative test for selected test sites.
Variations of the methods and systems consistent with features of the present invention previously described may be implemented without departing from the scope of the invention. For example, server 130 may be configured with software that automatically performs some or all of the analysis and decisions performed by a user operating client 118. For instance, server 130 may include software stored in memory 136 that, when executed by processing unit 134, analyzes the list of attributes ranked according to their impact on the performance metric values of the test sites (e.g., correlation and/or r-squared values). Server 130 may also execute software that selects the model and associated parameters (e.g., attributes, iterations, etc.) for predicting the performance levels of the test and non-test group sites. Additional, server 130 may execute software that selects one or more of the non-test group sites to implement the tested initiative based on the predicted performance values produced by the executed model. The above processes are not intended to be limiting and other procedures may be performed by software processes executed by server 130 and/or client 118 to supplement and/or compliment those decisions, data input, and analysis performed by a user.
Also, although the initiative analysis process is described as using web server and browser software for exchanging information with client 118 and server 120, methods and systems consistent with aspects of the invention may use any type of technology to allow a user to send and receive information from server 120. Further, although the above described aspects of the invention include communications between a client 118 and a server 120, methods and systems consistent with aspects of the invention are not limited to client-server network configurations. That is, a user associated with business network 110 may leverage a computing environment that is local (i.e., same computing system) to a system operated by the user.
Moreover, the virtual test processes described herein are not limited to the sequences, steps, and systems shown in
Further, methods, systems, and articles of manufacture, consistent with features of the present invention may be implemented using various network models, and is not limited to a particular computer and/or network environment. Furthermore, methods, systems, and articles of manufacture, consistent with features of the present invention are not limited to the implementation of systems and processes compliant any particular type of programming language. Any number of programming languages may be utilized without departing from the scope of the present invention.
Additionally, although aspects of the present invention are described as being associated with data stored in memory and other storage mediums, one skilled in the art will appreciate that these aspects can also be stored on or read from other types of computer-readable media, such as secondary storage devices, like hard disks, floppy disks, or CD-ROM; a carrier wave from the Internet; or other forms of RAM or ROM. Accordingly, the invention is not limited to the above described aspects of the invention, but instead is defined by the appended claims in light of their full scope of equivalents.
Bruce, Anthony Dean, D'Agostino, Mark James, Vaudo, Corey Christian, Zhao, Dacheng
| Patent | Priority | Assignee | Title |
| Patent | Priority | Assignee | Title |
| 4972504, | Feb 11 1988 | A C NIELSEN US , INC | Marketing research system and method for obtaining retail data on a real time basis |
| 5138638, | Jan 11 1991 | SHOPPERTRAK RCT CORPORATION | System for determining the number of shoppers in a retail store and for processing that information to produce data for store management |
| 5175797, | Sep 01 1989 | HITACHI, LTD , A CORP OF JAPAN | Learning type decision support system |
| 5227874, | Mar 10 1986 | QUEST NETTECH CORPORATION | Method for measuring the effectiveness of stimuli on decisions of shoppers |
| 5245533, | Dec 18 1990 | NCH PROMOTIONAL SERVICES, INC | Marketing research method and system for management of manufacturer's discount coupon offers |
| 5315093, | Feb 05 1992 | A C NIELSEN US , INC | Market research method and system for collecting retail store market research data |
| 5406477, | Aug 30 1991 | HEWLETT-PACKARD DEVELOPMENT COMPANY, L P | Multiple reasoning and result reconciliation for enterprise analysis |
| 5613217, | May 03 1995 | Telefonaktiebolaget LM Ericsson | Transceiver site selection a cellular communications system |
| 5692107, | Mar 15 1994 | Lockheed Missiles & Space Company, Inc. | Method for generating predictive models in a computer system |
| 5826252, | Jun 28 1996 | General Electric Company | System for managing multiple projects of similar type using dynamically updated global database |
| 6055524, | Oct 06 1997 | General Cybernation Group, Inc. | Model-free adaptive process control |
| 6078891, | Nov 24 1997 | Method and system for collecting and processing marketing data | |
| 6321206, | Mar 05 1998 | CGI TECHNOLOGIES AND SOLUTIONS INC | Decision management system for creating strategies to control movement of clients across categories |
| 6430539, | May 06 1999 | CALLAHAN CELLULAR L L C | Predictive modeling of consumer financial behavior |
| 6484158, | Dec 19 1997 | A C NIELSEN US , INC | Dynamic rule based market research database |
| 6553352, | May 04 2001 | ACOUSTIC, L P | Interface for merchandise price optimization |
| 6708156, | Apr 17 2000 | Michael von Gonten, Inc. | System and method for projecting market penetration |
| 6801818, | Mar 14 2001 | The Procter & Gamble Company | Distributed product development |
| 6934748, | Aug 26 1999 | Accenture Global Services Limited | Automated on-line experimentation to measure users behavior to treatment for a set of content elements |
| 6966061, | Sep 20 2001 | BellSouth Intellectual Property Corp. | System and method for managing initiative delivery process |
| 6970830, | Dec 29 1999 | General Electric Capital Corporation | Methods and systems for analyzing marketing campaigns |
| 7072863, | Sep 08 1999 | c4cast.com, Inc.; C4CAST COM, INC | Forecasting using interpolation modeling |
| 7080027, | Apr 17 2003 | TARGETRX, INC | Method and system for analyzing the effectiveness of marketing strategies |
| 7092896, | May 04 2001 | ACOUSTIC, L P | Interface for merchandise promotion optimization |
| 7155398, | Feb 19 2003 | International Business Machines Corporation | Cascaded planning of an enterprise planning model |
| 7213023, | Oct 16 2000 | UNIVERSITY OF NORTH CAROLINA AT CHARLOTTE, THE | Incremental clustering classifier and predictor |
| 7308497, | Aug 26 1999 | Accenture Global Services Limited | On-line experimentation |
| 7412398, | Jun 12 1997 | Method for analyzing net demand for a market area utilizing weighted bands | |
| 7451065, | Mar 11 2002 | GLOBALFOUNDRIES Inc | Method for constructing segmentation-based predictive models |
| 7526434, | Jan 30 2001 | Network based system and method for marketing management | |
| 7546246, | Jun 25 2002 | Bellsouth Intellectual Property Corporation | Methods and systems for change initiative management |
| 7548879, | Jul 18 2002 | NCR Voyix Corporation | Convenience store effectiveness model (CSEM) |
| 7599848, | Feb 13 2002 | SAP SE | System and methods and risk evaluation using an object measure-value in strategic planning |
| 7689456, | Dec 04 2001 | Kimberly-Clark Worldwide, Inc | System for predicting sales lift and profit of a product based on historical sales information |
| 7689459, | Sep 24 2003 | Industrial Technology Research Institute | Card with embedded bistable display having short and long term information |
| 7729931, | Mar 17 2003 | Verizon Patent and Licensing Inc | Systems and methods for comparing and improving sales performance over heterogeneous geographical sales regions |
| 7895072, | Jan 30 2004 | Applied Predictive Technologies | Methods, system, and articles of manufacture for developing analyzing, and managing initiatives for a business network |
| 7904327, | Apr 30 2002 | SAS INSTITUTE INC | Marketing optimization system |
| 8151247, | Oct 09 2006 | SAP SE | Test data management |
| 20010032105, | |||
| 20010042003, | |||
| 20020069099, | |||
| 20020133250, | |||
| 20020169665, | |||
| 20030018510, | |||
| 20030023841, | |||
| 20030046125, | |||
| 20030046126, | |||
| 20030050830, | |||
| 20030069659, | |||
| 20030069782, | |||
| 20030110080, | |||
| 20030130883, | |||
| 20030176931, | |||
| 20030177055, | |||
| 20030195793, | |||
| 20030212584, | |||
| 20040054511, | |||
| 20040093315, | |||
| 20040143477, | |||
| 20040143480, | |||
| 20040210471, | |||
| 20040243485, | |||
| 20040260624, | |||
| 20050039206, | |||
| 20050055275, | |||
| 20050060224, | |||
| 20050075921, | |||
| 20050080609, | |||
| 20050108082, | |||
| 20050200476, | |||
| 20060105775, | |||
| 20060195370, |
| Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
| Apr 27 2021 | Applied Predictive Technologies, Inc. | (assignment on the face of the patent) | / |
| Date | Maintenance Fee Events |
| Apr 27 2021 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
| Date | Maintenance Schedule |
| Jun 27 2026 | 4 years fee payment window open |
| Dec 27 2026 | 6 months grace period start (w surcharge) |
| Jun 27 2027 | patent expiry (for year 4) |
| Jun 27 2029 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Jun 27 2030 | 8 years fee payment window open |
| Dec 27 2030 | 6 months grace period start (w surcharge) |
| Jun 27 2031 | patent expiry (for year 8) |
| Jun 27 2033 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Jun 27 2034 | 12 years fee payment window open |
| Dec 27 2034 | 6 months grace period start (w surcharge) |
| Jun 27 2035 | patent expiry (for year 12) |
| Jun 27 2037 | 2 years to revive unintentionally abandoned end. (for year 12) |