A method for modeling a set of observed data comprises selecting a reference model and an alternative model as possible descriptions of the set of observed data, and storing an index function for measuring fit of models to data. The method further includes performing, by one or more processors, a simulated threshold-fitting for a first of the two models, deriving an initial simulated index for the second model for fitting the second model to the simulated data, and deriving an initial boundary for simulated index difference including calculating a difference between the threshold-fit simulated index for the first model and the initial simulated index for the second model. The method further includes determining, based on a comparison, whether to update a counter used in calculating a simulated p-value, and selecting, based on the simulated p-value, one of the reference and alternative models for modeling the set of observed data.
|
18. A system for model selection, the system comprising:
an comparison system, including one or more processors, for determining a simulated p-value that determines a threshold-fit simulated index for a first model and, while an early stop condition remains unsatisfied, iteratively compares an observed index difference to a boundary for a simulated index difference, the boundary for the simulated index difference based on a simulated index for a second model, the comparison system comprising:
an analyzer that:
selects a reference model as a possible description of a set of observed data, wherein the reference model includes a reference set of parameters,
selects an alternative model as another possible description of the set of observed data, wherein the alternative model includes an alternative set of parameters,
stores an index function for measuring fit of models to data,
calculates the observed index difference based on a difference in the index function for fitting the reference model to the observed data and for fitting the alternative model to the data, and
selects the first model and the second model, wherein the first model is one of the reference model and the alternative model and the second model is the other one of the reference model and the alternative model,
a simulator that generates a set of simulated data;
a first maximizer that performs an operation based on the set of simulated data, wherein the operation includes performing a simulated threshold-fitting for the first model by the one or more processors, wherein the simulated threshold-fitting for the first model includes varying a set of parameters for the first model for fitting the first model to the simulated data and deriving the threshold-fit simulated index for the first model as a value of the index function for the first model that satisfies a threshold criterion; and
a second maximizer that derives an initial simulated index for the second model as an initial value of the index function for fitting the second model to the simulated data; and
wherein the analyzer further:
derives an initial boundary for the simulated index difference, wherein deriving the initial boundary for the simulated index difference includes calculating a difference between the threshold-fit simulated index for the first model and the initial simulated index for the second model;
determines, based on a comparison of the initial boundary for the simulated index difference and the observed index difference, whether to update a counter used in calculating the simulated p-value; and
selects, based on the simulated p-value, one of the reference model and the alternative model for modeling the set of observed data.
19. A non-transitory computer-readable medium storing a computer program, wherein the computer program, when executed by one or more processors, causes the one or more processors to execute a method for model selection, the method comprising:
selecting a reference model as a possible description of a set of observed data, wherein the reference model includes a reference set of parameters;
selecting an alternative model as another possible description of the set of observed data, wherein the alternative model includes an alternative set of parameters;
storing an index function for measuring fit of models to data;
calculating an observed index difference based on a difference in the index function for fitting the reference model to the observed data and for fitting the alternative model to the data;
determining a simulated p-value using an comparison system that determines a threshold-fit simulated index for a first model and, while an early stop condition remains unsatisfied, iteratively compares the observed index difference to a boundary for a simulated index difference, the boundary for the simulated index difference based on a simulated index for a second model, determination of the simulated p-value comprising:
generating a set of simulated data;
selecting the first model and the second model, wherein the first model is one of the reference model and the alternative model and the second model is the other one of the reference model and the alternative model;
performing, by the one or more processors, an operation based on the set of simulated data, wherein the operation includes:
performing a simulated threshold-fitting for the first model by the one or more processors, wherein the simulated threshold-fitting for the first model includes varying a set of parameters for the first model for fitting the first model to the simulated data and deriving the threshold-fit simulated index for the first model as a value of the index function for the first model that satisfies a threshold criterion;
deriving an initial simulated index for the second model as an initial value of the index function for fitting the second model to the simulated data; and
deriving an initial boundary for the simulated index difference, wherein deriving the initial boundary for the simulated index difference includes calculating a difference between the threshold-fit simulated index for the first model and the initial simulated index for the second model;
determining, based on a comparison of the initial boundary for the simulated index difference and the observed index difference, whether to update a counter used in calculating the simulated p-value; and
selecting, based on the simulated p-value, one of the reference model and the alternative model for modeling the set of observed data.
1. A method for model selection, the method comprising:
selecting, by one or more processors, a reference model as a possible description of a set of observed data, wherein the reference model includes a reference set of parameters;
selecting, by the one or more processors, an alternative model as another possible description of the set of observed data, wherein the alternative model includes an alternative set of parameters;
storing, by the one or more processors, an index function for measuring fit of models to data;
calculating, by the one or more processors, an observed index difference based on a difference in the index function for fitting the reference model to the observed data and for fitting the alternative model to the data;
determining, by the one or more processors, a simulated p-value using an comparison system that determines a threshold-fit simulated index for a first model and, while an early stop condition remains unsatisfied, iteratively compares the observed index difference to a boundary for a simulated index difference, the boundary for the simulated index difference based on a simulated index for a second model, determination of the simulated p-value comprising:
generating, by the one or more processors, a set of simulated data;
selecting, by the one or more processors, the first model and the second model, wherein the first model is one of the reference model and the alternative model and the second model is the other one of the reference model and the alternative model;
performing, by the one or more processors, an operation based on the set of simulated data, wherein the operation includes:
performing a simulated threshold-fitting for the first model by the one or more processors, wherein the simulated threshold-fitting for the first model includes varying a set of parameters for the first model for fitting the first model to the simulated data and deriving the threshold-fit simulated index for the first model as a value of the index function for the first model that satisfies a threshold criterion;
deriving an initial simulated index for the second model as an initial value of the index function for fitting the second model to the simulated data; and
deriving an initial boundary for the simulated index difference, wherein deriving the initial boundary for the simulated index difference includes calculating a difference between the threshold-fit simulated index for the first model and the initial simulated index for the second model;
determining, by one or more processors, based on a comparison of the initial boundary for the simulated index difference and the observed index difference, whether to update a counter used in calculating the simulated p-value; and
selecting, by one or more processors based on the simulated p-value, one of the reference model and the alternative model for modeling the set of observed data.
2. The method of
3. The method of
4. The method of
5. The method of
deriving an updated boundary for the simulated index difference, wherein deriving the updated boundary for the simulated index difference includes deriving an updated simulated index for the second model as an updated value of the index function for fitting the second model to the simulated data and calculating a difference between the threshold-fit simulated index for the first model and the updated simulated index for the second model;
and wherein the method further comprises determining, based on a comparison of the updated boundary for the simulated index difference and the observed index difference, whether to update the counter.
6. The method of
7. The method of
determining that the initial lower boundary for the simulated index difference is smaller than the observed index difference; and
deriving an updated lower boundary for the simulated index difference, wherein deriving the updated lower boundary for the simulated index difference includes deriving an updated simulated index for the alternative model as an updated value of the index function for fitting the alternative model to the simulated data and subtracting the threshold-fit simulated index for the reference model from the updated simulated index for the alternative model;
and wherein the method further comprises increasing the counter if the updated lower boundary for the simulated index difference is larger than or equal to the observed index difference, and wherein the early stop condition is satisfied when the updated lower boundary for the simulated index difference is larger than or equal to the observed index difference.
8. The method of
increasing the counter if the simulated index difference is larger than or equal to the observed index difference; and
determining to leave the counter unchanged if the simulated index difference is smaller than the observed index difference.
9. The method of
10. The method of
determining that the initial upper boundary for the simulated index difference is not smaller than the observed index difference; and
deriving an updated upper boundary for the simulated index difference, wherein deriving the updated upper boundary for the simulated index difference includes deriving an updated simulated index for the reference model as an updated value of the index function for fitting the reference model to the simulated data and subtracting the updated simulated index for the reference model from the threshold-fit simulated index for the alternative model;
and wherein the method further comprises determining to leave the counter unchanged if the updated upper boundary for the simulated index difference is smaller than the observed index difference, and wherein the early stop condition is satisfied when the updated upper boundary for the simulated index difference is smaller than the observed index difference.
11. The method of
increasing the counter if the simulated index difference is larger than or equal to the observed index difference; and
determining to leave the counter unchanged if the simulated index difference is smaller than the observed index difference.
12. The method of
13. The method of
14. The method of
15. The method of
setting a weight value to 0.5 if an iteration of the iterative comparison is a first iteration of the iterative comparison;
otherwise calculating the weight value as a ratio of previous iterations in which the counter was decided to be increased; and
creating a random variable for which a probability of being one is the weight value; and
setting the first model to be the reference model if the random variable is one, otherwise setting the first model to be the alternative model.
16. The method of
|
The present disclosure relates generally to methods and systems for comparing parametric statistical models and in particular to likelihood ratio tests.
Empirical studies in many fields of science and technology collect a set of data and use the data to examine a theoretical model related to that field. Such empirical studies apply in a variety of areas, such as healthcare, drug development, finance, or stock market modeling and predictions.
In one example, a cancer drug development group identifies a family of drugs as candidates for inhibiting the growth of cancerous tumor cells. The drugs can be administered in one of three candidate carrier solutions. The group wants to identify an effective drug and a corresponding suitable family of carrier solutions. To that end, the group compares the drugs, two at a time, by performing a bioassay experiment on multiple equivalent tumor cells placed in multiple isolated slots of a bioassay plate. The group treats randomly selected subsets of the slots with each of the two drugs administered in one of the three carrier solutions. After a period of time, the cells are examined to determine the number of tumor cells that are alive in each slot. The effectiveness of each drug is measured by the number of cells that are killed during the time period. The group would like to determine whether there exists any variability in the effectiveness of drugs across the drug family, in the effectiveness of carrier solutions across the carrier family, and in the interactions between different drug and carrier solution combinations.
In another example, a clinical trial group studies a newly developed antihypertensive drug for lowering blood pressure in humans. To that end, the clinical trial selects a group of patients with a history of hypertension. Each patient visits the clinic five times. During each visit, the patient randomly receives one of the five options: a placebo and four different doses of the drug. The clinical trial monitors the blood pressure and other health parameters of each patient during the study. Based on the results, the scientists would like to determine the effectiveness of the drug and, if effective, a proper dose of the drug.
Many of these studies apply statistical modeling to solve the problem. In statistical modeling, researchers often attempt to describe the observed data with a model. In many cases the researchers know of multiple candidate models that may describe the data and want to find from among those candidates a model that describes the data in a best manner. In many cases, a candidate model is a parametric model, which includes some variable parameters.
For example, the cancer drug development example above may use a 2-way completely randomized analysis of variance (ANOVA) with interactions model. In this parametric model, the measured data are explained by equation (1):
yijk=μ+αi+bj+cij+εijk (1)
In equation (1), the index i runs over the values 1, and 2 for the two drugs; the index j runs over the values 1, 2, and 3 for the three carrier solutions; and the index k runs over the number of measurements. Further, the parameters and values are defined as follows:
yijk=kth measurement of response after treatment with drug i in carrier solution j
μ=mean response
αi˜N(0, σa2)=contribution of drug i to yijk
bj˜N(0, σb2)=contribution of carrier j to yijk
cij˜N(0, σc2)=contribution of (drug i, carrier j) interaction to yijk
εijk˜N(0,σ2)=contribution of error term to yijk
Moreover, the parameters αi, bj, cij and εijk are independent of each other.
The clinical trial example above, on the other hand, may use a the quadratic model shown in equation (2):
yij=αi+βiIij+γiIij2+εij (2)
In equation (2), the index i runs over the number of patients and the index j runs over 1 to 5 for the five visits. Further, the parameters and values are defined as follows:
yij=reduction in blood pressure on the jth visit for subject i
Iij=concentration of drug administered to subject I on the jth visit (Iij=0 for placebo)
αi=subject specific intercept
βi=subject specific slope
γi=subject specific quadratic coefficient
εij˜N(0,σ2)=error term which is independent for different i or j
In one example, the subject specific parameters above may be derived from the population level values as αi=α+αi; βi=β+bi; and γi=γ+ci in which α is population level intercept, β is population level slope, and γ is population level quadratic coefficient. Moreover,
where D is a symmetric positive semi-definite matrix, and
(αi, bi, ci)T is independent for various i and also independent of εij in equation (2).
Substituting the above values in equation (2) results in the model shown in equation (3):
yij=(α+αi)+(β+bi)Iij+(γ+ci)+εij (3)
In some studies, a researcher may propose different models. A first model is said to be nested in a second model if the first model can be derived from the second model by imposing constraints on the parameters of the second model. Is some embodiments, a researcher chooses between a base model and an alternative model. The base model can be more complex or less complex than the alternative model. In some embodiments, the base model is nested in the alternative model. In some other embodiments, the alternative model is nested in the base model. Further, in some embodiments, none of the base model and the alternative model is nested in the other model.
For instance, in the first example, a researcher may hypothesize that one or more of the parameters ai, bj, or cij, are not needed and thus can be set to zero. Similarly, in the second example, a researcher may define a reduced model for the clinical trial study by hypothesizing that the population average of γj is not needed in the model. The researcher may desire to compare these alternative reduced models with each other and with the full model of Equation (1) or Equation (2). In particular, the researcher may desire to compare the models for their accuracy in explaining the data.
In some embodiments, a method for modeling a set of observed data comprises selecting a reference model as a possible description of the set of observed data, wherein the reference model includes a reference set of parameters; selecting an alternative model as another possible description of the set of observed data, wherein the alternative model includes an alternative set of parameters; storing an index function for measuring fit of models to data; calculating an observed index difference based on a difference in the index function for fitting the reference model to the observed data and for fitting the alternative model to the data; generating a set of simulated data; selecting a first model and a second model, wherein the first model is one of the reference model and the alternative model and the second model is the other one of the reference model and the alternative model; performing, by one or more processors, an operation based on the set of simulated data, wherein the operation includes performing a simulated threshold-fitting for the first model by the one or more processors, wherein the simulated threshold-fitting for the first model includes varying the set of parameters for the first model for fitting the first model to the simulated data and deriving a threshold-fit simulated index for the first model as a value of the index function for the first model that satisfies a threshold criterion; deriving an initial simulated index for the second model as an initial value of the index function for fitting the second model to the simulated data; and deriving an initial boundary for simulated index difference, wherein deriving the initial boundary for simulated index difference includes calculating a difference between the threshold-fit simulated index for the first model and the initial simulated index for the second model; determining, based on a comparison of the initial boundary for simulated index difference and the observed index difference, whether to update a counter used in calculating a simulated p-value; and selecting, based on the simulated p-value, one of the reference model and the alternative model for modeling the set of observed data.
In some embodiments, the first simulated threshold-fitting includes a first maximization, and the first maximization includes deriving the threshold-fit simulated index for the first model as a maximum simulated index for the first model by varying the set of parameters for the first model and maximizing the index function for fitting the first model to the simulated data.
In some embodiments, the method further comprises storing observed fit parameters for the reference model based on maximizing the index function for fitting the reference model to the observed data, and storing observed fit parameters for the alternative model based on maximizing the index function for fitting the alternative model to the observed data.
In some embodiments, deriving the initial simulated index for the second model includes setting the set of parameters for the second model to the observed fit parameters for the second model and deriving the index function for fitting the second model to the simulate data.
In some embodiments, the operation further comprises deriving an updated boundary for simulated index difference, wherein deriving the updated boundary for simulated index difference includes deriving an updated simulated index for the second model as an updated value of the index function for fitting the second model to the simulated data and calculating a difference between the threshold-fit simulated index for the first model and the updated simulated index for the second model; and wherein the method further comprises determining, based on a comparison of the updated boundary for simulated index difference and the observed index difference, whether to update the counter.
In some embodiments, deriving an updated simulated index for the second model includes deriving a maximum simulated index for the second model by a second maximization, and the second maximization includes varying the set of parameters for the second model and maximizing the index function for fitting the second model to the simulated data. In some embodiments, performing the simulated threshold-fitting for the first model deriving an updated simulated index for the second model are performed in parallel.
In some embodiments, the first model is the reference model and the second model is the alternative model, and the initial boundary for simulated index difference is an initial lower boundary for a simulated index difference, and the determining whether to update the counter includes increasing the counter if the initial lower boundary for the simulated index difference is larger than or equal to the observed index difference.
In some embodiments, the operation further comprises determining that the initial lower boundary for the simulated index difference is smaller than the observed index difference; and deriving an updated lower boundary for the simulated index difference, wherein deriving the updated lower boundary for the simulated index difference includes deriving an updated simulated index for the alternative model as an updated value of the index function for fitting the alternative model to the simulated data and subtracting the threshold-fit simulated index for the reference model from the updated simulated index for the alternative model; and wherein the method further comprises increasing the counter if the updated lower boundary for the simulated index difference is larger than or equal to the observed index difference.
In some embodiments, deriving an updated simulated index for the alternative model includes deriving a maximum simulated index for the alternative model via maximizing the index function for fitting the alternative model to the simulated data, deriving an updated lower boundary for the simulated index difference includes deriving the simulated index difference by subtracting the threshold-fit simulated index for the reference model from the maximum simulated index for the alternative model, and the method further comprises increasing the counter if the simulated index difference is larger than or equal to the observed index difference; and determining to leave the counter unchanged if the simulated index difference is smaller than the observed index difference.
In some embodiments, the first model is the alternative model and the second model is the reference model, and the initial boundary for simulated index difference is an initial upper boundary for a simulated index difference, and the determining whether to update the counter includes determining to leave the counter unchanged if the initial upper boundary for the simulated index difference is smaller than the observed index difference.
In some embodiments, the operation further comprises determining that the initial upper boundary for the simulated index difference is not smaller than the observed index difference; and deriving an updated upper boundary for the simulated index difference, wherein deriving the updated upper boundary for the simulated index difference includes deriving an updated simulated index for the reference model as an updated value of the index function for fitting the reference model to the simulated data and subtracting the updated simulated index for the reference model from the threshold-fit simulated index for the alternative model; and wherein the method further comprises determining to leave the counter unchanged if the updated upper boundary for the simulated index difference is smaller than the observed index difference.
In some embodiments, deriving an updated simulated index for the reference model includes deriving a maximum simulated index for the reference model via maximizing the index function for fitting the reference model to the simulated data, deriving an updated upper boundary for the simulated index difference includes deriving the simulated index difference by subtracting the maximum simulated index for the reference model from the threshold-fit simulated index for the alternative model, and the method further comprises increasing the counter if the simulated index difference is larger than or equal to the observed index difference; and determining to leave the counter unchanged if the simulated index difference is smaller than the observed index difference.
In some embodiments, the method further comprises a plurality of simulation iterations, wherein each iteration of the plurality of simulation iterations includes the generating a set of simulated data for the iteration and further includes, based on the set of simulated data for the iteration, the performing an operation based on the set of simulated data and the determining whether to update the counter
In some embodiments, each iteration of the plurality of simulation iterations includes randomly deciding whether setting the first model to be the reference model or to be the alternative model. In some embodiments, each iteration of the plurality of simulation iterations includes deciding whether the first model is the reference model or the alternative model based on an outcome of one or more of the previous iterations of the plurality of simulation iterations. In some embodiments, each iteration further includes setting a weight value to 0.5 if the iteration is a first iteration of the plurality of simulation iterations; otherwise calculating the weight value as a ratio of previous iterations in which the counter was decided to be increased; and creating a random variable for which a probability of being one is the weight value; and setting the first model to be the reference model if the random variable is one, otherwise setting the first model to be the alternative model. In some embodiments, two iterations of the plurality of simulation iterations are performed in parallel. In some embodiments, the index function is a logarithm of a density function.
In some embodiments, a comparison system for modeling a set of observed data comprises an analyzer module configured to select a reference model as a possible description of the set of observed data, wherein the reference model includes a reference set of parameters, select an alternative model as another possible description of the set of observed data, wherein the alternative model includes an alternative set of parameters; store an index function for measuring fit of models to data; calculate an observed index difference based on a difference in the index function for fitting the reference model to the observed data and for fitting the alternative model to the data; and select a first model and a second model, wherein the first model is one of the reference model and the alternative model and the second model is the other one of the reference model and the alternative model; a simulator module configured to generate a set of simulated data; a first maximizer module configured to perform an operation based on the set of simulated data, wherein the operation includes performing a simulated threshold-fitting for the first model by the one or more processors, wherein the simulated threshold-fitting for the first model includes varying the set of parameters for the first model for fitting the first model to the simulated data and deriving a threshold-fit simulated index for the first model as a value of the index function for the first model that satisfies a threshold criterion; and a second maximizer module configured to derive an initial simulated index for the second model as an initial value of the index function for fitting the second model to the simulated data, wherein the analyzer module is further configured to derive an initial boundary for simulated index difference, wherein deriving the initial boundary for simulated index difference includes calculating a difference between the threshold-fit simulated index for the first model and the initial simulated index for the second model; determine, based on a comparison of the initial boundary for simulated index difference and the observed index difference, whether to update a counter used in calculating a simulated p-value; and select, based on the simulated p-value, one of the reference model and the alternative model for modeling the set of observed data.
In some embodiments, a non-transitory computer-readable medium stores a computer program, wherein the computer program, when executed by one or more computer processors, causes the one or more processors to execute a method for modeling a set of observed data.
The drawings are not necessarily to scale or exhaustive. Instead, emphasis is generally placed upon illustrating the principles of the inventions described herein. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. In the drawings:
The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or similar parts. Also, similarly-named elements may perform similar functions and may be similarly designed, unless specified otherwise. Numerous details are set forth to provide an understanding of the described embodiments. The embodiments may be practiced without these details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the described embodiments. While several exemplary embodiments and features are described here, modifications, adaptations, and other implementations may be possible, without departing from the spirit and scope of the invention. Accordingly, unless stated otherwise, the descriptions relate to one or more embodiments and should not be construed to limit the invention as a whole. Instead, the proper scope of the invention is defined by the appended claims.
As used in this disclosure, a set can include one or more elements. Further, as used in this disclosure, a first variable is an increasing function of a second variable if the first variable does not decrease and instead generally increases when the second variable increases.
A comparison system compares at least two candidate models at a time. In some embodiments, a researcher needs to test a hypothesis by comparing a reference model with an alternative model. In some embodiments, the reference and the alternative models are two reduced versions of a full model with different sets of parameters. In some embodiments, one of the reference and the alternative models is the full model and the other one is a reduced model derived from the full model. In some embodiments, the reference and alternative models are not reduced versions of each other or of a full model and are instead different models used to describe the same data.
In some embodiments, a null hypothesis states that the reference model adequately describes the observed data. An alternative hypothesis, on the other hand, states that the reference model does not describe the data as well as the alternative model.
In some embodiments, when applying a model to a set of data, the comparison system determines values for the parameters for which the model best fits the data. To that end, the system utilizes an index function, which measures the fit of the model to the data for a specific choice of the parameters. In some embodiments, the index function is a log of the likelihood function. In some embodiments, the likelihood function is a density function, that is, the density of data under a corresponding model. In some embodiment, the index function includes one or more, or a linear combination of, an AIC (Akaike Information Criteria) function, a BIC (Bavesian Information Criteria) function, sum of square errors, an sum of absolute deviations.
In various embodiments, a likelihood function measures the probability or the probability density for a given set of data as a function of one or more unknown parameters. These parameters may be estimated by finding the values that maximize the likelihood.
In various embodiments, the observed data are presented in various formats, such as multidimensional matrices, text files, spread sheets, lists, or tables. In some embodiments the observed data are formulated as a response vector y. In the first example above, for instance, the response vector includes elements yijk corresponding to the data for the cell indexed as i,j,k. In the second example above, the response vector includes elements yijk corresponding to the reduction in blood pressure for patient i during visit j.
In some embodiments, the two models are represented as models M1 and M2, which respectively include parameter sets represented as vectors φ1 and φ2. Moreover, the index function for the two models are represented by P(y|φ1, M1) and P(y|φ2, M2). Thus, P(y|φ1, M1) is the value of the index function for fitting model M1 to the response vector y with specific values selected for parameters φ1. Therefore, the null hypothesis, H0, states that the response y is a draw from the distribution P (y|φ1, M1) for some unknown φ1. The alternate hypothesis H1, on the other hand, states that the response y is a draw from the distribution P (y|φ2, M2) for some unknown φ2.
In some embodiments, the system computes the index value for different sets of values for the parameters and selects a set of values that provides the optimized index. In some embodiments, the optimized index is the largest index value. In some embodiments, an optimized index is the smallest index value. In such embodiments, a new index can be defined as the inverse or the negative of the existing index, such that that the optimized value for the new index is its largest value. For convenience, and without loss of generality, the optimized index will be called the maximized index.
Such set of values for the parameters in each model are called the fit parameters. Therefore, in the above formulation and in some embodiments, the fit parameters {circumflex over (φ)}1 and {circumflex over (φ)}2 satisfy maximization Equations (3a) and (3b)
{circumflex over (φ)}1=arg maxφ
{circumflex over (φ)}2=arg maxφ
Equation (3a) indicates that fit parameters {circumflex over (φ)}1 are the values of parameters φ1 that maximize index function P for fitting the first model M1 to a set of data y. Similarly, Equation (3b) indicates that fit parameters {circumflex over (φ)}2 are the values of parameters φ2 that maximize index function P for fitting the second model M2 to the set of data y. When data y are the observed data yo, the fit parameters are called observed fit parameters {circumflex over (φ)}1o and {circumflex over (φ)}2o:
{circumflex over (φ)}1o=arg maxφ
{circumflex over (φ)}2o=arg maxφ
Graph 110 shows that the set of observed fit parameter for the reference model is a value of μ=9.5. That is, for μ=9.5 the graph of the index function for fitting the reference model to the observed data reaches a maximum value of around −488.30. Graph 120, on the other hand, shows that the set of observed fit parameter for the alternative model is a value of B=1.12. That is, for B=1.12 the graph of the index function for fitting the alternative model to the observed data reaches a maximum value of around −487.12.
In some embodiments, the comparison system compares two models using the index function and the fit parameters. The system compares the two models by finding an observed index difference, that is, a difference in the index values for fitting each model to the observed data. To find the observed index difference, the system sets the parameters of each model to the observed fit parameters for the model.
In
In some embodiments, the observed index difference is called an observed test statistic TO. Observed statistic TO is used to test various hypotheses. In some embodiments the formulations of Equations 3(a)-(d), the observed index difference is calculated by Equation (4):
TO=2{log P(yo|{circumflex over (φ)}2o,M2)−log P(yo|{circumflex over (φ)}1o,M1)} (4)
In Equation (4), to find the observed index difference TO, the system subtracts the logarithm of the index function for the first model under the first set of observed fit parameters from the logarithm of the index function for the second model under the second set of observed fit parameters. TO is a quantification of how better the second model fits the observed data with respect to the first model. In various embodiments, instead of logarithms, the system uses another increasing function of the index function.
In some embodiments, if the alternative model has a smaller index value compared to the reference model, the alternative hypothesis will be rejected. If the alternative model has a larger index value, on the other hand, the null hypothesis is not necessarily rejected.
In general, a higher index value for a second model may indicate that the second model better describes the observed data. The higher index value for the second model, however, may not hold up under a different set of data. Instead, a higher index value for the second model may correspond to a statistical fluctuation of the index value for data that can be described as well or better by the first model.
Thus, in the above embodiments, a positive value for TO may or may not invalidate the null hypothesis. The system may determine the significance of the higher index value compared to statistically random fluctuations of the index function under the reference model. In some embodiments, the comparison system determines the significance of a positive TO by comparing it against a statistical distribution of the index difference under the null hypothesis. To that end, in some embodiments, the comparison system performs multiple simulations to generate random data based on the null hypothesis. For each set of randomly generated simulation data, the system finds the fit parameters and the maximum simulated index for each model under the fit parameters. For each simulation, the system then derives the difference in these index maxima, as a simulated index difference. The system then compares the derived simulated index differences with the observed index difference.
In the formulations of Equations (3) and (4), for example, in some embodiments the simulated index difference for one set of simulated data is calculated as TS from Equation (5):
TS=2{log P(ysim|{circumflex over (φ)}2s,M2)−log P(ysim|{circumflex over (φ)}1s,M1)} (5)
In Equation (5), ysim is the set of simulation data generated in one simulation. In one embodiment, for example, ysim is drawn from the density P(y|{circumflex over (φ)}1, M1). Further, in equation (5), {circumflex over (φ)}1s and {circumflex over (φ)}2s are fit parameters found by maximizing the index function for the first and second models for fitting the simulation data, and thus P(ysim|{circumflex over (φ)}2s, M2) and
P(ysim|{circumflex over (φ)}2s, M2) are simulated index maxima for the two models. The comparison model derives TS for each set of simulation data and compares the TS value with the TO. In particular, the comparison system calculates a p-value for testing the null hypothesis compared to the alternative hypothesis by finding the probability that TS is larger than or equal to TO, as shown in Equation (6):
P(TO)=Prob.(TS≧TO) (6)
In some embodiments, the comparison system approximates the p-value for TO from the results of a number of simulations using Equation (7):
In Equation (7), “i” is an index for the simulation running from 1 to the total number of simulations, nsim, and Tsi is the value of TS derived from the ith simulation. I (Tsi≧To) is a step function that has a value of one whenever a simulated index difference Tsi for a simulation is equal to or greater than the observed index difference To, and otherwise has a value of zero. Equation (7), thus, essentially finds the proportion of simulations in which Tsi≧To to the total number of simulations.
In some embodiments, the comparison system either rejects or fails to reject the null hypothesis based on the p-value. For example, in some embodiments, a significance level a is selected. The significance level indicates the allowed level of deviation from zero for the observed index difference under the null hypothesis. In various embodiments, a has values such as 0.10, 0.05, or 0.01. In some embodiments, the comparison system rejects the null hypothesis if the p-value is less than or equal to the significance level.
Graphs 210(a) and 220(a), for example, show the results of fitting the reference and alternative models to the first set of simulated data. Based on this fitting, the system determines simulated fit parameters and derives the corresponding simulated index difference. Graph 210(a) shows the values of the index function for fitting the reference model to the first set of simulated data for different values of the parameter μ of the reference model. Similarly, graph 220(a) shows the values of the index function for fitting the alternative model to the first set of simulated data for different values of the parameter B of the alternative model. Graph 210(a) shows that the index function for fitting the reference model to the first set of simulated data reaches a maximum value of −480.0 for the simulated fit parameter value of μ=9.1. Graph 220(a), on the other hand, shows that the index function for fitting the alternative model to the first set of simulated data reaches a maximum value of −479.0 for B=1.1. Thus, based on the first set of simulated data, the maximum simulated indexes for the reference and alternative models are respectively −480.0 and −479.0. The difference between these two simulated index maxima is the simulated index difference, shown by bar 202-a, and has a value of 1.0.
In some embodiments, the comparison system performs a set of iterations to find the maximum of the index function for each model.
In
In some embodiments, the iterations for each model are based on changing the value of the set of parameters for that model. In some embodiments, the iterations for the two models are performed independent of each other. In
In some embodiments, the comparison system uses the maxima of graphs 260 and 270 to derive the simulated index difference. In
Each simulated index difference can be compared with the observed index difference. Each graph in
Accordingly, graph 210(a) shows that for the first simulation, simulated index difference 202-a is smaller than observed index difference 102. Similarly,
By repeating the simulations, the system finds a distribution of simulated index differences. In
In some embodiments, the above process is computationally intensive. For example, as shown in
I/O module 310 receives information or commands from, and transmits outputs or results to, one or more external systems or users. In some embodiments, the received information includes information about various models, observed data, or the index function. In some embodiments, the outputs include results of comparing an observed index difference with a distribution of simulated index differences. In some embodiments, the outputs include a p-value for a null hypothesis against an alternative hypothesis derived based on observed and simulated data. In some embodiments, the outputs include a confidence interval on the true p-value.
Simulator module 320 derives simulated data. In some embodiments, simulator module 320 transmits the simulated data to first or second maximizers 340-1 or 340-2, or to analyzer module 330. In some embodiments, the simulations are based on the input data.
In various embodiments, first maximizer 340-1, second maximizer 340-2, or both perform the process of maximization of the index function for fitting a model to a set of observed data or a set of simulated data. In various embodiments, the first and second maximizers perform their respective maximizations in parallel or in series. In some embodiments, the maximizer receives the simulated data from simulator module 320 or receives the index function from analyzer module 330. In some embodiments, to perform a maximization, first maximizer 340-1 or second maximizer 340-2 calculates the value of the index function for fitting a model to the data for several different values of the set of parameters for the model, as depicted in
Analyzer module 330 analyzes the simulated data and maximization results and accordingly compares the two models. In some embodiments, analyzer module 330 compares two models by calculating a p-value for the observed index difference.
In some embodiments, the comparison system reduces the computational load for finding the p-value by as much as fifty percent. In some embodiments, in each simulation the comparison system eliminates the need for finding the maximum of the index function for at least one of the two models.
In block 402, the system performs initialization before starting simulations, according to some embodiments. In particular, the system finds the observed fit parameters for the reference model and for the alternative model. In various embodiments, the system finds the observed fit parameters by maximizing the index function for fitting the corresponding model to the observed data. The system uses the observed fit parameters to calculate the observed index difference (TO). The system initializes an incident counter. In some embodiments, the incident counter counts the number of times that a simulated index difference is equal to or greater than the observed index difference. In some embodiments, in each simulation the incident counter is the cumulative values of I(Tsi≧To) in Equation (7) for all simulation up to the present. That is, in a simulation j, the incident counter is the value of Σi=1j(Tsi≧To) as used in Equation (7). In some embodiments, the system initializes the incident counter by setting it to zero before starting the simulations.
After initialization in block 402, the system performs one or more simulation iterations. A generalized simulation loop, including one or more simulation iterations, is described by the process starting at block 404 and ending at block 424. A simulation iteration in the loop includes steps described in block 406, 408, 409, 410, 412, 414, 416, 418, 420, and 422. More specific embodiments of the simulation loop are described in
In block 404, the system generates a new set of simulated data for a s simulation iteration. In some embodiments, simulator module 320 generates the new set of simulated data under the null hypothesis with known parameters. In some embodiments, the simulator module generates the new set of simulated data as a random set of data based on the fitted reference model.
In block 406, the system selects one of the reference and alternative models as the first model to maximize. For ease of reference, in
In block 408, the system performs a first maximization. In some embodiments, the first maximizer module performs the first maximization. In some embodiments, the first maximization includes finding a maximum of the index function for fitting the first model to the simulated data. In some embodiments, the first maximization includes varying the set of parameters for the first model and finding a set of fit parameters for the first model. In some embodiments, the first maximization includes calculating the maximum simulated index for the first model in the manner described in
In some embodiments, the first maximizer module performs the first maximization by finding values of the parameters for which the index function reaches a threshold value or the number of iterations reaches a maximum iteration number. In some embodiments, the threshold value is a value of the index function that is equal to or close to a known maximum of the index function. In some embodiments, the first maximizer module ends the maximization iterations when the value of the index function reaches or exceeds the threshold value. In some embodiments, such an operation is called threshold-fitting and the resulting index value is called a threshold-fit index value. The maximizer module may use the threshold-fit index value as the maximum simulated index for the first model.
In block 409, the system initializes the second maximization. In some embodiments, block 409 is performed by the analyzer module. In some embodiments, to initialize the second maximization, the system calculates an initial simulated index for the second model. In some embodiments, the initial simulated index for the second model is the value of the index for the first iteration of the maximization iteration, as shown in
In block 410, the system derives an initial boundary value for the simulated index difference (Tb). The system derives the initial boundary value for the simulated index difference by finding a difference between the maximum simulated index for the first model and the initial simulated index for the second model.
In some embodiments, such as those detailed in
The system performs a second maximization iteration loop for the second model, which includes blocks starting at 412 and ending at 422, according to some embodiments. As explained above, in some embodiments the first iteration of the second maximization is performed in block 409 by initializing the set of parameters for the second model to the observed fit parameters. Further, as explained below in more detail, if necessary the system performs further iterations of the second maximization. The system may terminate the iterations, upon performing the first iteration or a subsequent iteration, if a conclusive decision can be made based on the results of the latest iteration. Otherwise, the system terminates the iterations upon reaching the last iteration.
In block 412, the system compares the observed index difference (TO) with the initial boundary value for the simulated index difference (Tb). In some embodiments, block 412 is performed by the analyzer module. In some embodiments, block 412 includes determining whether TO is greater than, equal to, or smaller than Tb.
In decision block 414, the system determines whether the comparison results in a conclusive decision based on the simulated data. In some embodiments, decision block 414 is performed by the analyzer module. In some embodiments, the system determines whether any further iterations of the second maximization is needed. In some embodiments, a conclusive decision includes determining that the relationship between the observed index difference and simulated index difference can be found without performing any further iterations of the second maximization. For example, in some embodiments such as those related to
If a conclusive decision can be made at this stage (decision block 414: Yes), in block 416 the system updates the incident counter based on that decision. As detailed below in relation to
If a conclusive decision cannot be made at this stage (decision block 414: No), in decision block 418 the system determines whether to end the second maximization iteration. In some embodiments, the second maximization iteration ends upon reaching its last iteration, such as last iterations 264 or 274 in
If the second maximization has not reached its last iteration (decision block 418: No), the system performs the next iteration of the second maximization in block 420. In some embodiments, the next iteration includes updating the values of the set of parameters for the second model and deriving a new value of index function for fitting the second model to the simulated data. The new value is stored as an updated simulated index for the second model. In some embodiments, such as those shown in
In block 422, the system updates the boundary value for the simulated index difference (Tb). In some embodiments, block 422 is performed by the analyzer module. In some embodiments, the system updates Tb by finding a difference between the maximum simulated index for the first model and the updated simulated index for the second model, as derived in block 420. The system then loops back to block 412, by comparing the updated Tb with the observed index difference TO.
If the second maximization reaches its last iteration (decision block 418: Yes), the system moves to block 416 and updates the incident counter based on the completion of the second maximization iteration. As detailed below in relation to
In some embodiments, the system performs the first maximization (block 408) and the second maximization (starting at block 409 and ending at block 422) in parallel. In some embodiments, the system drives Tb once one of the two maximizations are ended and based on the latest value of the index function in the second maximization. The system then decides whether to end the second maximization or continue it based on whether the system can make a conclusive decision based on the value of Tb.
In some embodiments, the system compares more than two models by using parallel computing. In doing so, each parallel thread performs the maximization of the index function for one of the models and the analyzer module compares the results of various maximizations at different stages of one or more maximizations in the manner described herein.
Upon exiting the maximization iteration for each simulation, in decision block 424 the system decides whether the simulation loop should be ended. In some embodiments, the system ends the simulation loop if a pre-set number of simulations is performed. In some embodiments, the system ends the simulation loop if the results of previous simulations indicate that a decision can be made about the hypotheses. In some embodiments, the system ends the simulation loop if the results of previous simulations indicates that the p-value has already exceeded the significance level a, or that the p-value can never reach the significance level a even if the remaining simulations all increase the incident counter. In some embodiments, for example, the simulations are stopped if a ratio of the incident counter to the number of already performed simulations reaches a threshold value. In some embodiments, the simulations are stopped if a ratio of the incident counter to the total number of simulations is below a lower threshold or exceeds an upper threshold.
If the system does not end the simulation loop (decision block 424: No), the system loops back to block 404 and starts a new simulation by generating a new set of simulated data. On the other hand, if the system ends the simulation loop (decision block 424: Yes), in block 426 the system derives the p-value. In some embodiments, the system derives the p-value based on the value of the incident counter. In some embodiments, the system derives the p-value by using Equation (7).
In block 436, the system selects the reference model as the first model and the alternative model as the second model to maximize. In other embodiments, as detailed below, the system makes such a selection by comparing a randomly generated number with a pre-set number or with a number derived from the results of previous iterations.
In block 438, the system performs the first maximization by deriving the maximum simulated index for the reference model. In various embodiments, such a first maximization is performed in the manner described in relation to graphs 210(a)-210(d) of
Referring to
Returning to
In
Returning to
For each simulation iteration, the initial lower boundary for the simulated index difference of this simulation iteration (TSLB) has a value that is generally lower than or at most equal to the simulated index difference (i.e., Tsi in Equation (7)). As seen in
Returning to
Such an early stop of the second maximization is shown in
If, on the other hand, TS(i)LB is not larger than or equal to TO (decision block 444: No), the system cannot conclusively decide the relationship between the values of Tsi and TO. The system thus proceeds to perform further maximization iterations for the alternative model.
In decision block 448 the system determines whether to end the maximization iteration for the alternative model. In some embodiments, this iteration ends upon reaching its last iteration, such as the last iteration 274 in
If the system decides not to end the maximization iteration for the alternative model (decision block 448: No), in block 450 the system performs the next iteration of the maximization for the alternative model. In some embodiments, the next iteration includes updating the values of the set of parameters for the alternative model and deriving a new value of the index function for fitting the alternative model to the simulated data. The system stores the new value as an updated simulated index for the alternative model. In some embodiments, such as the one shown in
Further maximization iterations for the alternative model result in values that are shown in graph 570. In some embodiments, each new iteration results in an updated value of the index function for the alternative model, which is larger than the index value in the previous iteration. As a result, as shown in graph 570 of
Returning to
In some embodiments, such as that shown in
Upon determining that the updated TSLB is larger than or equal to TO (decision block 444: Yes), in some embodiments the system ends the maximization iterations for the alternative model. Such a situation is shown in
The embodiments shown in
In some embodiments, TSLB remains below TO even when the system reaches the last maximization iteration for the alternative model (decision block 448: Yes). In such a situation, the system exits the maximization iteration without incrementing the incident counter. In some embodiments, such a situation occurs when the index function for the alternative model, such as graph 570 in
In block 466, the system selects the alternative model as the first model and the reference model as the second model. In other embodiments, as detailed below, the system makes such a selection by comparing a randomly generated number with a pre-set number or with a number derived from the results of previous iterations.
In block 468, the system performs the first maximization by deriving the maximum simulated index for the alternative model. In various embodiments, such a first maximization is performed in the manner described in relation to graphs 220(a)-220(d) of
Returning to
In
Returning to
The initial upper boundary for the simulated index difference (TSUB) has a value that is generally higher than or at least equal to the simulated index difference for this simulation iteration (i.e., Tsi in Equation (7)). As seen in
Returning to
Such an early stop of the second maximization is shown in
If, on the other hand, TSUB is not smaller than TO (decision block 474: No), the system cannot conclusively decide the relationship between the values of Tsi and TO. The system thus proceeds to perform further maximization iterations for the reference model.
In decision block 476 the system determines whether to end the maximization iteration for the reference model. In some embodiments, this iteration ends upon reaching its last iteration, such as the last iteration 264 in
If the system decides not to end the maximization iteration for the reference model (decision block 476: No), in block 480 the system performs the next iteration of the maximization for the reference model. In some embodiments, the next iteration includes updating the values of the set of parameters for the reference model and deriving a new value of the index function for fitting the reference model to the simulated data. The system stores the new value as an updated simulated index for the reference model. In some embodiments, such as the one shown in
Further maximization iterations of the reference model result in values that are shown in graph 660. In some embodiments, each new iteration results in an updated value of the index function for the reference model, which is larger than the index value in the previous iteration. As a result, as shown in graph 660 of
Returning to
In some embodiments, such as that shown in
Upon determining that the updated TSUB is smaller than TO (decision block 474: Yes), the system ends the maximization iterations for the reference model. Such a situation is shown in
Similar to the embodiments depicted in
In some embodiments, the TSUB remains larger than or equal to TO even when the system reaches the last maximization iteration for the reference model (decision block 476: Yes). In such a situation, the system exits the maximization iteration and increments the incident counter in block 478. In some embodiments, such a situation occurs when the index function for the reference models, such as graph 660 in
In various embodiments, at the start of each simulation iteration the system selects one of the reference and alternative models as the first model to maximize and the other one as the second model. In some embodiments, the system makes this selection based on the results of previous simulation iterations. In particular, in some embodiments, the system uses a biased random selection based on those results. In some embodiments, the system first determines a frequency value, which is the ratio of times in the previous iterations in which the simulated index difference Tsi was found to be larger than or equal to the observed index difference TO. Such iterations indicate the cases in which comparing the observed index difference with the lower boundary for the simulated index difference requires less or at most the same number of maximization iterations as compared to comparing it with the upper boundary for the simulate index difference. The system thus determines the probability of selecting the reference model as the first model based the frequency value, with a higher probability for a larger frequency value.
In some embodiments, the system generates a random number with a Bernoulli distribution for which the success probability is the frequency value. Then, the system selects the reference model as the first model if the random number is one, and otherwise selects the alternative model. According to some embodiments, if the system selects the reference model as the first model, the system performs the process of
The following is an exemplary high level software code that may be executed by a computer to implement the comparison according to some embodiments.
1:
Derive {circumflex over (φ)}1o, {circumflex over (φ)}2o and TO
2:
For i = 1 then to nsim do
3:
Simulate data ysim under H0: ysim
is drawn from the density P(y |{circumflex over (φ)}1o, M1)
4:
if i = 1 then
5:
f = 0.5
6:
else
7:
8:
end if
9:
Generate u ~Bernoulli (f)
10:
if u = 1 then
11:
TS(i)LB = 2{logP(ysim|{circumflex over (φ)}2o, M2) − maxφ
12:
if TS(i)LB ≧ TO then
13:
Record that TS(i) ≧ TO for iteration i by setting I (TS(i) ≧ TO) = 1
14:
else
15:
TS(i) = TS(i)LB + 2{maxφ
logP(ysim|{circumflex over (φ)}2o, M2)}
16:
If TS(i) ≧ TO then set I(TS(i) ≧ TO) = 1
otherwise set I(TS(i) ≧ TO) = 0
17:
end if
18:
else if u = 0 then
19:
TS(i)UB = 2{maxφ
20:
if TS(i)UB < TO then
21:
Record that TS(i) < TO for iteration i by setting I (TS(i) ≧ TO) = 0
22:
else
23:
TS(i) = TS(i)UB + 2{ysim|{circumflex over (φ)}1o, M1) − maxφ
24:
If TS(i) ≧ TO then set I(TS(i) ≧ TO) = 1 otherwise I(ts(I) ≧ to) = 0
25:
end if
26:
end if
27:
end for
28:
At line 1 of the code, the system derives the observed fits parameters {circumflex over (φ)}1o and {circumflex over (φ)}2o and the observed index difference TO. In some embodiments, the system derives these observed values using equations such as Equations (3c), (3d), and (4). A line 2, the system starts the simulation iterations. The simulation iteration loop starts at line 2 and ends at line 27.
At line 3, the system generates a set of simulated data under the null hypothesis (H0). At lines 4-8, the system calculates the frequency value, f. For the first iteration, the frequency value is 0.5, and otherwise it is the ratio of prior simulations in which the simulated index difference was larger than or equal to the observed index difference. At line 9, the system generates a random number based on a Bernoulli distribution with a probability of f. If the random number is 1, the system performs a first block of code at lines 11-17, in which the system finds the lower boundary for the simulated index difference. If, on the other hand, the random number is 0, the system performs a second block of code at lines 19-25, in which the system finds the upper boundary for the simulated index difference.
In the first block of code, at line 11, the system finds the initial value of TS(i)LB, the lower boundary for the simulated index difference for this simulation iteration indexed i. The system finds this initial value of TS(i)LB by finding the maximum simulated index for the reference model (i.e., finding maxφ
At line 12, the system compares the initial value of the lower boundary with the observed index difference TO. If the lower boundary is larger than or equal to the observed index difference, at line 13 the system sets the incident recorder I (TS(i)≧TO) to one for this simulation set.
If, on the other hand, the lower boundary is smaller than the observed index difference, the system derives the simulated index difference TS(i) at line 15. In particular, at line 15 the system finds the maximum simulated index for the alternative model (maxφ
At line 16, the system checks whether the simulated index difference is larger than or equal to the observed index difference (TS(i)≧TO). If the answer is yes, the system records the incident by setting the incident recorder I (Ts(i)≧TO) to one, and otherwise by setting it to zero.
In the second block of code, at line 19, the system finds the initial value of TS(i)UB, the upper boundary for the simulated index difference for this simulation. The system finds this initial value of TS(i)uB by finding the maximum simulated index for the alternative model (i.e., finding maxφ
At line 20, the system compares the initial value of the upper boundary with the observed index difference TO. If the upper boundary is smaller than the observed index difference, at line 21 the system sets the incident recorder I (TS(i)≧TO) to zero for this simulation set.
If, on the other hand, the upper boundary is not smaller than the observed index difference, the system derives the simulated index difference TS(i) at line 23. In particular, at line 23 the system finds the maximum simulated index for the reference model (maxφ
At line 24, the system checks whether the simulated index difference is larger than or equal to the observed index difference (TS(i)≧T). If the answer is yes, the system records the incident by setting I (TS(i)≧T) to one, and otherwise by setting it to zero.
At line 28, the system derives the p-value for the observed index difference P(TO) using Equation (7).
The disclosed systems or modules may be implemented via one or more computing devices or systems.
In various embodiments, processor 702 may be a microprocessor or a central processor unit (CPU) performing various methods in accordance to the embodiment. Memory 714 may include a computer hard disk, a random access memory (RAM), a removable storage, or a remote computer storage. In various embodiments, memory 714 stores various software programs executed by processor 702. Network adapter 712 enables system 700 to exchange information with external networks. In various embodiments, network adapter 712 includes a wireless wide area network (WWAN) adapter, or a local area network (LAN) adapter. I/O interfaces 716 may include keyboard 710, or a mouse, an audio input device, a touch screen, or an infrared input interface.
In various embodiments, one or more of modules disclosed in this disclosure are implemented via one or more processors executing software programs for performing the functionality of the corresponding modules. In some embodiments, one or more of the disclosed modules are implemented via one or more hardware modules executing firmware for performing the functionality of the corresponding modules. In various embodiments, one or more of the disclosed modules include storage media for storing data used by the module, or software or firmware programs executed by the module. In various embodiments, one or more of the disclosed modules or disclosed storage media are internal or external to the disclosed systems. In some embodiments, one or more of the disclosed modules or storage media are implemented via a computing “cloud”, to which the disclosed system connects via an internet and accordingly uses the external module or storage medium. In some embodiments, the disclosed storage media for storing information include non-transitory computer-readable media, such as a CD-ROM, a computer storage, e.g., a hard disk, or a flash memory. Further, in various embodiments, one or more of the storage media are non-transitory computer-readable media storing information or software programs executed by various modules or implementing various methods or flow charts disclosed herein.
In various embodiments, an I/O module is configured to receive inputs and to provide outputs. In some embodiments, the I/O module is an interface for receiving and sending data to a user or to another system. In various embodiments, the I/O module includes an input interface and an output interface. The input interface is configured to receive data, such as commands, and the output interface is configured to output information such as p-value or comparison results. In some embodiments, the I/O module includes one or more of an internet interface, a wireless interface, a data reader, a mouse, a keyboard, a display, a speaker, a touch screen, or a printer.
The foregoing description of the invention, along with its associated embodiments, has been presented for purposes of illustration only. It is not exhaustive and does not limit the invention to the precise form disclosed. Those skilled in the art will appreciate from the foregoing description that modifications and variations are possible in light of the above teachings or may be acquired from practicing the invention. For example, the steps described need not be performed in the same sequence discussed or with the same degree of separation. Likewise various steps may be omitted, repeated, or combined, as necessary, to achieve the same or similar objectives. Similarly, the systems described need not necessarily include all parts described in the embodiments, and may also include other parts not described in the embodiments. Accordingly, the invention is not limited to the above-described embodiments, but instead is defined by the appended claims in light of their full scope of equivalents.
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
7085690, | Jun 10 2000 | Unsupervised machine learning-based mathematical model selection | |
7474997, | Apr 16 2003 | Sony Corporation | Construction and selection of a finite mixture model for use in clustering and vector quantization |
8478534, | Jun 11 2003 | RESEARCH FOUNDATION OF THE STATE UNIVERSITY OF NEW YORK, THE | Method for detecting discriminatory data patterns in multiple sets of data and diagnosing disease |
20040236776, | |||
20120323834, | |||
20140258355, | |||
20150006605, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Mar 14 2013 | The MathWorks, Inc. | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Date | Maintenance Schedule |
Mar 08 2019 | 4 years fee payment window open |
Sep 08 2019 | 6 months grace period start (w surcharge) |
Mar 08 2020 | patent expiry (for year 4) |
Mar 08 2022 | 2 years to revive unintentionally abandoned end. (for year 4) |
Mar 08 2023 | 8 years fee payment window open |
Sep 08 2023 | 6 months grace period start (w surcharge) |
Mar 08 2024 | patent expiry (for year 8) |
Mar 08 2026 | 2 years to revive unintentionally abandoned end. (for year 8) |
Mar 08 2027 | 12 years fee payment window open |
Sep 08 2027 | 6 months grace period start (w surcharge) |
Mar 08 2028 | patent expiry (for year 12) |
Mar 08 2030 | 2 years to revive unintentionally abandoned end. (for year 12) |