A computer-implemented method of analyzing a dataset of pharmacovigilance data, includes determining a sample size-independent measure of association between two conditions of interest in the dataset of pharmacovigilance data; using a hypergeometric distribution to determine a measure of statistical unexpectedness between the conditions of interest in said dataset, wherein the distribution is based on an urn model under a hypothesis that the conditions are statistically independent; and displaying the measure of association with the measure of the statistical unexpectedness to identify a significant association between conditions of interest.
|
31. A computer program product on a computer readable storage medium that, when executed on a computing system, analyzes a dataset of pharmacovigilance data, the program product comprising:
(a) code for determining a sample size-independent measure of association between two conditions of interest in the dataset of pharmacovigilance data;
(b) code for analyzing a hypergeometric distribution to determine a measure of statistical unexpectedness between the conditions of interest in said dataset, wherein said distribution is based on an urn model under a hypothesis that said conditions are statistically independent; and
(c) code for displaying the measure of association with the measure of the statistical unexpectedness to identify a significant association between the conditions of interest.
1. A computer-implemented method of analyzing a dataset of pharmacovigilance data, comprising:
(a) determining a sample size-independent measure of association between two conditions of interest in the dataset of pharmacovigilance data on a suitably programmed computing device;
(b) analyzing a hypergeometric distribution to determine a measure of statistical unexpectedness between the conditions of interest in said dataset on a suitably programmed computing device, wherein said distribution is based on an urn model under a hypothesis that said conditions are statistically independent; and
(c) displaying the measure of association with the measure of the statistical unexpectedness to identify, a significant association between the conditions of interest on a suitably programmed computing device.
30. A system that includes a central processing unit (CPU) for analyzing a dataset of pharmacovigilance data, comprising:
(a) an input unit for accessing the dataset of pharmacovigilance data;
(b) a processing unit configured to:
(i) determining a sample size-independent measure of association between two conditions of interest in the dataset of pharmacovigilance data; and
(ii) analyzing a hypergeometric distribution to determine a measure of statistical unexpectedness between the conditions of interest in said dataset, wherein said distribution is based on an urn model under a hypothesis that said conditions are statistically independent; and
(c) a display unit for displaying the measure of association with the measure of the statistical unexpectedness to identify a significant association between the conditions of interest.
2. The computer-implemented method according to
where a total number of records in the dataset is N, a total number of records satisfying a first of the two conditions of interest is NA, a total number of records satisfying a second of the two conditions of interest is NB, and a total number of records satisfying both the first and second conditions of interest is NAB.
3. The computer-implemented method according to
RRAB<1 and CAB=P{NAB≦n}<θ RRAB>1 and C′AB=P{NAB≧n}<θ where θ represents a small significance threshold probability for significance, CAB represents the cumulative probability (CAB=P{NAB≦n}), and C′AB represents the complimentary probability C′AB=P{NAB≧n},
wherein statistical unexpectedness (UAB) of an observed response NAB is the reciprocal of the corresponding cumulative or complementary probability and defined as following
UAB=1/CAB if RRAB<1 and UAB=1/C′AB if RRAB>1. 4. The computer-implemented method according to
5. The computer-implemented method according to
6. The computer-implemented method according to
7. The computer-implemented method according to
8. The computer-implemented method according to
9. The computer-implemented method according to
10. The computer-implemented method according to
11. The computer-implemented method according to
12. The computer-implemented method according to
13. The computer-implemented method according to
14. The computer-implemented method according to
15. The computer-implemented method according to
16. The computer-implemented method according to
17. The computer-implemented method according to
18. The computer-implemented method according to
19. The computer implemented method according to
20. The computer implemented method according to
21. The computer implemented method according to
22. The computer implemented method according to
(a) taking a number of samples from the dataset;
(b) recalculating the reporting ratio and the measure of unexpectedness for each of the number of samples; and
(c) displaying a confidence box around an adverse event based on corresponding points for that adverse event for each of the calculated and recalculated reporting ratios and the measure of unexpectedness for each of the number of samples.
23. The computer implemented method according to
24. The computer implemented method according to
25. The computer implemented method according to
26. The computer implemented method according to
27. The computer implemented method according to
(a) taking a number of samples from the dataset;
(b) recalculating and displaying as points the reporting ratio and the measure of unexpectedness for each of the number of samples; and
(c) displaying points corresponding to the same pair of conditions across the number of samples in an integrated display by a vector.
28. The computer implemented method according to
29. The computer implemented method according to
|
This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. provisional application No. 60/621,078 filed on Oct. 25, 2004, and U.S. provisional application No. 60/645,609 filed on Jan. 24, 2005, the disclosures of which are incorporated herein in their entireties.
The present invention relates generally to the field of analyzing data for biological or medical tracking or monitoring purposes. More particularly, the present invention relates to analyzing data related to the safety of drugs, vaccines, medical devices, surgical procedures, and other therapeutic interventions for the treatment of disease in humans, animals, or agricultural plants.
Developers of pharmaceutical products, vaccines, medical products and the like are greatly concerned with the potential risk of harm to patients incidental to the use of these therapeutic interventions to treat disease. Agencies which regulate the sale and use of these products generally mandate a certain level of pharmacovigilance activity to measure and report adverse effects associated with the use of these products, in a way that attempts to ascertain whether or not a causal relationship exists between the use of the product and the adverse effect.
The evaluation of drug safety generally begins with in vitro toxicology studies in cultured cells and animals, and then proceeds to randomized, controlled clinical trials. In the present application, references to “drugs” may be understood to apply to vaccines, medical products, and the like as well. In a randomized, controlled clinical trial, a group of patients is divided into two groups. One half of the patients receive the drug under study, and the remainder receives a different “control” treatment, sometimes a placebo. More complex designs are possible as well, and often the assignment is done in a blind fashion so that neither the patient nor the investigator knows which patient has received which treatment until the conclusion of the study. In addition to comparing the efficacy of the drug under study versus the control, the relative safety of the two treatments is compared. This is done by compiling a table of all adverse events which occur. Adverse events are listed by a standardized code, such as the Medical Dictionary for Regulatory Activities (MedDRA) [Brown, 1999]. It should be noted that the complete citation for any referenced publication in provided at the end of this application in a separate section. For each adverse event code, the table includes the number of patients in the drug arm of the study who encountered that adverse event, and the number in the control arm of the study who encountered it as well. The adverse event table generally also includes percentages of the total population of each arm who experienced each type of adverse event. From this table, it is possible to calculate a relative risk for each event type associated with the drug as compared to the control.
Clinical trials must be conducted before a drug is marketed. Regulatory agencies such as the US Food and Drug Administration (FDA) decide to approve drugs based on evaluations of both the safety and the efficacy of the drug, as measured in clinical trials. Once the drug is on the market, a certain amount of additional “post-market” safety evaluation is required. Depending on the degree of risk associated with the drug, this may range from simple surveillance to specific additional studies, as determined by the regulatory agency.
Post-market safety studies are required because it is not possible to fully ascertain the safety of a drug in a controlled clinical trial. Clinical trials generally exclude groups of patients with serious diseases other than the one under study, since such patients would greatly complicate the interpretation of the results. Vulnerable groups such as children and pregnant women may also be excluded. However, all of these kinds of patients may be exposed to the drug once clinical trials are complete and it has been approved and is on the market. In addition, there is the possibility of evolving interaction with new drugs, or food, and lifestyle factors, which may not even exist at the time that the clinical trial is being performed. Drug interactions in general, and interactions between drugs and co-morbid conditions (e.g. worsening of diabetes in a diabetic cancer patient due to an anti-cancer drug), are of great concern in the field of pharmacovigilance.
Systems are in place for post-market pharmacovigilance, that is, the detection of safety signals from marketed drugs. A “safety signal” refers to a concern about an excess of adverse events compared to what would be expected if the occurrence of such events were independent of a drug's use. Regulatory agencies such as the FDA have established programs by which health care providers and patients can report adverse events to drug manufacturers and to the FDA itself. In some cases, these reports may be quite detailed, and may even include experimental re-challenge of a patient with a drug to check for reoccurrence of a non-serious adverse event, such as a rash. In this type of situation, a causal relationship between the drug and the adverse event does not need to rely on statistical techniques. However, in many cases, less detail is available. Reports are tallied into a table of Individual Safety Reports (ISRs), which generally include an approximate date of the event, the drugs that the patient was taking (very often more than one), and codes for the adverse event or events which occurred (also often more than one).
These ISRs are most often compiled into a matrix which indicates the prevalence of many possible drug-event combinations. Note that a complicating factor in using such data for pharmacovigilance is a lack of information about the size of the exposed population. With this type of data, pharmacovigilance methods generally rely on the detection of disproportionality of one drug-event combination compared to the rate at which the event occurs with other drugs. Data analysis methods which have been used with such data include the Multi-Item Gamma Poisson Shrinker (MGPS) [Du Mouchel, 2001; Szarfman 2002], the Proportional Reporting Ratio (PRR) method [Evans, 2001], and the Bayesian Confidence Propagation Neural Network (BCPNN) [Bate, 1998].
Deriving and tracking trends in pharmacovigilance data is an ongoing problem in view of the vast volume of data that needs to be analyzed together with the long timeframes over which much of this data is collected.
In certain aspects, the present invention relates to the visualization and detection of medically-important associations between drugs and adverse events, between different drugs, or between different adverse events (e.g., identification of syndromes), with or without the incorporation of relevant secondary data (e.g., demographic or clinical data).
In certain embodiments, the present invention provides a computer-implemented method of visualizing and analyzing a dataset of pharmacovigilance data, including: (a)determining a sample size-independent measure of association between two conditions of interest in the dataset of pharmacovigilance data; (b) determining a measure of statistical unexpectedness relating to the data on the relationship between the conditions of interest in said dataset, wherein the measure is based on an urn model under a hypothesis that the conditions are statistically independent; (c) visually displaying the measure of association with the measure of the statistical unexpectedness to aid in the determination of a significant association between the conditions of interest; and (d) tabulating the measures of association and of statistical unexpectedness to facilitate the generation of lists of conditions requiring further study.
In certain embodiments, the dataset comprises binary data and the measure of association comprises a reporting ratio RRAB defined as
where a total number of records in the dataset is N, a total number of records satisfying a first of the two conditions of interest is NA, a total number of records satisfying a second of the two conditions of interest is NB, and a total number of records satisfying both the first and second conditions of interest is NAB.
In certain embodiments, the significant association between the two conditions of interest is determined if the following criteria are satisfied
RRAB<1 and CAB=P{NAB≦n}<θ
RRAB>1 and C′AB=P{NAB≧n}<θ
where θ represents a small significance threshold probability for significance, CAB represents the cumulative probability (CAB=P{NAB≦n}), and C′AB represents the complimentary probability C′AB=P{NAB≧n}, wherein statistical unexpectedness (UAB) of an observed response NAB is the reciprocal of the corresponding cumulative or complementary probability and defined as following
UAB=1/CAB if RRAB<1 and UAB=1/C′AB if RRAB>1
In certain embodiments, the first condition of interest is a fixed reference condition, and the second condition of interest is range of relevant comparison conditions.
In certain embodiments the first condition of interest is a presence of a drug and the second condition of interest is an adverse event.
In certain embodiments, the reporting ratio or its logarithm is designated on the X-axis of a graph, and the statistical unexpectedness or its logarithm is designated on the Y-axis of the graph, and a point is plotted and labeled on the graph for each of a number of second conditions of interest.
In certain embodiments, a horizontal dotted line is plotted on the graph, designating a statistical significance threshold. For example, if it is desired to make a clear visual distinction for all first condition points corresponding to a statistical significance level of p=0.05, then the line is drawn at a Y-value corresponding to 20=1/0.05 on a log scale.
In certain embodiments, the significance threshold is adjusted for multiple comparisons wherein the significance threshold is adjusted using the Bonferroni correction wherein the significance threshold is adjusted to θ/M where M represents the number of comparisons between the first and second conditions of interest. Other corrections for multiple comparisons, besides the Bonferroni correction, may be used.
In certain embodiments, the step of displaying the measure of association includes separately indicating negative associations with reporting ratios less than one from positive associations with reporting ratios greater than one.
In certain embodiments, the present invention includes a preprocessing step in which the dataset is partitioned into subsets and then processing each subset of the dataset.
In certain aspects, the present invention provides a system for analyzing a dataset of pharmacovigilance data, including: an input unit for accessing the dataset of pharmacovigilance data; a processing unit configured for determining a sample size-independent measure of association between two conditions of interest in the dataset of pharmacovigilance data, and using a hypergeometric distribution to determine a measure of statistical unexpectedness between the conditions of interest in said dataset, wherein said distribution is based on an urn model under a hypothesis that said conditions are statistically independent; and a display unit for displaying the measure of association with the measure of the statistical unexpectedness to identify a significant association between the conditions of interest.
In certain embodiments, the present invention provides a computer program product on a computer readable medium that, when executed on a computing system, analyzes a dataset of pharmacovigilance data, the program product including: code for determining a sample size-independent measure of association between two conditions of interest in the dataset of pharmacovigilance data; code for using a hypergeometric distribution to determine a measure of statistical unexpectedness between the conditions of interest in said dataset, wherein said distribution is based on an urn model under a hypothesis that said conditions are statistically independent; and code for displaying the measure of association with the measure of the statistical unexpectedness to identify a significant association between the conditions of interest.
In view of some of the problems discussed earlier herein, techniques that improve deriving and tracking trends in pharmacovigilance data are of utmost importance so that, for example, adverse events caused by particular drugs can be discovered as soon as possible. Likewise, discovering positive associations, such as the unexpected beneficial effects of a drug to reduce the certain adverse events are also very valuable because these drugs may then be used for therapeutic purposes, or to reduce the number of adverse events in the future.
The ultimate goal of pharmacovigilance is the assessment of causality in drug-adverse event relationships, and this is not an entirely objective statistical exercise. Medical judgment and human pattern recognition currently play a significant role. Consequently, a pharmacovigilance tool does not necessarily need to make fully-automated decisions about the safety of drugs, or even attempt to create a one-dimensional ordered list of drugs based on their level of risk. A useful tool is one which presents drug-safety data in a visual manner which facilitates or improves expert medical judgment and decision-making. The role of the computer in such a system is to summarize an extremely large amount of data into a presentation of information that can be readily understood, and which accurately represents the statistical features and trends of the data for both large and small numbers of adverse events. This is the design philosophy embodied in the several embodiments of the present invention.
In certain embodiments, the present invention presents a computer implemented computational procedure that generates, from a given dataset D, a set of numbers useful in characterizing pairs of Conditions of Interest, described further herein, together with a display (such as a two-dimensional graphical display) to aid in interpreting these results (or numbers). In the example of displaying the results as a two dimensional graphical display, the horizontal axis of this graph is defined by an association measure, such as one previously used in pharmacovigilance studies, that is not sensitive to sample size effects. The vertical axis is based on the probability of the observed response under a statistical independence model, a quantity that is strongly dependent on sample size effects. Regions of this plot or display can be identified that correspond to unexpectedly strong positive associations (which may be due to harmful drug interactions or other undesirable phenomena), unexpectedly strong negative associations (which may be due to beneficial drug effects or to prescribing patterns like nonprescription of certain drugs for pregnant women), and results that are probably not of interest (e.g., associations that are not strong and/or not statistically significant).
In certain embodiments, an assumption underlying the present invention is described using the following data model: The given dataset D is assumed to consist of N records, each consisting of the following fields:
In certain embodiments, the method described here characterizes the relationship between pairs of Conditions of Interest, defined as collections of records that satisfy one of the following conditions:
The graphical display generated by the procedure described herein is based on the results obtained for a single Reference Condition of Interest (Condition A) and a family of Comparison Conditions of Interest (i.e., a set of Conditions B). As an illustrative example, taking the Reference Condition as a specific drug of interest, a useful set of Comparison Conditions is a set of all adverse events appearing in one or more records listing the drug of interest. More generally, Comparison Conditions B can be any well-defined set of Conditions of Interest, as defined above, whose possible association with Condition A is of interest. In certain embodiments, it is assumed that all Comparison Conditions B have at least one record in common with Reference Condition A (i.e., that NAB≧1), but one advantage of the method discussed herein is that it extends naturally to the case of non-coincident conditions (i.e., cases where NAB=0).
The following paragraphs first describe the analysis performed for the Reference Condition with each individual member of the set of Comparison Conditions. Then a graphical display constructed from all results in the set of Comparison Conditions is described.
Given two conditions of interest, Condition A and Condition B, the following two binary sequences are defined for every record in dataset D:
ak=1 if Record Rk satisfies Condition A, and ak=0 otherwise,
bk=1 if Record Rk satisfies Condition B, and bk=0 otherwise.
Quantitative measures of association between Conditions A and B can be defined in many different ways. See, for example, Agresti [Agresti, 2002], the relevant disclosure of which is incorporated herein. Most of these measures of association depend on the following four integers:
A
Not A
Total
B
NAB
NB − NAB
NB
Not B
NA − NAB
N − NA − NB + NAB
N − NB
Total
NA
N − NA
N
These numbers are related to the binary sequences {ak} and {bk} via the following simple expressions:
In certain embodiments, the relationship between Conditions A and B are characterized with two numbers, both derived from the four numbers NA, NB, NAB, and N defined above: (1) a traditional measure of association, and (2) a measure of statistical unexpectedness based on the probability of observing these four numbers under a null hypothesis that Conditions A and B are statistically independent.
The following paragraphs describe the details of determining a measure of association between two conditions of interest in step 220. Of the many available association measures between two binary sequences, the following three are described in the pharmacovigilance literature:
Unfortunately, the terminology used in the literature is not entirely standard. For example Gould [Gould, 2003] defines the first of these three measures as the “proportional reporting ratio,” while Du Mouchel [Du Mouchel, 1999] defines this first measure as the “nonstratified relative report rate.” The definition of “proportional reporting ratio” used herein is defined in Heeley [Heeley, 2002]. Gould cites a reference arguing that the Report Odds Ratio defined above has better resistance to the effects of under-reporting than the Reporting Ratio, but both the Report Odds Ratio and the Proportional Reporting Ratio as defined here suffer from the fact that their value becomes infinite when NB=NAB, and the Report Odds Ratio is also infinite whenever NA=NAB. This observation is important because both of these situations can and do arise in the analysis of pharmacovigilance data. For these reasons, the use of Reporting Ratio as defined above is advantageous in certain embodiments of the present invention. Finally, it should be noted that all of these association measures are invariant under a common rescaling of all counts by any constant multiplier.
The following paragraph discusses some of the processing performed in determining a measure of statistical unexpectedness in step 230. In certain embodiments, the statistical unexpectedness measure included in the method described herein is based on the idea of comparing the observed results with a random reference model. That is, under the assumption that Conditions A and B are not associated, the two sequences {ak} and {bk} should be statistically independent. It is possible to compute the probability pAB of observing the number NAB of simultaneous occurrences of Conditions A and B under this independence assumption, given the observed values for NA, NB, and N. This point is important because, unlike other methods (e.g., the GPS method described by DuMouchel 1999), the method described herein does not impose any distributional assumptions on the data values. This independence from distributional assumptions is a significant practical advantage, both because errors in specification of the data distribution in methods requiring distributional assumptions can cause those methods to perform poorly, and because changes in the original application scenario for those methods can require modifications of these distributional assumptions.
To compute the probability pAB, the method proceeds as follows. First, it should be noted that since the summation index k in sequence {ak} is arbitrary, nothing is changed by re-ordering these sums, defining a new summation index l such that
al=1 for l=1,2, . . . , NA and al=0 for l=NA+1,NA+2, . . . , N
Under this new labeling, the value of NAB is given by
The advantage of this expression is that it converts the statistical independence assumption into the following urn model: Consider the dataset D as an urn containing N balls, of which NA are white and the remaining N-NA are black. Under the statistical independence hypothesis, the NB records satisfying Condition B may be regarded as a random sample of NB balls drawn from the urn, of which NAB are white. It is a standard result that the probability of observing NAB=n is given by the hypergeometric distribution:
for max{0, NA+NB−N}≦n≦min{NA, NB}
The urn model is described in detail by Johnson [Johnson, 1992].
It should be noted that in typical pharmacovigilance applications, NA+NB<<N, so the lower limit on n is zero. It should be noted that expected value of NAB under this independence model is:
as explained in [Johnson, 1992].
Note that if NAB is equal to this expected value, the Reporting Ratio is equal to 1:
The practical importance of this observation lies in the fact that detection of significant associations involves the identification of Conditions A and B that have low probability under the statistical independence model. The hypergeometric distribution is unimodal, so small probabilities occur in the tails of the distribution. Hence, if the Reporting Ratio is less than 1, significant values of NAB fall in the left tail of the distribution, while if the Reporting Ratio is greater than 1, significant values of NAB fall in the right tail of the distribution. This observation leads to the following selection criteria for statistically unusual associations (i.e., associations that are inconsistent with the independence null hypothesis):
RRAB<1 and CAB=P{NAB≦n}<θ 1.
RRAB>1 and C′AB=P{NAB≧n}<θ 2.
Here, θ represents a small threshold probability for significance (e.g., θ=0.01 corresponds to a 1% significance threshold). The cumulative probability CAB=P{NAB≦n} is easily computed using standard procedures. For example, the S-Plus® statistical software package (Insightful Corp., Seattle, Wash.) contains built-in procedure phyper which computes this probability. The complimentary probability C′AB=P{NAB>n} can, in principle, be computed using this function and the following result, provided n>0:
P{NAB≧n}=1−P{NAB<n}=1−P{NAB≦n−1}
This procedure performs extremely poorly in practice, however, due to round-off problems: small differences between much larger numbers can be highly inaccurate. A much better alternative proposed herein takes advantage of the relations between cumulative probabilities described by Johnson, which provides a basis for computing C′AB using cumulative distribution function routines (e.g., the phyper built-in function in S-plus) with modified arguments.
In the procedure described here, the statistical unexpectedness of the observed response NAB is defined as the reciprocal of the appropriate cumulative probability:
UAB=1/CAB if RRAB<1 and UAB=1/C′AB if RRAB>1
Since pairs of conditions that are inconsistent with the statistical independence hypothesis are those with small associated cumulative probabilities, the statistical unexpectedness measure defined here will be large for these pairs of conditions. In particular, it should be noted that UAB≧1 in all cases, but this quantity is unbounded above.
In certain embodiments, the basic pharmacovigilance tool proposed herein is inspired by the “volcano plot,” commonly used to present microarray data analysis results as described by Wolfinger [Wolfinger, 2001]. In Wolfinger 2001, a measure of signal strength (specifically, the differences between two mean responses) is plotted against the p-value associated with a t-test of the hypothesis that the two means are equal. These values are plotted for each of a large number of genes characterized by the microarray experiment, and “interesting” genes are generally regarded as those exhibiting a sufficiently large difference in means (i.e., a strong enough signal) and sufficiently small t-test p-values (i.e., statistical significance at a specified level). In sharp contrast to the disclosure of Wolfinger 2001, in the method described here (which is exemplary of the processing that may be performed in step 240 of
Heeley 2002 describes a pharmacovigilance-related plot of probability versus a measure of association which differs from certain embodiments of this invention in at least three respects. First, Heeley 2002 uses the Proportional Reporting Ratio defined earlier in this document, which is infinite when NB=NAB, as noted earlier. Second, instead of statistical unexpectedness, these authors use Yates' continuity-corrected χ2 values computed from a test of statistical independence. The original χ2 statistic was proposed in 1900 by Karl Pearson, who derived its distribution for large samples, and Yates proposed his correction in 1934 as a computationally simpler approximation to Fisher's exact test, which is based on the hypergeometric distribution. Johnson 1992 notes that the approximate p-values associated with this corrected χ2 statistic are “appropriate for large NB, provided NA/N is not too small.” This condition is frequently violated in the phamacovigilance applications considered here and therefore provides another reason why the plot of Heeley 2002 would not work well with the pharmacovigilance applications considered herein. More generally, the article by Mielke [Mielke, 2001] argues against the use of the χ2 statistic in problems like the one considered herein that are equivalent to tests of independence in a 2×2 contingency table. Specifically, these authors note that:
It is well known that when expected cell frequencies are small, probability values based on the asymptotic χ2 probability distribution may be erroneous . . . .
These authors further note that the quality of the χ2 approximation depends on the following four problem characteristics:
These observations are important since in the pharmacovigilance applications considered herein, while N is typically quite large, relevant values of NA and NB often cover a very wide range. Also, it should be noted that the χ2 approximation can be expected to degrade increasingly as we move further into the tails of the distribution, corresponding to small probabilities or large statistical unexpectedness values. This behavior is particularly undesirable since it precisely these values that are of greatest interest in the pharmacovigilance application considered in the present invention.
Finally, a third important difference between the approach described herein in certain embodiments of the present invention and that of Heeley 1993 is that the method described herein adopts a correction for multiple comparisons. See generally Westfall [Westfall, 1993]. Specifically, it should note that the identification of Condition A/B pairs with “large statistical unexpectedness values” corresponds to a rejection of the hypothesis of statistical independence between Conditions A and B. For a fixed significance level θ, the probability of erroneously rejecting the independence hypothesis for any individual pair is θ, but the probability of erroneously rejecting the independence hypothesis for at least one of M distinct comparisons is larger than θ. Indeed, if all of the comparisons are statistically independent, this probability could be on the order of Mθ. One popular solution to this problem is the Bonferroni approximation, which replaces the original significance threshold value θ with the modified value θ/M. While the Bonferroni correction is known to be conservative [Westfall 1993], it has the advantage of simplicity and is incorporated into the procedure described herein in certain embodiments. Alternatively, other corrections for multiple comparisons may also be used as described in, for example, Westfall 1993, the relevant disclosure of which is incorporated herein. Specifically, the pharmacovigilance plots described herein include two horizontal reference lines (for example, the lines 125 and 130 in
Typically, pharmacovigilance methods focus on positive associations between two conditions of interest, often explicitly excluding the case of negative associations from consideration (e.g., DuMouchel 1999). An advantage of the method described here is that it assesses both positive and negative associations in the same plot. Specifically, note that significant negative associations correspond to Reporting Ratios less than 1 and high statistical unexpectedness, while significant positive associations correspond to Reporting Ratios greater than 1 with high statistical unexpectedness. To separate these two regions, a vertical line is included in the plot considered here at RRAB=1 (for example, line 120 in
In the case considered here, both the dataset D and Condition A are fixed, meaning that both N and NA are fixed. Under these conditions, note that pz is a monotonically decreasing function of NB, bounded above by:
Hence, the absence of some Comparison Condition B is significant at the level θ if pz is less than θ, which is true if the following condition holds:
It should be noted that since the arguments of the logarithms are less than 1, the logarithms are negative; and this is why the direction of inequality was reversed in the last step above. While this condition is only sufficient for pz<θ, and therefore conservative, the approximation is quite good when NB<<N, as is typically the case in most pharmacovigilance data of interest. As discussed earlier herein, a simple correction for multiple comparisons in this problem is to replace the threshold θ with the Bonferroni-corrected threshold θ/Nz. The practical significance of this result is that it gives a good approximation to the number of records satisfying Condition B required for a complete absence of records satisfying Conditions A and B together to be significant. One application for this result is the detection of potentially beneficial drug effects that lead to fewer occurrences of a particular adverse event than expected by random chance. Another possible application is the detection of fraud (e.g., deliberate under-reporting of a particular adverse event).
In the example shown in
In a useful variation of the pharmacovigilance map 500, as shown in
In another useful variation, the graph corresponds to a particular pair of drugs taken together by a patient, and each point corresponds to an adverse event. This form of graph can be used to ascertain whether any significant drug interactions exist between the pair of drugs under study.
In another useful variation, the graph (first condition) corresponds to a combination of a drug and adverse event occurring together, and each point corresponds to a second drug also taken by the patient. This form of graph can be used to ascertain which drugs might be responsible for a suspected drug interaction that causes the particular adverse event.
In another useful variation, the graph corresponds to a particular drug, and each point corresponds to a second drug taken in combination with the first drug. This form of graph can provide background information for pharmacovigilance investigations, by indicating what drugs are most commonly taken along with a drug under investigation.
In another useful variation, the graph corresponds to a particular adverse event, and each point corresponds to a second adverse event which occurs along with the first one. This form of graph is useful for the detection and investigation of syndromes, which are combinations of symptoms (adverse events in this case) which occur together. Syndromes can be indicative of causal relationships between drugs and adverse events.
Many further variations in the designation of first condition (defining the graph) and second condition (defining the points) are possible, for example, to investigate three-way drug interactions. These are all considered within the scope of this invention.
Relative to the Gamma Poisson Shrinker (GPS) procedure of DuMouchel and the Bayesian Confidence Propagation Neural Net (BCPNN) procedure of Bate, commonly-used methods in pharmacovigilance, one advantage of certain embodiments described here is the use of two distinct characterizations for each pair of Conditions of Interest, (1) a sample-size independent association measure like the Reporting Ratio together with (2) a statistical unexpectedness measure under a null hypothesis of statistical independence, rather than a single score for each condition. In producing a single score, loss of separate information about disproportionality and statistical significance occurs. This makes it more difficult to visualize the data and analyze it in a manner that allows for expert human judgment in the medical interpretation of pharmacovigilance data.
A second advantage of certain embodiments relative to the above-referenced known procedures is the much weaker statistical assumptions on which the certain embodiments are based. Specifically, the certain embodiments are based on the assumption that all records satisfying a particular Condition of Interest are equivalent (i.e., exchangeable), and the null hypothesis considered is one of statistical independence between Conditions of Interest. No data distributions need be specified in this method, in contrast to methods like GPS, for which underlying data distributions are specified as discussed in DuMouchel 1999.
A third advantage of the method described herein is simplicity: the basic mechanics of the urn model formulation can be easily explained to medical and pharmaceutical personnel with limited statistical backgrounds, in contrast to methods like MGPS that are based on Bayesian statistics.
A fourth advantage over both the previously-referenced methods and the simple Proportional Reporting Ratio (PRR) of Evans is the ability to properly visualize and draw implications from the occurrence of zero to four events. Methods utilizing the chi-squared distribution are not valid below five events. The Bayesian methods referenced above avoid this limitation through a complex procedure based on strong statistical assumptions. These methods do not produce valid results for zero events.
Some of the features in certain embodiments of the present invention include the following:
For pharmacovigilance data derived from a spontaneous-reporting system such as AERS, the horizontal axis of the graphs produced by certain embodiments represents the Proportional Reporting Rate (PRR) or optionally an Odds Ratio (OR). Drug-safety data from a clinical trial may also be visualized and analyzed using certain of the embodiments disclosed herein. In this case, the available data for each adverse event type consists of: nD, the number of patients experiencing the adverse event while taking the study drug D; ND, the total number of patients in the study taking drug D; nP, the number of patients experiencing the adverse event while taking a placebo; and NP, the total number of patients in the study taking a placebo. It is possible that the study contains multiple arms and involves multiple drugs and doses, but the principles described here hold in those cases. In place of the PRR, certain embodiments can be practiced using the Relative Risk (ζ) of an adverse reaction for the drug vs. placebo:
or the Odds Ratio (OR)
Under the null hypothesis, ζ=OR=1. Statistical unexpectedness can be calculated based on an urn model which essentially says that the placebo urn and the drug urn contain the same proportion of black balls. In other words,
This corresponds to Fisher's Exact Test for a 2×2 table. While this statistical test is well known, certain embodiments disclosed herein describe methods of utilizing it to visualize drug safety data in a number of variations. The utility of this may include side-by-side visual comparison of pre-market clinical trial safety data, and post-market, spontaneously-reported safety data as described above.
In some cases, drug-safety data is derived from patient medical records. In this case, the available data consists of: nD, the number of patients experiencing the adverse event within a given time interval while taking a drug of interest D; ND, the total number of patients in the medical-records database taking drug D within a given time interval; nX, the number of patients experiencing the adverse event within the given time interval while not taking drug D; and NX, the total number of patients in the study taking a placebo. Relative Risk, Odds Ratio, and statistical unexpectedness can be calculated as for clinical trial data, and visualized according to the methods of this invention. This can facilitate the side-by-side visual and analytic comparison of: 1) Pre-market safety data from clinical trials; 2) Post-market active surveillance data determined by examining a set of patient medical records; 3) Post-market passive surveillance data from a spontaneous-reporting system such as the FDA AERS system.
The basic method described thus far starts with a fixed dataset D. In certain embodiments, a preprocessing step is provided (as shown in step 210 of
In all of these additional embodiments, the basic analysis procedure described here is applied separately to each partition of the dataset D. One advantages of this partitioning is a relaxation of the exchangeability assumptions from global (i.e., all records satisfying the Conditions of Interest in the dataset are exchangeable) to local (i.e., all records satisfying the Conditions of Interest in each subset are exchangeable with other records in the subsets). This permits the detection and treatment of violations of global exchangeability assumptions like time-trends, local bursts of reporting activity (for example, following news reports of suspected drug reactions), or other forms of non stationary data or data inhomogeneity. A second advantage of this partitioning is the ability to compare across partitions to detect systematic dependencies on partitioning criteria. In addition, by comparing results from stratification-based partitioning with those for the complete dataset, it is possible to detect Simpson's paradox, in which associations differ strongly (e.g., reverse direction) between stratified and non-stratified data. See Agresti [Agresti, 2002]. Detection of Simpson's paradox is potentially useful because a necessary condition for its occurrence is a sufficiently strong association between the Conditions of Interest and the stratification variables. See Schield [Schield, 1999]. The identification of these associations can be useful in detecting possibly important confounding influences (e.g., gender-specific drug interactions).
In the embodiments considered above, conditions are plotted as points on a graph. In a further embodiment, these points may be surrounded by boxes, ovals, or similar shapes to indicate the degree of statistical certainty, i.e. the confidence limits, related to the position of the point. These will be referred to as “confidence boxes”, though they may have an oval or similar shape.
In the above embodiment, random samples are generated, each containing 80% of the records in the database. In a further embodiment, the samples are chosen so that each sample eliminates a randomly-chosen 20% of the drugs in the database. In other words, samples are chosen on a drug-by-drug basis, rather than on a record-by-record basis. Confidence boxes are generated as described above. The particular utility of this embodiment is for the detection of masking, also called cloaking, in spontaneous-reporting systems. The phenomenon of masking occurs when an extremely strong association between a drug and an adverse event suppresses the signals of disproportionality for other drugs with that adverse event. If 20 samples are generated with 20% elimination as described above, there is approximately a 99% likelihood that any given drug will be eliminated in at least one of the samples. If masking is occurring, elimination of an unduly influential drug will cause a large shift in the Reporting Rate and SU for several other drugs. This will produce a wide confidence boxes, warning of the potential for masking.
Confidence boxes are also useful in comparing graphs representing two or more drugs or vaccines, or other sets of conditions. Boxes which do not overlap indicate significant differences in the safety profile for the drugs under study.
Confidence boxes can be created for maps such as the map 500 shown in
Note that confidence boxes can also aid in the design of future clinical trials, or in planning for the integration of additional data sources, since a large box indicates a particular issue where further safety data is needed.
Plotted points may be colored in a variety of ways to indicate stratification of the data reflected in each point. For example, points could be colored to reflect the male/female ratio corresponding to the given drug-adverse event combination. Colors can be scaled in a variety of visually-meaningful ways. Arbitrary color codes can be used, for example: green=male, red=female. Colors can represent a “temperature” scale, so that higher values correspond to “hotter” colors. Rather than coding a variable directly into a color scale, it is possible to use the color to represent a Z-score or other statistical significance score using a color. Appropriate color scales are discussed by Montag and others [Montag, 1999].
Many other uses of color are possible. Points can be colored to indicate gender ratios, age ranges, underlying diseases, or many other factors of use in safety investigations. Color could be used to compare safety results from two different populations or geographic areas. It could also be used to highlight differences in safety profiles between two related drugs, or between two different studies of a single drug. This could include comparison of pre-market vs. post-market safety data for a given drug or vaccine.
In the embodiments considered thus far, points are plotted to represent the Reporting Rate (or relative risk or odds ratio) and statistical unexpectedness corresponding to various conditions. In another embodiment, vectors are drawn from one point to another, where the first point represents a condition in one stratum or data set, and the second point represents the same condition in a second stratum or data set. In this manner, the data sets can be compared, since longer vectors will correspond to greater discrepancies between the data sets, and shorter vectors will correspond to high similarity between the data sets. The data sets could be two clinical trials of the same or related drugs, or could include spontaneous-reporting data or medical record data.
Certain embodiments of the present invention provide for constructing and plotting trajectories for specified Conditions of Interest as partitioning parameters like those described previously herein are varied systematically. As a specific example, using the moving-window partitioning described above would provide the basis for time-dependent characterizations that could reveal changes in adverse event/drug associations with time as drugs are used for longer periods of time, as other drugs are approved or recalled, for example. Such plots could consist of a graph with two lines, one representing statistical unexpectedness, and the other representing an association measure, both plotted as a function of time. Alternatively, the plots could consist of a line representing a trajectory in an X-Y space of statistical unexpectedness vs. association measure, with time as a parameter.
The size of the moving window can be adjusted to optimize the sensitivity and specificity of the pharmacovigilance data. Larger window sizes lead to greater sensitivity, that is, higher likelihood of observing statistically significant effects, because greater number of events will be included within the window. Smaller windows lead to better specificity, that is, fewer spurious observations because they reduce the likelihood that happenstance temporal trends in the data will lead to false positive associations.
The points plotted in trajectories may reflect data points within a moving window or neighborhood as discussed above, or they may reflect a cumulative collection of data points. For example, the points of a trajectory might reflect each quarter within the years 2001-2004, or they might reflect cumulative data by quarter from the first quarter of 2001 to the last quarter of 2004.
Trajectories can be used to track emerging safety issues, by plotting the evolution of SU and log(RR) for a particular drug-event combination over time. This may be helpful in post-market surveillance, where the relative risk for a particular drug-event combination could be monitored using data from an electronic medical record system or other surveillance system.
In certain particular studies such as clinical trials, the independent variable in the moving window could represent the dose of a drug under study, rather than time. If an adverse event were dose-related, the corresponding trajectory would be seen to move in the direction of stronger association and greater statistical unexpectedness.
In certain embodiments, the present invention provides for incorporating medical and/or biological likelihood-of-association information included in pharmacovigilance databases. In some cases, it is possible to characterize the likelihood of a causal association between a drug and an adverse event for a particular patient. Strategies for this include “de-challenge,” where it is observed whether or not the adverse event is alleviated by discontinuing the drug, and “re-challenge,” where, when it is safe to do so, the suspect drug is re-administered to see if the adverse event worsens or re-occurs. In these cases, drug-adverse event associations may be semi-quantitatively labeled as “possible,” “probable,” “likely,” etc. In this invention, we would consider these labels to be numerical markings (“0,” “1,” “2,” “3,” etc.) on the balls in the urn models. The mathematics herein would then apply to the sums of the markings on the balls, rather than to the count of balls chosen, but would otherwise remain the same.
A related notion could be used to incorporate information about particular genes or biochemical pathways affected by certain drugs, and about those genes and pathways that are involved in certain adverse events. To accommodate this, the balls in the urn would be marked with numerical vectors, with each element of the vector corresponding to a particular gene or biochemical pathway. For balls corresponding to drugs, the components of the vector would represent the strength of influence of that drug on the particular gene expression or biochemical pathway. For balls corresponding to adverse events, the components of the vector would represent the influence of the particular gene expression or biochemical pathway on the occurrence of the adverse event.
Certain additional embodiments provide for the development of pharmacovigilance analyses based on a sequence of related analyses of the type described in the preceding paragraphs. For example, once a potential drug/reaction association is identified from an analysis similar to that illustrated in
One skilled in the art would recognize that the foregoing describes a typical computer system connected to an electronic network. It should be appreciated that many other similar configurations are within the abilities of one skilled in the art and it is contemplated that all of these configurations could be used with the methods and systems of the present invention. Furthermore, it should be appreciated that it is within the abilities of one skilled in the art to program and configure a networked computer system to implement the method steps of the present invention, discussed earlier herein. For example, such a computing system could be used to implement the method of analyzing and displaying pharmacovigilance data as discussed earlier herein.
The present invention also contemplates providing computer readable data storage means with program code recorded thereon (i.e., software) for implementing the method steps described earlier herein. Programming the method steps discussed herein using custom and packaged software is within the abilities of those skilled in the art in view of the teachings herein.
Other embodiments of the invention will be apparent to those skilled in the art from a consideration of the specification and the practice of the invention disclosed herein. It is intended that the specification be considered as exemplary only, with such other embodiments also being considered as a part of the invention in light of the specification and the features of the invention disclosed herein. Furthermore, it should be recognized that the present invention includes the methods disclosed herein together with the software and systems used to implement the methods disclosed here.
The present application refers to the following publications. The referenced portions of each of these publications are incorporated herein in their entireties.
Agresti A, Categorical Data Analysis 2nd Ed. J. W Wiley & Sons: New York, 2002.
Bate A et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharm 54:315-321, 1998.
Brown E G, Wood L, and Wood S. The medical dictionary for regulatory activities (MedDRA). Drug Safety 20(2):109-17, 1999.
DuMouchel W, Bayesian Data Mining in Large Frequency Tables with an Application to FDA Spontaneous Reporting System, American Statistician, 53:197-190, 1999.
Du Mouchel W and Pregibon D. Empirical Bayes screening for multi-item associations. Seventh ACM SigKDD International Conference on Knowledge Discovery and Data Mining. 2001.
Evans, S J W, Waller P C, and Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiology and Drug Safety 10:483-486, 2000.
Gordon A D, Classification (2nd. ed.), Chapman and Hall: Norwell, Mass., 1999.
Gould A L, Practical Pharmacovigilance Analysis Strategies, Pharmacoepidemiology and Drug Safety. 12:423-458, 2003.
Groetzner, J et al. Airway anastomosis complications in de novo lung transplantation with sirolimus-based immunosuppression. J Heart Lung Transplant. 23(5):632-8, 2004.
Heeley E, Wilton L V, and Shakir S A W, Automated Signal Generation in Prescription-Event Monitoring, Drug Safety 25:423-432, 2002.
Johnson N L, Kotz S, and Kemp A W, Univariate Discrete Distributions (2nd. ed.), J. W. Wiley & Sons: New York, 237-253, 1992.
Mielke P W Jr. and Berry K J, Permutation Methods: A Distance Function Approach. Springer Verlag: New York, 273, 2001.
Montage ED. The Use of Color in Multidimensional Graphical Information Display. IS&T/SID Seventh Color Imaging Conference, Scottsdale, Ariz., 1999.Downloaded from the Internet on Jul. 25, 2005 from [http://www.cis.rit.edu/people/faculty /montag/PDFs/paper43. pdf] the Center for Imaging Science RIT website with the extension “/people/faculty/Montag/PDFs/paper43.pdf”.
Schield M, Simpson's Paradox and Cornfield's Conditions, A
Szarfman A, Machado S G, and O'Neill R T. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA's spontaneous reports database. Drug Safety 25(6): 381-392, 2002.
U.S. Department of Health and Human Services, Food and Drug Administration, “Guidance for Industry: Food Pharmacovigilance Practices and Pharmacoepidemiologic Assessment”. Download from the Internet at [http://www.fda.gov/cber/gdlns/pharmacovig.pdf] the FDA government website with the extension “/cber/gdlns/pharmacovig.pdf”, 20 Jul. 2005.
Westfall P, and Young S, Resampling based Multiple Testing, J. W Wiley & Sons: New York, 1993.
Wolfinger R, et al., Assessing Gene Significance from cDNA Microexpression Data via Fixed Model, J. Computational Biology 8:625-637, 2001.
Hochberg, Alan M., Pearson, Ronald, Kingan, Robert J.
Patent | Priority | Assignee | Title |
11380440, | Sep 14 2011 | CERNER INNOVATION, INC | Marker screening and signal detection |
11817186, | Sep 14 2011 | Cerner Innovation, Inc. | Marker screening and signal detection |
11869671, | Sep 14 2011 | Cerner Innovation, Inc. | Context-sensitive health outcome surveillance and signal detection |
8428970, | Jul 13 2011 | Information record management system | |
9280612, | Dec 14 2012 | MICRO FOCUS LLC | Visualizing a relationship of attributes using a relevance determination process to select from candidate attribute values |
9607266, | Jul 23 2013 | Tata Consultancy Services Limited | Systems and methods for signal detection in pharmacovigilance using distributed processing, analysis and representing of the signals in multiple forms |
9779524, | Jan 21 2013 | MICRO FOCUS LLC | Visualization that indicates event significance represented by a discriminative metric computed using a contingency calculation |
Patent | Priority | Assignee | Title |
20020183965, | |||
20030046110, | |||
20040117126, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Oct 25 2005 | Prosanos Corp. | (assignment on the face of the patent) | / | |||
Dec 19 2005 | PEARSON, RONALD | PROSANOS CORP | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 017525 | /0215 | |
Dec 19 2005 | HOCHBERG, ALAN M | PROSANOS CORP | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 017525 | /0215 | |
Jan 04 2006 | KINGAN, ROBERT J | PROSANOS CORP | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 017525 | /0215 | |
Dec 21 2010 | PROSANOS CORPORATION | UBC LATE STAGE, INC | MERGER SEE DOCUMENT FOR DETAILS | 026193 | /0919 | |
Dec 21 2010 | PROSANOS CORPORATION | UBC LATE STAGE, INC | CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNMENT RECORDATION PREVIOUSLY RECORDED ON REEL 026193 FRAME 0919 ASSIGNOR S HEREBY CONFIRMS THE SERIAL NUMBER 11 966,439 SHOULD BE REMOVED FROM THIS RECORDATION 12 065631 AND 11 257,395 SHOULD REMAIN | 026588 | /0674 | |
Dec 27 2017 | UBC LATE STAGE, INC | MEDCO HEALTH SOLUTIONS, INC | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 044493 | /0388 | |
Apr 17 2018 | UBC LATE STAGE, INC | Wells Fargo Bank, National Association | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 046565 | /0614 | |
Feb 28 2020 | MEDCO HEALTH SOLUTIONS, INC | UBC LATE STAGE, INC | RELEASE OF SECURITY INTEREST IN PATENT COLLATERAL PREVIOUSLY RECORDED AT REEL 044493, FRAME 0388 | 052068 | /0304 | |
Feb 28 2020 | UBC LATE STAGE, INC | GOLUB CAPITAL LLC, AS ADMINISTRATIVE AGENT | PATENT SECURITY AGREEMENT | 052068 | /0051 |
Date | Maintenance Fee Events |
Feb 02 2010 | ASPN: Payor Number Assigned. |
Jul 05 2013 | STOL: Pat Hldr no Longer Claims Small Ent Stat |
Aug 30 2013 | REM: Maintenance Fee Reminder Mailed. |
Sep 24 2013 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Sep 24 2013 | M1554: Surcharge for Late Payment, Large Entity. |
Jul 19 2017 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Jul 20 2021 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Jul 20 2021 | M1556: 11.5 yr surcharge- late pmt w/in 6 mo, Large Entity. |
Date | Maintenance Schedule |
Jan 19 2013 | 4 years fee payment window open |
Jul 19 2013 | 6 months grace period start (w surcharge) |
Jan 19 2014 | patent expiry (for year 4) |
Jan 19 2016 | 2 years to revive unintentionally abandoned end. (for year 4) |
Jan 19 2017 | 8 years fee payment window open |
Jul 19 2017 | 6 months grace period start (w surcharge) |
Jan 19 2018 | patent expiry (for year 8) |
Jan 19 2020 | 2 years to revive unintentionally abandoned end. (for year 8) |
Jan 19 2021 | 12 years fee payment window open |
Jul 19 2021 | 6 months grace period start (w surcharge) |
Jan 19 2022 | patent expiry (for year 12) |
Jan 19 2024 | 2 years to revive unintentionally abandoned end. (for year 12) |