Background Consider a meta-analysis where a ‘head-to-head’ comparison of diagnostic tests for a disease of interest is intended. Assume there are two or more tests available for the disease, where each test has been studied in one or more papers. Some of the papers may have studied more than one test, hence the results are not independent. Also the collection of tests studied may change from one paper to the other, hence incomplete matched groups.

Methods We propose a model, the proportional odds ratio (POR) model, which makes no assumptions about the shape of OR_p, a baseline function capturing the way OR changes across papers. The POR model does not assume homogeneity of ORs, but merely specifies a relationship between the ORs of the two tests.

One may expand the domain of the POR model to cover dependent studies, multiple outcomes, multiple thresholds, multi-category or continuous tests, and individual-level data.

Results In the paper we demonstrate how to formulate the model for a few real examples, and how to use widely available or popular statistical software (like SAS, R or S-Plus, and Stata) to fit the models, and estimate the discrimination accuracy of tests. Furthermore, we provide code for converting ORs into other measures of test performance like predictive values, post-test probabilities, and likelihood ratios, under mild conditions. Also we provide code to convert numerical results into graphical ones, like forest plots, heterogeneous ROC curves, and post test probability difference graphs.

Formulation

To compare two diagnostic tests i and j, we want to estimate the difference in their performance. However, in reality such difference may vary from one paper (study) to the other. Therefore ∆_i,j,p = PERF_i,p – PERF_j,p , where the difference ∆ depends on paper index p, where PERF_i,p is observed performance of test i in paper p. To simplify notation, assume that a single number measures performance of each test in each paper. We relax this assumption later, allowing for the distinction between the two types of mistakes (FNR and FPR, or equivalently TPR and FPR). We decompose the differences

(1)             ∆_i,j,p = PERF_i,p – PERF_j,p = δ_i,j + δ_i,j,p ,

where δ_i,j is the ‘average’ difference between the two tests, and δ_i,j,p is deviation of the observed difference within paper p from the average δ_i,j. The δ_i,j is an estimator for the difference between performance of the two tests. Note by using deviation parameterization (similar to an ANOVA model) [12, pp.51 & 45] we explicitly accept and account for the fact that the observed difference varies from one paper to the other, while estimating the ‘average’ difference. This is similar to a random-effects approach where a random distribution is assumed for the ∆_i,j,p and then the mean parameter for the distribution is estimated. In other words, one does not need to assume ‘homogeneous’ difference of the two tests across all the papers, and then estimate the ‘common’ difference [13].

The observed test performance, PERF, may be measured in several different scales, such as paired measures sensitivity and specificity, positive and negative predictive values, likelihood ratios, post test odds, and post test probabilities for normal and abnormal test results; as well as single measures such as accuracy, risk or rate ratio or difference, Youden’s index, area under ROC curve, and odds ratio (OR). When using OR as the performance measure, the marginal logistic regression model

(2)             logit(Result_pt) = b₀ + b₁*Disease_pt + b₂*PaperID_pt + b₃*Disease_pt*PaperID_pt + b₄*TestID_pt + b₅*Disease_pt*TestID_pt + b₆*TestID_pt*PaperID_pt + b₇*Disease_pt*TestID_pt*PaperID_pt

implements the decomposition of the performance. Model (2) is fitted to the (repeated measures) grouped binary data, where the 2-by-2 tables of gold-standard versus test results are extracted from each published paper. In the model (2) Result is an integer-valued variable for positive test result (depending on software choice, for grouped binary data, usually Result is replaced by number of positive test results over the total sample size, for each group); Disease is an indicator for actual presence of disease, ascertained by the gold standard; PaperID is a categorical variable for papers included in the meta-analysis; and TestID is a categorical variable for tests included. Regression coefficients b₂ to b₇ can be vector valued, meaning having several components, so the corresponding categorical variables should be represented by suitable number of indicator variables in the model. Indexes p and t signify paper p and test t. They define the repeated measures structure of the data [10]. Note model (2) fits the general case where there are two or more tests available for the disease, where each test has been studied in one or more papers. Some of the papers may have studied more than one test; hence the results are not independent. Also the collection of tests studied may change from one paper to the other, hence incomplete matched groups.

From model (2) one can show that

LOR_pt = b₁ + b₃*PaperID_pt + b₅* TestID_pt + b₇*TestID_pt*PaperID_pt

and therefore the difference between performance of two tests i and j, measured by LOR, is

LOR_pi – LOR_pj = b₅* TestID_pi – b₅* TestID_pj + b₇*TestID_pi*PaperID_pi – b₇*TestID_pj*PaperID_pj

where we identify δ_i,j of the decomposition model (1) with the b₅* TestID_pi – b₅* TestID_pj, and identify δ_i,j,p with b₇*TestID_pi*PaperID_pi – b₇*TestID_pj*PaperID_pj .