Adjustment for comorbidities is common in performance benchmarking and risk prediction. Despite the exponential upsurge in the number of articles citing or comparing Charlson, Elixhauser, and their variants, no systematic review has been conducted on studies comparing comorbidity measures in use with administrative data. We present a systematic review of these multiple comparison studies and introduce a new meta-analytical approach to identify the best performing comorbidity measures/indices for short-term (inpatient + ≤ 30 d) and long-term (outpatient+>30 d) mortality. Articles up to March 18, 2011 were searched based on our predefined terms. The bibliography of the chosen articles and the relevant reviews were also searched and reviewed. Multiple comparisons between comorbidity measures/indices were split into all possible pairs. We used the hypergeometric test and confidence intervals for proportions to identify the comparators with significantly superior/inferior performance for short-term and long-term mortality. In addition, useful information such as the influence of lookback periods was extracted and reported. Out of 1312 retrieved articles, 54 articles were eligible. The Deyo variant of Charlson was the most commonly referred comparator followed by the Elixhauser measure. Compared with baseline variables such as age and sex, comorbidity adjustment methods seem to better predict long-term than short-term mortality and Elixhauser seems to be the best predictor for this outcome. For short-term mortality, however, recalibration giving empirical weights seems more important than the choice of comorbidity measure. The performance of a given comorbidity measure depends on the patient group and outcome. In general, the Elixhauser index seems the best so far, particularly for mortality beyond 30 days, although several newer, more inclusive measures are promising.