Why causal estimation needs causal evaluation, and how Inverse Propensity Weighting and Doubly Robust scores change model comparisons
What if your uplift metric ranked an imperfect model above the true treatment-effect ranking? In a simulation where we know the answer, Qini can do exactly that, and following its recommendation means choosing a ranking with lower true causal gain.
Let’s start with the classical uplift problem. Suppose a company needs to decide which customers of theirs to send marketing messages to. The best ranking of customers would be based on the expected benefit from sending them a marketing message, which depends on the value of the action we try to nudge them to and the shift in the probability of performing this action due to the receiving the communication. Of course, the company doesn’t know these probabilities, so it uses an imperfect ranking based on their estimates.
Let’s take a hypothetical situation in which we know the real probabilities, that means we know the real perfect ranking of the customers in terms of the efficiency of communication. Suppose we decided to compare it with the imperfect ranking based on our estimates of these probabilities. Funny thing — it occurs that the Qini metric (which is often used for uplift measurement and will be described further) will assign better score to the imperfect ranking. How could it be?
Throughout this article, Qini and IPW are calibrated to match DR at one reference: the true-CATE ranking in the higher-selectivity, seed-42 holdout. The constants stay fixed for all learners, bootstrap resamples, assignment policies and tuning results. This aligns the reference scores, but does not adjust for confounding or change the ordering of models within a metric. In this example, calibrated Qini still favors Linear DR over the true-CATE ranking, whereas known causal gain favors true CATE. The calibration formula is given below.
The reason is that it’s often not enough to use a causal model that accounts for the confounding. If we have the treatment assignment dependent on the customer characteristics, we also need to adjust the metric to evaluate this causal model, as the Qini metric still can contain the selection bias.
In this article, we will go through the whole evolution of our attempts to measure the causal effect — from the uplift bars, through the Qini metric towards the IPW and doubly robust scores to evaluate the models. We will use six common learners from S learner to DML learner, and will see how we can assess their estimates depending on different degrees of confounding.
The targeting question
For a given treatment T (a marketing message) and an outcome Y (a customer’s actual purchase) a customer’s Conditional Average
Treatment Effect (CATE) is given by:
CATE can be perceived as the average treatment effect (ATE) for a given customer with profile x. It can be used for customer targeting since high CATE shows that the the effect of the treatment on this customer is higher. Just to highlight — it doesn’t mean the customer with high CATE has the highest outcome or the probability of making some action. It means that if contacted, the customer has the highest uplift in this outcome or probability compared to the situation when the customer isn’t contacted.
Usually, companies have limited budgets for their marketing activities, so the ranking is instrumental to understand how to optimally distribute the budget for these activities.
In this article, we will compare the 6 models based on the synthetic data using different metrics. In our case we know the true treatment effect, which is usually not the case. This approach allows us to make a comparison and study the uplift metrics in detail.
We write the propensity as:

It represents the difference in treatment probability, depending on the user profile. Irrespective of the metric and evaluation method and the model — we need some number of conditions to be satisfied so that our causal estimator works: consistency, no relevant interference, no unmeasured confounding and overlap.
Balanced group sizes can hide selective assignment
We simulate 1 million individual synthetic customers for which we have several characteristics: age, income, past spending over time, time since last contact with the customer. We simulate a factual communication policy that depends on the two dimensions: age and last contact activity,

where a is the intercept for calibration, and Z is the standardized product of age and last contact activity. We have provided the executable code for the outcome in the companion.
To simulate this type of data, the following parameters are fixed for the communication policy: the probability is to be between 0.1 and 0.9, with 50% treated on average. The inactivity and age of the customer are used to predict the probability of communication with the customer.
The purchase probability is also based on a logistic function and dependent on customer characteristics. Moreover, the treatment effect itself is nonuniform and impacts different customers in a different way.
Below in the code you may see the simulation algorithm:
import numpy as np from scipy.special import expit def message_probability(age, days, strength=2.5): # Population: independent, uniformly distributed ages and inactivity days. population_product = ( np.arange(18, 71)[:, None] * np.arange(366)[None, :] ).ravel() mean = population_product.mean() std = population_product.std() z_population = (population_product - mean) / std # Choose intercept a so the population's average propensity is 0.5. lo, hi = -20.0, 20.0 for _ in range(70): a = (lo + hi) / 2 if expit(a - strength * z_population).mean() < 0.5: lo = a else: hi = a a = (lo + hi) / 2 z = (np.asarray(age) * np.asarray(days) - mean) / std return 0.1 + 0.8 * expit(a - strength * z) def purchase_probability(age, income, spending, days, treatment): # Baseline purchase log-odds. baseline = ( -1 - 0.02 * age + 0.00005 * income + 0.00003 * spending + 0.01 * days - 0.000001 * age * income - 0.0000005 * spending * days ) # Treatment changes log-odds differently across customer profiles. treatment_effect = ( 0.5 - 0.011 * days + 0.05 * age - 0.00004 * days * age ) return expit(baseline + treatment * treatment_effect) # Generate customers and assign messages. rng = np.random.RandomState(42) n = 1_000_000 age = rng.randint(18, 71, n) income = np.clip(rng.normal(50_000, 15_000, n), 0, None) spending = np.clip(rng.normal(1_000, 500, n), 0, None) days = rng.randint(0, 366, n) propensity = message_probability(age, days) treatment = rng.binomial(1, propensity) # Potential purchase probabilities and known conditional treatment effects. p0 = purchase_probability(age, income, spending, days, treatment=0) p1 = purchase_probability(age, income, spending, days, treatment=1) true_cate = p1 - p0 # Observe one outcome under each customer's assigned treatment. purchase_probability_observed = np.where(treatment == 1, p1, p0) purchase = rng.binomial(1, purchase_probability_observed) For every profile we calculate p₁(x), p₀(x) and TrueCATE=p₁(x)−p₀(x). As the treatment groups in the simulation are almost of equal size the mean of the true-CATE values for the treated customers is approximately +0.191 and for the controls −0.003. The observed purchase-rate difference is 0.119. The ATE for this simulation is 0.094 (all numbers are rounded to 3 FP digits).
Below are the graphs which allow us to have a high-level understanding of the data we have generated:

Six rankings on the same held-out customers
We trained on 700k customers and then evaluated on the 300k held-out customers using seed 42 for this example. Each of the S, T, X, DA, DR, DML estimators was implemented using both linear and “boosted” base models. Note that all of the estimators above (S,T,X,DA,DR,DML) have lengthy derivations and are material for another article. Here I will just provide the high-level intuition of the algorithms.

The models used for this article are the EconML implementations (the S- and T-learners as well as the DA-, DR-, and DML-learners with a linear and XGBoost base model).
Our starting point will be an S-learner as the simplest and the most intuitive model. Its holdout MSE against the true CATE is 0.0006, with correlation 0.9964. That tells us its effect estimates are quite accurate in this simulation. But in reality we don’t know the true effect, so we need another way to assess targeting.
Looking at uplift without knowing the truth
The most obvious way to do this is an uplift barplot. We sort customers by the CATE, divide them into buckets and compare treatment and control outcomes for the buckets.

The graphs look useful: the right-hand buckets have larger observed differences than the left-hand ones. However, those differences are not quite the true effects. Grouping by predicted CATE does not necessarily make the treatment and control customers comparable inside each bucket — they can be very different in their characteristics.
The plot helps us inspect a model, but we should not interpret its bar differences as causal effects or use only them to decide who should receive treatment.
Quite often we just need one number, which allows us to compare different estimators. Something similar to the ROC AUC score we sometimes use to assess classification models. That’s where the Qini AUC comes into play.
From uplift bars to the Qini curve
Instead of having a bar plot that compares uplift for a number of different buckets of customers, look at the single customer with the highest predicted value of treatment. Then add the next customer with the highest predicted value, then the next customer, and so on. For each customer added in the group we calculate the total positive outcomes for the treatment and control.
Rₜ(k) and R꜀(k) denote the positive-outcome totals for the treatment group and the control group, respectively, for the top k customers. Let Nₜ (k) and N꜀ (k) denote the arm sizes for the treatment group and the control group.
The Qini metric is then defined as:

Let p̂=Nₜ(N)/N be the observed treatment share in the evaluation sample. First, define q(f)=Q(k)/(Np̂), where f=k/N, and let A_Q be its area above the line connecting the start and end points:

Zero is the reference area for random targeting; finite-sample scores can fluctuate around it. The normalization by N and horizontal fraction f=k/N are already included above, so we do not divide this area by N² again. Dividing by the treatment share supplies the causal-gain unit convention, but does not remove selection bias.
For the additional calibration, let r₀ denote the true-CATE ranking in the higher-selectivity, seed-42 holdout. We use DR as the reference score and calculate the fixed divisors:


All displayed evaluation areas and differences then use:

All three reference scores are therefore start as 0.0826. The curve below is also divided by c_Q, so its area matches the tables. This is an oracle-assisted calibration for this simulation: it requires knowing the true-CATE ranking and does not establish that the scores estimate the same quantity accurately. They are calibrated only for comparability purposes. The calibration preserves each evaluator’s model ordering. We freeze these constants everywhere, including within bootstrap resamples.
Numbers are rounded for display. Differences and confidence intervals are calculated before rounding, so equal displayed scores need not be identical.
Here is the S-learner’s Qini curve for the illustration:

Why can Qini prefer the wrong ranking
The main problem is that the Qini metric in its evaluation still uses the observed treatments and outcome variables. Remember that the customers were selected for treatment based on their characteristics; thus, the treatment/control differences still contain selection bias.
A causal learner estimates the treatment effect. If done correctly, it is potentially unconfounded. But the metric is still confounded, as it is observational! We’ve seen it already at the graph with the uplift bars — the difference was not matching the true effect.
This mismatch might be an insignificant calculation artifact if it wasn’t for its potential effect on qualitative conclusions based on the analysis. In our example, when we compare the models, the Linear DR learner gives us better metrics than the “oracle” model, which knows the true CATE. This is indicative of a large problem. Now we know we can’t trust Qini, because it can choose a wrong estimator for us.
Therefore, we now understand that we need to improve not just a learner, but the evaluation as well. The ideal situation of course — randomized treatment assignment. But when we have only biased observational data — we can use Inverse-Propensity Weighting (IPW) or Doubly Robust (DR) score which we discuss below.
Adjusting the evaluation with IPW and DR
Inverse-propensity weighting gives each observed outcome a weight based on how likely its assignment was:

Doubly robust scoring also uses the outcome prediction and the residual, but weights the residual by the inverse propensity score of the customer.

In the formulas above, T means the treatment flag, Y — the outcome variable. ê(X_i) — the propensity model (probability of being treated), μ̂₀(X_i) and μ̂₁(X_i) — the models of the outome prediction under treatment and control.
Usually, the DR-learner uses the same pseudo-outcome as you see above to fit effects, but here we are not looking for it — we are evaluating a trained ranking. The ranking can come from any model — not just the DR-learner.
Next step — we sort customers by the model-ranked scores, summing up the corresponding scores:

These are the uncalibrated IPW and DR gain curves. IPW and DR adjust for confounding under their respective assumptions. Under randomized assignment, treatment-share-normalized Qini targets the same causal-gain convention. For display throughout this article, we additionally divide the IPW curve and area by the fixed reference divisor c_IPW; DR stays unchanged. Calibration is separate from the causal adjustment.
For the adjusted methods, we estimate a propensity model and two outcome models separately from the main ranking model. They are called the nuisance models. They are estimated using the observable variables X, T, and Y. For the nuisance models, we use three-fold crossfitting, which guarantees that for both predictions the customer was excluded from the training sample for the models. All rankings use the same evaluation scores.
Running the following Python code in the companion directory trains the S-learner and, in addition, fits separate evaluation models.
import numpy as np from sklearn.model_selection import train_test_split from uplift_lib import ( generate_dataset, FEATURES, model_factories, crossfit_evaluation_nuisance, adjusted_scores, gain_curve, qini_auc, area, ) data = generate_dataset(1_000_000, seed=42, assignment="selection_high") train, test = train_test_split( np.arange(len(data)), test_size=0.3, random_state=42 ) X = data[FEATURES].to_numpy() y = data.MadeOrder.to_numpy() t = data.TreatmentGroup.to_numpy() model = model_factories(family="xgb", seed=42, threads=4)["S"]() model.fit(y[train], t[train], X=X[train]) ranking = model.effect(X[test]).ravel() evaluation = data.iloc[test].reset_index(drop=True) observed = evaluation[FEATURES + ["TreatmentGroup", "MadeOrder"]] nu, audit = crossfit_evaluation_nuisance(observed, seed=2042) ipw = adjusted_scores( evaluation.MadeOrder, evaluation.TreatmentGroup, nu.e_hat ) dr = adjusted_scores( evaluation.MadeOrder, evaluation.TreatmentGroup, nu.e_hat, nu.mu0_hat, nu.mu1_hat, ) c_q = 1.5445466434481507 # Frozen high-policy seed-42 reference. c_ipw = 1.0005932054691182 print( "Calibrated Qini", qini_auc(evaluation.MadeOrder, evaluation.TreatmentGroup, ranking) / evaluation.TreatmentGroup.mean() / c_q, ) print("Calibrated IPW", area(*gain_curve(ranking, ipw)) / c_ipw) print("DR", area(*gain_curve(ranking, dr))) The errors in the evaluation of different estimators are close to each other, so we better look at the error of the score of each ranking with respect to the causal gain of that ranking, which is known from the data generation process (thanks to our synthetic dataset!).

After calibration, Qini’s residual errors across the six boosted rankings range from +0.0002 to +0.0008. Calibrated IPW errors range from +0.0001 to +0.0002, and DR errors from +0.0001 to +0.0002. Qini favors the wrong ranking in the Linear-DR comparison below.
The disagreement matters for model choice, too:

The “true gain” is the area above the endpoint diagonal of the cumulative true-CATE curve for each ranking. IPW and DR produce scores close to these known gains and favor the true-CATE ranking over Linear DR. They do not reproduce every small ordering difference among the boosted learners. Calibrated Qini still favors Linear DR over the true-CATE ranking.
When you use these methods — keep in mind that IPW still requires a correct propensity model, and DR still requires a correct propensity OR both outcome models. Both scorers require consistency, no unmeasured confounding and overlap. Estimated models and finite samples leave residual error, so these approaches are not automatically unbiased scores for every dataset. So, you use it, but be careful as always.
Are the differences between models convincing
We have only compared point estimates for the different models so far. It might be interesting to see how large the differences are between the estimates for another random set of evaluation customers (i.e. different realization of the holdout sample of 300k customers with their respective multiplicities). In other words — are these differences statistically significant and consistent?
In order to investigate this we conducted 1,000 paired bootstrap resamples of the 300,000 customers in the holdout data set. For each of the resampled evaluation data sets, we computed the statistics of interest for all models and all evaluation methods. The resulting values’ 2.5th and 97.5th percentiles are provided in brackets below.
These are the conditional 95% percentile intervals. Note that we hold the trained models and nuisance functions fixed and resampling only the evaluation part. This means we don’t guarantee the 95% coverage for the full procedure bootstrap. The experiment is repeated later with 3 different seeds to ensure the conclusions are robust.

Note that overlying intervals do not directly answer questions of whether one model is better than another. More relevant are the intervals for the paired differences between the two models under consideration (how much one model is better than another). Below we compared each of the models with the oracle truth.

Which estimator would we choose here?
Choosing a CATE estimator is a broader topic that deserves a separate article. For this example, I would choose the XGBoost S-learner as a simple starting point. It has the simplest estimation structure among the six learners considered: one outcome model, with treatment included as an input feature. The DML estimator has the highest DR-gain point estimate among the six boosted learners, but its advantage over S is small: the paired S-minus-DML difference is -0.00006, with a conditional 95% interval which includes zero. That does not prove equivalence and this is not the claim that S learner is always the best. But if the difference in the 5th digit is not critical for your application, I would choose the S-learner for this specific case. Simplicity wins.


The comparison of S with T has a positive DR-area difference of +0.0002 with a Conditional 95% Interval of [+0.00003, +0.0003]. So, within these models and this fixed set of nuisance fits, S has a statistically significant DR-area difference compared with T. The paired intervals for the comparisons with X, DA, DR, and DML all include zero, so there is no evidence of a consistently higher DR area for S.
We don’t use Qini here anymore, because we’ve seen before that its confoundedness affects its conclusions even on a qualitative scale. Even increasing bootstrap sample for the Qini won’t make it unbiased, so we can stop using it.
Does the assignment policy change the result
We have so far followed one data generation to model comparison example. Let’s now slightly change the assignment policy and check if the results are stable under different assignments.
We will replace 2.5 in the propensity rule by 0.5 and by 1, affecting the selection bias and we will call it “low” and “middle” selectivity. The population treatment share will still be 50% as we have calibrated our assignment in this way. We will also refit all the learners for 3 different seeds for robustness.
For this figure, we rank customers directly by their known true CATE. No S, T or other learner supplies the ranking. We hold the customer profiles and this perfect ordering fixed across assignment policies within each seed, then change which customers receive treatment and which outcomes we observe. The question is whether each evaluator can accurately score the same ranking as assignment becomes more selective.
All three evaluators’ point estimates will coincide in the “higher” selection policy by construction because of the calibration we’ve done for comparability. The valibration is made based on the true CATE and we are considering the true CATE ranking in this case. What is interesting is how the rankers behave in case of the changes in the assignment policy.
We see that the DR and IPW behave quite stable, and the Qini is not stable at all and changing a lot depending on the assignment policy. This is one more evidence of how it can be influenced by different levels of selection bias.

All three runs of increased selectivity show the same reversal between Linear DR and true CATE as was seen in the single run of increased selectivity. The expected-count diagnostic is also well-behaved. This constitutes more empirical evidence for the phenomenon than a single lucky or unlucky run. (Of course, three runs do not guarantee to cover the variability in all possible datasets.)
Does a small amount of tuning change the comparison?
What about hyperparameter tuning? The evaluator also affects which hyperparameters do we select! We gave each of the six boosted learners 15 configurations, including its default. The original 700,000 training customers were split into 500,000 for fitting candidates and 200,000 for validation. Then we selected each learner’s configuration using cross-fitted DR gain on that validation sample, without using true CATE.
Then we have frozen the selected hyperparameters before generating a fresh 300,000-customer test sample from the ”high” selectivity policy. Both default and selected models were refitted on the same original 700,000 training customers, then evaluated on these new test customers with shared cross-fitted evaluation models. The earlier article holdout was not used in this tuning comparison. So, do not attribute differences from earlier tables to tuning alone. Below you will see the illustrated effect of the tuning on the three metrics considered before and on the RMSE metric which is calculated relative to the true CATE which we know as we have generated our dataset (it’s omitted from other sections, because in reality we never know the true CATE).



The direction of the effect from our tuning is not clear for S, X, DA, DR, DML learners as the intervals include zero, which corresponds to no change in the values. But for T-learner the paired intervals all favor tuning which also improves the CATE RMSE from 0.0328 to 0.0237. S-learner has the most extreme behavior where the DR point estimate decreased, and the CATE RMSE increased from 0.0243 to 0.0316 on the fresh test sample. So our tuning in case of the S-learner was not very fruitful.
To be clear, it’s still hard to compare the tuning of such different models, as the number of potential hyperparameter changes is different for a simple S-learner and for the DML. But the idea of this section is to show that the hyperparameter runing doesn’t change the ranking effect of confounded metric (Qini) we’ve shown before.
What to do when comparing uplift models
In conclusion, let’s try to formulate — what should we do when comparing the uplift models quality? First evaluate the assignment to treatment in your experiment. If the assignment to treatment is not random (as in most cases) then a familiar uplift plot or Qini curve don’t work anymore. Use both an evaluation design and the estimator which assumptions still hold (don’t require random assignment).
Using IPW and a DR evaluator of causal gain on the the data gave scores close to the known causal gain and kept the estimators ranking correct. I would opt in for the DR evaluation, well, because it’s Doubly Robust, so it’s still valid even if one of its assumptions failed.
Ultimately, for your marketing campaign (or any other application of the uplift model) look at what’s important for your case: the budget, cost and incremental value for a given budget. Area under the curve is interesting but not what you ar optimizing for as the business objective.
Code and reproducibility
The separate evaluation companion contains evaluation.ipynb, the shared model and metric implementation, all numerical tables, paired bootstrap draws, figures and the tuning protocol. Its default notebook run analyzes the completed experiments and independently checks IPW/DR formulas from stored observed rows and nuisance predictions.
python reproduce.py --output regenerated recreates all baseline fits, evaluator fits, diagnostics, resamples, tuning and figures in a new directory. The single code example above is supplied as article_example.py. The README distinguishes these full fitting steps from quick notebook analysis. Three-policy seed ranges and fixed-fit bootstrap intervals retain their separate meanings.
References
[1] PyWhy, EconML Meta-Learners, version 0.16.0 documentation.
[2] E. H. Kennedy, Towards optimal doubly robust estimation of heterogeneous causal effects (2020; revised 2023), arXiv.
[3] scikit-uplift contributors, Metric implementation, version 0.5.1 documentation.
[4] grf contributors, Assessing heterogeneity with RATE, documentation. Our own conditional row-bootstrap procedure is used in this article instead of grf’s implementation of RATE for causal heterogeneity inference.