The curve

Here is the whole phenomenon in one curve. It traces the out-of-sample risk of plain ridgeless least squares as the number of parameters grows relative to the sample size: the risk climbs as the model approaches just enough capacity to interpolate, blows up at that interpolation threshold, and then — against every classical instinct — falls a second time, eventually dipping below the null risk.

In math notation, we fix , sweep , fit ridgeless least squares at each step, and trace the out-of-sample risk against (isotropic case , signal norm held fixed for a fair comparison). The risk climbs toward , blows up at the interpolation boundary, then descends a second time for .

Hastie et al. (2022) make this exact in the limit with :

Bias up, variance down

Why does the risk fall again past ? The two components move in opposite directions.

The bias rises with for a reason familiar from the regime: the minimum-norm solution lives in the row space of , an -dimensional slice of . As grows that slice shrinks relative to the whole, so more of falls in the directions the data never see and the min-norm rule zeroes out. More parameters, more of left on the table, more bias.

The variance falls with , which feels backwards — surely more parameters means more variance? Not for the minimum-norm interpolator. Extra columns give the solver more directions to spread the fit across, so the minimum-norm solution to generally has smaller norm as grows, and a lower-norm interpolator is smoother and more stable — effectively more regularized, for free.

The spike at is where both forces peak. With the model has just barely enough capacity to interpolate and no slack to do it gracefully: is nearly singular and the variance term explodes. The interpolation threshold is not a sweet spot — it is the worst place to be.

It’s mostly a regularization story

The dramatic part of the curve (the divergence at ) is an artifact of insisting on interpolation, and interpolation is a choice, not a law. Ridgeless least squares is just ridge regression with the penalty dialed to zero, the one setting a statistician would never choose on purpose.

Hastie et al. (2022) prove it: optimally-tuned ridge dominates minimum-norm least squares at every and every SNR, well-specified or not. Its risk has no spike — proper smooths the curve into a single well-behaved descent, with the global minimum (under misspecification) sitting right around , exactly where ridgeless is worst. And the optimal penalty is estimable: leave-one-out cross-validation recovers it asymptotically.

Double descent is real and the math is beautiful, but the headline (more parameters fix overfitting) gets it backwards. The second descent just shows the minimum-norm interpolator picking up a hidden, growing dose of regularization as rises. You need not climb out to to enjoy it; a ridge penalty and a cross-validation loop dial it in directly, skipping the peak entirely. Less “interpolation is secretly fine” than “under-regularization is bad, and overparametrization happens to re-regularize you on the way out.”

When overparametrization genuinely helps

That said, the strong claim — that the risk’s global minimum can live deep in the overparametrized regime — is true, but only under structure worth knowing.

In the isotropic, well-specified case it essentially never pays: even when creates a local minimum past , the global minimum stays underparametrized, and the risk only approaches the null risk as . Overparametrization buys nothing you couldn’t get more cheaply.

Two things change the verdict.

Misspecification: if the truth has components your features cannot represent, adding features improves the approximation, and with enough signal the global minimum can cross the boundary.

Structure in the covariance: when aligns with the leading eigenvectors of — the “latent space” model, where each feature carries fresh information about a few latent drivers — the risk can fall monotonically across the whole overparametrized regime, bottoming out as . This is what large networks actually look like, and no accident: a linear model with a learned feature map is exactly the lazy-training (“neural tangent kernel”) linearization of an overparametrized network. Not an analogy — the same story.

The central subspace

The central subspace turns a vague wish (“reduce without losing information”) into a well-defined target. Under mild regularity conditions it exists, is unique, and satisfies the conditional-independence condition above. Estimating it, rather than any particular basis, is what frees SDR from needing a model: two regressions as different as

can share the same central subspace, and an SDR method estimates that common structure without knowing which regression generated the data. Once you have an estimated basis , plotting against , a sufficient summary plot, is often the single most informative diagnostic you can draw, because by construction it hides nothing relevant.

Sliced inverse regression

The foundational method, and still the first one to reach for, is sliced inverse regression (SIR), introduced by Li (1991). Its trick is to run the regression backwards. Modeling in high dimensions is hard; modeling is easy, because is typically one-dimensional. SIR studies how the mean of shifts as varies: it slices the range of into bins, computes the mean of within each slice, and asks which directions those slice-means move along.

Under a mild linearity condition on the distribution of , the centered inverse-regression curve

lies in the central subspace, up to a covariance rescaling. Estimating it reduces to a generalized eigenvalue problem: the leading eigenvectors of the between-slice covariance, relative to , span (a subset of) . The eigenvalues themselves suggest — a sharp drop tells you how many directions to keep.

Where SIR fails, and what comes after

SIR has one famous blind spot, and it is instructive. Because it only tracks how the mean of moves with , it is blind to dependence that shifts the variance while leaving the mean fixed.

The textbook example is with symmetric : the slice means of barely move, SIR sees nothing, and reports an empty direction. Sliced average variance estimation (SAVE, Cook and Weisberg 1991) was built for exactly this case — it examines the slice-wise covariance of and so detects symmetric, curved dependence that SIR misses. The tradeoff is that SAVE needs more data to work well and leans on a second regularity assumption, the constant-covariance condition.

That tension (capture more structure versus require weaker assumptions) organizes the rest of the field. Later methods (directional regression, minimum discrepancy approaches, and the semiparametric formulations of Ma and Zhu) chip away at the linearity and constant-covariance conditions, extend SDR to the conditional mean alone (the central mean subspace), or push into the high-dimensional regime with sparsity. For most applied work, though, SIR is the right starting point and SAVE the right thing to try when SIR comes up empty despite an obvious relationship in the data.

The scalar score becomes a density

With binary treatment, the propensity score is a single number in . Its magic is dimension reduction: Rosenbaum and Rubin showed that conditioning on that one scalar is enough to remove covariate bias, no matter how many covariates went into it.

When is continuous, “the probability of receiving treatment” is no longer a probability — it is a density. The natural object is the conditional density of the treatment given covariates,

and the generalized propensity score (GPS) is this density evaluated at the dose a unit actually received:

This is the one change that ripples through everything. The score is still a scalar random variable — but it is a scalar you get by plugging into a density, not by reading off a probability, and it now depends on which dose you are asking about.

What the GPS balances

The binary score’s defining property is balance: within strata of equal , treatment is independent of . The GPS inherits a version of this, but with a twist worth internalizing:

Read carefully: balance holds at each dose level separately, conditioning on the score evaluated at that same . You use as many propensity scores as there are levels of the treatment, yet you never use more than one at a time. This is the mental model shift.

Notably, this balancing property is mechanical — it falls out of the definition and does not itself require unconfoundedness. Combine it with weak unconfoundedness and you get the payoff: assignment is unconfounded given the GPS.

Recovering the curve

The estimation strategy is where the GPS earns its keep, and it is pleasingly concrete.

The step that trips people up is the third one. You do not average over , the score at each unit’s own realized dose. You average over , the score evaluated at the target dose for everyone. Getting this wrong silently biases the curve. Note also that has no causal meaning on its own — the regression coefficients are not effects. Only after the averaging step does become interpretable, which is a familiar theme from interpreting regression adjustments causally.

Multiple implementations

The GPS defined above is Hirano and Imbens’ route, and it is the cleanest place to build intuition. But the same core idea (reducing the covariates to a low-dimensional score summarizing treatment assignment) shows up in a few different shapes.

When you actually need it

The single most useful question is: does a downstream decision depend on the magnitude of the probability, or only on its order?

If you only need to rank — show the top- most likely churners, sort leads by score, compute AUC — calibration is irrelevant. Any monotonic transform of the score leaves the ranking, and hence AUC, unchanged. Recalibrating buys you nothing here and, with isotonic regression, can even hurt by introducing ties.

Calibration starts to matter the moment a probability feeds arithmetic. Expected-value decisions () need the to be right, not just ordered. Thresholding at a fixed operating point (act when ) assumes means something. Combining model outputs with other probabilities, feeding them to a Bayesian update, or reporting them to a human who will treat as “nearly certain” — all of these break under miscalibration. This is closely related to the distinction I drew between causal and predictive models: the use case dictates which properties of the output you are allowed to trust.

Diagnosing miscalibration

The workhorse diagnostic is the reliability diagram: bin predictions by score, and for each bin plot the mean predicted probability (-axis) against the observed fraction of positives (-axis). Perfect calibration lies on the line. Curves below the diagonal signal over-confidence; curves above it, under-confidence. It is one of the plots I think everyone should have in their back pocket.

The two panels below show the contrast. On the left, a well-calibrated model hugs the diagonal — its stated probabilities match observed frequencies. On the right, an over-confident model: it sits above the diagonal for low scores and below it for high ones, the telltale crossing pattern of a classifier whose probabilities are pushed too hard toward the extremes.

For a single number, use a proper scoring rule—a loss function that is minimized if and only if the predicted probability equals the true probability. The Brier score is the most intuitive: it is simply the mean squared error of the probabilities,

and log loss is its logarithmic cousin. But mind the trap: a low Brier score does not guarantee good calibration. According to Murphy’s decomposition, the Brier score is a sum of reliability (calibration), resolution (how much the predictions differ from the base rate), and an inherent uncertainty term.

A model that is “sharp” but miscalibrated—one that confidently predicts for a group that is actually positive—can still achieve a lower Brier score than a perfectly calibrated model that conservatively predicts for everyone. In other words, better discrimination can mask poor calibration in a single aggregate metric. To know if your probabilities are actually trustworthy, you cannot rely on the Brier score alone; pair it with a reliability diagram.

Miscalibration varies by model

Not all classifiers misbehave the same way. Logistic regression is typically well-calibrated out of the box: minimizing log loss with the canonical logit link enforces a balance property that ties predicted probabilities to observed frequencies.

Naive Bayes, on the other hand, is over-confident — its independence assumption compounds evidence it should not, pushing scores toward and .

Random forests and bagged ensembles, while extremely popular, are under-confident: averaging many trees pulls probabilities toward the middle, since it is rare for all trees to agree at the extremes, producing a telltale sigmoid reliability curve.

Max-margin methods like SVMs distort further still, because they optimize the decision boundary and never really estimate probabilities at all. Knowing the direction of a model’s bias tells you which fix will suit it.

Fixes

Calibration is a post-processing step: fit a monotonic map from raw scores to calibrated probabilities on held-out data. The two classic choices trade flexibility against data hunger.

The Kronecker Product

Given an matrix and a matrix , their Kronecker product is the block matrix you get by multiplying every entry of by the entire matrix :

A small example makes the pattern obvious:

Working with Kronecker products

Two algebraic facts do almost all the work in applications. The first is the mixed-product rule, valid whenever the ordinary products are conformable.

The second is its corollary for inverses, That second identity is the reason the operation is so beloved: inverting a giant matrix collapses into inverting its two small factors separately. Transposition behaves just as cleanly,

Where They Show Up

Why the “F > 10” rule broke

The rule of thumb traces to Staiger and Stock (1997), who suggested as a rough indicator that weak-instrument distortions were tolerable. Stock and Yogo (2005) made it rigorous by defining “weak” through worst-case performance and tabulating critical values under two distinct criteria.

• The bias criterion asks how large the worst-case 2SLS bias is relative to OLS; its critical values sit near across a range of instrument counts, which is where the folklore comes from.
• The size criterion asks how badly a nominal 5% -test can over-reject; its critical values are a different animal, rising from about with one instrument to nearly with thirty. Already the single number “10” is doing two incompatible jobs.

The deeper problem is that the entire Stock–Yogo apparatus was derived under homoskedasticity. The critical values depend on a Kronecker-product structure in the covariance matrix that simply does not hold once errors are heteroskedastic, clustered, or serially correlated — which is to say, in essentially every applied setting.

The effective F-statistic

The fix for measuring strength is due to Montiel Olea and Pflueger (2013), who introduced what they call the effective -statistic. The conventional first-stage and even the naive robust Wald have, as Andrews, Stock, and Sun (2019) put it, “no theoretical justification” for gauging instrument strength under heteroskedasticity — they target the wrong population object or misstate its variance.

The effective replaces the variance term with a heteroskedasticity-robust analogue:

where is the instrument second-moment matrix and the robust variance of . It collapses to the usual under homoskedasticity and, crucially, “measures the right object and gets the standard errors right on average.” In the just-identified case it coincides with the robust first-stage .

The practical instruction is simple: whenever you report a first-stage as a strength diagnostic, it should be the effective one, compared against the appropriate Montiel Olea–Pflueger critical values rather than a remembered “10.”

The Anderson–Rubin test

Measuring strength is only half the battle; the harder question is what to report for when strength is in doubt. The oldest and most robust answer is the Anderson–Rubin (AR) test, dating to 1949. Instead of forming , AR inverts a test. To test , construct the residual and ask whether it is correlated with the instrument. Under the null,

and the coefficient on is , which equals zero precisely when is the true (since ). So the AR statistic is just the squared -statistic for in the regression of on , and under it is distributed regardless of the value of — even at . That is the whole trick: testing whether a coefficient is zero in a clean regression does not care how strong the instrument is.

The confidence set is the collection of nulls the test fails to reject,

The key point is that AR’s occasional unbounded intervals are not a defect: under very weak instruments, any honest procedure must sometimes admit that the data cannot pin down, while AR still works well when identification is strong.

Fix 1: abandon the -test

The Annual Review tradition — Andrews, Stock, and Sun (2019) as the canonical statement — draws the natural conclusion: in the just-identified case, report AR intervals, and that’s it. Keane and Neal (2024) push this to its sharp edge and argue for abandoning the 2SLS -test even when instruments are strong. Their argument is a power asymmetry that the older literature, fixated on bias and size, had missed.

The mechanism is the 2SLS standard error itself. The estimated structural variance entering is a quadratic function of that is minimized at the OLS estimate. So whenever a draw of lands near OLS, its standard error is artificially small, and the -test is correspondingly eager to declare significance. The upshot is a test with inflated power to detect false positives in the direction of the OLS bias and almost no power to detect true effects lying away from OLS. Keane and Neal show this distortion persists at first-stage values of , , even — far above any conventional threshold — and conclude that the bar for the should be raised to roughly , and that one should simply use AR throughout. The AR standard error, by contrast, is built around a variance minimized at rather than at OLS, which is exactly what kills the asymmetry.

Fix 2: fix the -test instead

Lee, McCrary, Moreira, and Porter (2022) accept the diagnosis and reject the prescription. Their objection is practical: practitioners know and trust the -ratio, the entire reporting apparatus is built around it, and throwing it out is a heavy ask. The -statistic is not normal under weak instruments — true — but its non-normal distribution is known, and depends on the data only through the observed first-stage . So rather than discard it, replace the constant with a critical value that is a smooth, decreasing function of the first stage. They call this the tF procedure.

The headline number is sobering. For the constant to deliver a genuine 5% test, you do not need , or even Stock and Yogo’s — you need

Below that, is too small and you must inflate.

Operationally the procedure is a lookup: estimate your usual (robust, clustered) 2SLS standard error, read the adjustment factor off their Table 3 at your observed , and multiply. A few anchor points at the 5% level make the magnitude vivid:

First-stage	critical value	SE multiplier

At the celebrated , an honest 95% interval is about wider than the one most papers report, and can still be shorter than AR when both intervals are bounded because it reserves its “we cannot learn ” verdict for the rare cases where .

Three things people call “sequential”

Untangling the vocabulary is half the battle, so let me be explicit about three ideas that are routinely conflated.

Classical sequential testing concerns a single hypothesis whose data accumulate over time, with the sample size not fixed in advance. Wald’s sequential probability ratio test is the canonical example: keep sampling until the evidence is decisive, then stop. The hard part is that naive optional stopping destroys the type I error guarantee—if you re-test after every new observation and stop the first time you see , you will eventually cross that line even when the null is true.

Always-valid p-values are the modern device that fixes optional stopping. They are the inner process: a single experiment, monitored continuously, with a validity guarantee that survives stopping at any time for any reason.

Online multiple testing is a different problem entirely. Here a stream of distinct hypotheses arrives over time, each tested once, and the goal is to control a false discovery criterion across the whole stream while deciding each case in real time. This is the outer process.

Robertson, Wason and Ramdas call this the inner/outer framing: each monitored experiment emits one valid -value, and the outer online-testing procedure controls error rates across the resulting stream.

Always-valid -values: the inner process

Peeking breaks a fixed- -value because validity applies to one pre-committed look, not to the event “ at some point along the way.” Repeated looks turn that into a union of events, whose null probability can be much larger than .

An always-valid -value is constructed precisely so that the union is controlled:

The usual construction runs through martingales and e-values. Ville’s inequality bounds the probability that the evidence process ever crosses a threshold, which gives the optional-stopping guarantee. The confidence-interval analog is a confidence sequence: intervals that cover the true parameter simultaneously at all sample sizes with probability .

The payoff is continuous monitoring with valid inference: you can stop when the evidence is decisive and still control type I error. The cost is conservatism. Under the null, always-valid -values are typically stochastically larger than uniform, a fact ADDIS later turns into an advantage.

Error rates for the outer process

Now zoom out to the stream. Which error rate should an online procedure control? The false discovery proportion up to time is

and the headline quantity, the false discovery rate, is its expectation,

The FDR is the right default for most applied work: it has a long track record in genetics, an intuitive reading as the expected share of discoveries that are wrong, and it scales gracefully to large streams.

Two variants show up constantly. The marginal FDR, replaces the expectation of a ratio with a ratio of expectations. It is not identical to the FDR, but it is far more tractable, and many online algorithms can only be proven to control the mFDR. Treat it as a pragmatic stand-in when an FDR proof is unavailable for your setting. The false discovery exceedance goes the other way and controls a tail: This is the right tool when the FDP can swing far from its mean—few hypotheses, or heavy dependence—and you want a guarantee about the realized proportion, not just its average. Finally, the familywise error rate, the probability of any false rejection, remains the standard in confirmatory clinical trials where regulators demand it.

The defining constraint of the online setting is informational: when deciding on , you know only the past rejections, not the future and not even the eventual number of tests. The crudest response is alpha-spending—a Bonferroni-type split with . It controls the FWER, and hence the FDR, but at a ruinous price: the levels must shrink toward zero, so the power to reject collapses as grows. A procedure that becomes blind to discoveries simply because they arrive late is not viable. This is what motivates everything that follows.

Alpha-investing

The breakthrough idea, originating with Foster and Stine (2008) and generalized by Aharoni and Rosset (2014), reframes testing as the management of an error budget called alpha-wealth. You start with wealth . Testing a hypothesis costs you some wealth —an investment. Making a discovery pays you back a reward . The wealth evolves as and must stay nonnegative, which constrains how aggressively you can test ().

The intuition for why a reward on rejection is legitimate—rather than a cheat—is worth internalizing. Look back at

Each genuine rejection enlarges the denominator of the FDP, buying room for continued testing without the power collapse of alpha-spending. This family is known as generalized alpha-investing (GAI); GAI++ (Ramdas et al., 2017) refines the rewards to guarantee FDR control, whereas earlier rules controlled only the mFDR.

LORD, SAFFRON, ADDIS

The wealth metaphor is intuitive but it was a “statistical” reframing by Ramdas et al. (2017) that produced the algorithms practitioners actually use. The idea is to maintain a running overestimate of the FDP and spend wealth to keep that estimate below —exactly the logic that powers Benjamini–Hochberg offline.

LORD (Javanmard and Montanari, 2018), and its uniform improvement LORD++, set each test level from a fixed, non-increasing sequence that sums to one, allocating a slice of the initial wealth plus a slice of the reward earned at each past rejection : Daunting at a glance, but readable: the first term spends a fraction of the starting budget, and every prior rejection injects fresh budget that is then doled out over future tests on the same schedule . LORD++ never spends more than it has earned, which is exactly why it keeps the FDP estimate below . It is best understood as the online analog of Benjamini–Hochberg, and under independence (more precisely, when the null -values are conditionally super-uniform) it controls the FDR; Chen and Arias-Castro (2021) further show it is asymptotically as powerful as BH in a Gaussian model.

SAFFRON (Ramdas et al., 2018) makes LORD++ adaptive. It picks a threshold and refuses to spend wealth on “candidate” -values larger than —since a large -value was never going to be rejected anyway, why pay to test it? By estimating the proportion of true nulls and conserving wealth accordingly, SAFFRON delivers more power than LORD++ whenever a meaningful fraction of hypotheses are non-null with strong signals.

ADDIS (Tian and Ramdas, 2019) adds discarding on top of adaptivity. It explicitly throws away the most conservative nulls—the largest -values—before testing. Here is where the earlier observation pays off: always-valid -values are conservative under the null. A procedure that exploits conservative nulls is therefore a natural partner for an inner process built on continuous monitoring. The two layers of the inner/outer framing don’t just compose—they reinforce each other.

Why cannot be identified

Here is the crux. Every row of lives in by definition, which means annihilates anything orthogonal to the row space: . Multiply the model through and the consequence is immediate —

The component contributes nothing to , and therefore nothing to . The data are generated by alone. Whatever value takes — zero, enormous, anything — the distribution of is exactly the same. No estimator can distinguish between two coefficient vectors that agree on their row-space part and differ off it, because they produce statistically identical data.

When and has full column rank, the row space is all of , the projection is the identity, vanishes, and this issue never arises — which is why we rarely think about it. But the moment , the row space is a proper subspace of , the orthogonal complement is non-trivial, and a whole -dimensional flat of coefficient vectors fits the data equally well. That is non-identifiability, stated geometrically: is unknowable precisely along the directions the design never probes.

Non-unique coefficients, unique predictions

Now the payoff. The fitted-value vector is

and this object is unique even though is not. Geometrically, fitting projects onto the column space ; that projection is a single, well-defined point regardless of which of the infinitely many least-squares solutions you happened to compute. The same is true of the row-space part of the coefficients: is identifiable and estimable, while is not. So the situation is not “everything is lost,” but rather a clean partition — the row-space shadow of is recoverable, its orthogonal complement is forever hidden, and predictions live entirely in the recoverable part.

This is why high-dimensional regression is not the doomed exercise the equation-counting heuristic suggests. The coefficients are a fiction the data cannot fully resolve, but the model’s predictions are perfectly real. It is also a useful corrective to a habit many of us have — reading individual coefficients as if they meant something. With , an individual is whatever your estimator’s tie-breaking rule decided to make it; only its projection onto the row space carries information. (This tension between estimating coefficients and estimating predictions shows up even in well-posed problems; I have written about it in the difference between causal and predictive models.)

Four ways to pick a

If infinitely many coefficient vectors fit, every estimator is really a rule for choosing one — equivalently, a rule for deciding what to do off the row space, where the data are silent. Christensen lays out four that practitioners actually use, and the unifying thread is that they are all functions of the ordinary least-squares fit; they differ only in how they pin down the ambiguous directions.

1. Minimum-norm least squares. Among all least-squares solutions, take the one with the smallest . This is exactly — the solution that sets the orthogonal component to zero and lives entirely in the row space . It is unique, it is what gradient descent from the origin converges to, and it is the natural reference point for the others.

2. Penalized (regularized) least squares. Minimize for a penalty — ridge (), the LASSO (), elastic net, and so on. The penalty breaks the tie by preferring small or sparse coefficients, which generally makes the estimate unique.

3. Spectral shrinkage. Working from the singular value decomposition of , shrink the components of the fit associated with each eigen-direction by a chosen factor. Ridge regression is the canonical special case (it shrinks the direction with eigenvalue by ), and as ridge converges to the minimum-norm solution.

4. Principal component regression. Keep only the leading principal components of and regress on those, discarding the low-variance directions entirely. This is a hard-thresholding cousin of spectral shrinkage — zero weight below the cutoff, full weight above.

The differences among them are real but secondary to the shared structure: each one estimates the identifiable part from the data and then imposes some discipline on the unidentifiable part. For the theory behind when these choices recover the truth, see my post on the LASSO’s theoretical guarantees.

Two Nuisances, Three Estimators

The reason TMLE is easy to place on a mental map is that it shares its ingredients with methods you already use. G-computation (regression adjustment) models only the outcome regression and then averages the predicted difference. Inverse probability weighting (IPW) models only the propensity score and reweights the observed outcomes. Each leans entirely on getting one of the two nuisances right.

TMLE uses both. It starts from an outcome-regression prediction and then nudges it using the propensity score, in a way designed specifically for the ATE. That extra step is what buys double robustness: the final estimate is consistent if either or is correctly estimated, not necessarily both. If both are consistent, the estimator is also efficient. This is the practical payoff, and it is why the method is worth the extra bookkeeping.

The Recipe: Four Steps

Here is the whole procedure for the ATE, stripped to its essentials. One technical preliminary: because the targeting step runs through a logistic regression, the continuous outcome is first rescaled to the unit interval , and everything is undone at the end.

The only genuinely new object is the clever covariate . Note that it looks a lot like an inverse-probability weight — and that is no accident — but here it enters as a regressor in the fluctuation rather than as a weight. Step 3 is the “targeting”: it tilts the initial fit just enough to remove first-order bias in the direction that matters for the ATE, while leaving the rest of the outcome model alone.

Why Bother With the Targeting Step?

A fair question is why we do not simply stop after step 1 and report the average difference in predicted outcomes — that is exactly G-computation. The answer is robustness. A plain outcome-model plug-in is only as good as that one model; if it is misspecified, the estimate is biased, full stop. The targeting step injects information from the propensity score so that a mistake in one of the two models can be absorbed by the other. In the paper’s simulation, when either the outcome or the exposure model is deliberately misspecified, TMLE stays essentially unbiased, while G-computation and IPW pick up large bias from the same misspecification.

Two practical notes follow from this. First, TMLE is a substitution (plug-in) estimator — the ATE is computed by plugging fitted values into the target functional — which makes it more stable than estimating-equation methods like IPW when propensity scores get close to or . Second, because the recipe never cares how and are estimated, you should estimate them with flexible machine learning rather than hand-specified regressions. In practice this means an ensemble (the super learner), which lets the data choose among a library of ML methods instead of betting on one functional form.

Use Case #1: Noncompliance

In a trial where everyone complies, , and the difference in mean outcomes by assignment is the average treatment effect. Noncompliance breaks this.

Suppose treatment effects are constant and equal to , so that

Then a little algebra on the assignment-based comparison gives

The intuition is that the ITT effect gets diluted: when only a fraction of those assigned actually take treatment, the assignment-versus-assignment comparison mixes in untreated people on the treatment side and crossovers on the control side, shrinking the apparent effect toward zero. If compliance is, say, , the ITT understates the true effect by more than half.

The fix falls right out of the identity. Dividing the reduced form by the first stage recovers the effect:

This ratio of differences in means is the heart of IV analysis — the Wald estimator. When effects are heterogeneous, the LATE theorem (Imbens and Angrist, 1994) tells us this same ratio identifies not some universal but the average effect for compliers:

Use Case #2: Selection Bias

The second problem IVs solve is subtler, and it is where practitioners most often go wrong. Faced with noncompliance, the tempting move is a per-protocol or as-treated analysis: just compare people by the treatment they actually received, . The trouble is that is not randomized. The people who comply, cross over, or refuse differ systematically, so comparing by treatment received reintroduces exactly the confounding that randomization was designed to eliminate.

The IV approach sidesteps this entirely by comparing groups defined by randomized assignment , not by . The comparison is apples-to-apples — the complier subpopulation on the treated side is statistically identical to the complier subpopulation on the control side, differing only in the luck of the draw. The estimate is a per-protocol effect, but one purged of selection bias. As a bonus, although individual compliers cannot be identified, their average baseline characteristics can be computed, letting the analyst check how representative they are of the broader patient population.

Prediction Error

This is the most basic guarantee. It says the Lasso will predict “well enough” — the in-sample prediction error will not exceed a certain bound.

Formally, with the penalty chosen at the canonical rate , one can show that with high probability

This is the so-called slow rate.

What is remarkable is how little it requires: essentially no conditions on the design matrix beyond standardizing the columns. All you need is for to be large enough to dominate the noise — which is exactly what the scaling ensures. The term is the price of searching over candidate predictors, and it enters only logarithmically, which is why the Lasso tolerates enormous .

If you are willing to assume more about the design (see the next section), the bound sharpens to the fast rate,

which scales with the number of true signals rather than with .

Estimation Error

Here the question shifts from predictions to the coefficients themselves — how close is to ? This is a harder problem, and it requires a condition on the design matrix.

The standard assumption is the restricted eigenvalue (RE) condition. Loosely, it requires that behave like a well-conditioned matrix — bounded away from singularity — but only over the directions that matter, namely vectors that are approximately sparse. We cannot ask to be invertible outright, since makes that impossible; the restriction to a sparse cone is what rescues the argument.

Under the RE condition with constant , and again ,

The intuition is that strong correlations among predictors flatten the objective function in certain directions, making the coefficients poorly identified even when predictions are fine. The RE condition rules out the worst of this. When it holds, the error decays at essentially the rate an oracle who knew would achieve, up to the factor.

You know to never take the coefficients — or the implied -values — from a Lasso fit too seriously without an extreme degree of caution. The shrinkage that makes the Lasso work also biases every non-zero coefficient toward zero, and standard inference ignores the selection step entirely. I have written about hypothesis testing in linear ML models elsewhere on the blog; the estimation bounds above are about rates, not about valid confidence intervals.

Support Recovery

This is the most intriguing guarantee and, fittingly, the most expensive. It is relevant for model selection rather than prediction: it asks whether the Lasso picks out exactly the right set of variables, , where .

The goal is selection consistency,

Two strong conditions are typically needed. The first is the irrepresentable condition, which limits how strongly the irrelevant predictors () can be correlated with the relevant ones ():

for some . This is close to necessary, not merely sufficient — if it fails, the Lasso tends to bring spurious variables into the model. The second is a beta-min condition: the smallest non-zero coefficient must be detectably large, . Signals weaker than the noise floor simply cannot be distinguished from zero.

The practical implication is sobering. Exact support recovery is fragile: it demands that the truly important variables be only mildly correlated with the unimportant ones, and that no real signal be too small. Both are easy to violate with observational data. This is also why I treated this topic separately in my post on the oracle property — standard Lasso generally does not achieve oracle-style selection, and the fixes (adaptive Lasso, nonconvex penalties like SCAD and MCP) exist precisely to relax the irrepresentable condition.

The First-Generation P-Value

Start with the object everyone already knows. We observe a dataset, and the -value is the probability of seeing data at least as extreme — weirder, loosely speaking — than what we actually observed, if the null hypothesis were true.

A large -value means the data are unsurprising under the null and therefore offer no evidence against it. Famously, this is not the probability that the null is true; getting at that requires a Bayesian apparatus and a prior. This single number has carried an enormous load for nearly a century, and that is precisely the source of trouble.

The Problem

The classical -value conflates two things a practitioner usually wants to keep separate: the size of an effect and the precision with which it is estimated. A tiny, scientifically meaningless effect will produce an arbitrarily small -value once the sample is large enough, because precision alone drives significance. In the era of big data this is not a hypothetical — it is the default failure mode.

The deeper issue is the point null itself. The hypothesis exactly is almost never true and rarely even interesting. Two treatments are essentially never identical to infinite precision; the relevant question is whether they differ by an amount that matters. Statistical significance, as classically defined, simply does not speak to that question.

The Main Idea

The SGPV, which I will denote , replaces the point null with the interval null and then measures how much the interval estimate overlaps it. It is a proportion, not a tail-area probability:

where is the length of the overlap between the two intervals. When the interval estimate is reasonably precise (specifically, ), the correction term is and the formula reduces to the clean fraction — the share of the interval estimate that falls inside the null zone. The correction factor only kicks in when is very wide relative to the null, capping the statistic to reflect that the data are simply too imprecise to say much.

The interpretation is direct:

• : the interval estimate does not overlap the null at all. The data are incompatible with a trivial effect and support a scientifically meaningful one.
• : the interval estimate lies entirely inside the null. The data support only the null premise — something a classical -value can never tell you.
• : the data are inconclusive, with being the maximally inconclusive case.

Quick Example

A quick worked example from the paper makes it concrete. In a study of 100 smokers and 100 non-smokers, suppose 65 smokers and 50 non-smokers develop lung cancer. The odds ratio is with a 95% confidence interval of . Take the interval null to be odds ratios between and — associations too weak to care about. The interval estimate overlaps the null only slightly, and the SGPV works out to : suggestive, but not conclusive. Bump the smokers’ cancer count from 65 to 70 and the odds ratio rises to with interval , which no longer touches the null. Now , and we would report a scientifically meaningful association.

Pros and Cons

The appeal is that the SGPV bakes scientific relevance into the inference itself, rather than leaving it as an afterthought to be eyeballed from a confidence interval. By construction it can support the null, it indicates when the data are inconclusive instead of forcing a verdict, and — a genuinely attractive frequency property — its Type I error rate shrinks toward zero as the sample grows, instead of hovering at forever. That directly attacks the big-data significance problem and tends to yield lower false discovery rates.

The interval null must be specified in advance, and pinning down the width of “scientifically trivial” is a real judgment call that invites disagreement and can feel arbitrary. The statistic is a proportion rather than a probability, which makes it unfamiliar and harder to slot into existing reporting conventions. And it competes in a crowded field: practitioners who care about false discoveries already reach for false discovery rate (FDR) or family-wise error rate (FWER) procedures.

Q-Q Plot

A Q-Q plot compares sample quantiles to theoretical quantiles from a reference distribution. In practice that reference is usually the normal distribution, which makes the plot a fast diagnostic for residual checks and distributional shape. If the points line up, the sample is broadly consistent with the reference. If they bend away from the line, that tells you where the mismatch lives: skewness shows up as asymmetric curvature, while heavy tails pull the extremes away from the line. One can also use Q-Q plots to compare two empirical distributions, but I’d argue there are better ways to do that.

What I like about Q-Q plots is that they force you to think about where a distribution departs from a model, not just whether a normality test rejects. The downside is that they are easy to overread in small samples and less useful if you do not have a meaningful reference distribution in mind. Unlike traditional statistical tests, Q-Q plots do not spit out a -value, so you have to interpret the plot yourself.

ggplot(iris, aes(sample = Sepal.Length)) +
  stat_qq(color = "#66c2a5", size = 2) +
  stat_qq_line(color = "black", linewidth = 0.8) +
  theme_minimal() +
  labs(
    title = "Q-Q Plot of Sepal Length",
    x = "Theoretical Quantiles",
    y = "Sample Quantiles"
  )

stats.probplot(iris["sepal_length"], dist="norm", plot=plt)
plt.title("Q-Q Plot of Sepal Length")
plt.xlabel("Theoretical Quantiles")
plt.ylabel("Sample Quantiles")
plt.show()

Violin Plot

A violin plot combines a boxplot with a smoothed (symmetric) density estimate. That makes it useful when a plain boxplot feels too compressed. Two groups can have similar medians and quartiles but very different shapes, and a violin plot makes that visible immediately. In the iris data, it is a quick way to see that species differ not only in central tendency but in how concentrated or dispersed their sepal lengths are.

The main drawback is that the density is smoothed, so small samples can look more structured than they really are. It can also be sensitive to the smoothing parameters (bandwidth more than kernel type). Violins also become noisy if you cram in too many categories. Still, when I want a compact distribution comparison across a handful of groups, violin plots are often a strict upgrade over boxplots.

ggplot(iris, aes(x = Species, y = Sepal.Length, fill = Species)) +
  geom_violin(trim = FALSE) +
  geom_boxplot(width = 0.12, fill = "white", outlier.shape = NA) +
  theme_minimal() +
  labs(title = "Violin Plot of Sepal Length by Species")

sns.violinplot(data=iris, x="species", y="sepal_length", inner="box")
plt.title("Violin Plot of Sepal Length by Species")
plt.xlabel("")
plt.ylabel("Sepal Length")
plt.show()

ECDF Plot

The empirical cumulative distribution function shows the share of observations less than or equal to a given value. That sounds modest, but it is one of the cleanest ways to compare distributions because it avoids arbitrary bin choices and displays the full sample directly. When one ECDF sits to the right of another, you can read that as a first-order stochastic dominance story, at least visually.

The ECDF is defined as

Do you remember that the PDF is the derivative of the CDF? Yes, CDF is really central to probability theory and understanding any variable at hand. In microeconomic theory classes, ECDFs are used to establish stochastic dominance relationships. I like ECDFs because they are honest. They show every observation’s contribution to the distribution without smoothing it away. The tradeoff is that they are less familiar to many audiences and can look busy when too many groups are overlaid. For side-by-side distribution comparison, though, they are hard to beat.

ggplot(iris, aes(Sepal.Length, color = Species)) +
  stat_ecdf(linewidth = 1) +
  theme_minimal() +
  labs(
    title = "ECDF of Sepal Length by Species",
    x = "Sepal Length",
    y = "Empirical CDF"
  )

sns.ecdfplot(data=iris, x="sepal_length", hue="species")
plt.title("ECDF of Sepal Length by Species")
plt.xlabel("Sepal Length")
plt.ylabel("Empirical CDF")
plt.show()

Ridgeline Plot

Ridgeline plots stack several density curves vertically, which makes them especially useful when you want to compare many related distributions at once. The variables, however, need to be on more-or-less the same scale for the plot to make sense. They are common in cohort analysis and time-based comparisons, but they also work well for grouped exploratory analysis like the species differences in iris.

Their advantage is compactness: you can compare several distributions without the visual clutter of heavy overlap. Their weakness is that they are still density plots, so the same caution about smoothing applies. I use ridgelines when I want a plot that is more expressive than small multiples but less chaotic than overlaying five or six densities in one panel.

ggplot(iris, aes(x = Sepal.Length, y = Species, fill = Species)) +
  geom_density_ridges(alpha = 0.7, color = "white") +
  theme_ridges() +
  labs(
    title = "Ridgeline Plot of Sepal Length by Species",
    x = "Sepal Length",
    y = NULL
  )

species_order = ["setosa", "versicolor", "virginica"]
x_grid = np.linspace(iris["sepal_length"].min() - 0.3,
                     iris["sepal_length"].max() + 0.3, 300)
offsets = [0.0, 1.0, 2.0]

fig, ax = plt.subplots(figsize=(7, 5))
for offset, species in zip(offsets, species_order):
    subset = iris.loc[iris["species"] == species, "sepal_length"]
    kde = stats.gaussian_kde(subset)
    density = kde(x_grid)
    density = density / density.max() * 0.8
    ax.fill_between(x_grid, offset, offset + density, alpha=0.7)
    ax.plot(x_grid, offset + density, color="black", linewidth=0.8)
    ax.text(x_grid.min() - 0.02, offset + 0.12, species, ha="right")

ax.set_title("Ridgeline Plot of Sepal Length by Species")
ax.set_xlabel("Sepal Length")
ax.set_yticks([])
plt.show()

Hexbin Plot

Scatterplots are great until they are not. Have you tried a scatterplot with a million points? It’s slow and it’s hard to see anything. Once the sample gets large enough, overplotting hides the very structure you want to see. Hexbin plots solve that by aggregating points into small hexagonal cells and coloring those cells by count. You give up the exact point cloud, but in return you get a much clearer view of where the data are concentrated.

The iris data are too small to truly need a hexbin, which is worth saying out loud. But the plot still illustrates the logic well. On genuinely large datasets, this is often the right substitute for a scatterplot, as is the binscatter I have written about elsewhere. The cost is that rare points and local outliers become less visible, so it is better for density structure than for point-level inspection.

ggplot(iris, aes(Sepal.Length, Petal.Length)) +
  geom_hex() +
  theme_minimal() +
  labs(
    title = "Hexbin Plot of Sepal vs Petal Length",
    x = "Sepal Length",
    y = "Petal Length"
  )

plt.hexbin(
    iris["sepal_length"],
    iris["petal_length"],
    gridsize=14,
    cmap="YlOrRd"
)
plt.xlabel("Sepal Length")
plt.ylabel("Petal Length")
plt.title("Hexbin Plot of Sepal vs Petal Length")
plt.colorbar(label="Count")
plt.show()

Corrgram

A corrgram turns a correlation matrix into something you can actually read. Before fitting a regression, building a clustering pipeline, or running PCA, I almost always want to know which variables are moving together and which are largely independent. A corrgram gives that answer in a single glance.

The upside is speed: strong blocks, redundancies, and likely multicollinearity jump out immediately. The downside is that (Pearson) correlation is a blunt summary. It only captures linear association, ignores conditional relationships, and can be badly distorted by outliers. Presumably, one can move away from Pearson correlation and do the same plot with other correlation measures. Corrgrams also don’t work well with too many variables. So I treat corrgrams as a screening device, not as evidence of mechanism. Used that way, they are extremely effective.

corr_matrix <- cor(iris[, 1:4])

corrplot(
  corr_matrix,
  method = "color",
  type = "upper",
  tl.col = "black",
  tl.srt = 45
)

corr = iris.drop(columns=["species", "target"]).corr()

sns.heatmap(corr, annot=True, cmap="RdBu_r", center=0)
plt.title("Correlation Matrix (Corrgram)")
plt.show()

In the iris data, the corrgram immediately tells you that petal length and petal width are carrying very similar information. That is exactly the kind of thing you want to know before moving on to feature engineering, PCA, or a predictive model.

Classical PCA

Classical PCA is the benchmark because its optimization problem is clean and its geometry is transparent. The first component is the unit vector that captures the most sample variance; the second is the best such vector orthogonal to the first; and so on. If the singular values of are , then the proportion of variance explained by the first components is

In practice, two issues matter more than the derivation. First, PCA is extremely sensitive to scaling. If one variable is measured in dollars and another in percentages, the dollar variable may dominate the first component unless the data are standardized. Second, variance is not the same thing as signal. A noisy feature with large variance can easily drive the first component. I treat classical PCA as a compression tool, not as an automatic discovery engine.

library(stats)

set.seed(1988)
n <- 300
p <- 8

# Simulate a low-rank signal with two latent factors
latent_factors <- matrix(rnorm(n * 2), n, 2)
loadings_true <- matrix(c(
  0.9,  0.0,
  0.8,  0.1,
  0.7, -0.1,
  0.6,  0.2,
  0.0,  0.8,
  0.1,  0.7,
 -0.1,  0.6,
  0.2,  0.5
), nrow = p, byrow = TRUE)

X <- latent_factors %*% t(loadings_true) + matrix(rnorm(n * p, sd = 0.3), n, p)
colnames(X) <- paste0("feature_", 1:p)

# Standardize before PCA because variables may be on different scales
pca_fit <- prcomp(X, center = TRUE, scale. = TRUE)

# Explained variance ratio
explained_var <- pca_fit$sdev^2 / sum(pca_fit$sdev^2)
round(explained_var[1:4], 3)

# First two loading vectors
round(pca_fit$rotation[, 1:2], 3)

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

np.random.seed(1988)
n = 300
p = 8

latent_factors = np.random.randn(n, 2)
loadings_true = np.array([
    [0.9,  0.0],
    [0.8,  0.1],
    [0.7, -0.1],
    [0.6,  0.2],
    [0.0,  0.8],
    [0.1,  0.7],
    [-0.1, 0.6],
    [0.2,  0.5],
])

X = latent_factors @ loadings_true.T + np.random.randn(n, p) * 0.3

scaler = StandardScaler()
X_std = scaler.fit_transform(X)

pca = PCA(n_components=4)
pca.fit(X_std)

print("Explained variance ratio:", np.round(pca.explained_variance_ratio_, 3))
print("First two loading vectors:\n", np.round(pca.components_[:2].T, 3))

Sparse PCA

Sparse PCA modifies the loading vectors so that many coordinates are exactly zero. A convenient way to write the idea is

or, equivalently, with an penalty on the loadings. The point is not to improve the mathematics of PCA. The point is to make the components readable.

This matters when is large and the classical loading vectors spread small weight across almost every variable. In genomics, marketing, or text applications, that is often useless from a substantive perspective. Sparse PCA forces each component to be built from a smaller set of variables. The tradeoff is that you lose some variance explained, orthogonality becomes less clean, and the components can be more sensitive to tuning choices. In practice, I reach for Sparse PCA when interpretation matters at least as much as compression.

# install.packages("elasticnet")
library(elasticnet)

set.seed(1988)
n <- 200
p <- 12

latent_factor <- rnorm(n)
X <- cbind(
  latent_factor + rnorm(n, sd = 0.2),
  0.9 * latent_factor + rnorm(n, sd = 0.2),
  0.8 * latent_factor + rnorm(n, sd = 0.2),
  0.7 * latent_factor + rnorm(n, sd = 0.2),
  matrix(rnorm(n * (p - 4)), n, p - 4)
)

X <- scale(X)

# Ask for two sparse components with at most 4 nonzero loadings each
spca_fit <- spca(X, K = 2, type = "predictor", sparse = "varnum", para = c(4, 4))

round(spca_fit$loadings[, 1:2], 3)

import numpy as np
from sklearn.decomposition import SparsePCA
from sklearn.preprocessing import StandardScaler

np.random.seed(1988)
n = 200
p = 12

latent_factor = np.random.randn(n)
X = np.column_stack([
    latent_factor + np.random.randn(n) * 0.2,
    0.9 * latent_factor + np.random.randn(n) * 0.2,
    0.8 * latent_factor + np.random.randn(n) * 0.2,
    0.7 * latent_factor + np.random.randn(n) * 0.2,
    np.random.randn(n, p - 4)
])

X_std = StandardScaler().fit_transform(X)

spca = SparsePCA(n_components=2, alpha=1.0, random_state=1988)
spca.fit(X_std)

print("Sparse loadings:\n", np.round(spca.components_.T, 3))

Kernel PCA

Kernel PCA keeps the variance-maximization logic but applies it in a nonlinear feature space. Instead of diagonalizing the covariance matrix of , we diagonalize a centered kernel matrix

where might be a radial basis function kernel or a polynomial kernel. PCA is then performed on the centered version of rather than on the original variables.

This is useful when the data lie on a curved manifold rather than near a linear subspace. The classic example is concentric circles: ordinary PCA sees almost no useful low-dimensional linear structure, while Kernel PCA can often unfold the geometry. The price is interpretability. Classical PCA gives loading vectors in the original variables; Kernel PCA gives components in an implicit feature space. In practice, that makes it more of a nonlinear embedding method than a variable-summary tool. It is also sensitive to kernel choice and scale, so I do not treat it as a push-button replacement for standard PCA.

# install.packages("kernlab")
library(kernlab)

set.seed(1988)
n <- 300
angles <- runif(n, 0, 2 * pi)
radius <- rep(c(1, 2), each = n / 2) + rnorm(n, sd = 0.05)

X_circle <- cbind(
  radius * cos(angles),
  radius * sin(angles)
)

kpca_fit <- kpca(
  x = X_circle,
  kernel = "rbfdot",
  kpar = list(sigma = 5),
  features = 2
)

head(rotated(kpca_fit))

import numpy as np
from sklearn.decomposition import KernelPCA

np.random.seed(1988)
n = 300
angles = np.random.uniform(0, 2 * np.pi, size=n)
radius = np.repeat([1.0, 2.0], repeats=n // 2) + np.random.randn(n) * 0.05

X_circle = np.column_stack([
    radius * np.cos(angles),
    radius * np.sin(angles),
])

kpca = KernelPCA(n_components=2, kernel="rbf", gamma=5)
X_embedded = kpca.fit_transform(X_circle)

print(np.round(X_embedded[:5], 3))

Probabilistic PCA

Probabilistic PCA (PPCA) replaces the deterministic projection view with a latent variable model:

Here is a -dimensional latent factor and is isotropic Gaussian noise. Under maximum likelihood, the estimated subspace coincides with classical PCA in a particular limit, but the formulation buys you something important: a likelihood, uncertainty quantification, and a principled way to deal with missing values.

That makes PPCA attractive when PCA is part of a generative modeling workflow rather than just a preprocessing step. I especially like it when the data matrix has moderate missingness and I do not want to impute first and hope for the best. The main caveat is the isotropic-noise assumption. If feature-specific noise levels differ substantially, PPCA can be too restrictive and factor analysis may be the better model.

# install.packages("pcaMethods")
library(pcaMethods)

set.seed(1988)
n <- 150
p <- 6

latent_factors <- matrix(rnorm(n * 2), n, 2)
loadings_true <- matrix(rnorm(p * 2), p, 2)
X <- latent_factors %*% t(loadings_true) + matrix(rnorm(n * p, sd = 0.2), n, p)

# Introduce missing values
missing_index <- sample(length(X), size = 0.1 * length(X))
X[missing_index] <- NA

ppca_fit <- pca(X, method = "ppca", nPcs = 2, seed = 1988)

# Completed data and estimated scores
X_completed <- completeObs(ppca_fit)
scores(ppca_fit)

# pip install ppca-py
import numpy as np
from ppca import PPCA

np.random.seed(1988)
n = 150
p = 6

latent_factors = np.random.randn(n, 2)
loadings_true = np.random.randn(p, 2)
X = latent_factors @ loadings_true.T + np.random.randn(n, p) * 0.2

# Introduce missing values
missing_mask = np.random.rand(n, p) < 0.10
X[missing_mask] = np.nan

ppca = PPCA(n_components=2)
ppca.fit(X)

scores, score_cov = ppca.posterior_latent(X)
X_imputed = ppca.sample_missing(X, n_draws=1)[0]

print("Estimated noise variance:", round(ppca.noise_variance_, 4))
print("First five latent scores:\n", np.round(scores[:5], 3))

Truncated PCA

This last flavor is a little different. Truncated PCA does not change the statistical target. It changes the computation. Instead of computing the full singular value decomposition, we directly approximate the top singular vectors:

When and are large, or when is sparse, that distinction matters a lot. If all you want are the first few components, computing the full decomposition is wasted effort.

For practitioners, this is often the most useful PCA variant of all because it makes the classical method scale. The catch is conceptual rather than mathematical: randomized or truncated PCA is not discovering a different notion of component. It is approximating the same principal subspace more cheaply. If the approximation error is small, great. If not, you have a computational shortcut, not a new estimator.

# install.packages("irlba")
library(irlba)

set.seed(1988)
n <- 1000
p <- 200

latent_factors <- matrix(rnorm(n * 5), n, 5)
loadings_true <- matrix(rnorm(p * 5), p, 5)
X_large <- latent_factors %*% t(loadings_true) + matrix(rnorm(n * p, sd = 0.5), n, p)

# Fast approximation to the first 5 principal components
pca_fast <- prcomp_irlba(X_large, n = 5, center = TRUE, scale. = TRUE)

pca_fast$sdev^2 / sum(pca_fast$sdev^2)

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

np.random.seed(1988)
n = 1000
p = 200

latent_factors = np.random.randn(n, 5)
loadings_true = np.random.randn(p, 5)
X_large = latent_factors @ loadings_true.T + np.random.randn(n, p) * 0.5

X_large = StandardScaler().fit_transform(X_large)

# Randomized SVD computes an approximate leading subspace
pca_fast = PCA(n_components=5, svd_solver="randomized", random_state=1988)
pca_fast.fit(X_large)

print(np.round(pca_fast.explained_variance_ratio_, 3))

Manski Bounds

Manski (1990) is the natural starting point because it assumes almost nothing beyond bounded outcomes. Suppose , let , and define

Then the missing counterfactual means satisfy

and

Combining them gives sharp bounds on the ATE:

where

and

These bounds are usually wide, and that is exactly the point. Manski bounds tell you what the data alone can support before you add structure. In practice, I treat them as the baseline honesty check.

Tightening Manski: MTR, MTS, and MIV

The usual next step is to ask whether credible qualitative restrictions can narrow the interval. Manski and Pepper (2000) study three of the most useful ones. My first job market paper as a PhD candidate employed these restrictions to tighten the Manski bounds in the context of the labor market impact of immigration.

First, under Monotone Treatment Response (MTR), treatment weakly helps everyone:

MTR tightens the bounds by ruling out any configuration in which treatment hurts some units, so the lower bound rises and negative treatment effects become harder or impossible to sustain. For example, under MTR, cannot be below (each control’s missing is at least that unit’s observed ), not merely ; and cannot exceed .

Second, under Monotone Treatment Selection (MTS), treated units are systematically stronger than untreated units in terms of their potential outcomes. MTS tightens the bounds by imposing an ordering on who selects into treatment, so the observed outcomes in one group become informative about the missing potential outcomes in the other. For example, under MTS, is bounded below by , not merely .

Third, under a Monotone Instrumental Variable (MIV) assumption, an ordered variable shifts potential outcomes in a known direction:

In words, MIV lets us use the ordering in to intersect bounds across instrument values, which can noticeably shrink the identified set. These assumptions get more powerful as the data scientist combines them together. In some cases, the resulting interval can be informative.

Balke-Pearl Bounds for Noncompliance

Balke and Pearl (1997) address randomized assignment with imperfect compliance. Instead of jumping directly to LATE under exclusion and monotonicity, they ask a broader question: what does the observed joint distribution of imply about the population treatment effect under weaker assumptions?

The answer is a sharp nonparametric bound obtained by optimizing over all latent compliance-response types consistent with the observed data:

This is best viewed as a separation between what the experiment identifies and what extra assumptions identify. Balke-Pearl bounds are often much wider than a LATE estimate, but they answer a different question. LATE is a point-identified effect for compliers under stronger structure. Balke-Pearl bounds are partial-identification statements about broader causal quantities. When the policy question is about the full eligible population rather than compliers, that distinction matters.

Lee Bounds for Sample Selection

Lee (2009) is the method I see most often in practice because the intuition is so transparent. Suppose treatment is randomized, but outcomes are only observed for selected units. Wages observed only for employed workers is the canonical example. If treatment changes employment, comparing observed wages across treatment arms is contaminated by selection.

Lee’s key assumption is monotone selection: treatment can move selection in only one direction for every unit. If treatment raises the probability of observation, then the treated group contains some “extra” observed units relative to control. Those units must be trimmed away from one tail or the other of the treated outcome distribution.

Let indicate whether the outcome is observed and suppose . The excess selected share in the treated group is

Trimming a fraction from the upper tail gives one bound; trimming it from the lower tail gives the other.

I like Lee bounds because they are easy to explain and easy to audit. The practical warning is equally simple: if treatment plausibly pushes some units into the sample and others out, the monotone-selection logic breaks. ## Bottom Line

• Bounds are not a consolation prize. They are the right estimand when the data do not support point identification.
• Manski bounds are the benchmark because they show what your design identifies before assumptions start doing the heavy lifting.
• Monotonicity restrictions, Lee trimming, and Balke-Pearl bounds can be very informative, but only when their substantive assumptions are defensible.
• Wide bounds are often the most important empirical result in the paper because they reveal how little the design alone can rule out.

Regression Is a Projection, Not a Causal Model

This is the first point I would emphasize in practice. Writing down

does not, by itself, assume homogeneous treatment effects or even invoke potential outcomes. It simply defines the best linear predictor of given and . If the goal is prediction, that is the end of the story. (For a related distinction about what randomness this projection is defined over, see my note on OLS with fixed versus random .)

For causal interpretation, however, we need more. Under random assignment or selection on observables, plus enough structure on how outcomes vary with , the projection coefficient may line up with a causal estimand. Under constant treatment effects and correct linear adjustment, that estimand is often the ATE. Once treatment effects vary with , the coefficient generally becomes a weighted average of heterogeneous effects rather than the plain sample average.

Aronow and Samii: Asymptotic View

Aronow and Samii (2016) show that regression-adjusted estimators need not be representative of the sample as a whole. In large samples, the estimand targeted by regression can be written as a weighted average of conditional treatment effects, where the weights depend on how treatment assignment varies with covariates and on the linear adjustment built into the regression.

The key practical point is that OLS does not weight covariate strata equally. These weights are proportional to residualized treatment variation (via FWL), not to the precision of outcome estimates. In particular, they do not correspond to inverse-variance weights in general. So even under ignorability, the regression coefficient need not correspond to the ATE for the empirical covariate distribution. It is often better understood as an ATE for an implicit reweighted population. That is a subtle point, but it matters whenever overlap is uneven or the linear model fits some regions of the covariate space much better than others.

Chattopadhyay and Zubizarreta: Finite-Sample View

One limitation of the Aronow-Samii perspective is that it is asymptotic. Chattopadhyay and Zubizarreta (2023) go further by showing that common linear regression estimators admit exact finite-sample weighting representations. For a regression-adjusted ATE estimator,

where the weights are functions of only and , not the realized outcomes.

This is useful for two reasons. First, it makes regression adjustment look less mysterious: OLS is implicitly constructing a weighted comparison between treated and control outcomes. Second, the implied weights can be inspected directly. In their framework, the weights clarify when regression adjustment achieves exact balance on included covariates, how dispersed the weights are, and whether the regression is targeting a population that still looks like the observed sample. That is a much more practical diagnostic than simply reporting a coefficient table.

Słoczyński: Heterogeneous Effects View

Słoczyński (2022) asks what the OLS coefficient means when treatment effects are heterogeneous. His central result is that the coefficient on treatment is generally not the ATE. Instead, it is a convex combination of two group-specific effect parameters that, under additional conditions, can be interpreted as the ATT and the ATU. The striking part is the weighting: the smaller treatment arm gets the larger implicit weight.

So if treated units are rare, OLS tends to lean toward effects for treated units. If treated units are common, it leans toward effects for untreated units. The exact formula depends on the specification and on how treatment assignment varies with covariates, but the qualitative message is robust: heterogeneity changes the target, and OLS can overweight the effect for the smaller group.

This is one of those results that sounds surprising at first and obvious in hindsight. Regression learns treatment effects from residual variation in treatment status. When one group is small, comparisons involving that group carry disproportionate identifying content. The practical implication is straightforward: if you care specifically about the ATE or ATT, you should not assume OLS is giving it to you just because the regression includes controls.

Angrist and Pischke: Saturated Model View

The cleanest interpretation of regression comes from saturated models with discrete covariates, an approach emphasized by Angrist and coauthors. If takes only a small number of values and the regression fully saturates those cells, then OLS is just averaging within-cell treatment-control differences. In that case, regression is a dressed-up version of exact matching.

That perspective is helpful because it shows where the causal content comes from. The coefficient is credible when comparisons are being made within genuinely comparable covariate cells. But it also shows the limitation immediately: with continuous or high-dimensional covariates, literal saturation is impossible and the argument breaks down. At that point, OLS is no longer exact within-cell adjustment. It is a parametric approximation that extrapolates across covariate values. That is often reasonable, but it is no longer harmless.

Standard Lasso

The standard Lasso solves the following optimization problem:

The appeal of Lasso is straightforward: it trades a convex penalty for exact zeros in the solution. In moderately high dimensions, this often works surprisingly well as a first pass.

The main issue shows up when predictors are correlated. Lasso will typically pick one variable from a correlated group and ignore the rest, and which one it picks can be unstable across folds or small perturbations of the data. At the same time, all coefficients are shrunk, including the large ones, which introduces bias that doesn’t go away even with large samples.

In practice, I treat standard Lasso as a baseline rather than a final model. If it’s stable and predictive, great. If not, it’s usually pointing to a structural issue in the design.

library(glmnet)

# Simulate data
set.seed(1988)
n <- 100
p <- 20
X <- matrix(rnorm(n * p), n, p)
beta_true <- c(3, -2, 1.5, rep(0, p - 3))  # Only 3 non-zero coefficients
y <- X %*% beta_true + rnorm(n)

# Fit standard Lasso
lasso_fit <- glmnet(X, y, alpha = 1)  # alpha = 1 for Lasso

# Cross-validation to select lambda
cv_fit <- cv.glmnet(X, y, alpha = 1)
lambda_opt <- cv_fit$lambda.min

# Get coefficients at optimal lambda
coef(cv_fit, s = "lambda.min")

from sklearn.linear_model import Lasso, LassoCV
from sklearn.datasets import make_regression
import numpy as np

# Simulate data
np.random.seed(1988)
X, y, coef_true = make_regression(n_samples=100, n_features=20, 
                                   n_informative=3, coef=True, 
                                   noise=1.0, random_state=123)

# Fit Lasso with cross-validation
lasso = LassoCV(cv=5, random_state=123)
lasso.fit(X, y)

# Display results
print(f"Optimal lambda: {lasso.alpha_:.4f}")
print(f"Number of non-zero coefficients: {np.sum(lasso.coef_ != 0)}")
print(f"Selected coefficients:\n{lasso.coef_[lasso.coef_ != 0]}")

Adaptive Lasso

Adaptive Lasso extends the standard Lasso by using data-driven weights for each coefficient: where and comes from an initial estimator like OLS or Ridge.

The idea here is to penalize coefficients unevenly. Variables that look important in a first-stage model get penalized less, while weaker ones get pushed harder toward zero. This reduces the bias on large coefficients and improves variable selection consistency under certain conditions—one of the ingredients of the oracle property.

In practice, Adaptive Lasso is less about prediction and more about recovering a meaningful support. If you care about which variables are selected—not just the predictive accuracy—it’s often worth the extra step.

# Continue from previous example
library(glmnet)

# Step 1: Get initial estimates using Ridge
ridge_fit <- glmnet(X, y, alpha = 0)  # alpha = 0 for Ridge
cv_ridge <- cv.glmnet(X, y, alpha = 0)
beta_init <- as.vector(coef(cv_ridge, s = "lambda.min"))[-1]  # Remove intercept

# Step 2: Compute adaptive weights
gamma <- 1  # Common choice
weights <- 1 / (abs(beta_init) + 1e-8)^gamma  # Add small constant to avoid division by zero

# Step 3: Fit Adaptive Lasso
adaptive_lasso <- glmnet(X, y, alpha = 1, penalty.factor = weights)
cv_adaptive <- cv.glmnet(X, y, alpha = 1, penalty.factor = weights)

# Compare coefficients
cat("Standard Lasso non-zero:", sum(coef(cv_fit, s = "lambda.min")[-1] != 0), "\n")
cat("Adaptive Lasso non-zero:", sum(coef(cv_adaptive, s = "lambda.min")[-1] != 0), "\n")

from sklearn.linear_model import Ridge, Lasso

# Step 1: Get initial estimates using Ridge
ridge = Ridge(alpha=1.0)
ridge.fit(X, y)
beta_init = ridge.coef_

# Step 2: Compute adaptive weights
gamma = 1
weights = 1 / (np.abs(beta_init) + 1e-8)**gamma

# Step 3: Fit Adaptive Lasso (manual implementation via weighted penalty)
# Scale features by weights
X_weighted = X / weights

# Fit Lasso on weighted features
adaptive_lasso = Lasso(alpha=0.1)
adaptive_lasso.fit(X_weighted, y)

# Transform back to original scale
adaptive_coef = adaptive_lasso.coef_ / weights

print(f"Standard Lasso non-zero: {np.sum(lasso.coef_ != 0)}")
print(f"Adaptive Lasso non-zero: {np.sum(adaptive_coef != 0)}")

Relaxed Lasso

Relaxed Lasso separates selection from estimation. First, run Lasso to pick variables; then refit on that subset, either with OLS or partial shrinkage via a parameter . At you recover Lasso, and at you get post-selection OLS.

The point is to reduce shrinkage bias. Lasso is good at finding the support but tends to underestimate large coefficients. Relaxing the penalty after selection keeps sparsity while improving estimates.

In practice, this works well when you trust the selected variables but want better coefficient accuracy. The main risk is overfitting if too many variables are selected, so it’s worth tuning both and .

library(glmnet)
library(relaxnet)  # For relaxed Lasso

# Fit relaxed Lasso using glmnet (has built-in support)
relaxed_fit <- glmnet(X, y, alpha = 1, relax = TRUE)
cv_relaxed <- cv.glmnet(X, y, alpha = 1, relax = TRUE)

# Manual two-stage approach
# Stage 1: Standard Lasso selection
lasso_coef <- coef(cv_fit, s = "lambda.min")[-1]
selected <- which(lasso_coef != 0)

# Stage 2: OLS on selected variables
if (length(selected) > 0) {
  X_selected <- X[, selected]
  ols_fit <- lm(y ~ X_selected)
  
  # Compare coefficients
  cat("Lasso coefficients (selected):\n")
  print(lasso_coef[selected])
  cat("\nRelaxed (OLS) coefficients:\n")
  print(coef(ols_fit)[-1])
}

# Manual two-stage relaxed Lasso
from sklearn.linear_model import LinearRegression

# Stage 1: Lasso selection
lasso_coef = lasso.coef_
selected = np.where(lasso_coef != 0)[0]

print(f"Lasso selected {len(selected)} variables")

# Stage 2: OLS on selected variables
if len(selected) > 0:
    X_selected = X[:, selected]
    ols = LinearRegression()
    ols.fit(X_selected, y)
    
    # Compare coefficient magnitudes
    print(f"\nLasso coefficients (mean abs): {np.abs(lasso_coef[selected]).mean():.4f}")
    print(f"Relaxed coefficients (mean abs): {np.abs(ols.coef_).mean():.4f}")
    
    # Often relaxed coefficients are larger in magnitude

Square-root Lasso

Square-root Lasso, also known as Scaled Lasso, modifies the objective function to:

The crucial difference from standard Lasso is using the norm directly (without squaring) in the loss term. This seemingly small change has important consequences: the estimator becomes scale-invariant, meaning you don’t need to estimate or know the error variance to set the penalty parameter appropriately. In standard Lasso, the optimal choice of depends on the unknown noise level, but square-root Lasso eliminates this dependence.

This variant is particularly valuable when you have unknown or heteroskedastic error variance, making it robust to variance misspecification. The scale-invariance also simplifies tuning: you can use theoretically-motivated choices for without prior knowledge of the noise level. In practice, this often translates to more stable selection across different datasets and makes the method especially appealing in settings where variance estimation is challenging or the homoskedasticity assumption is questionable.

library(scalreg)  # For square-root Lasso

# Fit square-root Lasso
sqrt_lasso <- scalreg(X, y)

# Compare with standard Lasso
cat("Standard Lasso selected:", sum(coef(cv_fit, s = "lambda.min")[-1] != 0), "variables\n")
cat("Square-root Lasso selected:", sum(sqrt_lasso$coefficients != 0), "variables\n")

# Square-root Lasso is not in sklearn, but we can implement a simple version
from sklearn.linear_model import LassoLars
from scipy.optimize import minimize

# Manual implementation using CVXPY (if available)
try:
    import cvxpy as cp
    
    # Define variables
    beta = cp.Variable(X.shape[1])
    
    # Define objective: ||y - X*beta||_2 + lambda * ||beta||_1
    lambda_sqrt = 0.1
    objective = cp.Minimize(cp.norm(y - X @ beta, 2) + lambda_sqrt * cp.norm(beta, 1))
    
    # Solve
    prob = cp.Problem(objective)
    prob.solve()
    
    sqrt_lasso_coef = beta.value
    print(f"Square-root Lasso selected {np.sum(np.abs(sqrt_lasso_coef) > 1e-6)} variables")
    
except ImportError:
    print("Square-root Lasso requires cvxpy package")
    print("Install with: pip install cvxpy")

Elastic Net

Elastic Net blends and regularization by minimizing:

This is often reparametrized as where controls the mixing between and penalties.

Elastic Net fixes a key issue with Lasso: when predictors are highly correlated, Lasso tends to pick one arbitrarily and ignore the rest. Adding an penalty induces a grouping effect, so correlated variables enter or leave together, while the term still enforces sparsity.

This makes it a better default in settings with multicollinearity—common in practice. The mixing parameter controls the trade-off: closer to 1 behaves like Lasso, closer to like Ridge. In practice, moderate values (e.g. ) work well, with cross-validation refining the choice.

library(glmnet)

# Create correlated predictors to demonstrate Elastic Net advantage
set.seed(1988)
n <- 100
X_base <- matrix(rnorm(n * 5), n, 5)
# Add correlated predictors
X_corr <- cbind(X_base, X_base[, 1:2] + matrix(rnorm(n * 2, sd = 0.1), n, 2))
beta_true <- c(2, -1.5, 0, 0, 0, 2.2, -1.3)  # True coefficients for correlated pairs
y_corr <- X_corr %*% beta_true + rnorm(n)

# Fit Elastic Net with alpha = 0.5 (equal mix of $\ell_1$ and $\ell_2$)
elastic_fit <- cv.glmnet(X_corr, y_corr, alpha = 0.5)

# Compare with pure Lasso (alpha = 1)
lasso_corr <- cv.glmnet(X_corr, y_corr, alpha = 1)

cat("Elastic Net coefficients:\n")
print(coef(elastic_fit, s = "lambda.min"))
cat("\nLasso coefficients:\n")
print(coef(lasso_corr, s = "lambda.min"))

from sklearn.linear_model import ElasticNet, ElasticNetCV

# Create correlated predictors
np.random.seed(1988)
n = 100
X_base = np.random.randn(n, 5)
X_corr = np.hstack([X_base, X_base[:, :2] + np.random.randn(n, 2) * 0.1])
beta_true = np.array([2, -1.5, 0, 0, 0, 2.2, -1.3])
y_corr = X_corr @ beta_true + np.random.randn(n)

# Fit Elastic Net with l1_ratio = 0.5 (equal mix)
elastic = ElasticNetCV(l1_ratio=0.5, cv=5)
elastic.fit(X_corr, y_corr)

# Compare with Lasso
lasso_corr = LassoCV(cv=5)
lasso_corr.fit(X_corr, y_corr)

print("Elastic Net coefficients:")
print(elastic.coef_)
print(f"\nElastic Net selected {np.sum(elastic.coef_ != 0)} variables")
print(f"Lasso selected {np.sum(lasso_corr.coef_ != 0)} variables")

Group Lasso

Group Lasso extends the penalty to operate on predefined groups of variables: where represents the coefficients belonging to group , and is the norm applied within each group.

The key insight is that the norm within groups combined with summation across groups creates a sparsity-inducing penalty at the group level. Either all coefficients in a group are set to zero, or all are kept (though possibly shrunk). This “all or nothing” behavior respects the natural grouping structure in your data.

Group Lasso is useful when variables come in meaningful groups. A common example is categorical features encoded as dummies—you usually want to include or exclude the whole variable, not individual levels. Similar structure appears in multi-task settings or grouped scientific measurements.

Instead of sparsity at the coefficient level, Group Lasso selects entire groups while allowing dense coefficients within them. This makes the model align better with how features are constructed.

library(grpreg)

# Create data with natural groups
# Suppose we have 3 categorical variables with 3, 4, and 5 levels
set.seed(1988)
n <- 100
X1 <- model.matrix(~ factor(sample(1:3, n, replace = TRUE)) - 1)
X2 <- model.matrix(~ factor(sample(1:4, n, replace = TRUE)) - 1)
X3 <- model.matrix(~ factor(sample(1:5, n, replace = TRUE)) - 1)
X_grouped <- cbind(X1, X2, X3)

# Define groups (which columns belong to which group)
groups <- c(rep(1, 3), rep(2, 4), rep(3, 5))

# True model: only group 1 and 3 are relevant
beta_true <- c(2, -1, 1.5, rep(0, 4), 1, -0.5, 0.8, 1.2, -1)
y_grouped <- X_grouped %*% beta_true + rnorm(n)

# Fit Group Lasso
group_lasso <- cv.grpreg(X_grouped, y_grouped, group = groups, penalty = "grLasso")

cat("Group Lasso coefficients by group:\n")
coefs <- coef(group_lasso, s = "lambda.min")[-1]
for (g in unique(groups)) {
  cat(sprintf("Group %d: %d non-zero out of %d\n", 
              g, sum(coefs[groups == g] != 0), sum(groups == g)))
}

from sklearn.linear_model import MultiTaskLasso
# Note: True group Lasso requires specialized packages
# We'll demonstrate with a simplified example

# Simulate grouped structure
np.random.seed(1988)
n = 100
# Create 3 groups with 3, 4, 5 features each
X_grouped = np.random.randn(n, 12)
groups = np.array([1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3])

# True coefficients (group 1 and 3 active, group 2 zero)
beta_true = np.array([2, -1, 1.5, 0, 0, 0, 0, 1, -0.5, 0.8, 1.2, -1])
y_grouped = X_grouped @ beta_true + np.random.randn(n)

# For true Group Lasso, would need package like 'group-lasso' or 'celer'
# Here we show conceptual grouping with manual implementation
print("For Python Group Lasso, install specialized packages:")
print("  pip install group-lasso")
print("  pip install celer")

Fused Lasso

Fused Lasso adds a penalty on differences between adjacent coefficients:

This method introduces two types of penalties: the standard penalty encourages overall sparsity (setting coefficients to zero), while the fusion penalty encourages adjacent coefficients to be equal. The fusion penalty means that nearby coefficients in the ordering are pulled toward each other, creating piecewise-constant patterns in the coefficient profile.

Fused Lasso is useful when features have a natural ordering and coefficients are expected to vary smoothly or in blocks. Instead of treating coefficients independently, it encourages both sparsity and similarity between neighbors, leading to piecewise-constant patterns.

This shows up in time series, spatial data, or ordered genomic features. The two penalties control the trade-off: drives sparsity, while controls how strongly adjacent coefficients are fused.

library(genlasso)

# Simulate data with ordered features (e.g., time series or spatial)
set.seed(1988)
n <- 100
p <- 50

# Create design matrix with ordered features
X_ordered <- matrix(rnorm(n * p), n, p)

# True coefficients with piecewise constant structure
beta_true <- c(rep(0, 10), rep(2, 15), rep(0, 10), rep(-1.5, 10), rep(0, 5))
y_ordered <- X_ordered %*% beta_true + rnorm(n)

# Fit Fused Lasso
fused_fit <- fusedlasso(y_ordered, X_ordered)

# Get coefficients at a specific lambda
lambda_idx <- 50  # Example index
coefs_fused <- coef(fused_fit, lambda = fused_fit$lambda[lambda_idx])$beta

# Visualize coefficient profile
plot(coefs_fused, type = "s", 
     main = "Fused Lasso Coefficient Profile",
     xlab = "Feature Index", ylab = "Coefficient",
     col = "blue", lwd = 2)
lines(beta_true, col = "red", lty = 2, lwd = 2)
legend("topright", c("Estimated", "True"), 
       col = c("blue", "red"), lty = c(1, 2))

# Fused Lasso implementation using sklearn and custom penalty
from sklearn.linear_model import Lasso
import matplotlib.pyplot as plt

# Simulate ordered features
np.random.seed(1988)
n, p = 100, 50
X_ordered = np.random.randn(n, p)

# Piecewise constant true coefficients
beta_true = np.concatenate([
    np.zeros(10), np.full(15, 2), np.zeros(10), 
    np.full(10, -1.5), np.zeros(5)
])
y_ordered = X_ordered @ beta_true + np.random.randn(n)

# Standard Lasso (for comparison)
lasso_ordered = Lasso(alpha=0.1)
lasso_ordered.fit(X_ordered, y_ordered)

# For true Fused Lasso, specialized packages needed
# Conceptual visualization
plt.figure(figsize=(10, 5))
plt.plot(beta_true, 'r--', label='True', linewidth=2)
plt.plot(lasso_ordered.coef_, 'b-', label='Standard Lasso', alpha=0.7)
plt.xlabel('Feature Index')
plt.ylabel('Coefficient Value')
plt.title('Coefficient Profile: Fused Lasso Encourages Piecewise Constant Structure')
plt.legend()
plt.grid(alpha=0.3)
plt.show()

print("For true Fused Lasso in Python, consider packages:")
print("  skfda (functional data analysis)")
print("  or implement using cvxpy with fusion penalty")

Graphical Lasso

Graphical Lasso applies penalization to the estimation of precision matrices (inverse covariance matrices): where is the precision matrix, is the sample covariance matrix, and ensures positive definiteness.

Graphical Lasso shifts the focus from regression to covariance structure, estimating a sparse precision matrix. A zero entry means variables and are conditionally independent given the rest, so the model directly encodes a network of relationships.

This is useful when the goal is to recover dependency structure rather than predict an outcome—common in genomics, finance, or neuroscience. The penalty enforces sparsity, leading to interpretable graphs where most connections are absent. In practice, the main challenge is tuning to balance fit and sparsity.

library(glasso)
library(igraph)

# Simulate multivariate data
set.seed(1988)
n <- 100
p <- 10

# Create a sparse precision matrix (true network structure)
Theta_true <- matrix(0, p, p)
diag(Theta_true) <- 1
# Add some conditional dependencies
Theta_true[1, 2] <- Theta_true[2, 1] <- 0.5
Theta_true[2, 3] <- Theta_true[3, 2] <- 0.4
Theta_true[4, 5] <- Theta_true[5, 4] <- 0.6
Theta_true[7, 8] <- Theta_true[8, 7] <- 0.3

# Generate data from this precision matrix
Sigma <- solve(Theta_true)
X_network <- MASS::mvrnorm(n, mu = rep(0, p), Sigma = Sigma)

# Compute sample covariance
S <- cov(X_network)

# Fit Graphical Lasso
glasso_fit <- glasso(S, rho = 0.1)  # rho is the penalty parameter

# Extract estimated precision matrix
Theta_est <- glasso_fit$wi

# Visualize network
# Create adjacency matrix (thresholded)
adj_matrix <- (abs(Theta_est) > 0.01) * 1
diag(adj_matrix) <- 0

# Plot network
graph_obj <- graph_from_adjacency_matrix(adj_matrix, mode = "undirected")
plot(graph_obj, 
     main = "Estimated Conditional Dependence Network",
     vertex.size = 20,
     vertex.label.cex = 0.8)

from sklearn.covariance import GraphicalLassoCV
import networkx as nx
import matplotlib.pyplot as plt

# Simulate multivariate data
np.random.seed(1988)
n, p = 100, 10

# True sparse precision matrix
Theta_true = np.eye(p)
Theta_true[0, 1] = Theta_true[1, 0] = 0.5
Theta_true[1, 2] = Theta_true[2, 1] = 0.4
Theta_true[3, 4] = Theta_true[4, 3] = 0.6
Theta_true[6, 7] = Theta_true[7, 6] = 0.3

# Generate data
Sigma = np.linalg.inv(Theta_true)
X_network = np.random.multivariate_normal(np.zeros(p), Sigma, size=n)

# Fit Graphical Lasso with cross-validation
glasso = GraphicalLassoCV(cv=5)
glasso.fit(X_network)

# Get estimated precision matrix
Theta_est = glasso.precision_

# Visualize network
plt.figure(figsize=(10, 5))

# Create adjacency matrix (thresholded)
adj_matrix = (np.abs(Theta_est) > 0.01).astype(int)
np.fill_diagonal(adj_matrix, 0)

# Plot using networkx
G = nx.from_numpy_array(adj_matrix)
pos = nx.spring_layout(G, seed=123)

plt.subplot(1, 2, 1)
nx.draw(G, pos, with_labels=True, node_color='lightblue', 
        node_size=500, font_size=10, font_weight='bold')
plt.title('Estimated Network Structure')

# Show precision matrix heatmap
plt.subplot(1, 2, 2)
plt.imshow(Theta_est, cmap='RdBu_r', vmin=-1, vmax=1)
plt.colorbar(label='Precision Matrix Entry')
plt.title('Estimated Precision Matrix')
plt.tight_layout()
plt.show()

print(f"Sparsity: {np.sum(np.abs(Theta_est) < 0.01) / p**2:.2%}")

Definition

An estimator is said to have the oracle property if it can do two things:

• Selection consistency:
• Asymptotic efficiency: which is the same limiting distribution as the OLS estimator that knows in advance.

If the support were known, estimation reduces to low-dimensional OLS on . That estimator is unbiased, efficient, and easy to analyze. Some of you will remember the Gauss-Markov theorem from your econometrics course which states that, the OLS estimator is the best linear unbiased estimator (BLUE) under homoskedasticity.

This is an appealing theoretical benchmark for sparse estimators. You can hardly do better than that.

Which Methods Achieve It

Classical LASSO does not generally satisfy the oracle property. Its penalty introduces shrinkage bias that persists asymptotically.

Nonconvex penalties (e.g., SCAD and MCP) were explicitly designed to achieve the oracle property under regularity conditions. Adaptive LASSO can also achieve it when weights are constructed from a root- consistent pilot estimator.

The key mechanism is reduced shrinkage for large coefficients while still penalizing small ones.

Practical Implications

The oracle property is always asymptotic. There are never such guarantees in finite samples. It requires conditions such as:

• correct model specification,
• suitable signal strength (minimum nonzero coefficient size),
• regularity conditions on the design matrix,
• appropriate tuning parameter rates.

I have written about the theoretical guarantees of LASSO and the design conditions they rely on before. In finite samples, especially when signals are weak or highly correlated, procedures that theoretically satisfy the oracle property may not outperform simpler methods. In practice, prediction risk often matters more than exact support recovery.

There is also a conceptual point: the oracle benchmark assumes that the “true” model is sparse and well-defined. In many modern applications, sparsity is an approximation rather than a literal truth.

Skewed Sampling Distributions

Symmetry of the interval reflects symmetry of the sampling distribution, not symmetry of the data.

Consider estimating a proportion from a binomial model. The MLE is . For moderate and near or , the distribution of is visibly skewed. A Wald interval,

may extend below or above . That is a red flag: the procedure ignores the geometry of the parameter space.

Score intervals and logit-transformed intervals are asymmetric in precisely because they respect this skewness and the constraint. The asymmetry is not a flaw—it is the correction.

Nonlinear Transformations

Suppose for a nonlinear . Even if is approximately normal, the distribution of

is generally not symmetric in finite samples.

A first-order delta method approximation gives

which suggests a symmetric interval in -space. However, mapping that interval back to -space via typically produces asymmetry.

This is routine in practice. Log-scale confidence intervals for positive parameters (e.g., rate ratios, hazard ratios) are symmetric in but asymmetric in . The asymmetry reflects curvature in .

Likelihood-Based Intervals

Likelihood-ratio intervals solve

where is the log-likelihood. When is not quadratic as is common in small samples or near boundaries, the resulting set is not symmetric around .

The quadratic approximation that underlies Wald intervals is a second-order Taylor expansion. If the likelihood is skewed, the quadratic approximation inherits bias, and symmetric intervals misrepresent uncertainty.

Bootstrap Percentile Intervals

Bootstrap percentile intervals are defined directly from empirical quantiles of the bootstrap distribution:

where are bootstrap replicates.

No symmetry is imposed. If the empirical distribution of is skewed, the interval is skewed. This is often desirable: the procedure adapts to the shape of the sampling distribution.

The percentile method is not universally optimal, but it makes the asymmetry explicit instead of suppressing it.

Fixed : Classical Linear Model

In the classical setup, is treated as fixed (non-stochastic). Then, all randomness comes from the error term .

Conditional on ,

Inference is therefore conditional inference. Confidence intervals and -tests are statements about the distribution of given this specific design matrix, .

This framework is natural in designed experiments, where is literally chosen by the researcher.

Random : Econometric View

In most observational settings, is random. We observe i.i.d. draws from an unknown joint distribution, . Under standard regularity conditions, the same OLS estimator, , satisfies

The asymptotic variance becomes

Under homoskedasticity, this simplifies to

The algebra mirrors the fixed- case, but the interpretation changes: we are no longer conditioning on a specific realization of ; we are averaging over its distribution.

What Actually Changes?

Three things matter.

First, the object of inference. With fixed , inference is conditional on the design. With random , inference is about repeated sampling of .

Second, exogeneity assumptions. In the fixed- model, we require . In the random- case, we need the same condition, but it now constrains the joint distribution: it says that once we know the regressors, there is no systematic remaining signal in the error term. Violations become statements about endogeneity, meaning is statistically related to omitted factors inside . Whether the resulting coefficient carries a causal meaning at all is a separate question I take up in what OLS estimates in causal inference.

Third, robustness. Heteroskedasticity-robust standard errors are naturally derived in the random- framework, where the conditional variance may depend on . In other words, different parts of the regressor distribution can come with different noise levels, so inference has to account for that variation rather than rely on a single common variance.

What does not change is the formula for . Nor does unbiasedness depend on being fixed; it depends on the conditional mean-zero assumption.