Full search page →

Logistic separation and finite estimates

Optional · supplementary note 1 of 1 for OPT 05 · Approximation, Statistical Estimation and Experiment Design. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

A convex likelihood can improve indefinitely along a parameter direction without having a finite minimizer. Logistic separation is a practical example of the existence distinction introduced in OPT 01. Software documentation explicitly distinguishes finite estimates from separated data [6].

Consider two scalar observations (x,y)=(−1,0)(x,y)=(-1,0) and (1,1)(1,1) with no intercept, and model Pr⁡(Y=1∣x)=1/(1+e−wx)\Pr(Y=1\mid x)=1/(1+e^{-wx}). Both observed labels become more likely as the coefficient ww increases. Their summed negative log-likelihood is

ℓ(w)=2log⁡(1+e−w),ℓ′(w)=−21+ew.\ell(w)=2\log(1+e^{-w}),\qquad \ell'(w)=-\frac{2}{1+e^w}.

Here w∈Rw\in\mathbb R is the fitted coefficient, xx the fixed feature and yy the binary label. The objective is positive at every finite ww and approaches zero as w→+∞w\to+\infty. Thus the parameters diverge while the objective has a finite unattained infimum. This is not an objective unbounded below.

Adding λw2/2\lambda w^2/2 with λ>0\lambda>0 makes this scalar objective strongly convex and growing at both tails, giving a unique finite minimizer. It changes the estimator; it does not recover a missing unregularized estimate. In a larger model, an unpenalized direction requires a separate existence check [1].

Code · Python · 14 lines
import numpy as np
from scipy.optimize import brentq
from scipy.special import expit

loss = lambda w: 2*np.logaddexp(0, -w)
values = loss(np.array([0., 2., 10., 20.]))
assert np.all(values > 0) and np.all(np.diff(values) < 0)
lam = 0.2
derivative = lambda w: -2*expit(-w) + lam*w
solution = brentq(derivative, 0, 2/lam)
curvature = 2*expit(solution)*expit(-solution) + lam
assert abs(derivative(solution)) < 1e-12 and curvature >= lam
print("Unregularized losses:", values)
print("Penalized coefficient / gradient:", solution, derivative(solution))

The shrinking unregularized gradient is not evidence of a finite parameter estimate. Inspect coefficient growth and the existence assumptions as well as the loss. The regularized Newton example in OPT 06 instead uses an explicit positive quadratic penalty.

References

  1. Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press, 2004. Reading: Chapter 6, Sections 6.1–6.5; Chapter 7, Sections 7.1–7.5. Official book, free PDF.

  2. SciPy developers, “scipy.optimize.minimize” and “scipy.optimize.minimize_scalar,” SciPy API documentation, accessed 2026. Multivariate interface, scalar interface.

  3. SciPy developers, “scipy.optimize.linprog,” SciPy API documentation, accessed 2026. Official documentation.

  4. Sebastian Pokutta, “The Frank-Wolfe Algorithm: A Short Introduction,” Jahresbericht der Deutschen Mathematiker-Vereinigung, 126, 2024. Open-access survey.

  5. Michael Schaller, Alberto Bemporad and Stephen Boyd, “Learning Parametric Convex Functions,” manuscript, 2025. Official manuscript page.

  6. SAS Institute, “Existence of Maximum Likelihood Estimates,” SAS/STAT 12.1 User’s Guide, accessed 2026-11-05. Official documentation.