Logistic separation and finite estimates
Optional · supplementary note 1 of 1 for OPT 05 · Approximation, Statistical Estimation and Experiment Design. Nothing in the main text depends on it; citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
A convex likelihood can improve indefinitely along a parameter direction without having a finite minimizer. Logistic separation is a practical example of the existence distinction introduced in OPT 01. Software documentation explicitly distinguishes finite estimates from separated data [6].
Consider two scalar observations and with no intercept, and model . Both observed labels become more likely as the coefficient increases. Their summed negative log-likelihood is
Here is the fitted coefficient, the fixed feature and the binary label. The objective is positive at every finite and approaches zero as . Thus the parameters diverge while the objective has a finite unattained infimum. This is not an objective unbounded below.
Adding with makes this scalar objective strongly convex and growing at both tails, giving a unique finite minimizer. It changes the estimator; it does not recover a missing unregularized estimate. In a larger model, an unpenalized direction requires a separate existence check [1].
Code · Python · 14 lines
import numpy as np
from scipy.optimize import brentq
from scipy.special import expit
loss = lambda w: 2*np.logaddexp(0, -w)
values = loss(np.array([0., 2., 10., 20.]))
assert np.all(values > 0) and np.all(np.diff(values) < 0)
lam = 0.2
derivative = lambda w: -2*expit(-w) + lam*w
solution = brentq(derivative, 0, 2/lam)
curvature = 2*expit(solution)*expit(-solution) + lam
assert abs(derivative(solution)) < 1e-12 and curvature >= lam
print("Unregularized losses:", values)
print("Penalized coefficient / gradient:", solution, derivative(solution))The shrinking unregularized gradient is not evidence of a finite parameter estimate. Inspect coefficient growth and the existence assumptions as well as the loss. The regularized Newton example in OPT 06 instead uses an explicit positive quadratic penalty.
References
-
Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press, 2004. Reading: Chapter 6, Sections 6.1–6.5; Chapter 7, Sections 7.1–7.5. Official book, free PDF.
-
SciPy developers, “scipy.optimize.minimize” and “scipy.optimize.minimize_scalar,” SciPy API documentation, accessed 2026. Multivariate interface, scalar interface.
-
SciPy developers, “scipy.optimize.linprog,” SciPy API documentation, accessed 2026. Official documentation.
-
Sebastian Pokutta, “The Frank-Wolfe Algorithm: A Short Introduction,” Jahresbericht der Deutschen Mathematiker-Vereinigung, 126, 2024. Open-access survey.
-
Michael Schaller, Alberto Bemporad and Stephen Boyd, “Learning Parametric Convex Functions,” manuscript, 2025. Official manuscript page.
-
SAS Institute, “Existence of Maximum Likelihood Estimates,” SAS/STAT 12.1 User’s Guide, accessed 2026-11-05. Official documentation.