Finding the Lipschitz Constant of a Function

We've already explored Lipschitz functions, which have a bounded rate of change. Now, let's focus on how to find the Lipschitz constant—the key parameter that determines how "steep" a function can be.

What is the Lipschitz Constant?

A function $f : R^{n} \to R$ is Lipschitz continuous with constant $B$ if there exists a finite number $B \geq 0$ such that:

| f (x) - f (y) | \leq B ∥ x - y ∥, \forall x, y \in D (f) .

The Lipschitz constant $B$ is the smallest possible value that satisfies this inequality for all $x, y$ . It essentially tells us how fast the function can change—a small $B$ means slow growth, while a large $B$ means faster variation.

Finding the Lipschitz Constant for a Function

There are different ways to compute the Lipschitz constant depending on whether we are dealing with:

Lipschitz continuity of function values ( $B$ )
Lipschitz continuity of the gradient (smoothness condition, $L$ )

1. Lipschitz Constant for Function Values

To find the Lipschitz constant $B$ for a function $f$ , we evaluate:

B = sup_{x \neq y} \frac{| f (x) - f (y) |}{∥ x - y ∥}

This means that $B$ is the maximum rate of change of the function over all possible pairs of points.

Recall:

The supremum (or least upper bound) of a set is the smallest number that is greater than or equal to all elements in the set.

Example 1

$f (x) = x^{2}$ on $R$

To find the Lipschitz constant $B$ , we compute:

B = sup_{x \neq y} \frac{| x^{2} - y^{2} |}{| x - y |}

Using the difference of squares identity:

\frac{| x^{2} - y^{2} |}{| x - y |} = | x + y | .

Since $| x + y |$ can be arbitrarily large as $x, y \to \infty$ , $f (x) = x^{2}$ is not Lipschitz on $R$ . However, on a bounded domain, say $[- a, a]$ , we get:

B = sup_{x, y \in [- a, a]} | x + y | = 2 a .

Thus, $f (x) = x^{2}$ is Lipschitz on a bounded interval, with $B = 2 a$ .

2. Lipschitz Constant for the Gradient (Smoothness Constant)

A function is $L$ -smooth (i.e., its gradient is Lipschitz) if:

∥ \nabla f (x) - \nabla f (y) ∥ \leq L ∥ x - y ∥, \forall x, y \in R^{n} .

To find $L$ , we compute:

L = sup_{x} ∥ J f (x) ∥

where $J f (x)$ is the Jacobian (matrix of partial derivatives) of $f$ at $x$ .

Example 2

$f (x) = \frac{1}{2} x^{2}$

Compute the gradient: $\nabla f (x) = x .$
Compute the Lipschitz constant of the gradient: $L = sup_{x} | \nabla f (x) | .$

Since $| \nabla f (x) | = | x |$ grows indefinitely, $f (x)$ is not smooth on $R$ . But on a bounded domain $[- a, a]$ , we get:

L = sup_{x \in [- a, a]} | x | = a .

So, for a bounded interval, the function is $a$ -smooth.

Lipschitz Constant from the Hessian

For twice-differentiable functions, the Lipschitz constant of the gradient can be found using the Hessian matrix $H_{f} (x)$ . The Lipschitz constant $L$ is given by:

L = sup_{x} λ_{max} (H_{f} (x)),

where $λ_{max} (H_{f} (x))$ is the largest eigenvalue of the Hessian matrix.

Example 3

$f (x) = x^{2} + y^{2}$

The Hessian is:

H_{f} (x, y) = [\begin{matrix} 2 & 0 \\ 0 & 2 \end{matrix}] .

The eigenvalues of this matrix are both 2, so:

L = 2.

This means $f (x) = x^{2} + y^{2}$ is 2-smooth.

Summary

Type of Lipschitz Constant	Formula	Meaning
Lipschitz function values ( $B$ )	$B = sup_{x \neq y} \frac{\| f (x) - f (y) \|}{\| x - y \|}$	Controls how fast function values change.
Lipschitz gradient ( $L$ -smoothness)	$L = sup_{x} \| J f (x) \|$	Controls how fast the gradient changes.
Hessian-based Lipschitz constant	$L = sup_{x} λ_{max} (H_{f} (x))$	Controls the curvature of the function.