Hessian matrix

Hessian matrix: Understanding the second-order behavior of functions

The Hessian matrix is a mathematical tool used to analyze the curvature of a function. It helps us determine whether a function is convex, concave, or neither by looking at its second-order derivatives. In optimization, the Hessian is particularly useful because it tells us if a function has a unique minimum and whether standard optimization methods (like gradient descent) will work efficiently.

convex and concave functions.png|500

What is the Hessian matrix

The Hessian of a function $f : R^{n} \to R$ is a square matrix containing all the second-order partial derivatives of the function. It is written as:

H_{f} (x) = \nabla^{2} f (x) = [\begin{matrix} \frac{\partial^{2} f}{\partial {x_{1}}^{2}} & \frac{\partial^{2} f}{\partial x_{1} \partial x_{2}} & \dots & \frac{\partial^{2} f}{\partial x_{1} \partial x_{n}} \\ \frac{\partial^{2} f}{\partial x_{2} \partial x_{1}} & \frac{\partial^{2} f}{\partial {x_{2}}^{2}} & \dots & \frac{\partial^{2} f}{\partial x_{2} \partial x_{n}} \\ ⋮ & ⋮ & ⋱ & ⋮ \\ \frac{\partial^{2} f}{\partial x_{n} \partial x_{1}} & \frac{\partial^{2} f}{\partial x_{n} \partial x_{2}} & \dots & \frac{\partial^{2} f}{\partial {x_{n}}^{2}} \end{matrix}]

Each entry of this matrix represents the rate of change of one partial derivative with respect to another variable. The diagonal elements represent how the function curves along each coordinate direction, while the off-diagonal elements describe how the curvature changes between different variables.

Why is the Hessian important

The Hessian matrix is used to:

Determine convexity: If the Hessian satisfies certain conditions, the function is convex.
Classify critical points: It tells us whether a point is a local minimum, maximum, or a saddle point.
Optimize functions: In machine learning, economics, and physics, the Hessian is used to efficiently minimize or maximize functions.

Understanding the Hessian in one dimension (1D case)

In one dimension, the Hessian reduces to a single second derivative:

H_{f} (x) = \frac{d^{2} f}{d x^{2}}

Here’s how to interpret it:

If $\frac{d^{2} f}{d x^{2}} > 0$ , the function is convex (curves upwards like a bowl).
If $\frac{d^{2} f}{d x^{2}} < 0$ , the function is concave (curves downward like an upside-down bowl).
If $\frac{d^{2} f}{d x^{2}} = 0$ , the function might be linear or have an inflection point (a change in curvature).

Convex And Concave Functions And Inflection Points General.png|500

Example 1: quadratic function

Consider the function

f (x) = x^{2}

Its first derivative is:

\frac{d f}{d x} = 2 x

Its second derivative is:

\frac{d^{2} f}{d x^{2}} = 2

Since $2 > 0$ , the function is convex — its graph forms a parabolic bowl that always curves upwards.

Convex And Concave Functions And Inflection Points Good Example.png|400

Example 2: cubic function

Consider the function

f (x) = x^{3}

The first derivative is:

\frac{d f}{d x} = 3 x^{2}

The second derivative is:

\frac{d^{2} f}{d x^{2}} = 6 x

This function is not convex everywhere because the second derivative depends on $x$ :

For $x > 0$ , we have $\frac{d^{2} f}{d x^{2}} > 0$ , so it is convex.
For $x < 0$ , we have $\frac{d^{2} f}{d x^{2}} < 0$ , so it is concave.
At $x = 0$ , $\frac{d^{2} f}{d x^{2}} = 0$ , meaning the function has an inflection point where it transitions from concave to convex.

Convex And Concave Functions And Inflection Points Bad Example.png|400

This shows that convexity is not just about checking one point — it must hold everywhere.

Understanding the Hessian in two dimensions (2D case)

For a function of two variables $f (x, y)$ , the Hessian is a $2 \times 2$ matrix:

H_{f} (x, y) = [\begin{matrix} \frac{\partial^{2} f}{\partial x^{2}} & \frac{\partial^{2} f}{\partial x \partial y} \\ \frac{\partial^{2} f}{\partial y \partial x} & \frac{\partial^{2} f}{\partial y^{2}} \end{matrix}]

To determine convexity, we check if this matrix is positive semidefinite using the leading principal minors test (or checking all eigenvalues 🤷‍♀️):

The first leading principal minor (the first diagonal element) must be nonnegative:
$\frac{\partial^{2} f}{\partial x^{2}} \geq 0$
The determinant of the Hessian matrix must be nonnegative:
$det (H_{f}) = (\frac{\partial^{2} f}{\partial x^{2}}) (\frac{\partial^{2} f}{\partial y^{2}}) - {(\frac{\partial^{2} f}{\partial x \partial y})}^{2} \geq 0$

These two conditions ensure that the function is convex in two dimensions.

Example 1: convex function in 2D

Consider:

f (x, y) = x^{2} + y^{2}

First, compute the second derivatives:

\frac{\partial^{2} f}{\partial x^{2}} = 2, \frac{\partial^{2} f}{\partial y^{2}} = 2, \frac{\partial^{2} f}{\partial x \partial y} = 0

The Hessian matrix is:

H_{f} (x, y) = [\begin{matrix} 2 & 0 \\ 0 & 2 \end{matrix}]

Check the conditions:

The first leading principal minor is $2 \geq 0$ ✅
The determinant is $2 \cdot 2 - 0^{2} = 4 \geq 0$ ✅

Since both conditions hold, the function is convex.

Geometric intuition:
This function represents a bowl-shaped surface in 3D, confirming convexity.

Example 2: non-convex function in 2D

Consider:

f (x, y) = x^{2} - y^{2}

The Hessian matrix is:

H_{f} (x, y) = [\begin{matrix} 2 & 0 \\ 0 & - 2 \end{matrix}]

Check the conditions:

The first leading principal minor is $2 \geq 0$ ✅
The determinant is $2 \cdot (- 2) - 0^{2} = - 4$ , which is negative ❌

Hyperbolic Paraboloid.png

Since the determinant is negative, the Hessian is not positive semidefinite, meaning the function is not convex — it has a saddle point.

Summary of the Hessian Matrix and Convexity

Dimension	Hessian Form	Convexity Condition
1D	$\frac{d^{2} f}{d x^{2}}$	$\frac{d^{2} f}{d x^{2}} \geq 0$
2D	$[\begin{matrix} \frac{\partial^{2} f}{\partial x^{2}} & \frac{\partial^{2} f}{\partial x \partial y} \\ \frac{\partial^{2} f}{\partial y \partial x} & \frac{\partial^{2} f}{\partial y^{2}} \end{matrix}]$	Determinant test: $D > 0$ , $\frac{\partial^{2} f}{\partial x^{2}} > 0$
nD	$\nabla^{2} f (x)$	Matrix is positive semidefinite

So, basically the idea of nD case is the same as 2D case, we need to proove that the Hessian matrix is positive semidefinite. We can choose any method to do that: minors test or eigenvalues test.

Final Takeaways

The Hessian matrix captures the second-order behavior of a function.
In one dimension, convexity is determined by checking if $\frac{d^{2} f}{d x^{2}} \geq 0$ .
In higher dimensions, the function is convex if the Hessian matrix is positive semidefinite.
The determinant test helps determine convexity in 2D.
The Hessian is widely used in machine learning, physics, and optimization algorithms to ensure stability and efficiency.