Intro to Convex Functions

We’ve set the stage by understanding what optimization means — finding the best solution to a problem. But to truly master optimization, we need to understand the building blocks that shape these solutions. Before we dive into convex functions, we must first meet their geometric backbone: convex sets.

Convex sets are like the terrain on which optimization takes place — they shape where solutions can be found and how we navigate towards them. Without understanding the landscape, we can't understand how functions behave within it.

The Cauchy – Schwarz Inequality

The Cauchy – Schwarz inequality is a fundamental tool in mathematics, especially in the study of convex functions. It states:

| x \cdot y | \leq ∥ x ∥ ∥ y ∥

This inequality means that the absolute value of the dot product between two vectors $x$ and $y$ is always less than or equal to the product of their magnitudes. In other words, the angle between two vectors can never violate this limit — it sets the stage for understanding geometric relationships in convex sets and spaces.

Remark

For more deep understanding, you can refer to this page: Cauchy – Schwarz Inequality

Convex Sets

A convex set is a collection of points where any line segment between two points in the set remains entirely within the set. This is the geometric definition of "no shortcuts" — everything inside the boundary is smooth and connected. Formally, a set $C$ is convex if:

λ x + (1 - λ) y \in C for all x, y \in C and λ \in [0, 1]

This formula means that if you pick any two points in the set, the entire straight path (line segment) connecting them stays inside the set.

Figure 2.1.png|600

Remark

This formula was derived from coordinate geometry, when we want to divide a line segment into two parts according to some ratio.

Visual Understanding of Convexity

Convex shape: The line between any two points stays inside the boundary (e.g., a hexagon).
Not convex shape: The line between two points "leaks" outside the shape (e.g., a shape with an inward dent or a hole).

Figures 2.2 and 2.3.png|600

This geometric difference is essential because convex sets guarantee easier optimization — the shortest path is always safe!

Examples of Convex Sets

Convexity shows up everywhere in geometry and optimization. Here are common examples:

The empty set: A trivial convex set (nothing to break the rule!).
A single point: With only one point, any line between points is just that point.
A line or line segment: Always straight — no curves or dents.
A subspace (like a plane through the origin): Infinite, flat, and smooth.
A hyperplane: A flat surface extending infinitely in multiple dimensions.
Balls and ellipses: Round shapes where every line between points stays within the boundary.

Properties of Convex Sets: Special Behavior

Convex sets have unique and useful mathematical properties:

Intersections of convex sets are always convex:
If you overlap multiple convex sets, their common area remains convex.
Example: Overlapping circles still creates a "smooth" region without dents.
Projections onto convex sets are unique:
If you drop a point onto a convex set from above (like casting a shadow), there will be only one closest point — a unique solution.

Why Convexity Matters

Convexity simplifies the search for solutions in optimization problems. In a convex set, there are no traps, no false minima, and the shortest route to the solution is direct. That's why understanding convex sets is a superpower for solving complex optimization problems!