D.48 About Lagrangian multipliers
This note will derive the Lagrangian multipliers for an example
problem. Only calculus will be used. The example problem will be to
find a stationary point of a function of four variables if there
are two constraints. Different numbers of variables and constraints
would work out in similar ways as this example.
The four variables that example function depends on will be
denoted by , , , and .
The two constraints will be taken to be equations of the form
0 and 0, for
suitable functions and . Constraints can always be
brought in such a form by taking everything in the constraint’s
equation to the left-hand side of the equals sign.
So the example problem is:
Stationarize means to find locations where the function has a minimum
or a maximum, or any other point where it does not change under small
changes of the variables as long as these satisfy
The first thing to note is that rather than considering to be a
function of , you can consider it instead to be
to be a function of and and only two additional variables from
, say and :
The reason you can do that is that you should in principle be able to
reconstruct the two missing variables and given ,
, , and .
As a result, any small change in the function , regardless of
constraints, can be written using the expression for a total
At the desired stationary point, acceptable changes in variables are
those that keep and constant at zero; they have 0 and
0. So for to be stationary under all acceptable changes
of variables, you must have that the final two terms are zero for any
changes in variables. This means that the partial derivatives in the
final two terms must be zero since the changes and
can be arbitrary.
For changes in variables that do go out of bounds, the change in
will not be zero; that change will be given by the first two
terms in the right-hand side. So, the erroneous changes in due to
going out of bounds are these first two terms, and if we subtract
them, we get zero net change for any arbitrary change in
In other words, if we
penalize the change in for
going out of bounds by amounts and at the rate above,
any change in variables will produce a penalized change of zero,
whether it stays within bounds or not.
The two derivatives at the stationary point in the expression above
are the Lagrangian multipliers or penalty factors, call them
. In those terms
for whatever is the change in the variables , and
that means for whatever is the change in the original variables
. Therefore, the change in the penalized
is zero whatever is the change in the variables
In practical application, explicitly computing the Lagrangian
multipliers and as the derivatives of
function is not needed. You get four equations by putting
the derivatives of the penalized with respect to through
equal to zero, and the two constraints provide two more
equations. Six equations is enough to find the six unknowns
through , and .