Optimierungsgrundlagen

The following content is provided under a Creative Commons license.

Your support will help MIT OpenCourseWare continue to offer high-quality educational

resources for free.

To make a donation or to view additional materials from hundreds of MIT courses, visit

MIT OpenCourseWare at ocw.mit.edu.

Well, OK, I'm happy to be back.

And I'm really happy about the project proposals that are coming in.

This is, like, OK, this is a really a good part of the course.

And so keep them coming.

And I'm happy to give whatever feedback I can on those proposals

and do make a start.

They're really good.

And if summer completed before the end of the semester,

and we can produce a chance to offer you a chance to report on them,

that's good to, so well done with those proposals.

So today, I'm jumping to Part 6.

So Part 6 and Part 7 are optimization,

which is the fundamental algorithm that goes into deep learning.

So we've got to start with optimization.

Everybody has to get that picture.

And then Part 7 will be the structure of CNN's,

convolutional neural nets and all the kinds of applications.

And so can we start with optimization?

So first, could I get the basic facts about three terms of a Taylor series?

So that's the typical.

It's seldom that we would go up to third derivatives in optimization.

So that's the most useful approximation to a function.

Everybody recognizes it.

So I'm thinking of F as just one function and X as just one variable.

But now I really want to go to more variables.

So what do I have to change if F is a function of more variables?

So now I'm thinking of X as well.

No, let me see.

Yeah, I want n variables here.

X is X1 up to Xn.

So what?

I just to get the word straight.

So we can begin on optimization.

So what will be the similar step?

So the function F at X, remember X is n variables.

Okay, now what do I have?

Delta X.

So what's the point about Delta X now?

It's a vector.

Delta X1 to Delta Xn.

And what about the derivative of F?

It's a vector 2.

The derivative of F with respect X1.

The derivative of F with respect X2 and so on.

So what do I have?

What do I have to change about that?

I know those guys are vectors.

So it's their dot product.

So it's Delta X transpose.

That vector times this DFDX.

So what's the, so this now I'm replacing this by all the derivatives.

The, it's the gradient.

So the gradient of F at X is the derivatives, let's see.

It's essentially get the notation straight here.

Yeah.

So it'll be the partial derivatives of the function F.

So grad F is the partial derivatives of F with respect to X1 down

to partial derivative of respect to Xn.

Okay, good.

That's the linear term.

And now what's the quadratic term?

One half.

Now Delta X is into the scalar anymore.

It's a vector.

So I'm going to have Delta X transpose.

And Delta X.

And what goes in between is the second derivatives.

But I've got a function of n variables.

And so what does this, so now I have a matrix of second derivatives.

Right?

And I'll call it H.

This is the matrix of second derivatives.

H, J, K is the second derivative of F with respect to X, J, and X, K.

And what's the name for this guy?

The Hessian, Hessian matrix.

How the Hessians got into this picture?

I don't know.

The only Hessians I know are the ones who fought in the revolutionary war for somebody.

Who, which side were they on?

I think maybe the wrong side, the French were on our side.

And anyway, Hessian matrix.

And what are the facts about that matrix?

Well, the first fact is that it's the key fact is it's symmetric.

Yeah.

Okay.

And again, it's an approximation.

And so, and everybody recognizes that if we have a function, if n is very large,

and we have a function of many variables, then we had n derivatives to compute here,

and about half n squared derivatives using the half comes from the symmetry.

But the key point is the n squared variables, the derivatives to compute there.

So, computing the gradient is feasible if n is small or moderately large.

Actually, by using automatic differentiation, the key idea of back propagation, back prop.

You can do, you can speed up the computation of derivatives quite amazingly,

but still for the size of deep learning problems, that's how to reach.

Okay.

So, that's the picture, and then I will want to solve, use this to solve equations.

Let me, there's a parallel picture for a vector F.

So, now, this is a vector function.

This is f1 of x up to fn of x, and x is x1 to xn.

So, I have n functions of n variables, n functions of n variables.

Well, that's exactly what I have in the gradient.

That's the state of these two as parallel.

The parallel being f is corresponds to the gradient of f, n functions of n variables.

Okay.

Okay.

Now, maybe what I'm after here is to solve f equals zero.

I'm going to think about the f at x plus delta x.

So, it starts with f of x, and then we have the correction times the matrix of first derivatives.

And what's the name for that matrix of first derivatives?

Well, if I'm just given n functions, and what am I after here?

I'm looking for the Jacobian.

So, here we'll go the Jacobian.

This is the Jacobian named after Jacobi.

Jacobian matrix.

And what are its entries?

Jay, the jk entry, is the derivative of the j function with respect to the case variable.

Okay.

And I'm stopping at first order there.

Okay.

So, these are the sort of facts of calculus.

Fact of 1802, you could say.

Multi-variable calculus.

That's the point.

Notice that we're doing just like the first half of 1802, just differential calculus.

Derivatives.

Taylor series.

We're not doing multiple nicals.

That's not part of our world here.

Okay.

So, that's the background.

Now, I want to look at optimization.

Okay.

Okay.

So, over here, I want to optimize.

Well, over here, let me try to minimize f of x.

And I'll be in the vector case here.

And over here, I want to solve f equals zero.

And of course, that means f of 1 equals zero all the way along to f n equals zero.

Here, I have n equations n n unknowns.

Let me start with that one.

And I'll start with Newton's method.

Newton's method to solve these n equations in n unknowns.

Okay.

So, Newton's method.

Which is often not presented in 1802.

That's a crime.

Because that's the big application of the gradients in Jacobians.

Okay.

So, I'm trying to solve n equations in n unknowns.

And so, I want f at x plus delta x to be zero.

Right?

So, I want f of x plus delta x to be zero.

So, f of x plus delta x is I'm putting in a zero.

I'm just copying that equation.

Is f at the where I am.

Let me use k for the case iteration.

So, I'm at a point x k.

I want to get to a point x k plus 1.

And so, f of x plus j at that point times delta x,

which is x k plus 1 minus x k.

Good.

That's Newton's method.

Of course, e zero isn't quite true.

Well, zero will be true for the,

I'm constructing x k plus 1 here.

I'm constructing x k plus 1.

Okay.

So, let me just rewrite that night,

and we've got Newton's method.

So, we're looking for this change.

x k plus 1 minus x k.

I'll put it on this side as plus x k.

So, that's this.

Now, I have to invert that and put it on the other side of the equation.

So, that'll go with a minus.

This guy will be inverted.

And f at x k.

So, that's Newton's method.

Natural.

Let me just repeat that.

You see where the x k plus 1 minus x k is sitting.

Right?

And I moved f x k to the other side with a minus sign.

And then I multiplied through by j inverse.

So, I got that.

So, that's Newton's method.

For a system of equations.

But over there, I'm going to write down Newton's method

for minimizing a function.

This is such basic stuff that we have to begin here.

Let me even begin with a,

extremely straightforward example of Newton's method here.

Suppose my function,

I suppose I've only got one function actually.

Suppose I only have one function.

So, I'm, so suppose my function is x squared minus 9.

And I want to solve f of x equals 0.

I want to find this square root of 9.

Okay.

So, what is Newton's method for it?

We just should see.

My point is just to see how Newton's method is written.

And then rewrite it a little bit so that we see that convergence.

Okay.

So, of course, the Jacobian is 2x.

So, Newton's method says that xk plus 1.

I'm just going to copy that Newton's method minus 1 over 2xk.

Right?

That's the derivative.

Times f of xk, which is xk squared minus 9.

Okay.

We followed the formula where this determines xk plus 1.

And let's simplify it.

So, here I have xk minus that looks like a half of xk.

So, I think I have a half of xk.

And then this times this is 9.

I have of 1 over xk.

Is that right?

Half of xk from this stuff.

And plus 9.

I have of 1 over xk.

Okay.

Can I just check that I know the answer is 3.

Can I be sure that I get the right answer 3.

So, if xk was exactly 3, then of course, I expect xk plus 1 to stay at 3.

So, do what does that happen?

So, half of 3 and 9 half of 1.

Third, what's that?

Half of 3 and 9 half of 1.

Third.

That's 3 halves and 3 halves.

That's 6 halves and that's 3.

Okay.

So, we've checked that the method is consistent, which just means we kept the algebra straight.

But then the really important point about Newton's method is to discover how fast it converges.

So, now let me do xk plus 1 minus 3.

So, now I'm looking at the error, which is I hope approaching 0.

Is it approaching 0?

How quickly is it approaching 0?

These are the fundamental questions of optimization.

So, I'm going to subtract 3 from both sides.

Somehow.

Okay.

From here.

I guess I'm going to subtract 3.

Can I just, so I was just checking that it was correct.

Okay.

Now, so xk plus 1 minus 3.

I'm going to subtract 3 from both sides.

I'm going to subtract 3 there.

And then I hope that that box is what goes down here, right?

Subtracted 3 from both sides.

So, I'm looking at the, I'm hoping it now, hoping things go to 0.

Okay.

So, what do I have there?

Let me factor out the 1 over xk.

So, what do I have then left?

1 over xk.

So, there's a 9 halves from there.

1 over xk.

So, I really have a half of xk squared, because I've divided by an xk.

And this minus 3, I better put minus 3xk, because I'm dividing by the xk.

I claim that that's, now I've got it.

And let's see.

Let me take out the 2 to forget these 2s and make that a 6.

So, I have 1 over 2 xk times 9 plus xk squared minus 6.

Anything good about that?

We hope so.

We hope that that is something attractive.

So, this is again.

This is the error at step k plus 1.

And it's 1 over 2 xk times this thing in brackets 9 plus xk squared minus 6xk.

And we recognize that as xk minus 3 squared.

xk squared minus 6 of them plus 9, that's xk minus 3 squared.

Okay, that was the goal, of course.

That's the goal that shows why Newton's method is fantastic.

If you can execute it, if you can start near enough, notice that.

So, how do I describe this great equation?

It says that the error is squared at every step.

Square at every step.

So, if I'm converging to a limit, it will satisfy the, it will be 3.

Or I guess minus 3, is that possible?

Yeah, minus 3 is another solution here.

So, we've got 2 solutions.

The Newton's method could converge to 3.

Am I right? It could converge to minus 3.

So, I could, I'd have a similar equation sort of centered at minus 3.

Or what does it always do one of those?

It could blow up.

So, those are sort of regions of attraction.

They're all the starting points that approach 3.

And the whole point of that equation is with quadratic convergence.

The error being squared at every step.

It zooms in on 3.

Then there's a whole, all these starting points that would go to minus 3.

And then there's starting points that would blow up.

So, and those, maybe for this very simple problem,

that the picture is not too difficult to sort out those 3 regions.

But if we had, and this is allowing for a vector.

Two equations, or in equations, then we're in n variables.

And as a really, you get beautiful pictures.

You get some of the type of pictures that gave rise to these books on fractals.

The picture books on fractals.

For these basins of attraction, does the starting point lead you to one of the solutions,

or does it lead you to infinity?

Here, that would be interesting to just draw it for this.

Okay, but the essential point is the quadratic convergence.

If it's close enough, you see that it has to be close.

If xK is, if x0 is pretty near 3, then this is about 1, 6 of that.

And there'll be a good region of attraction in this case.

Okay, so that's a Newton's method for equations.

And now I want to do Newton's method.

I just want to convert all those words over to Newton's method for optimization.

So remember, the, these words were solving F equals 0.

This board is minimizing capital F, and what's the connection between them?

Well, of course, this corresponds to solving the gradient equals 0.

At a minimum, if I'm minimizing, I'm finding a point where all the first derivatives are 0.

So that'll be the match between these.

That this rad F in this picture is the small F in that picture.

Okay.

Now, I guess here I have, and this is sort of the heart of our applications to deep learning.

We have very complicated loss functions to minimize.

The functions of thousands are hundreds of thousands of variables.

Okay.

So that means that we would like to use Newton's method but often we can't.

So let me, so I need to put down here two methods.

One that doesn't involve those high second derivatives and one a Newton's that does.

So, so first I'll write down a method that does not.

So, so method 1.

And this will be steepest descent.

And what is that?

That says that xk plus 1, the new x, is the old x minus steepest descent means that I move into

the steepest direction, which is the direction of the gradient of F.

I move some distance and I better have freedom to decide what that distance should be.

So this is a step size S or in the language of deep learning.

It's often called the learning rate.

So obviously learning rate.

Okay.

So is that so, and it's natural to choose Sk.

We're going along, do you see where what this right hand side looks like?

I'm at a point in end dimensions.

We're all, we're in end dimensions here.

We have functions of end variables.

There's a vector.

There's a direction to move down the steepest slope of the graph.

And here's a distance to move.

And we will stop.

We'll have to get off this step.

If we stay on it, it'll swing, it'll take us off to infinity.

You would like to choose Sk so that you minimize capital F.

You take the point on this line.

So this is a line in Rn.

And for all the points on that line in that direction, F has some value.

And what you expect is that initially, because you chose it sensibly,

the value of the graph will drop.

But then at a certain point it will turn back on you and increase.

So that would be the natural stopping point.

That would call that an exact line search.

So exact line search would be exact line search is the best sense.

Of course that would take time to compute.

You're probably in deep learning.

That's time you can't afford.

So you fix the learning rate S.

Maybe you choose 0.01 to be pretty safe.

Okay.

So that's method one steepest descent.

Now, method two will be Newton's method.

So we have xk plus one minus something times delta F.

And now I'm going to do the right thing.

I'm going to live right here.

And the right thing is the Hessian.

The second derivative.

This was cheap.

We just took the direction and went along it.

Now we're getting really the right direction by using the second derivative.

So that's h inverse.

Okay.

And what I've done is to set that to 0.

So that would be.

You see that's Newton's method.

It's totally parallel to this guy.

Actually, I'm really happy to have these two on the board parallel to each other

because you have to kind of keep straight.

Are you solving equations or are you minimizing functions?

And you're using different letters in the two problems.

But now you see how they match.

The Jacobian of.

So again, the matches at think of F as the gradient of F.

That's the way you should think of it.

So the Jacobian for of the gradient is the Hessian.

The Jacobian of the gradient is the Hessian.

And that makes sense because we take the first derivative of the first derivative

of the second derivative.

Only we're doing matrix y.

So that's.

So the Jacobian of the gradient.

We're doing a vector matrix sentence instead of a scalar sentence.

The Jacobian of the gradient is a Hessian.

Yeah.

Right.

Okay.

So that's what I wanted to start with.

Just to get those basic facts down.

And so the basic facts were the three term Taylor series.

And then the basic algorithms followed naturally from it by setting F at the new point to zero.

If that's what you were solving.

Or by assuming you had the middle.

Right.

Good.

Good.

Okay.

Now what?

Now we have to think about solving these problems.

We have them studying.

Do they converge?

What rate do they converge?

Well that's the rate of convergence is like what I.

Why I took this separated off this example.

So the convergence rate for Newton's method will be quadratic.

The error gets squared.

And of course that means super fast convergence.

If you start close enough.

The rate of convergence for a steepest descent is of course not.

You're not squaring errors here because you're just taking some number instead of the inverse of the correct matrix.

So you can't expect super speed.

So linear rate of convergence would be right.

You expect that the error.

You would like to know that the error is multiplied by at every step by some constant below one.

That would be a linear rate compared to being multiplied by being squared at every step.

Okay.

And so this will be our basic.

This is the basic formula that we've built on.

For large scale.

For really large scale problems.

And there are methods.

Of course people are going to come up with methods that.

There's sort of a cheap Newton's method.

11burg Markwart.

And it's in the notes at the end of this section.

At the end of the six point four that we'll get to.

So 11burg Markwart is sort of cheap man's Newton's method.

It does not compute the Hessian.

But it says, okay, I, from the gradient I can see a sort of one term in the Hessian.

So it grabs that term.

But it's not fully second order.

Okay.

So now we have to think about problems.

And I guess the whole, the message here is.

And our starting point has to be convexity.

Convexity is the key word for this.

For these, for these problems.

For the function that we want to minimize.

If that's a convex function.

Well first of all, the convex function is likely to have one minimum.

And the picture that's in our mind of steepest descent is picture of a ball.

A ball is the graph of a convex function.

So I'm turning to convexity now.

I'll leave that board there because that's pretty crucial.

And speak about the idea of convexity.

Convex function.

Convex set.

So let's call the function f of x.

And the typical convex that will be, I'll call it k.

Okay.

So we just want to remember what does that word convex mean?

And how do you know if you have a convex function or a convex set?

Okay, let me start with convex set.

So because here's my general problem.

Convex minimization.

Which you hope to have and in many applications you do have.

So you minimize a convex function for points in a convex set.

So that's like the ideal situation.

That's the ideal situation.

Okay, get something on your side.

Something powerful convexity.

The function is convex.

And so let me draw a convex function.

The graph.

Okay, so I'll draw a convex function.

So a ball.

So that's a graph of f of x.

And then so here are the x's.

Let me put x1 and x2 in the base.

And the graph of f of x1 and x2 appear.

Okay, actually I'm over there.

I should be calling this function f, I think.

Is that right?

Yeah, little f would be the gradient of this guy.

Yeah, I think so.

Okay.

So now I'm minimizing over x and over certain x is not all x's.

I might be minimizing for example.

K might be the set where a x equals b.

K might be in that case a subspace.

Or a shifted subspace when I said subspace.

But then 1806 is reminding me in my mind that I only have a subspace when b is zero.

But you know the word for a subspace that sort of moved over.

So I'll just put that word down.

A bunch of words to learn for this topic.

But they're worth learning.

Okay.

So it's like a plane but not necessarily through the origin.

If b is zero, it doesn't go through.

If b is not zero, it doesn't go through the origin.

Okay.

Anyway, or I have some other convex.

Let me just put this convex at k in the base.

And did I make it convex?

I think pretty likely I did.

So now what's, well, the convex sets the constraint.

So this is the constraint.

Is that x must be x is in the set k.

Okay.

And I drew it as a convex blob.

Here was an example where it's a flat, not a blob, but a flat plane.

But let me come back to what this convex mean.

What's a convex set?

We have to do that.

Should have done that before.

In the notes, I had the fun of figuring out if I took a triangle.

Is that a convex set?

Let's just be sure.

So what's a convex set?

That is a convex set because if I take any two points in the set,

and draw the line between them, it stays in the set.

So that's convexity.

Any line from x1 to x2 stays in the set.

Okay, good.

So here's my little exercise to myself.

What if I took the union of two triangles?

All I wanted to get you to do is to sort of think visualize convex

and not convex possibilities.

Suppose I have one triangle even if it was obtuse.

That's still a convex, right?

No problem.

But now, what if I put those two triangles together?

Take their union.

Well, if I take them sitting with a big gap between, like I've lost.

I've never had a chance.

Because if I took one of my, if it was the union of these two,

well, you know what I'm going to say.

If I took that point, and that point, of course, it goes outside and stupid.

What about, but what if the triangles,

what if that triangle that lower triangle kind of overlaps the upper triangle?

Is that a convex set?

You're everybody's right, saying no.

How do I say that the union of those two triangles is not a convex set?

Guys, tell me where to pick two points.

Where the line goes out.

Well, I take one from that corner and one from that corner and the line between them went outside.

So union is usually not convex.

Well, if I think of the union of two sets, my mind actually automatically goes to the other corresponding possibility,

which is the intersection of the two sets.

So if I take the intersection of the two sets.

Now, what's the deal with that?

What was the, when I had two triangles, two separated triangles,

what can we say about the intersection of those two triangles?

It's empty.

So should we regard the empty set as a convex set?

Yes.

Isn't it?

Yes.

It's vacuous.

So it hasn't got any problems, right?

Okay.

But now is the intersection is always convex.

I'm assuming the two sets that we start with are.

Now, that's an important fact.

That the intersection of convex sets, let's just draw a picture that shows an example.

So what's the intersection, just this part, and it's convex?

Okay.

Can you give me a little proof that the intersection is convex?

So I take two points in the intersection.

Let me start the proof.

To test if something's convex, how do you test it?

You take two points in the set in the intersection.

And you want to show that the line between them is in the intersection.

Okay.

Why is that?

So take two points.

Take x1 in the intersection of two sets here.

And that's the symbol for intersection.

And we've got another point in the intersection.

And now we want to look at the line between them.

The line from x1 to x2, what's the deal with that one?

Is that fully in x1 in k1?

Why is it fully in k1?

I took two points in the intersection.

I'm looking at the line between them.

And I'm asking is it in the first set k1?

And the answer is yes.

Because those points were in k1 and k1's convex.

And is that line between them in k2?

Yes, same reason.

The two endpoints were in k2.

So the line between them is in k2.

So the intersection of convex sets is always convex.

The intersection of convex sets is convex.

Good.

So you'll see in the notes these possibilities with two triangles.

Sometimes you can take the union but not very often.

Okay.

Now what's the next thing I have to do?

Convex functions.

We got convex sets.

What are convex functions?

And we're good.

Right.

Because this is our prototype of a problem.

And I now would know what it means to be for that F to be.

I'm sorry.

I now know what it means for the set k to be convex set.

But now I have to look at the other often more important part of the problem.

What's the function I'm minimizing?

And I'm looking for functions with this kind of a picture.

Okay.

The coolest way is to connect the definition of a convex function

to the definition of a convex set.

This is really the nicest way.

It's a little quick.

It just switches by you.

Tell me.

Do you see a convex set in that picture?

Do you see a convex set in that picture?

That's a picture of a graph of a convex function.

It's a picture of a bowl.

Is the points on that surface?

Is that a convex set?

No.

Certainly not.

No.

But where is a convex set to be found here in that picture?

Yes.

Yes.

Yes.

The points on and above the bowl.

The inside the bowl we could say.

These points.

So convex function, yes.

A function's convex when the points on and above the graph.

Our convex set.

You could say, okay, mathematicians are just being lazy.

Having got one definition straight for a convex set.

Now, they're just using that to give a easy definition of a convex function.

Actually, it's quite useful for functions that could maybe equal infinity, sort of generalized function.

But it's not the quickest way to tell if the function is convex.

It's not our usual test for convex functions.

So now I want to give such a test.

Now I definition of convex function.

Of a smooth convex.

Yeah.

Yeah.

The, this fact, don't, I don't.

I don't.

I don't.

I don't.

I don't.

I don't.

I don't.

I don't.

I don't.

I don't.

I don't know if it's graph that the really official French name for the set above the graph is the epic graph.

But I won't even write that word down.

Okay.

Why do I come back to that for a minute?

Because I would like to think about two functions, F1 and F2.

Out of two functions, I can always create the minimum or the maximum.

So suppose I have two convex functions, convex functions F1 and F2.

Okay.

Then I could choose the minimum.

I could choose my new function.

So I call it little M for minimum.

M of X is the minimum of F1 and F2.

And I could choose a maximum function, which would be the maximum of F1 of X and F2 of X at the same point X.

If I, it's just natural, it's saying, okay, I have two functions.

I've got a bowl and I've got another bowl.

And suppose they're both convex.

So I'm just stretching you to think here.

I've got the graphs of two convex functions.

And I would like to consider the minimum of those two functions.

And also the maximum of those two functions.

I believe life is good.

One of these will be convex and the other won't.

And can you identify which one is convex and which one is not convex?

What about the minimum?

Is that a convex function?

So just look at the graph.

What is the minimum look like?

The minimum is this guy until they meet somehow on some surface and then this guy.

Is that convex?

We have like one minute to answer that question.

Absolutely no.

It's got this bad king in it.

What about the maximum of the two functions?

So the maximum is the one that's above all the points that are,

or things that are above or on there's the maximum function.

That was the minimum function.

It had a king.

The maximum function is like that.

And it is convex.

So maximum yes minimum no.

And we could have the maximum of 1500 functions.

If the 1500 functions are all convex, the maximum will be.

Because it's the part way above everybody's graph.

And that would be the graph of the maximum.

OK.

Good.

And now finally let me just say how do you know whether a function is convex?

How to test?

How to test?

OK.

So let me take just a function of one variable.

What's the test you learned in calculus?

Freshman calculus actually.

Just show that this is a convex function.

What's the test for that?

Second derivative should be positive or possibly zero.

So second derivative, greater equals zero.

Everyone.

That's this convex.

Final question.

Suppose f is a vector.

So this is a vector.

And so I have n functions of n variable.

No, I don't.

I have one function.

But I'm in n variables.

So this is what's the test for convexity test.

So it would be passed, for example, by x1 squared plus x2 squared.

Would it be passed by?

So here would be the question.

Would it be passed by x transpose some symmetric matrix s?

That would be a quadratic, a pure quadratic, would it be convex?

What would be the test?

I'm looking for an n-dimensional equivalent of positive second derivative.

The n-dimensional equivalent of positive second derivative is convexity.

And we have to recognize what's the test.

So I could apply it to this function, or I could apply it to any function of n variables.

And I maybe should be here.

What's the test here?

Here I have a matrix instead of a number.

So what's the requirement going to be?

Positive definite, or semi definite.

Or semi definite, just as here.

So the test is positive semi definite, Hessian.

And here the Hessian is actually that s because the second derivatives will produce,

I'll put a half in there.

The second derivatives will produce s equal to Hessian age.

So this here the s.

So positive semi definite, Hessian in general, second derivative matrix for a quadratic.

So it's convex problems that where we're going to get farther with.

We run into no saddle points.

We run into no local minimum.

Once we found the minimum, it's the global minimum.

These are the good problems.

OK.

Again, happy to see you today, and I look forward to Wednesdays.

Once we found the minimum, it's the global minimum.

These are the good problems.

OK.

Again, happy to see you today, and I look forward to Wednesdays.