Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Convex Optimization (2004) [pdf]

web.stanford.edu

Post reply on HN
Print view View original

51–56 of 56 posts

Re: Convex Optimization (2004) [pdf]

#51

Post by gms7777 »

Solvency wrote: post #49
FooBarBizBazz wrote: post #44

Earlier quoted context omitted.

> This is one of those things where the math is actually pretty simple, but the notation is incredibly opaque If you're not used to it, this kind of notation looks like hieroglyphics. If you are used to it, every vague English-language technical document you see floating around your workplace just reads like a bunch of flailing-arm hand-waving.

What do you mean by that?

Usually, language that tries to simplify or put things in layman's terms is missing important detail for fully specifying the problem. It's like watching the 3 minute version of a recipe on YouTube and thinking you have a good understanding of how it works, but then when you go to make it, you realize they didn't tell you if they used whole-wheat or all-purpose flour, or if they bake at 350 or 450, or in a glass or aluminum pan. If you try to follow the recipe, you might end up with something significantly different. It's not that the quick, intuitive version isn't useful, you can gain a lot of insight and get a high level picture often much quicker than you would reading the complete, fully specified recipe. But if what you want to do is reproduce the recipe exactly, you need the fully specified version. Personally when I'm writing technical documents, I prefer to include both the intuitive overview and the fully specified technical version.

Re: Convex Optimization (2004) [pdf]

#52

Post by tomrod »

dfan wrote: post #9
Solvency wrote: post #5

"A mathematical optimization problem, or just optimization problem, has the form minimize f0(x) subject to fi(x) ≤ bi , i = 1, . . . , m. (1.1) Here the vector x = (x1, . . . , xn) is the optimization variable of the problem, the function f0 : R n → R is the objective function, the functions fi : R n → R, i = 1, . . . , m, are the (inequality) constraint functions, and the constants b1, . . . , bm are the limits, or…

From the introduction: "The only background required of the reader is a good knowledge of advanced calculus and linear algebra. If the reader has seen basic mathematical analysis (e.g., norms, convergence, elementary topology), and basic probability theory, he or she should be able to follow every argument and discussion in the book." It's a graduate-level course. If that paragraph is arcane, the book is probably a f…

We used the book in an advanced undergrad course. It's definitely doable once you have some real analysis and matrix analysis under your belt.

Re: Convex Optimization (2004) [pdf]

#53

Post by TrackerFF »

uptownfunk wrote: post #15

Does this have any application to SOTA ML?

Dunno if support vector machines are still considered "SOTA", but that would be one of the most obvious examples of a convex optimization problem (and probably a good starting point for students trying to tie ML and convex optimization)

Re: Convex Optimization (2004) [pdf]

#54

Post by cpgxiii »

Convex optimization can be a really amazing tool. We use optimization extensively, both on actually convex problems, and on non-convex-but-practically-solvable problems in robotics.

The math surrounding optimization is great; however, the reality of optimization tools is still very poor. A competitive optimizer is a massive project, and outside of a number of mostly limited/specialized solvers, the effective tools are all proprietary and very expensive (e.g. SNOPT, Gurobi, Mosek, CPLEX, etc). How solveable and stable your problems can be depends on these tools, and effective problem formulation (e.g. what and how many constraints you use, how you compute gradients) is essentially a black art learned through hard experience.

There's a great example of the complexity difference between optimization and other tools in the world of motion planning for robots: we expect that any semi-competent undergrad can implement search- and sampling-based planners (e.g. A* or RRT), implementing a good optimizer for trajectory optimization is a multi-million dollar project.

The world of optimization desperately needs a MuJoCo-DeepMind moment, where a large interested company buys one of the major commercial optimization providers and makes their tools free and open source. This would really be transformative to the field.

Re: Convex Optimization (2004) [pdf]

#55

Post by i_don_t_know »

optbuild wrote: post #50
fiforpg wrote: post #20

Earlier quoted context omitted.

Come on, prior to this people read Nocedal & Wright, which is still very much a standard text on nonlinear optimization, and there were well-known implementations of nonlinear optimization algorithms written by these people in Fortran. These are most likely hiding in any modern LBFGS library you are looking at, including Scipy etc. It is rather that more people understand these algorithms now and more people wrote im…

Which book by Nocedal and Wright? Can someone link to it?

https://link.springer.com/book/10.1007/978-0-387-40065-5

Re: Convex Optimization (2004) [pdf]

#56

Post by FooBarBizBazz »

Solvency wrote: post #49
FooBarBizBazz wrote: post #44

Earlier quoted context omitted.

> This is one of those things where the math is actually pretty simple, but the notation is incredibly opaque If you're not used to it, this kind of notation looks like hieroglyphics. If you are used to it, every vague English-language technical document you see floating around your workplace just reads like a bunch of flailing-arm hand-waving.

What do you mean by that?

Heh. I make a comment about unclear/inexplicit communication... using a metaphor that itself has little specific concrete meaning. And you write this reply. "Oh no, I have been deconstructed!", I say, laughing, in the voice of the Wicked Witch of the West as she is melting....

Or maybe you were really asking. That's the thing about the Internet. Only the FBI knows you're a dog, and nobody knows what the dog really means.

But if it really was a question, gms7777's sibling reply is good.

Post reply on HN