Black box optimization competition • Hacker News Board

Re: Black box optimization competition

#21

Post by darkmighty » Mon, Apr 27, 2015, 4:01 PM UTC

jpfr wrote: post #20

darkmighty wrote: post #19

Earlier quoted context omitted.

The point is, I don't even need to look up your techniques (although I did out of respect) to know there really isn't such a case; what I stated is a simple, almost trivial principle (apparently it has a name [1] as some pointed out). Mathematics models data, and you can't model without assumptions. It's like developing a theory which can't have axioms. For example, kernel regression probabilistic model is a terrible…

You are fighting a mathematically pure interpretation of black boxes that are making no assumptions at all. Your observations are correct. But nobody actually interprets the term "black box" the way you deem wrong. Taken from here [1]: White-box models: This is the case when a model is perfectly known; it has been possible to construct it entirely from prior knowledge and physical insight. Grey-box models: This is th…

Fair enough. I wasn't not familiar with the literature to be honest, it was just a remark.

I still dislike the term and concept, but it's hard to argue with a conventional definition. I believe assumptions should be made as clear as possible and the term seems like a futile attempt at hiding them.

Re: Black box optimization competition

#22

Post by murbard2 » Mon, Apr 27, 2015, 5:14 PM UTC

pilooch wrote: post #12

murbard2 wrote: post #10

It's a little strange that they do not have a track that gives gradient information, given that it is often a real world possibility. Also, this basically allows unlimited time between eval... So this becomes a contest about - coming up with a distribution over R^n -> R function - finding the optimal evaluation points to do Bayesian update I predict the winner will use some a mixture of Gaussian processes with variou…

You can compute the gradient, it just has a high budget cost. The usual winner is a flavor of CMA-ES, though they may have picked up the functions to avoid this.

You're missing my point. In many real world problems, it is cheap to compute the gradient. Thus, black box optimization methods which can use gradient information are inherently valuable, and it is surprising that they do not have a track that would allow showcasing those.

Re: Black box optimization competition

#23

Post by nullc » Wed, Apr 29, 2015, 1:09 AM UTC

murbard2 wrote: post #22

pilooch wrote: post #12

Earlier quoted context omitted.

You can compute the gradient, it just has a high budget cost. The usual winner is a flavor of CMA-ES, though they may have picked up the functions to avoid this.

You're missing my point. In many real world problems, it is cheap to compute the gradient. Thus, black box optimization methods which can use gradient information are inherently valuable, and it is surprising that they do not have a track that would allow showcasing those.

In a great many real world problems, including most of the most expensive ones gradients are _not_ available, or can only be expensively computable... even if your objective is differentiable, automatic differentiation isn't cheap on non-trivial functions.

Experiences differ, but in mine the most common place to find objectives with gradients is in optimizer challenges.

That said; sure, there should be a track that gives you the gradients. I agree that it would be nice if there were another track.

Re: Black box optimization competition

#24

Post by nullc » Wed, Apr 29, 2015, 1:13 AM UTC

cshimmin wrote: post #3

Wish I had seen something about this sooner. The competition began in January and ends on the 30th of this month.

That was my thought too! but instead of even click on the HN comments I went and wrote a contestant. Within a couple hours all my runs will have completed; assuming no big power failure I'll make the deadline! :)

(Uh, no doubt I won't do well, since I had no time to ... like.. actually test my code on any functions except a couple trivial trials. :P ... I hope they put up some kind of ranking information as soon as it closes. I have no idea if my results are awful or merely bad :) (and I probably shouldn't share best numbers before it closes) )

Re: Black box optimization competition

#25

Post by nullc » Wed, Apr 29, 2015, 1:17 AM UTC

rer0tsaz wrote: post #7

obstinate wrote: post #2

Seems really interesting. Too mathy for my skillset. If I may, I propose that the organizers remove the restriction on disassembling the client library or intercepting network connections. This restriction seems like it cannot benefit the organizers, unless the protocol is insecure. People are going to ignore this rule anyway, and you can't stop them or even detect them doing it. So why put it in there? It's only goi…

It's probably insecure, because you don't want to do 75,850,000 sequential evaluations over a network. It would take over a week for a single track with even just 10ms response time.

They run over the network (no, I wasn't trying to cheat, but while stracing to debug my own code, I could see it sending for each query).

Each track is split up into 1000 tests; you can run the separate tests concurrently; which more or less eliminates the latency as the largest test permits 'only' about 400k queries.

Re: Black box optimization competition

#26

Post by nullc » Wed, Apr 29, 2015, 1:34 AM UTC

darkmighty wrote: post #19

jpfr wrote: post #17

Earlier quoted context omitted.

Yes, there is such a thing. There exist many more techniques than trivially assuming some "template" function and fitting the function parameters against the data. Have a look at nonparametric modelling techniques. For example kernel regression or gaussian processes. You either don't make any assumptions, or you take an uninformative prior that distributes over all possible results. This competition evokes modelling,…

The point is, I don't even need to look up your techniques (although I did out of respect) to know there really isn't such a case; what I stated is a simple, almost trivial principle (apparently it has a name [1] as some pointed out). Mathematics models data, and you can't model without assumptions. It's like developing a theory which can't have axioms. For example, kernel regression probabilistic model is a terrible…

Look at it this way: Many interesting problems in engineering have expensive to evaluate objectives with generally unknown structure and noisy multi-modal results, but are still piece-wise smooth. It's true that in the space of all possible functions virtually none meet these criteria, but many practically interesting ones do.

If your function really is some a random oracle, then, indeed, no optimizer will do well against it. OTOH, none will do (relatively) poorly either.

Effective optimization techniques can explore a function generally and exploit similarities to known models or at least any smoothness they can find. Ineffective techniques will just it caught in local minima or fail to exploit smoothness or "obvious" structure.

Powerful "generic" optimizers are a tool which is important for industry. But the common ways they are benchmarked potentially allows for overfitting in the design phase, this contest is intended to correct that, and provide a potentially better assessment of how general these optimizers are.