Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Black box optimization competition

bbcomp.ini.rub.de

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Black box optimization competition

#11

Post by enkico »

darkmighty wrote: post #5

I really dislike the term "Black box optimization". There's no such thing. You have to make assumptions about your function, so in the end this is just rewarding people whose optimizers happen to match the chosen functions; but those functions are not made explicit whatsoever. That doesn't make any sense. For example, if the output/input are floating point numbers than you can assume the domain/range is [-M,M]. Other…

what is "typical" for one can be "rare" for the other, "black box" suggests that the participants do not know what is inside, the organizers on their side should make sure that the content is of some interest for the "real-world problems/applications"

what you are describing is related to the "no free lunch theorem", something one can attempt to deal with to get things working "in practice"

Re: Black box optimization competition

#12

Post by pilooch »

murbard2 wrote: post #10

It's a little strange that they do not have a track that gives gradient information, given that it is often a real world possibility. Also, this basically allows unlimited time between eval... So this becomes a contest about - coming up with a distribution over R^n -> R function - finding the optimal evaluation points to do Bayesian update I predict the winner will use some a mixture of Gaussian processes with variou…

You can compute the gradient, it just has a high budget cost.

The usual winner is a flavor of CMA-ES, though they may have picked up the functions to avoid this.

Re: Black box optimization competition

#13

Post by pilooch »

darklajid wrote: post #9

Whoa. The servers for this competion are about 8km away. That's the most 'local' content I've ever seen on HN. Unfortunately I have to agree with obstinate here. The pure math is too much for me and reverse engineering (still daunting, but interesting/possible) is not acceptable. If any HN person wins this contest, I offer beers close to the black box :)

Shameless plug: anyone interested should be able to get baseline results and above easily by using libcmaes. I am one of the authors with no time to compete, but am interested in reports on how it goes. Also if you are a researcher or a student the lib should let you experiment easily with various custom strategies.

https://github.com/beniz/libcmaes

Re: Black box optimization competition

#14

Post by darkmighty »

enkico wrote: post #11
darkmighty wrote: post #5

I really dislike the term "Black box optimization". There's no such thing. You have to make assumptions about your function, so in the end this is just rewarding people whose optimizers happen to match the chosen functions; but those functions are not made explicit whatsoever. That doesn't make any sense. For example, if the output/input are floating point numbers than you can assume the domain/range is [-M,M]. Other…

what is "typical" for one can be "rare" for the other, "black box" suggests that the participants do not know what is inside, the organizers on their side should make sure that the content is of some interest for the "real-world problems/applications" what you are describing is related to the "no free lunch theorem", something one can attempt to deal with to get things working "in practice"

The organizers making sure it has some real world relevance is what I would equate with problems being "typical". In practice, you may find ill-characterized "Typical problems" and solve them, but as I said, a truly "Black box optimization" would not make sense; hence I dislike the term (and the general problem statement).

Re: Black box optimization competition

#15

Post by silentvoice »

darkmighty wrote: post #5

I really dislike the term "Black box optimization". There's no such thing. You have to make assumptions about your function, so in the end this is just rewarding people whose optimizers happen to match the chosen functions; but those functions are not made explicit whatsoever. That doesn't make any sense. For example, if the output/input are floating point numbers than you can assume the domain/range is [-M,M]. Other…

Maybe a better term should be "blind" rather than black-box. I think the goal is simply to hold optimization to the same level of reproducibility that is expected of most scientific fields today, and if a researcher is allowed to introduce a hundred tunable parameters that makes their algorithm converge on all the standard test cases then they haven't created a reproducible optimizer - they have created a benchmark solver.

Re: Black box optimization competition

#16

Post by ramgorur »

1. You do not know what the function looks like, even there is no gradient information

2. You have a fixed number of probes M

2. Among M, You have N number probes to get the silhouette of the function (exploration).

3. Then from the rest of the (M - N) trials, you need to find the optima (exploiation).

Sounds more like a pseudo-science than a math problem to me.

Re: Black box optimization competition

#17

Post by jpfr »

darkmighty wrote: post #5

I really dislike the term "Black box optimization". There's no such thing. You have to make assumptions about your function, so in the end this is just rewarding people whose optimizers happen to match the chosen functions; but those functions are not made explicit whatsoever. That doesn't make any sense. For example, if the output/input are floating point numbers than you can assume the domain/range is [-M,M]. Other…

Yes, there is such a thing.

There exist many more techniques than trivially assuming some "template" function and fitting the function parameters against the data.

Have a look at nonparametric modelling techniques. For example kernel regression or gaussian processes. You either don't make any assumptions, or you take an uninformative prior that distributes over all possible results.

This competition evokes modelling, optimisation and the exploration/exploitation tradeoff. I'm sure there will be very interesting theory behind the winning entries...

Re: Black box optimization competition

#18

Post by Houshalter »

ramgorur wrote: post #16

1. You do not know what the function looks like, even there is no gradient information 2. You have a fixed number of probes M 2. Among M, You have N number probes to get the silhouette of the function (exploration). 3. Then from the rest of the (M - N) trials, you need to find the optima (exploiation). Sounds more like a pseudo-science than a math problem to me.

Huh? Who said it was a math problem? And pseudoscience? Most real world optimization problems are like this. Sometimes don't get gradient information or unlimited trials.

The point of the task is to reward methods that work efficiently with limited trials and domain information, rather than who can run hillclimbing on the biggest computer or hand tune the parameters the best.

Re: Black box optimization competition

#19

Post by darkmighty »

jpfr wrote: post #17
darkmighty wrote: post #5

I really dislike the term "Black box optimization". There's no such thing. You have to make assumptions about your function, so in the end this is just rewarding people whose optimizers happen to match the chosen functions; but those functions are not made explicit whatsoever. That doesn't make any sense. For example, if the output/input are floating point numbers than you can assume the domain/range is [-M,M]. Other…

Yes, there is such a thing. There exist many more techniques than trivially assuming some "template" function and fitting the function parameters against the data. Have a look at nonparametric modelling techniques. For example kernel regression or gaussian processes. You either don't make any assumptions, or you take an uninformative prior that distributes over all possible results. This competition evokes modelling,…

The point is, I don't even need to look up your techniques (although I did out of respect) to know there really isn't such a case; what I stated is a simple, almost trivial principle (apparently it has a name [1] as some pointed out).

Mathematics models data, and you can't model without assumptions. It's like developing a theory which can't have axioms. For example, kernel regression probabilistic model is a terrible model (assumption) with very large error for a large class of distributions[2], and so on. We're talking about picking the best technique; this technique is going to pick some assumptions arbitrarily that will or will not work well based on an unclear choice of the organizers. That's why I would prefer if they stated instead "Functions with some real world relevance", or "Typical functions", or maybe "Poorly behaved functions", and so on.

[1] http://en.wikipedia.org/wiki/No_free_lunch_in_search_and_opt...

[2] On the wikipedia page you can see they do make assumptions on f to minimize the squared error for choosing the kernel. It's inevitable.

Re: Black box optimization competition

#20

Post by jpfr »

darkmighty wrote: post #19
jpfr wrote: post #17

Earlier quoted context omitted.

Yes, there is such a thing. There exist many more techniques than trivially assuming some "template" function and fitting the function parameters against the data. Have a look at nonparametric modelling techniques. For example kernel regression or gaussian processes. You either don't make any assumptions, or you take an uninformative prior that distributes over all possible results. This competition evokes modelling,…

The point is, I don't even need to look up your techniques (although I did out of respect) to know there really isn't such a case; what I stated is a simple, almost trivial principle (apparently it has a name [1] as some pointed out). Mathematics models data, and you can't model without assumptions. It's like developing a theory which can't have axioms. For example, kernel regression probabilistic model is a terrible…

You are fighting a mathematically pure interpretation of black boxes that are making no assumptions at all. Your observations are correct. But nobody actually interprets the term "black box" the way you deem wrong.

Taken from here [1]:

White-box models: This is the case when a model is perfectly known; it has been possible to construct it entirely from prior knowledge and physical insight.

Grey-box models: This is the case when some physical insight is available, but several parameters remain to be determined from observed data. It is useful to consider two subcases.

1. Physical modeling: A model structure can be built on physical grounds, which has a certain number of parameters to be estimated from data. This could, for example, be a state-space model of given order and structure.

2. Semiphysical modeling. Physical insight is used to suggest certain nonlinear combinations of measured data signal. These new signals are then subjected to model structures of black-box character.

Black-box models: No physical insight is available or used, but the chosen model structure belongs to families that are known to have good flexibility and have been 'successful in the past'.

[1] http://www.sciencedirect.com/science/article/pii/00051098950...

Post reply on HN