Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

blog.demofox.org

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#31

Post by densh »

jhsvsmyself wrote: post #28
wenc wrote: post #17

I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basically techniques on how to conduct experiments efficiently and reducing the number of runs, thus reducing overall cost).... yet I've never had occasion to ever apply it. One area where I feel it might be relevant to my work is in hyperparameter tuning, especia…

I just took this class last semester. I've been trying to find things to do with what I learned but I haven't found anything yet.

For coffee affectionados out there: picking best coffee setup settings is very similar to a physics experiment, and you can totally use the knowledge of experiment design theory there.

Settings:

* Kinds of beans at various levels of roasting available to you.

* Grind size on a grinder with controllable grind size in steps.

* Ratio of ground coffee weight to water weight used for brewing.

* For filter coffee: preinfusion time, brew time or flow rate (see V60 brewer).

* For espresso coffee: preinfusion time & pressure, brew time & pressure or flow rate (see Flair Espresso).

* Water temperature.

* Water mineral composition.

* For milk-based coffee drinks: kinds of milk, milk percentage as ratio of the total drink volume, steaming duration, final milk temperature.

All of these create a huge factorial space of possible configurations, especially if you've go into flow or pressure profiling. If you frame it as an experiment you might isolate some of the variables that make the most impact tailored to your coffee preferences.

And here is an example of a scientifically inclined paper in this space if you went to get inspired: https://www.cell.com/matter/fulltext/S2590-2385(19)30410-2?_...

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#32

Post by onos »

wenc wrote: post #17

I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basically techniques on how to conduct experiments efficiently and reducing the number of runs, thus reducing overall cost).... yet I've never had occasion to ever apply it. One area where I feel it might be relevant to my work is in hyperparameter tuning, especia…

Random search beats grid search!

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#33

Post by samvher »

wenc wrote: post #17

I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basically techniques on how to conduct experiments efficiently and reducing the number of runs, thus reducing overall cost).... yet I've never had occasion to ever apply it. One area where I feel it might be relevant to my work is in hyperparameter tuning, especia…

I'm in biotech (as a software engineer/data scientist) and here this stuff is used a lot. Experiments are very hands on and expensive in terms of inputs so you want to be efficient with your use of them.

Bayesian approaches (e.g. optimization with Gaussian processes) tend to be very serial in nature: later sampling points are decided from the information you get earlier on. But these biological experiments can take days or weeks so you want to run them in parallel as much as possible. DoE is excellent for that.

I'm speccing out a D-optimal design as we speak (though I'm not sure this is what we will use for the particular project I'm on).

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#34

Post by samvher »

thereisnospork wrote: post #25
wenc wrote: post #17

I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basically techniques on how to conduct experiments efficiently and reducing the number of runs, thus reducing overall cost).... yet I've never had occasion to ever apply it. One area where I feel it might be relevant to my work is in hyperparameter tuning, especia…

I've at times made fairly extensive use of d-optimal designs in formulation type research and development, usually in an iterative process, pruning and adding variables as well as refining ranges. Lot of footguns, but the approach can drastically decrease optimization time. In my experience it needs the right combination of: too many factors to make full factorial reasonable, too much (expected) covariance to trust s…

What are the footguns you've run into with D-optimal designs?

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#35

Post by cgearhart »

wenc wrote: post #17

I studied Design of Experiments (DOE) in college -- it was part of the core curriculum in my major. The content feels like it ought to be very useful (it's basically techniques on how to conduct experiments efficiently and reducing the number of runs, thus reducing overall cost).... yet I've never had occasion to ever apply it. One area where I feel it might be relevant to my work is in hyperparameter tuning, especia…

I used a lot of these techniques as an operations research analyst in the early 2010’s.

I also wondered the same thing about DoE for hyperparameter tuning. Always felt like it was a case of grid search/random search/bayes opt being “good enough” and easier. But maybe DoE would be worth it for something like LLMs where training runs for a month on 25k GPUs.

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#36

Post by thereisnospork »

samvher wrote: post #34
thereisnospork wrote: post #25

Earlier quoted context omitted.

I've at times made fairly extensive use of d-optimal designs in formulation type research and development, usually in an iterative process, pruning and adding variables as well as refining ranges. Lot of footguns, but the approach can drastically decrease optimization time. In my experience it needs the right combination of: too many factors to make full factorial reasonable, too much (expected) covariance to trust s…

What are the footguns you've run into with D-optimal designs?

Mainly around bounding variables, especially when coupled with overly ambitious scope/number of tests. The short version is that its really easy to wind up with say a 80 experiment test where half or more of the combinations are invalid or perform poorly to the degree they don't adequately lend confidence to the prediction in the region of the global maximum. Leads to wasted work and the poor predictive ability doesn't lead well into focused future work. There are also the usual statistical foot guns of p-hacking and such.

For a concrete example, consider a bread baking optimization, considering time, temperature, and baking soda %. This gives a cubic design space, for which the naively (ignoring expected covariance of temperature*time) optimal design for 9 experiments is the cube's corners + the center point. If, for instance, your predetermined t_max always results in a briquette instead of a loaf, ~half of your experimental data is going to be worthless.

With more nuance, and temp_max is genuinely the highest reasonable temperature there are still two problems:

a. covariance is likely to drive combinations into 'invalid' territory (e.g. temp_max + time_max is likely to be invalid, or 2/9ths of your experimental. temp_min / time_min is also likely invalid for another 2/9ths).

b. predictive power / linearity of response over the ranges specified. Even if the combinations aren't invalid, if they are all the min/max combinations are poorly performing (due to overly wide, but valid, boundaries) you can wind up with poor predictive performance.

Covariance can be accounted for, design space trimmed (e.g. to a cube with a corner or two cut off), and bounds set conservatively but that is all tricky manual intervention that relies on knowledge of the problem domain and scaling factors of the underlying physics. When the problem isn't well understood it is easy to make errors in assumptions, those errors have a high cost, and if the problem was well understood a DOE probably wouldn't be necessary.

edit: for a less trivial example of a suitable problem for a d-optimal DOE, but with tricky bounding / underlying physics, consider: a 4 part formulation of fumed silica, cyanoacrylate, isopropanol, and water to make a gap-filling/quick-setting adhesive.

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#37

Post by pfedak »

penteract wrote: post #9

Something about this doesn't make sense - from the C=A case: > we wouldn’t be able to tell the difference in results between only doing option B (doubling the butter) and doing all three options: adding an egg, doubling the butter, and adding nuts on top. However, the experiments listed include exactly those 2 experiments (Test 1 and Test 3 in the table with a column titled C=A), so unless I'm missing something, "wou…

As best I can tell from reading https://www.stat.purdue.edu/~yuzhu/stat514s2006/Lecnot/fracf... the unstated assumption here is that we're going to do linear regression where (critically) the "-" case for each condition is -1, and the "+" is +1. This has the surprising-to-me effect of making "I", which looks like it might be the control group based on notation, actually a positive recipient of AC interaction (and any even-order interaction). You can think of this as a change in basis in how you parse out the effects, where we're talking about

1 -1

-1 1

(like a covariance matrix) instead of

0 0

0 1

for an interaction.

I have a gut feeling it's done this way mostly because the tools being used expect things to be expressed this way rather than any conscious choice by experimenters. Through this lens, if you test

I, B, AC, ABC

every experiment has a positive effect from AC interaction, and taking B-I, which we might think of as the effect from B, is in this paradigm also sensitive to the ABC interaction and the AB and BC interactions. The "real" effect from B would be approximated as (B + ABC - AC - I)/2, which is exactly the same as the effect from ABC interaction (which is positive when an odd number of its constituents are positive...).

I'm pretty sure this is just a difference in mathematical perspective - you can represent exactly the same data, but the coefficients (i.e. effect values) will change, and there's a different notion of what you know vs don't know. Maybe there's a more convincing reason to do this when you have more than two "levels", but from the presentation in TFA it just feels like overcomplicating things with a confusing prior about how effects work.

It also seems like the given example is just bad. If the parameters are numeric and there's not a reasonable "control", this perspective feels much more natural.

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#38

Post by penteract »

pfedak wrote: post #37
penteract wrote: post #9

Something about this doesn't make sense - from the C=A case: > we wouldn’t be able to tell the difference in results between only doing option B (doubling the butter) and doing all three options: adding an egg, doubling the butter, and adding nuts on top. However, the experiments listed include exactly those 2 experiments (Test 1 and Test 3 in the table with a column titled C=A), so unless I'm missing something, "wou…

As best I can tell from reading https://www.stat.purdue.edu/~yuzhu/stat514s2006/Lecnot/fracf... the unstated assumption here is that we're going to do linear regression where (critically) the "-" case for each condition is -1, and the "+" is +1. This has the surprising-to-me effect of making "I", which looks like it might be the control group based on notation, actually a positive recipient of AC interaction (and any…

Thanks for the explanation and the link.

So the goal is to find real numbers f_I, f_A, ...f_ABC such that

result = f_I + f_A*v_A + f_B*v_B + f_AB*v_A*v_B + ... + f_ABC*v_A*v_B*v_C

where v_A is 1 or -1 depending on whether A is present or absent in the experiment. f_AB is being abbreviated to AB, which is causing some confusion, since when heading a column, AB means v_A*v_B. The article should say that we can't tell the difference between the effect associated with B and the effect associated the 3-way interaction (for this definition of the effect associated with the 3-way interaction).

Re: Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

#39

Post by nxobject »

mtu wrote: post #16
esafak wrote: post #4

Does any experimentation platform support this?

JMP is used industrially for design of experiments: https://www.jmp.com/en_us/software/capabilities/design-of-ex... R also seems to have some libraries for things related to design of experiments.

JMP is also a very criminally underrated and flexible tool to do data analysis with – both exploratory and modelling. It's my preferred tool of choice over R, especially after they added a structural equation modelling platform.

It's a pity they got rid of pricy perpetual licensing quite a few years ago for a no less pricy subscription model.

Post reply on HN