Fractional Factorial Experiment Design: There Are Too Many Experiments to Do

Let's say you're testing different formulations for cake but because you don't have time to do the full cartesian product of all possible variations over sugar content x fat content x type of flour, you just pick a handful, and it turns out that every time you're testing a high-sugar variation you are also testing spelt flour -- that's an alias because if you really like these particular cakes there's no way to know whether it was the spelt or the sugar that did it.

What experimental design brings to the table is a disciplined way to figure out what variables will be aliased and to make sure that these aliases are mostly harmless, either with the help of subject matter knowledge (you can taste sugar so it's fine if it's aliased with other things) or mathematically (let's try to avoid aliasing sugar content with fat content, but instead alias sugar content with the interaction effect of sugar, fat and flour type because that higher-order interaction is unlikely to matter over and above the first-order effects.)

Fractional designs in particular are typically used in agriculture and industrial settings, places where you want to try to optimize over very many factors at once but cannot afford to test every variation. It is not common in web analytics because (1) we usually assume that one particular change to a site or app will be independent of another change elsewhere, so there is no need to test them concurrently to see if a particular combo stands out and (2) if we did want to test combinations of variables concurrently, there's usually enough users or visitors to just test the entire grid and not worry about picking a selection of variations.

> Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much.

I guess you're right in that if there were very many interactions between variables, then the aliasing of a fractional design would be an incredible nuisance, but on the other hand if there were no dependence between any of the variables at all then there would be no point to any sort of factorial design as you could just test each variable sequentially.

> Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs)

I have to admit, I have never ever seen anyone calculate a D-optimal or G-optimal design for anything -- that said, I only graduated as a statistician 6 or 7 years ago. As you say, there might be some use to it for continuous variables but for factors, given that latin squares (etc.) are known to be optimal so you can just grab that off the shelf.

I've at times made fairly extensive use of d-optimal designs in formulation type research and development, usually in an iterative process, pruning and adding variables as well as refining ranges. Lot of footguns, but the approach can drastically decrease optimization time. In my experience it needs the right combination of: too many factors to make full factorial reasonable, too much (expected) covariance to trust single-vaeiable optimization, and cheap enough to commit to doing 8+ experiments before a major evaluation.

Similarly:

> Lastly, I = AC means that we can’t tell the difference between doing nothing at all, compared to adding an egg and adding nuts on top.

But test 0 and test 2 are doing nothing and adding an egg & nuts respectively. What are we missing here?

On further reading, it seems like "doing all three options" should be interpreted to mean "the interaction of all three options"? So we aren't able to tell if an improvement of taste comes from B alone or from the interaction of A, B, and C. I'm mostly guessing though.