Genetic Algorithms in Multivariate Email Optimization

Post by dlss » Sat, Mar 15, 2014, 2:54 PM UTC

erickerr wrote: post #3

dlss wrote: post #2

> The minimum threshold odds [to stop showing a variation] are calculated by 40% / number enabled variations Uh... How did you choose those cutoffs? Looks like you have a significant chance of making the wrong choice. Also: > Once enough data is collected to start making conclusions (1000 sends per variation) You should check out the Bayesian solution to the Multi-Armed Bandit problem. It's very close to what you are…

Thanks a lot for the feedback. This is a first pass implementation, but I agree that more thought should be put into the cutoff threshold, specifically for when there are only initially 2 (or maybe 3) variations. We considered a weighted decision approach but 1) were turned off by posts like http://visualwebsiteoptimizer.com/split-testing-blog/multi-a... and 2) wanted to keep moving parts to a minimum for V1. Any tho…

Hey Eric,

It's a good v1 for sure, congrats!

I would ignore any non-baysian MAB posts out there. The formulation used by other approaches is one that considers an infinite number of repeated trials, which is basically an insane assumption. Epsilon greedy and UCB1 aren't optimal except with that assumption.

You should check out:

  - http://www.economics.uci.edu/~ivan/asmb.874.pdf
  - https://www.youtube.com/watch?v=vz3D36VXefI
  - http://www.cs.cmu.edu/~deepay/mywww/papers/nips08-mortal.pdf (good benchmarks)

+1 that VWO's blog post is dumb :p

FWIW you are doing a weighted decision approach, it's just that you've constrained your weights to be either 0 or 1...

Cheers,

David

Re: Genetic Algorithms in Multivariate Email Optimization

Re: Genetic Algorithms in Multivariate Email Optimization