Edit for clarity:
When I say you had to know how to program I mean you know to know the environment and not fail. It’s not about knowing some tool, framework, or pretend gimmick. If your test executes properly you are helping the employer form massive revenue making decisions. When it break you break existing functionality in production and nobody will forgive you. If you know what you are doing you can squeeze absolutely anything you can imagine in an A/B test and most of it can be tested in a browser console.