Whom the gods would destroy, they first give real-time analytics (2013)
1–10 of 66 posts
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#2[dead]
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#3If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#4If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?
MAB and its friends like contextual MAB has always been the dream. Closing the loop so analytics data is pushed back to the decision point in code and isn't a one-way pipe to some dashboard is the hardest part though. For non-technical reasons.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#5If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?
MAB and its friends like contextual MAB has always been the dream. Closing the loop so analytics data is pushed back to the decision point in code and isn't a one-way pipe to some dashboard is the hardest part though. For non-technical reasons.
Sort of a generalized PEBCAK
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#6If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?
How well does that dodge the problem? I'd imagine a multi armed bandit should stay such that it is always sampling from many fair coins, as it were. I would be delighted to read a study on that.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#7And… near real-time with high uptime is relatively more costly to build / maintain / deploy / operate than batch — so save your org the cost!
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#8Just use Amplitude
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#9Just use Amplitude
Last I used Amplitude it was insanely expensive. Is that not still the case?
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#10I also think real time is mostly useless (aside from for alerting, which probably is a different tool), but I don't think the one day delay is much of (if any) protection against the experimental pitfalls described.