Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask Google to choose your next domain

rvmenu.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Ask Google to choose your next domain

#21

Post by KnowledgeSponge »

sillysaurus2 wrote: post #11
joelrunyon wrote: post #9

I spent a lot of money on Google in the RV industry (to the tune of $500,000/month for a certain fortune 500 company) & there are a lot of issues with this post: First up, his stats across the board aren't even close. His impressions vary by a factor of 4 for the different ads. From that alone, I bet he forgot to turn of the "optimize my ads" setting from Google and switch on "rotate evenly". The "optimize my ads" fe…

Completely offtopic, but I can't resist: What's it like being in a position to choose how $500k per month is spent? Wow. That's equivalent to about forty engineers' salaries per year. EDIT: I visited your website and am not sure what to think. http://joelrunyon.com/ Is it true that those sorts of marketing tactics will convince people to entrust you with how to spend $500k/mo? (Personal branding is rather important.…

Not the parent, but I managed and eventually helped lead the paid search group at one of the top search agencies. We had clients with 7 figure budgets all the way up to 9 figure budgets (or essentially unlimited as long as we were within certain parameters).

It's a bit scary and heart attack inducing if something goes majorly wrong (like going dark for a day because someone forgot to check the flight dates in the account after you took it over from the previous agency). But like the OP said, it just becomes numbers.

The cool thing is you not only get much better support and escalation paths at Google, but you can test things at a very large scale and get answers VERY quickly. And little things that might not make a huge difference on smaller accounts, suddenly equate to tens or hundreds of thousands of dollars difference in total revenue. But again, all things are relative, so %'s might not change much.

Re: Ask Google to choose your next domain

#22

Post by KnowledgeSponge »

sp332 wrote: post #2

The ads started on January 2 and took 6 days to hit 98% statistical confidence. No! You have to pick the time first, wait that amount of time, and later analyze how much confidence you achieved. With random variations, you could have a run of any size of positive or negative results, and you could just wait until you get the result you want. Sorry but tons of people make this mistake with A/B tests.

So since Optimizely basically lets the test run until it "reaches statistical significance" with their Chance to Beat Baseline number, would you say they are doing it wrong?

Methodology here: https://help.optimizely.com/hc/en-us/articles/200039895-Chan...

I've often run tests that seem to be in the 90%+ chance to beat baseline, but the graphs just didn't look like they were finished or had enough data, so I'd let it run a bit longer. Sure enough, big changes in percent lift, and also big changes in confidence dropping down below 90% (sometimes MUCH lower), and occasionally coming back up.

Re: Ask Google to choose your next domain

#23

Post by 204NoContent »

sp332 wrote: post #8
204NoContent wrote: post #5

Earlier quoted context omitted.

Weird... I always thought that if you are running a split test in parallel (all at the same time), then you can figure out the number of samples needed to compare the branches with statistical confidence. I mean it makes sense to me. As the number of samples increases for each split test the distribution shifts from a binomial distribution to a gaussian distribution by the central limit theorem, and that happens arou…

Here's a good explanation: http://www.evanmiller.org/how-not-to-run-an-ab-test.html

Thanks for the link to the blog post. It raised an important point worthy of inspection. I ran some numbers and "peeking" after the first 1000 trials does change the outcome. The chance that the outcome will reverse from declaring branch A the winner with 95% confidence to declaring branch B the winner with 95% confidence is rather small, less than 10%. However, if you lower your requirements to 80% confidence then the chance of the winner swapping increases to over 50%! For reference, I used the Wilson approximation for binomial distributions. I'm sure the Wald approximation fares worse.

Re: Ask Google to choose your next domain

#25

Post by joelrunyon »

KnowledgeSponge wrote: post #21
sillysaurus2 wrote: post #11

Earlier quoted context omitted.

Completely offtopic, but I can't resist: What's it like being in a position to choose how $500k per month is spent? Wow. That's equivalent to about forty engineers' salaries per year. EDIT: I visited your website and am not sure what to think. http://joelrunyon.com/ Is it true that those sorts of marketing tactics will convince people to entrust you with how to spend $500k/mo? (Personal branding is rather important.…

Not the parent, but I managed and eventually helped lead the paid search group at one of the top search agencies. We had clients with 7 figure budgets all the way up to 9 figure budgets (or essentially unlimited as long as we were within certain parameters). It's a bit scary and heart attack inducing if something goes majorly wrong (like going dark for a day because someone forgot to check the flight dates in the acc…

Exactly - the fact that Google actually pays attention to you is a nice plus. If I had an issue with a campaign, I had a direct phone number for someone who could fix things.

That said, at that point, support had already started to go downhill & we found the "reps" usually knew very little / less than us about campaigns. Interestingly enough, we found that BING reps were much more helpful as they were eagerly trying to make up market share.

[EDIT]: You also get access to betas & other google "tests" they're running with adwords that most normal advertisers don't. Most of them are designed to make Google more money, but every once in a while they'd come up with some cool stuff that'd be really helpful.

Re: Ask Google to choose your next domain

#26

Post by joelrunyon »

nostromo wrote: post #20
joelrunyon wrote: post #9

I spent a lot of money on Google in the RV industry (to the tune of $500,000/month for a certain fortune 500 company) & there are a lot of issues with this post: First up, his stats across the board aren't even close. His impressions vary by a factor of 4 for the different ads. From that alone, I bet he forgot to turn of the "optimize my ads" setting from Google and switch on "rotate evenly". The "optimize my ads" fe…

I can't imagine that Google doesn't weigh significance when picking a winning ad. Is that really the case?

Oftentimes they start "optimizing" whatever gets the most impressions out of the gate.

They'll do it on such a small sample size that statistical significance is impossible.

Google is more interested in you putting out ads that get more clicks (money for them) than they are about the ROI that you get on those ads.

That's why the default settings for all campaigns now are all settings that benefit Google rather than the advertiser.

Re: Ask Google to choose your next domain

#27

Post by joelrunyon »

caseyf7 wrote: post #24

Why wouldn't you get the domain with the most impressions and then work to improve the click-thru rate? (other than Joel's great point about Google controlling impressions)

Because you have no reason to assume that the one with the most impressions is the top performing ads.

Because of the way impression share is distributed (and because we don't know what ad rotation setting OP had on), the top impression ad might only have the most impressions because Google decided to show it more rather than it actually being superior on any actual performance metric.

Re: Ask Google to choose your next domain

#28

Post by michaelbuckbee »

joelrunyon wrote: post #9

I spent a lot of money on Google in the RV industry (to the tune of $500,000/month for a certain fortune 500 company) & there are a lot of issues with this post: First up, his stats across the board aren't even close. His impressions vary by a factor of 4 for the different ads. From that alone, I bet he forgot to turn of the "optimize my ads" setting from Google and switch on "rotate evenly". The "optimize my ads" fe…

I wonder if "optimize my ads" is performing a Multi Armed Bandit optimization approach [1] - which would actually be helpful.

https://support.google.com/analytics/answer/2844870?hl=en

Re: Ask Google to choose your next domain

#29

Post by pbhjpbhj »

Perhaps ... "rv menu" appears to perform well because those looking for food menus for when they are off in their RV are interested in just _looking_ at RVs? Did they mainly click the expensive ones? [That's what I'd do, like "ooh, fancy; I'd like one of those"]

Interest, indeed clicks, don't mean revenue will follow unless they were actual sales actions.

Now, it could also be that you can sell RV hire to people researching for an RV trip using "RV menu" ...

Re: Ask Google to choose your next domain

#30

Post by User8712 »

So, were these ads actually appearing evening across the same sites or search results, and in the same locations? If rvmenu.com appeared more frequently in one location than the others, that would skew the entire results.

I'm not convinced with the data you posted. The sample is small, and there's a lot of unknowns in the testing parameters. I wouldn't advise anyone to take this approach when selecting a domain.

Post reply on HN