Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things we've learned about building products

newsletter.posthog.com

Post reply on HN
Print view View original

41–50 of 121 posts

Re: Things we've learned about building products

#41

Post by stetrain »

porridgeraisin wrote: post #38
simonw wrote: post #32

Earlier quoted context omitted.

I think A/B testing is one of the most expensive ways of getting feedback on a product feature. - You have to make good decisions about what you're going to test - You have to build the feature twice - You have to establish a statistically robust tracking mechanism. Using a vendor helps here, but you still need to correctly integrate with them. - You have to test both versions of the feature AND the tracking and test…

> You have to build the feature twice Why though? Can't you have it dynamically look up whether the experiment is active for the current request and if so behave a certain way? And the place it looks up from can be updated however?

But you have to implement and test both sides of that "if" statement, both behaviors. Thus "build the feature twice"

Re: Things we've learned about building products

#42

Post by simonw »

stetrain wrote: post #41
porridgeraisin wrote: post #38

Earlier quoted context omitted.

> You have to build the feature twice Why though? Can't you have it dynamically look up whether the experiment is active for the current request and if so behave a certain way? And the place it looks up from can be updated however?

But you have to implement and test both sides of that "if" statement, both behaviors. Thus "build the feature twice"

Right: you have to take responsibility for implementing (and testing and short-term maintaining) two branches.

Re: Things we've learned about building products

#43

Post by eddythompson80 »

light_triad wrote: post #36
simonw wrote: post #32

Earlier quoted context omitted.

I think A/B testing is one of the most expensive ways of getting feedback on a product feature. - You have to make good decisions about what you're going to test - You have to build the feature twice - You have to establish a statistically robust tracking mechanism. Using a vendor helps here, but you still need to correctly integrate with them. - You have to test both versions of the feature AND the tracking and test…

Some teams think they can A/B test their way to a great product. It can become a socially acceptable mechanism to avoid having opinions and reduce friction. Steve Blank's quote about validating assumptions: "Lean was designed to inform the founders’ vision while they operated frugally at speed. It was not built as a focus group for consensus for those without deep convictions" Is the Lean Startup Dead? (2018) https:/…

Any sort of political/PR fallout in any organization can be greatly limited or eliminated if you just explain a change as an "experiment" rather than something deliberate.

"We were just running an experiment; we do lots of those. We'll stop that particular experiment. No harm no foul" is much more palatable than "We thought we'd make that change. We will revert it. Sorry about that".

With the former people think: "Those guys are always experimenting with new stuff. With experimentations comes hiccups, but experimentation is generally good"

With the later; now people would wanna know more about your decision-making process. How and why that decision was made. What were the underlying reasons? What was your end goal with such a change? Do you actually have a plan or are you just stumbling in the dark?

Re: Things we've learned about building products

#44

Post by bluGill »

marcosdumay wrote: post #33
bluGill wrote: post #21

Earlier quoted context omitted.

Most A/B tests should not be done in a production like way. Grab some post-it notes and and sketch out both A and B: then watch users work with it. For a lot of what you want to know the above will give better information than a 100% polished A/B test in production. When people see a polished product they won't give you the same type of feedback as they will for an obvious quick sketch. The UI industry has gone wrong…

You are pushing for bit of too much of name-misuse. A/B tests are something what the entire point is that you run them on the real world, and gather how real users react. Design-time user testing has been a thing for much longer than A/B tests. They are a different thing. I mean, your point stands. But you can't do A/B tests on anything that is not production, those your are recommending are a different kind of tests…

I'll accept your definition correction. However I think my point still stands: there are better things than A/B testing to get the information you need.

Re: Things we've learned about building products

#45

Post by MrLeap »

the__alchemist wrote: post #26

Thought on this one: > Trust is also built with transparency. Work in public, have discussions in the open, and document what you’re working on. This gives everyone the context they need, and eliminates the political squabbles that plague many companies. This seems prone to feedback loops; it can go both directions. If there are political squabbles, discussion may be driven private, to avoid it getting shut down or d…

I've seen this. Takes management vigilantly guarding the commons from excessive drive bys and divebombs.

It takes a lot less energy to throw shit than it does to clean shit. There's infinite signals. Big egos take a lot of energy to repel and redirect to maintain it. I think it's absolutely worth it when it's possible, but yeah.

You wouldn't think so until you've done it, but it's really hard to get 6+ adults together where everyone's primary goal in that team is to make a thing. Seems like there's always one or more people who want to peck others, build fiefdoms, hold court.

Re: Things we've learned about building products

#46

Post by rapfaria »

adamgordonbell wrote: post #40
skyyler wrote: post #37

In the first image in the article, what is a "SuperDay"? Is this like a trial day where you're invited to do a day of work for free?

They pay you for it, but it is a trial work day. Story time. I interviewed for a job at posthog. I knew that I really loved their communication style. But I hadn't used their product and didn't know a ton about them except that their writing is fantastic. The 'product for engineers' focus that they is cool but when I had an interview, it was clear that I wasn't a 'product for engineering' person. When they proposed t…

> And I sort of said I had some more questions before we moved on to the Super Day.

> And they basically just said: we don't think it's going to work out.

Ouch, so their tight-knit, no-shortcuts hiring process is only thorough for them, not for the engineer applying.

Re: Things we've learned about building products

#47

Post by scottishbee »

Chyzwar wrote: post #5

18. Instead, forcing PRs into day of work unit, it is better to be minimum testable increment. Some features just need more work to be tested. Forcing everything into tiny tickets make both planning tedious and often introduce bugs in half finished features. 22. I saw design system fail in many companies. It is very hard to get right people and budget for this to succeed. For most startups are better to pick existing…

27. I don't think you do disagree. Read point 29: Hire and rely on product engineers. They have full-stack technical skills needed to build a product along with customer obsession. Yes, this means they need to talk to users, do user interviews, recruit tests for new features, collect feedback, do support, and respond to incidents.

Re: Things we've learned about building products

#48

Post by zabzonk »

dang wrote: post #30
zabzonk wrote: post #2

Sorry, what "successful products"? And I have to say that "Technical Content Marketer " is one of the most dubious job titles I have ever seen.

Posthog is pretty successful! But since it isn't strictly necessary, perhaps we can make this thread more meaningful by removing success from the title above.

Well, I'd never heard about it before today, but that may be just my bad, as I'm not really much into web stuff. But when I see something like this (from the top of their website):

> The single platform to analyze, test, observe, and deploy new features

my reaction is "Wha?"

But what the hell - though I do think the job title is bad, but then I've had a few of those myself.

Re: Things we've learned about building products

#49

Post by adamgordonbell »

rapfaria wrote: post #46
adamgordonbell wrote: post #40

Earlier quoted context omitted.

They pay you for it, but it is a trial work day. Story time. I interviewed for a job at posthog. I knew that I really loved their communication style. But I hadn't used their product and didn't know a ton about them except that their writing is fantastic. The 'product for engineers' focus that they is cool but when I had an interview, it was clear that I wasn't a 'product for engineering' person. When they proposed t…

> And I sort of said I had some more questions before we moved on to the Super Day. > And they basically just said: we don't think it's going to work out. Ouch, so their tight-knit, no-shortcuts hiring process is only thorough for them, not for the engineer applying.

Perhaps, but it wasn't a bad experience. I've come to value hiring processes and I guess employers where they known how to swiftly make decisions.

Re: Things we've learned about building products

#50

Post by dang »

zabzonk wrote: post #48
dang wrote: post #30

Earlier quoted context omitted.

Posthog is pretty successful! But since it isn't strictly necessary, perhaps we can make this thread more meaningful by removing success from the title above.

Well, I'd never heard about it before today, but that may be just my bad, as I'm not really much into web stuff. But when I see something like this (from the top of their website): > The single platform to analyze, test, observe, and deploy new features my reaction is "Wha?" But what the hell - though I do think the job title is bad, but then I've had a few of those myself.

I never let startups get away with "The" on Hacker News when I see it. I always replace it with "A" :)
Post reply on HN