31–40 of 80 posts
Rather we should optimize for user understanding or satisfaction - whichever fits. These are harder to capture but FAR more beneficial to the consumers of content.
Earlier quoted context omitted.
The danger is it's all local optima. A/b test shows lift on "i created a rust copilot" vs "rust debugging toolkit". Ultimately.. it's one big "so what" outcome. These are distractions. A product either has market fit or it doesn't. The big companies are especially susceptible to these distractions because they have the budget to blow chasing micro funnel optimizations. It sounds reasonable, but in my experience i agr…
Yes but just because you have product market fit doesn’t mean you shouldn’t try to optimize copy so that it reaches as big an audience as possible
1) Is copy even important? I think it is. If this post was titled, "Auto-tune experimentation for short-form content optimization", it might make half the audience confused about the product. In fact, the 1-liner we use for HN is very different from when we talk to VCs, because the audience is different with different goals & backgrounds. I guess the point I am making is that messaging has to be contextualized, depending on users, platform, and goals.
2) PMF vs copy - I agree that the two are orthogonal. Copy is not going to solve for the lack of a PMF (and it shouldn't). Exactly the point above - the goal is to help more and more users comprehend what you do, hopefully in a way that's more personalized to them.
Why is all text lowercase on your website? I wouldn't want this in my product :)
Have you thought about a specialized product to automate A/B tests for form submission? Not just copy, but also layout and which fields you collect. Direct response marketers and leadgen services could really use this.
Earlier quoted context omitted.
Yes but just because you have product market fit doesn’t mean you shouldn’t try to optimize copy so that it reaches as big an audience as possible
Yeah - two themes here - 1) Is copy even important? I think it is. If this post was titled, "Auto-tune experimentation for short-form content optimization", it might make half the audience confused about the product. In fact, the 1-liner we use for HN is very different from when we talk to VCs, because the audience is different with different goals & backgrounds. I guess the point I am making is that messaging has to…
That's the challenge: conversion funnel is complex with many factors. and largest one of them, in simple terms, is PMF.
if we measure downstream like clicks or inbound leads etc, that's more aligned with "discovery of PMF" and that's good. But it should be stacked ranked as so, it's not driving the needle. it's exploratory.
Meta, TikTok and Google have all the data. Not even a sampling. They can exactly answer what is the most effective copy for possibly anything that anyone wants to do. No A/B test required. So why don't they offer that? I know it's not because the data is proprietary or private, because basically all the information you need is visible on Facebook Ad Library, more than enough to answer most questions about authoring c…
Appreciate the candid comments and opinions here. I'll break it down and go over them - 1. Having access to data is not the problem we're after (most companies have the first-party data in-house). The key challenges are around having a platform that fundamentally separates strings (copies) from code, and lets you update them effortlessly, based on inferences from that data. So, I am not sure I understand why this is…
Which audiences am I optimizing copy for? Where do they come from? Some Google, Meta TikTok or Apple owned channel right?
Google has indexed every website. Meta has every ad. Can't they just tell me what copy to use? Why don't they? I mean, they know! They know what copy works best, for pretty much everything. They can sort by clickthrough rate, revenue due to the purchase data they have, they have everything! You talk about SMB - they know every SMB! They know your margin and your COGS and whatever because they in aggregate they observe rational spending where all the cost is eaten by marketing; they know your potential market, etc. They know all this. They don't need to run tests. They can look at very recent, weeks old, historic data, and they have way more than enough samples to answer these questions to more or less the same degree of certainty and scientific rigor that any SMB doing it themselves as a first party can do.
I mean if they wanted to, they could run the A/B tests for you! Google could "just" serve a different web page with slightly altered copy. And see if more people "click" or "convert" or whatever. They have better technology, 1,000,000x more data... Why don't they do this? You wouldn't even need UX. It could just happen, you would check a box, and they would do this.
> fundamentally separates strings (copies) from code... and lets you update them effortlessly... The ability to make scientific edits without re-deployments and accelerating continuous iterations based on user feedback...
You keep talking about UX for developers and product managers. These are UX things. It doesn't actually matter. The existence or non-existence of what you're talking about doesn't correlate to higher or lower conversions, it isn't a scientific opinion on the practice of optimization, it is just a bunch of UX patterns to achieve it, but it could be achieved in many ways, perhaps with even better UX. Like in the example I gave, where Google "just" does this for you, which is the best UX because there is no UX, you don't need to separate strings from code, and you don't need to update them, because you don't need to do anything. Google could just do this. They own the channel, they see everything, they have the technology.
So why don't Google and Meta and Apple offer an automatic optimization product? You ought to have an opinion, it can't just be, "I don't know." I mean the sort of obvious answer is that "optimization doesn't really work" instead of "three paragraphs of bullshit."
> Curious why you think A/B test results are fictional? Getting stat sig results is probably the surest way to conclude results. Perhaps there's a different angle you are talking about here.
Well one reason I am very confident they are fictional is because the people who own the channels for a decade haven't offered a tool to do this.
I mean maybe they will. Maybe it was a technology problem, but I don't believe so. You could have Markov Chained your way through 5 word long taglines and whatever. They didn't need to way for generative AI to create valid test strings for people's websites. Indeed they could just let you copy the best performing taglines they see in their systems. Why. Don't. They?!
> Given the number of users that get exposed to every change a consumer company as large as Twitter makes
Another POV is that every change they made was bad. They thought they were a product organization, and they are really a backend engineering organization, where the best decisions are all based on first principles or executives' opinions, not on some unknowable measurement about audiences.
I'm no lawyer, but the statement "to productize the tooling we started building at Twitter" is troublesome to me. Do you have some protection from Twitter/X making a IP claim against this? I.e., can you prove you scrapped "the tooling we started building at Twitter" and began fresh?
The idea transpired from some of the early tools at Twitter. The actual product has nothing in it that would be considered Twitter's IP afaik. A couple of other companies we spoke to use similar techniques to optimize copy testing. Good call out on the language though - we could have worded it better!
An opportunity to dogfood your product!