Your biggest next TODO is to rethink how you price the usage and make it 1000% simpler.
Removing stuff is never obvious yet often better
31–40 of 196 posts
Re: Removing stuff is never obvious yet often better
#32> Before long, a dedicated Slack channel was created, which accrued over 550+ messages representing opinions from every corner of the company. Another few thousand words and dozens of hours were spent in meetings discussing what we should add to the calculator to fix it. This is a symptom of over hiring. Too many people removes agency. When people lose sight of what's actually important and feel that they must reach…
> This is a symptom of over hiring. Perhaps. It's also a symptom of bike shedding which can happen with as few as two people.
As few as one - I'm pretty sure I manage that all on my own sometimes!
Re: Removing stuff is never obvious yet often better
#33> Before long, a dedicated Slack channel was created, which accrued over 550+ messages representing opinions from every corner of the company. Another few thousand words and dozens of hours were spent in meetings discussing what we should add to the calculator to fix it. This is a symptom of over hiring. Too many people removes agency. When people lose sight of what's actually important and feel that they must reach…
How did you reach the conclusion that the company has too many employees from that one sentence?
Re: Removing stuff is never obvious yet often better
#34I don't know if this calculator was good or bad, but the rationale sounds superficially ridiculous. > Visitors who didn't see the calculator were 16% more likely to sign up and 90% more likely to contact us than those who saw it. There was no increase in support tickets about pricing, which suggests users are overall less confused and happier. Of course if you hide the fact that your product might cost a lot of money…
> Whether they are better off depends on whether they end up getting a bill they are unhappy with later at some unspecified future date, or not. How is that a function of the overly-simplified and often-wrong calculator? If the user is never there to be happy/unhappy about it in the first place, then how would you test this anyway? By closing the loop and increasing engagement, you are increasing the chance that you…
> often-wrong
The author was very careful with their words: they didn't say the calculator was wrong. They said it was confusing and sensitive to small adjustments. It's likely that the same confusion and variable sensitivity exists during usage. IMHO they should have bit the bullet and revised the pricing model.
Re: Removing stuff is never obvious yet often better
#35The general message is interesting. This specific bit kinda gives pause though: > One slight misinterpretation and wrong input and you'd get an estimate that's overstated by as much as 1,000x. Does it also mean that in real world usage, one slight misinterpretation or misevaluation of your metrics and you're liable to 1000x more than you planned to ? I totally see this as a reality of online billing systems. I've mis…
(Author here) > Does it also mean that in real world usage, one slight misinterpretation or misevaluation of your metrics and you're liable to 1000x more than you planned to? Unlikely. You can see why in these two examples that really happened: One user I spoke with said they assumed "queries per second" is calculated by (number of searches) x (top-k for each search), where "top-k" is the number of results they want…
Thanks !
> where "top-k" is the number of results they want back.
Some systems will do that, so I get the confusion. I think the YouTube API for instance has a quota system that takes internal operations into account, so getting back 10 results in a query effectively weights 10+ credits.
I better understand the kind of issues you are facing, as these kind of subtilities are inherently hard to explain.
For better or worse, that's another advantage of starting with a small trial account and actually see how the operations are billed for typical operations.
Re: Removing stuff is never obvious yet often better
#36> Before long, a dedicated Slack channel was created, which accrued over 550+ messages representing opinions from every corner of the company. Another few thousand words and dozens of hours were spent in meetings discussing what we should add to the calculator to fix it. This is a symptom of over hiring. Too many people removes agency. When people lose sight of what's actually important and feel that they must reach…
How did you reach the conclusion that the company has too many employees from that one sentence?
Two lines of reasoning.
First: A lot of time was spent building consensus.
No individual felt they had unilateral power to remove the calculator. Instead the behavior was to seek approval. That's probably because of unclear ownership, which often happens because of too many people.
Second: Too many cooks in the kitchen.
At any stage of a company there's a limited amount of important work. Work that is mission critical and provides outsized value to the company.
When people feel their own work is not important they seek other work that appears to be important. So, you get the behavior of a lot of opinions on a pricing calculator.
Re: Removing stuff is never obvious yet often better
#37> In an internal poll, 7 of every 10 people in the company thought the version with the calculator would do better. An interesting and pretty classic dynamic - I liked the article overall but I think this point didn't get the highlighting it deserved. If 30% of the people involved think that the calculator is a bad idea that signals a potentially huge problem even if the majority think it is fine. Be alert to the pol…
I agree in principle, but I am struggling on how you could quantifiably evaluate the contentiousness of a change. No feature will ever get 100% consensus. 30% does not seem great, but is it meaningfully different from 20%? Even better if you have mixed incentives: sales wants any and all dark patterns enabled, customer support is sick of issuing refunds because the cart auto adds extended warranty to the purchase, et…
> No feature will ever get 100% consensus. 30% does not seem great, but is it meaningfully different from 20%?
Nobody is saying a feature should be automatically removed when it has 30 % detractors, just that it is a useful signal to investigate further.
The specific % threshold doesn't matter. Pick one that makes you chase false positives rarely enough. The exact number will vary from organisation to organisation.
Re: Removing stuff is never obvious yet often better
#38YAGNI: You aren't gonna need it. “Perfection is achieved not when there is nothing left to add, but when there is nothing left to take away” “Less is more” “The value of a creative product doesn’t lie in how much is there, but in how much has been discarded.” rttm: Reduce to the max "Kill your darlings" ...
It feels redundant to agree with this comment. I will anyway.
"Any fool can make something complicated. It takes a genius to make it simple."
"I apologize for such a long letter - I didn't have time to write a short one."
--
As for the calculator, I think it points to a bigger problem. Customers need to know what the platform will charge and a way to compare platforms in general. If the only way to truly know how much something will cost is to run their code on it, then maybe that's the thing that someone needs to implement.
There are big issues with this in the most-naive implementation in that people can easily abuse the ability to run code. That suggests that perhaps we need a benchmark-only environment where benchmarks themselves are the only thing allowed out of the environment. This may require a fair-amount of engineering/standards effort but could be a game-changer in the space.
A framework for being able to run this on many platforms to compare performance and pricing would lead to customers generating packages for vendors to compete. Though, I suppose it could also hide some devilish details like step-changes in rates.
This same framework would be useful for other things too, like testing how implementation changes affect future bills. Also, how pricing changes between vendors might become more-advantageous over time.
Of course, the sales folks might balk because they would rather have a conversation with everyone they can. Maybe I'm just advocating for a more advanced and complex calculator? ¯\_(ツ)_/¯
Re: Removing stuff is never obvious yet often better
#39May be the problem is just having so many people (the real cost).
Pinecone is just generally thought as something extremely overpriced:
https://www.timescale.com/blog/pgvector-vs-pinecone/
Until they can solve the feeling (or cut down on their own overhead), I don't see that great future for the company.
Re: Removing stuff is never obvious yet often better
#40i think of this lesson often (i don't remember who told me) elon building something at spacex and ruthlessly cutting out pieces they initially thought were needed. less complexity meant cheaper and faster construction so = more tests. i use this in day to day to life: making plans, buying things, managing others - reducing has led to more accomplished.
This is how China Lake made the Sidewinder missile also, and a large part of how Skunk Works operated under Kelly Johnson. Also to some extent Toyota.
I believe shrinking the feedback loops is the most effective form of envelope-pushing.