Reliability: It’s not great • Hacker News Board

Re: Reliability: It’s not great

#461

Post by pbronez » Wed, Mar 08, 2023, 12:46 PM UTC

snapetom wrote: post #345

eyelidlessness wrote: post #301

Earlier quoted context omitted.

When I was in an engineering IC version of this role, I longed for someone like you to have my back in management and have customers’ backs too. If I had a time machine and a magic lamp I’d team up with you. As a future representation of past me, I can tell you: 1. Everything it’s making you feel is valid. 1b. If you’re feeling burnt out, please listen to it. It gets worse if you let it. 2. While I can’t hire you now…

Man, you don't know how this impacted me. My boss is supportive, but he's also under heavy fire. Like I mentioned, my peers are rightfully skeptical. My team are a bunch of sharp, good guys, but haven't had any good guidance of mentorship in years, if not decades. They're all different, but what the have in common is that they've been screwed and judged unfairly thanks to past incompetence. That just pisses me off. T…

Agree with GP - your post makes me want to hire you.

One trick you might try: write future press releases. This helps you look beyond the immediate problems and focus on the destination. For example:

“Q3 2023: ACME CO released version Z today, which dramatically simplifies our engine to focus on core user needs. ‘It does thing I want an doesn’t crash anymore’ says Key Buyer #1”

By writing this down, you can put the vision in front of everyone. Then check it against actual progress to see how you’re doing.

Re: Reliability: It’s not great

#462

Post by cowl » Wed, Mar 08, 2023, 2:35 PM UTC

trilobyte wrote: post #438

cowl wrote: post #355

Earlier quoted context omitted.

Most of these platform have reached their critical mass to stay in business (and time to attract the customers like you) only because of the free tier. Sorry but even a low cost tier is too much for most wanting to give a try to a new infrastructure/stack. Most of these adoptions come from hobby projects trying it first and then recommending to use it in a professional setting. In a professional setting yes. you can…

Do you have examples of companies in this space that actually reached break-even? Heroku never hit profitability as far as I remember and with the Salesforce acquisition the question of profitability is moot. AWS is a counter-example to using a free tier as a GTM strategy. AWS did not start with a free-tier offering for S3 or EC2, that only came years later. By then they already had significant traction in the market…

I don't know if they have reached break-even. I know that they are in business for a long time and are clearly getting investments based on the usage and the potential. Usage that would not be there were it not for the Free time contributed to test-drive their offering and the free "evangelizing" done by those "free-loaders". AWS/Google/Microsoft are in a different league because they can afford the waiting game for actual Enterprise Evaluations or can afford massive sale forces (Azure case). Smaller players can not afford it and have to rely on "recommendations" based on past experiences.

Re: Reliability: It’s not great

#463

Post by Dangeranger » Wed, Mar 08, 2023, 2:37 PM UTC

snapetom wrote: post #321

zamnos wrote: post #160

Earlier quoted context omitted.

Can you help me in a detailed sense - what did you tell customers? did you literally say there's product is "completely over-engineered by a bunch of intellectual engineers with no focus, no discipline, and no oversight"? That seems a little over-honest to me but of course I wasn't there.

It depends on the audience. First off, it helps I've been 15+ years as an engineer, 5 as an engineering manager, and throughout have the community contributions in the field on my resume. I instantly spotted the problems when I was given an architecture diagram on day 1 and discussed what I would do differently. All that gives credibility. If it's internal audience, I am brutally honest. The organization needs to kno…

If you have any interest in writing a series of blog posts about your experiences in the “turnaround” and linking to them in your profile I believe you would have an audience. Your explanations are clear and your experience is worthwhile. Just food for thought.

Re: Reliability: It’s not great

#464

Post by LunaSea » Wed, Mar 08, 2023, 4:09 PM UTC

mike_d wrote: post #459

LunaSea wrote: post #458

Earlier quoted context omitted.

If you have customers in the EU than the court has jurisdiction. If the company doesn't comply, fines will be directly taken from customer payments for example.

Again - regardless of if a domestic court believes they have jurisdiction, any court case not brought in the venue of the defendant is effectively meaningless as you cannot be granted meaningful relief. If the destination bank account is outside the EU, they can't touch it without cooperation from the defendant countries courts - which requires you to file in the defendants venue. If an EU country unilaterally seized…

> Again - regardless of if a domestic court believes they have jurisdiction, any court case not brought in the venue of the defendant is effectively meaningless as you cannot be granted meaningful relief.

Of course you can, you simply reach for assets within the border of said member country or the EU. As I mentioned in my previous comment, you can for example get the funds from outgoing payments by customers of said company. You can also freeze accounts, prevent ownership or investments by any citizen of that country as well.

> If the destination bank account is outside the EU, they can't touch it without cooperation from the defendant countries courts - which requires you to file in the defendants venue. If an EU country unilaterally seized intra-bank remittance they would be cut off from the international banking system without hesitation.

There is nothing unilateral about a country seising money as payment of a fine from a company. This is a standard tool that every countries' IRS equivalent agency have in their tool belt.

> You seem to really be grasping at straws here, but the EU is not some all powerful entity that can enforce its laws outside its jurisdiction.

I never said that EU is all powerful, however, if business is done within the EU, EU countries have the power to access any and all funds going to the US for companies that do not comply.

They can also decide to block said service as a punitive measure.

Re: Reliability: It’s not great

#465

Post by LunaSea » Wed, Mar 08, 2023, 4:12 PM UTC

tpxl wrote: post #460

LunaSea wrote: post #414

Earlier quoted context omitted.

Google Chrome through telemetry and account history synchronisation which log PII in URLs and searched. Google Search will see PII go by if your marketing team is researching leads on LinkedIn for example. > And if you think there are no companies without AWS and Microsoft 360, you need to expand your horizon. I work for one such company, and so do many of my peers. And that's great. What is the services stack your c…

I honestly can't tell if you're trolling or you said 'AWS' and 'Microsoft 360' and meant cloud and managed email. > What kind of alternatives do you use for your email, browser, centralised data storage, etc. ? There are plenty of browser alternatives (firefox, safari, vivaldi, even chromium). There are dozens if not hundreds of email providers, and you can even provide your own. You can 'centralize data storage' on…

> I honestly can't tell if you're trolling or you said 'AWS' and 'Microsoft 360' and meant cloud and managed email.

I meant both clouds and managed email / storages services.

> safari

Don't both Firefox and Safari have telemetry and various ping back services?

> There are dozens if not hundreds of email providers, and you can even provide your own.

> You can 'centralize data storage' on disks on hardware you own, on premises or colocated. You could even use one of the dozens to hundreds of managed service and cloud providers.

Sure you can, I'm just saying that it is rarely if ever done in medium to large companies.

Re: Reliability: It’s not great

#466

Post by mixmastamyk » Wed, Mar 08, 2023, 7:38 PM UTC

otterley wrote: post #424

JeremyNT wrote: post #421

Earlier quoted context omitted.

This is especially true for scaling. A solution that works great for your current deployment may be completely unworkable for 2x your current deployment. You just won't know until you fall off the cliff. The armchair quarterback can opine that you should have just hired experts in XYZ domains from the start to design robust systems that can scale to arbitrary sizes, but most orgs don't need to scale to arbitrary size…

While I largely agree with you, this isn’t one of those cases. If Fly wasn’t supposed to scale in due course to this size, it probably wouldn’t have been funded. If your business model is predicated on you scaling, yes, you should hire appropriately in anticipation of that. Besides, I’m not even necessarily talking about hiring here - even consulting would have been sufficient to avoid this catastrophe.

Yes, although it's rarely possible to know which bottlenecks will hurt the most up front. Unless you've done the same thing before, which is not the case with anyone pushing boundaries.

Basically this is an argument around so-called premature optimization. Good to have issues now while it is mostly enthusiasts that are the customers. Guessing that this bump will be forgotten in five years? And not like AWS et al don't have outages occasionally that they learn from.

Re: Reliability: It’s not great

#467

Post by nazka » Wed, Mar 08, 2023, 8:30 PM UTC

gazby wrote: post #406

popcorncowboy wrote: post #380

Earlier quoted context omitted.

> I'm pretty sure I can negotiate salary, education, bonus, and what not if I can pull this off. Do this up front. Do it as soon as you possibly can. You will lose a huge amount of negotiating leverage if you "wait until you show them". I cannot stress this enough.

Just to reinforce this suggestion, this was my first thought also. Go get your comp mate.

Same! It’s the same principle as CEOs and stock options. For instance Musk that set several goals (x% market share, x unit sold…) and the hardest they get the more he will receive compensations.

I am not sure how much you could negotiate but you can have something like that and being metric based. X% customers happy, x% rating change, x% customers retained when they were close to leave. Then you make the math of the revenue and profit and it’s hard to say no.

Re: Reliability: It’s not great

#468

Post by philsnow » Wed, Mar 08, 2023, 9:09 PM UTC

ocimbote wrote: post #357

azurelake wrote: post #169

Earlier quoted context omitted.

> who doesn't want instantaneous global writes I want to gently note since I see a lot of misunderstanding around Spanner and global writes: Global writes need at least one round trip to each data center, and so they're still subject to the speed of light.

> so they're still subject to the speed of light. I giggled. Good witty comment, bravo.

I get the impression that you think "still subject to the speed of light" is some kind of hyperbole or something, like if you were on a freeway and saw a sign that said "end speed limit" and thought to yourself "welp, still can't go faster than c".

But when you're working on distributed systems that span the planet (say multi-master setups where ~every region can read and even write with low latency), you start thinking of the distance between your datacenters not in miles or kilometers but in milliseconds. The east coast and west coast of the US are at least 14 milliseconds apart:

  % units "2680 miles" "c ms"
  2680 miles = 14.386759 c ms

and that's not counting non-optimal routing, switching delays, or the speed of light in fiber (only 70% of c). Half of the circumference of the earth (~12500 miles) is likewise 67 milliseconds away absolute best case (unless you can somehow make fiber go through the earth).

Re: Reliability: It’s not great

#469

Post by mike_d » Thu, Mar 09, 2023, 8:44 AM UTC

LunaSea wrote: post #464

mike_d wrote: post #459

Earlier quoted context omitted.

Again - regardless of if a domestic court believes they have jurisdiction, any court case not brought in the venue of the defendant is effectively meaningless as you cannot be granted meaningful relief. If the destination bank account is outside the EU, they can't touch it without cooperation from the defendant countries courts - which requires you to file in the defendants venue. If an EU country unilaterally seized…

> Again - regardless of if a domestic court believes they have jurisdiction, any court case not brought in the venue of the defendant is effectively meaningless as you cannot be granted meaningful relief. Of course you can, you simply reach for assets within the border of said member country or the EU. As I mentioned in my previous comment, you can for example get the funds from outgoing payments by customers of said…

> Of course you can, you simply reach for assets within the border of said member country or the EU.

Which is exactly what I said. If the US company has an EU subsidiary you sue in that venue that can grant you relief. There are US tax implications of holding foreign assets, so the 1% of US companies with overseas interests create a foreign subsidiary, the other 99% have absolutely nothing within the reach of the EU.

> There is nothing unilateral about a country seising money as payment of a fine from a company.

Funds in transit belong to the sender until they arrive in the destination account. The EU would be seizing the funds of an innocent third party (the customer), and the target company would just shrug and say "your payment didn't arrive send it again." The EU cannot seize a transaction in flight and also compel the target company to honor it against their books.

> if business is done within the EU, EU countries have the power to access any and all funds going to the US for companies that do not comply.

See above. Taking money from random EU customers I guess is something they could do, but I imagine their citizenry would be none too pleased about it.

Let me try to simplify it for you: the EU cannot take what is not in EU jurisdiction without the cooperation of the foreign court. If a company says they were complying with their domestic law which violated EU law, they would likely not receive the cooperation of domestic courts to grant relief.

Re: Reliability: It’s not great

#470

Post by LunaSea » Thu, Mar 09, 2023, 7:35 PM UTC

mike_d wrote: post #469

LunaSea wrote: post #464

Earlier quoted context omitted.

> Again - regardless of if a domestic court believes they have jurisdiction, any court case not brought in the venue of the defendant is effectively meaningless as you cannot be granted meaningful relief. Of course you can, you simply reach for assets within the border of said member country or the EU. As I mentioned in my previous comment, you can for example get the funds from outgoing payments by customers of said…

> Of course you can, you simply reach for assets within the border of said member country or the EU. Which is exactly what I said. If the US company has an EU subsidiary you sue in that venue that can grant you relief. There are US tax implications of holding foreign assets, so the 1% of US companies with overseas interests create a foreign subsidiary, the other 99% have absolutely nothing within the reach of the EU.…

Let me make it simpler for you.

If say Google were to not follow the GDPR for example, even if they didn't have any European subsidiaries, the EU or a member country would simply make all Google customers pay their subscription fees to them instead of Google as fine payment for the fine. Customers would see no service disruption.