Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

blog.cloudflare.com

Post reply on HN
Print view View original

291–300 of 304 posts

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#291

Post by hinkley »

nine_k wrote: post #139
lpapez wrote: post #68

This is the right way to deliver software. Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs. In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.

This assumes that you have plenty of cash to burn in the process, which is approximately correct for VC-backed ventures, and for offshoots of large corporations that play a lomg game.

I have a mountain of bad things to say about Microsoft, but they definitely had loss leader products that never made a dime but pulled you into their ecosystem. So they funneled money from a couple of cash cows back into the other projects to keep it enticing.

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#292

Post by hinkley »

chii wrote: post #260
lpapez wrote: post #68

This is the right way to deliver software. Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs. In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.

> start generating profit, and then you can start optimizing your costs. this assumes you can generate profit before you can get optimized - what if profit generation is only possible with optimized software? A lot of online MMO-style games tend to require such optimizations as they scale into the size required to generate profit. Or, in the current era of ai, the cost of the capital investment is far exceeding the a…

I've seen a lot of people relearn the same couple of lessons over an over again.

One, most of the things you believe scale linearly scale at nlogn because of physics. Storage is much worse, scaling at n^1.5. What fit in a single 4U case is orders of magnitude slower when it no longer fits in a single data center. Inserting records into a table with a billion rows is much slower than a table with ten thousand. Wipe the surprised Pikachu look off of your face.

Second, every architectural choice needs to be reviewed every time traffic increases by an order of magnitude, and preferably before so that you have time to stew on possible solutions and allow serendipity the time it needs to find you a better answer. If you have made too many architectural decisions, you will find there aren't enough hours in a month to keep up, even if you never sleep. That's the real reason we practice Principle of Least Power. Not security, but job security. Keep the system small enough that you have time to keep the plates spinning while also implementing new features to attract or retain customers.

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#293

Post by robotresearcher »

fc417fc802 wrote: post #287
robotresearcher wrote: post #276

Earlier quoted context omitted.

No, it's using a fridge in the understanding of the fundamental limitations of fridges. You asked ‘why would I want to use a service that serves stale data’. The answer is you don’t want to. But if you have a distributed system your only choices are about the nature of the possible staleness, since global instantaneous consistency is known to be impossible due to fundamental features of the universe.

In that case you're (intentionally?) misinterpreting what I originally said. The person I replied to was metaphorically suggesting turning the temperature dial on the fridge way up and I metaphorically asked why I would want to eat at his house if he's going to be serving me spoiled food. My original question could also be described as asking why I would choose to shop at a grocer where items on the shelf are routine…

Your additional explanation does help clarify your meaning.

Serving up records with TTLs in the past is so deeply weird that I didn't even consider that was what the grandparent had in mind.

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#294

Post by AlotOfReading »

esseph wrote: post #289
AlotOfReading wrote: post #279

Earlier quoted context omitted.

I picked the convention center as city center (union station would also be a reasonable choice) and I was only able to find 2 improved lots under $300k. Notably, neither is quite 3+ acres or within 20 minutes of the things you listed, and both are on septic. And this is in a metro where home prices are 36% below national median. [0] https://www.realtor.com/realestateandhomes-detail/3711-N-67t... [1] https://www.realt…

First if you us the convention center, you place the housing center off of where most of the housing actually is. Maybe 40% is in Kansas, with another 60% in the metro being on the Missouri side, roughly. The population density of the metro drags further south though if you look at it on a heatmap. That said, look around: https://www.zillow.com/kansas-city-mo/?category=SEMANTIC&sea... Between $175k and $350k center o…

    First if you us the convention center, you place the housing center off of where most of the housing actually is.
Convention centers are usually downtown. Mean population centers reflect the surrounding geography and can drift pretty far from an "idealized" circular urban area. My area's is out on an island for example, nowhere near downtown. Jackson county's is east of Raytown. I'm not sure why you'd specifically want to live within 20 minutes of the mean population center, so it wasn't the obvious interpretation of your original statement.

    Between $175k and $350k center on KCMO alone there are over 760 results.
No, there are 17 results once you filter for 2+ acres. Most of them are mislabeled or unimproved property. If you filter those out too (because this is a housing discussion), what remains within 20-30ish minutes are the two I linked.

I realize multiple acres of land is a bit of a silly requirement, but you're the one who suggested there are tons of houses with "several acres within 20 minute drive to a lot of US city downtown areas". That's why I was surprised.

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#295

Post by fc417fc802 »

inigyou wrote: post #204
otterley wrote: post #193

Earlier quoted context omitted.

It's impossible to know how often participants chose to stick to the standards anyway despite it not being in their best interest. Do you have any specific examples?

Linux routers responding ICMP from 192.0.0.8 when not having an IP address (currently being standardized). Clouds using class E address space (240.0.0.0-255.255.255.254). Routers that drop IP packets with options. The entire concept of NAT until the existing behavior was eventually standardized. Unnumbered interfaces.

I think all of those are examples of "[doing] things against agreed-upon standards [with] a compelling reason and [with] due consideration to the adverse consequences".

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#296

Post by esseph »

AlotOfReading wrote: post #294
esseph wrote: post #289

Earlier quoted context omitted.

First if you us the convention center, you place the housing center off of where most of the housing actually is. Maybe 40% is in Kansas, with another 60% in the metro being on the Missouri side, roughly. The population density of the metro drags further south though if you look at it on a heatmap. That said, look around: https://www.zillow.com/kansas-city-mo/?category=SEMANTIC&sea... Between $175k and $350k center o…

First if you us the convention center, you place the housing center off of where most of the housing actually is. Convention centers are usually downtown. Mean population centers reflect the surrounding geography and can drift pretty far from an "idealized" circular urban area. My area's is out on an island for example, nowhere near downtown. Jackson county's is east of Raytown. I'm not sure why you'd specifically wa…

> That would buy you thousands of square feet and often several acres within 20 minute drive to a lot of US city downtown areas.

This is my direct quote.

Let me break this down:

1. That would buy you thousands of square feet

True, clearly seen in the search results.

2. and often several acres

True, you just filtered down to several results currently on the market.

> I realize multiple acres of land is a bit of a silly requirement

Why? I live slightly further than 20min outside of town. There are massive, beautiful homes with tons of land out here, and tons have gigabit fiber from GFiber or several of the other providers in the area.

But that's not a requirement if you don't want it to be! There are clearly tons of homes available in the city with 3-4+ bedrooms and 2+ bathrooms in the $200 and $300ks is the point.

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#297

Post by Retric »

otterley wrote: post #282
Retric wrote: post #281

Earlier quoted context omitted.

You’re assuming performance has zero impact on customer retention, spending, etc which is demonstrably false. Further future costs aren’t bound by the current customer base or fiscal quarter. Insufficiently optimized code kills companies in highly competitive markets.

Nobody’s assuming that. A well-functioning business compares opportunity costs with all those considerations in mind.

“Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.”

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#298

Post by inigyou »

fc417fc802 wrote: post #295
inigyou wrote: post #204

Earlier quoted context omitted.

Linux routers responding ICMP from 192.0.0.8 when not having an IP address (currently being standardized). Clouds using class E address space (240.0.0.0-255.255.255.254). Routers that drop IP packets with options. The entire concept of NAT until the existing behavior was eventually standardized. Unnumbered interfaces.

I think all of those are examples of "[doing] things against agreed-upon standards [with] a compelling reason and [with] due consideration to the adverse consequences".

And the compelling reason is always profit.

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#299

Post by foltik »

kevincox wrote: post #285
rfgplk wrote: post #51

Frankly weird that they were resorting to high level containers for this in the first place. Also, this line struck me as odd > Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads. jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first…

It sounds like you are just writing your own allocator? That sounds great, but why is it going to be better than jemalloc? There are many reasons a specialized allocator can be better, but just saying "write your own" doesn't really add much value to the discussion.

If you squint, even their TLV encoding in a Box is kind of a specialized arena allocator.

Re: Saving 100 terabytes of memory by optimizing 1.1.1.1's DNS cache

#300

Post by otterley »

inigyou wrote: post #298
fc417fc802 wrote: post #295

Earlier quoted context omitted.

I think all of those are examples of "[doing] things against agreed-upon standards [with] a compelling reason and [with] due consideration to the adverse consequences".

And the compelling reason is always profit.

No it isn’t. Sometimes it’s security, or making the most out of limited resources.

Besides, not all profitable ventures are bad.

Post reply on HN