201–210 of 305 posts
I wonder at their scale, why wouldn’t it make sense to store the entries lightly compressed in memory?
Alternatively a btree somehow they can take advantage of prefix compression
One of my proudest professional moments was when me and three others managed to reduce memory load of the game Wavetale from 20+GiB to under 3GiB so we could port it to Nintendo Switch. The 100 TiB number almost gives me vertigo. Though in this context it was "just" 50%
Earlier quoted context omitted.
Why would anyone want to use a DNS resolver that tampered with records on a large scale? The TTL is intentionally set by the originator of the record. Or alternatively, if you don't tamper why would I want to use a service that serves stale data?
Every distributed system serves stale data. You can define away ‘stale’ by picking a consistency model, but look inside the consistency machinery and you will see fresher data you aren’t allowed to have yet.
Earlier quoted context omitted.
It was the fact that adhering to the standard was in the best interest of each participant. When it isn't, they don't.
It's impossible to know how often participants chose to stick to the standards anyway despite it not being in their best interest. Do you have any specific examples?
Earlier quoted context omitted.
The way I usually prevent having to scale out to a bazillion systems is never getting more than 10 users.
The art of premature optimizations
This is the right way to deliver software. Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs. In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
Its a yes if you do not know the domain space, query patterns well enough and also if the cost of optimization or time for optimization may have detrimental impact to business. In this case it most likely means that the crowd in the room did not anticipate much on this in early phases and no one in the room pointed these things out. The irony is that these performance and disk numbers are heavily discussed as a part…
Pretty sure they were joking.
leaves
Earlier quoted context omitted.
Or optimize a bit earlier and prevent having to scale out to a bazillion systems.
Remember that everything has an opportunity cost. Running a lot of servers might cost $10 million annually, but if the product team had to choose between a project that would recoup $5 million of that vs. an opportunity to earn $50 million ARR for the same amount of work, the logical answer would be obvious.
One of my proudest professional moments was when me and three others managed to reduce memory load of the game Wavetale from 20+GiB to under 3GiB so we could port it to Nintendo Switch. The 100 TiB number almost gives me vertigo. Though in this context it was "just" 50%
I'd love to hear what was taking up that 17GiB if you can share, even if it is commonly optimized things like packing, still fun to hear about
We had to implement a streaming system and figure out a good way of chunking the world. This was a challenge because everything was on water and you could see nearby islands quite far away. An intern called Tommi did a great job identifying a good strategy and writing the system.
We also had to reduce the density and model complexity of a lot of environmental details such as rocks, vegetation, and stuff like pots and clotheslines. This was done largely by sorting things by memory size and frequency of use and identifying outliers.
One of the biggest issues was actually really silly: the journal fetched Portrait images and names of characters by referencing the actual NPC and having them embedded there. Meaning the journal, which was always loaded, would pull every NPC involved in a quest into memory including their behaviors, textures, and models.
On my end I also found a lot of silly details wasting hundreds of megabytes. Special render passes using huge textures and render targets, poor structuring of the render pipeline caused memory increases, several key shaders referenced huge textures which weren't necessary, ...
My blog posts about the project[0] mainly focus on rendering performance because that's where I spent more time and it contains more interesting content for discussion. But reducing memory was an ongoing concern with countless little improvements over the 1.5 year porting process