It Can Happen to You • Hacker News Board

Re: It Can Happen to You

#381

Post by kaba0 » Thu, Mar 04, 2021, 6:51 PM UTC

quelsolaar wrote: post #306

kaba0 wrote: post #209

Earlier quoted context omitted.

I think it goes both ways in that you either go full low level and write yourself everything (for questionable benefits), or you use a (possibly higher level) language with sane standard library, but the important thing is the quality of said library.

I find that writing everything yourself, especially simple things like a text2inteeger parser, is very valuable because it takes very little time and it levels up your understanding of the system. I'm starting to believe that you rarely understand something until you have implemented it. Therefor implementation is the best way to learn.

I’m totally with you here, it is a really invaluable learning tool. But similarly to science with “standing on the shoulders of giants”, we would not be anywhere if everyone started everything from scratch. Like, it’s okayish to reimplement a vector or something, but even a sort gets harder (especially if you want to make one that is performant both on a few element list and longer ones). And algorithms are just one thing, will you also learn into the totally foreign world of eg. audio processing?

Basically the only silver bullet for productivity increase is shared code. We just have to place higher emphasis on software correctness/quality instead of the functionality churn in certain parts of the software world.

Re: It Can Happen to You

#382

Post by froh » Thu, Mar 04, 2021, 6:53 PM UTC

mkeeter wrote: post #27

Blog author here! Thanks to HN for warning me about sscanf at exactly the right time – within a day of me trying to load some ASCII STLs and noticing it was slow... Linked deep in the Twitter replies [1], there's an open glibc issue about this, dating back to 2014: https://sourceware.org/bugzilla/show_bug.cgi?id=17577 C doesn't have any requirements on the complexity of sscanf, so it might not be a bug per se, but it…

printf and scanf match nicely with their format specifiers, so the serialization and deserialization can be maintained nicely in lockstep.

to avoid the quadratic overheating sttlen you can simply use fmemopen(3), which makes the temporary sscanf FILE object explicit and persistent for the whole parse, and needs just one strlen call.

https://news.ycombinator.com/item?id=26343149

Re: It Can Happen to You

#383

Post by arcturus17 » Thu, Mar 04, 2021, 6:55 PM UTC

didibus wrote: post #369

arcturus17 wrote: post #228

Earlier quoted context omitted.

In what sense is recursion a form of looping? I'm thinking of the memory model for each - one mutates some variables (at least some sort of index, unless you're doing an infinite loop) in a single frame, while the other accumulates function calls and stack frames until you bottom out, then return values are "folded" back up. Semantically, from a caller's perspective, they can achieve the exact same thing, but aside f…

> In computer science, a loop is a programming structure that repeats a sequence of instructions until a specific condition is met. That's the general definition at least I've always been most aware of. I don't want to claim it is the most common one, cause I don't really have numbers and who is the authority on comp-sci definitons? But I do feel it is at least a somewhat common academic definition for looping. That…

There's no denying that from that definition they are the same. It's just after you've debugged enough loops and recursions you can't help but think they are quite different!

Re: It Can Happen to You

#384

Post by akkartik » Thu, Mar 04, 2021, 7:01 PM UTC

kaba0 wrote: post #379

akkartik wrote: post #221

Earlier quoted context omitted.

I'm not sure I follow your question, but the idea is that the computer the end user receives has a single opinionated language for programming it. Kinda like Basic for microcomputers. The end user is of course welcome to build their own languages. That is encouraged! But multiple languages make the computer more difficult for others to comprehend. My goal is to convince you that, all things being equal, a computer wi…

My question is regarding why would a single language codebase be easier to comprehend/have fewer bugs, security holes? In terms of a single program it makes sense, but I seldom read the source code of a library for example I depend on - if it has a good public API, it could be written in anything for all I care. Not trying to dismiss the idea at all, just I don’t yet see “the light”, so to say.

Yeah, these are good questions and to be fair your questions are shared by many.

My basic worldview is that we need to have 100-1000x more people reading and auditing open source. The original value of open source was in the ability of people to read the source. If we don't use that ability then we don't really get the benefit.

The world today focuses on APIs and ignores implementation. I would argue that's the biggest source of problems today, with security holes, data breaches, user-hostile UX and so on.

If you accept that being able to read sources matters, hopefully it makes sense why reducing the number of languages matters. Every new language you add is more code to read, another language to learn and become proficient in, a new source of gotchas and subtleties to spend 10 years learning. Another set of tools, another set of moving parts that might not build on your computer because of some subtle version mismatch.

It's a hard problem. So let's make it easier for ourselves by relying on fewer languages, and being more thoughtful about the dependencies we introduce into our projects.

Re: It Can Happen to You

#385

Post by abetlen » Thu, Mar 04, 2021, 7:05 PM UTC

froh wrote: post #380

simias wrote: post #320

I think the really embarrassing part for Rockstar is that they didn't bother to investigate what took 5+ minutes to load in their star product, a simple profiling would've made the issue obvious. So either they knew and they didn't care, or they didn't know and they didn't care. That being said both for GTA and for TFA the issue is a very similar sscanf call: sscanf(data, "%f", &f); I already posted a similar comment…

scanf and printf have complementary format specifiers, which can make maintaining serialization and parsing of regular data a breeze... the proper remedy is to simply wrap the string to parse with fmemopen(3), which makes the temporary FILE object explicit and persistent for the whole parse, and needs just one strlen call. https://news.ycombinator.com/item?id=26343149

Cool trick, thanks for sharing. I don't get why there isn't a suitable snscanf function that takes the buffer length as an argument and returns the number of bytes parsed?

Re: It Can Happen to You

#386

Post by koalahedron » Thu, Mar 04, 2021, 7:23 PM UTC

rkagerer wrote: post #47

mkeeter wrote: post #27

Blog author here! Thanks to HN for warning me about sscanf at exactly the right time – within a day of me trying to load some ASCII STLs and noticing it was slow... Linked deep in the Twitter replies [1], there's an open glibc issue about this, dating back to 2014: https://sourceware.org/bugzilla/show_bug.cgi?id=17577 C doesn't have any requirements on the complexity of sscanf, so it might not be a bug per se, but it…

Hey, Matt, neat to see you here and congrats on making the front page! Recognize you from the Formlabs forums & conferences. Love that notion of professional empathy underscoring your message in the blog post.

I know Matt too, primarily from him rejecting my Libfive PRs for being "too 1337".

But seriously, Matt, I might have a project for an in-person event regarding printing robots. Stay tuned--to what channel I don't know.

Re: It Can Happen to You

#387

Post by tdeck » Thu, Mar 04, 2021, 7:23 PM UTC

kaba0 wrote: post #377

oblio wrote: post #294

Earlier quoted context omitted.

They were designed when an extra byte or so per string cost you a lot of money. Nowadays, when 99% of the systems anyone will program start at 1MB RAM and 90% probably start at 512MB, they're a liability for almost no benefit.

You’ve got an extra byte either way, the \0 at the end. Which in many cases will make you copy a string because you can’t just “point” into a string literal and say take n chars from there. Of course I am not that old so I don’t have enough expertise — but seeming that every other language even at the time decided against it is pretty telling.

I think your parent was referring to the cost of storing a 2-byte string length instead of a 1-byte terminator. In the 1970s and 1980s, 2 bytes would likely be the minimum storage needed for the length of a general purpose string implementation. Although there were some language environments (e.g. Pascal) that had counted strings with a max length of 255.

Re: It Can Happen to You

#388

Post by IanNorris » Thu, Mar 04, 2021, 7:38 PM UTC

simias wrote: post #320

I think the really embarrassing part for Rockstar is that they didn't bother to investigate what took 5+ minutes to load in their star product, a simple profiling would've made the issue obvious. So either they knew and they didn't care, or they didn't know and they didn't care. That being said both for GTA and for TFA the issue is a very similar sscanf call: sscanf(data, "%f", &f); I already posted a similar comment…

Not excusing this but there are likely a few mitigating factors here.

* Tight deadlines result in shipping code that's barely tested and may have resulted in minimal code reviews on it. * The original article mentioned how the ids were always unique. It may have been intended to load content from multiple sources or to allow patching of content on disk (or repurposed entirely from a different game). Or it could well be an oversight/over-engineering. * It may even be a general purpose json parser from another project that had never been tested with data of this size until after launch. * It probably wasn't always this bad. Likely when the game launched the loading times were much more reasonable as the amount of in-app-purchases was an order of magnitude smaller.

Typically most of the IAPs will be added much later, so much of the profiling work would have been done with this code having a much smaller json block.

When the game was shipped the dev team will likely have been shrunk significantly as the bulk of the team moves to a new project leaving a smaller team with a focus more on the content itself and the engine team that likely deal with and spot stuff like this will probably have their attention elsewhere.

Don't work for R*, have shipped many high budget titles though including live services.

Re: It Can Happen to You

#389

Post by kaba0 » Thu, Mar 04, 2021, 8:00 PM UTC

tdeck wrote: post #387

kaba0 wrote: post #377

Earlier quoted context omitted.

You’ve got an extra byte either way, the \0 at the end. Which in many cases will make you copy a string because you can’t just “point” into a string literal and say take n chars from there. Of course I am not that old so I don’t have enough expertise — but seeming that every other language even at the time decided against it is pretty telling.

I think your parent was referring to the cost of storing a 2-byte string length instead of a 1-byte terminator. In the 1970s and 1980s, 2 bytes would likely be the minimum storage needed for the length of a general purpose string implementation. Although there were some language environments (e.g. Pascal) that had counted strings with a max length of 255.

Fair enough; but actually it can be more memory efficient as well because of the better reusability of substrings (in case of null-terminated ones only the end can be reused)

Re: It Can Happen to You

#390

Post by mcguire » Thu, Mar 04, 2021, 8:49 PM UTC

nyanpasu64 wrote: post #137

lostcolony wrote: post #125

Earlier quoted context omitted.

...I fully plan to use "O(whatever)". Not sure for what. But, yes. (naive) Quicksort's amortized complexity being O(nlogn), but its O(n^2) on already sorted data, is all I ever needed to learn to take away that lesson. When sorting already sorted data is worse than sorting randomized data, it's a quick realization that "amortized cost" = "read the fine print".

Quicksort as O(n log n) is not amortized complexity, but average runtime for random data.

Something that is amortized complexity:

    vector.push(x)

Most of the time, it's O(1). Sometimes it's O(n). If you double the size of the backing array when it runs out of space, it's amortized O(1).