Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Drop millions of allocations by using a linked list

github.com

Post reply on HN
Print view View original

101–110 of 158 posts

Re: Drop millions of allocations by using a linked list

#101

Post by mbrock »

steveklabnik wrote: post #97
mbrock wrote: post #44

Semi-related: does anyone know why installing gems is so ridiculously slow? What is the thing doing? Downloading tarballs, yes, but then? It's a dynamic language; there is no compilation or verification! Why can I install Ruby packages using apt almost immediately, when gem/bundle install takes half a coffee break? I'm growing more impatient with the years. I have measured out my life with slow software. We talk abou…

This may sound a bit glib, but the reason it's so slow is because basically every Rubyist goes "Does anyone know why installing gems is so slow? What is this thing doing?" And then _takes a coffee break instead of figuring it out_. There are very, very, very few people who actually do any work on core infrastructure projects. I don't blame them. The codebase of rubygems is... not exactly welcoming. I myself did some…

Yeah, good point. Still, it seems likely that a number of people have done at least some work on profiling and optimizing without success. Hasn't Gem slowness been an issue since, like, 2006? The reason I haven't even tried is mostly that it seems likely to be difficult and treacherous. But I might be wrong.

Note that I wasn't only complaining. I just figured that there is some real reason for the slowness and that this reason might be known by the community (and I didn't find much googling for "why is ruby gems slow"). Call it preliminary research...

Re: Drop millions of allocations by using a linked list

#102

Post by jdmichal »

jokoon wrote: post #76

I still can't see the real usefulness of linked lists, the idea of having a data container that doesn't have a transparent indexing algorithm sounds ill-advised. Linked lists should be named "linked graphs" instead. There is so much relevant science to learn about CPU caches, than there is about using a container which is based on nested pointer indirections.

If you're going to change the name, "unary trees" makes a lot more sense. "Linked graph" does not imply the 1-child-per-node linearity requirement of a linked list.

Re: Drop millions of allocations by using a linked list

#103

Post by nirvdrum »

engendered wrote: post #84
rfrey wrote: post #51

I'm really surprised by the amount of smugness in the comments here. A bit of good-natured teasing, followed by a wheelbarrow full of "ruby-devs" this and "web-devs" that. Take off your Hats of Superior Coding. Any one of us, regardless of honorific titles, could have made this mistake, and you know it. Being steeped in CS Fundamentals does not immunize you against bugs. Congratulations to tenderlove for finding the…

Is this legitimately a bug, though? It could be tagged as a performance defect, but there is code like this through projects across the land. I think the reason this rubs some people the wrong way is that implementations like this can be the result of the "no premature optimizations!" philosophy and its advocacy. I've encountered this firsthand at a number of organizations, and it apparently was somewhat endemic in t…

Not to get all philosophical, but "legitimately a bug" is a hard question to answer. Some people strictly consider correctness to be the definition of a bug. Others extend the definition to mean when things don't work the way they probably should (e.g., performance or usability). Performance is an especially interesting one to me because if it's not really a bug, then does a performance regression constitute a bug even if it's producing the same results? If someone is relying on that timing or it otherwise affects interaction with the code, its execution speed is a functional component.

In my experience, projects that adopt a slightly broader definition of "bug" have a better track record of improving on those fronts. Nobody like to have bugs around, after all.

Re: Drop millions of allocations by using a linked list

#104

Post by steveklabnik »

mbrock wrote: post #101
steveklabnik wrote: post #97

Earlier quoted context omitted.

This may sound a bit glib, but the reason it's so slow is because basically every Rubyist goes "Does anyone know why installing gems is so slow? What is this thing doing?" And then _takes a coffee break instead of figuring it out_. There are very, very, very few people who actually do any work on core infrastructure projects. I don't blame them. The codebase of rubygems is... not exactly welcoming. I myself did some…

Yeah, good point. Still, it seems likely that a number of people have done at least some work on profiling and optimizing without success. Hasn't Gem slowness been an issue since, like, 2006? The reason I haven't even tried is mostly that it seems likely to be difficult and treacherous. But I might be wrong. Note that I wasn't only complaining. I just figured that there is some real reason for the slowness and that t…

> Still, it seems likely that a number of people have done at least some work on profiling and optimizing without success.

You'd think that, wouldn't you? The reality is... not that. :/

And yeah, this isn't really about you specifically, sorry if it came off that way. This is just a general problem in the Ruby OSS world. I can't speak to too many other OSS worlds, as the Ruby one is the one I've been most involved in. If you're looking for more research, Andre Arko is one of the other people who's actually doing work in this area, and he's given a number of talks about why Bundler is slow: https://vimeo.com/67807956 and related. He's been making a lot of strides, but it's not easy.

Re: Drop millions of allocations by using a linked list

#105

Post by Sir_Cmpwn »

aidos wrote: post #17
perdunov wrote: post #6

Earlier quoted context omitted.

Really. Every time I whine publicly how web programming people aren't familiar with even basic CS, I get a slap. But really, I should move to web programming. I'll be an expert computer scientist there, probably.

Annnndddd.... what reaction do you expect? "Oh, please come and do web development so we can bask in the glow of your self-righteousness and infinite knowledge of computers." /snark (apologies for being offensive, but good lord, what a silly statement - unless I missed the joke) I stand before you as a counterpoint to your foolish generalisation, and guess what, I know plenty of other people that don't fit your stere…

I'm a senior dev in both domains and I can safely say that your comment is bullshit.

Re: Drop millions of allocations by using a linked list

#106

Post by evincarofautumn »

kybernetyk wrote: post #73
mkopinsky wrote: post #70

Earlier quoted context omitted.

How do you keep track of the technical debt? Ticket system?

//TODO: //FIXME: ;)

Wink all you like, but I get a lot of value from greppable, well written fixmes directly in the source they pertain to. If I’m working on a feature and I discover some odd misbehaviour, there is often a comment right there in the source, explaining precisely what I need to do next.

Re: Drop millions of allocations by using a linked list

#107

Post by mbrock »

steveklabnik wrote: post #104
mbrock wrote: post #101

Earlier quoted context omitted.

Yeah, good point. Still, it seems likely that a number of people have done at least some work on profiling and optimizing without success. Hasn't Gem slowness been an issue since, like, 2006? The reason I haven't even tried is mostly that it seems likely to be difficult and treacherous. But I might be wrong. Note that I wasn't only complaining. I just figured that there is some real reason for the slowness and that t…

> Still, it seems likely that a number of people have done at least some work on profiling and optimizing without success. You'd think that, wouldn't you? The reality is... not that. :/ And yeah, this isn't really about you specifically, sorry if it came off that way. This is just a general problem in the Ruby OSS world. I can't speak to too many other OSS worlds, as the Ruby one is the one I've been most involved in…

Cool, thanks, that's good info! I'll have to investigate these talks.

Re: Drop millions of allocations by using a linked list

#108

Post by nirvdrum »

Dewie wrote: post #75
robmccoll wrote: post #35

Earlier quoted context omitted.

You should be careful with the term engineer. By definition, engineering is the application of scientific and mathematical knowledge to solving practical problems. Without knowing and understanding the science and math behind computing and software, one can hardly claim to be a software engineer.

So many programmers have this weird inferiority complex when it comes to the term "engineer". Not you, but those who think that most programming can never be called "engineering" because people don't die if you introduce a software bug[1] (as if the only kinds of modern "engineers" have to do with immediately safety-critical things). I prefer the plain "programmer" myself, but I don't see the big deal unless "enginee…

For me it's not a matter of severity of consequences, but rather in all other professional fields to be called an engineer you need to pass a PE exam. Having not taken that exam, it just feels like taking a cheap shortcut because it's not widely regulated.

Re: Drop millions of allocations by using a linked list

#109

Post by nirvdrum »

lmm wrote: post #40
rosspanda wrote: post #33

Earlier quoted context omitted.

A agree somewhat, but I've seen 10 box systems that could run on raspberry pie with good code

I've seen them. I've worked on them. But again, where's the cost/benefit?

I'm hardly what most environmentalists would call an "environmentalist", but one cost here is the increase in carbon footprint. Of course, to the company the cost/benefit analysis errs on the side of just spinning up more boxes. But from a larger perspective, taking some extra time to make more efficient use of machines could have a drastic impact. Many optimizations don't require months to implement. Many of those are even avoidable with a bit of foresight.

Re: Drop millions of allocations by using a linked list

#110

Post by moe »

steveklabnik wrote: post #97
mbrock wrote: post #44

Semi-related: does anyone know why installing gems is so ridiculously slow? What is the thing doing? Downloading tarballs, yes, but then? It's a dynamic language; there is no compilation or verification! Why can I install Ruby packages using apt almost immediately, when gem/bundle install takes half a coffee break? I'm growing more impatient with the years. I have measured out my life with slow software. We talk abou…

This may sound a bit glib, but the reason it's so slow is because basically every Rubyist goes "Does anyone know why installing gems is so slow? What is this thing doing?" And then _takes a coffee break instead of figuring it out_. There are very, very, very few people who actually do any work on core infrastructure projects. I don't blame them. The codebase of rubygems is... not exactly welcoming. I myself did some…

The codebase of rubygems is... not exactly welcoming.

What a very polite way to put it.

IMHO that whole mess (rubygems + bundler) would ideally be replaced from scratch, removing the need for bundler in the process.

If any generous sponsor wants to improve Ruby as a whole, that's where their money should go. Imagine the productivity gains if everyones test-cycle was suddenly >10% faster, and nobody would have to waste energy on bundler/rbenv/rvm issues anymore.

Perhaps we could even fix the deployment nightmare in the process, with e.g. jar-style packaging, but now I'm really dreaming...

Post reply on HN