Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Drop millions of allocations by using a linked list

github.com

Post reply on HN
Print view View original

111–120 of 158 posts

Re: Drop millions of allocations by using a linked list

#111

Post by steveklabnik »

moe wrote: post #110
steveklabnik wrote: post #97

Earlier quoted context omitted.

This may sound a bit glib, but the reason it's so slow is because basically every Rubyist goes "Does anyone know why installing gems is so slow? What is this thing doing?" And then _takes a coffee break instead of figuring it out_. There are very, very, very few people who actually do any work on core infrastructure projects. I don't blame them. The codebase of rubygems is... not exactly welcoming. I myself did some…

The codebase of rubygems is... not exactly welcoming. What a very polite way to put it. IMHO that whole mess (rubygems + bundler) would ideally be replaced from scratch, removing the need for bundler in the process. If any generous sponsor wants to improve Ruby as a whole, that's where their money should go. Imagine the productivity gains if everyones test-cycle was suddenly >10% faster, and nobody would have to wast…

I may or may not have threatened to do this after having had one too many drinks, but when I sobered up, I... sobered up. ;)

While I love a good 'burn the world down re-write,' it's a _lot_ of work and isn't guaranteed to succeed. It's been tried before, and in other languages too: check out wheels in Python.

That said, Rubygems did replace what came before it, and Gemcutter replaced what came before it... so it could be done. It's just non-trivial.

Re: Drop millions of allocations by using a linked list

#112

Post by icebraining »

pointernil wrote: post #87
icebraining wrote: post #58

Earlier quoted context omitted.

What social changes would you expect from the pricing policy changes of marginal EC2 instances?

"social changes from the pricing policy changes of marginal EC2 instances" sounds ridiculous when given this context, right? It changes when the context are not "marginal EC2 instances" but instead energy and resources burning machines, used (often) by ignorant software developers and their organizations allowed and actually encouraged, partly even actively driven into such purely self beneficial behavior models. For…

It sounds ridiculous because Amazon doesn't really control anything. The only result of their pricing change would be the switching to a competitor.

It changes when the context are not "marginal EC2 instances" but instead energy and resources burning machines, used (often) by ignorant software developers and their organizations allowed and actually encouraged, partly even actively driven into such purely self beneficial behavior models.

Fair enough, and I agree with you that more reflection is needed on the ethics of our industry. No argument there.

That said, I think you're miscalculating the result of such a switch. The fact is that servers are pretty efficient.

Say one of the developers commutes to work, doing ~12 miles each way on her/his Prius. If he works for two days optimizing the code, the energy cost of his commute will be ~130kWh.

With that same energy, you can run a PowerEdge R420 on full power (CPU benchmark) for almost 40 days! And remember that each of those would power a bunch of EC2 instances.

The reason EC2 instances are cheap is because they're actually cheap, both in terms of energy and resources.

Re: Drop millions of allocations by using a linked list

#113

Post by [deleted] »

lmm wrote: post #41
twsted wrote: post #23

Earlier quoted context omitted.

The statement was a little strong, but in my experience it contains some truth. People who has never used a lower-level programming language can't often recognize these performance issues.

I've often found the opposite; people who use C get excited about using >> rather than / because it saved a few cycles on very old compilers, but fail to notice where they could've made it a million times faster by using a hash table rather than a linked list.

[deleted]

Re: Drop millions of allocations by using a linked list

#114

Post by riffraff »

moe wrote: post #110
steveklabnik wrote: post #97

Earlier quoted context omitted.

This may sound a bit glib, but the reason it's so slow is because basically every Rubyist goes "Does anyone know why installing gems is so slow? What is this thing doing?" And then _takes a coffee break instead of figuring it out_. There are very, very, very few people who actually do any work on core infrastructure projects. I don't blame them. The codebase of rubygems is... not exactly welcoming. I myself did some…

The codebase of rubygems is... not exactly welcoming. What a very polite way to put it. IMHO that whole mess (rubygems + bundler) would ideally be replaced from scratch, removing the need for bundler in the process. If any generous sponsor wants to improve Ruby as a whole, that's where their money should go. Imagine the productivity gains if everyones test-cycle was suddenly >10% faster, and nobody would have to wast…

I'm reasonably sure some of the rubygems maintainers stated that in $FUTURE_VERSION bundler's functionality will be folded back in rubygems itself.

Rubygems is a very old project and it was basically stalled for a long time, it has already made quite big strides recently.

But I don't think it's rubygems' role to fix what rvm/rbenv fix (multiple rubies). Jars don't do that either :)

Re: Drop millions of allocations by using a linked list

#115

Post by duaneb »

agentultra wrote: post #89
rfrey wrote: post #51

I'm really surprised by the amount of smugness in the comments here. A bit of good-natured teasing, followed by a wheelbarrow full of "ruby-devs" this and "web-devs" that. Take off your Hats of Superior Coding. Any one of us, regardless of honorific titles, could have made this mistake, and you know it. Being steeped in CS Fundamentals does not immunize you against bugs. Congratulations to tenderlove for finding the…

Smugness: the dark-side of hubris. Hubris being one of the three virtues [0]. I like to remember the past of computer programming as though it was once friendly and receptive to people of all skill levels. I owe quite a lot to the geeks who came before me and answered my stupid questions, gave me powerful tools to learn with, and accepted my contributions; flawed as they were. Without making a few mistakes along the…

You can be proud of your achievements without being smug.

Also, hubris being a virtue is crap. It may be decent for your self esteem but it makes you miserable to be around. Case in point: Larry Wall.

Re: Drop millions of allocations by using a linked list

#116

Post by icebraining »

pointernil wrote: post #87
icebraining wrote: post #58

Earlier quoted context omitted.

What social changes would you expect from the pricing policy changes of marginal EC2 instances?

"social changes from the pricing policy changes of marginal EC2 instances" sounds ridiculous when given this context, right? It changes when the context are not "marginal EC2 instances" but instead energy and resources burning machines, used (often) by ignorant software developers and their organizations allowed and actually encouraged, partly even actively driven into such purely self beneficial behavior models. For…

Now what could the social change be, the more expensive computing devices could allow for in those regions?

The social change in those regions would be the factories closing down and moving to other countries, as the low prices would be no longer so relevant, and so the workers would return to the famine and poverty of the 60s and 70s that they are just beginning to escape.

Re: Drop millions of allocations by using a linked list

#117

Post by bcg1 »

tr352 wrote: post #82
jph wrote: post #26

Earlier quoted context omitted.

Java had the same kind of issue for years. In Java and Ruby, the standard library uses an object `+` operator to mean concatenation (not numeric addition), and the implementation did immediate data copies, rather than doing reference linking and copy-on-write (or immutability).

But in Java this happens only with strings doesn't it?

Sort of... technically only the '+' operator only applies to strings... however it is essentially the same issue as the pull request, from what I can gather (I assume the issue comes from allocating a new array with size length+1 and copying the original each time a record is added). Same thing happens with java.util.ArrayList.add() and its cousins though, and from my experience people rarely use the constructor specifying an initial capacity, so the default gets used even if it is would obviously be woefully small (default is 10 BTW, in case you're curious & lazy :).

Also one might argue that the problem is actually much worse with strings, because string concatenation is so common and the syntactic sugar of the '+' operator for strings encourages the "wrong" way.

Re: Drop millions of allocations by using a linked list

#118

Post by moe »

steveklabnik wrote: post #111
moe wrote: post #110

Earlier quoted context omitted.

The codebase of rubygems is... not exactly welcoming. What a very polite way to put it. IMHO that whole mess (rubygems + bundler) would ideally be replaced from scratch, removing the need for bundler in the process. If any generous sponsor wants to improve Ruby as a whole, that's where their money should go. Imagine the productivity gains if everyones test-cycle was suddenly >10% faster, and nobody would have to wast…

I may or may not have threatened to do this after having had one too many drinks, but when I sobered up, I... sobered up. ;) While I love a good 'burn the world down re-write,' it's a _lot_ of work and isn't guaranteed to succeed. It's been tried before, and in other languages too: check out wheels in Python. That said, Rubygems did replace what came before it, and Gemcutter replaced what came before it... so it coul…

I... sobered up.

Same here, still have my napkin notes. It's actually one of my bucket list projects, if there wasn't the dreadful food-on-table constraint...

check out wheels in Python.

Valid point. Python is indeed a good example for an even worse situation. In fairness, most languages are still worse off than Ruby even now. I wouldn't trade maven, CPAN, etc. for bundler with all its warts.

However, there's also languages pulling ahead. npm seems to be slowly getting there (after a rough start) and the Go experience (godep), while still in flux and not directly comparable, is also something to draw lessons from.

Re: Drop millions of allocations by using a linked list

#119

Post by bcg1 »

Ao7bei3s wrote: post #71
DiabloD3 wrote: post #62

Earlier quoted context omitted.

Not sure why parent is getting downvoted, there is a serious problem in the Ruby community that very few of them have read GoF, TAoCP, and/or K&R.

Stop jumping to conclusions, and stop throwing around buzzwords. GoF is a (somewhat C++-centric) book about design patterns that's completely unrelated to the discussion at hand, may not be the best resource to learn about design patterns and has nothing to do with CS. TAoCP is more like an encyclopedia; actually reading through even one chapter takes a significant amount of effort (if you want to get anything out of…

+10 if I could, re "Expert C Programming"; excellent read, even if you aren't a C programmer

Re: Drop millions of allocations by using a linked list

#120

Post by aidos »

Sir_Cmpwn wrote: post #105
aidos wrote: post #17

Earlier quoted context omitted.

Annnndddd.... what reaction do you expect? "Oh, please come and do web development so we can bask in the glow of your self-righteousness and infinite knowledge of computers." /snark (apologies for being offensive, but good lord, what a silly statement - unless I missed the joke) I stand before you as a counterpoint to your foolish generalisation, and guess what, I know plenty of other people that don't fit your stere…

I'm a senior dev in both domains and I can safely say that your comment is bullshit.

What does any of that even mean?

Edit Could you clarify? Which part of what I said is "bullshit"?

Post reply on HN