Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's wrong with 2006 programming?

antirez.com

Post reply on HN
Print view View original

41–50 of 51 posts

Re: What's wrong with 2006 programming?

#41

Post by ntoshev »

wmf wrote: post #35
ntoshev wrote: post #27

Earlier quoted context omitted.

Be sure to check PHK's comment (#5) madvise + mincore syscalls could be used to get the kernel to preload the pages asynchronously.

Not really; because mincore() does not generate events you'd have to poll to find out when the page-in has finished, which seems pretty wasteful. If I was doing disk I/O from an event-driven program I would rather use the Flash approach of using a small number of worker threads to perform blocking I/O. PHK says "I consider this a deficiency in POSIX standards, not in the concept of VM.", but you have to go to product…

I agree you have to make it work with the real and not ideal kernel :) Another thing PHK misses is that single-threaded approach solves the database concurrency problems for free, and people start to use it more - e.g. Stonebraker in his VoltDB.

Polling mincore wouldn't be that bad as it would only happen between the commands and at least the core of the algorithm would be simple: before executing a command, check if it's data are in memory with mincore, if not ask to load them with madvise, and go to the next command.

What does Flash use worker threads for?

Re: What's wrong with 2006 programming?

#42

Post by shin_lao »

wmf wrote: post #2

Just to amplify his point, if you want your program to take page faults as PHK suggests, it has to be multithreaded. If you choose event-driven concurrency you can't afford to take page faults in mmap() or read(). When you make the threads vs. events decision you're implicitly making a bunch of related decisions about I/O and scheduling as well; a hybrid approach (like using events and mmap) won't work well.

Actually the hybrid approach works extremely well, it's just more delicate to architecture.

The idea is to process in parallel everything you can and have the services ("processors") communicate asynchronously by events.

Re: What's wrong with 2006 programming?

#45

Post by jacquesm »

fleitz wrote: post #22

Stop thinking of RAM/disk/etc as storage systems and start thinking of them as retrieval systems. Then stop thinking of your data costs as $/GB (storage systems) and start thinking of your data costs as $/(IO/sec/GB) (retrieval systems). I know everyone these days seems to think that removing a structured query language parser from a database makes every other problem go away, but realistically RDBMS vendors spend mi…

> SSDs for the stuff you can't afford to keep on an SSD

I think you meant 'in RAM' there.

Re: What's wrong with 2006 programming?

#47

Post by wmf »

ntoshev wrote: post #41
wmf wrote: post #35

Earlier quoted context omitted.

Not really; because mincore() does not generate events you'd have to poll to find out when the page-in has finished, which seems pretty wasteful. If I was doing disk I/O from an event-driven program I would rather use the Flash approach of using a small number of worker threads to perform blocking I/O. PHK says "I consider this a deficiency in POSIX standards, not in the concept of VM.", but you have to go to product…

I agree you have to make it work with the real and not ideal kernel :) Another thing PHK misses is that single-threaded approach solves the database concurrency problems for free, and people start to use it more - e.g. Stonebraker in his VoltDB. Polling mincore wouldn't be that bad as it would only happen between the commands and at least the core of the algorithm would be simple: before executing a command, check if…

Flash used worker threads for any operation that could potentially block, like disk reads.

http://www.usenix.org/event/usenix99/full_papers/pai/pai_htm...

Re: What's wrong with 2006 programming?

#48

Post by antirez »

jacquesm wrote: post #46
antirez wrote: post #44

Earlier quoted context omitted.

Fixed in the article, sorry I was not aware that the "Varnish guy" was a well known programmer.

It's like referring to Donald Knuth as 'the TeX guy'.

With all the respect, I wish Poul-Henning Kamp the biggest of the fame, but it is hard to compare him with a world wide myth as Donald Knuth.

Re: What's wrong with 2006 programming?

#49

Post by mkramlich »

jasonwatkinspdx wrote: post #36
fleitz wrote: post #22

Stop thinking of RAM/disk/etc as storage systems and start thinking of them as retrieval systems. Then stop thinking of your data costs as $/GB (storage systems) and start thinking of your data costs as $/(IO/sec/GB) (retrieval systems). I know everyone these days seems to think that removing a structured query language parser from a database makes every other problem go away, but realistically RDBMS vendors spend mi…

This has been well understood since antiquity (in the CS world at least). Read Jim Gray's "The 5 minute rule" and the more recent papers that cite it. Most likely the access frequency of your objects is not such to demand them residing in L2. Ultimately there's no need to use a commercial database either, as there are compelling open source alternatives, though if your needs are very specific, a commercial database m…

> This has been well understood since antiquity (in the CS world at least).

Yes I believe it was first Cicero that pointed this out. Or perhaps even Aristotle. ;)

Re: What's wrong with 2006 programming?

#50

Post by dasil003 »

jules wrote: post #32
dasil003 wrote: post #29

Earlier quoted context omitted.

You can't take any of his arguments at face value? Like the blocking argument?

No, performance of complex systems is really really hard to predict.

Yes, but again, why do we assume Salvatore is just pulling this stuff out of his ass?
Post reply on HN