Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cache is the new RAM

blog.memsql.com

Post reply on HN
Print view View original

41–50 of 97 posts

Re: Cache is the new RAM

#41

Post by thomasrynne »

zzleeper wrote: post #26
temuze wrote: post #7

The database I want still doesn't exist. Here's what I want: - Easy sharding, a la Elasticsearch. I want virtual shards that can be moved node to node and an easy to understand primary/replica shard system for write/reads. I want my DB nodes to find each other with an easy discovery system with plugins for AWS/Azure/Digital Ocean etc. - Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQ…

Quick question about the pony. I mean, about Cstore. What do you think is the best option for doing that nowadays? (By that I mean columnar stores that are hopefully optimized with the usual tricks like inlined compression. I know Vertica works but if are there cheaper (free?) alternatives?

I don't know if this is the kind of thing you had in mind but you could try monetdb. I ran some tests a while ago and it performed very well.

Re: Cache is the new RAM

#42

Post by platz »

mpweiher wrote: post #40
jandrewrogers wrote: post #13

A couple points I would make with respect to the article: - In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler. In-memory databases are generally only faster if the disk-backed database uses mmap() for cache replacement or similarly terrible I/O scheduling. The big advantage of in-memory databases is that you avoid the enormously complicated implementation task…

> In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler. Hmm...Michael Stonebraker would probably disagree with you on that. For his different newer projects, he claims and somewhat convincingly shows 10x or more performance improvements. The analysis is that ~10% of a typical disk-based system is useful work. By radically simplifying as a result of doing away wit…

It's not just the disk store, also avoiding the time spent locking and unlocking in row-based architectures (i.e. using lock-free algorithms)

http://www.se-radio.net/2013/12/episode-199-michael-stonebra...

Re: Cache is the new RAM

#43

Post by icelancer »

GrinningFool wrote: post #39

> It pisses me off to no end these developers have to out-think SQL and re-invent the whole damn new wheel for "efficiency's" sake. Worked with someone who did this a couple years ago. I keep trying to get him to explain why he was reinventing sql, and never did get a clear answer to that. > just waggles their dick around saying Sorry to be the one to say it, but I don't think that part was necessary or appropriate t…

It's more of a mental one than a physical one, but point taken.

Re: Cache is the new RAM

#44

Post by mpweiher »

platz wrote: post #42
mpweiher wrote: post #40

Earlier quoted context omitted.

> In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler. Hmm...Michael Stonebraker would probably disagree with you on that. For his different newer projects, he claims and somewhat convincingly shows 10x or more performance improvements. The analysis is that ~10% of a typical disk-based system is useful work. By radically simplifying as a result of doing away wit…

It's not just the disk store, also avoiding the time spent locking and unlocking in row-based architectures (i.e. using lock-free algorithms) http://www.se-radio.net/2013/12/episode-199-michael-stonebra...

"...single-threaded, lock-free, doesn’t require disk I/O in the critical path, ..."

IIRC, the single-threadedness is made possible in large part by not having I/O on the critical path. (When you wait on I/O, a single threaded design stalls completely, so either you multi-thread somehow or your throughput dies).

:-)

Re: Cache is the new RAM

#46

Post by maerF0x0 »

lallysingh wrote: post #6
maerF0x0 wrote: post #4

Amazon doesnt expose much of these statistics (how fast of ram do i get with a M3.large or a c3.med etc) . Does this mean real performance is for those who own their servers?

It's easy enough to find out. Create an array of linked-list nodes, and have them point to each other randomly (hint: std::random_shuffle w/an array of indices). Write a routine that traverses the list N times. Time the routine for larger lists. You should see a jump as your list gets larger than each stage of cache.

Seems to me that would be very dependent on neighbours and their current workloads?

Re: Cache is the new RAM

#47

Post by nehan »

temuze wrote: post #7

The database I want still doesn't exist. Here's what I want: - Easy sharding, a la Elasticsearch. I want virtual shards that can be moved node to node and an easy to understand primary/replica shard system for write/reads. I want my DB nodes to find each other with an easy discovery system with plugins for AWS/Azure/Digital Ocean etc. - Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQ…

re: materialized views -- you can kind of fake them in PostgreSQL by creating a view, populating it, and then defining triggers on insert, delete, and updates to your source tables.

Re: Cache is the new RAM

#48

Post by phusion »

icelancer wrote: post #30
temuze wrote: post #7

The database I want still doesn't exist. Here's what I want: - Easy sharding, a la Elasticsearch. I want virtual shards that can be moved node to node and an easy to understand primary/replica shard system for write/reads. I want my DB nodes to find each other with an easy discovery system with plugins for AWS/Azure/Digital Ocean etc. - Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQ…

I have a LOT fewer requests than you, but this one is just so irritating: >- Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQL client a say "go! You already know how to use this!". If I want a new feature, then dammit, build on top of SQL the way PostgreSQL has. Odds are, regardless if its some JSON API or SQL, my language will have a client for it that will be superior than writing r…

Almost blasted my screen with yerba mate after the dick waggling remark, kudos.

Re: Cache is the new RAM

#49

Post by Ntrails »

cturner wrote: post #31
temuze wrote: post #7

The database I want still doesn't exist. Here's what I want: - Easy sharding, a la Elasticsearch. I want virtual shards that can be moved node to node and an easy to understand primary/replica shard system for write/reads. I want my DB nodes to find each other with an easy discovery system with plugins for AWS/Azure/Digital Ocean etc. - Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQ…

It's impractical to write maintainable code in the forms of SQL I'm familiar with. But maybe someone here knows better. You should be able to abstract logic out to blocks for the duration of the query. Along these lines: cset means select id, venue_id from customer select venue_name from venue where id = cset.venue_id cset would be scoped to your connection. Instead of pages of copy-and-paste queries where the levels…

I mean, you can also use CTEs for that but it is of course limited.

For all of the flaws of managing SQL, it's when I get handed giant swathes of dynamic SQL that I die a little inside.

Re: Cache is the new RAM

#50

Post by peterwwillis »

GrinningFool wrote: post #39

> It pisses me off to no end these developers have to out-think SQL and re-invent the whole damn new wheel for "efficiency's" sake. Worked with someone who did this a couple years ago. I keep trying to get him to explain why he was reinventing sql, and never did get a clear answer to that. > just waggles their dick around saying Sorry to be the one to say it, but I don't think that part was necessary or appropriate t…

How about 'flails their variously-gendered techno-boner' ?

(Honestly, I don't get why foul language would be off-limits here, as long as you're not being rude)

Post reply on HN