Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cache is the new RAM

blog.memsql.com

Post reply on HN
Print view View original

31–40 of 97 posts

Re: Cache is the new RAM

#31

Post by cturner »

temuze wrote: post #7

The database I want still doesn't exist. Here's what I want: - Easy sharding, a la Elasticsearch. I want virtual shards that can be moved node to node and an easy to understand primary/replica shard system for write/reads. I want my DB nodes to find each other with an easy discovery system with plugins for AWS/Azure/Digital Ocean etc. - Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQ…

It's impractical to write maintainable code in the forms of SQL I'm familiar with. But maybe someone here knows better.

You should be able to abstract logic out to blocks for the duration of the query. Along these lines:

    cset means select id, venue_id from customer

    select venue_name from venue where id = cset.venue_id
cset would be scoped to your connection.

Instead of pages of copy-and-paste queries where the levels are all mixed up with one another. You can get something similar by polluting the namespace (e.g. views, temporary tables) but that's hacky and not the same as good.

Re: Cache is the new RAM

#32

Post by icelancer »

wyager wrote: post #25
temuze wrote: post #7

The database I want still doesn't exist. Here's what I want: - Easy sharding, a la Elasticsearch. I want virtual shards that can be moved node to node and an easy to understand primary/replica shard system for write/reads. I want my DB nodes to find each other with an easy discovery system with plugins for AWS/Azure/Digital Ocean etc. - Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQ…

>Fucking SQL. I don't want to learn your stupid DSL... If I want a new feature, then dammit, build on top of SQL What I hear is "I don't know anything better than the incredibly cut-rate solution I've been taught how to abuse, so I'm opposed to any improvements." SQL is a human interface language. I absolutely don't understand the desire that people have to use it for IPC. The tremendous set of SQL injection vulnerab…

It isn't what I am willing or able to learn; I am a good software developer that can pick up your new DSL or whatever crap you want to invent.

But guess what? The guys making $70k/year doing DB analysis and reporting who have little to no software development background can't. And they are the ones that live in the hell you've created.

Re: Cache is the new RAM

#33

Post by activus »

temuze wrote: post #7

The database I want still doesn't exist. Here's what I want: - Easy sharding, a la Elasticsearch. I want virtual shards that can be moved node to node and an easy to understand primary/replica shard system for write/reads. I want my DB nodes to find each other with an easy discovery system with plugins for AWS/Azure/Digital Ocean etc. - Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQ…

These would be great. One thing I had need for at a previous job was a combined search / compute engine.

We basically had a search cluster that would return ids that would get joined into the actual data in another system, then we'd do heavy post-processing and push it off to another system for slicing and dicing.

Not sure if it's any use to anyone else, but a system that allowed people to push search queries into a cluster, do some user defined post-processing and then have the engine pipe that to an endpoint would have been interesting for that use case.

I believe there's some Hadoop eco-system bits that do a bit of this now (Impala comes to mind).

Re: Cache is the new RAM

#34

Post by sehrope »

temuze wrote: post #27
jsmeaton wrote: post #24

Earlier quoted context omitted.

(I'm coming from the perspective of having to use Oracle at work - and I'm happy to be corrected if my experiences do not match your wishlist) > Easy sharding, a la Elasticsearch Oracle RAC lets you add nodes to the cluster. Helps with CPU - doesn't really help with disk contention > Fucking SQL. If I want a new feature, then dammit, build on top of SQL the way PostgreSQL has. There is no good story here for Oracle.…

I come from a PSQL background where materialized views are manually refreshed :) After skimming Oracle's docs, I'd settle for something as simple as: CREATE MATERIALIZED VIEW my_mview REFRESH FAST ON COMMIT THROTTLE 10s AS ( ... ) Does Oracle use deltas to refresh these mviews? It'd super cool if it did! For example, if the mview is a sum of a bunch of rows, then it could use the logged changes on those rows and its…

Oracle supported both full refresh and incremental refresh as of at least 10 years ago (maybe longer).

Fast refresh requires creating "MATERIALIZED VIEW LOGS" on the source table(s) and covers most (but not all) aggregations/groupings in the mview SQL. Once it's setup, DML to the source tables gets logged to the mview logs and a "fast" refresh of the mview uses it to incrementally update the mview's data.

Re: Cache is the new RAM

#35

Post by wyager »

icelancer wrote: post #32
wyager wrote: post #25

Earlier quoted context omitted.

>Fucking SQL. I don't want to learn your stupid DSL... If I want a new feature, then dammit, build on top of SQL What I hear is "I don't know anything better than the incredibly cut-rate solution I've been taught how to abuse, so I'm opposed to any improvements." SQL is a human interface language. I absolutely don't understand the desire that people have to use it for IPC. The tremendous set of SQL injection vulnerab…

It isn't what I am willing or able to learn; I am a good software developer that can pick up your new DSL or whatever crap you want to invent. But guess what? The guys making $70k/year doing DB analysis and reporting who have little to no software development background can't. And they are the ones that live in the hell you've created.

>And they are the ones that live in the hell you've created.

I didn't create SQL ;)

Re: Cache is the new RAM

#36

Post by adlpz »

cturner wrote: post #31
temuze wrote: post #7

The database I want still doesn't exist. Here's what I want: - Easy sharding, a la Elasticsearch. I want virtual shards that can be moved node to node and an easy to understand primary/replica shard system for write/reads. I want my DB nodes to find each other with an easy discovery system with plugins for AWS/Azure/Digital Ocean etc. - Fucking SQL. I don't want to learn your stupid DSL. I want to give coworkers a SQ…

It's impractical to write maintainable code in the forms of SQL I'm familiar with. But maybe someone here knows better. You should be able to abstract logic out to blocks for the duration of the query. Along these lines: cset means select id, venue_id from customer select venue_name from venue where id = cset.venue_id cset would be scoped to your connection. Instead of pages of copy-and-paste queries where the levels…

Well that's why (in my opinion) you should _generate_ the queries, not _write_ them. Use an ORM or whatever flavor of basically a SQL generator you prefer. Or even easier, use any programming language to concatenate around some strings and build your SQL.

The power of using SQL comes from the fact that, no matter what, it's always there. You can always write/buy/find another client for your DB in another language, and as long as it generates SQL, it's all good.

Re: Cache is the new RAM

#37

Post by toomim »

jandrewrogers wrote: post #13

A couple points I would make with respect to the article: - In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler. In-memory databases are generally only faster if the disk-backed database uses mmap() for cache replacement or similarly terrible I/O scheduling. The big advantage of in-memory databases is that you avoid the enormously complicated implementation task…

Can you give some refs to make these statements more real?

I'm trying to understand whether an in-memory db would be faster than postgres, for instance. What's the postgres I/O scheduler, and is it good? Are there benchmarks somewhere showing the difference?

Why does the I/O scheduler make a difference for in-memory vs. disk databases? Is this a subsystem that caches the database in memory? Are you saying that, with proper caching, a disk-based database is as efficient as an in-memory db?

Re: Cache is the new RAM

#38

Post by nemo44x »

This article is full of so much logical fallacy I'm surprised it made it here. And it's an advertisement none the less.

Creates a red herring by stating he's been doing this a long time and has seen it all.

Creates straw man after straw man in the trashing of memory caches (avoids their use cases), Dynamo (there's a good reason tons of people use various NoSQL Databases) and Hadoop (C'mon, now).

He also creates more logical fallacy in calling various concepts silver bullets that ended up having problems. I don't think anyone serious about technology thinks replication, sharding, load balancing "solves everything". Nothing is a silver bullet and anyone who says something is is selling you something...

And then he fails to really address the MemSQL uses replication, sharding (in a limited sense since the core SQL concept of a JOIN is wrecked here and they have a big warning on their troubleshooting page about an error you users must see often).

SQL is great but I have plenty of great reasons to use other data stores. SQL isn't a silver bullet for data.

Point is, he is calling MemSQL a silver bullet and is obviously trying to sell something while ripping plenty of great ideas and concepts by picking the worst implementations of them and largest misunderstandings of them.

Re: Cache is the new RAM

#39

Post by GrinningFool »

> It pisses me off to no end these developers have to out-think SQL and re-invent the whole damn new wheel for "efficiency's" sake.

Worked with someone who did this a couple years ago. I keep trying to get him to explain why he was reinventing sql, and never did get a clear answer to that.

> just waggles their dick around saying

Sorry to be the one to say it, but I don't think that part was necessary or appropriate to this community. Also, using the collective gender-neutral "their" in conjunction with dick waggling is just funny, now that I think about it.

Re: Cache is the new RAM

#40

Post by mpweiher »

jandrewrogers wrote: post #13

A couple points I would make with respect to the article: - In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler. In-memory databases are generally only faster if the disk-backed database uses mmap() for cache replacement or similarly terrible I/O scheduling. The big advantage of in-memory databases is that you avoid the enormously complicated implementation task…

> In-memory databases offer few advantages over a disk-backed database with a properly designed I/O scheduler.

Hmm...Michael Stonebraker would probably disagree with you on that. For his different newer projects, he claims and somewhat convincingly shows 10x or more performance improvements. The analysis is that ~10% of a typical disk-based system is useful work. By radically simplifying as a result of doing away with the disk-store, you remove those overheads.

A similar argument (also with benchmarks) is made by the RAMCloud people, they claim up to 1000x perf. increase over disk-based storage for a data-center.

Since I am mostly a dabbler (but have also managed to outperform RDBMs by factor of 1000 or more using RAM-based techniques), I would be curious as to what these papers get wrong.

http://downloads.voltdb.com/datasheets_collateral/technical_...

http://cs-www.cs.yale.edu/homes/dna/papers/vldb07hstore.pdf

https://ramcloud.atlassian.net/wiki/display/RAM/RAMCloud

Post reply on HN