Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RediSQL – A Redis module that provides a functional SQL database

github.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: RediSQL – A Redis module that provides a functional SQL database

#41

Post by antirez »

siscia wrote: post #40
arielm wrote: post #39

Earlier quoted context omitted.

I completely agree. As someone who uses redis sql heavily I understand both have their use cases and wouldn't want to merge both to begin with. On top of that, this looks like a simple proxy, so it's only making things less efficient. OP, I'd suggest explaining the benefits better. Maybe I missed it but I don't really get this at all.

The modules is born just to fulfill one of our needs: we had several micro-services using redis and no DB at all, but then we find ourselves in the situation where we needed SQL capabilities, redis module where being announced and we give them a shoot. What you mean by efficiency ? Operation at the second? Sure it is less efficient than Postgres or of SQLite embed in your application. However if you mean convenience…

Hello siscia. What's not clear to me if this is intended for production or if it's an example. I understand the general problem of proving SQLite with a network interface/API, and I guess, but I don't know, there are many solutions for that. If you are trying to solve the problem of SQLite not having a network interface by default, allowing the access from Redis via the Redis protocol, then the implementation is not optimal AFAIK because it does not use any threading but stops Redis while executing the queries in SQLite. Redis modules allow for non blocking commands doing the bulk of the work in threads, so I would suggest at least this approach if this is what you are doing. IMHO also there should be a more sensible conversion of data types from the SQL reply to the Redis protocol.

If Redis has to be stopped during SQL queries IMHO it's better to use another network layer with SQLite.

Re: RediSQL – A Redis module that provides a functional SQL database

#43

Post by gtrubetskoy »

siscia wrote: post #13
stevekemp wrote: post #11

It's a cute project, I remember you posted it before. I remember somebody else had a go at this a few years back https://groups.google.com/forum/#!topic/redis-db/TLfwGfldgUU

It is a little different from Threadis since, in my understanding, threadis is more a redis fork. While rediSQL is a redis module. Using a redis module get you A LOT of thing from free and you really stand on the shoulder of giants.

> threadis is more a redis fork. While rediSQL is a redis module.

Thredis author here - Redis did not support modules back then, had I done this today it prolly would have been a module (though it's been a while since I hacked on Redis and I don't know how modules work)

Re: RediSQL – A Redis module that provides a functional SQL database

#44

Post by siscia »

antirez wrote: post #41
siscia wrote: post #40

Earlier quoted context omitted.

The modules is born just to fulfill one of our needs: we had several micro-services using redis and no DB at all, but then we find ourselves in the situation where we needed SQL capabilities, redis module where being announced and we give them a shoot. What you mean by efficiency ? Operation at the second? Sure it is less efficient than Postgres or of SQLite embed in your application. However if you mean convenience…

Hello siscia. What's not clear to me if this is intended for production or if it's an example. I understand the general problem of proving SQLite with a network interface/API, and I guess, but I don't know, there are many solutions for that. If you are trying to solve the problem of SQLite not having a network interface by default, allowing the access from Redis via the Redis protocol, then the implementation is not…

Ciao Salvatore,

I claim that the module is alpha code.

There is a non-blocking implementation on the way. The problem that I am facing is the tradeoff between using a single thread or multiple threads, but I came to the conclusion that it depends on the use cases. (SQLite on write lock a whole table no a row at the time, so I am expecting a single lockless thread to be faster on write intensive workload and multiple thread be faster on read intensive).

What you suggest as conversion of data types from SQL reply to redis ? Right now a select return a nested array, the outer for the rows the inners for the columns.

Thanks

Re: RediSQL – A Redis module that provides a functional SQL database

#45

Post by gtrubetskoy »

alfanick wrote: post #18
aalekhN wrote: post #15

Earlier quoted context omitted.

Have you thought of ways to implement queries involving LIKE operations ? I worked on a similar project some time ago and one of the limitation for such implementation I encountered was use of glob-style patterns in redis.

This is just a bridge to in-memory sqlite accessible from redis connection – it does not emulate sql for redis data - both databases are completely independent, unfortunately.

That's not hard to do, I did that in Thredis:

  redis 127.0.0.1:6379> set foo bar
  OK

  redis 127.0.0.1:6379> sql "select * from foo"
  1) 1) "key"
     2) "val"
  2) 1) "1"
     2) "bar"

Re: RediSQL – A Redis module that provides a functional SQL database

#46

Post by siscia »

gtrubetskoy wrote: post #43
siscia wrote: post #13

Earlier quoted context omitted.

It is a little different from Threadis since, in my understanding, threadis is more a redis fork. While rediSQL is a redis module. Using a redis module get you A LOT of thing from free and you really stand on the shoulder of giants.

> threadis is more a redis fork. While rediSQL is a redis module. Thredis author here - Redis did not support modules back then, had I done this today it prolly would have been a module (though it's been a while since I hacked on Redis and I don't know how modules work)

Hi :)

sure, I took a deep look at your project and really appreciate you open source it.

Do you have any feedback to share?

Cheers,

Simone

Re: RediSQL – A Redis module that provides a functional SQL database

#47

Post by antirez »

siscia wrote: post #44
antirez wrote: post #41

Earlier quoted context omitted.

Hello siscia. What's not clear to me if this is intended for production or if it's an example. I understand the general problem of proving SQLite with a network interface/API, and I guess, but I don't know, there are many solutions for that. If you are trying to solve the problem of SQLite not having a network interface by default, allowing the access from Redis via the Redis protocol, then the implementation is not…

Ciao Salvatore, I claim that the module is alpha code. There is a non-blocking implementation on the way. The problem that I am facing is the tradeoff between using a single thread or multiple threads, but I came to the conclusion that it depends on the use cases. (SQLite on write lock a whole table no a row at the time, so I am expecting a single lockless thread to be faster on write intensive workload and multiple…

Hey, glad to know you are going multi-threaded on that, since I believe it will be very easy. To start you could just use a thread per request (not just to start, probably even in the long run), immediately block the client via the module API to do this, fire the thread, wait for the thread to complete the work and accumulate the reply, and re-surrect the blocked client, feeding the reply. AFAIK this can work very well.

About the reply, returning a nested array as you are doing may actually be enough after all, even if the reply need to be parsed accordingly to the exact query. The alternative would be to annotate the reply with field names and things like that.

Another thing that probably could be improved: to provide direct support (by passing multiple arguments) to quote strings to pass to SQL, in order to allow the user to have an easy path to avoid security issues.

Re: RediSQL – A Redis module that provides a functional SQL database

#49

Post by siscia »

antirez wrote: post #47
siscia wrote: post #44

Earlier quoted context omitted.

Ciao Salvatore, I claim that the module is alpha code. There is a non-blocking implementation on the way. The problem that I am facing is the tradeoff between using a single thread or multiple threads, but I came to the conclusion that it depends on the use cases. (SQLite on write lock a whole table no a row at the time, so I am expecting a single lockless thread to be faster on write intensive workload and multiple…

Hey, glad to know you are going multi-threaded on that, since I believe it will be very easy. To start you could just use a thread per request (not just to start, probably even in the long run), immediately block the client via the module API to do this, fire the thread, wait for the thread to complete the work and accumulate the reply, and re-surrect the blocked client, feeding the reply. AFAIK this can work very we…

I tried to spawn a new thread for each request and it was embarrassedly slow, maybe I was doing something wrong from my side, but I believe I will settle for a pool.

About the reply the only thing that I see manageable is to include the name of the columns as first row, I was considering the idea but I am not completely sold yet.

I was thinking to bind a particular statement to a key in order to provide the possibility to have standard queries between different clients

  REDISQL.CREATE_STATEMENT insert_user "INSERT INTO users VALUES($1, $2, $3)" 
  > OK
  REDISQL.EXEC_STATEMENT insert_user name, password, email
  > OK
or with multiple insert in the same transaction

  REDISQL.CREATE_TRANSACTION
  > "REDISQL.TRANSACTION_12345"
  REDISQL.EXEC_STATEMENT "REDISQL.TRANSACTION_12345" insert_user name1 password1 email1
  > OK
  REDISQL.EXEC_STATEMENT "REDISQL.TRANSACTION_12345" insert_user name2 password2 email2
  > OK
  REDISQL.EXEC_STATEMENT "REDISQL.TRANSACTION_12345" insert_user name3 password3 email3
  > OK
  REDISQL.COMMIT_TRANSACTION "REDISQL.TRANSACTION_12345"
  > 3 # as the number of statement executed

Re: RediSQL – A Redis module that provides a functional SQL database

#50

Post by badminton1 »

Redis and NoSQL data stores in general are for merge-on-write not merge-on-read. That's why most of them are non relational (no relations, foreign keys or joins).

This project encourages merge-on-read which is exactly the problem k/v stores try to solve in another way (merge-on-write).

The problem with relational data stores is that once you have relations then you need consistency and then you need moving operations to the data store itself (stored procedures), and transactions, etc... and suddenly you need an army of certified DBAs and a $1+ million dollar budget on datastores alone.

It's not the right way to go. Feel free to have SQL but adding joins is a bad idea.

Post reply on HN