Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ReadySet Core: next-generation SQL caching, freely available

readyset.io

Post reply on HN
Print view View original

51–59 of 59 posts

Re: ReadySet Core: next-generation SQL caching, freely available

#51

Post by zasdffaa »

School-Cotton wrote: post #50
zasdffaa wrote: post #35

There are too many questions here. What does it not do? What's the overhead of monitoring the main DB and how's it done - triggers? Does it need schema changes? What about race conditions - can you guarantee none? What's the memory overhead you need for the cache? Can you control what gets cached? > It can serve millions of reads per second on a single node ... I'm not a network guy but that seems just astonishing -…

> That's an unsolved problem in general, surely? It's not. Materialize (my employer) incrementally maintains views too, using tech (Differential Dataflow) that has existed for almost 10 years: https://cs.stanford.edu/~matei/courses/2015/6.S897/readings/... . ReadySet is based on Noria (Jon Gjengset's Ph.D thesis, explained for non-experts here: https://jon.thesquareplanet.com/noria-in-simpler-terms.pdf ). Taking a re…

We may be talking very different things. From the postgres docs, a sample materialised view https://www.postgresqltutorial.com/postgresql-views/postgres...> (I did a few tweaks as marked)

   CREATE MATERIALIZED VIEW rental_by_category
   AS
   SELECT c.name AS category,
     sum(p.amount) AS total_sales
    FROM (((((payment p
      JOIN rental r ON ((p.rental_id = r.rental_id)))
      JOIN inventory i ON ((r.inventory_id = i.inventory_id)))
      JOIN film f ON ((i.film_id  f.film_id)))  -- tweak
      JOIN film_category fc ON ((f.film_id = fc.film_id)))
      JOIN category c ON ((fc.category_id
It can materialise and efficiently (read: incrementally) maintain the result set of that??

Re: ReadySet Core: next-generation SQL caching, freely available

#52

Post by [deleted] »

zasdffaa wrote: post #51
School-Cotton wrote: post #50

Earlier quoted context omitted.

> That's an unsolved problem in general, surely? It's not. Materialize (my employer) incrementally maintains views too, using tech (Differential Dataflow) that has existed for almost 10 years: https://cs.stanford.edu/~matei/courses/2015/6.S897/readings/... . ReadySet is based on Noria (Jon Gjengset's Ph.D thesis, explained for non-experts here: https://jon.thesquareplanet.com/noria-in-simpler-terms.pdf ). Taking a re…

We may be talking very different things. From the postgres docs, a sample materialised view https://www.postgresqltutorial.com/postgresql-views/postgres... > (I did a few tweaks as marked) CREATE MATERIALIZED VIEW rental_by_category AS SELECT c.name AS category, sum(p.amount) AS total_sales FROM (((((payment p JOIN rental r ON ((p.rental_id = r.rental_id))) JOIN inventory i ON ((r.inventory_id = i.inventory_id))) JOI…

[deleted]

Re: ReadySet Core: next-generation SQL caching, freely available

#53

Post by School-Cotton »

zasdffaa wrote: post #51
School-Cotton wrote: post #50

Earlier quoted context omitted.

> That's an unsolved problem in general, surely? It's not. Materialize (my employer) incrementally maintains views too, using tech (Differential Dataflow) that has existed for almost 10 years: https://cs.stanford.edu/~matei/courses/2015/6.S897/readings/... . ReadySet is based on Noria (Jon Gjengset's Ph.D thesis, explained for non-experts here: https://jon.thesquareplanet.com/noria-in-simpler-terms.pdf ). Taking a re…

We may be talking very different things. From the postgres docs, a sample materialised view https://www.postgresqltutorial.com/postgresql-views/postgres... > (I did a few tweaks as marked) CREATE MATERIALIZED VIEW rental_by_category AS SELECT c.name AS category, sum(p.amount) AS total_sales FROM (((((payment p JOIN rental r ON ((p.rental_id = r.rental_id))) JOIN inventory i ON ((r.inventory_id = i.inventory_id))) JOI…

I believe the non-equijoin will cause problems for Materialize today (I don’t work on our optimizer team, so I’m not 100% sure and don’t take this as authoritative). We might turn that into a cross join followed by a filter.

I will answer that for sure later today when I’m back at my desk.

If you changed that back to an equals sign, yes, we could incrementally maintain your query.

Re: ReadySet Core: next-generation SQL caching, freely available

#54

Post by zasdffaa »

School-Cotton wrote: post #53
zasdffaa wrote: post #51

Earlier quoted context omitted.

We may be talking very different things. From the postgres docs, a sample materialised view https://www.postgresqltutorial.com/postgresql-views/postgres... > (I did a few tweaks as marked) CREATE MATERIALIZED VIEW rental_by_category AS SELECT c.name AS category, sum(p.amount) AS total_sales FROM (((((payment p JOIN rental r ON ((p.rental_id = r.rental_id))) JOIN inventory i ON ((r.inventory_id = i.inventory_id))) JOI…

I believe the non-equijoin will cause problems for Materialize today (I don’t work on our optimizer team, so I’m not 100% sure and don’t take this as authoritative). We might turn that into a cross join followed by a filter. I will answer that for sure later today when I’m back at my desk. If you changed that back to an equals sign, yes, we could incrementally maintain your query.

Thanks. I'd be very interested.

I originally had added the HAVING clause to nastify it further, because this would cause values to appear and disappear, so to handle that you (probably?) have to materialise the entire result of the GROUP BY before applying a HAVING. Which is doable could cause some overhead.

I guess I could see it work for INSERT-only tables, with a lot of headache, but throw in UPDATE and DELETE and it could become awful.

Also we have to agree on what 'incrementally' means :)

Re: ReadySet Core: next-generation SQL caching, freely available

#55

Post by School-Cotton »

zasdffaa wrote: post #54
School-Cotton wrote: post #53

Earlier quoted context omitted.

I believe the non-equijoin will cause problems for Materialize today (I don’t work on our optimizer team, so I’m not 100% sure and don’t take this as authoritative). We might turn that into a cross join followed by a filter. I will answer that for sure later today when I’m back at my desk. If you changed that back to an equals sign, yes, we could incrementally maintain your query.

Thanks. I'd be very interested. I originally had added the HAVING clause to nastify it further, because this would cause values to appear and disappear, so to handle that you (probably?) have to materialise the entire result of the GROUP BY before applying a HAVING. Which is doable could cause some overhead. I guess I could see it work for INSERT-only tables, with a lot of headache, but throw in UPDATE and DELETE and…

Updates and deletes work fine, because Differential Dataflow stores everything as a (row, timestamp, cardinality) tuple. Materializes uses signed integers as its cardinality type. Thus deletes are modeled as (“the row”, t2, -1), which will cancel out with (“the row”, t1, +1), and thus nothing will be returned for that row when the view is queried at any time >= t2. Eventually compaction will cause these cancelled rows to be annihilated entirely and so there will be no lingering space impact.

Re: ReadySet Core: next-generation SQL caching, freely available

#56

Post by cpursley »

marzoeva wrote: post #47
latchkey wrote: post #45

Earlier quoted context omitted.

From the blog post "Rather than forcing developers to switch to a key-value store"... > need Consul, Zookeeper and Redis to make this fly A hard dependency on 3 key value stores?

We need either Consul or Zookeeper (for leader election). No dependency on Redis! That part of the blog post refers to ReadySet having a SQL interface, rather than a key-value one.

Elixir/Erlang is not the most memory efficient, but it could be used for something like this without the need for Redis for Consul/Zookeeper.

Re: ReadySet Core: next-generation SQL caching, freely available

#57

Post by School-Cotton »

zasdffaa wrote: post #54
School-Cotton wrote: post #53

Earlier quoted context omitted.

I believe the non-equijoin will cause problems for Materialize today (I don’t work on our optimizer team, so I’m not 100% sure and don’t take this as authoritative). We might turn that into a cross join followed by a filter. I will answer that for sure later today when I’m back at my desk. If you changed that back to an equals sign, yes, we could incrementally maintain your query.

Thanks. I'd be very interested. I originally had added the HAVING clause to nastify it further, because this would cause values to appear and disappear, so to handle that you (probably?) have to materialise the entire result of the GROUP BY before applying a HAVING. Which is doable could cause some overhead. I guess I could see it work for INSERT-only tables, with a lot of headache, but throw in UPDATE and DELETE and…

I checked the plan. This will involve a cross join, and maintaining the entire result of the group by before the having (the former could possibly be avoided with some future work in Materialize to allow range queries; the latter is probably essential). Without your tweaks, the only required state is the join indexes.
Post reply on HN