Live data from Hacker News

Show HN: A Schemaless Data Store Within Your SQL Database

schemafreesql.com

21–30 of 47 posts

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#21

This sounds like a lot of reinvention of the JSON/JSONB support that Postgres has already had since 9.2/9.4. They have a slide deck with a variety of in-depth examples. https://wiki.postgresql.org/images/7/7c/JSON-by-example_FOSD... CREATE INDEX review_review_jsonb ON reviews USING GIN (review_jsonb); -- Select data with JSON SELECT review_jsonb#>> '{product,title}' AS title , avg((review_jsonb#>> '{review,rating}'):…

[deleted]

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#22
post #8
post #3

Interesting project! I'm curious about how this compares to other tech: - How is data stored internally? - What are the tradeoffs with this implementation? - Do you generate indexes on data as it comes in? - How are nested keys handled? - Are indexes eventually consistent?

Thanks! - How is data stored internally? During our beta we will be hosting the databases and the underlying table structures are not viewable. However, on release, you will have full access to inspect the tables and indexes as they will be created within your own database. - What are the tradeoffs with this implementation? Depends on what you compare it to, but let's assume we are looking at the tradeoffs of SFSQL v…

And just to be clear. This is NOT a mutually exclusive solution. Just as people use a combination of different databases for different projects and even within the same project, SFSQL can be used alongside other solutions. It is especially convenient to use as an additional datastore when you are already storing your data in a SQL DB because there is no additional devops beyond what was already in place for that SQL DB.

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#23
post #20

Earlier quoted context omitted.

Looks like chrome pw gen does not generate a special character we have removed that restriction. Please try again.

I think it does but not always. It works now, thanks! I don't know if you've seen https://news.ycombinator.com/showhn.html one of the things it talks about is minimizing 'try it' friction and this one felt pretty high, even without the password thing. People bounce at not just account creation but also stuff like 'beta access' especially if it's not clear from the 'beta access' verbiage if it means 'access right now'…

I see, thanks we will change the verbiage.

We tried to remove friction by allowing beta account creation without a password and one click demo account creation.

You can start using the beta service without creating a password, maybe that was not clear? We have "beta" there just so you know it's not production ready. We can put the beta verbiage in the account creation page description but not in the Create Account Button call to action.

Thanks for the feedback.

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#24
post #16
post #8

Earlier quoted context omitted.

Thanks! - How is data stored internally? During our beta we will be hosting the databases and the underlying table structures are not viewable. However, on release, you will have full access to inspect the tables and indexes as they will be created within your own database. - What are the tradeoffs with this implementation? Depends on what you compare it to, but let's assume we are looking at the tradeoffs of SFSQL v…

It sounds like you have single-column indexes on every column and then would rely on an intersecting query plan (e.g. bitmap heap scan) if you query multiple columns. This isn't just slightly slower, it depends a lot on your data how slow this is. It's potentially linear in the length of single-column matches. E.g. a query plan for (region='us' AND product='1234' AND year='2021') could scan the region index for all '…

That WOULD be horrible. But we arent doing that. In essence we found that JOINS using optimized indexes are less costly than full table scans (on unindexed attributes).

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#26
post #18

This sounds like a lot of reinvention of the JSON/JSONB support that Postgres has already had since 9.2/9.4. They have a slide deck with a variety of in-depth examples. https://wiki.postgresql.org/images/7/7c/JSON-by-example_FOSD... CREATE INDEX review_review_jsonb ON reviews USING GIN (review_jsonb); -- Select data with JSON SELECT review_jsonb#>> '{product,title}' AS title , avg((review_jsonb#>> '{review,rating}'):…

Glad you asked about the JSON datatype. The Postgres JSON type is a great addition and certainly works well. Especially for situations where you have some common traits shared across something like 'products' (e.g. price), store those common traits in columns and then use the JSON type to store the uncommon columns (e.g. 'flavor'). Where SFSQL really helps is: - When new attributes are created that need indexing, you…

How do you protect against user input that produces a psychopathic volume of indexes?

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#27
post #7

Are you unrolling the nested JSON data structures and storing as traditional K/Vs in an EAV pattern? Possibly using one table for each datatype or using a sparse table? I'd be curious how this performs for complex queries - does this rely heavily on index intersection?

Yes, an EAV pattern is being used. The end result in query speed with the particular table designs and indexes is essentially just like what index intersection gives you but without the setup. And of course EAV is not a pattern that you can easily roll out by hand when you need it.

Please try this demo which is a sample of some queries against Clinical Trials data. https://schemafreesql.com/demo.html#clinicalTrial Although this demo data set is a limited size, the same queries when run against the entire clinicalTrial dataset performed very respectably.

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#30
post #18

Earlier quoted context omitted.

Glad you asked about the JSON datatype. The Postgres JSON type is a great addition and certainly works well. Especially for situations where you have some common traits shared across something like 'products' (e.g. price), store those common traits in columns and then use the JSON type to store the uncommon columns (e.g. 'flavor'). Where SFSQL really helps is: - When new attributes are created that need indexing, you…

How do you protect against user input that produces a psychopathic volume of indexes?

I guess that would be on the developer side to sanitize/limit/clean in some way, like limiting the number of key-value pairs, object depth, etc?
Post reply on HN