Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A Schemaless Data Store Within Your SQL Database

schemafreesql.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#41

Post by mdellavo »

etirk wrote: post #37
mdellavo wrote: post #33

Earlier quoted context omitted.

EAV is a pattern you can easily roll out - https://docs.sqlalchemy.org/en/14/orm/examples.html#module-e... Another commenter illustrated some of the issues you will run into with this pattern at scale. It would probably help to benchmark some complex queries on a sizable data set. And compare against mongo, postgres jsonb, vanilla eav, clickhouse, etc. Without much information to go on, it's hard to know what this is…

I agree with getting some benchmarks up there. Until that's done, I can share what some preliminary tests revealed. Just to give you more of feel for where this tool might fit. We imported the complete Clinical Trials dataset (380k docs ~ 200 attributes each) into SFSQL, Mongo and a Postgres JSON column. Import speed with raw documents: Mongo and Postgres win hands down. Why? Very little processing to be done on thei…

what about applying a GIN index on the JSONB?

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#42

Post by etirk »

mdellavo wrote: post #41
etirk wrote: post #37

Earlier quoted context omitted.

I agree with getting some benchmarks up there. Until that's done, I can share what some preliminary tests revealed. Just to give you more of feel for where this tool might fit. We imported the complete Clinical Trials dataset (380k docs ~ 200 attributes each) into SFSQL, Mongo and a Postgres JSON column. Import speed with raw documents: Mongo and Postgres win hands down. Why? Very little processing to be done on thei…

what about applying a GIN index on the JSONB?

Did not try that kind of index on postgres. Thx. Will keep that on the list for benchmarking.

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#43

Post by AtlasBarfed »

Why isn't it horizontally scalable? Are there tons of joins?

I wrote a JSON/Document/Property Graph database front end with Cassandra and DynamoDB backends, so that is ... kinda ... schemaless.

I ended up doing a lot of things that RethinkDB and OrientDB did but didn't scale like cassandra. I was trying to do a streaming document layer. Also a JSON document database with simple relations is a property graph database.

But Elassandra and similar projects kinda did the same thing so I dropped it. But I came out of it thinking it shouldn't be hard to scale JSON, which is basically schemaless.

I had schema-optional things too where certain named properties/elements of a JSON doc would be rendered onto a fixed table and schema, are you.guys doing anything like that? I think a schema optional or options for partial schemas for documents would enable the transition of schemaless data to schema'd stores in enterprise system evolution.

Do you guys intermix all documents like graph databases do, which seems a bit insane if you have to do some large scale table scan or migration of data and have to sift through data? Or build an index on a heap of documents with only some of them needing indexing?

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#44

Post by rmbyrro »

dfragnito wrote: post #36
rmbyrro wrote: post #35

Earlier quoted context omitted.

I don't mean to be negative, but just two cents of feedback: I believe (could be wrong) many developers would be worried about using this. Seems to me you're a small shop (?). AWS can get away with a proprietary DB API like DynamoDB's. They are big and can provide strong assurances. Since SFSQL is just starting out, by a small shop, it would be a high risk to rely on it being a closed source, proprietary API. Might b…

In a self hosted model developers would host SFSQL on their own infrastructure, which could be AWS, Google, Azure, DO, etc.. There is no reliance on us. Yes we are maintaining the API and underlying implementation. Software is a Service thus the subscription model.

Yeah, I totally agree that providing the self-hosted option is a big plus.

Covers maintainability concerns at least in the short term. Some might still be worried about long-term security & bug fixing, support, etc.

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#45

Post by rmbyrro »

Hey, quick question: would this by any chance work as an AWS Lambda container image [1] connected to Aurora Serverless [2] (MySQL or Postgres), perhaps using the Data API [3]?

If yes, I think this is a big plus for teams looking for a managed infra.

[1] https://docs.aws.amazon.com/lambda/latest/dg/lambda-images.h...

[2] https://aws.amazon.com/rds/aurora/serverless/

[3] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#46

Post by dfragnito »

rmbyrro wrote: post #45

Hey, quick question: would this by any chance work as an AWS Lambda container image [1] connected to Aurora Serverless [2] (MySQL or Postgres), perhaps using the Data API [3]? If yes, I think this is a big plus for teams looking for a managed infra. [1] https://docs.aws.amazon.com/lambda/latest/dg/lambda-images.h... [2] https://aws.amazon.com/rds/aurora/serverless/ [3] https://docs.aws.amazon.com/AmazonRDS/latest/Aur…

It would not work in its current state but we have plans for a Lambda compatable offering. We are also looking at fly.io and other "serverless" function providers.

Re: Show HN: A Schemaless Data Store Within Your SQL Database

#47

Post by productceo »

>Web App Development Environment we developed back in 1999. It was comprised of an IDE, Web Server, Object Store, Virtual File System, Template System, and polyglot (Java, JavaScript, and Python).

When you made this, why did you believe building this was what people needed?

And what did you end up observing that made you discontinue the project?

Post reply on HN