Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databases should contain their own Metadata – Use SQL Everywhere

floedb.ai

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Databases should contain their own Metadata – Use SQL Everywhere

#31

Post by evanelias »

tkejser wrote: post #25
matheusalmeida wrote: post #18

Earlier quoted context omitted.

The key difference is that it's not just about schema metadata (tables, indexes, views, columns, etc...). PostgreSQL is fabulous regarding this. Even native types are part of the catalog (pg_catalog). Things are great in your DB... until they aren't. The post is about making observability a first-class citizen. Plans and query execution statistics, for example, queryable using a uniform interface (SQL) without the ne…

Thank you and yes! By making the entire architecture of the database visible via system objects - you allow the user to form a mental model of how the database itself works. Instead of it being just a magic box that runs queries - it becomes a fully instrumented data model of itself. Now, you could say: "The database should just work" and perhaps claim that it is design error when it doesn't. Why do I need instrument…

This sounds conceptually similar to performance_schema [1] in MySQL or MariaDB, which is a built-in feature originally introduced in MySQL 5.5 (2010). Or perhaps the easier-to-use sys schema [2], which wraps performance_schema among other things, introduced in MySQL 5.7 (2015).

It's great to have that observability functionality, but I don't really understand the purpose of writing a new DBMS from scratch to add this though. Why not get something merged into Postgres core?

[1] https://dev.mysql.com/doc/refman/8.4/en/performance-schema.h...

[2] https://dev.mysql.com/doc/refman/8.4/en/sys-schema.html

Re: Databases should contain their own Metadata – Use SQL Everywhere

#32

Post by tkejser »

evanelias wrote: post #31
tkejser wrote: post #25

Earlier quoted context omitted.

Thank you and yes! By making the entire architecture of the database visible via system objects - you allow the user to form a mental model of how the database itself works. Instead of it being just a magic box that runs queries - it becomes a fully instrumented data model of itself. Now, you could say: "The database should just work" and perhaps claim that it is design error when it doesn't. Why do I need instrument…

This sounds conceptually similar to performance_schema [1] in MySQL or MariaDB, which is a built-in feature originally introduced in MySQL 5.5 (2010). Or perhaps the easier-to-use sys schema [2], which wraps performance_schema among other things, introduced in MySQL 5.7 (2015). It's great to have that observability functionality, but I don't really understand the purpose of writing a new DBMS from scratch to add this…

Merging to PostgreSQL core for something that needs to run on top of a Petabytes of data in the cloud, on Iceberg, with an advanced query planner and a high speed SIMD engine.... AND trying to squeeze into the "pg_" naming mess?

I don't think so...

And yes, its conceptually similar to MySQL and also conceptually similar to SQL Servers implementation from 1997. That's by the design. Obviously, we are not writing a new DBMS from scratch just to add system objects.

Have a look at some of the other blogs on that site to see what we are up to. Basically, we want to give you an experience that resemblers that instrumentation you got used to from the on-premise databases, but one that can run on top of Iceberg in the cloud.

Re: Databases should contain their own Metadata – Use SQL Everywhere

#33

Post by evanelias »

tkejser wrote: post #32
evanelias wrote: post #31

Earlier quoted context omitted.

This sounds conceptually similar to performance_schema [1] in MySQL or MariaDB, which is a built-in feature originally introduced in MySQL 5.5 (2010). Or perhaps the easier-to-use sys schema [2], which wraps performance_schema among other things, introduced in MySQL 5.7 (2015). It's great to have that observability functionality, but I don't really understand the purpose of writing a new DBMS from scratch to add this…

Merging to PostgreSQL core for something that needs to run on top of a Petabytes of data in the cloud, on Iceberg, with an advanced query planner and a high speed SIMD engine.... AND trying to squeeze into the "pg_" naming mess? I don't think so... And yes, its conceptually similar to MySQL and also conceptually similar to SQL Servers implementation from 1997. That's by the design. Obviously, we are not writing a new…

> something that needs to run on top of a Petabytes of data in the cloud, on Iceberg, with an advanced query planner and a high speed SIMD engine

Part of my confusion was that this blog post makes no mention whatsoever of any of those things!

It gave me the (incorrect) impression that this observability functionality was the purpose of the product. And it is worded in a way which makes no mention of prior art in built-in DBMS observability.

Looking at the other threads here, I don't think I'm the only one who was confused about that. A couple intro paragraphs to the product might help a lot.

Re: Databases should contain their own Metadata – Use SQL Everywhere

#34

Post by tkejser »

whynotmaybe wrote: post #27
zabzonk wrote: post #4

Not clear if the author realises that all commercial SQL database engines support querying of the database's metadata using SQL. Or maybe I have misunderstood - I only skimmed the article.

I don't think it's as easy to do the example in the article just by using information_schema. > Which tables have a column with the name country where that column has more than two different values But on their product page, the definition of floesql left me puzzled > It uses intelligent caching and LLVM-based vectorized execution to deliver the query execution speed your business users expect. > With its powerful qu…

INFORMATION_SCHEMA is a good start, but it does not get you to full metadata flexibility. The columns you need just aren't there. It is good to have a standard for the metadata - but the standard isn't ambitious enough (a point I also make in the blog and as you observe, the sample query isn't possible on Information Schema alone)

The Floe engine is a full database on top of Iceberg and Delta storage. The system views are just the tip of the iceberg. We will be blogging more about what we are building.

Re: Databases should contain their own Metadata – Use SQL Everywhere

#36

Post by tkejser »

evanelias wrote: post #33
tkejser wrote: post #32

Earlier quoted context omitted.

Merging to PostgreSQL core for something that needs to run on top of a Petabytes of data in the cloud, on Iceberg, with an advanced query planner and a high speed SIMD engine.... AND trying to squeeze into the "pg_" naming mess? I don't think so... And yes, its conceptually similar to MySQL and also conceptually similar to SQL Servers implementation from 1997. That's by the design. Obviously, we are not writing a new…

> something that needs to run on top of a Petabytes of data in the cloud, on Iceberg, with an advanced query planner and a high speed SIMD engine Part of my confusion was that this blog post makes no mention whatsoever of any of those things! It gave me the (incorrect) impression that this observability functionality was the purpose of the product. And it is worded in a way which makes no mention of prior art in buil…

That's fair feedback and I shall take that into account in future blogs.

Thanks for letting me know - you can stare yourself blind on that stuff

Re: Databases should contain their own Metadata – Use SQL Everywhere

#37

Post by whynotmaybe »

tkejser wrote: post #34
whynotmaybe wrote: post #27

Earlier quoted context omitted.

I don't think it's as easy to do the example in the article just by using information_schema. > Which tables have a column with the name country where that column has more than two different values But on their product page, the definition of floesql left me puzzled > It uses intelligent caching and LLVM-based vectorized execution to deliver the query execution speed your business users expect. > With its powerful qu…

INFORMATION_SCHEMA is a good start, but it does not get you to full metadata flexibility. The columns you need just aren't there. It is good to have a standard for the metadata - but the standard isn't ambitious enough (a point I also make in the blog and as you observe, the sample query isn't possible on Information Schema alone) The Floe engine is a full database on top of Iceberg and Delta storage. The system view…

Good, execution planning for majors DBMS didn't receive any ground breaking evolution because it can be considered a "solved" problem but I'm always curious about new ways to address it.
Post reply on HN