Live data from Hacker News

Show HN: Marmot – Single-binary data catalog (no Kafka, no Elasticsearch)

github.com

21–24 of 24 posts

Re: Show HN: Marmot – Single-binary data catalog (no Kafka, no Elasticsearch)

#21

If single binary is a selling point, why not use sqlite instead of postgres?

Postgres has a lot of features such as trigram-based search which is pretty essential if I don't want to use a dedicated search indexer. It's also much better at handling concurrent writes than SQLite.

Re: Show HN: Marmot – Single-binary data catalog (no Kafka, no Elasticsearch)

#23

[flagged]

Hey, that's a good question! At the moment, it treats the latest run as the desired state. So any new changes to a schema will simply overwrite the old version. I'd like to version these so people can navigate schema versions in the UI. If using a plugins, they currently are triggered either via the CLI or a schedule on the UI, so updates will only appear in the catalog after a plugin has run.

I'd also love to have some native integrations beyond Airflow. Once I've matured the existing plugin ecosystem a bit more, it's high on my list (along with column-level lineage).

Re: Show HN: Marmot – Single-binary data catalog (no Kafka, no Elasticsearch)

#24
post #6

Hey there, Great to see Marmot here and I'm a huge fan of your project. Recently, we deployed a catalog but we went with open-metadata https://open-metadata.org/ another amazing project. What we missed on marmot was existing integrations with Airflow and other plugins like Tableau, PowerBI etc as well as other features such as sso, mcp etc. We're an enterprise and needed a more mature product. Fingers crossed marmot…

That's great to know, I wasn't aware anybody even attempted to used it yet! I'm currently in the process of overhauling the Plugin system, it's been quite hard to test some enterprise closed-source integrations like Tableau and Snowflake to build out plugins. SSO is sort kind of available, but undocumented, it currently only supports Okta but I'm working on fleshing out a lot of this in the next big release (along wi…

We gave it a proper deployment and were blown away by the speed but in the end we need a lot of features. SSO/SAML is really important for not just access but also governance. We also miss the Snowflake and dbt plugin among others.

I saw the plugin system but having never written any production ready go code, it doesn't make sense to just use an LLM to generate code and pull requests which you then need to spend time reviewing.

Marmot is a wonderful project and I'm sure it'll be worth the wait.

Post reply on HN