[deleted]
Choose DuckDB rather than SQLite
51–60 of 65 posts
Re: Choose DuckDB rather than SQLite
#52DuckDB is modern but written in C++ and crashes in production more than SQLite, which is old and it doesn’t really crash. you have to build in resilience to use DuckDb.
Wait seriously? That's a dealbreaker if true.
Re: Choose DuckDB rather than SQLite
#53I ran this through an AI checker and it flagged half of it immediately. @dang, I know Substack just enabled Pangram integration, is there anyway you could get Y Combinator to spring for a Pangram subscription for the front page or something ?
AI comments aren't allowed on HN. AI submissions are.
And shouldn't be.
Re: Choose DuckDB rather than SQLite
#54How are these two DB engines even comparable other than at the edges? They handle two completely separate workload types: one is more a general-purpose DB engine and the other is specifically for columnar datasets, analytics and the like -- of course a hand-tuned DB engine is going to destroy SQLite on any reasonable benchmark: SQLite wouldn't be optimized for that hand-tuned use-case whereas something like DuckDB is…
SQLite is _the_ tool of choice for local SQL databases with minimal overhead. If you needed a single file DB for an OLAP workload, SQLite was still the best option even if the technology wasn't an ideal fit. Duckdb is exciting specifically because SQLite/duckdb aren't comparable; we can stop shoehorning OLAP into an OLTP database. I ran into this myself; I tried using SQLite to store the results of whole-internet rDN…
Sure, but this is what I'm getting at. :) SQLite is a general-purpose engine that is absolutely amazing at what it does and where it's used (and in many cases even where it isn't appropriate). But TFA was about comparing the two which just doesn't work because one is hand-tuned for a specific use-case while the other is not.
Re: Choose DuckDB rather than SQLite
#55How are these two DB engines even comparable other than at the edges? They handle two completely separate workload types: one is more a general-purpose DB engine and the other is specifically for columnar datasets, analytics and the like -- of course a hand-tuned DB engine is going to destroy SQLite on any reasonable benchmark: SQLite wouldn't be optimized for that hand-tuned use-case whereas something like DuckDB is…
With indexes SQLite is very very fast even for aggregation at medium scales. And DuckDB is reasonably fast for even single record writes. ~1000x slower than SQLite, but that's still pretty fast if you're only doing a few hundred writes per second or batching.
Agreed. And DuckDB certainly has a very nice API. But I wouldn't use it for a general dataset (think document storage).
Re: Choose DuckDB rather than SQLite
#56Too bad Android doesn't have JDBC APIs. Getting the native binaries compiled on Android is the easy part, but there is no straightforward way to access it from Java. The Room APIs are tied to SQLite as well.
You could use the framework built-in sqlite, or you could drop in h2 and use that. It's much more advanced than sqlite.
Re: Choose DuckDB rather than SQLite
#57wait, it means you should not save all traces to Langfuse?
what is the primary usecase of Langfuse? exporting some subset of all traces for some quality checks etc.?
Re: Choose DuckDB rather than SQLite
#58Earlier quoted context omitted.
Let's keep discussion bar high. I've just checked their website, and they state "relying on ClickHouse to power these analytics use cases". That's not OLTP. https://clickhouse.com/comparison/postgresql Fair to say, seeing 1000x w/o any trace of proof won't help me to choose.
You new to the world, right? ClickHouse proven being a beast for analytics, tracing and even some basic metrics storage (VM won of course, but still CH is probably 2nd or 3rd)
* for other readers, VM here most likely refer to Victoria Metrics
Re: Choose DuckDB rather than SQLite
#59Earlier quoted context omitted.
AI comments aren't allowed on HN. AI submissions are.
Yes, I am asking if that particular policy could be changed. A little AI here and there is fine, to each their own, but I would prefer not to read posts that are overwhelmingly so.
It's one thing for us to detect and autokill generated comments on our own site; it's our site and we can set the rules and run software on our own servers to process the comments and handle things in the way we and the community are happy with.
It's a big additional leap to for our software to try to reach into others' sites, get through any anti-bot defenses they may be running, try to scrape their content and evaluate on whether it's sufficiently human-authored to be on HN.
There's generally a wider range of LLM involvement with a long-form post than the typical, relatively brief HN comment, which then opens the way for more debate on HN about "how much" LLM influence the post has and how much should be allowed on HN. Part of what we're trying to optimize for on HN is minimizing offtopic/meta discussion, so we don't want to encourage this kind of debate.
Our heuristic about article quality is largely unchanged from before LLMs were an issue: if an article is badly written, it shouldn't be on HN, and should be flagged.
Re: Choose DuckDB rather than SQLite
#60Earlier quoted context omitted.
Yes, I am asking if that particular policy could be changed. A little AI here and there is fine, to each their own, but I would prefer not to read posts that are overwhelmingly so.
It's one thing for us to detect and autokill generated comments on our own site; it's our site and we can set the rules and run software on our own servers to process the comments and handle things in the way we and the community are happy with. It's a big additional leap to for our software to try to reach into others' sites, get through any anti-bot defenses they may be running, try to scrape their content and eval…
There are AI-detection tools which exist. Even if those won't run against all URLs, using them where possible should provide some utility. HN already penalises sites based on various criteria, and if it takes hand-pasting some examples from a site to find that it is/isn't using AI slop, that's another option.
As to what should be tested: front-page items, possibly even a subset of those (top 10--15 of 30). That's going to be a limited set of items per day, though more than just 30. (I don't know how many items cycle through the front page on a daily basis, though I believe daily submissions as of 2022 were about 1,000/day (https://web.archive.org/web/20220116193045/https://whaly.io/...>)).
Working this into the HN story-processing lifecycle might be a good call.
I'd much prefer not seeing a bunch of AI slop in submissions, by way of generated output. AI as part of the resarch process I think I could live with.
AI-generated content seems, definitionally, not to be intellectual in nature, and would seem to go against HN's prime directive. It also seems to make HN lose its collective mind, which has long been another mod consideration.