When data bricks contain neon, do they glow in the dark?
Databricks acquires Neon
211–220 of 234 posts
Re: Databricks acquires Neon
#212I really like databricks at the beginning, Spark clusters magically appear whenever I needed them. No more thoughts were given on how to store the data, etc.. After a few years, I've grown, so did my skill set and my employer. Databricks simply couldn't grow with us at the rate that we needed. No, not because of scaling, but the toolings such as AI/ML, what the market is demanding us to do. Faster iterations and experiments, less cost.
Databricks now reminds me of Oracle. Still a great product, but it's a melting delicious ice cream.
Re: Databricks acquires Neon
#213Not too familiar with Neon other than the basics - its premise is that you use S3 as bottomless storage for Postgres and it’s otherwise the same as standard Postgres right? And this is all open source? Why are people paying? Can’t you use a cloud provider and have them host this for you?
> you use S3 as bottomless storage for Postgres [...] Why are people paying? It's vastly more complicated to do this efficiently than you might imagine. Postgres' internal architecture is built around a very different set of assumptions (pages, WAL, local disk etc.) than what the S3 API offers.
Would "essential internal abstraction, with the introduction of read stream APIs" evolve into full storage abstraction?
Re: Databricks acquires Neon
#214I’ve loved Neon and now I’m a little worried. Are there any alternatives?
[Disclaimer: I work for Xata] As it happens, we've just launched our new Xata platform ( https://xata.io/ ) which has some of the key Neon features: instant copy-on-write branching and separation of storage and compute. As an extra twist, we also can do anonymization (PII masking) between your production database and developer branches. The way we do copy-on-write branches is a bit different. We haven't done any modi…
Hi @tudorg - do the Xata copy-on-write branches work like Neon in that you effectively get an isolated Postgres cluster, allowing you to test roles, extensions, creating/dropping DBs, altering schema & data of existing DBs, etc? I looked in the docs but it wasn’t clear to me.
Re: Databricks acquires Neon
#215Earlier quoted context omitted.
Flink. It has more momentum than Spark right now.
"momentum" is a tricky word. Zig has more momentum than C++, but will it ever overtake the language? I'd bet not.
Well its not a tricky word it just wrong. Velocity maybe. Or more probably acceleration.
Re: Databricks acquires Neon
#216Earlier quoted context omitted.
[Disclaimer: I work for Xata] As it happens, we've just launched our new Xata platform ( https://xata.io/ ) which has some of the key Neon features: instant copy-on-write branching and separation of storage and compute. As an extra twist, we also can do anonymization (PII masking) between your production database and developer branches. The way we do copy-on-write branches is a bit different. We haven't done any modi…
Hi @tudorg - do the Xata copy-on-write branches work like Neon in that you effectively get an isolated Postgres cluster, allowing you to test roles, extensions, creating/dropping DBs, altering schema & data of existing DBs, etc? I looked in the docs but it wasn’t clear to me.
Yes, exactly, you get a new isolated Postgres database, just started from essentially a snapshot at the time of the fork. You can test any types of changes.
Re: Databricks acquires Neon
#217Earlier quoted context omitted.
Hi @tudorg - do the Xata copy-on-write branches work like Neon in that you effectively get an isolated Postgres cluster, allowing you to test roles, extensions, creating/dropping DBs, altering schema & data of existing DBs, etc? I looked in the docs but it wasn’t clear to me.
Yes, exactly, you get a new isolated Postgres database, just started from essentially a snapshot at the time of the fork. You can test any types of changes.
Great, thanks for confirming
Re: Databricks acquires Neon
#218Someone deeper into the Databricks world correct me if this sounds like nonsense, but after thinking about this more my theory is that this is a defensive move for Databricks.
Smaller companies don't usually need Databricks until they grow and become larger with more complex needs, and enough data that queries start to take a long time to run.
As bare metal gets so much faster the point where you hit scaling issues with a DB becomes further and further away.
As this trend continues more companies might decide they don't need Databricks they just need a few Postgres replicas and nothing else.
Neon is kind of like an insurance policy against this outcome.
Re: Databricks acquires Neon
#219Earlier quoted context omitted.
> My last gig wouldn’t allow open source software or any company that might not be there in a decade I bet they had VMware all over the place.
Yup. And OpenShift. And Red Hat for Linux. And SAP. And IBM. But you know what? Tiny group of people relative to the impact, revenue and competitors. If we needed skills we clicked “buy”, 100 consultants would arrive who are experts, sort it and we’d move on. Not scratching around looking for people who know what we use and needing to learn 50 different open source tools. Coming from a much looser universe I learned…
I've worked at a Swedish streaming company, we bought a filesystem from IBM called "GPFS" or something before it was renamed. Well we had shit performance, reliability and everything else that could be bad. It was running on FC switches recommended by IBM, config from IBM, everything from IBM and it was an absolute fucking joke, metadata timed out regularly and the solution was "deal with it". I think my superiors should've been harder claiming we haven't received what we bought but we're Swedish so we'd rather just fall over and die.
Re: Databricks acquires Neon
#220I’ve loved Neon and now I’m a little worried. Are there any alternatives?