Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Sucks About Erlang (2008)

damienkatz.net

Post reply on HN
Print view View original

81–90 of 105 posts

Re: What Sucks About Erlang (2008)

#81

Post by facetube »

ermouth wrote: post #78
facetube wrote: post #29

Earlier quoted context omitted.

God, where to start. Map/reduce views ("secondary indexes") are currently built by serializing JSON down a pipe to an external process called couchjs that links against a seven year old version of Mozilla Spidermonkey (1.8.5, released on March 31st 2011, and the only version it currently works with). This external process then pipes the map results back, again as JSON. Forget zerocopy; we're serializing and deseriali…

> Nothing is streaming Except CouchDB entire REST API, where nearly every endpoint streams result row by row, which is great. > Performance (and likely security, due to age of the engine) in this subsystem is extremely poor As of security: wrong guess. Go compare number of critical CouchDB CVE for last, say, 5 years, with any other DB you want. As for query server: indexing is slow for the reasons you pointed out. Re…

> Except CouchDB entire REST API, where nearly every endpoint streams result row by row

I'm talking about the internals of couchjs.

> As of security: wrong guess. Go compare number of critical CouchDB CVE

I'm talking about Spidermonkey 1.8.5, which is seven years old. Multiple CVEs have been reported including a full RCE, which was patched in later versions. If you believe these are patched in libmozjs185, I'd love to see a link to the source package showing where the fix was applied. Because I can't find it.

Further: https://www.cvedetails.com/vulnerability-list/vendor_id-45/p.... Another full RCE in CouchDB.

> Performance in this area depends much more on how branchy is the json doc being processed

Citation needed. There's an issue specifically about this in the CouchDB issue tracker if you'd like to read more.

> As for query server: indexing is slow for the reasons you pointed out.

Right. It is not performant. That's my point.

Curious what kind of database sizes you're working with too; you won't see this stuff with a few hundred documents.

Re: What Sucks About Erlang (2008)

#82

Post by facetube »

damienkatz wrote: post #77
facetube wrote: post #30

Earlier quoted context omitted.

It doesn't use real node-local/shard-local MVCC AFAICT. It's O_APPEND on flat files.

It _is_ append only. But doesn’t use flat files. The main storage files have 2 different immutable btrees to find documents (by id and by sequence num). And it does have mvcc for both storage and secondary indexes. Internally it has transactions but they aren’t exposed to the user, as multi document transactions don’t make sense for the replicated document model.

Thanks for the clarification. Can I ask how the MVCC is implemented (e.g. WAL, MVTO, MVRC, etc.), or if there's documentation around that might give some additional insight?

Re: What Sucks About Erlang (2008)

#83

Post by knrz »

__sr__ wrote: post #72
latch wrote: post #17

Earlier quoted context omitted.

We've been using Elixir almost exclusively for a couple years, and I can't speak highly enough about it. There are clearly problems where beam is ill-suited, but otherwise, I almost feel religious about it. I've worked for many years with many other languages and runtimes, but beam, OTP and Elixir (and the synergy between them) are better tools. As a foundation, message passing and isolation are the only realistic we…

Is Elixir being used extensively outside Silicon Valley? Can you list some well known success stories? Looking at job listings and open source projects, it doesn’t seem to be widely used. In fact, I don’t know anyone outside Silican Valley who has built anything non-trivial in Elixir. I am not saying it is bad. But I’d like to know about it before I dive in.

I've built major crawling infrastructure at [#paid](http://hashtagpaid.com) with Elixir.

My anecdatum. :)

Re: What Sucks About Erlang (2008)

#84

Post by lostcolony »

__sr__ wrote: post #72
latch wrote: post #17

Earlier quoted context omitted.

We've been using Elixir almost exclusively for a couple years, and I can't speak highly enough about it. There are clearly problems where beam is ill-suited, but otherwise, I almost feel religious about it. I've worked for many years with many other languages and runtimes, but beam, OTP and Elixir (and the synergy between them) are better tools. As a foundation, message passing and isolation are the only realistic we…

Is Elixir being used extensively outside Silicon Valley? Can you list some well known success stories? Looking at job listings and open source projects, it doesn’t seem to be widely used. In fact, I don’t know anyone outside Silican Valley who has built anything non-trivial in Elixir. I am not saying it is bad. But I’d like to know about it before I dive in.

Not publicly documented, but CNN's ingest scheduling system (basically the backend that says "at this time, record this incoming feed, on this encoder, associate it with this metadata", and all the ensuing complexity that comes with finding a free encoder, responding in the event of hardware failure, allowing users to plan recordings that there isn't space for, etc) is in Erlang. Because it needed to be bullet proof; if it goes down, CNN doesn't record incoming video.

Had Elixir been mature at the time, we would have used it instead.

Re: What Sucks About Erlang (2008)

#85

Post by forgottenpass »

lostcolony wrote: post #84
__sr__ wrote: post #72

Earlier quoted context omitted.

Is Elixir being used extensively outside Silicon Valley? Can you list some well known success stories? Looking at job listings and open source projects, it doesn’t seem to be widely used. In fact, I don’t know anyone outside Silican Valley who has built anything non-trivial in Elixir. I am not saying it is bad. But I’d like to know about it before I dive in.

Not publicly documented, but CNN's ingest scheduling system (basically the backend that says "at this time, record this incoming feed, on this encoder, associate it with this metadata", and all the ensuing complexity that comes with finding a free encoder, responding in the event of hardware failure, allowing users to plan recordings that there isn't space for, etc) is in Erlang. Because it needed to be bullet proof;…

That's interesting. Do you guys have to build a lot of custom tooling in TV? Are CNN's requirements particularly novel compared to other stations with live programming and remote feeds?

Re: What Sucks About Erlang (2008)

#86

Post by damienkatz »

facetube wrote: post #82
damienkatz wrote: post #77

Earlier quoted context omitted.

It _is_ append only. But doesn’t use flat files. The main storage files have 2 different immutable btrees to find documents (by id and by sequence num). And it does have mvcc for both storage and secondary indexes. Internally it has transactions but they aren’t exposed to the user, as multi document transactions don’t make sense for the replicated document model.

Thanks for the clarification. Can I ask how the MVCC is implemented (e.g. WAL, MVTO, MVRC, etc.), or if there's documentation around that might give some additional insight?

Here’s a link that describes the Couchstore file format. It’s the storage engine for Couchbase and is essentially the same design as CouchDB. https://github.com/couchbaselabs/couchstore/wiki/Format

Re: What Sucks About Erlang (2008)

#87

Post by facetube »

damienkatz wrote: post #86
facetube wrote: post #82

Earlier quoted context omitted.

Thanks for the clarification. Can I ask how the MVCC is implemented (e.g. WAL, MVTO, MVRC, etc.), or if there's documentation around that might give some additional insight?

Here’s a link that describes the Couchstore file format. It’s the storage engine for Couchbase and is essentially the same design as CouchDB. https://github.com/couchbaselabs/couchstore/wiki/Format

Thanks a lot; this is super-helpful.

Re: What Sucks About Erlang (2008)

#88

Post by HumanDrivenDev »

daleharvey wrote: post #52
HumanDrivenDev wrote: post #24

Earlier quoted context omitted.

I don't quite get pouchdb. Do you need to store multiple revisions of the same document in a browser database? Locks/transactions seem like a better approach here for writing to data client side.

Not entirely sure what you misunderstand, PouchDB stores multiple revisions because thats how CouchDB's sync protocol works, the point of PouchDB was to match CouchDB semantics. PouchDB (as CouchDB) provides options to control how much you track (revs_limit, auto_compaction) and there are improvements we could do to handle tracking less information better We use transactions under the hood actually writing data to in…

I mean I both get it, and don't get it. CouchDB is all about master-master, and pouchdb just brings that to the browser. But at the same time I don't understand why you'd want a 'master' at the edge of the network, or the overhead of multiple revisions when a users browser isn't going to have tones of concurrent connections.

I fully admit I'm not very well-versed on databases or distributed computing, happy to have things clarified.

Re: What Sucks About Erlang (2008)

#89

Post by HumanDrivenDev »

facetube wrote: post #46
HumanDrivenDev wrote: post #41

Earlier quoted context omitted.

I don't know that much about databases, could you go into more detail?

From the point of view of a single cluster or node (ignoring replication for a moment), it just doesn't have any strong notion of transactions. There's no write-ahead log, no rollback, and no situation where you'd be able to operate in a mode equivalent to something like `SERIALIZABLE` on a relational database.

oh right, I understand what you mean now.

I guess most people should know that going in. Are their good master-master systems out there with local transactions?

Re: What Sucks About Erlang (2008)

#90

Post by facetube »

facetube wrote: post #45
facetube wrote: post #29

Earlier quoted context omitted.

God, where to start. Map/reduce views ("secondary indexes") are currently built by serializing JSON down a pipe to an external process called couchjs that links against a seven year old version of Mozilla Spidermonkey (1.8.5, released on March 31st 2011, and the only version it currently works with). This external process then pipes the map results back, again as JSON. Forget zerocopy; we're serializing and deseriali…

For balance: there is a newer query/index system called Mango in Apache CouchDB 2.0+, that IIRC is internal and doesn't rely on any external view server. It wasn't in 1.7.1, though, so if you're coming from there, it's very much a "switch query APIs to get tolerable performance" situation.

Also for balance: it's unclear if there are CVEs. There may not be any that are exploitable due to the age of the library (i.e. the bugs landed post-release). However, it's also very tricky to rule these out one by one, and the dependency is slated for removal from some distros.

There may be no material security issue. But the age of the software does not instill confidence, and newer engines are likely far more performant.

Post reply on HN