Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Sucks About Erlang (2008)

damienkatz.net

Post reply on HN
Print view View original

91–100 of 105 posts

Re: What Sucks About Erlang (2008)

#91

Post by lostcolony »

forgottenpass wrote: post #85
lostcolony wrote: post #84

Earlier quoted context omitted.

Not publicly documented, but CNN's ingest scheduling system (basically the backend that says "at this time, record this incoming feed, on this encoder, associate it with this metadata", and all the ensuing complexity that comes with finding a free encoder, responding in the event of hardware failure, allowing users to plan recordings that there isn't space for, etc) is in Erlang. Because it needed to be bullet proof;…

That's interesting. Do you guys have to build a lot of custom tooling in TV? Are CNN's requirements particularly novel compared to other stations with live programming and remote feeds?

I've since left, but yes and no. Video at that scale requires some unique hardware, which in turn requires some unique software. But for broadcasters of that scale, many of the problems are the same.

We actually had, prior to this system, been relying on one built by a French company, that cost hundreds of thousands, possibly millions, per year in licensing. Updates were infrequent, often included regressions, but it mostly worked.

This started as a way to get some of the smaller bureaus to have some basic functionality, and grew into a system that was deployed to every bureau, eventually replacing what we had in Atlanta (that previously mentioned system). It was extremely popular, and the business stakeholders even characterized it as the biggest success they've had from our department.

One of our group's VPs showed it at an industry trade show, just offhandedly, and an exec from another broadcaster indicated regret of having just signed a deal to purchase the aforementioned system above, rather than attempting to roll their own (seeing the success CNN had).

It's not a key differentiator for CNN, but they also don't have software sales as a core competency. So they have basically best of breed software sitting there unnoticed. shrug C'est la vie.

It also has the benefit of, since it's Erlang, largely just working without issue. There's not many there still with Erlang knowledge, but that isn't really much issue for another 20 years if it continues just working without issue.

Re: What Sucks About Erlang (2008)

#92

Post by facetube »

facetube wrote: post #81
ermouth wrote: post #78

Earlier quoted context omitted.

> Nothing is streaming Except CouchDB entire REST API, where nearly every endpoint streams result row by row, which is great. > Performance (and likely security, due to age of the engine) in this subsystem is extremely poor As of security: wrong guess. Go compare number of critical CouchDB CVE for last, say, 5 years, with any other DB you want. As for query server: indexing is slow for the reasons you pointed out. Re…

> Except CouchDB entire REST API, where nearly every endpoint streams result row by row I'm talking about the internals of couchjs. > As of security: wrong guess. Go compare number of critical CouchDB CVE I'm talking about Spidermonkey 1.8.5, which is seven years old. Multiple CVEs have been reported including a full RCE, which was patched in later versions. If you believe these are patched in libmozjs185, I'd love t…

Correction: the library is too old for that specific CVE to be exploitable. There may be others, there may not be. I don't know – and that's actually the problem.

The chief complaint here is that it's very hard to reason about the security of seven year old software that distros are pulling out of their package repositories citing security issues (e.g. Alpine Linux). Advisories tend to say "versions before X", and you're left to either wonder or read tickets/code. Not everyone wants to sign on for that, and it's work that isn't going toward writing your app.

Re: What Sucks About Erlang (2008)

#93

Post by skolsuper »

newlaptopnewacc wrote: post #60
latch wrote: post #22

Earlier quoted context omitted.

My point is that you can achieve many of the same benefits of microservices within a single app (or "vm" in erlang terms). So you get isolation (beam processes), decoupling (message passing) and resilience (supervisors) without having to deploy and maintain N different apps and the pain points that come with that (monitoring, discovery, integration testing, debugging, ...) Our company has other systems, in other lang…

I have to ask someone to bring kubernetes into the comparison -redundant with Erlang?

I also want to know this. We have a team of 3 people maintaining our microservices infra (services written in NodeJS), out of ~15 developers. It feels like a waste, these (monitoring, logging, deployment etc) are all problems that have been solved many times over.

Re: What Sucks About Erlang (2008)

#94

Post by jake_morrison »

newlaptopnewacc wrote: post #60
latch wrote: post #22

Earlier quoted context omitted.

My point is that you can achieve many of the same benefits of microservices within a single app (or "vm" in erlang terms). So you get isolation (beam processes), decoupling (message passing) and resilience (supervisors) without having to deploy and maintain N different apps and the pain points that come with that (monitoring, discovery, integration testing, debugging, ...) Our company has other systems, in other lang…

I have to ask someone to bring kubernetes into the comparison -redundant with Erlang?

I would say they are complementary. Kubernetes is a standard solution for packaging and deploying apps. If you run a lot of apps, you want your ops team to be able to manage them without knowing all the details. You get better utilization of your servers by allowing them to deploy standard units and manage resources at a higher level.

You can put an Erlang/Elixir app into a container and deploy it, but containerization doesn't provide as much value as it may for other languages. The Erlang VM doesn't depend much on the operating system. The mix build tool tightly manages runtime library dependencies, and building a release combines the VM and the app's libraries into a single tarball that you can deploy as a unit. So we don't find it particularly useful to use Docker/Kubernetes to deploy.

Erlang/Elixir makes it very easy to take advantage of all the CPU cores available, so you generally benefit from having one instance with lots of CPUs instead of lot of little instances. And it can be a lot cheaper to do this in a dedicated server environment instead of the cloud.

So on the whole, Kubernetes is not that big a win for Erlang, but they work fine together.

Re: What Sucks About Erlang (2008)

#95

Post by daleharvey »

HumanDrivenDev wrote: post #88
daleharvey wrote: post #52

Earlier quoted context omitted.

Not entirely sure what you misunderstand, PouchDB stores multiple revisions because thats how CouchDB's sync protocol works, the point of PouchDB was to match CouchDB semantics. PouchDB (as CouchDB) provides options to control how much you track (revs_limit, auto_compaction) and there are improvements we could do to handle tracking less information better We use transactions under the hood actually writing data to in…

I mean I both get it, and don't get it. CouchDB is all about master-master, and pouchdb just brings that to the browser. But at the same time I don't understand why you'd want a 'master' at the edge of the network, or the overhead of multiple revisions when a users browser isn't going to have tones of concurrent connections. I fully admit I'm not very well-versed on databases or distributed computing, happy to have t…

I had to make that distinction fairly early on wether PouchDB would be an edge client or a full node and I decided full node, tradeoffs either way but pros of being a full node:

1. Additional (p2p) use cases, honestly I dont use CouchDB much these days, I use express-pouchdb-server so I can embed pouchdb into my server easier than any other database, there is also pouchdb-server and p2p projects like Thali (http://thaliproject.org/)

2. CouchDB existed, sync is super hard, writing an edge client would have meant designing it myself, writing a couchdb clone meant I needed to make a lot less decisions about how it worked while being very confident it would work, The test suite runs the same tests against pouchdb/couchdb to ensure compatibility (and replication via each configuration)

Re: What Sucks About Erlang (2008)

#96

Post by pulsarpietro »

Tomte wrote: post #53
pulsarpietro wrote: post #47

Is it right to say that Erlang and Prolog are actually that close as stated at the beginning of the article ? As far as I am aware Erlang does not run the unification algorithm and it is very different, you can, for instance, assign variables. I don't think it is right to compare the two as close languages as it is claimed but I would be interested in your opinion

It's true that Erlang is not a logic-based language like Prolog, but the first implementation of Erlang was written in Prolog.

what is it written in now ?

Re: What Sucks About Erlang (2008)

#98

Post by ermouth »

facetube wrote: post #81
ermouth wrote: post #78

Earlier quoted context omitted.

> Nothing is streaming Except CouchDB entire REST API, where nearly every endpoint streams result row by row, which is great. > Performance (and likely security, due to age of the engine) in this subsystem is extremely poor As of security: wrong guess. Go compare number of critical CouchDB CVE for last, say, 5 years, with any other DB you want. As for query server: indexing is slow for the reasons you pointed out. Re…

> Except CouchDB entire REST API, where nearly every endpoint streams result row by row I'm talking about the internals of couchjs. > As of security: wrong guess. Go compare number of critical CouchDB CVE I'm talking about Spidermonkey 1.8.5, which is seven years old. Multiple CVEs have been reported including a full RCE, which was patched in later versions. If you believe these are patched in libmozjs185, I'd love t…

> I'm talking about the internals of couchjs.

_list functions stream, and was specifically designed to be able to stream. However, streaming does not help much in this case, moreover too small chunks dramatically reduce already awful _list perf.

Taking this in account I see no value for views to stream. Sending all emitted KVs for the doc to Erlang bits in one turn seems both much more predictable and safe.

> Multiple CVEs have been reported including a full RCE, which was patched in later versions

Last CVEs has nothing to do with Spidermonkey. BTW they may be patched without upgrade, with 5LOC long design document in Erlang.

The main reason of last CVEs is inconsistency in parsing improper JSONs by Erlang parser. Namely, most JSON parsers process '{"abc":1, "abc":2}' as {abc:2}, but old jiffy parses it as {abc:1}. BTW severe inconsistencies in parsing JSON are pretty common across implementations, please read http://seriot.ch/parsing_json.php for more details.

> Citation needed. There's an issue specifically about this in the CouchDB issue tracker if you'd like to read more.

I‘ll give you no cite, sorry, because we discovered the effect during internal tests. Reason is simple: accessing values in deep branchy JSONs is generally faster in JS, because it‘s native format. JSON in Erlang is a monster like {[{>, 1}]} for {abc:1}, which, when has a lot of levels and nodes at each level, performs bad for selecting random node.

> Curious what kind of database sizes you're working with too

We sometimes measure number of docs with M postfix ) Not very often, however. In my humble opinion, if you plan to have CouchDB with, say, 100M docs in a single bucket, you probably chose wrong solution.

BTW, same for large kitchen DBs for buckets with, say, 10K docs.

Re: What Sucks About Erlang (2008)

#99

Post by facetube »

ermouth wrote: post #98
facetube wrote: post #81

Earlier quoted context omitted.

> Except CouchDB entire REST API, where nearly every endpoint streams result row by row I'm talking about the internals of couchjs. > As of security: wrong guess. Go compare number of critical CouchDB CVE I'm talking about Spidermonkey 1.8.5, which is seven years old. Multiple CVEs have been reported including a full RCE, which was patched in later versions. If you believe these are patched in libmozjs185, I'd love t…

> I'm talking about the internals of couchjs. _list functions stream, and was specifically designed to be able to stream. However, streaming does not help much in this case, moreover too small chunks dramatically reduce already awful _list perf. Taking this in account I see no value for views to stream. Sending all emitted KVs for the doc to Erlang bits in one turn seems both much more predictable and safe. > Multipl…

> I‘ll give you no cite, sorry, because we discovered the effect during internal tests

We discovered the issue in internal tests and reported it upstream where it was confirmed; there's nothing to discuss here.

> BTW they may be patched without upgrade, with 5LOC long design document in Erlang.

I don't have to hand-patch other database systems. Further, as of now there are no functioning packages for multiple versions of Ubuntu. Multiple competitors do not have any of these problems.

> Last CVEs has nothing to do with Spidermonkey

It was a full RCE that didn't have vendor packages ready in time.

Sorry, but using seven year old language runtimes is daft. It might be fine for you, but it's not appropriate in environments where you care about security and performance, or care about the overhead of making your team reason about these things unnecessarily.

> We sometimes measure number of docs with M postfix

Yeah, we did this in 2003 on commodity hardware, and even it built indexes faster than CouchDB builds map/reduce indexes. Fix the external view server protocol or be honest with people and kill it off – the status quo is unacceptable.

Finally, here's a five year old issue that's still open admitting what I just explained: https://issues.apache.org/jira/browse/COUCHDB-1743

Re: What Sucks About Erlang (2008)

#100

Post by facetube »

exikyut wrote: post #68
facetube wrote: post #29

Earlier quoted context omitted.

God, where to start. Map/reduce views ("secondary indexes") are currently built by serializing JSON down a pipe to an external process called couchjs that links against a seven year old version of Mozilla Spidermonkey (1.8.5, released on March 31st 2011, and the only version it currently works with). This external process then pipes the map results back, again as JSON. Forget zerocopy; we're serializing and deseriali…

> You can write map/reduce views in Erlang itself as an alternative, but that engine is disabled by default due to security concerns – it's "not sandboxed". I get the impression the author(s) of this DB were still at the stage of needing to brow-beat the infrastructure into submission, or somesuch :S

One of the stated goals on their Spidermonkey replacement issue (N.B. it still won't fix the view server protocol performance, but at least it'll be a modern engine) is "Get my C skills back out of the closet", and will require https://github.com/jquery/esprima.

It's depressing.

Post reply on HN