Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We decided to go for the big rewrite

tech.channable.com

Post reply on HN
Print view View original

21–30 of 75 posts

Re: We decided to go for the big rewrite

#21

Post by mark_l_watson »

I have also found Spark (and Hadoop before that) a little clunky to prototype and develop on, but when you need to handle very large data sets with good throughout performance then systems like Spark/Hadoop are great. One problem they had was maintaining infrastructure, and to be honest, when I used mapreduce as a contractor at Google or AWS Elastic MapReduce as a consultant I didn’t have to deal too much with infrastructure.

Anyway, it makes sense that they backed off using Spark and HDFS - makes sense given the size of their datasets.

The original poster mentioned that their data analytics software is written in Haskell. I would like to see a write up on that.

EDIT: I see that they do have two articles on their blog on their use of Haskell.

Re: We decided to go for the big rewrite

#22

Post by StreamBright »

Darkstryder wrote: post #5

> Prematurely designing systems “for scale” is just another instance of premature optimization > Examples abound: (...) using a distributed database when Postgres would do This is the only part of the article that bugged me a little, because in my experience the choice between single-machine and distributed databases is not so much about scale as it is about availability and avoiding a single point of failure. Even i…

You can use a replication setup for Postgres which gives you a pretty good availability.

Also, Spark has some single point of failures that are not obvious at first.

https://gist.github.com/aseigneurin/3af6b228490a8deab519c6ae...

Re: We decided to go for the big rewrite

#23

Post by nicoburns »

royjacobs wrote: post #11

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres? I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again. Edi…

I think it is, but it's a worthwhile point to reiterate. It's also worth pointing out that it can be worth doing a rewrite to a less scalable architecture if it simplifies the architecture and provides more flexibility.

The company I work is currently undergoing a similar rewrite from firebase to Postgres (although we're doing a gradual migration), and it's amazing how much code we have been able to throw away, and how much quicker we can move in the parts of the code that have been migrated.

Re: We decided to go for the big rewrite

#24

Post by bluedino »

Our company runs on a pile of VBA/Access. At least it talks to a MariadDB server on Linux.

The biggest problems are trying to run/develop this code on machines that were made in the last ten years, the other is that it's a horrible, horrible codebase. Code practices from the early 90's.

To make things worse, objects are 'evil', all HTML/SQL/XML is built by appending strings, there's no data sanity checks.....

I started a proof of concept replacement system that was written in Python and ran on the web.

It was met with "We can go with a web based system, since if anything changes in the browser we'll be up shit creek."

:-/

Re: We decided to go for the big rewrite

#25

Post by barking »

z3t4 wrote: post #12

"A creature from another dimension would see us as noise as our atoms are constantly being replaced" Don't wait for a big rewrite. Constantly keep deleting and rewriting. Just make sure you are solving real problems while doing it.

This feels like a far safer path to take

Re: We decided to go for the big rewrite

#26

Post by nicoburns »

Darkstryder wrote: post #5

> Prematurely designing systems “for scale” is just another instance of premature optimization > Examples abound: (...) using a distributed database when Postgres would do This is the only part of the article that bugged me a little, because in my experience the choice between single-machine and distributed databases is not so much about scale as it is about availability and avoiding a single point of failure. Even i…

> Depending on the service and its users, this mandatory down-time might or might not be acceptable.

True. But if it is acceptable, then it may well be a good trade-off to make.

Re: We decided to go for the big rewrite

#27

Post by Darkstryder »

duijf wrote: post #17
Darkstryder wrote: post #5

> Prematurely designing systems “for scale” is just another instance of premature optimization > Examples abound: (...) using a distributed database when Postgres would do This is the only part of the article that bugged me a little, because in my experience the choice between single-machine and distributed databases is not so much about scale as it is about availability and avoiding a single point of failure. Even i…

Re availability: We had a hard time keeping the system based on Spark available. There were days when the cluster would freak out multiple times in a single day. The 'fix' would be: restart a bunch of spark workers. We spent a lot of time debugging/finding this out (some parts documented in [1]) but couldn't work out what the problem was. (EDIT: Assuming there even was a single problem.) In this particular case, I'd…

Yeah, I agree. It was more of a general comment, because you seem to have one Postgres instance for every client, which is already a big step against SPOF.

At $previous_job we had a "one service" = "one MySQL instance" policy. Every time a MySQL server would go down all clients would all lose access to that service at the same time. It was stressful and much less robust than your setup.

Re: We decided to go for the big rewrite

#28

Post by mannykannot »

Darkstryder wrote: post #5

> Prematurely designing systems “for scale” is just another instance of premature optimization > Examples abound: (...) using a distributed database when Postgres would do This is the only part of the article that bugged me a little, because in my experience the choice between single-machine and distributed databases is not so much about scale as it is about availability and avoiding a single point of failure. Even i…

I had the same reaction... this statement seems to be an over-generalization, but it can be resolved by being careful about what 'premature' means. In this case, "we can trivially shard datasets of different projects over different servers, because they are all independent of one another", so it seems the scaling issue had a solution from the outset.

Warnings about the risks of premature optimization should not stop people thinking about issues that are likely to arise in future, and what they might do about them. On the other hand, this does not mean that you should necessarily implement that solution now.

Re: We decided to go for the big rewrite

#30

Post by Darkstryder »

nicoburns wrote: post #26
Darkstryder wrote: post #5

> Prematurely designing systems “for scale” is just another instance of premature optimization > Examples abound: (...) using a distributed database when Postgres would do This is the only part of the article that bugged me a little, because in my experience the choice between single-machine and distributed databases is not so much about scale as it is about availability and avoiding a single point of failure. Even i…

> Depending on the service and its users, this mandatory down-time might or might not be acceptable. True. But if it is acceptable, then it may well be a good trade-off to make.

I agree completely. But it is a trade-off you need to be aware of, and need to be confident it will still be true in the future.

I've had situations where a client wanted a 99.95% availability SLA for our SaaS service instead of 99.5% as we were providing at the time.

He thought it was a minor change on our side, but it required completely rethinking our architecture from the ground up.

Post reply on HN