Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We decided to go for the big rewrite

tech.channable.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: We decided to go for the big rewrite

#41

Post by mumblemumble »

mason55 wrote: post #39
mumblemumble wrote: post #33

Earlier quoted context omitted.

There's that, but Spark is also a little bit tricky because it has such a big feature set that it's not just attractive as a big data tool. On paper, it can be attractive as a tool for easy single-node data parallelism, for easy streaming data processing, for easy machine learning on the Java platform, stuff like that. I'm looking at migrating off of Spark, too, and finding that Spark is still the only way to get a d…

Curious what you recommend instead of Spark.

That's a question that has no single answer. Spark is a giant omnibus project with all sorts of bells and whistles, and the viable alternatives are going to vary wildly, depending on which subset of them you actually need/use.

If you want a single, simple answer that's easy to sell to management, I'd refer you back to the last paragraph of my original post.

Re: We decided to go for the big rewrite

#42

Post by s_Hogg »

kod wrote: post #40
royjacobs wrote: post #11

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres? I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again. Edi…

> "Sure, Scala will work great! It's future-proof and everyone will love it!" Scala is still the least-bad option for a JVM language. Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

> Scala is still the least-bad option for a JVM language.

Source, please?

> Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

Why is that?

Re: We decided to go for the big rewrite

#43

Post by royjacobs »

kod wrote: post #40
royjacobs wrote: post #11

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres? I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again. Edi…

> "Sure, Scala will work great! It's future-proof and everyone will love it!" Scala is still the least-bad option for a JVM language. Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

I would argue that in a sufficiently large organization you're better off using plain Java or perhaps Kotlin. The latter hits the sweet spot between 'expressive' and 'unreadable' whereas Scala can miss the mark.

I'm sure if you are very disciplined when writing Scala then this will not happen, etc etc. But the fact that people need to decide upfront which parts of Scala to use and which ones to avoid seem like red flags to me (and in fact were red flags, in my experience).

Re: We decided to go for the big rewrite

#44

Post by smilliken »

Darkstryder wrote: post #5

> Prematurely designing systems “for scale” is just another instance of premature optimization > Examples abound: (...) using a distributed database when Postgres would do This is the only part of the article that bugged me a little, because in my experience the choice between single-machine and distributed databases is not so much about scale as it is about availability and avoiding a single point of failure. Even i…

Downtime is not mandatory while upgrading to a new PostgreSQL version. I've upgraded from 9.0 to 11 and most versions in between without downtime on large busy databases.

There's not a single approach that works for every case, but they all involve a replica and upgrading one at a time.

Re: We decided to go for the big rewrite

#45

Post by lifeisstillgood »

>>> One of our main reasons for choosing Apache Spark had been its ability to handle very large datasets (larger than what you can fit into memory on a single node) and its ability to distribute computations over a whole cluster of machines ... We cannot fit all of our datasets in memory on one node, but that is also not necessary, since we can trivially shard datasets of different projects over different servers, because they are all independent of one another.

So this seems to be the massive takeaway - if you need to operate on a whole dataset that is larger than one node's memory capacity then you have to go distributed. Else it still seems an overhead barely worth the effort.

So Google: dataset is all web pages on the internet - yes that's too large go distributed.

Tesco / Walmart : dataset might be all the sales for a year. Probably too large. But could you do with sales per week? per day?

having the raw data of all your transactions etc lying around waiting for your spiffo business query sounds good but ... is it?

I would be interested in hearing folks' cut-off points for going full Big Data vs "we don't really need this"

Re: We decided to go for the big rewrite

#46

Post by [deleted] »

mason55 wrote: post #39
mumblemumble wrote: post #33

Earlier quoted context omitted.

There's that, but Spark is also a little bit tricky because it has such a big feature set that it's not just attractive as a big data tool. On paper, it can be attractive as a tool for easy single-node data parallelism, for easy streaming data processing, for easy machine learning on the Java platform, stuff like that. I'm looking at migrating off of Spark, too, and finding that Spark is still the only way to get a d…

Curious what you recommend instead of Spark.

[deleted]

Re: We decided to go for the big rewrite

#47

Post by macspoofing »

royjacobs wrote: post #11

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres? I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again. Edi…

>Not trying to come off as too snarky

I had a similar thought. Postgres is a great choice, but then they also went with Haskell ... I look forward to another blog post in 2-3 years detailed all the ways that Haskell failed them and that at the day they should have just gone with an industry standard language.

Re: We decided to go for the big rewrite

#49

Post by apta »

kod wrote: post #40
royjacobs wrote: post #11

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres? I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again. Edi…

> "Sure, Scala will work great! It's future-proof and everyone will love it!" Scala is still the least-bad option for a JVM language. Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

With Java getting value types, record types, and pattern matching it will end up giving all the other JVM languages a run for their money. Today, Kotlin is much more approachable than Scala, not to mention better tooling. Once Java catches up though, it will be a different story.

Re: We decided to go for the big rewrite

#50

Post by goto11 »

Is the current system is so badly architected that it cannot be refactored gradually or rewritten piecemeal? Then the forces which caused these problems will also be in effect during the rewrite, so it will end in the same place when it reaches feature parity.

I can only think of a few places where a full rewrite is justified:

* You lost the source code

* The application is almost purely integration with some 3'rd party platform or component, and you need to replace that platform. (E.g. you are developing a registry cleaner and need to port it to Mac)

* You don't have any customers or users yet and time-to-market is not a concern.

* You are not a business and are writing the code purely for your own enjoyment

But these are business level considerations. For individual developers there may be compelling reasons to push for a rewrite:

* You find it more fun to work on green-field projects than to perform maintenance development.

* The new platform is more exciting or looks better on the CV than the old

Post reply on HN