Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We decided to go for the big rewrite

tech.channable.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: We decided to go for the big rewrite

#61

Post by wvenable »

Darkstryder wrote: post #5

> Prematurely designing systems “for scale” is just another instance of premature optimization > Examples abound: (...) using a distributed database when Postgres would do This is the only part of the article that bugged me a little, because in my experience the choice between single-machine and distributed databases is not so much about scale as it is about availability and avoiding a single point of failure. Even i…

You don't need a distributed database to have replication and a hot backup. Stackoverflow runs this kind of configuration -- if it works for them, it can work for you.

Re: We decided to go for the big rewrite

#62

Post by jimbokun »

We decided to go for a Big Rewrite for a completely different reason. The initial license for the proprietary NoSQL database we had negotiated was about to expire, and the company was going to charge us an order of magnitude more to renew.

So we immediately set out redesigning our system to use other, fully open source technologies. Also gave us an opportunity to reconsider architecture decisions that had not scaled well. In our case, moving from a monolith to microservices has had major benefits. Maybe the biggest being able to quickly see which microservice is the bottleneck and needs to be scaled up to handle the load. With the monolith, if it got slow, it was very difficult to figure out which part of the workload was making it slow.

Re: We decided to go for the big rewrite

#63

Post by goto11 »

ryanelfman wrote: post #53
goto11 wrote: post #50

Is the current system is so badly architected that it cannot be refactored gradually or rewritten piecemeal? Then the forces which caused these problems will also be in effect during the rewrite, so it will end in the same place when it reaches feature parity. I can only think of a few places where a full rewrite is justified: * You lost the source code * The application is almost purely integration with some 3'rd pa…

That's not necessarily true. Those forces will still be there but the learning of what not to do has also been had. People can improve and learn over time.

Yes but if you have improved over time, then at least the most recently developed parts of the code would be high-quality and would not need rewriting. So you would only need to rewrite some encapsulated legacy parts of the codebase, which is completely different from a full rewrite.

Re: We decided to go for the big rewrite

#64

Post by mbo »

apta wrote: post #49
kod wrote: post #40

Earlier quoted context omitted.

> "Sure, Scala will work great! It's future-proof and everyone will love it!" Scala is still the least-bad option for a JVM language. Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

With Java getting value types, record types, and pattern matching it will end up giving all the other JVM languages a run for their money. Today, Kotlin is much more approachable than Scala, not to mention better tooling. Once Java catches up though, it will be a different story.

When are higher-kinded types roadmapped for Java?

Re: We decided to go for the big rewrite

#65

Post by apta »

mbo wrote: post #64
apta wrote: post #49

Earlier quoted context omitted.

With Java getting value types, record types, and pattern matching it will end up giving all the other JVM languages a run for their money. Today, Kotlin is much more approachable than Scala, not to mention better tooling. Once Java catches up though, it will be a different story.

When are higher-kinded types roadmapped for Java?

I'm not aware that they are. It's a nice feature, but it might introduce unnecessary indirection is used inappropriately, that and the vast majority of languages get by without it without any significant hindrance.

Re: We decided to go for the big rewrite

#67

Post by claytonjy »

mason55 wrote: post #39
mumblemumble wrote: post #33

Earlier quoted context omitted.

There's that, but Spark is also a little bit tricky because it has such a big feature set that it's not just attractive as a big data tool. On paper, it can be attractive as a tool for easy single-node data parallelism, for easy streaming data processing, for easy machine learning on the Java platform, stuff like that. I'm looking at migrating off of Spark, too, and finding that Spark is still the only way to get a d…

Curious what you recommend instead of Spark.

I'm not a user myself, but just from watching the chatter in the world it seems dask and rapids are quickly overtaking spark in mindshare, or at least enthusiasm, and that spark is following hadoop/mapreduce to the Big Data Graveyard.

Re: We decided to go for the big rewrite

#68

Post by julienfr112 »

kod wrote: post #40
royjacobs wrote: post #11

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres? I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again. Edi…

> "Sure, Scala will work great! It's future-proof and everyone will love it!" Scala is still the least-bad option for a JVM language. Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

Yeah the team is productive. But only in the 20% left for real work between the interminable arguing over monads or circe or play or sbt or whatever.

Re: We decided to go for the big rewrite

#69

Post by alexpotato »

steve_adams_86 wrote: post #51
alexpotato wrote: post #31

Rewriting an application can mean different things: 1. "We are going to start over from scratch and rewrite the whole thing!" Joel Spolksy famously said to "Never do this!" 2. "We are going to slowly refactor the whole codebase." This can, eventually, lead you to a place where none of the original code is there so it's like a rewrite but much simpler. 3. "We are going to slowly add new places to replace the old syste…

I'm not sure about the resume part. My experience refactoring legacy stuff into more modern stuff is actually a big talking point for my resume in my experience. People are always interested in why I did it, how I did it, why didn't I just build anew, etc. Maybe that's because most jobs I apply for tend to have at least one 'legacy' system laying around that needs some help. But this is somewhat common outside of the…

Should have been more specific as to which industry I was referring to regarding the resume comment.

I worked for a big bank for 5 years supporting large trading systems. After the 3rd attempt to retire a legacy trading system by rolling out a new system, someone pointed out that if you are a mid level manager you don't get any resume points for refactoring old systems.

Hence, you keep trying to roll out new systems even if those systems have bugs like "switch all buy orders to sells" (true story).

Re: We decided to go for the big rewrite

#70

Post by kod »

julienfr112 wrote: post #68
kod wrote: post #40

Earlier quoted context omitted.

> "Sure, Scala will work great! It's future-proof and everyone will love it!" Scala is still the least-bad option for a JVM language. Anyone who can't be productive in Scala (not "better java", not "worse haskell", Scala) isn't someone you want on your team anyway.

Yeah the team is productive. But only in the 20% left for real work between the interminable arguing over monads or circe or play or sbt or whatever.

Doing real work in Scala for a decade. Never argued about any of that.
Post reply on HN