We decided to go for the big rewrite • Hacker News Board

Re: We decided to go for the big rewrite

#11

Post by royjacobs » Fri, Oct 04, 2019, 11:21 AM UTC

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres?

I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again.

Edit: Not trying to come off as too snarky, though I've found that this type of thing is pretty common in startups where everyone from the CTO on down has some experience but not a lot of experience. I've fallen into that trap too, at some point saying "Sure, Scala will work great! It's future-proof and everyone will love it!" cue crickets

[0] https://tech.channable.com/posts/2017-02-24-how-we-secretly-...

Re: We decided to go for the big rewrite

#12

Post by z3t4 » Fri, Oct 04, 2019, 11:25 AM UTC

"A creature from another dimension would see us as noise as our atoms are constantly being replaced"

Don't wait for a big rewrite. Constantly keep deleting and rewriting. Just make sure you are solving real problems while doing it.

Re: We decided to go for the big rewrite

#13

Post by baud147258 » Fri, Oct 04, 2019, 11:26 AM UTC

gtsteve wrote: post #2

This doesn't really sound like a big-bang rewrite as such but an incremental development process. The sort of rewrite that would be a major strategic mistake is where you start with a new repository and begin re-implementing the entire product, but this seems all perfectly sensible to me. This just sounds like the sort of incremental Ship of Theseus [0] development that many of us are doing. The product I'm working o…

The company where I work is doing (and is close to finishing) a big rewrite of one the product. The base version is deployed on-prem at our clients, the new one will be cloud service, with the same features. But the on-prem version is still worked on and supported (I'm working on it) alongside the cloud one, the cloud one is to offer a less cumbersome alternative.

Re: We decided to go for the big rewrite

#14

Post by duijf » Fri, Oct 04, 2019, 11:31 AM UTC

royjacobs wrote: post #11

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres? I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again. Edi…

> Reading this article it seems like yet another example of "you don't have big data".

Yes definitely. This thing was a big exercise in "You aren't going to need it".

> so in the end it's mostly...just an app talking to Postgres?

Yes. We solve problems for our customers. It's nice if we can do that without also creating problems for ourselves :)

> seems like they've been jumping on bandwagons before

Just wanted to point out that this system is also written in Haskell. We didn't really switch bandwagons.

Re: We decided to go for the big rewrite

#15

Post by barking » Fri, Oct 04, 2019, 11:39 AM UTC

baud147258 wrote: post #13

gtsteve wrote: post #2

This doesn't really sound like a big-bang rewrite as such but an incremental development process. The sort of rewrite that would be a major strategic mistake is where you start with a new repository and begin re-implementing the entire product, but this seems all perfectly sensible to me. This just sounds like the sort of incremental Ship of Theseus [0] development that many of us are doing. The product I'm working o…

The company where I work is doing (and is close to finishing) a big rewrite of one the product. The base version is deployed on-prem at our clients, the new one will be cloud service, with the same features. But the on-prem version is still worked on and supported (I'm working on it) alongside the cloud one, the cloud one is to offer a less cumbersome alternative.

Is the on-prem one Windows based? Also by less cumbersome do you mean the cloud one has fewer features or is just easier to get up and running with it because it's a browser based. Are the company expecting your on-prem one to fade away over time etc? Could you give any information of what each was written in and an order of magnitude, hundres of thousands of loc etc. Sorry for all the questions but I'm nosey! No problem if you can't.

Re: We decided to go for the big rewrite

#16

Post by royjacobs » Fri, Oct 04, 2019, 11:45 AM UTC

duijf wrote: post #14

royjacobs wrote: post #11

Reading this article it seems like yet another example of "you don't have big data". Most of the features that are unique to Spark (or Spark-like setups) were not needed, so in the end it's mostly...just an app talking to Postgres? I'm not sure, but reading other articles[0] on the blog seems like they've been jumping on bandwagons before, so it's probably good to come back on those decisions every now and again. Edi…

> Reading this article it seems like yet another example of "you don't have big data". Yes definitely. This thing was a big exercise in "You aren't going to need it". > so in the end it's mostly...just an app talking to Postgres? Yes. We solve problems for our customers. It's nice if we can do that without also creating problems for ourselves :) > seems like they've been jumping on bandwagons before Just wanted to po…

>Just wanted to point out that this system is also written in Haskell. We didn't really switch bandwagons.

Fair enough! In the end it's great that you've been able to take a step back and look at your ultimate goal (i.e. the customer problem solving). That doesn't always happen enough, and it's a good sign for your IT teams :) Also the fact that you were able to get the time to actually do this.

Re: We decided to go for the big rewrite

#17

Post by duijf » Fri, Oct 04, 2019, 11:46 AM UTC

Darkstryder wrote: post #5

> Prematurely designing systems “for scale” is just another instance of premature optimization > Examples abound: (...) using a distributed database when Postgres would do This is the only part of the article that bugged me a little, because in my experience the choice between single-machine and distributed databases is not so much about scale as it is about availability and avoiding a single point of failure. Even i…

Re availability: We had a hard time keeping the system based on Spark available. There were days when the cluster would freak out multiple times in a single day. The 'fix' would be: restart a bunch of spark workers. We spent a lot of time debugging/finding this out (some parts documented in [1]) but couldn't work out what the problem was. (EDIT: Assuming there even was a single problem.)

In this particular case, I'd take the single point of failure over the previous situation.

That being said: we have successfully used PostgreSQL's fail-overs multiple times. In my experience, they work quite alright.

[1]: https://tech.channable.com/posts/2018-04-10-debugging-a-long...

Re: We decided to go for the big rewrite

#18

Post by baud147258 » Fri, Oct 04, 2019, 11:49 AM UTC

barking wrote: post #15

baud147258 wrote: post #13

Earlier quoted context omitted.

The company where I work is doing (and is close to finishing) a big rewrite of one the product. The base version is deployed on-prem at our clients, the new one will be cloud service, with the same features. But the on-prem version is still worked on and supported (I'm working on it) alongside the cloud one, the cloud one is to offer a less cumbersome alternative.

Is the on-prem one Windows based? Also by less cumbersome do you mean the cloud one has fewer features or is just easier to get up and running with it because it's a browser based. Are the company expecting your on-prem one to fade away over time etc? Could you give any information of what each was written in and an order of magnitude, hundres of thousands of loc etc. Sorry for all the questions but I'm nosey! No pro…

The on-prem is windows-based, we're delivering new version as msi. The goal of the cloud version is be easier to get up and running, since there's no installation steps and (I guess) less configuration to do. The company might want the on-prem to fade, but it won't happen in the short term since we still have clients. The on-prem is 15-20 years old, but not always actively worked on and with ~1M Loc (including sql scripts, setup, config files, but not the test suite). The cloud project started last year and I think there's already clients using it, but I don't have access to the code, so I have no idea for the loc.

Re: We decided to go for the big rewrite

#19

Post by roland35 » Fri, Oct 04, 2019, 11:53 AM UTC

I am currently evaluating if we should rewrite a large chunk of our embedded controller which handles motion control right now, so this is a timely write-up! I think the lessons here are the same in embedded code - we can keep the existing black box (consultant written!) code for now while the new motion control code is written in a parallel branch. The more modular the project the easier this is luckily!

Re: We decided to go for the big rewrite

#20

Post by barking » Fri, Oct 04, 2019, 12:03 PM UTC

baud147258 wrote: post #18

barking wrote: post #15

Earlier quoted context omitted.

Is the on-prem one Windows based? Also by less cumbersome do you mean the cloud one has fewer features or is just easier to get up and running with it because it's a browser based. Are the company expecting your on-prem one to fade away over time etc? Could you give any information of what each was written in and an order of magnitude, hundres of thousands of loc etc. Sorry for all the questions but I'm nosey! No pro…

The on-prem is windows-based, we're delivering new version as msi. The goal of the cloud version is be easier to get up and running, since there's no installation steps and (I guess) less configuration to do. The company might want the on-prem to fade, but it won't happen in the short term since we still have clients. The on-prem is 15-20 years old, but not always actively worked on and with ~1M Loc (including sql sc…

Thanks for the info. I am involved with a Windows based product and we've thought now and again about doing something like this. We have a competitor that is browser and cloud based but the product seems to be a lot less featured. It feels like they are quite limited in what locally connected hardware (cameras, modems, card readers, barcode readers, printers and scanners) they can interact as well as seamless bridges to other locally installed software. With desktop software this is fairly straightforward but I don't know if this is/was a problem for cloud software.