Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: I have to analyze 100M lines of Java – where do I start?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#111

Post by jacquesm »

fleitz wrote: post #101
sergiosgc wrote: post #26

For rewrite from scratch projects, I always start by identifying the use cases covered by the application. You don't need the code for that. Just run the application and identify what it is that it does. Then, work backwards. For each use case, use the existing code as specification of the use case behavior. At 100 million lines, I'd suspect this is either an extremely large project, where a rewrite from scratch is i…

Yup, start the profiler up, use the application, check the call graph. At 100 MLOC the code base is probably a complete mess. Also, simian is probably your friend as it will identify large chunks of duplicate code. As well source control can be your friend, the older the source is the more likely it is to contain useful code. The files with the most changes will usually be where the bugs are.

Why did this get downvoted? It's actionable and on topic.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#112

Post by PeterisP »

Divide and conquer.

Find a way to split it into something like 10 pieces of 10m LOC each in a way where you can understand and [re-]document the data and control flow between them.

Repeat with further subdivision, as much as you have people.

Then, if you really need to re-build this from scratch, do it per component - first, make automated integration tests for its functionality, and only then attempt to rebuild that part of the system.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#113

Post by johan_larson »

sergiosgc wrote: post #26

For rewrite from scratch projects, I always start by identifying the use cases covered by the application. You don't need the code for that. Just run the application and identify what it is that it does. Then, work backwards. For each use case, use the existing code as specification of the use case behavior. At 100 million lines, I'd suspect this is either an extremely large project, where a rewrite from scratch is i…

Identify the users. Figure out who actually does anything with the system. They'll be invaluable as you try to determine what part of the system are active.

Look for a test suite. You'll need one once you start making changes, to keep from breaking anything. If necessary create one based on actual jobs run of the system. You want integration tests, for big parts of the system, rather than unit tests.

Once you have a test suite, start with dead code analysis. Any codebase as big as this one will have a lot of accumulated cruft that is just getting in the way. Delete it.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#114

Post by andrewchambers »

rebuild 100M lines from scratch? sounds impossible to me.

You can re implement the applications that are causing problems maybe one at a time maybe.

I don't understand what you want to get syntax trees for, but it sounds like you are gonna need to store them in a database and do queries on it if there really is info that you need.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#115

Post by atmosx »

crpatino wrote: post #110
stronglikedan wrote: post #105

Earlier quoted context omitted.

I could be wrong, but I think the "M" means thousands in this case. I know it means thousands when dealing with inventory UOMs. 100 million lines just seems like a lot, even for a code generator, and not something that would be turned over to a single team.

I don't think there are that many companies with ~10^8 LOC code bases in the world, let alone for a single product[1]: Facebook(webapp): ~3 10^7 LOC Linux Kernel: ~2 10^7 LOC Windows XP: ~5*10^7 LOC My impression is that 10 million LOC codebase is relatively common... but past that the size of the organization (company or volunteers) needed becomes a major sorting criteria. [1] http://www.wired.com/2013/04/facebook-w…

Yes but these project are no dumped to a team to re-write them all the often, are they?

The way I get this, his team is limited in personnel and resources and kinda overwhelmed by the project. I'm sure Facebook/Linux (OS devs)/Microsoft are not exactly out of developers.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#117

Post by V-2 »

As for your first step - with a project that size (and thus of substantial age, I assume), there's surely tons and tons of dead code. I'd throw it away first. Slim the thing down. The very process of identifying unused code will already familiarize you roughly with the conceptual "shape" of the application

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#118

Post by johan_larson »

grey-area wrote: post #57
user1241320 wrote: post #10

Earlier quoted context omitted.

We'd love to know what these lines do. For example what part of this codebase deals with the DB and what part does not. And then go deeper. The final goal is to re-do what these lines do :(

100, 000, 000 lines of code is a huge amount and would take you over 1000 days just to read at 1 line per second, and 1000 man-years to fully understand. If your final goal is to rewrite all of it you are probably doomed to fail. You should first ask yourself (and your clients) some simple questions about why this insane project has been dumped on you and what the goal is: What is the order of priority of services -…

Yes, a "rewrite" is the wrong way to think about this project. A typical programmer might be able to produce 10K lines of production code in a year, which means it would take 10,000 engineer-years to rewrite it all.

The OP needs to think in terms of stewardship, not complete reconstruction, and make improvements by small steps as a gradual process.

I talked about a similar problem here: http://short-sharp.blogspot.ca/2012/08/fixing-broken-codebas...

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#119

Post by joshdance »

I don't know if you want to re-write it. Having to identify all the use cases for a system that large will be horrific. Especially since people have build onto the broken ways. And a large customer will require that it be exactly like it was before so you have to recreate the old broken way and the new shiny way. Who decided it needed to be re-written?

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#120

Post by n1ghtmare_ »

crpatino wrote: post #110
stronglikedan wrote: post #105

Earlier quoted context omitted.

I could be wrong, but I think the "M" means thousands in this case. I know it means thousands when dealing with inventory UOMs. 100 million lines just seems like a lot, even for a code generator, and not something that would be turned over to a single team.

I don't think there are that many companies with ~10^8 LOC code bases in the world, let alone for a single product[1]: Facebook(webapp): ~3 10^7 LOC Linux Kernel: ~2 10^7 LOC Windows XP: ~5*10^7 LOC My impression is that 10 million LOC codebase is relatively common... but past that the size of the organization (company or volunteers) needed becomes a major sorting criteria. [1] http://www.wired.com/2013/04/facebook-w…

I realize Facebook is not a trivial software, but seriously - this large? I heard this before and I just don't get it.
Post reply on HN