Ask HN: I have to analyze 100M lines of Java – where do I start? • Hacker News Board

Ask HN: I have to analyze 100M lines of Java – where do I start?

#1

Post by user1241320 » Tue, Sep 02, 2014, 1:20 PM UTC

As part of a huge "let's see what's going on here and re-build this from scratch" they dumped the whole code repository on me and my team.

We've started parsing it and tried to work on extracting abstract syntax trees and all that.

Any idea would help us a great deal.

Thanks.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#2

Post by mml » Tue, Sep 02, 2014, 1:30 PM UTC

Funny, I have 50,000 individual Java apps to analyze. I started with a copy/paste detector. Pmd has a free one. Good luck!

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#3

Post by bilalhusain » Tue, Sep 02, 2014, 1:30 PM UTC

Please elaborate. Why is there need for extracting ASTs? Is it a single 100M line of Java source file? AFAIK Java has limitations on method size. If code is already organized into file and methods try to come up with some sort of UML representation. (I am assuming you are trying to understand the code base, not profiling or doing code analysis.)

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#4

Post by stuaxo » Tue, Sep 02, 2014, 1:31 PM UTC

Start with a static analysis tool - it will find lots of small possible bugs, by fixing them you will get good coverage of the code and insight into the structure + it will be better at the end.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#5

Post by jacquesm » Tue, Sep 02, 2014, 1:33 PM UTC

Callgraph. Then document the larger chunks, working your way down.

It's like having a map versus having no map at all.

And 100M lines? Are you sure there is no code generator at work here?

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#6

Post by bjackman » Tue, Sep 02, 2014, 1:33 PM UTC

You haven't really described your goals: What do you want to extract from your analysis? Metrics to tell you what's "wrong" with the existing code base? Some sort of model of the system's semantics?

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#7

Post by darrelld » Tue, Sep 02, 2014, 1:34 PM UTC

Start at the main function. See how things get setup and walk through the code from there. Keep notes on the structure and flow of things (if any). There isn't really an easy way to do this unless it had been documented properly before.

AstroGrep is a good Windows based tool that allows you to search within file so you could use it to find which files spit out a particular output to screen.

Not sure what you mean by using ASTs though.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#8

Post by jacquesm » Tue, Sep 02, 2014, 1:34 PM UTC

bjackman wrote: post #6

You haven't really described your goals: What do you want to extract from your analysis? Metrics to tell you what's "wrong" with the existing code base? Some sort of model of the system's semantics?

At a guess, understanding and ability to fix/change, that's the usual with projects like these, bringing the code back into a state of maintenance which could be described as 'under control'.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#9

Post by dmead » Tue, Sep 02, 2014, 1:34 PM UTC

why would you need the ast?

if given a large chunk of code to maintain i'll usually run doyxgen on it to generate the xmlish kind of chart that it makes. at least it gives me a roadmap to start, but it's not super great.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#10

Post by user1241320 » Tue, Sep 02, 2014, 1:35 PM UTC

bjackman wrote: post #6

You haven't really described your goals: What do you want to extract from your analysis? Metrics to tell you what's "wrong" with the existing code base? Some sort of model of the system's semantics?

We'd love to know what these lines do. For example what part of this codebase deals with the DB and what part does not. And then go deeper.

The final goal is to re-do what these lines do :(