Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: I have to analyze 100M lines of Java – where do I start?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 126 posts

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#11

Post by jacquesm »

darrelld wrote: post #7

Start at the main function. See how things get setup and walk through the code from there. Keep notes on the structure and flow of things (if any). There isn't really an easy way to do this unless it had been documented properly before. AstroGrep is a good Windows based tool that allows you to search within file so you could use it to find which files spit out a particular output to screen. Not sure what you mean by…

AST = abstract syntax tree.

With 100M lines your process will take many years.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#12

Post by MaxBarraclough »

bilalhusain wrote: post #3

Please elaborate. Why is there need for extracting ASTs? Is it a single 100M line of Java source file? AFAIK Java has limitations on method size. If code is already organized into file and methods try to come up with some sort of UML representation. (I am assuming you are trying to understand the code base, not profiling or doing code analysis.)

Seconded.

> I have to analyse [...]

> We've started parsing it and tried to work on extracting abstract syntax trees and all that.

Why? How will this help? What are you really after?

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#13

Post by user1241320 »

jacquesm wrote: post #5

Callgraph. Then document the larger chunks, working your way down. It's like having a map versus having no map at all. And 100M lines? Are you sure there is no code generator at work here?

It's code that's been developed and it's been running for decades, I'm afraid.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#15

Post by ufmace »

A few ideas that have worked for me in the past:

Map the control flow. This code/app/whatever is doing something in production right now. What tells it to start? How does the control flow from the start point to the stuff that takes in data to the stuff that writes the output or does whatever this app does? Whatever the options for how it works are, where are they set, how do they make it into the core of the application to affect whatever it does?

Map the data flow. Input must be coming into this thing somewhere. Find where it reads it in, where it writes it out, and how it gets from one to the other, what data structures and methods it passes through on the way.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#16

Post by jacquesm »

user1241320 wrote: post #10
bjackman wrote: post #6

You haven't really described your goals: What do you want to extract from your analysis? Metrics to tell you what's "wrong" with the existing code base? Some sort of model of the system's semantics?

We'd love to know what these lines do. For example what part of this codebase deals with the DB and what part does not. And then go deeper. The final goal is to re-do what these lines do :(

> The final goal is to re-do what these lines do

That is quite possibly a huge mistake. (And a very costly one too!)

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#17

Post by user1241320 »

MaxBarraclough wrote: post #12
bilalhusain wrote: post #3

Please elaborate. Why is there need for extracting ASTs? Is it a single 100M line of Java source file? AFAIK Java has limitations on method size. If code is already organized into file and methods try to come up with some sort of UML representation. (I am assuming you are trying to understand the code base, not profiling or doing code analysis.)

Seconded. > I have to analyse [...] > We've started parsing it and tried to work on extracting abstract syntax trees and all that. Why? How will this help? What are you really after?

Not a single line. The whole Java codebase.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#18

Post by jacquesm »

user1241320 wrote: post #13
jacquesm wrote: post #5

Callgraph. Then document the larger chunks, working your way down. It's like having a map versus having no map at all. And 100M lines? Are you sure there is no code generator at work here?

It's code that's been developed and it's been running for decades, I'm afraid.

How does that stop you from making a callgraph?

(On the off chance that you don't know what that is: http://en.wikipedia.org/wiki/Call_graph)

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#20

Post by sp332 »

As a first pass, try deleting as much code as possible :) If there are files or whole projects that aren't needed anymore, they're just slowing down your analysis. Also some dead-code analysis could be helpful, at least in broad strokes. You could instrument the code with a test coverage tool, then run the code instead of the tests to see what code gets reached.

Edit: You could also look for duplicated code, and quickly refactor that to just be in one place.

Post reply on HN