Ask HN: I have to analyze 100M lines of Java – where do I start?
21–30 of 126 posts
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#22Start at the main function. See how things get setup and walk through the code from there. Keep notes on the structure and flow of things (if any). There isn't really an easy way to do this unless it had been documented properly before. AstroGrep is a good Windows based tool that allows you to search within file so you could use it to find which files spit out a particular output to screen. Not sure what you mean by…
AST = abstract syntax tree. With 100M lines your process will take many years.
Am I missing something or are you saying that making an AST like a compiler will help you understand a huge codebase better and faster?
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#23Earlier quoted context omitted.
It's code that's been developed and it's been running for decades, I'm afraid.
How does that stop you from making a callgraph? (On the off chance that you don't know what that is: http://en.wikipedia.org/wiki/Call_graph )
It doesn't. I was just answering about the code generator thing you mentioned.
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#24If you're a UML guy here's a great option http://www.altova.com/umodel/uml-reverse-engineering.html
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#25Earlier quoted context omitted.
AST = abstract syntax tree. With 100M lines your process will take many years.
Am I missing something or are you saying that making an AST like a compiler will help you understand a huge codebase better and faster?
No, an AST will not help but a call graph certainly would (it shows you how the various routines are organized in graph form, who calls who).
An AST for 100M lines would be absolute madness, a call graph just might work and I'm somewhat hoping that it turns out to be either a ton of generated or duplicated code.
I also wonder if the OP isn't out of his depth based on the question(s) asked.
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#26For rewrite from scratch projects, I always start by identifying the use cases covered by the application. You don't need the code for that. Just run the application and identify what it is that it does. Then, work backwards. For each use case, use the existing code as specification of the use case behavior.
At 100 million lines, I'd suspect this is either an extremely large project, where a rewrite from scratch is inadvisable, or that there is a code generator at work. If it is the latter, you want to analyze the code generating source, not the end result.
Anyhow, generically, for a first contact with a new code base, code coverage tools are a good start, as is a call graph debug run of the project. It'll let you spot dead code as well as hot code (code being called at every run of the application). It'll highlight the important and non-important code parts, allowing you to read less code and get a grasp on the architecture.
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#27You haven't really described your goals: What do you want to extract from your analysis? Metrics to tell you what's "wrong" with the existing code base? Some sort of model of the system's semantics?
We'd love to know what these lines do. For example what part of this codebase deals with the DB and what part does not. And then go deeper. The final goal is to re-do what these lines do :(
[deleted]
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#28[deleted]
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#29Earlier quoted context omitted.
How does that stop you from making a callgraph? (On the off chance that you don't know what that is: http://en.wikipedia.org/wiki/Call_graph )
It doesn't. I was just answering about the code generator thing you mentioned.
100M lines of java code developed 'line-by-line' would make it one of the largest software projects that I've ever heard about.
Without telling you directly that you should disqualify yourself (after all I don't know you), if you don't have the knowledge about the tools employed to deal with medium sized projects (say up to 1M lines) how on earth will you deal with 100 times as much?
Re: Ask HN: I have to analyze 100M lines of Java – where do I start?
#30Source to UML: http://www.architexa.com/
Getting call paths: https://github.com/gousiosg/java-callgraph
Line coverage from instrumented jars: http://emma.sourceforge.net/
For this type of request, I'd push back and say, let's identify very small parts of this and begin rewriting those one at a time in an isolated project. Kind of an agile rewrite that will combine the legacy project with the slowly rewritten one. Use the tools to identify parts of the project than can be isolated. Build new interfaces or services to let the old project communicate with the new one. Get a history of the source repository to see where recent edits are and prioritize those to be rewritten first (presuming they want a rewrite to lower maintenance costs).