Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: I have to analyze 100M lines of Java – where do I start?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 126 posts

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#81

Post by jerven »

jacquesm wrote: post #76
jerven wrote: post #69

Earlier quoted context omitted.

Don't try to figure out how the code does what does yet. Figure out what systems exists inside it: 1. What kind of modules? 2. Which servers/hardware? 3. Which databases/datastores? 4. What systems talk to what? 5. What test systems exist or existed? 6. Which api/frameworks where used? 7. Who is currently working on them/maintaining it? 8. Is anyone left who used to? 9. Why is a rewrite on the table? 10. Is there any…

Code complexity does not increase linearly. 100 M lines is stupendous.

Not really, my project is 3 FTE for 8 years. Double it to 16 years and then multiple the number of developers by 100.

Consider a large enterprise having 300 developers in multiple teams, I am not at all surprised that they can manage to write 100 million lines or so. Also I think this is really a system of systems, and in my experience probably has large parts developed by the lowest bidding firms. Which when software is developed for 10 years or more means more than one way to do the same thing.

Also having to deal with lots off ancient systems and working around weird bugs probably fixed years ago. You know things like bugs in Java 1.2 on HP-UX and stuff like that, or errors in Oracle 7i etc...

Plus functional duplication because team A did not know subteam C2 build the same thing...

Editing my comment instead of replying to the excellent comment by @jacquesm as hn does not allow me to reply to the reply.

Actually completing the transfer of a codebase like that is unlikely to a new team without much much more of a handover. But some high/middle frustrated with the current system manager asking a team to start rebuilding before it gets shut down a few months/years later is very possible. An other plausible option is a corporate take over... But then I would expect a very experienced team to work on it who do not need to ask HN for this kind of thing.

I personally have been in a situation where code moved between companies and no documentation or old developers where available. Not as large as this only a 1 or 2 million lines of code/xml. But I am no longer surprised by the stupid acts that large corporations can perform.

And this must be systems not just one. You can't build a single jar file of 50+ millions of lines of code than could have loaded in a JVM around 1.3.1 on even high end hardware for the time.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#82

Post by jacquesm »

jerven wrote: post #81
jacquesm wrote: post #76

Earlier quoted context omitted.

Code complexity does not increase linearly. 100 M lines is stupendous.

Not really, my project is 3 FTE for 8 years. Double it to 16 years and then multiple the number of developers by 100. Consider a large enterprise having 300 developers in multiple teams, I am not at all surprised that they can manage to write 100 million lines or so. Also I think this is really a system of systems, and in my experience probably has large parts developed by the lowest bidding firms. Which when softwar…

How realistic does it sound to you that a codebase that size would be transferred to a new team without any of the old team, outside of a hack of a bank or a reversal of some outsourcing decision or something like that?

Typically the value of such a codebase is determined by the quality of the team maintaining it and the degree to which it is documented.

Complexity of software constructs is not linear and enough books have been written about simply multiplying and dividing manyears and lines of code that I don't think we need to hash that out all over again. See 'the mythical man-month' and many similar books and articles.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#83

Post by MaxBarraclough »

user1241320 wrote: post #17
MaxBarraclough wrote: post #12

Earlier quoted context omitted.

Seconded. > I have to analyse [...] > We've started parsing it and tried to work on extracting abstract syntax trees and all that. Why? How will this help? What are you really after?

Not a single line. The whole Java codebase.

That doesn't answer my question.

How does parsing ASTs help accomplish the goal? What is the actual goal here? What is meant by 'analyse'?

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#84

Post by tcopeland »

mml wrote: post #2

Funny, I have 50,000 individual Java apps to analyze. I started with a copy/paste detector. Pmd has a free one. Good luck!

Wow, a blast from my distant past... a link to the copy/paste detector:

http://pmd.sourceforge.net/pmd-5.1.3/cpd-usage.html

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#85

Post by EtienneK »

1) Focus on the functional use-cases and not code.

2) Identify integration points to other systems and ask why they are there

3) Realize that a "big-bang" rebuild never works and that it's better to break up the system into smaller pieces and replace them piece by piece.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#86

Post by radicalbyte »

klibertp wrote: post #59
radicalbyte wrote: post #44

Do you and your team have experience of Java development? Your question sounds like something that someone with either no real experience and/or no experience of an object-oriented language would ask. 100 million lines is a lot of code. Why do you need to "parse it to extra the AST"? That's crazy. Do you have the original design documents and architectural documentation? If you do, read it.

Downvoted. Even if the design docs would exist, it would take months to read them, without any guarantee that they correspond to the reality. Meanwhile, automated analysis of actual code can give you at least high-level overview of the codebase and maybe a hint where to start digging. Getting AST is a first step required for most automated tools to do their work. EDIT: I acted too rashly and downvoted your post befor…

Hahaha, no problem.

Static Analysis would be my second step, but first I'd have a look at the architectural documentation. I can't imagine that a project of this size wouldn't at least have a Powerpoint explaining the structure and concepts of the code.

Then it's time to start using tools.

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#87

Post by rkalla »

sergiosgc wrote: post #26

For rewrite from scratch projects, I always start by identifying the use cases covered by the application. You don't need the code for that. Just run the application and identify what it is that it does. Then, work backwards. For each use case, use the existing code as specification of the use case behavior. At 100 million lines, I'd suspect this is either an extremely large project, where a rewrite from scratch is i…

This, 100x this.

You and your team don't just have to build an understanding of the code (e.g. frameworks, patterns, DB's, etc.) but the app itself so you can truly understand it's purpose.

Your rewrite won't (hopefully?) also be 100mm lines so being able to understand the high level purpose of the app completely and then diving deep from there, you may (hopefully) find many places where the system can be simplified.

Are you really not exaggerating? 100mm lines of source? Yiiiikes...

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#88

Post by radicalbyte »

jerven wrote: post #69
user1241320 wrote: post #47

Earlier quoted context omitted.

Any human line-by-line/application-by-application analysis is (for this particular discussion) out of the scope. The size of the thing and the way we thought we were going to work is quite different. For instance, suppose we produce AST for all the routines/pieces of logic/you_name_it we wanted to then find similar patterns or clusters that would give us hint to then work on a "pareto-like" way. As already stated it'…

Don't try to figure out how the code does what does yet. Figure out what systems exists inside it: 1. What kind of modules? 2. Which servers/hardware? 3. Which databases/datastores? 4. What systems talk to what? 5. What test systems exist or existed? 6. Which api/frameworks where used? 7. Who is currently working on them/maintaining it? 8. Is anyone left who used to? 9. Why is a rewrite on the table? 10. Is there any…

When you go above the 1m-or-so LOC it gets much easier to move to more data-driven designs.

Of course the OP could be including all test cases, data etc in his LOC; in which case you could easily reach the hundred-million LOC mark...

Re: Ask HN: I have to analyze 100M lines of Java – where do I start?

#90

Post by dprice1 »

dugmartin wrote: post #58

Understanding the "shape" of a codebase is something I've always been interested in and I started building a tool to help me understand and traverse code here: http://sherlockcode.com/ However I don't think it would scale to 100M lines of code. I have run Linux through it and it was acceptable (both in run times and browse times). At 100M lines of code you need some way to see an overall "map" of the codebase and the…

This thing seems like a good start but I have a bug report for you. For me at least, it's a deal-breaker. When browsing a source file, (firefox 32.0 on macos) pageup/pagedown/spacebar and up/down arrows do not scroll the code, even when the code pane has focus. Pressing any of these give focus to the search box. I need to be able to use keyboard navigation at least for scrolling.
Post reply on HN