61–70 of 126 posts
Earlier quoted context omitted.
developed in java and running for decades? how long has the language been around?
I started using Java in early '95 (having begged a copy from someone at Sun) and I seemed to be one of the first people writing stuff outside of Sun.
You haven't really described your goals: What do you want to extract from your analysis? Metrics to tell you what's "wrong" with the existing code base? Some sort of model of the system's semantics?
We'd love to know what these lines do. For example what part of this codebase deals with the DB and what part does not. And then go deeper. The final goal is to re-do what these lines do :(
Earlier quoted context omitted.
I started using Java in early '95 (having begged a copy from someone at Sun) and I seemed to be one of the first people writing stuff outside of Sun.
I started in August 1995 when I first heard about it, which I thought was when they released it but it may have had a variety of trickle releases.
However, I did think Java was rather good and when I co-founded a start-up in mid '95 we positioned ourselves as a "Java company" - which was no bad thing in the long term as we were in a reasonable position when Netscape, Novell and IBM later decided they wanted to support it. Indeed our 2 round of VC investment was led by Novell - quite unusual for a UK company at the time...
Earlier quoted context omitted.
AST = abstract syntax tree. With 100M lines your process will take many years.
Am I missing something or are you saying that making an AST like a compiler will help you understand a huge codebase better and faster?
In general having AST is always better than having a plain text file, unless you want to read it. But then you can easily dump AST back to text whenever you want.
Yeah, making AST will help you analyse your codebase programmatically which in turn will let you understand the codebase better and faster. This is some very basic programming knowledge, I think. Or is it not? Some comments here don't know what AST even is - is this the state of PL knowledge in the mainstream? Lisp and Smalltalk people would be very, very sad if it was so.
Earlier quoted context omitted.
Am I missing something or are you saying that making an AST like a compiler will help you understand a huge codebase better and faster?
"making an AST like a compiler" will give you a semantic - as contrasted with textual - understanding of the code. Especially with how much data Java encodes in its source, this makes it a very good base for running automated analysis and visualisation tools, perhaps written by yourself. In general having AST is always better than having a plain text file, unless you want to read it. But then you can easily dump AST…
You'd be better off to start with just the build scripts and build tools.
ASTs are great for increasing understanding of much smaller projects but for something this size you'd likely end up with very little to show for your effort except the crashlogs of your tools.
You need to go 'coarse' before you can go 'fine' on something this magnitude.
This is not a 3 week project, just mapping the thing properly will take (man)years.
If the idea is to rebuild the application I would start at the beginning: what is the input and output? What does the user see? What are the various service hooks? How are they called? When are they called? Why are they called?
Then I would look at how the overall code is organized. What modules are there? Are there core utility modules that seem to be called by everything else? What are those doing? What are the most used business function modules?
Then I would look at the build process. What external dependencies are there? What are they used for? Are there modern alternatives? What about internal dependencies? Does the build process look organized and sane or a chaotic mess cobbled together over the years?
Do you have logs? What is the most utilized part of the application?
Then I would look at the database. What tables seem to be the most important (if you could get usage stats from a running and used application that could help, but otherwise you could look at which tables are keyed off of the most)? What data is most critical? What modules interact with that data? What tables are essential for supporting this data?
Answering these questions will start to fill out a nice 30,000 ft view of the application and how it is actually used.
You are going to get the most bang for your re-implementation buck by identifying and replacing often used utilities (especially if they are custom built or built before a good de-facto standard was formed for that particular task) with modern, well known, alternatives. Then follow the execution path of the most often used modules and the modules that work with the most critical data and work down the list.
With a 100 million line application, you are looking at many years to understand all of it and many years to re-implement. To get anything useful in a reasonable amount of time you are going to have to boil it down as much as possible, then break what's left down into independent functional areas and tackle it an area at a time.
The code is important, but if it were me, I'd try to analyze how the users and processes work before I'd dig into the nitty gritty of the code too much if at all possible. I'd build the smallest functional unit from what I deem to be the most important and critical module(s) trying to cut as much cruft from the application and database as possible. I'd get users and processes to start banging on the new app as soon as possible. I'd keep the old application up and running and available to analyze (not for the users but for the developers and analysts) as the team works down the most often used parts. I would not try to analyze the whole mess in one go beyond finding waypoints as described above. If possible I'd also try to get users to understand that the old way is not necessarily the right way. Much pain has been caused trying to make new systems work exactly like the old systems when the new systems don't face the same constraints. It is just too tempting to say 'make it work like it did'.
Earlier quoted context omitted.
It doesn't. I was just answering about the code generator thing you mentioned.
100M lines of java code developed 'line-by-line' would make it one of the largest software projects that I've ever heard about. Without telling you directly that you should disqualify yourself (after all I don't know you), if you don't have the knowledge about the tools employed to deal with medium sized projects (say up to 1M lines) how on earth will you deal with 100 times as much?
If it's not generated, then it may be "versioned" that way. I saw projects where the entire codebase was copied over to new directories tens of times, and no previous "version" was ever deleted.
But, after giving it some thought, 100 mloc is really MUCH, like in 50x more than anything I directly worked with. It does sound kind of improbable, but hey, it could happen :)
Earlier quoted context omitted.
"making an AST like a compiler" will give you a semantic - as contrasted with textual - understanding of the code. Especially with how much data Java encodes in its source, this makes it a very good base for running automated analysis and visualisation tools, perhaps written by yourself. In general having AST is always better than having a plain text file, unless you want to read it. But then you can easily dump AST…
I disagree an AST would help with a project this size, it's just unmanageable. You'd be better off to start with just the build scripts and build tools. ASTs are great for increasing understanding of much smaller projects but for something this size you'd likely end up with very little to show for your effort except the crashlogs of your tools. You need to go 'coarse' before you can go 'fine' on something this magnit…
Yeah, I started commenting before the realization of how HUGE this thing would be hit me, sorry :)
With a codebase like that, it's better to look at it through the users' eyes, rather than trying to reverse engineer the business from the code. Things that look like bugs in the code may actually be features for the users, or may have been absorbed so long ago that they've fundamentally changed the nature of the business. You don't need to understand the whole codebase. It will take years. Best to focus on what the…
Any human line-by-line/application-by-application analysis is (for this particular discussion) out of the scope. The size of the thing and the way we thought we were going to work is quite different. For instance, suppose we produce AST for all the routines/pieces of logic/you_name_it we wanted to then find similar patterns or clusters that would give us hint to then work on a "pareto-like" way. As already stated it'…
1. What kind of modules?
2. Which servers/hardware?
3. Which databases/datastores?
4. What systems talk to what?
5. What test systems exist or existed?
6. Which api/frameworks where used?
7. Who is currently working on them/maintaining it?
8. Is anyone left who used to?
9. Why is a rewrite on the table?
10. Is there any way you can work on smaller pieces at a time?
11. What are the pain points of the current users (will tell you what area to focus on)?
12. Can you document what comes in and out?
If you do want to go down the static analysis path, start with existing tools before trying to build your own. If needed get external help for this.
A 100 Million lines of code is not so bizarre. The project I work on is currently about 300,000 lines and a project some 300 times larger is quite imaginable for me.
Earlier quoted context omitted.
BINGO!
BINGO all you want but if you're at liberty to disclose such things (or to confirm them) you should have included it in your original write up.