Review open patches
Ask HN: How to understand the large codebase of an open-source project?
31–40 of 49 posts
Re: Ask HN: How to understand the large codebase of an open-source project?
#32You can use the debugger on low level api calls to get pretty much anywhere in the codebase. If you want to find whats changing a label to "foo" you can hook into every set_Text call and put a conditional breakpoint on all label changes to break on "foo", then just go up the callstack to find the logic. This strategy works on network interfaces and file interfaces as well. I abused this on our 2M+ SLOC legacy codebas…
A variation on this is instrumenting an codebase with profiler flamegraphs, which I find a lot more straightforward to drill in/out with than stepping through functions one at a time.
I normally use manual profiling libraries - I need an excuse to try out orbit, which uses automatic instrumentation for similar purpouses: https://www.youtube.com/watch?v=L8w0qI8qzvM
A little lower fidelity in some ways, but faster iteration than what I've been doing in others...
Re: Ask HN: How to understand the large codebase of an open-source project?
#33This is one of the times when an IDE (hat-tip to Jetbrains, but many other are available) really comes into its own. Fast navigation facilitates understanding. If you really are a vim die-hard, "exuberant ctags" is an excellent tool in this area.
Re: Ask HN: How to understand the large codebase of an open-source project?
#34I'd like to point out that if it is truly large, then deep intimate knowledge of specific parts will never be truly had. Those who do know the entire project understand the flow and architecture of it but the details are blurred.
To add, close mentorship with a core member of the team is one of the ways to get around this hurdle. Anything less formal than that, in many cases, it's fairly impossible (unless you have some very similar experience you can draw from).
Re: Ask HN: How to understand the large codebase of an open-source project?
#35You have to be realistic about what you expect here. Suppose it takes you 1/100th of the time it took them to write the code to understand it. Some of these projects have had hundreds of people working on them for years. It may take you months to get a basic grasp of things.
At my current company it often takes 6 months for experienced people to become productive, and that's with a helping hand.
IDEs are nice, but grep remains the best tool. You tend to need to find things in XML files and config as well.
Re: Ask HN: How to understand the large codebase of an open-source project?
#36You have to be realistic about what you expect here. Suppose it takes you 1/100th of the time it took them to write the code to understand it. Some of these projects have had hundreds of people working on them for years. It may take you months to get a basic grasp of things. At my current company it often takes 6 months for experienced people to become productive, and that's with a helping hand. IDEs are nice, but gr…
It depends heavily on code quality.
In general, I think that high quality code attracts high quality pull requests. That's because high quality code is easy to understand and easy to change because the core structure of the code is fundamentally sound and well suited to the problem that it is solving.
Re: Ask HN: How to understand the large codebase of an open-source project?
#37Earlier quoted context omitted.
I suppose the difference is you'd normally use a debugger to find out why the code isn't doing what it's supposed to, rather than using it to find out what the code is supposed to be doing in the first place. I don't consider it abuse either, though.
Agreed, and riffing on that a bit — I find the name "debugger" is actually troublesome when teaching newcomers (I work with kids of various ages). I see the debugger much more like a "REPL for a compiled language" than a "bug removal tool". I try to teach people to think of it as an interactive inspection tool, not as (merely) a thing to fix broken programs.
"REPL for a compiled language", or "Binary REPL", I like that.
Besides that, it's times like these when I realise how useful IDEs are. Instead of needing to use grep (or something similar), I can simply right click on a variable and choose 'Find all references' (this is in VS, but I'm sure many of the leading IDEs will have this feature). When I use the command line it's to save myself time.
Re: Ask HN: How to understand the large codebase of an open-source project?
#38- git-extras has some nice features... "git summary" and "git effort". These commands show: most active users, most active files (by active days), etc.
- gource can be used to visualize the activity in a repository.
- sloccount and cloc can be used to count lines of code.
- For C/C++/C#, you can run Doxygen, and ask it to generate documentation for undocumented entities. This can make give you another perspective on the code base.
- In runtime there are various tools you can use to audit what a program does... On Linux you've got strace, lsof, wireshark and many others... On Windows you've got Process Monitor from Sysinternals, as well as wireshark.
Re: Ask HN: How to understand the large codebase of an open-source project?
#39My personal copy-paste summary of a similar topic on HN some time ago (https://news.ycombinator.com/item?id=9784008):
# Getting familiar with a new codebase
### Use the right tools
- grep, ack, ag, global search (Visual Assist)
- doxygen, javadocs
- sourcegraph, pfff (facebook), open-grok, SourceInsight
- Proper IDE, REPL
- chronon (dvr for java)
- SWAG (Software Architecture Group)
- Static code analysis
### Use the repository
- Find most relevant (frequently, recently edited) files
- Find dependancy graphs
- Get basic information like which languages are used for what
- Use good source control so that you don't have to worry about breaking things
- Look at commits, in general or for specific issues
- Browse the directory structure, packages, modules, namespaces etc.
- Use "blame" to see when things changed
### Ask questions
- Talk to the customer, find out the purpose of the application
- Pair up with another developer who is more familiar with the code
### Read the documentation
- Look at use cases, diagrams describing architecture, call graphs, user docs
- Understand the problem domain
- Add more documentation as your knowledge grows
- Comments and docs might be wrong!
### Browse the code
- Skim around to get a general idea and a feeling for where things are
- Look at public interfaces, header files first
- Find out which libraries are used
- Take some important public API or function in the UI and follow the code from there. Find implementations of functions, dive into related functions and data structures until you understand how the it's done. Then work your way back out.
- Use tools to quickly find declarations, definitions, references and calls of variables/functions/etc., usage patterns
- Find the entry point of the program
- Figure out the state machine of the program
- Focus on your particular issue
- Use a large, vertical screen with small font size with a pane to show file/class structure
### Take notes
- Use pencil and paper to write down summaries, relationships between classes, record definitions, core functions and methods
- Write a glossary: Function names, Datatypes, prefixes, filenames
- Document everything you understand and don't understand
- Use drawings to create a mental model
### Look at the data
- Find out how the data is stored in the database
### Build the project
- First make sure you can build it and run it
### Use the debugger, profiler and logging
- Set breakpoints, poke around the code, change variables, inspect local variables, stack traces, ...
- Watch the initialization process
- Start from main() and see where it goes
- Find hotspots with the profiler
- Set logging level to max/add logging and use the output to go through the code
### Edit the code
- Adopt the existing coding style
- Try to recreate and fix small bugs, make sure you understand the implications of the fix to the rest of the program first
- Tidy up the code according to the common standard after talking with the team
- Make the code clearer (best with tests)
- Add TODO comments
- Add comments describing what you think the code does
- Hack some feature into the code, then try to not break other stuff, build up a mental model over time, re-write the feature properly
### Use Tests
- Run the tests, make sure they are all passing
- Create new tests
- Browse the tests as an examples reference
Re: Ask HN: How to understand the large codebase of an open-source project?
#40Step by step, part by part, and IMHO it's no different than the onboarding process on any new project. I usually try first to understan a general idea of the whole project, where is what (the structure), a bird's eye view of the business logic and the supporting DB structures. And then with time you dive deeper in areas where work needs to be done. If the code is structured properly usually you can start working on a few related parts without a need to know much about the rest of the system. And tests are there to give you the confidence to refactor and change things freely...