Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On navigating a large codebase

blog.royalsloth.eu

Post reply on HN
Print view View original

41–50 of 139 posts

Re: On navigating a large codebase

#41

Post by Gehinnn »

sillysaurusx wrote: post #24

I navigate codebases by cat'ing all the files together (prefixed with filename) and piping it into vim. I was shocked how much I learn by this seemingly-horrible technique. For example, the python files that actually get deployed are often quite different from the ones in source control. For tensorflow, at least. I regularly read 1M+ lines of code this way. Not an exaggeration; vim scales, nothing else does.

If you read/scan 10 lines a second, you still need over 24 hours non-stop to read a 1M+ code base. I doubt a random file ordering is helpful! Especially if you lose code navigation features like "go to definition".

Re: On navigating a large codebase

#42

Post by gambiting »

mtzet wrote: post #16

The advice about using both grep /and/ the IDE is very good. Often they are framed as in opposition to each other, but in reality they're just tools. IDE's are great when they work, but it's entirely possible to make it confused. I keep hearing to get better IDE's, especially from Java developers who seem to have nicer IDE's than us C++ schmucks, but even the best IDE will not save you when your program is really an…

Speaking of IDEs - I work in video games development, huge codebase that's over a decade old, heavily templated C++ code - I've switched off the IDE "suggestions" long time ago, visual studio is just wrong about incorrect/missing code like 90% of the time. Just hit compile and read the errors, I have files that VS shows as nearly entirely wrong, squiggly lines everywhere, and yet they compile and link fine. And the opposite where VS doesn't see any issue at all but they don't build. Or they build fine using MSVC but not in Clang, or vice versa, and VS has no idea.

Re: On navigating a large codebase

#43

Post by lstamour »

aste-risk wrote: post #38
lstamour wrote: post #9

Earlier quoted context omitted.

Yep, Sourcetrail can do that, for the languages it supports. (It has an SDK so additional languages can be added, with effort.) Give it a method, another method, and it will draw a line from point A to B (with all the functions in between) using static analysis plus you can explore before and after to see what calls what. You can even see field usage though there it can be confused sometimes, understanding varies by…

Can you use sourcetrail on properitery codebases as well? I see it's GPL and according to my understanding, it's okay to use it on properitery software as long as you don't make any modifications to the sourcetrail software itself. Are there any hidden commercial licenses before I try it out on my company's codebase?

I’m not a lawyer but if you’re not embedding GPL code output into your code, you’re fine. Using GPL code to write or reason about code under a different license is not the same thing as having GPL software output a copy of its own GPL-licensed code, for example: https://softwareengineering.stackexchange.com/questions/5221...

The only other risk is letting your company’s proprietary code be visible by third-parties but Sourcetrail runs locally on your computer and can run completely offline.

As to Sourcetrail’s licensing— it previously had a closed license and was supported by a startup with a number of employees. It recently went open source and can be supported financially through Patreon: https://www.sourcetrail.com/blog/open_source/

Re: On navigating a large codebase

#44

Post by mattmanser »

bluefirebrand wrote: post #3

This is a great article. I felt like it was describing a job I recently left, especially this piece: > It’s fine to have less experienced people working on a large system as long as they have the elders overseeing their work. In the world where senior titles are handed left and right, that is often not the case and it’s how you end up with a very fragile system that is suitable for a replacement as soon as it was bui…

Use the IDE for intellisense? Great idea except the database models are in a different project...

If for whatever reason you can't link the project properly, as a tip for the future, I've ended up just making a separate copy of the project which included a reference to the uncompiled project.

And in the worst instance I actually used a reverse generated project from the binary. I actually gradually refactored that auto-generated code into more readable code too!

Re: On navigating a large codebase

#45

Post by sillysaurusx »

Gehinnn wrote: post #41
sillysaurusx wrote: post #24

I navigate codebases by cat'ing all the files together (prefixed with filename) and piping it into vim. I was shocked how much I learn by this seemingly-horrible technique. For example, the python files that actually get deployed are often quite different from the ones in source control. For tensorflow, at least. I regularly read 1M+ lines of code this way. Not an exaggeration; vim scales, nothing else does.

If you read/scan 10 lines a second, you still need over 24 hours non-stop to read a 1M+ code base. I doubt a random file ordering is helpful! Especially if you lose code navigation features like "go to definition".

You'd be surprised.

Suppose I want to "go to definition" for a class named Saver.

  /^class Saver\>
19 times out of 20, this works. It's also instant; my vim will likely get me there faster than your IDE's go to definition functionality. (Looking at you, pycharm!)

Here's my flow.

  >>> import tensorflow as tf
  >>> tf.train.Saver
  
  >>> from tensorflow.python.training import saver
  >>> saver
Then I open /usr/local/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py.

Suppose I want to know: Where are all the places that Saver is used in all of tensorflow?

  time find . -type f -name '*.py' | xargs merge | ft py
  Vim: Reading from stdin...

  real 0m0.930s
  user 0m0.243s
  sys 0m0.414s

  /\
Boop: https://i.imgur.com/EJ5bZSW.png

Literally every usage of Saver in all of Tensorflow.

So let's say you're interested in a specific line. This one, for example:

  # being added to the GLOBAL_VARIABLES collection, so that Saver()
Boop: https://i.imgur.com/YSAzhCJ.png

I did that by highlighting "being added to the GLOBAL_VARIABLES", ctrl-c, then pressing "u" to undo the :%v//d, then / followed by ctrl-v.

That might sound hard, but with muscle memory I don't even think about it -- it's like explaining how you open a can of food. Do you really think about where you place your fingers, or the pressure of your nail on the flap of the can? No, you just open it up. Same thing here; it's automatic.

Way faster than IDEs, and I get just as much (or more) info.

I'd love to use pycharm, but the slowness keeps pushing me back to this technique.

Re: On navigating a large codebase

#46

Post by disgruntledphd2 »

Gehinnn wrote: post #41
sillysaurusx wrote: post #24

I navigate codebases by cat'ing all the files together (prefixed with filename) and piping it into vim. I was shocked how much I learn by this seemingly-horrible technique. For example, the python files that actually get deployed are often quite different from the ones in source control. For tensorflow, at least. I regularly read 1M+ lines of code this way. Not an exaggeration; vim scales, nothing else does.

If you read/scan 10 lines a second, you still need over 24 hours non-stop to read a 1M+ code base. I doubt a random file ordering is helpful! Especially if you lose code navigation features like "go to definition".

With the language server protocol, this works reasonably well in Python code bases for me in Emacs.

Re: On navigating a large codebase

#48

Post by sverhagen »

sydd wrote: post #32
bluefirebrand wrote: post #3

This is a great article. I felt like it was describing a job I recently left, especially this piece: > It’s fine to have less experienced people working on a large system as long as they have the elders overseeing their work. In the world where senior titles are handed left and right, that is often not the case and it’s how you end up with a very fragile system that is suitable for a replacement as soon as it was bui…

> Code comments? Nah. This is one of my biggest gripes. Someone (I think uncle bob) said that good code is self-documenting, which is bs in 95% of the cases. Yeah, you don't need to document the convertMinsToSecs() method, but most real life codebases are full with edge cases, shortcuts, temporary solutions, half-complete reorganizations. So people use this for writing no comments at all, whereas a few words of comme…

I think Uncle Bob got cancelled /s, but politics aside, I don't think he was necessarily wrong about good code documenting itself, but it came with a lot of direction about what then exactly constitutes good code. If people don't bother to hone the skills of good types and methods, well named and with clear responsibilities, of course they're not clearing the bar to drop the comments. Having grown more senior, I believe I have gotten a little better at expressing meaning and intent through the code itself, and I'm surely writing a lot less comments because of it, which seems a win overall.

Re: On navigating a large codebase

#49

Post by valenterry »

abhishekjha wrote: post #22

And then you try navigating an Akka Framework based source with their ask pattern. No way to navigate where the control flow would go with inheritance in the mix.

Yes! Maybe it changed a bit with akka typed, but I stick with: use as few actors as you need to get your job done.
Post reply on HN