A project with a single 11,000-line code file • Hacker News Board

Re: A project with a single 11,000-line code file

#321

Post by cupofpython » Mon, Apr 04, 2022, 3:32 PM UTC

notreallyserio wrote: post #135

wk_end wrote: post #98

Earlier quoted context omitted.

I don’t think this dichotomy is helpful. I’m presently working at a startup that’s trying to dig itself out of a hole created by the first CTO, who in doing things “quickly” created an MVP so buggy, inefficient, crash-prone, and unmaintainable that we can’t retain customers or engineers. As always, there’s a balance to be struck, and ways to operate quickly that don’t sacrifice quality too much.

I'm kind of surprised that you can't find engineers interested in creating a new implementation of an existing application that is actually used by people. I think that might be my dream role.

That type of role typically requires senior engineers at a junior salary

Re: A project with a single 11,000-line code file

#322

Post by manquer » Mon, Apr 04, 2022, 4:41 PM UTC

disgruntledphd2 wrote: post #294

manquer wrote: post #241

Earlier quoted context omitted.

If work with not easily deterministic code typically but not always ML models like say speech-to-text or Face-Recognition or classification/ recommendation systems or network performance dependent applications like video conferencing that wouldn't be that feasible . Almost nothing is impossible to test yes, however to know and be able to mock the data for each test case can be extremely hard and at some point not wor…

> If work with not easily deterministic code typically but not always ML models like say speech-to-text or Face-Recognition or classification/ recommendation systems or network performance dependent applications like video conferencing that wouldn't be that feasible Nah, most ML systems (actually doing something in the world) are mostly just ordinary code, which can be tested like any other code (as you put it into f…

I haven't heard of any method to test the model apart from statistical analysis of reference /training data.

The model is what gets continually updated and is the critical path that needs coverae, Testing interfaces are trivial and at times not critical to test if already running in production for a while (you probably have already caught most/all issues and know what to test or take care of in a interface rewrite).

It is not about impossible, here is an example, let's say you are working on English speech-to-text model, the next version works better in your set of benchmarks.

It could for example perform very poorly (compared to your previous model) for accented English or mixed with other languages, for older people or in noisy environments like a car, or for for specific subjects like medical/legal dictation and so on and since your benchmarks originally didn't cover these types of scenarios you wouldn't know one way or another.

These were real cases all added to speech-to-text models after user feedback and adequate demand being identified and research effort put in, and now training/benchmark data includes these. There are plenty of scenarios not yet solved (mixing two languages is active area of research) or not included because user feedback didn't capture it, of not yet worth solving.

Neural network testing is hard because by design they have millions(and these days billions) of parameters as inputs and you cannot feasibly test every possible outcome, you will not know what all things to check until people start using your app in ways you never thought off.

NN /ML is not hard requirement this is true for any complex systems. Shazam type fingerprinting for example is just spectrography and Fourier transforms, NN is just newest tool devs use. All complex systems with thousands and above parameters have same problems

Re: A project with a single 11,000-line code file

#323

Post by squeaky-clean » Mon, Apr 04, 2022, 4:43 PM UTC

Jtsummers wrote: post #222

squeaky-clean wrote: post #214

Earlier quoted context omitted.

Maybe it's the editors that need updating. It would be pretty neat to have the chunks of functions / namespaces model as lots of tiny separate files on the disk, but a sort of view-layer so that you can view all the related ones in one virtual file. You could even then have multiple virtual files that include the same raw files. For example in a video game your player, monsters, health potions, and attacks could all…

Look up "class browser", in particular "smalltalk class browser" to see examples, current and historic, of editors supporting that kind of approach to navigating codebases.

Maybe I'm looking up the wrong thing but these basically list the classes and method names and hotlink to the code?

I mean something that would dump them all into, one contiguous "file" from the editor's perspective. Included components (I don't want to say classes because it could work in a functional language or maybe you just don't include full classes but certain methods) wouldn't have to be coupled to the main class you're editing. Like if you have a decoupled event system you could pull in just the events relevant to the idea you're working on. You could have different views depending on what idea you're working on and save them as their own file.

To use the gamedev analogy again you could have MonsterCombat.view MonsterAI.view MonsterAnimations.view which would all expose different subsets of the Monster class and various related methods from other classes/modules.

Re: A project with a single 11,000-line code file

#324

Post by squeaky-clean » Mon, Apr 04, 2022, 4:48 PM UTC

jamesfinlayson wrote: post #256

squeaky-clean wrote: post #214

Earlier quoted context omitted.

Maybe it's the editors that need updating. It would be pretty neat to have the chunks of functions / namespaces model as lots of tiny separate files on the disk, but a sort of view-layer so that you can view all the related ones in one virtual file. You could even then have multiple virtual files that include the same raw files. For example in a video game your player, monsters, health potions, and attacks could all…

I might be remembering wrong but I thought Visual C++ 6 might have had a class browser as an alternative to a file browser. Maybe modern Visual Studio has it too?

I don't really mean a class browser but something that completely abstracts the underlying code organization and let's you create a sort of meta-file that includes all the related code for various ideas.

Like you could have all of the code from class A except the debug related methods, a few methods from class B, just a few functions from a static MathUtils class, so on.

Maybe your ClassAUnitTest meta-file could include some of the ugliest methods from the class you're testing but not the entire class.

Re: A project with a single 11,000-line code file

#325

Post by squeaky-clean » Mon, Apr 04, 2022, 4:50 PM UTC

cyber_kinetist wrote: post #235

squeaky-clean wrote: post #209

Earlier quoted context omitted.

The Player controller from the game Celeste is a single 5600 line file that includes things like systems only used in the tutorial. I honestly don't think it's as bad as some of the criticism it got when the code was released makes it seem, but it certainly could be better looking code. But ultimately, Maddy Thorson isn't selling a block of code. They're selling a game and it has extremely satisfying control of the c…

I remember sharing that Player.cs code from Celeste in the gamedev subreddit, and getting all kinds of weird novice comments about how the code doesn't adhere to 'OOP Principles' or 'there isn't any unit tests' or 'you should split it into multiple files with 100 lines each' or 'you should use an ECS to make a real game'. Laster on Noel Berry did give a response explaining the various design choices behind their code…

I hadn't seen the new Readme update (even though I Googled the repo again to link the cs file lol). Thanks!

Re: A project with a single 11,000-line code file

#326

Post by MauroIksem » Mon, Apr 04, 2022, 4:56 PM UTC

pnt12 wrote: post #313

formerly_proven wrote: post #143

Earlier quoted context omitted.

It always seemed surprising to me how some of the big Oblivion and Skyrim mods would get by with fairly few bugs despite there being no way to have automated tests and some of them having 10k lines of scripting (or much more in some cases) spread around dozens or hundreds of quests (quests in the CE engine are not just the quests you as a player see, but also a huge number of invisible quests because quest state mach…

For a single modder, it may have to do with being a single developer, working on it over a long period of time and being a passion project. For a while, I was the only developer working in a small module of a bigger project: I started the code base, discussed requirements with clients, implemented the needed features, tested the whole product end to end. I developed a very good instinct about it and about what any ch…

This description has been life the last 4 years.

Re: A project with a single 11,000-line code file

#327

Post by benibela » Mon, Apr 04, 2022, 5:07 PM UTC

activitypea wrote: post #54

I don't get the obsession with file length. What's the benefit of having 100 files with one 50-line function per file, over having a 5000 file with 100 functions? Obviously not counting extreme cases where the file size would break some editors' buffers

It depends on your tooling

I just searched for the largest code files on my system and found a 100k file

I opened it on the online repo and gitlab did not want to show it at first. When I clicked on show anyways, Firefox broke trying to load the website and I had to restart it. (then I could not post on HN anymore due to the noprocrast setting there)

When I opened that file in an IDE, it was shown quickly without any issues. But there is a notable delay when typing, so 100k lines are too much.

Other IDEs might already fail with smaller files

Although when the IDE has a "search in the open file" and not a "search in all files of the project" feature, one file is much easier to use than multiple files

Re: A project with a single 11,000-line code file

#328

Post by smaudet » Mon, Apr 04, 2022, 5:15 PM UTC

KwisaksHaderach wrote: post #279

gofreddygo wrote: post #114

Earlier quoted context omitted.

Funnily enough, I have recently had great success by reversing the "best practices" on a distributed "micro services" architecture application into a single big Java file. Best practices were the usual suspects DRY, IOC, SQL + NoSQL, separation of concerns, config files over code, composition over inheritance, unexplainable overlapping annotations, dozens of oversimplified components doing their own thing, and some $…

This application must be really really simple ;), so no database?

You'd be surprised the garbage overhead brings into an app architecture...

These best practices really only make sense in large organizations, i.e. Conways law.

After all, you can't really ask 100 developers to all add code to one file in a couple weeks - they will spend a month or so just resolving conflicts...

100 file repos are designed so that 100 developers can edit them (in theory), and have relatively few conflicts, not because its better code.

As another anecdote, I find whenever I do solo code I can easily spin out thousands of lines of code within a week (includes testing), when doing code on a project with many other devs my rate drops into the range of maybe 200 a week, just because so much time is spent interlinking other code, finding fixing bugs and tests strewn across many files...

Re: A project with a single 11,000-line code file

#329

Post by dopp0 » Mon, Apr 04, 2022, 5:49 PM UTC

exdsq wrote: post #164

sillysaurusx wrote: post #117

Earlier quoted context omitted.

Ironically, this is a description of hacker news itself. https://github.com/shawwn/arc/blob/arc3.1/news.arc (HN has indentation, though.) It’s important to realize that this is good design. It’s hard to separate yourself from the time you live in, but the rewards are worthwhile.

The only readability issue I have with that is the functions expected arguments. Add some types and I’d be very happy to work on it. I believe Facebook uses a single directory of files now as best practice? With the file names including namespaces. That was an HN comment from ages ago so could be wrong or misinterpreted.

> I believe Facebook uses a single directory of files now as best practice Do you have any sources for this?

Re: A project with a single 11,000-line code file

#330

Post by smaudet » Mon, Apr 04, 2022, 5:56 PM UTC

pnt12 wrote: post #313

formerly_proven wrote: post #143

Earlier quoted context omitted.

It always seemed surprising to me how some of the big Oblivion and Skyrim mods would get by with fairly few bugs despite there being no way to have automated tests and some of them having 10k lines of scripting (or much more in some cases) spread around dozens or hundreds of quests (quests in the CE engine are not just the quests you as a player see, but also a huge number of invisible quests because quest state mach…

For a single modder, it may have to do with being a single developer, working on it over a long period of time and being a passion project. For a while, I was the only developer working in a small module of a bigger project: I started the code base, discussed requirements with clients, implemented the needed features, tested the whole product end to end. I developed a very good instinct about it and about what any ch…

Fewer (in charge) developers makes for a more stable product - even open source products fair better when they have a long term maintainer versus ownership changes frequently.

The other thing is style - when used well, state machines don't require testing. There's nothing to test - either your machine works or it doesn't, there is no point testing state transitions because that is the fundamental job of the state machine. You may as well test that addition works.

Ofc, they must be used well - problems may be difficult or overly complex to model as a state machine or even set of state machines - the pattern excels for small problems, less for large ones.