Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kraken Technologies: How we organise our large Python monolith

blog.europython.eu

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Kraken Technologies: How we organise our large Python monolith

#11

Post by globalreset »

gyulai wrote: post #10

I quite like layered dependency structures. I even have a lua codebase where I've linearized the sourcecode files into a particular order. The layout looks like this: 01_foo/ 03_foo.lua 04_bar.lua 15_whatever.lua 02_bar/ 22_whatever.lua 56_something.lua It has always bothered me in codebases that they don't have a "beginning" and an "end", and therefore it's difficult for a reader to know where to start when they jus…

That sounds like one of the things they could write about in dailywtf.

First, who has time to read a project from start to beginning. Second, as the time changes the dependencies between parts will change as well.

Re: Kraken Technologies: How we organise our large Python monolith

#12

Post by banashark »

gyulai wrote: post #10

I quite like layered dependency structures. I even have a lua codebase where I've linearized the sourcecode files into a particular order. The layout looks like this: 01_foo/ 03_foo.lua 04_bar.lua 15_whatever.lua 02_bar/ 22_whatever.lua 56_something.lua It has always bothered me in codebases that they don't have a "beginning" and an "end", and therefore it's difficult for a reader to know where to start when they jus…

This was always one of my favorite parts of f#. It’s not only mandated for structures within a file (no hoisting), but also for files within a project.

Reading and understanding an f# code base has always been vastly more simple to me than the incredible indirection of some other codebases (trying to find where the provider for the factory of the abstract class member that’s an interface implemented by 10 almost exactly similar c# classes? Good luck /s)

Re: Kraken Technologies: How we organise our large Python monolith

#13

Post by walthamstow »

bitdivision wrote: post #3
photonthug wrote: post #2

Never heard of https://import-linter.readthedocs.io/ before. Not sure if I like this type of solution, but it's interesting, and certainly the problem is real.

The author, David Seddon, is also the author of the package [0]. So I'm guessing Kraken was an initial user. 0: https://import-linter.readthedocs.io/en/stable/authors.html

Interesting that he doesn't blow his own horn about being the author of the linting tool. He simply describes it as 'an open source tool'

Re: Kraken Technologies: How we organise our large Python monolith

#14

Post by barefeg »

CraigJPerry wrote: post #4

This is java circa 2009 flavoured, this model was a popular approach in the java community around that timeframe. Was enforcer the name of the maven plugin? I can’t remember now, but it became fairly popular. I wonder if this might be susceptible to the same issues that we had in java, specifically it doesn’t prevent accidental duplication, it makes refactoring harder/slower and it’s problematic for the team when it…

What is the current flavor in Java?

Re: Kraken Technologies: How we organise our large Python monolith

#15

Post by gyulai »

globalreset wrote: post #11
gyulai wrote: post #10

I quite like layered dependency structures. I even have a lua codebase where I've linearized the sourcecode files into a particular order. The layout looks like this: 01_foo/ 03_foo.lua 04_bar.lua 15_whatever.lua 02_bar/ 22_whatever.lua 56_something.lua It has always bothered me in codebases that they don't have a "beginning" and an "end", and therefore it's difficult for a reader to know where to start when they jus…

That sounds like one of the things they could write about in dailywtf. First, who has time to read a project from start to beginning. Second, as the time changes the dependencies between parts will change as well.

Who has time to write documentation? As the software evolves, the documentation will need changing. Who has time to write unit tests? As the modules evolve, the unit tests will need updating. Why do any of these things?

A well-written piece of code that actually solves a complex problem is a demanding mistress and doesn't appreciate the "who has time..." attitude.

Among the software engineering good practices that I engage in, I'm finding this to be one of the highest bang-for-the-buck ones.

For example, literate programming is a thing that many people believe is worth the effort, at least in certain scenarios and at least for certain parts of codebases: But, if you can make your codebase more readable without requiring additional prose to be written, then that's a comparatively high bang-for-the-buck alternative.

In practice, linear structure doesn't need all that much updating at all: You may have noticed that I don't use strictly sequential numbers. Early in the life of the codebase, I will generally skip lots of numbers, so that I can fill in the gaps with code that comes later. Mostly, all it really takes is to be intentional about where in the linear order to insert new code.

Also, note that it's still useful, even if no one ever goes on a mission to read the codebase in its entirety. If the codebase as a whole has a linear structure, that also imposes a linear structure on any subset of it. If you have five sourcecode files that are somehow pertinent to something you're doing, you immediately know the order in which to look at them.

Re: Kraken Technologies: How we organise our large Python monolith

#16

Post by ianandrich »

photonthug wrote: post #5
CraigJPerry wrote: post #4

This is java circa 2009 flavoured, this model was a popular approach in the java community around that timeframe. Was enforcer the name of the maven plugin? I can’t remember now, but it became fairly popular. I wonder if this might be susceptible to the same issues that we had in java, specifically it doesn’t prevent accidental duplication, it makes refactoring harder/slower and it’s problematic for the team when it…

> this happens because the rules enforcement can only happen effectively if the rules themselves are perfect and complete and that gets harder to guarantee with scale and speed of iteration. IMHO, project-level assertions about initialization-order feels a little bit dirty, but potentially practical as long as there's only a few rules. Surely even a bazillion line monolith can still often be thought of in terms of a…

Have you had any experience using deal on personal projects or professionally yet?

Re: Kraken Technologies: How we organise our large Python monolith

#17

Post by globalreset »

gyulai wrote: post #15
globalreset wrote: post #11

Earlier quoted context omitted.

That sounds like one of the things they could write about in dailywtf. First, who has time to read a project from start to beginning. Second, as the time changes the dependencies between parts will change as well.

Who has time to write documentation? As the software evolves, the documentation will need changing. Who has time to write unit tests? As the modules evolve, the unit tests will need updating. Why do any of these things? A well-written piece of code that actually solves a complex problem is a demanding mistress and doesn't appreciate the "who has time..." attitude. Among the software engineering good practices that I…

Large part of biggest crimes I've seen in software comes down to picking some allegedly good property and optimizing for it irrespective of costs and limitations it involves. Encapsulation, modularity, flexibility, readability, purity, testability - you name it. Some dubious, some reasonable - no matter what, it usually involves costs and drawbacks.

I have 99 big problems developing software, and not being able to read it from start to end ain't one.

I'm a bit jealous for people advocating such things, being able to work on software tiny enough to even be able to think this is practical. Literate programming could work for tiny software that can be approximately flattened to a 1d "story": "beginning -> end" with only minor detours.

Most of projects I worked with was large enough that things are optimized precisely for not having to understand large parts that were abstracted away (and the problem is to try to do it without introducing too much complexity).

Re: Kraken Technologies: How we organise our large Python monolith

#18

Post by gyulai »

globalreset wrote: post #17
gyulai wrote: post #15

Earlier quoted context omitted.

Who has time to write documentation? As the software evolves, the documentation will need changing. Who has time to write unit tests? As the modules evolve, the unit tests will need updating. Why do any of these things? A well-written piece of code that actually solves a complex problem is a demanding mistress and doesn't appreciate the "who has time..." attitude. Among the software engineering good practices that I…

Large part of biggest crimes I've seen in software comes down to picking some allegedly good property and optimizing for it irrespective of costs and limitations it involves. Encapsulation, modularity, flexibility, readability, purity, testability - you name it. Some dubious, some reasonable - no matter what, it usually involves costs and drawbacks. I have 99 big problems developing software, and not being able to re…

It's not so much about the size of the codebase, it's more about the tradeoff of how much code there is, versus how much intrinsic/real complexity.

By "real" complexity, I mean to exclude complexity for complexity's sake, or complexity that's a side effect of a piece of software being concerned with itself and its own structure rather than with solving any problem that exists outside of it. -- We can surely all agree that the latter is bad.

Academic prototypes, for example, tend to solve highly complex problems with small codebases, but the problems are often contrived or they are tiny pieces of systems that would have to be much larger to be of any use.

Software systems you encounter in your run-of-the-mill software engineering business often have huge codebases, but the large size is not due to the fact that there's any problem there of any real complexity. The system just needs to solve trivial problems, and there are just a lot of them.

But you can certainly have systems that need to solve many problems and some of those problems have high intrinsic complexity. In such a scenario, it's important to recognize that there's a part of your system that's "extra special" due to its complexity, and needs to be approached in a way that's different from how you approach most of your engineering. You might need to write a library in a literate programming style, and then use that library from your other code, which isn't written in literate style. Engineering organizations are seldom able to have that kind of flexibility. The opinion you're likely to get is "we don't do literate programming here". The outcome is that such organizations are just not capable of writing such systems, because a system is only as good as the worst job you've done at solving any of the problems it raises.

I'd also like to point out that you're guilty of a bit of a strawman argument there, because all I did was point out a technique I sometimes find useful, and you responded as if I was advocating applying it "irrespective of costs and drawbacks" by pointing out that this often ends up as "crimes against software".

Re: Kraken Technologies: How we organise our large Python monolith

#19

Post by unusable_link »

I'm very sorry for the offtopic but how am I supposed to read the article if they covered the button to close the cookie popup by the button to subscribe to their newsletter? [0]

I tried clicking the bottom and the corner and it just doesn't work. Is it a dark pattern? Is it done deliberately to make people click on the newsletter button more?

[0] https://ibb.co/dg2krqL

Re: Kraken Technologies: How we organise our large Python monolith

#20

Post by kgeist »

>Enforcing layering with Import Linter

We use a similar tool for PHP - deptrac, it checks that our architectural decisions about layers and modules, and their interactions, are not violated. If they're violated, the build fails.

Interestingly, we don't use similar tools for our microservices. I'm not sure whether it's an oversight or it's just unnecessary in a microservice.

Post reply on HN