Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kraken Technologies: How we organise our large Python monolith

blog.europython.eu

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Kraken Technologies: How we organise our large Python monolith

#21

Post by IshKebab »

gyulai wrote: post #10

I quite like layered dependency structures. I even have a lua codebase where I've linearized the sourcecode files into a particular order. The layout looks like this: 01_foo/ 03_foo.lua 04_bar.lua 15_whatever.lua 02_bar/ 22_whatever.lua 56_something.lua It has always bothered me in codebases that they don't have a "beginning" and an "end", and therefore it's difficult for a reader to know where to start when they jus…

I also work on a codebase that has to have a particular order (because the compiler doesn't support any kind of modules or interfaces).

It's a nightmare. Very difficult to follow.

Re: Kraken Technologies: How we organise our large Python monolith

#22

Post by shoo »

CraigJPerry wrote: post #4

This is java circa 2009 flavoured, this model was a popular approach in the java community around that timeframe. Was enforcer the name of the maven plugin? I can’t remember now, but it became fairly popular. I wonder if this might be susceptible to the same issues that we had in java, specifically it doesn’t prevent accidental duplication, it makes refactoring harder/slower and it’s problematic for the team when it…

> prevent accidental duplication

sometimes sections of code can be syntactically duplicate at some point in time but aren't really semantically duplicate.

e.g. in terms of the example in the post, of having different client-specific logic. at times two different clients may end up with syntactically duplicate code in their corresponding client-specific layers. but unless this code reflects that some detail specific to both clients is actually subject to the same pressures and constraints, what is syntactically duplicate today may not be tomorrow when client A's code needs to change due to pressure P that does not also impact client B, so the syntactically duplicate code is no longer syntactically duplicate.

factoring out syntactically duplicate common code between things that need to change at different times for different reasons is unhelpful as it introduces unnecessary coupling.

not sure if i've understood the context of "prevent accidental duplication" but reasons for change and avoiding unnecessarily coupling is something that jumps into mind when considering multiple clients or multiple regions

Re: Kraken Technologies: How we organise our large Python monolith

#23

Post by shoo »

photonthug wrote: post #2

Never heard of https://import-linter.readthedocs.io/ before. Not sure if I like this type of solution, but it's interesting, and certainly the problem is real.

Yeah, the problem is certainly real. I remember working with a much smaller python monorepo some years ago (probably 100x fewer python modules but the beginnings of multiple client-specific variants that depended upon a swathe of core libraries) and we were already starting to see places it would be valuable to declare constraints banning certain types of imports that broke our desired module architecture and enforce those constraints during CI without needing humans to notice the violations during code review.

Re: Kraken Technologies: How we organise our large Python monolith

#24

Post by ericjmorey »

loveparade wrote: post #8
mint2 wrote: post #6

Note, this is not the crypto kraken, it’s a different kraken for those who ignore crypto related stuff

Why does it matter? Just because I don't like Google doesn't mean they suck at writing Python code. The same goes for anything crypto.

People are emotional and sometimes addressing an emotional concern helps. In this case, people who are exhausted by cryptocurrency culture would be more willing to read this write-up being assured that it's not cryptocurrency related.

Re: Kraken Technologies: How we organise our large Python monolith

#25

Post by skinnyarms »

shoo wrote: post #22
CraigJPerry wrote: post #4

This is java circa 2009 flavoured, this model was a popular approach in the java community around that timeframe. Was enforcer the name of the maven plugin? I can’t remember now, but it became fairly popular. I wonder if this might be susceptible to the same issues that we had in java, specifically it doesn’t prevent accidental duplication, it makes refactoring harder/slower and it’s problematic for the team when it…

> prevent accidental duplication sometimes sections of code can be syntactically duplicate at some point in time but aren't really semantically duplicate. e.g. in terms of the example in the post, of having different client-specific logic. at times two different clients may end up with syntactically duplicate code in their corresponding client-specific layers. but unless this code reflects that some detail specific t…

Maven Enforcer does have a rule for duplicate class _names_ (which includes the entire namespace), but I think the author was more talking about it's enforcement of duplicate dependencies. For example, it will detect when you have different (transitive) versions of the same library referenced by your code or dependencies.

Re: Kraken Technologies: How we organise our large Python monolith

#26

Post by fijiaarone »

CraigJPerry wrote: post #4

This is java circa 2009 flavoured, this model was a popular approach in the java community around that timeframe. Was enforcer the name of the maven plugin? I can’t remember now, but it became fairly popular. I wonder if this might be susceptible to the same issues that we had in java, specifically it doesn’t prevent accidental duplication, it makes refactoring harder/slower and it’s problematic for the team when it…

You can see duplication in the one example they provide —- between clients and territories.

Every client has to have duplicate modules for every territory that they operate in and if they begin to operate in another territory, that has to be copied in — or they have to keep a copy of all territory modules for every client. I think it is probably the former, since they talk about pruning and ignored imports.

Re: Kraken Technologies: How we organise our large Python monolith

#27

Post by samsquire »

gyulai wrote: post #10

I quite like layered dependency structures. I even have a lua codebase where I've linearized the sourcecode files into a particular order. The layout looks like this: 01_foo/ 03_foo.lua 04_bar.lua 15_whatever.lua 02_bar/ 22_whatever.lua 56_something.lua It has always bothered me in codebases that they don't have a "beginning" and an "end", and therefore it's difficult for a reader to know where to start when they jus…

I really like this idea, it reminds me of my "entrypoints" folder idea. The idea is you put the file where everything is registered or started in one place. Such as int main() {} and your dependency inversion container or controllers.

Re: Kraken Technologies: How we organise our large Python monolith

#28

Post by dezgeg »

febed wrote: post #9

IMHO the customer specific stuff should have been forked out into a separate repo instead of cluttering up the main one

Hard disagree, splitting to multiple repos has a heavy cost; biggest of them the pain if you ever need to commit something to both repos at same time. Splitting the codebase to multiple repos shouldn't be done lightly.

Re: Kraken Technologies: How we organise our large Python monolith

#30

Post by electroly »

gyulai wrote: post #10

I quite like layered dependency structures. I even have a lua codebase where I've linearized the sourcecode files into a particular order. The layout looks like this: 01_foo/ 03_foo.lua 04_bar.lua 15_whatever.lua 02_bar/ 22_whatever.lua 56_something.lua It has always bothered me in codebases that they don't have a "beginning" and an "end", and therefore it's difficult for a reader to know where to start when they jus…

In F# you have to order your source files and it sucks, hard. It's not typically considered a benefit but rather an unfortunate side effect of how F# works that you have to just live with.
Post reply on HN