Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kraken Technologies: How we organise our large Python monolith

blog.europython.eu

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Kraken Technologies: How we organise our large Python monolith

#31

Post by gyulai »

electroly wrote: post #30
gyulai wrote: post #10

I quite like layered dependency structures. I even have a lua codebase where I've linearized the sourcecode files into a particular order. The layout looks like this: 01_foo/ 03_foo.lua 04_bar.lua 15_whatever.lua 02_bar/ 22_whatever.lua 56_something.lua It has always bothered me in codebases that they don't have a "beginning" and an "end", and therefore it's difficult for a reader to know where to start when they jus…

In F# you have to order your source files and it sucks, hard. It's not typically considered a benefit but rather an unfortunate side effect of how F# works that you have to just live with.

> [...] It's not typically considered a benefit[...]

...see sibling comment, where the commenter does seem to imply they rather like it.

Re: Kraken Technologies: How we organise our large Python monolith

#32

Post by electroly »

gyulai wrote: post #31
electroly wrote: post #30

Earlier quoted context omitted.

In F# you have to order your source files and it sucks, hard. It's not typically considered a benefit but rather an unfortunate side effect of how F# works that you have to just live with.

> [...] It's not typically considered a benefit[...] ...see sibling comment, where the commenter does seem to imply they rather like it.

I stand by my statement. Sibling comment is not typical. There's a reason that file order is rare among programming languages. Languages never decide to mandate this out of the blue when their language design doesn't otherwise require file order; it's something you only mandate if you can't implement orderless files. You get a bunch of nice properties when they're not strictly ordered, like compiling files in parallel, rebuilding an earlier file without having to rebuild a later file, etc. In F#, compilation is forced to start at the first modified file and proceed linearly to the end of the project.

OP's technique, notably, avoids these problems by numbering the files but not actually mandating at the compiler/language level that they be ordered. Here they are acting as a hint to the programmer but the files are still, as far as the language is concerned, orderless. That would seem to be a nice compromise if you like the idea of an ordered codebase. More of a literate programming technique.

Re: Kraken Technologies: How we organise our large Python monolith

#33

Post by gyulai »

electroly wrote: post #32
gyulai wrote: post #31

Earlier quoted context omitted.

> [...] It's not typically considered a benefit[...] ...see sibling comment, where the commenter does seem to imply they rather like it.

I stand by my statement. Sibling comment is not typical. There's a reason that file order is rare among programming languages. Languages never decide to mandate this out of the blue when their language design doesn't otherwise require file order; it's something you only mandate if you can't implement orderless files. You get a bunch of nice properties when they're not strictly ordered, like compiling files in paralle…

Is that just because F# is a typed language?

My personal experience has been that it's a pain to implement this as soon as you have strong typing and try to avoid "unsafe" casts. For example, in modern Java, if you have some kind of a Container over elements of type T, and a matching ContainerIterator, you will want some way in each of the type declarations to refer to the other type declaration, so you need cyclical imports.

If you look at it through the lens of a dynamically-typed language like Python 3.4 or Lua, the problem just disappears. This is the case to a lesser extent even for Java before it had generics, and collections were just collections of Object, and you used "unsafe" casts all the time. I am finding this practice not the slightest bit limiting in Lua, nor do I get the feeling that the readability of my code suffers in any way at all. (Rather the contrary, as I've pointed out before).

Re: Kraken Technologies: How we organise our large Python monolith

#34

Post by electroly »

gyulai wrote: post #33
electroly wrote: post #32

Earlier quoted context omitted.

I stand by my statement. Sibling comment is not typical. There's a reason that file order is rare among programming languages. Languages never decide to mandate this out of the blue when their language design doesn't otherwise require file order; it's something you only mandate if you can't implement orderless files. You get a bunch of nice properties when they're not strictly ordered, like compiling files in paralle…

Is that just because F# is a typed language? My personal experience has been that it's a pain to implement this as soon as you have strong typing and try to avoid "unsafe" casts. For example, in modern Java, if you have some kind of a Container over elements of type T, and a matching ContainerIterator , you will want some way in each of the type declarations to refer to the other type declaration, so you need cyclica…

I wouldn't blame the static type system; this is more of a language design decision of its own. IMO, F# does it this way because ML did it this way. I think it's more of a cultural norm than a technical decision. There is a certain theoretical purity to requiring ordered declarations across a whole project.

It's valuable to compare F# against C++ here. F# and C++ both require ordered declarations within a file (while C# and Java do not), but F# requires the files themselves to be ordered, too, while C++ does not. Both are statically typed languages. C++ avoids requiring a file-level order by defining file-level compilation units and a system for importing and exporting symbols between compilation units to allow each file to be processed separately with the imports/exports resolved during a final linking phase. There's complexity involved in accomplishing this. F# has not done it; an F# project is the same as concatenating all of the source files together, and thus declarations must be ordered across files just like they are within a file.

Re: Kraken Technologies: How we organise our large Python monolith

#35

Post by mint2 »

loveparade wrote: post #8
mint2 wrote: post #6

Note, this is not the crypto kraken, it’s a different kraken for those who ignore crypto related stuff

Why does it matter? Just because I don't like Google doesn't mean they suck at writing Python code. The same goes for anything crypto.

I personally almost skipped it, thinking it might be a crypto company puff piece masquerading as a python show and tell. crypto is known for puffery and it’s exhausting. But this article turned out to be not even from a crypto company, instead just having the misfortune of sharing name.

Re: Kraken Technologies: How we organise our large Python monolith

#36

Post by calderwoodra »

Code layers makes sense, but organizing your code in layers does not.

For example, I much prefer this:

  project/
    feature_1/
      view.py
      service.py
      model.py
    feature_2/
      view.py
      service.py
      model.py
over this:

  project/
    views/
      feature_1_view.py
      feature_2_view.py
    services/
      feature_1_service.py
      feature_2_service.py
    models/
      feature_1_models.py
      feature_2_models.py
And their import-linter library doesn't support this very well.

Re: Kraken Technologies: How we organise our large Python monolith

#37

Post by pxul »

unusable_link wrote: post #19

I'm very sorry for the offtopic but how am I supposed to read the article if they covered the button to close the cookie popup by the button to subscribe to their newsletter? [0] I tried clicking the bottom and the corner and it just doesn't work. Is it a dark pattern? Is it done deliberately to make people click on the newsletter button more? [0] https://ibb.co/dg2krqL

Yes this bugged me too. Was able to clear the cookie pop-up by rotating my phone.

Re: Kraken Technologies: How we organise our large Python monolith

#38

Post by photonthug »

ianandrich wrote: post #16
photonthug wrote: post #5

Earlier quoted context omitted.

> this happens because the rules enforcement can only happen effectively if the rules themselves are perfect and complete and that gets harder to guarantee with scale and speed of iteration. IMHO, project-level assertions about initialization-order feels a little bit dirty, but potentially practical as long as there's only a few rules. Surely even a bazillion line monolith can still often be thought of in terms of a…

Have you had any experience using deal on personal projects or professionally yet?

So far I'm working it into personal projects gradually. Runtime checks are at least as useful as something like pydantic's validation, and more expressive, because validation is basically "pre" but now deal can provide the "post". DBC via pre/post might be "just" syntactic sugar on asserts, but notation matters: it looks great, it reduces cognitive load. It easily does stuff that's impossible or just awkward with types, so having an alternate way to express constraints available probably keeps me out of certain rabbit holes.

Really looking forward to trying verification ( https://deal.readthedocs.io/basic/verification.html ) but I think I need to hit some critical mass of annotation first.

Re: Kraken Technologies: How we organise our large Python monolith

#39

Post by nick-of-time »

unusable_link wrote: post #19

I'm very sorry for the offtopic but how am I supposed to read the article if they covered the button to close the cookie popup by the button to subscribe to their newsletter? [0] I tried clicking the bottom and the corner and it just doesn't work. Is it a dark pattern? Is it done deliberately to make people click on the newsletter button more? [0] https://ibb.co/dg2krqL

I used uBlock's element zapper to kill it. Mobile Firefox supports it. To exit zapper mode oncxe you're done (I had to look this up) swipe right twice.

Re: Kraken Technologies: How we organise our large Python monolith

#40

Post by david-j-vujic »

Some have already mentioned Polylith, but what might not be known is that it is available for Python too (I’m the maintainer of the Python tooling).

What is mentioned in the article reminds of the Polylith Architecture, even if Polylith probably has a more simplistic view on code: a flat structure, made of something called “bricks” - that are small namespace packages. These ones are combined into features, the features are combined into apps or services.

You can choose how to deploy your artifacts - as a single Monolith, or several microservices. Since the code is not coupled to the built artifact, it isn’t a big thing to change it from one type to another.

Docs here if you want to know more: https://davidvujic.github.io/python-polylith-docs/

Post reply on HN