Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parse, Don't Validate (2019)

lexi-lambda.github.io

Post reply on HN
Print view View original

261–270 of 288 posts

Re: Parse, Don't Validate (2019)

#261

Post by kortex »

nooorofe wrote: post #242
kortex wrote: post #84

This principle is how pydantic[0] utterly revolutionized my python development experience. I went from constantly having to test functions in repls, writing tons of validation boilerplate, and still getting TypeErrors and NoneTypeErrors and AttributeErrors left and right to like...just writing code. And it working ! Like one time I wrote a few hundred lines of python over the course of a day and then just ran it... a…

There is one thing concerns me about type annotation. With annotation Python code looks like Java code, but with it I don't have advantages of Java (like speed). Besides I see things like `x: typing.Union[str, typing.Any]` - maybe better no annotation.

> x: typing.Union[str, typing.Any]

That's absolutely a valid and useful annotation. It tells me, and autocomplete, that "x" is probably a str, more likely than not, but I need to be aware that it might not be.

Re: Parse, Don't Validate (2019)

#262

Post by kortex »

JPKab wrote: post #155
kortex wrote: post #84

This principle is how pydantic[0] utterly revolutionized my python development experience. I went from constantly having to test functions in repls, writing tons of validation boilerplate, and still getting TypeErrors and NoneTypeErrors and AttributeErrors left and right to like...just writing code. And it working ! Like one time I wrote a few hundred lines of python over the course of a day and then just ran it... a…

Curious, but how does pydantic compare to marshmallow? I'm currently using marshmallow in a project, specifically using the functionality that builds parsers from dataclasses. I was curious what the differences were.

Personal opinion: pydantic crushes Marshmallow. Not even a fair fight. Pydantic is more performant, has better mypy/linter integration, and more powerful data model. We had a project where we pre-emptively used marshmallow to marshall/validate data. Had to remove that and solely use it at the ORM layer because of performance (and it still struggles).

I haven't used pydantic's ORM integration, but I don't hesitate to use pydantic models everywhere as business logic classes unless I need ludicrous speed.

That's all opinion, but I'd definitely give pydantic a swing.

Re: Parse, Don't Validate (2019)

#263

Post by kortex »

ukj wrote: post #258
kortex wrote: post #77

Earlier quoted context omitted.

I mean, GP said "you can't have routers" and maybe I'm being dense by interpreting that as "never or almost never," but even with a generous "too hard to be practical," I still don't think it's correct. And I explicitly said "escape hatch" meaning language feature. You don't need that much indirection to get routers in Haskell, Rust, Go, C, C++... like I fail to see how implementing routers are a barrier in strict ty…

What C++, Haskell, Rust, Go etc. call "dynamic dispatch" is what a router calls "static routing". The defining characteristic is that the lookup table is pre-determined and immutable at runtime. What routers call "dynamic routing" is having the lookup table mutable at runtime. There can be no equivalent to that in type-safe languages because when you mutate the dispatch table you lose type-safety.

> What routers call "dynamic routing" is having the lookup table mutable at runtime.

> There can be no equivalent to that in type-safe languages because when you mutate the dispatch table you lose type-safety.

That's exactly what I mean by escape hatches. Rust has unsafe. You can write a WTF FastInverseSqrt in Rust. You shouldn't 99% of the time. But you can.

Rust also has Box, Rc, Arc, and other tools. I'm not fluent enough in rust to know how to, but I'm quite confident and will eat my hat if you can't accomplish what is effectively a mutable vtable dispatch in Rust.

But also...why? Yeeting a function call into the void without any type knowledge seems way more bug-prone than interface/trait-based dispatch, to what benefit? Save developer time? I've spent countless dev-days of my life I won't get back debugging exactly this kind of dynamic-dispatched, json-dsl, frankly loosey-goosey untyped bullshit. There's so many better ways.

Re: Parse, Don't Validate (2019)

#264

Post by kortex »

mixedCase wrote: post #139
ukj wrote: post #115

Earlier quoted context omitted.

>I do dynamic dispatch all the time in strongly typed languages. I believe semantics is getting in our way of communicating. You don't do dynamic dispatch ___ALL___ the time. You only do it at runtime. And you only do static type safety at compile time. Those are different times. You can't have both of those features at the __SAME TME__, therefore you can't have both features ALL the time. They are mutually exclusive…

"All the time" is an english expression, please, look it up before going all-caps Python name mangle convention on me. Yes, of course dynamic dispatch is a runtime phenomenom, that's the dynamic part of it. But there's nothing stopping the code that performs dynamic dispatch from being strongly typed. Strong types are instructions used to prove that the code holds certain properties, they are a separate program from…

> "All the time" is an english expression, please, look it up before going all-caps Python name mangle convention on me.

I know reaction comments are discouraged on HN, but this had me in stitches. Top-tier gourmet dig right there.

Re: Parse, Don't Validate (2019)

#265

Post by kortex »

mixmastamyk wrote: post #202
hardwaregeek wrote: post #177

Earlier quoted context omitted.

Agreed. It's like saying "oh well I just fly the airplane really carefully". A lot of codebases eclipse the point where one person can understand the whole system. Testing, static analysis and tooling are what allows us to keep the plane flying.

Agreed with the end of your post. However, the top post approaches religious dogma. I argue against that even if one has some good points.

> However, the top post approaches religious dogma.

I don't deny it. Join the cult of Static Python. We have cookies! And lower stress levels!

I usually wrap that spiel with my caveat "this depends greatly on your neurotype, style, environment, and other things." I have ADHD and my brain struggles with keeping bits of state in memory, so having to remember the type of every variable without my IDE tracking it for me is a huge performance drain.

However, I would contend even if your neurotype supported that mental workflow... it isn't actually better. Humans on average can handle 7 +/- 2 "pieces" of information in focus. Why spend any of your precious half-dozen pieces of salient consciousness on something a machine is really good at doing?

Re: Parse, Don't Validate (2019)

#266

Post by kortex »

Scarbutt wrote: post #184
kortex wrote: post #171

Earlier quoted context omitted.

No this was like over a week, and 100% due to the tooling. Pydantic, pycharm, black, mypy, and flake8. Pretty much went from "type hints here and there" to "what happens if I try writing python as if it were (95%) statically typed." I'd been testing well before this point but it's not the same as writing test. The development process is totally different when you write structured types first and then write your logic…

The development process is totally different when you write structured types first and then write your logic. 10/10 would recommend. Unless you were writing very small throwaway scripts, in what world where you writing your logic first and thinking about your data structures later?

The world of data science, ML and computer vision research. It's very academic-heavy, which has two effects. There's an insulation between commercial software dev and it, which results in a lot of NIH, a lot of reinforcement of bad habits, and a lag in propagation of best practices. Second, and related to this, there is a tendency to just piecemeal hack towards the solution, rather than architect the system from the ground up.

It's not zero consideration of data structures, it's mostly a focus on the main data type (arrays and data frames) and not really thinking about typed records, data models and such. The majority of types are float, str, dict, np.ndarray, pd.DataFrame. No dataclasses, minimal classes, and when classes are used, it's Java101 style "all the bad parts of OOP" programming. Sadly, I've spent years in this space before learning better.

Re: Parse, Don't Validate (2019)

#267

Post by kortex »

jimmaswell wrote: post #178
JPKab wrote: post #150

Earlier quoted context omitted.

It's an immediate tell when someone makes statements like the one you're replying to. It immediately tells me that they've never worked on large software projects, and if they have they haven't worked on ones that lasted more than a few months. I apologize to folks reading this for my rather aggressive tone but I've been writing software for a long time in numerous languages, and people with the unit tests as an afte…

I've worked on large scale projects for a long time. A large portion of the kind of code I've written is impractical or impossible to actually "unit test" e.g. Unity3D components or frontend JS that interacts with a million things. When something weird is going on I'll have to dig in with console logs and breakpoints. On certain backend code where I am able to do unit tests, they do catch the occasional edge case log…

> A large portion of the kind of code I've written is impractical or impossible to actually "unit test" e.g. Unity3D components or frontend JS that interacts with a million things.

Opinion: This is actually a symptom of what is (imho) a pervasive problem lodged deep in the collective consciousness of software dev: OOP with fine-grained objects. I blame (early) Java in large part for exacerbating this mentality. Encapsulation of state with mutator methods in particular. It sprays state all over the application, encourages mutation in place over immutability, coupling, validating-not-parsing, and makes it nigh-well impossible to write good tests.

It's really hard to write objects that enforce all invariants under every mutation. And when you have state strewn everywhere, it's impossible to test every nook and cranny. The combinatorial space explodes.

Objects are helpful for encapsulating state when they are course-grained, mutations are atomic, coupling occurs in one place, state changes are auditable, and the entire state can be replayed/set at once, to enable mock tests and subsystem integration tests. AKA, things like databases, reactors, and persistent data structures.

Re: Parse, Don't Validate (2019)

#268

Post by wabain »

ukj wrote: post #257
justinpombrio wrote: post #125

Earlier quoted context omitted.

Are you sure you know what dynamic dispatch is? Java has dynamic dispatch, and it is a statically typed language. In Java, it's often called "runtime polymorphism". https://www.geeksforgeeks.org/dynamic-method-dispatch-runtim... And using it doesn't give up any of Java's type safety guarantees. The arguments and return type of the method you call (which will be invoked with dynamic dispatch) are type checked.

In English there seems to be the eternal confusion between what things are and what we call them. When a router does lookups from a static table it's "static routing". When Java does lookups from a static table it's "dynamic dispatch". The same type of computation is being characterised as both "static" and "dynamic". When a router does lookups from a dynamic table it's "dynamic routing" - there is no equivalent in J…

I don't think this analogy quite holds together. A router doing lookups from a table is implementing a static routing strategy from a control plane perspective; it's using statically configured values instead of using dynamic information about the network topology gleaned using a routing protocol like BGP. But an implementation of that strategy in terms of table lookups is dynamic—it's walking a data structure to retrieve values which were specified in the runtime configuration, not at compile time.

The reason that "dynamic dispatch" in Java, etc. is called that is that the instance of the dispatch table to use is chosen dynamically, rather than being fixed at the callsite. It's true that Java doesn't let the shape of the dispatch table change at runtime, but that's not what dynamic vs static refers to conventionally in this context. The ability to dynamically add and remove methods from a class is something which you typically only get in dynamically typed languages but dynamic dispatch and dynamic typing are not the same thing.

In particular, while a full-featured routing information base implementation will usually use some form of dynamic dispatch to customize the behavior of routes originated through different protocols, it's very uncommon for an implementation to rely on dynamic typing which adds or mutates the methods associated with different entities. That's simply a different kind of tool used for different purposes. It's something which can be helpful in object-relational mapping, for instance, because you can create methods based on a dynamic database schema. The RIB is not going to have a schema like that which changes at runtime.

Re: Parse, Don't Validate (2019)

#269

Post by ukj »

wabain wrote: post #268
ukj wrote: post #257

Earlier quoted context omitted.

In English there seems to be the eternal confusion between what things are and what we call them. When a router does lookups from a static table it's "static routing". When Java does lookups from a static table it's "dynamic dispatch". The same type of computation is being characterised as both "static" and "dynamic". When a router does lookups from a dynamic table it's "dynamic routing" - there is no equivalent in J…

I don't think this analogy quite holds together. A router doing lookups from a table is implementing a static routing strategy from a control plane perspective; it's using statically configured values instead of using dynamic information about the network topology gleaned using a routing protocol like BGP. But an implementation of that strategy in terms of table lookups is dynamic—it's walking a data structure to ret…

>But an implementation of that strategy in terms of table lookups is dynamic—it's walking a data structure to retrieve values which were specified in the runtime configuration, not at compile time.

That's precisely the point. You can specify part of the routing table at compile/configure time - the rest gets generated at runtime.

The data/control plane distinction is conceptual. It doesn't hold in memory when the router is handling its own network traffic - it has a single routing table/world-view.

My own routing table is shared by the data plane AND control plane.

At some point you will receive an external data (routing update) which requires runtime validation, you will do reflection and update your own routing table (ring 0 address space) based on external events.

>The RIB is not going to have a schema like that which changes at runtime.

The schema need not change. The entries/relations between objects changing is sufficient to violate type-safety.

Route add( *str1, *str2) to Number.add().

Re: Parse, Don't Validate (2019)

#270

Post by ukj »

kortex wrote: post #263
ukj wrote: post #258

Earlier quoted context omitted.

What C++, Haskell, Rust, Go etc. call "dynamic dispatch" is what a router calls "static routing". The defining characteristic is that the lookup table is pre-determined and immutable at runtime. What routers call "dynamic routing" is having the lookup table mutable at runtime. There can be no equivalent to that in type-safe languages because when you mutate the dispatch table you lose type-safety.

> What routers call "dynamic routing" is having the lookup table mutable at runtime. > There can be no equivalent to that in type-safe languages because when you mutate the dispatch table you lose type-safety. That's exactly what I mean by escape hatches. Rust has unsafe . You can write a WTF FastInverseSqrt in Rust. You shouldn't 99% of the time. But you can. Rust also has Box, Rc, Arc, and other tools. I'm not flue…

Well isn't that's precisely my point!

"unsafe" turns off your compiler's guard rails.

You CAN accomplish mutable vtable dispatch and in doing so you will undermine Rust's type-safety!

Yeeting a function call into the void is what async network clients usually do. You will soon get a result. Maybe.

Post reply on HN