Cold Showers: For when people get too hyped up about things • Hacker News Board

Re: Cold Showers: For when people get too hyped up about things

#211

Post by dahauns » Sat, Jul 25, 2020, 2:28 PM UTC

archarios wrote: post #128

jdm2212 wrote: post #77

Earlier quoted context omitted.

The study IMHO is reality. Every large, actively maintained system I've ever heard of uses a strongly typed language, or was written in a dynamically typed language but has converted to a gradually typed language. You can't safely refactor without compile time type checking, and you can't maintain a non-trivial system over the long-term if you can't refactor it.

Just write tests and suddenly refactoring is possible without static typing. Those tests will also cover type checking concerns implicitly.

Repeat after me:

Tests are existentially quantified, types are universally quantified.

(Yeah, existential types, I know. Shut up. ;) )

Re: Cold Showers: For when people get too hyped up about things

#212

Post by dahauns » Sat, Jul 25, 2020, 2:35 PM UTC

pavel_lishin wrote: post #123

pixelrevision wrote: post #27

Earlier quoted context omitted.

I think it’s the wrong metric to look at. Static typing still leaves plenty of room for bugs. The capability and discipline of the team would likely be more of a factor than the type system so the studies would be hard to get right. However I do think static typing provides an enormous benefit to picking up code that is 5 years old and written by someone else. The ability to see “this is a nonnullable int32 value typ…

I would change that "5 years" to "5 months" (or maybe even 5 weeks), and argue that you can just as easily be that "someone else" on a long enough time-scale (and I don't mean 5 years.)

That hits closer to home than I'd like to admit. :)

Re: Cold Showers: For when people get too hyped up about things

#213

Post by catalogia » Sat, Jul 25, 2020, 2:40 PM UTC

pantaloony wrote: post #181

rmdashrfstar wrote: post #179

Earlier quoted context omitted.

Gotta love all the HN armchair gatekeepers in this thread

Taking amusement in catching little social lies isn’t exactly gatekeeping. More like sport. I don’t give a damn whether people call themselves Stoics or not and whether they’re sincere—whatever that means—or not, but I’m very sure, specifically, that there are a lot more fans of Meditations than people who’ve read it or even meaningfully read about it, which is funny in that “oh boy, aren’t we humans goofy” sort of w…

One of my hobbies is asking christians if they know about the time god told a bear to maul some kids for making fun of a bald guy. When they say they aren't I dab, tip my fedora, then moonwalk out of the room. Aren't humans goofy?

Re: Cold Showers: For when people get too hyped up about things

#214

Post by tristor » Sat, Jul 25, 2020, 3:05 PM UTC

barrkel wrote: post #135

tristor wrote: post #103

Earlier quoted context omitted.

"serialized" here doesn't really mean processed in serial, it means "serializable" in the context of database information theory. Databases have special concurrency control requirements in order to create hard guarantees on database consistency. You can process queries in parallel and still have a serializable result, because of transaction coordination. Doing this on one server is much easier than doing this across…

There's an enormous leap between something which is slow on SQLite and something which requires etl into a data warehouse or similar tech, columnar store etc. I mean, at least three orders of magnitude, minimum. It's just a ludicrous argument. SQLite is fine for a file format, and in very specific dumb CRUD scenarios it's just about ok. But it's not worth sticking with if you need anything interesting over any volume…

> Ironically, you're more representative of the "we'll need Hadoop" crowd.

That seems an especially odd assertion to make. What is your basis for this comment? I'd suggest it makes clear you know nothing about me.

Re: Cold Showers: For when people get too hyped up about things

#215

Post by tristor » Sat, Jul 25, 2020, 3:17 PM UTC

johncolanduoni wrote: post #177

tristor wrote: post #103

Earlier quoted context omitted.

"serialized" here doesn't really mean processed in serial, it means "serializable" in the context of database information theory. Databases have special concurrency control requirements in order to create hard guarantees on database consistency. You can process queries in parallel and still have a serializable result, because of transaction coordination. Doing this on one server is much easier than doing this across…

There are MVCC systems with serializable, strictly serializable, and even externally consistent transactions. FoundationDB and Spanner are both externally consistent (with geo-replication in Spanner’s case). CockroachDB is serializable, though not strictly serializable. Single-master Postgres can do serializable transactions as well.

I'd be very interested in you providing an example of an MVCC system which is fully serializable. Conventional wisdom is, that while possible, it is prohibitively expensive to ensure a snapshot isolation system like MVCC is fully serializable, and it is explicitly most expensive for analytics / OLAP workloads because you must keep track of the read set of every transaction. It is possible for such a system to exist, then, but it would cost such a performance penalty as to be a painfully bad choice for a workload where OLAP and OLTP would be mixed, bringing us back to the point I originally made.

In most cases, snapshot isolation is sufficient, and some database vendors even conflate snapshot isolation with serializable, but they're not the same thing. I'd be hesitant to believe any vendor claims that they implement serializable MVCC without validating it via testing. As we've been shown by Jepsen, database vendors make many claims, some of which are unsubstantiated. Spanner is very cool technology, however I have personally heard some very interesting claims from folks on the Spanner team that would violate the laws of physics, so again, without a demonstration of their claims, I'd take them with a grain of salt.

Re: Cold Showers: For when people get too hyped up about things

#216

Post by mkoubaa » Sat, Jul 25, 2020, 5:00 PM UTC

hwayne wrote: post #90

fmakunbound wrote: post #61

Earlier quoted context omitted.

In addition to that, I also feel calling ourselves engineers is a stretch.

I used to think this, and then I interviewed people who did both traditional and software engineering professionally, and now I'm not so sure. I did a first draft of what I learned here: https://www.youtube.com/watch?v=3018ABlET1Y I'm hoping to have a written version by the end of September.

I'm a mechanical engineer who writes software for mechanical engineers. I find that the work of MechE moves slower due to operational issues. if there could move fast and break things cheaply to get to market faster they would. all that matters is that the final product is tested and hardened which is something that software shops mostly do anyways

not to mention things like generative design and process automation are getting us to that point.

Re: Cold Showers: For when people get too hyped up about things

#217

Post by johncolanduoni » Sat, Jul 25, 2020, 6:44 PM UTC

tristor wrote: post #215

johncolanduoni wrote: post #177

Earlier quoted context omitted.

There are MVCC systems with serializable, strictly serializable, and even externally consistent transactions. FoundationDB and Spanner are both externally consistent (with geo-replication in Spanner’s case). CockroachDB is serializable, though not strictly serializable. Single-master Postgres can do serializable transactions as well.

I'd be very interested in you providing an example of an MVCC system which is fully serializable. Conventional wisdom is, that while possible, it is prohibitively expensive to ensure a snapshot isolation system like MVCC is fully serializable, and it is explicitly most expensive for analytics / OLAP workloads because you must keep track of the read set of every transaction. It is possible for such a system to exist,…

Both the FoundationDB docs and Spanner whitepapers are very clear that their definitions of strict serializability match with the conventional academic one. FoundationDB does keep track of the read set of every transaction: there are nodes dedicated to doing that in their architecture. You can even add things to the read set of each transaction without actually reading them to get additional guarantees about ordering (e.g. to implement table-level locks). FoundationDB (and obviously Spanner as well) don’t have any massive penalties for this; FoundationDB clusters can handle millions of operations per second with low single digit read and commit latencies: https://apple.github.io/foundationdb/performance.html.

If your condition for believing these claims is approval from Jepsen (i.e. Kyle Kingsbury), he apparently didn’t both testing FoundationDB because their test suite is “waaaay more rigorous”: https://twitter.com/aphyr/status/405017101804396546. In particular their test suite is able to produce reproducible tests of any variation in messaging (dropped messages, reordered messages) across their single-threaded nodes, which is extremely useful in narrowing down places where serialization violations can hide. He also seems to believe Spanner’s claims: https://www.youtube.com/watch?v=w_zYYF3-iSo

I’m not sure where this “conventional wisdom” about serializability having an unavoidable large performance hit is coming from; the databases I mentioned are very well known in the distributed systems field and not some random vendors making outlandish claims.

Re: Cold Showers: For when people get too hyped up about things

#218

Post by anonymoushn » Sat, Jul 25, 2020, 9:15 PM UTC

dahauns wrote: post #210

anonymoushn wrote: post #185

Earlier quoted context omitted.

How many hundreds of LOC would you like to write to support serializing and deserializing JSON for an endpoint that has a schema with around 20 fields, some of which are nested? If you are using Spring and Jackson, you will get to write around 300 LOC across 8 files before you get your hands on a single deserialized object. In any sane language you would use a library that enforces an arbitrary JSON schema to get the…

Why would you write any of this yourself? This is the classic use case for code generation. (And IMO one of the few justified ones.)

It seems like the more common approach among people who use Java is to write the 300 LOC across the 8 files then use the library to generate JSON schema, rather than the other way around. I wrote it myself because I did not want to tell my team that they had been doing things wrong for years before trying their approach once.

Re: Cold Showers: For when people get too hyped up about things

#219

Post by rowanG077 » Sat, Jul 25, 2020, 10:08 PM UTC

nafey wrote: post #136

albntomat0 wrote: post #131

Earlier quoted context omitted.

They are not the same, at least according to the definitions I'm familiar with. Static/dynamic is whether type checking is done at compile time or run type. Strong/weak is how flexible the language is with type conversion. Another explanation: https://en.hexlet.io/courses/intro_to_programming/lessons/ty...

Are these dimensions orthogonal though? For example is there any Strongly typed dynamic language?

Yes. Completely orthogonal.

Strongly typed dynamic: Python

Weakly typed dynamic: Javascript

Strongly typed static: Haskell

Weakly typed static: C

Re: Cold Showers: For when people get too hyped up about things

#220

Post by julienfr112 » Sun, Jul 26, 2020, 9:31 AM UTC

api wrote: post #26

The big data one is outstanding. If you don't have more data than can fit on a reasonably large hard drive, you do not have big data and you are likely able to process it faster and cheaper on one system. Today that threshold would be around 10TiB.

It take hours to read a whole 10TiB hard drive. Depending on use cases, it can be unacceptable.