Cold Showers: For when people get too hyped up about things • Hacker News Board

Re: Cold Showers: For when people get too hyped up about things

#101

Post by unilynx » Fri, Jul 24, 2020, 3:49 PM UTC

castorp wrote: post #86

bob1029 wrote: post #71

Earlier quoted context omitted.

We aren't running any reports on our databases like this. I would argue it is a bad practice in general to mix OLTP and OLAP workloads on a single database instance, regardless of the specific technology involved. If we wanted to run an aggregate that could potentially impact live transactions, we would just copy the SQLite db to another server and perform the analysis there. We have some telemetry services which ope…

> At the end of the day, you still have to go to disk on writes, and this must be serialized against reads for basic consistency reasons. No, absolutely not. That's why modern databases use a thing called multi version concurrency control. You can run (multiple) queries on the same table that is updated by multiple transactions at the same time without one blocking the others (assuming the write transactions don't bl…

I think this is specifically because random reads are scaling much better than writes, even though you won't see it on standard "that many MB/s" benchmarks where read is 'just' a few multiples of the write performance.

Persisting a transaction to the database is still (and especially in MVCC): "send data write". "wait for write to be flushed". "toggle metadata bit to mark write as completed". "wait for bit to be completed" which still serialises transaction commits while reads can complete in parallel as fast as the device can handle.

Especially now that the reads and writes don't have to share the disk head, it makes sense for random reads to keep on scaling better than writes.

Re: Cold Showers: For when people get too hyped up about things

#102

Post by b5n » Fri, Jul 24, 2020, 3:54 PM UTC

castorp wrote: post #86

bob1029 wrote: post #71

Earlier quoted context omitted.

We aren't running any reports on our databases like this. I would argue it is a bad practice in general to mix OLTP and OLAP workloads on a single database instance, regardless of the specific technology involved. If we wanted to run an aggregate that could potentially impact live transactions, we would just copy the SQLite db to another server and perform the analysis there. We have some telemetry services which ope…

> At the end of the day, you still have to go to disk on writes, and this must be serialized against reads for basic consistency reasons. No, absolutely not. That's why modern databases use a thing called multi version concurrency control. You can run (multiple) queries on the same table that is updated by multiple transactions at the same time without one blocking the others (assuming the write transactions don't bl…

> Mixing OLTP and OLAP becomes increasingly "normal"

Just because it's "normal" doesn't mean it's correct. Just because you can doesn't mean you should.

All hail bob1029:

> We aren't running any reports on our databases like this. I would argue it is a bad practice in general to mix OLTP and OLAP workloads on a single database instance, regardless of the specific technology involved.

Re: Cold Showers: For when people get too hyped up about things

#103

Post by tristor » Fri, Jul 24, 2020, 3:56 PM UTC

castorp wrote: post #86

bob1029 wrote: post #71

Earlier quoted context omitted.

We aren't running any reports on our databases like this. I would argue it is a bad practice in general to mix OLTP and OLAP workloads on a single database instance, regardless of the specific technology involved. If we wanted to run an aggregate that could potentially impact live transactions, we would just copy the SQLite db to another server and perform the analysis there. We have some telemetry services which ope…

> At the end of the day, you still have to go to disk on writes, and this must be serialized against reads for basic consistency reasons. No, absolutely not. That's why modern databases use a thing called multi version concurrency control. You can run (multiple) queries on the same table that is updated by multiple transactions at the same time without one blocking the others (assuming the write transactions don't bl…

"serialized" here doesn't really mean processed in serial, it means "serializable" in the context of database information theory. Databases have special concurrency control requirements in order to create hard guarantees on database consistency. You can process queries in parallel and still have a serializable result, because of transaction coordination. Doing this on one server is much easier than doing this across a cluster of servers.

So in your case, MVCC is what you're talking about, which is not the same level of consistency guarantee as serializable, rather it is based on snapshot isolation. Some database vendors consider them effectively the same isolation level because the anomalies associated to other common non-serializable isolation levels aren't typically present in most MVCC implementations, but there's a lot more complexity here than you are acknowledging.

Mixing OLTP and OLAP workloads on the same database is pretty much always a bad idea. This is why it's common practice to use ETL jobs to move data from an OLTP optimized database like Postgres or MySQL to a separate database for OLAP (which could be another MySQL or PG instance, or could be something like ClickHouse or another columnar database optimized for OLAP). Just because you /can/ do something, doesn't mean you /should/ do something...

Re: Cold Showers: For when people get too hyped up about things

#104

Post by fmakunbound » Fri, Jul 24, 2020, 4:03 PM UTC

hwayne wrote: post #90

fmakunbound wrote: post #61

Earlier quoted context omitted.

In addition to that, I also feel calling ourselves engineers is a stretch.

I used to think this, and then I interviewed people who did both traditional and software engineering professionally, and now I'm not so sure. I did a first draft of what I learned here: https://www.youtube.com/watch?v=3018ABlET1Y I'm hoping to have a written version by the end of September.

Fair enough! You make some great points there which changed my perspective.

Re: Cold Showers: For when people get too hyped up about things

#105

Post by ozim » Fri, Jul 24, 2020, 4:09 PM UTC

hwayne wrote: post #90

fmakunbound wrote: post #61

Earlier quoted context omitted.

In addition to that, I also feel calling ourselves engineers is a stretch.

I used to think this, and then I interviewed people who did both traditional and software engineering professionally, and now I'm not so sure. I did a first draft of what I learned here: https://www.youtube.com/watch?v=3018ABlET1Y I'm hoping to have a written version by the end of September.

This is great. People romanticize construction, mechanical and other engineering like there would be no failures in those disciplines. Buildings collapse, machines break down in unforeseen circumstances. My pet theory is that in software it is just a lot easier to create a lot of stuff, so it is also a lot easier to create issues.

You can add that in eastern Europe you can get engineering degree which is "technical bachelor" from technical university, so I am software engineer as it is printed in my diploma.

Re: Cold Showers: For when people get too hyped up about things

#106

Post by nicoburns » Fri, Jul 24, 2020, 4:21 PM UTC

bob1029 wrote: post #79

saberience wrote: post #76

Earlier quoted context omitted.

Are you living on another planet? Are you seriously suggesting people replace SqlServer or MySQL with SQLite? If you can come to my company and replace our 96-core SqlServer boxes with SQLite I'll pay you any salary you ask for.

I can assure you that I live on the same planet as everyone else posting here. Whether or not I could perform this miracle depends entirely on your specific use cases. Many people who have this sort of reaction are coming from a place where there is heavy use of the vendor lock-in features such as SSIS and stored procedures. If you are ultimately just trying to get structured business data to/from disk in a consisten…

Is SQLite likely to be faster than postgres? In terms of ease of use / admin overhead I consider them mostly equivalent. I thought the main problem with SQLite was it was slow tih concurrent writers. Whereas the "bigger" SQL databases have code that allows concurrent writes.

Re: Cold Showers: For when people get too hyped up about things

#107

Post by orangecat » Fri, Jul 24, 2020, 4:22 PM UTC

GlennS wrote: post #23

quonn wrote: post #10

I wish it would be possible to have better studies for that. I believe that static typing has huge benefits as software scales. I also believe that the type system of TypeScript is actually stronger in practice than the Java or C# one (despite theoretical weaknesses). It has the right tradeoffs (e.g. structural equivalence, being able to type strings, being able to check that all cases are handled, etc.) It would be…

People who like static typing seem to really like static typing. I'm honestly not convinced it helps that much. And it seems to cost a lot to me. I like database and API schemas though. And I like clojure.spec and function preconditions a lot.

In my experience both the benefits and costs of static typing are overstated. It doesn't mean that your code works if it compiles, and it doesn't mean that you can get rid of half your tests. But it's great for refactoring, and it's a useful form of documentation that can't lie, unlike comments. And if you're using a reasonable language it's not much additional work to add types; often with type inference it's zero work.

Re: Cold Showers: For when people get too hyped up about things

#108

Post by Noos » Fri, Jul 24, 2020, 4:24 PM UTC

gonzo41 wrote: post #8

moby_click wrote: post #5

There should be a section about the benefits of cold showers.

Is taking cold showers like a machismo SV thing?

No, it is more of a euphemism for "sobering up." Existed for quite awhile, usually used for drunks though. Has minor connotations of being an oddball health nut-people extolling the benefits of being immersed into chilly water.

Re: Cold Showers: For when people get too hyped up about things

#109

Post by nicoburns » Fri, Jul 24, 2020, 4:26 PM UTC

GlennS wrote: post #23

quonn wrote: post #10

I wish it would be possible to have better studies for that. I believe that static typing has huge benefits as software scales. I also believe that the type system of TypeScript is actually stronger in practice than the Java or C# one (despite theoretical weaknesses). It has the right tradeoffs (e.g. structural equivalence, being able to type strings, being able to check that all cases are handled, etc.) It would be…

People who like static typing seem to really like static typing. I'm honestly not convinced it helps that much. And it seems to cost a lot to me. I like database and API schemas though. And I like clojure.spec and function preconditions a lot.

I hated statix typing until I used Rust. Rust has Sum types (super powered enums), which provide what I was missing from dynamically typed langauges in languages like Java, C#, etc: namely the ability to have an "or" type (e.g. this is an integer or a string, and I want to be able to branch on that at runtime).

That, plus type inference makes the static typing pretty painless.

Re: Cold Showers: For when people get too hyped up about things

#110

Post by rwoerz » Fri, Jul 24, 2020, 4:29 PM UTC

ozim wrote: post #105

hwayne wrote: post #90

Earlier quoted context omitted.

I used to think this, and then I interviewed people who did both traditional and software engineering professionally, and now I'm not so sure. I did a first draft of what I learned here: https://www.youtube.com/watch?v=3018ABlET1Y I'm hoping to have a written version by the end of September.

This is great. People romanticize construction, mechanical and other engineering like there would be no failures in those disciplines. Buildings collapse, machines break down in unforeseen circumstances. My pet theory is that in software it is just a lot easier to create a lot of stuff, so it is also a lot easier to create issues. You can add that in eastern Europe you can get engineering degree which is "technical b…

Yes, software is immaterial and thus not constrained by laws of physic (except speed of light). It is comparably easy to change but also comparably hard to specify and model in advance.