Cold Showers: For when people get too hyped up about things • Hacker News Board

Re: Cold Showers: For when people get too hyped up about things

#71

Post by bob1029 » Fri, Jul 24, 2020, 2:35 PM UTC

hans_castorp wrote: post #45

bob1029 wrote: post #42

Earlier quoted context omitted.

SQLite is incredible. If you are struggling to beat the "one query per second" meme, try the following 2 things: 1. Only use a single connection for all access. Open the database one time at startup. SQLite operates in serialized mode by default, so the only time you need to lock is when you are trying to obtain the LastInsertRowId or perform explicit transactions across multiple rows. Trying to use the one connectio…

> If at this point you are still finding that SQLite is not as fast or faster than SQL Server, MySQL, et.al., then I would be very surprised. What about large aggregation queries, that are parallelized by modern DBMS? Does it still scale that well if you have many concurrent read and write transactions (e.g. on the same table)?

We aren't running any reports on our databases like this. I would argue it is a bad practice in general to mix OLTP and OLAP workloads on a single database instance, regardless of the specific technology involved.

If we wanted to run an aggregate that could potentially impact live transactions, we would just copy the SQLite db to another server and perform the analysis there. We have some telemetry services which operate in this fashion. They go out to all of the SQLite databases, make a copy and then run analysis in another process (or on another machine).

I am not aware of any hosted SQL technology which is capable of magically interleaving large aggregate queries with live transactions and not having one or both impacted in some way. At the end of the day, you still have to go to disk on writes, and this must be serialized against reads for basic consistency reasons. After a certain point, this is kinda like trying to beat basic information theory with ever-more-complex compression schemes. I'd rather just accept the fundamental truth of the hardware/OS and have the least amount of overhead possible when engaging with it.

Re: Cold Showers: For when people get too hyped up about things

#72

Post by sgt101 » Fri, Jul 24, 2020, 2:35 PM UTC

Yet again the "we don't need big data" demoed on an example that fits on a disk. Big data is north of 30TB.

Re: Cold Showers: For when people get too hyped up about things

#73

Post by wenc » Fri, Jul 24, 2020, 2:36 PM UTC

Taikonerd wrote: post #58

I read this and thought, "oh, the author is calling out formal verification as overhyped? Hillel Wayne ( https://hillelwayne.com/ ) is going to be angry! Wait, who wrote this..."

Me too.

I recall that F Scott Fitzgerald said "the test of a first-rate intelligence is the ability to hold two opposed ideas in mind at the same time and still retain the ability to function."

Re: Cold Showers: For when people get too hyped up about things

#74

Post by milquetoastaf » Fri, Jul 24, 2020, 2:40 PM UTC

fmakunbound wrote: post #61

rwoerz wrote: post #28

We software engineers are still more like alchimists rather than chemists. That list reminds me of [1], which rants about this state of affairs and [2] that puts many beliefs to the test. [1] https://youtu.be/WELBnE33dpY [2] https://www.oreilly.com/library/view/making-software/9780596...

In addition to that, I also feel calling ourselves engineers is a stretch.

Agreed. If we were actually engineers we'd all be writing TLA+ and Coq to verify the correctness of what we do and there'd be a formal cert program codified by the state

Re: Cold Showers: For when people get too hyped up about things

#75

Post by jdm2212 » Fri, Jul 24, 2020, 2:45 PM UTC

j_m_b wrote: post #52

whatl3y wrote: post #19

Earlier quoted context omitted.

I've recently learned and ported a few projects from Javascript to TypeScript and I'll vouch that, so far, it is much better and easier to reason about my code and what it's doing. I also feel I need less test cases to adequately test my code. In saying that, I'm interested in if there is any accepted, peer reviewed literature with quantifiable data as to whether strongly typed languages are "better" (whatever the st…

How much would you attribute that to the fact that you rewrote the code? I know my second pass is always better.

JS -> TS is usually just a matter of adding types, so I think by "ported" the previous commenter just meant adding types and tweaking as needed to make the type system happy.

Re: Cold Showers: For when people get too hyped up about things

#76

Post by saberience » Fri, Jul 24, 2020, 2:49 PM UTC

bob1029 wrote: post #42

Multicomp wrote: post #16

I love that sqlite article. It seems like "everyone" is certain that sqlite can only be used for up to a single query per second, anything more and you need to spin up a triple sharded postgres or Hadoop cluster because it 'needs to scale'. I love being able to show that study, if you properly architect your sqlite system and am willing to purchase hardware, you can go a long long way, much further than almost all co…

SQLite is incredible. If you are struggling to beat the "one query per second" meme, try the following 2 things: 1. Only use a single connection for all access. Open the database one time at startup. SQLite operates in serialized mode by default, so the only time you need to lock is when you are trying to obtain the LastInsertRowId or perform explicit transactions across multiple rows. Trying to use the one connectio…

Are you living on another planet? Are you seriously suggesting people replace SqlServer or MySQL with SQLite?

If you can come to my company and replace our 96-core SqlServer boxes with SQLite I'll pay you any salary you ask for.

Re: Cold Showers: For when people get too hyped up about things

#77

Post by jdm2212 » Fri, Jul 24, 2020, 2:51 PM UTC

quonn wrote: post #10

I wish it would be possible to have better studies for that. I believe that static typing has huge benefits as software scales. I also believe that the type system of TypeScript is actually stronger in practice than the Java or C# one (despite theoretical weaknesses). It has the right tradeoffs (e.g. structural equivalence, being able to type strings, being able to check that all cases are handled, etc.) It would be…

The study IMHO is reality. Every large, actively maintained system I've ever heard of uses a strongly typed language, or was written in a dynamically typed language but has converted to a gradually typed language. You can't safely refactor without compile time type checking, and you can't maintain a non-trivial system over the long-term if you can't refactor it.

Re: Cold Showers: For when people get too hyped up about things

#78

Post by red_admiral » Fri, Jul 24, 2020, 2:54 PM UTC

First link (formal verification) currently gives a 404 for me. Someone not wanting the extra publicity?

Re: Cold Showers: For when people get too hyped up about things

#79

Post by bob1029 » Fri, Jul 24, 2020, 2:56 PM UTC

saberience wrote: post #76

bob1029 wrote: post #42

Earlier quoted context omitted.

SQLite is incredible. If you are struggling to beat the "one query per second" meme, try the following 2 things: 1. Only use a single connection for all access. Open the database one time at startup. SQLite operates in serialized mode by default, so the only time you need to lock is when you are trying to obtain the LastInsertRowId or perform explicit transactions across multiple rows. Trying to use the one connectio…

Are you living on another planet? Are you seriously suggesting people replace SqlServer or MySQL with SQLite? If you can come to my company and replace our 96-core SqlServer boxes with SQLite I'll pay you any salary you ask for.

I can assure you that I live on the same planet as everyone else posting here.

Whether or not I could perform this miracle depends entirely on your specific use cases. Many people who have this sort of reaction are coming from a place where there is heavy use of the vendor lock-in features such as SSIS and stored procedures.

If you are ultimately just trying to get structured business data to/from disk in a consistent manner and are seeking the lowest latency and highest throughput per request, then SQLite might be what you are looking for.

The specific core counts or other specifications are meaningless. SQLite scales perfectly on a single box, and if you have some good engineers you might even be able to build a clustering protocol at the application layer in order to tie multiple together. At a certain point, writing your own will get cheaper than paying Microsoft for the privilege of using SQL Server.

Re: Cold Showers: For when people get too hyped up about things

#80

Post by therufa » Fri, Jul 24, 2020, 2:57 PM UTC

quonn wrote: post #10

I wish it would be possible to have better studies for that. I believe that static typing has huge benefits as software scales. I also believe that the type system of TypeScript is actually stronger in practice than the Java or C# one (despite theoretical weaknesses). It has the right tradeoffs (e.g. structural equivalence, being able to type strings, being able to check that all cases are handled, etc.) It would be…

In my personal opinion (based on personal observation) static typing helps to become more lazy and trusting since it enables features such as autocomplete, type hinting and so on where one basically gives away understanding of detail. Don't get me wrong! I love being lazy and trusting because it allows to leverage more code than I'd be able to produce on my own, but usually it's also the source of many of my own mistakes and failures. I strongly believe that I'm not the only one.