Stages of denial in encountering K • Hacker News Board

Re: Stages of denial in encountering K

#421

Post by Veedrac » Wed, Mar 11, 2020, 10:07 PM UTC

geocar wrote: post #417

Veedrac wrote: post #416

Earlier quoted context omitted.

Sorry, it's a bit weird not sharing a lexicon. I get the impression Q is a whole different genealogy of programmers. In Python, although you can just use shelve to store data on disk, in practice this is considered a bad idea beyond very simple cases. Valuable data wants the guarantees that real databases provide, like ACID. shelve doesn't provide this, and IIUC nor does kdb+. So if you're handling 50 billion events…

> ... like ACID. shelve doesn't provide this, and IIUC nor does kdb+. So if you're handling 50 billion events a day, live, and you need these to persist, you'd use SQL or something similar. That would then ultimately determine how you add and manipulate records. … ACID is overrated. You can get atomicity, consistency, isolation and durability easily with kdb as I'll illustrate. I appreciate you won't understand every…

I mean, sure, if your problem is such that a strategy like that works for you, I'm not going to tell you otherwise. You can log incoming messages and dump data out to files easily with Python too. I wouldn't want to call that a ‘database’, though, since it's no more than a daily archive.

Re: Stages of denial in encountering K

#422

Post by clarry » Wed, Mar 11, 2020, 10:22 PM UTC

tangentstorm wrote: post #418

at_a_remove wrote: post #339

Earlier quoted context omitted.

Probably not, no. I could easily see myself becoming dejected after printing out a few cheat sheets and buying some relevant books. I would have attempted some exercises and done whatever toy projects with an increasing feeling of dread that I was simply aping what was before me only to find what I thought I had learned had slipped away in a weekend. Should I get much further than that, I would then try to solve an a…

That's a pretty good set of things to visualize if you want to feel defeated before you even get started. I write K for a living, and J for fun. J is WAY crazier, and I learned it specifically because it was so weird and crazy. I kinda wanted to remember what it was like to be a beginner again. In all these languages, you can always fall back to doing things the "old way"... They all pretty much support structured an…

> I write K for a living, and J for fun.

Do you have a clear preference for either? If you could choose J at work, would you? Would you write K for fun?

> (Also for what it's worth, the communities around vector languages do tend to be pretty friendly and welcoming...)

If you've got a site / mailing list / IRC channel in mind, I'd like to know about it.

Re: Stages of denial in encountering K

#423

Post by geocar » Thu, Mar 12, 2020, 12:09 AM UTC

Veedrac wrote: post #421

geocar wrote: post #417

Earlier quoted context omitted.

> ... like ACID. shelve doesn't provide this, and IIUC nor does kdb+. So if you're handling 50 billion events a day, live, and you need these to persist, you'd use SQL or something similar. That would then ultimately determine how you add and manipulate records. … ACID is overrated. You can get atomicity, consistency, isolation and durability easily with kdb as I'll illustrate. I appreciate you won't understand every…

I mean, sure, if your problem is such that a strategy like that works for you, I'm not going to tell you otherwise. You can log incoming messages and dump data out to files easily with Python too. I wouldn't want to call that a ‘database’, though, since it's no more than a daily archive.

Yes! "databases" are all overrated too. Slow expensive pieces of shit. No way you could do 50 billion inserts from Python to SQL server on a single core in a day!

I'm so glad k isn't a "database" like that.

Re: Stages of denial in encountering K

#424

Post by carapace » Thu, Mar 12, 2020, 4:24 AM UTC

Nursie wrote: post #414

carapace wrote: post #409

Earlier quoted context omitted.

I can't agree. (BTW, I don't like K, and I don't use it, but if I did I'd like to think I'd be productive in it.) This little system runs circles around entire sub-industries of other software. The fact that it exists and uses one or two orders of magnitude less time and code than other systems is significant. It's like axes vs. chainsaws. If the job is to log a forest the latter will be better than the former.

Less code is entirely irrelevant unless for some reason you're concerned about a few k of storage for your source, less time appears to be speculation. If people can make working, maintainable software other ways, particularly if those other ways have large, established ecosystems of dependencies, then they are likely doing it right regardless of how much you like apl.

Again, I don't like APL.

Less code -> fewer bugs, all other things equal, eh?

> large, established ecosystems of dependencies

...are actually a symptom of failure to refactor. Ideally, over time, you code base approaches the Kolmogorov complexity of the problems it solves.

> If people can make working, maintainable software other ways ...

...but it takes 10x-100x more code, RAM, CPU power, developer time, etc. than if you hadda used K, then you're leaving money on the table.

Re: Stages of denial in encountering K

#425

Post by Veedrac » Thu, Mar 12, 2020, 7:16 AM UTC

geocar wrote: post #423

Veedrac wrote: post #421

Earlier quoted context omitted.

I mean, sure, if your problem is such that a strategy like that works for you, I'm not going to tell you otherwise. You can log incoming messages and dump data out to files easily with Python too. I wouldn't want to call that a ‘database’, though, since it's no more than a daily archive.

Yes! "databases" are all overrated too. Slow expensive pieces of shit. No way you could do 50 billion inserts from Python to SQL server on a single core in a day! I'm so glad k isn't a "database" like that.

It's a bit unfair to compare the speed of wildly inequivalent things. RocksDB would be more comparable, but even there it is offering much stronger resilience guarantees, multicore support, and gives you access to all your data at once.

Calling them expensive is ironic AF. Most of them are free and open source.

Re: Stages of denial in encountering K

#426

Post by geocar » Thu, Mar 12, 2020, 7:57 AM UTC

Veedrac wrote: post #425

geocar wrote: post #423

Earlier quoted context omitted.

Yes! "databases" are all overrated too. Slow expensive pieces of shit. No way you could do 50 billion inserts from Python to SQL server on a single core in a day! I'm so glad k isn't a "database" like that.

It's a bit unfair to compare the speed of wildly inequivalent things. RocksDB would be more comparable, but even there it is offering much stronger resilience guarantees, multicore support, and gives you access to all your data at once. Calling them expensive is ironic AF. Most of them are free and open source.

> It's a bit unfair to compare the speed of wildly inequivalent things.

Yes, but I understand you keep doing it because you don't understand this stuff very well yet.

> RocksDB would be more comparable

How do you figure that? RocksDB is not a programming language.

If you combine it with a programming language like C++, it can, with only 4x the hardware just about keep up 1/6th of my macbook air[1].

RocksDB might be more comparable to gdbm, but it's not even remotely like q or k.

[1]: And that's taking facebook's benchmarks at their word here, ignoring how utterly synthetic these benchmarks read: https://github.com/facebook/rocksdb/wiki/Performance-Benchma...

> much stronger resilience guarantees,

You're mistaken. There is no resilience guarantee offered by rocksdb. In q I can backup the checkpoints and the logs independently. It is trivial to get whatever level of resilience I want out of q just by copying regular files around. RocksDB requires more programming.

> gives you access to all your data at once

You're mistaken. This is no problem in q. All of the data is mmap'd as soon as I access it (if it isn't mmap'd already).

> Calling them expensive is ironic AF. Most of them are free and open source.

If they require 4x the servers, they're at least 4x as expensive. If it takes 20 days to implement instead of 5 minutes, then it's over 5000x as expensive.

No, calling that "free" is what's ironic, and believing it is moronic.

Re: Stages of denial in encountering K

#427

Post by Veedrac » Thu, Mar 12, 2020, 8:20 AM UTC

geocar wrote: post #426

Veedrac wrote: post #425

Earlier quoted context omitted.

It's a bit unfair to compare the speed of wildly inequivalent things. RocksDB would be more comparable, but even there it is offering much stronger resilience guarantees, multicore support, and gives you access to all your data at once. Calling them expensive is ironic AF. Most of them are free and open source.

> It's a bit unfair to compare the speed of wildly inequivalent things. Yes, but I understand you keep doing it because you don't understand this stuff very well yet. > RocksDB would be more comparable How do you figure that? RocksDB is not a programming language. If you combine it with a programming language like C++, it can, with only 4x the hardware just about keep up 1/6th of my macbook air[1]. RocksDB might be m…

> How do you figure that? RocksDB is not a programming language.

I'm comparing to the code you showed. You're using the file system to dump static rows of data. All your data munging is on memory-sized blocks at program-level. Key-value stores are the comparable database for that.

> You're mistaken. This is no problem in q. All of the data is mmap'd as soon as I access it (if it isn't mmap'd already).

Yes, because you're working on the tail end of small, immutable data tables, rather than an actually sizable database with elements of heterogeneous sizes.

> In q I can backup the checkpoints and the logs independently. It is trivial to get whatever level of resilience I want out of q just by copying regular files around.

Yes, because you don't want much resilience.

---

What you're doing here is incredibly simplistic. It's not proper resiliency, it's not scalable to more complex problems, and it's not scalable to larger workloads. An mmap'ed table and an actual database are different things.

It works fine for you, but for many other people it's not.

Re: Stages of denial in encountering K

#428

Post by geocar » Thu, Mar 12, 2020, 1:22 PM UTC

Veedrac wrote: post #427

geocar wrote: post #426

Earlier quoted context omitted.

> It's a bit unfair to compare the speed of wildly inequivalent things. Yes, but I understand you keep doing it because you don't understand this stuff very well yet. > RocksDB would be more comparable How do you figure that? RocksDB is not a programming language. If you combine it with a programming language like C++, it can, with only 4x the hardware just about keep up 1/6th of my macbook air[1]. RocksDB might be m…

> How do you figure that? RocksDB is not a programming language. I'm comparing to the code you showed. You're using the file system to dump static rows of data. All your data munging is on memory-sized blocks at program-level. Key-value stores are the comparable database for that. > You're mistaken. This is no problem in q. All of the data is mmap'd as soon as I access it (if it isn't mmap'd already). Yes, because yo…

> You're using the file system to dump static rows of data

That's what MySQL, PostgreSQL, SQL Server, and Oracle all do. They write to a logfile (called the "write ahead log") then periodically (and concurrently) process it into working sets that are checkpointed (checked) in much the same way. It's a lot slower because they don't know what is actually important in the statement except what they can deduce from analysis. Whilst that analysis is slow, they do this so that structural concerns can be handed off to a data expert (often called a DBA), since most programmers have no fucking clue how to work with data.

That can work for small data, but it doesn't scale past around the 5bn inserts/day mark currently, without some very special processing strategies, and even then, you don't get close to 50bn.

> All your data munging is on memory-sized blocks at program-level.

That is literally all a computer can do. If you think otherwise, I think you need a more remedial education than the one I've been providing.

> What you're doing here is incredibly simplistic. It's not proper resiliency, it's not scalable to more complex problems, and it's not scalable to larger workloads. An mmap'ed table and an actual database are different things.

Yes, except for everything you said, nothing you said is true in the way that you meant it.

Google.com does not query a "database" that looks any different from the one I'm describing; Bigtable was based on Arthur's work. So was Apache's Kafaka and Amazon's Kinesis. Stream processing is undoubtedly the future, but it started here:

https://pdfs.semanticscholar.org/15ec/7dd999f291a38b3e7f455f...

Not only does this strategy get used for some of the hardest problems and biggest workloads, it's quite possibly the only strategy that can be used for some of these problems.

Resiliency? Simplistic? I'm not even sure you know what those words mean. Putting "proper" in front of it is just weasel words...

Re: Stages of denial in encountering K

#429

Post by Veedrac » Thu, Mar 12, 2020, 1:41 PM UTC

geocar wrote: post #428

Veedrac wrote: post #427

Earlier quoted context omitted.

> How do you figure that? RocksDB is not a programming language. I'm comparing to the code you showed. You're using the file system to dump static rows of data. All your data munging is on memory-sized blocks at program-level. Key-value stores are the comparable database for that. > You're mistaken. This is no problem in q. All of the data is mmap'd as soon as I access it (if it isn't mmap'd already). Yes, because yo…

> You're using the file system to dump static rows of data That's what MySQL, PostgreSQL, SQL Server, and Oracle all do. They write to a logfile (called the "write ahead log") then periodically (and concurrently) process it into working sets that are checkpointed (checked) in much the same way. It's a lot slower because they don't know what is actually important in the statement except what they can deduce from analy…

Ight imma head out, this is clearly not a good direction for the discussion.

Re: Stages of denial in encountering K

#430

Post by tluyben2 » Thu, Mar 12, 2020, 4:32 PM UTC

goto11 wrote: post #294

tluyben2 wrote: post #260

Earlier quoted context omitted.

Matter of taste; I cannot imagine having to program Java (give me Clojure/Scala any time if jvm is required) for a living anymore; deciphering a lovely 200000 class codebase where every call in every 30 lines bring you into a deep spelunking trying to figure out where/how/what and hoping it's not in some ancient undocumented .jar etc. So I have very much the opposite of what you have. Then again, I have been doing k/…

> the brilliant imagination of people who like to use design patterns 'just a bit' wrong for everything. I assume you are comparing this to deciphering and maintaining K-code written by developers who uses all the idioms just a bit wrong?

Well that would be an interesting test; it is not that easy to do that while it is absolutely trivial (I would say run-of-the-mill) to abuse design patterns.