Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arthur Whitney releases an open-source subset of K with MIT license

shakti.com

Post reply on HN
Print view View original

141–150 of 234 posts

Re: Arthur Whitney releases an open-source subset of K with MIT license

#141

Post by dzaima »

Qem wrote: post #135
doug-moen wrote: post #129

FYI: This only implements a subset of K (I'd estimate 1/3). Calling it a 'release' is an overstatement. The docs state that it is a work in progress. It's also quite buggy (it's easy to get a segmentation fault). The version I saw in January was about 1/3 the size of this version, and also buggy. I hope that the final version of this code is less buggy and more usable. If you want to learn the K language, don't use t…

> Any of the other open source K projects are better than this (more complete, less buggy, better documented). One thing that puzzles me, about array languages, is that despite several open source implementations already existing, like J, its surprisingly difficult to find them packaged in Linux repositories. For example, you can't just "apt install J", or "apt install gnu-apl" on Ubuntu. In J case, it seems the defa…

You can 'apt install apl' for GNU APL. Most open-source array languages though either have very few users, and/or are moving quite fast and thus an apt-packaged version would likely be rather out-of-date quite basically always. Though, for example, nix has J, BQN, uiua, GNU APL, and Dyalog APL (based on quick searches), so the barrier to entry to apt also is presumably rather high.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#142

Post by rscho »

oefrha wrote: post #128
rscho wrote: post #120

Earlier quoted context omitted.

For using a heap of libs to compute something that has been done a thousand times? No. For writing a completely new data processing pipeline from scratch? Much, much faster! Array langs have decent performance, but that's not why they are used. They are used because you develop faster for the kind of problem they're good at. People always conflate those two aspects. I use J for scientific research.

Seems weird to switch to develop faster and complain about people conflating the two aspects when this thread is clearly talking about runtime performance, triggered by the benchmark claims: > real-sql(k) is consistently 100 times faster (or more) than redshift, bigquery, snowflake, spark, mongodb, postgres, .. > same data. same queries. same hardware. anyone can run the scripts.

I'm not switching anything. Just trying to add to the conversation. BTW, for simpler queries I have no doubt the benchmarks are correct. I anticipate it would not hold for more beefy queries.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#145

Post by freedomben »

oefrha wrote: post #128
rscho wrote: post #120

Earlier quoted context omitted.

For using a heap of libs to compute something that has been done a thousand times? No. For writing a completely new data processing pipeline from scratch? Much, much faster! Array langs have decent performance, but that's not why they are used. They are used because you develop faster for the kind of problem they're good at. People always conflate those two aspects. I use J for scientific research.

Seems weird to switch to develop faster and complain about people conflating the two aspects when this thread is clearly talking about runtime performance, triggered by the benchmark claims: > real-sql(k) is consistently 100 times faster (or more) than redshift, bigquery, snowflake, spark, mongodb, postgres, .. > same data. same queries. same hardware. anyone can run the scripts.

> Seems weird to switch to develop faster and complain about people conflating the two aspects when this thread is clearly talking about runtime performance, triggered by the benchmark claims

It doesn't look to me like GP switched to develop and complained of conflation. The switch happened higher up the thread by wood_spirit, and GP just continued the conversation (and called out the tendency to conflate, without calling out a specific person).

On a meta note, I wish this trend of saying "it seems weird" and then calling out some fallacy or error would die. Fallacies are extremely common and not "weird", and it comes off as extremely condescending.

It happens quite frequently on HN (and surely other places, though I don't regularly patronize those). So to be clear, this isn't critcism levelled at you exclusively. (I even include myself as target for this criticism, as I've used the expression previously on HN as well, before I thought more about it).

Firstly, in this case and in most cases where that expression is used, it's actually weird to call it weird[1]. Fallacies, logic errors, and other mistakes are extremely natural to humans. Even with significant training and effort, we still make those mistakes routinely.

Secondly, it seems like it's often used as a veiled ad hominem or insult. It's entirely superfluous to add. In this case you could have just said "you complained about people conflating the two aspects and then conflated them yourself." (It still wouldn't have been correct as GP didn't conflate them, but it would have been more direct and clear).

Thirdly, it comes off as condescending[2]. It's sort of like saying, "whoa dude, we're all normal and don't make mistakes, but you're weird and do make mistakes." In reality, we all do it so it's not weird at all.

[1]: https://www.merriam-webster.com/dictionary/weird

  1: of strange or extraordinary character : ODD, FANTASTIC
  2: of, relating to, or caused by witchcraft or the supernatural : MAGICAL
[2]: The irony of this is not lost on me. I can definitely see how this comment might also come off as condescending. I don't intend it to be, but it is a ridiculously long comment for such a simple point. It also included a dictionary check which is also frequently a charactersitc of condescending comments. I don't intend it to be condescending, merely reflective of self-analytical, but as is my theme here, we all make mistakes :-)

Re: Arthur Whitney releases an open-source subset of K with MIT license

#146

Post by breck »

bravura wrote: post #131
breck wrote: post #121

Earlier quoted context omitted.

Agreed. Pushing through until you can think in array languages is well worth it! In my experience one of the top 30 highest ROI mental circuits you can develop. That being said, I'm not convinced that the extremely minimal syntax is essential. I think it can be done another way ;)

Can you please share a few of the other 30 highest ROI mental circuits to develop?

Good question.

I should make a ranked list.

In regards to programming, my top 30 would include:

ScrollSets: https://breckyunits.com/scrollsets.html

RegEx.

The dataflow paradigm as popularized by dplyr would be on there.

HIT ranking: https://breckyunits.com/hits.html

Re: Arthur Whitney releases an open-source subset of K with MIT license

#147

Post by tromp »

tasuki wrote: post #140
tromp wrote: post #76

Earlier quoted context omitted.

An interpreter for BLC, including tokenizing, parsing, and evaluation, can be written in as few as 29 bytes of BLC (and 650 bytes of C).

John, do tell more about BLC please!

There's more at http://tromp.github.io/cl/cl.html including the LispNYC talk I gave last year.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#148

Post by philipov »

omnicognate wrote: post #132

I'm going to have to take another run at learning this corner of computing soon, but it's a prospect I'm not relishing. Everything about it rubs me up the wrong way. If you'd like an antidote, have a read of Gerald Jay Sussman's books, where you'll see profound concepts from maths and physics captured in succinct and expressive (as opposed to merely terse) code, accompanied by eloquent explanations devoid of boasts o…

Which one do you recommend?

The first one I found was "Structure and Interpretation of Computer Programs"

https://web.mit.edu/6.001/6.037/sicp.pdf

Re: Arthur Whitney releases an open-source subset of K with MIT license

#149

Post by wood_spirit »

magicalhippo wrote: post #93
anonzzzies wrote: post #73

Earlier quoted context omitted.

> a sign that a programmer hasn't worked on anything substantial Maybe you want to check who he is?

I meant I don't know as in I haven't actually worked with him. Based on what I've seen before as well as the supplied code, I would be very skeptical to have him on my team.

This is a bit of a tangent but it’s a thought experiment that I recently heard:

Data pipeline A is written and maintained by a team in a type safe language with extensive unit tests.

Data pipeline B be was written long ago by a scientist who has since left, in sql in a day.

Both compute the same dataset, but B gets the answer correct.

Which is the better pipeline, and why?

Re: Arthur Whitney releases an open-source subset of K with MIT license

#150

Post by pixelmonkey »

dchest wrote: post #43

Everyone here charges too little for software. the free(MIT license) version is shakti.com/k/k.zip the supported version(supported and 10 to 100 times faster) is $100K per month(minimum) https://groups.google.com/g/shaktidb/c/5SPufca3mo4

k is widely used by a handful of big investment banks and big hedge funds for quant/finance stuff. There are only a handful of such companies in the world but they are extremely price insensitive, especially with regard to technology that lets them get a market edge. I suspect this is the dynamic that kx, the company, tapped into over the years. I also suspect this open source release is mainly because investment banks have come around on their desire for open source (rather than proprietary) software over the years, at least on some teams. You can see the open source release doc explicitly positions k vs Python, pandas, and polars.

For example, I have an old friend from a major investment bank who used to work on an internal (proprietary) pub/sub system but who, these days, works on integrations between that system and Apache Kafka.

Post reply on HN