Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arthur Whitney releases an open-source subset of K with MIT license

shakti.com

Post reply on HN
Print view View original

81–90 of 234 posts

Re: Arthur Whitney releases an open-source subset of K with MIT license

#81

Post by eigenvalue »

eggy wrote: post #62
eigenvalue wrote: post #28

This language is very popular among quant finance people associated with Morgan Stanley. I don’t see the appeal myself. Maybe it helps prevent people stealing the code since it’s so awful looking to work with! At one point I had to learn it and I think I’ve totally forgotten it now— it’s like my brain repressed it. Not my cup of tea, that’s for sure.

Most people I know who actually learn an array language like k or j usually grow to appreciate the expressiveness and cleverness of these languages. Typically, people have your reaction who have only looked at it and tried it very briefly. I'm surprised. Why did you have to learn it? Where?

Was working at a quant pod at Millennium for a bit where they used it. I was ultimately able to use it but everything took me 20x longer than using Numpy/Pandas. The irony was that the Python code was shorter because there were so many more library functions and better abstractions and syntax. So it was slow and unintuitive for zero benefit whatsoever.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#82

Post by NKosmatos »

He might be a smart person, with a very high IQ and on a different level than the rest of us, but by writing with this style, with no comments, with no proper capitalization/style and with this attitude, he’s putting me off (IMHO).

Oftentimes, the way something is presented and how the language is used, might be as important as the thing itself ;-)

Re: Arthur Whitney releases an open-source subset of K with MIT license

#83

Post by mlochbaum »

JonChesterfield wrote: post #77
stefano wrote: post #61

Those are some very big claims with respect to performance. Has anyone outside of the author been able to reproduce the claims, considering you need to pay 100k/month just to do it? I also wonder if the commercial version has anti-benchmark clauses like some database vendors. I've always seen claims that K is much faster than anything else out there, but I've never seen an actual independent benchmark with numbers. E…

Array languages are very fast for code that fits sensibly into arrays and databases spend a lot of compute time getting correctness right. 100x faster than postgres on arbitrary data sounds unlikely-to-impossible but on specific problems might be doable.

Yes, when I took a look at shakti's database benchmarks before, they seemed entirely reasonable with typical array language implementation methods. I even compared shakti's sum-by benchmarks to BQN group followed by sum-each, which I'd expect to be much slower than a dedicated implementation, and it was around the same order of magnitude (like 3x slower or something) when accounting for multiple cores. I was surprised that something like Polars would do it so slowly, but that's what the h2o benchmark said... I guess they just don't have a dedicated sum-by implementation for each type. I think shakti may have less of an advantage with more complicated queries, and doing the easy stuff blazing fast is nice but they're probably only a small fraction of the typical database workload anyway.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#84

Post by secondary_op »

And I though that Skala, FP (monad in X is just a monoid in the category of endofunctors of X) people are pretentious sect, this is so much worse, given their customer base, I immediately presume that this is nothing more than approach to deliberately make things unintelligible for as many people as possible, so that your white collar hedge fund guy would have heart stroke just by glancing at this source code, not even trying to read or understand it, that is one way to treat people and make business, it is despicable. Industry needs to formalise this into well known phenomena much like Security through obscurity [1] so that kind hearted pragmatic people avoid this like a plague.

[1] https://en.wikipedia.org/wiki/Security_through_obscurity

Re: Arthur Whitney releases an open-source subset of K with MIT license

#85

Post by wood_spirit »

stefano wrote: post #61

Those are some very big claims with respect to performance. Has anyone outside of the author been able to reproduce the claims, considering you need to pay 100k/month just to do it? I also wonder if the commercial version has anti-benchmark clauses like some database vendors. I've always seen claims that K is much faster than anything else out there, but I've never seen an actual independent benchmark with numbers. E…

Quick for building a website? Probably not.

Quick for evaluating some idea you just had if you are a quant? Yes absolutely!

So imagine you have a massive dataset, and an idea.

For testing out your idea you want that data to be in an “online analytical processing” (OLAP) kind of database. These typically store the data by column not row and other tricks to speed up crunching through reads, trading off write performance etc.

There are several big tech choices you could make. Mainstream king is SQL.

Something that was trendy a few years ago in the nosql revolution was to write some scala at the repl.

It is these that K is competing with, and being faster than.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#86

Post by Dudhbbh3343 »

stefano wrote: post #61

Those are some very big claims with respect to performance. Has anyone outside of the author been able to reproduce the claims, considering you need to pay 100k/month just to do it? I also wonder if the commercial version has anti-benchmark clauses like some database vendors. I've always seen claims that K is much faster than anything else out there, but I've never seen an actual independent benchmark with numbers. E…

They at least used to have free evaluation licenses that were good for a month. Our license was even unlocked for unlimited cores.

I doubt they'd give them out to a random individual or small startup, but maybe still possible for a serious potential customer.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#87

Post by JonChesterfield »

sph wrote: post #52
PufPufPuf wrote: post #40

The website reads like an edgy script-kiddy blog. Is K actually a useful project, or is it just a passion project of someone who happens to be sort of famous?

> The website reads like an edgy script-kiddy blog. The code does, as well. Either Mr. Whitney's brain is not wired like a regular homo sapiens sapiens, or the entire thing smells of "I am smarted than you and I don't need to lower myself to your level." I do not buy for a single second that for Mr. Whitney debugging IOCCC-level obfuscated code is easier than plain C code. One writes "normal code" because one will ha…

I worked with a guy who could handle loops nested half a dozen deep with data dependencies woven through the structure with exactly the same apparent cognitive overhead as for i = 0, N. The sort of structure you get when you arrange a difficult calculation to match the cache hierarchy of the target machine. Didn't do comments or variable names with much enthusiasm.

He was superb at finding errors at code review. As in looking through code someone else had written and pulling out the mistakes. Presumably everything looked completely trivial to him, regardless of how tangled the control flow had got.

Whitney may be similar.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#88

Post by defrost »

magicalhippo wrote: post #59
rwillink wrote: post #17

He showed K @ Royal Society & bunch of apl & aplus guys were there. Someone asked , where are the comments? AW said comments get out of date with all the changes, if you can’t read the code you Shldnt be working on it. We then all looked at each other..

> if you can’t read the code you Shldnt be working on it I don't know this AW guy, but to me that's a huge red flag and a sign that a programmer hasn't worked on anything substantial. Ie non-trivial stuff that's maintained by a team over time. Being able to read the code is irrelevant, as the comments should tell you why the code is doing what it's doing. For example, yeah I trivially can see the code is doing a retr…

for reference: https://en.wikipedia.org/wiki/Arthur_Whitney_(computer_scien...

Re: Arthur Whitney releases an open-source subset of K with MIT license

#89

Post by raverbashing »

wood_spirit wrote: post #71
eigenvalue wrote: post #28

This language is very popular among quant finance people associated with Morgan Stanley. I don’t see the appeal myself. Maybe it helps prevent people stealing the code since it’s so awful looking to work with! At one point I had to learn it and I think I’ve totally forgotten it now— it’s like my brain repressed it. Not my cup of tea, that’s for sure.

Thinking about problems and data manipulation in the way array languages enable is hard but, once you have pushed through the what feels like a barrier of mainstream programming language thinking that is stopping you from “grokking” it, it is a sudden moment of clarity and then you “get it”. Perhaps a half way house is sql. The difference between ORM-style CRUD and a power user using window functions to make the data…

But tbh it looks worse than matlab

Re: Arthur Whitney releases an open-source subset of K with MIT license

#90

Post by wood_spirit »

eigenvalue wrote: post #81
eggy wrote: post #62

Earlier quoted context omitted.

Most people I know who actually learn an array language like k or j usually grow to appreciate the expressiveness and cleverness of these languages. Typically, people have your reaction who have only looked at it and tried it very briefly. I'm surprised. Why did you have to learn it? Where?

Was working at a quant pod at Millennium for a bit where they used it. I was ultimately able to use it but everything took me 20x longer than using Numpy/Pandas. The irony was that the Python code was shorter because there were so many more library functions and better abstractions and syntax. So it was slow and unintuitive for zero benefit whatsoever.

But how did your perf compare to the best of the K kicking quants around you? Were they too being less productive than they would have been in python?

I’m not saying they were right or better. Horses of courses. Array languages do my head in and my choice is sql.

Post reply on HN