Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arthur Whitney releases an open-source subset of K with MIT license

shakti.com

Post reply on HN
Print view View original

151–160 of 234 posts

Re: Arthur Whitney releases an open-source subset of K with MIT license

#151

Post by freedomben »

rscho wrote: post #142
oefrha wrote: post #128

Earlier quoted context omitted.

Seems weird to switch to develop faster and complain about people conflating the two aspects when this thread is clearly talking about runtime performance, triggered by the benchmark claims: > real-sql(k) is consistently 100 times faster (or more) than redshift, bigquery, snowflake, spark, mongodb, postgres, .. > same data. same queries. same hardware. anyone can run the scripts.

I'm not switching anything. Just trying to add to the conversation. BTW, for simpler queries I have no doubt the benchmarks are correct. I anticipate it would not hold for more beefy queries.

You came by it honestly! The initial conflation (and therefore context switch) happened further up-thread.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#152

Post by omnicognate »

philipov wrote: post #148
omnicognate wrote: post #132

I'm going to have to take another run at learning this corner of computing soon, but it's a prospect I'm not relishing. Everything about it rubs me up the wrong way. If you'd like an antidote, have a read of Gerald Jay Sussman's books, where you'll see profound concepts from maths and physics captured in succinct and expressive (as opposed to merely terse) code, accompanied by eloquent explanations devoid of boasts o…

Which one do you recommend? The first one I found was "Structure and Interpretation of Computer Programs" https://web.mit.edu/6.001/6.037/sicp.pdf

That's a good place to start but it's primarily about programming itself. In "Structure and Interpretation of Classical Mechanics" and "Functional Differential Geometry" he applies his approach of using computer programs as a way of communicating concepts to humans to some fascinating maths and physics topics.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#153

Post by deusum »

rscho wrote: post #117
PufPufPuf wrote: post #40

The website reads like an edgy script-kiddy blog. Is K actually a useful project, or is it just a passion project of someone who happens to be sort of famous?

Oh, it's just racking millions of dollars from big bank users every year. Nothing a script-kiddy couldn't achieve...

Script kiddies bust their butts phishing and installing black market ransomware. This Whitney fellow is probably sitting in his office somewhere expecting people to just throw $100k (per month!) at him. ;-)

Re: Arthur Whitney releases an open-source subset of K with MIT license

#154

Post by anonu »

Chirono wrote: post #96
stefano wrote: post #61

Those are some very big claims with respect to performance. Has anyone outside of the author been able to reproduce the claims, considering you need to pay 100k/month just to do it? I also wonder if the commercial version has anti-benchmark clauses like some database vendors. I've always seen claims that K is much faster than anything else out there, but I've never seen an actual independent benchmark with numbers. E…

I used to use K professionally inside a hedge fund a few years back. Aside from the terrible user experience (if your code isn’t correct you will often just get ‘error’ or ‘not implemented’ with no further detail), if the performance really was as stellar as claimed, then there wouldn’t need to be a no benchmark clause in the license. It can be fast, if your data is in the right formats, but not crazy fast. And easy…

I've been using kdb/q since 2010. Started at a big bank and have used it ever since.

Kdb/q is like minimalist footwear. But you can run longer and faster with it on. There's a tipping point where you just "get it". It's a fantastic language and platform.

The problem is very few people will pay 100k/month for shakti. I'm not saying people won't pay and it won't be a good business. But if you want widespread adoption you need to create and an ecosystem. Open sourcing it is a start. Creating libraries and packages comes after. The mongodb model is the right approach IMO

Re: Arthur Whitney releases an open-source subset of K with MIT license

#156

Post by rscho »

tiffanyh wrote: post #106
Chirono wrote: post #96

Earlier quoted context omitted.

I used to use K professionally inside a hedge fund a few years back. Aside from the terrible user experience (if your code isn’t correct you will often just get ‘error’ or ‘not implemented’ with no further detail), if the performance really was as stellar as claimed, then there wouldn’t need to be a no benchmark clause in the license. It can be fast, if your data is in the right formats, but not crazy fast. And easy…

Would you recommend K? Is something else better (if so what)?

Are you a quant, or doing very specialized numerical things that aren't in libs on 2D datasets? Then 10X yes. If not, no. Everything else will be better

Re: Arthur Whitney releases an open-source subset of K with MIT license

#157

Post by tresclow »

wood_spirit wrote: post #149
magicalhippo wrote: post #93

Earlier quoted context omitted.

I meant I don't know as in I haven't actually worked with him. Based on what I've seen before as well as the supplied code, I would be very skeptical to have him on my team.

This is a bit of a tangent but it’s a thought experiment that I recently heard: Data pipeline A is written and maintained by a team in a type safe language with extensive unit tests. Data pipeline B be was written long ago by a scientist who has since left, in sql in a day. Both compute the same dataset, but B gets the answer correct. Which is the better pipeline, and why?

And how do you know B's answer is correct?

Re: Arthur Whitney releases an open-source subset of K with MIT license

#158

Post by chongli »

tiffanyh wrote: post #106
Chirono wrote: post #96

Earlier quoted context omitted.

I used to use K professionally inside a hedge fund a few years back. Aside from the terrible user experience (if your code isn’t correct you will often just get ‘error’ or ‘not implemented’ with no further detail), if the performance really was as stellar as claimed, then there wouldn’t need to be a no benchmark clause in the license. It can be fast, if your data is in the right formats, but not crazy fast. And easy…

Would you recommend K? Is something else better (if so what)?

I think the main reason to use any of these array languages (for work) is job security. Since it's so hard to find expert programmers if you can get your employer to sign off on an array language for all the mission-critical stuff then you can lock in your job for life! How can they possibly replace you if they can't find anyone else who understands the code?

Otherwise, I don't see anything you can do in an array language that you couldn't do in any other language, albeit less verbosely. But I believe in this case a certain amount of verbosity is a feature if you want people to be able to read and understand the code. Array languages and their symbol salad programs are like the modern day equivalent of medieval alchemists writing all their lab notes in a bespoke substitution cipher. Not unbreakable (like modern cryptography) but a significant enough barrier to dissuade all but the most determined investigators.

As an aside, I think the main reason these languages took off among quants is that investing as an industry tends toward the exultation of extremely talented geniuses. Perhaps unintelligible "secret sauce" code has an added benefit of making industrial espionage more challenging (and of course if a rival firm steals all your code they can arbitrage all of your trades into oblivion).

Re: Arthur Whitney releases an open-source subset of K with MIT license

#160

Post by magicalhippo »

wood_spirit wrote: post #149
magicalhippo wrote: post #93

Earlier quoted context omitted.

I meant I don't know as in I haven't actually worked with him. Based on what I've seen before as well as the supplied code, I would be very skeptical to have him on my team.

This is a bit of a tangent but it’s a thought experiment that I recently heard: Data pipeline A is written and maintained by a team in a type safe language with extensive unit tests. Data pipeline B be was written long ago by a scientist who has since left, in sql in a day. Both compute the same dataset, but B gets the answer correct. Which is the better pipeline, and why?

Without more context or additional assumptions there's no good answer to that.

If you just care about today, then clearly B is better because it provides the correct result today.

Also, just because A is written in a type-safe language and has extensive unit tests doesn't in itself mean it's any less complex and undecipherable than B.

I can think of several takes, with different assumptions, leading to very different perspectives.

One take could be this:

Lets assume A has been written by a competent team, using good practices. Lets also assume the problem of incorrect answers in A has been known for some time and has been investigated a fair bit. That is, it's not just a trivial bug that's not been caught yet.

Since A doesn't work one could reasonably assume B is complex and difficult to understand, otherwise A's team should be able to find their error based on studying the SQL in B. Otherwise it indicates A's team is not competent, which goes against our previous assumption.

Given that, one could reasonably assume changing B will be very difficult.

Thus if one cares about maintaining and evolving the pipeline due to changing demands over many years, then it's likely A is better, as the bug in A producing the wrong answer should be fixable by a competent team.

Again, just one take of many possible...

An alternate, more trivial take could be that team A were given an incorrect specification. So while they implemented the specification correctly, B actually implements something slightly differently.

We see this one with customers all the time. Where they think the old system does X but it does in fact do something slightly different, so when we implement the new system as requested, the customer files a bug report because it doesn't do what the old system actually did.

Post reply on HN