Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arthur Whitney releases an open-source subset of K with MIT license

shakti.com

Post reply on HN
Print view View original

121–130 of 234 posts

Re: Arthur Whitney releases an open-source subset of K with MIT license

#121

Post by breck »

wood_spirit wrote: post #71
eigenvalue wrote: post #28

This language is very popular among quant finance people associated with Morgan Stanley. I don’t see the appeal myself. Maybe it helps prevent people stealing the code since it’s so awful looking to work with! At one point I had to learn it and I think I’ve totally forgotten it now— it’s like my brain repressed it. Not my cup of tea, that’s for sure.

Thinking about problems and data manipulation in the way array languages enable is hard but, once you have pushed through the what feels like a barrier of mainstream programming language thinking that is stopping you from “grokking” it, it is a sudden moment of clarity and then you “get it”. Perhaps a half way house is sql. The difference between ORM-style CRUD and a power user using window functions to make the data…

Agreed. Pushing through until you can think in array languages is well worth it! In my experience one of the top 30 highest ROI mental circuits you can develop.

That being said, I'm not convinced that the extremely minimal syntax is essential. I think it can be done another way ;)

Re: Arthur Whitney releases an open-source subset of K with MIT license

#122

Post by papercrane »

BeefWellington wrote: post #103
papercrane wrote: post #63

Earlier quoted context omitted.

Whitney is famous for writing code like this, it's been his coding style for decades. For example, he wrote an early J interpreter this way in 1989. There's also a buddy allocator he wrote at Morgan Stanley that's only about 10 lines of C code. https://code.jsoftware.com/wiki/Essays/Incunabulum https://github.com/tavmem/buddy/blob/master/a/b.c

When writing code in this manner, lines of code is kind of a meaningless metric. You could put the entire Linux kernel in one very long line of C.

It was meant just as an indicator of the code density. The actual line lengths are ~80 chars.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#123

Post by Qem »

eigenvalue wrote: post #81
eggy wrote: post #62

Earlier quoted context omitted.

Most people I know who actually learn an array language like k or j usually grow to appreciate the expressiveness and cleverness of these languages. Typically, people have your reaction who have only looked at it and tried it very briefly. I'm surprised. Why did you have to learn it? Where?

Was working at a quant pod at Millennium for a bit where they used it. I was ultimately able to use it but everything took me 20x longer than using Numpy/Pandas. The irony was that the Python code was shorter because there were so many more library functions and better abstractions and syntax. So it was slow and unintuitive for zero benefit whatsoever.

> was ultimately able to use it but everything took me 20x longer than using Numpy/Pandas.

You can try klongpy, a K-like array language implementation that runs atop numpy: https://pypi.org/project/klongpy/

Re: Arthur Whitney releases an open-source subset of K with MIT license

#124

Post by rscho »

frankjr wrote: post #32
cloudbonsai wrote: post #12

Earlier quoted context omitted.

You may not believe it, but that's how K/Q/J people write C code. Bonus: Go visit and do "View Source" on that website. Even HTML has fragrance of K.

I don't understand how. How do you debug something this? How do you go about fixing a bug? There are no docs nor tests. It seems like one would spend hours just trying to understand what's going on. Is there a good source on their methodology? Because my mind is blown.

You do actually spend hours to understand what's going on. Basically, the idea is that your implementation will be so short and condensed that you'll just rewrite from scratch to fix bugs.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#125

Post by lmeyerov »

Chirono wrote: post #96
stefano wrote: post #61

Those are some very big claims with respect to performance. Has anyone outside of the author been able to reproduce the claims, considering you need to pay 100k/month just to do it? I also wonder if the commercial version has anti-benchmark clauses like some database vendors. I've always seen claims that K is much faster than anything else out there, but I've never seen an actual independent benchmark with numbers. E…

I used to use K professionally inside a hedge fund a few years back. Aside from the terrible user experience (if your code isn’t correct you will often just get ‘error’ or ‘not implemented’ with no further detail), if the performance really was as stellar as claimed, then there wouldn’t need to be a no benchmark clause in the license. It can be fast, if your data is in the right formats, but not crazy fast. And easy…

The last point is spot on... Pandas on GPUs (cudf) gets you both the perf + usability, without having to deal with issues common to stack/array languages (k) and lazy languages (dask, polars). My flow is pandas -> cudf -> dask cudf , spark, etc.

More recently, we have been working on GFQL with users at places like banks (graph dataframe query language), where we translate down to tools like pandas & cudf. A big "aha" is that columnar operations are great -- not far from what array languages focus on -- and having a static/dynamic query planner so optimizations around that helps once you hit memory limits. Eg, dask has dynamic DFS reuse of partitions as part of its work stealing. More SQL-y tools like Spark may make plans like that ahead of time. In contrast, that lands more on the user if they stick with pandas or k, eg, manual tiling.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#126

Post by rscho »

sph wrote: post #69
fragmede wrote: post #57

Earlier quoted context omitted.

haven't you ever written code with single letter variable names and it makes sense to you? and then been forced to read somebody else's code with single character variable names and found it completely inscrutable? this is just that on (a lot of) steroids

No I don't write entire C programs with single letter variables, because there is no way where "c" is more readable than "cnt" or "count". With the usual exception of "for(int i", x, and y variable inside small scopes. If I was paid by the hour to write C, then I'd use single letter variables too, but I'm too lazy to do twice the work, when I can make my life simpler. Simplicity is a virtue, there is nothing interest…

The language structure of APLs is far simpler than C. Basically, you learn a few dozen primitives and you're off to the races. That's part of what makes APLs appealing.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#127

Post by dan-robertson »

IshKebab wrote: post #95
wood_spirit wrote: post #85

Earlier quoted context omitted.

Quick for building a website? Probably not. Quick for evaluating some idea you just had if you are a quant? Yes absolutely! So imagine you have a massive dataset, and an idea. For testing out your idea you want that data to be in an “online analytical processing” (OLAP) kind of database. These typically store the data by column not row and other tricks to speed up crunching through reads, trading off write performanc…

I would probably use Matlab for that sort of stuff tbh. Is K faster than Matlab?

I think some things will be faster and the real difference is more in things that are easy to express in the language. Eg solving differential equations may be easier in matlab and as-of joins in k.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#128

Post by oefrha »

rscho wrote: post #120
IshKebab wrote: post #95

Earlier quoted context omitted.

I would probably use Matlab for that sort of stuff tbh. Is K faster than Matlab?

For using a heap of libs to compute something that has been done a thousand times? No. For writing a completely new data processing pipeline from scratch? Much, much faster! Array langs have decent performance, but that's not why they are used. They are used because you develop faster for the kind of problem they're good at. People always conflate those two aspects. I use J for scientific research.

Seems weird to switch to develop faster and complain about people conflating the two aspects when this thread is clearly talking about runtime performance, triggered by the benchmark claims:

> real-sql(k) is consistently 100 times faster (or more) than redshift, bigquery, snowflake, spark, mongodb, postgres, ..

> same data. same queries. same hardware. anyone can run the scripts.

Re: Arthur Whitney releases an open-source subset of K with MIT license

#129

Post by doug-moen »

FYI: This only implements a subset of K (I'd estimate 1/3).

Calling it a 'release' is an overstatement. The docs state that it is a work in progress. It's also quite buggy (it's easy to get a segmentation fault). The version I saw in January was about 1/3 the size of this version, and also buggy. I hope that the final version of this code is less buggy and more usable.

If you want to learn the K language, don't use this version. Any of the other open source K projects are better than this (more complete, less buggy, better documented). This project is good if you want to learn more about the Arthur Whitney C coding style, because it is so small. Other projects written in this style (some open source K implementations, the J language) are huge by comparison.

Post reply on HN