Viewing profile — entilzha

Great to see our paper here again! Since the paper release, we've also released model weights here for anyone interesting in building on top of it: https://huggingface.co/facebook/…

At least I wasn't aware of this work, but thanks for the refs! I'm always curious to read papers from 10-20+ years ago that have similarly inspired ideas. If it makes sense, we'll …

I don't believe so, or at least if someone tried it didn't work well enough that I remember :). Some of the motivation for the architecture changes in encoding patches stemmed from…

(Author here) If I understand your question right, this is one of the reasons BPE is nice and the parent liked it. For any character sequence, provided the characters are in the al…

(Author Here) Good description! Maybe what parent got mixed up on is an alternate way to view this is trying to chunk bytes to have roughly similar information. EG we initially tri…

(Author Here) Related thought, I think BPE is quite a good, cheap inductive bias to have in a model, which is part of what made it challenging to scale better against. I also suspe…

(Author Here) There is at least some work on character based modeling, but it hasn’t scaled well before. The challenge I think with something more adhoc for exceptional tokens is t…

(Author Here) In editing we couldn’t find a good place for this so cut it in the current version, but at one point had discussed a parallel with information density of speech as de…

(Author Here) Not sure what you mean by implicit? If you mean just treat bytes as tokens, one issue you run into is your sequence lengths get quite long, so compared to a regular t…

Author here :), I do think it’s a good direction to look into! That said, aside from it being a bit too much to do at once, you’d also have to be careful about how you distributed …

I tried a few a while back. What I really want is as close to 1-1 to obsidian UI as possible. I found with some of the plugins that it could be hit/miss on working correctly. If I …

While you’re here, a killer feature for me would be the ability to privately host obsidian sites (similar to publish). Even if it required subscribing to publish to download a tarb…

Any thoughts on how to access/modify on mobile without making it too cumbersome? I often think about todo on walk/train, but could see making it a computer only thing.

Totally agree on not trusting any one doctor. Nowadays, I basically assume doctors are narrow minded experts and do the broader thinking myself (by reading widely). It’s the old sa…

Not entirely true. For good utilization you need both GPU/TPU ops to be fast (written in C), but that won’t get you far if your input pipeline (possibly written in python) is slow.…

I’m a co-author and would be happy to answer questions about our work!

Anyone know if puppy works with deep learning libraries like pytorch/tensorflow, or if there are plans to do so? Not looking for numerical speed ups, but for speed ups in preproces…

A similar issue got me from recommending android and google services to friends/family to actively discouraging them (and migrating off of every service I could feasibly do). The W…

You could actually simplify even more with the trick used in a comment farther down ( https://github.com/0101/pipetools ). That way you would implement __or__ on for example `pipe`…

True, but for better or worse python has support for spark/sklearn/pandas/numpy/plotting/deep learning/web which makes it a good jack of all trades language for data/research scien…

Seq doesn't return a generator. At its core there is a concept of Lineage taken from Apache Spark. Essentially, when you do something like seq(data).map(func) it builds on a list o…

Data interchange between the two is relatively seeemless. seq(df) converts a pandas dataframe into a sequence of tuples/namedtuples. Calling .to_pandas(columns=cols) will convert a…

For operations where vectorization pandas will be much faster since its based on numpy. That being said, there are things which are very awkard to do in pandas which are very easy …

Thanks on name! Fair enough on docs being a bit scarce. The main intent is to being able to easily make a pandas dataframe a sequence of tuple/namedtuples, and convert a sequence o…