Viewing profile — pwendell

We do this for a lot of our customers (fine tuned to save cost when inference volume is high). Right now for internal coding we are using off-the-shelf models but we are considerin…

Thank you for the feedback. We wrote this because after discussing with some of our peer companies, I realized everyone was roughly doing similar things. And I thought it would be …

The difficulty of evaluating coding agents is indeed a really big challenge. We built evals on our own codebase and shared some information about that to allow other companies to r…

The issue is the growth rates can cause costs to drastically change quickly. If you have 1,000 employees and the average is spending $100/month you're at a $1.2M run rate. But sudd…

I authored this - happy to answer any questions.

15k is the entire corpus we have right now. Hopefully others can join up in releasing additional samples that can be merged in over time. We'll definitely keep iterating on Dolly a…

Yes this was a very surprising result... that the relatively small uptraining was able to unlock so much latent knowledge in the model.

Yes, this.

Yes the issue is that some of the training data is arguably tainted with some noncommercial license (it's nuanced, discussed below in my comment). We are releasing weights to peopl…

No I haven't heard of that, we'll engage with that team. This is exactly what we need will look into it.

Yes it's nuanced, but will be simplified going forward. This uses a fully open source (liberally licensed) model and we also open sourced (liberally licensed) our own training code…

Full source code is up here now: https://github.com/databrickslabs/dolly Sorry it took us a day to get the external repo setup.

The DataFrame is an evolution of the RDD model, where Spark knows explicit schema information. The core Spark RDD API is very generic and assumes nothing about the structure of the…

Very likely in Spark 1.4. Hortonworks has been helping out with this, we just need some internal refacotring to the API to make it work.

No worries. Hopefully you'll reconsider using it!

Hey - sorry you had a bad experience. That bug was filed as a "minor" issue with only one user ever reporting it, so it didn't end up high up in our triage. We didn't merge the pul…

You can call GraphX algorithms right now from within the Streaming API, for instance compute a graph on a windowed view of data. Online graph algorithms aren't there yet (probably …

Yeah, you can load Parquet data directly into SchemaRDD's in 1.1 and get the type conversion, including use of nested types. That's the long term solution for all of our storage in…