71–80 of 86 posts
Next step is to train a model exclusively on leaks of proprietary Microsoft source code. Fair use, right Microsoft?
And if they want to sue for that, they will be shooting themseleves in the foot by proving their Fair-use argument isn't real.
Earlier quoted context omitted.
In theory the 2B model should be somewhere in between, whenever it gets fixed.
Assuming that it's just that hardcoded check I should be able to add it today. But if that check is load-bearing (i.e. if it relies on those assumptions elsewhere in the code) it could be a bit more painful. Edit: sadly, it's not that easy. Removing that check lets the 2B model load, but it produces gibberish. I've opened an issue with FasterTransformer here, and will also try to debug it further myself, but unfortun…
Earlier quoted context omitted.
Oof. GitLab is really the only contender to GitHub. Both gitea and sr.ht don’t have search features. So we are really starting from square one here
If you're working with code that fits on a single machine I recommend using ripgrep for brute force search - it's shockingly fast. I built my own web frontend for it a while back: https://simonwillison.net/2020/Nov/28/datasette-ripgrep/
Earlier quoted context omitted.
Oof. GitLab is really the only contender to GitHub. Both gitea and sr.ht don’t have search features. So we are really starting from square one here
Square one is great then. Let's move to square two "feature complete" by implementing search :-) Honestly, Gitea's feature set is quite impressive. sr.ht I'm less familiar. Of course GitLab is another option. I didn't mention it because it is well known, and probably less interesting / approachable to someone who would like to start from scratch, because it is more complex and there's this open core aspect.
Now GitHub / Microsoft, who are producing tools that integrate themselves more and more into the programmers' workflow, will now more opportunities to enforce this kind of fringe ideology.
Dystopian predictions; the following words will be replaced:
Parent / Child Inheritance Class Binary Invalid
Earlier quoted context omitted.
To do this in a way that’s actually useful is hard. Microsoft has had Intellisense, JetBrains have autocomplete in Intellij and other products. Both have big teams and decades of work put into them, and still and great, hence the ML approach being tried now.
Is it really that difficult? If I type app.get('/', then looking for literal occurrences and presenting me a menu of all the literal completions I have used before (no ML required at all) would already be a huge win.
What if it's not indented the same amount, is it the same?
What if it's not a GET, but a POST? Both use the req/res handlers so you'd want the prediction for both.
What if it's not for the path /, but for some other path. As you can only have one handler for the root, most handlers will be for different paths.
Maybe you can write these edge cases in for the Express situation and get req/res prediction, but that's a bunch of work to automate ~10 characters and it only works for this specific use-case. It doesn't work for any other languages, frameworks, libraries, or use-cases.
There are 2 ways to do this well: 1) static analysis of code to understand what is allowed to come next, and predicting based on that (Intellij, Intellisense, etc) or 2) ML or statistical analysis to determine what's likely to come next without knowing much about the rules (Copilot, Tabnine, Intellicode). Both of these approaches are hard to do right, but have a high pay off.
Awesome work, I made a similar project which is making a cost effective and privacy focused alternative to OpenAI text generation that can be switched to in a one line/easy way because it's API compatible https://text-generator.io/blog/over-10x-openai-cost-savings-... also works for generating code too so would be excited for someone to try that out too.
Does it work with natural language to sql?
### migrations/user_add_free_credit.sql adds free credits field to user table
This is awesome! How is the quality vs. GitHub Copilot for Python or JavaScript?
The largest Python model (codegen-16B-mono) should be pretty competitive! The biggest thing that it lacks is the ability to make use of context after the cursor. Unfortunately, that capability will require retraining the models: https://arxiv.org/abs/2207.14255
I'd be interested in how to convert models without retaining, or minimal retaining, I'd have thought it would just work with iterative mask tokens at least for models trained with MLM