Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up in the Python community? – April 2023

bitecode.substack.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: What's up in the Python community? – April 2023

#41

Post by simonw »

ineedasername wrote: post #34

>Most new Python projects are still about AI Are there any that allow a user to run an LLM with GPT-3-ish capabilities on a single pc w/ Where "reasonable" is something like no more than a few minutes to get the output. A little longer wouldn't be too bad either since you could script something that submits prompts automatically and let things run in the background or something. Trying to search for such a thing-- if…

Yeah, that's possible now. You don't actually need a GPU to run inference against some of the smaller language models - I've heard reports of them running on a RaspberryPi.

The trick for the moment is to skip Python though. lambda.cpp and its many variants are the ones that I've heard working best.

I suggest starting with LLaMA 7B or Alpaca. More notes here: https://simonwillison.net/tags/homebrewllms/

This one is the easiest to get working I think: https://github.com/nomic-ai/gpt4all

Re: What's up in the Python community? – April 2023

#42

Post by theaiquestion »

mathisfun123 wrote: post #15
qbasic_forever wrote: post #8

I was browsing the most popular pip packages recently and was also downright shocked how much AI stuff is up at the top. I will be very curious if much of these things are still popular or relevant in a year.

So weird - you really think AI/ML is a flash in the pan when it comes to Python? You realize pytorch, tensorflow, pandas, numpy, scikit-learn are all approaching 10 years on pypi?

I think this is more comparable to the bajillion JS frameworks that popped up between like 2015 and 2020.

Same kinda deal with the big frameworks either surviving (likely with large rewrites for improvements) or succeeded by a replacement. But it feels like that's died down a bit.

> pytorch, tensorflow, pandas, numpy, scikit-learn

I think it's likely these will either survive or be replaced by a successor - but we wont be seeing 100 new ai packages a day.

Re: What's up in the Python community? – April 2023

#43

Post by int_19h »

ineedasername wrote: post #34

>Most new Python projects are still about AI Are there any that allow a user to run an LLM with GPT-3-ish capabilities on a single pc w/ Where "reasonable" is something like no more than a few minutes to get the output. A little longer wouldn't be too bad either since you could script something that submits prompts automatically and let things run in the background or something. Trying to search for such a thing-- if…

None of the models out in the open really have GPT-3-ish capabilities, not even llama-65b.

That aside, https://github.com/ggerganov/llama.cpp is the best option given the constraints that you describe. However, you will still need considerable amount of system RAM for larger models - 20 Gb for llama-30b, 40 Gb for llama-65b. Swap is an option here, but performance with it is abysmal.

These things are huge, and I don't think there is any way around that. You can play tricks with quantization, and larger models seem to be more tolerant of it, but even if, as some claim, 65b could be quantized to 2-bit while retaining decent performance, that would still need 20 Gb of RAM (CPU or GPU) to load - never mind additional requirements for actual inference.

The breakthrough here is more likely to come from new hardware that is highly optimized for those LLMs - basically just matmul for various bitnesses all the way down to 2-bit plus as much fast memory as can be packed into it for a given price point.

Re: What's up in the Python community? – April 2023

#44

Post by ensignavenger »

nologic01 wrote: post #39

While the hyperbole around AI might calm down a bit, the broader 'data science' domain is unlikely to diminish in importance and may generate new hypes. And in the short to medium term Python will certaintly play a core role in these developmemts. Yet what I am missing in the news is some hint of where Python is going longer term. What will 4.0 look like? Which of the know limitations will be addressed? Will it ever…

Some good questions, but many of them don't have answers yet.

> What will 4.0 look like?

Right now there has been no decisions made on a 4.0. It may be just another release, or it may remove some features, but it almost certainly won't be a major backwards compatability-breaking release like 3.0 was.

> Which of the know limitations will be addressed?

That depends on what the contributors want to put effort into. :) Speeding it up is currently getting a lot of attention. Each new release brings significant performance improvements.

> Will it ever be performant on a standalone basis?

As stated above, each new release is faster, but it is impossible to predict where it will end up, and even more impossible to know what you consider "performant". For many users, it is already performant :)

> Will it ever be native on mobile?

That is a question for the owners of the mobile platforms, as they generally decide what languages get first party "native" support on their platforms.

Re: What's up in the Python community? – April 2023

#45

Post by ineedasername »

LanternLight83 wrote: post #36
ineedasername wrote: post #34

>Most new Python projects are still about AI Are there any that allow a user to run an LLM with GPT-3-ish capabilities on a single pc w/ Where "reasonable" is something like no more than a few minutes to get the output. A little longer wouldn't be too bad either since you could script something that submits prompts automatically and let things run in the background or something. Trying to search for such a thing-- if…

If you're really alright with a few minutes delay, you might be able to run =<13b param. models on CPU

Any particular project that's prebuilt to not throw "out of ram" errors for it? I just want to play around with things without the restrictions ChatGPT. The OpenAI API is much less restrictive but can get pricey fast if I want quality above the lowest tier, at least for hobbyist purposes. I ran up a $10 charge in a few hours-- that's what I had budgeted for playing with a new "toy".

It get's very tedious prompt engineering to convince ChatGPT to respond to some perfectly reasonable prompts without and an answer that amounts to "aww shucks, I'm just a simple LLM and couldn't possible generate an answer to that sort of thing". Recently I asked it-- after about 20 prompts back and forth-- tp "analyze the personality of the person submitting these prompts" (They were prompts about the nature of AI, and how did it know that it was not a simplified LLM being run & controlled by a true advanced AGI, etc). It took about a dozen tried to get it to (finally) "Generate a fictional bio in the form of and RPG character sheet based on the prompts".

Even if it takes a few minutes and the quality is lacking a bit lacking and unpolished, I don't want to have to argue with the LLM to get a response.

Re: What's up in the Python community? – April 2023

#46

Post by ineedasername »

int_19h wrote: post #43
ineedasername wrote: post #34

>Most new Python projects are still about AI Are there any that allow a user to run an LLM with GPT-3-ish capabilities on a single pc w/ Where "reasonable" is something like no more than a few minutes to get the output. A little longer wouldn't be too bad either since you could script something that submits prompts automatically and let things run in the background or something. Trying to search for such a thing-- if…

None of the models out in the open really have GPT-3-ish capabilities, not even llama-65b. That aside, https://github.com/ggerganov/llama.cpp is the best option given the constraints that you describe. However, you will still need considerable amount of system RAM for larger models - 20 Gb for llama-30b, 40 Gb for llama-65b. Swap is an option here, but performance with it is abysmal. These things are huge, and I don'…

What about something that's maybe more along the lines of a GPT-2.5?

Re: What's up in the Python community? – April 2023

#47

Post by Spivak »

axutio wrote: post #29
Spivak wrote: post #12

Earlier quoted context omitted.

By volume of different projects probably not, but by usage probably. Once winners get decided and paths get treaded the ecosystem will probably be smaller but I bet the user count will be bigger. Look, even if you're the most biggest skeptic of all the AI stuff you can still do useful things with it that you couldn't before like present full-English interfaces and get embeddings for arbitrary text which is a leap com…

Is this a setup you've already built? Could you outline what you've glued together for this bot?

Oh sure! So the OCR is a statically linked build of tesseract based on [1] and pytesseract [2] which is a super thin wrapper but easier than writing it yourself. The I stole/modified the prompt from [3] to get the bot to write Python programs that do date calculations. Then I used [4] to take the output in case the Llm didn't use the format I asked. I run the Python code it generates in a rootless container that uses the Lambda RIE [5] because I was too lazy to make my own thing. So I'm mildly lied about the v8 isolate because everyone gets the implication and who wants to hear about RIE and seccomp profiles.

[1] https://github.com/wingedrhino/static-tesseract

[2] https://pypi.org/project/pytesseract/

[3] https://python.langchain.com/en/latest/modules/agents/tools/...

[4] https://pypi.org/project/dateparser/

[5] https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm...

Re: What's up in the Python community? – April 2023

#48

Post by ineedasername »

simonw wrote: post #41
ineedasername wrote: post #34

>Most new Python projects are still about AI Are there any that allow a user to run an LLM with GPT-3-ish capabilities on a single pc w/ Where "reasonable" is something like no more than a few minutes to get the output. A little longer wouldn't be too bad either since you could script something that submits prompts automatically and let things run in the background or something. Trying to search for such a thing-- if…

Yeah, that's possible now. You don't actually need a GPU to run inference against some of the smaller language models - I've heard reports of them running on a RaspberryPi. The trick for the moment is to skip Python though. lambda.cpp and its many variants are the ones that I've heard working best. I suggest starting with LLaMA 7B or Alpaca. More notes here: https://simonwillison.net/tags/homebrewllms/ This one is th…

Thanks! I think have my weekend free time planned out now. Or a least a Saturday. I don't do AI at work but playing with code on my time off, even for fun, is a little too work like to really relax. I needs me some mindless entertainment too.

Re: What's up in the Python community? – April 2023

#49

Post by modo_ »

roflyear wrote: post #6
gverrilla wrote: post #5

Question from a beginner: I'm deploying a python project to aws (beanstalk + lambda, among others), and I've noticed I can only use python versions 3.9 and 3.8. Why is that? Is 3.10 not considered stable?

I don't know why it is, but 3.10 and 3.11 are considered stable (and I would suggest you use 3.11 if you can).

I was just looking into this earlier. A lot of major packages have explicitly stated 3.11 readiness, but we're still only at 45% of the top 360 packages.

That's not to say the remaining 55% are broken on 3.11, just tread with caution. For comparison, 71% explicitly support 3.10.

https://pyreadiness.org/3.11/

Re: What's up in the Python community? – April 2023

#50

Post by bart_spoon »

roflyear wrote: post #6
gverrilla wrote: post #5

Question from a beginner: I'm deploying a python project to aws (beanstalk + lambda, among others), and I've noticed I can only use python versions 3.9 and 3.8. Why is that? Is 3.10 not considered stable?

I don't know why it is, but 3.10 and 3.11 are considered stable (and I would suggest you use 3.11 if you can).

I would not use 3.11 yet. I’ve had lots of issues with packages not supporting it and other headaches that still need to be ironed out. I’ve had no issues with 3.10.
Post reply on HN