Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SpaCy 3.0

github.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: SpaCy 3.0

#61

Post by pabs3 »

t-vi wrote: post #36
syllogism wrote: post #18

Earlier quoted context omitted.

I'm sorry that this conflicted with your plans, but I feel strongly that distributing Python libraries via system package managers such as apt is very bad for users. The pain is felt especially by users who are relatively new to Python, who will end up with their system Python in a confusing state that is difficult to correct. We of course encourage anyone to clone the repo or install from an sdist if they want to co…

That is sad, I would use SpaCy more if it had Debian packages (in particular in Debian). Python stuff packaged by Debian seems to work very well for me and has been for years.

At $work we have internal packages of SpaCy and dependencies, not yet updated to 3.0 though. It was relatively easy to package using py2dsc from stdeb.

Re: SpaCy 3.0

#62

Post by tsujp »

This sort of this could be used to create a "plain language" CLI right? So you could have the usual flags etc and then a separate version for less tech literate people that allows something like "list hidden files" (I know `ls -a` isn't particularly hard to remember I just need a contrived example).

Re: SpaCy 3.0

#63

Post by hadsed »

liminal wrote: post #54

I'm curious what sort of NLP use cases people are solving. How are people finding business value in these models and pipelines? We have looked at a number of uses and have found it hard to make a case for ROI. Wondering what's been working for folks.

It helps to be able to deliver value at a low bar. This stuff takes a lot of time and energy to improve upon, and it won't "just work". Search is usually a good place to start. Just make one little process a tiny bit better. Make sure you're tracking data very well, because if you're not already at a place to be making data-driven decisions you can't possibly take advantage of machine learning. After all that, just iterate until you've found a better problem to tackle with your newfound capabilities :)

Re: SpaCy 3.0

#64

Post by nocajar »

@hannibal Would love to discuss how we could extend spacy as a powerful engine to also support processing from layouted documents. Just imagine how powerful it would be if you could throw a PDF document into it and it would preprocess it to text + layout, e.g. Paragraph and perform the next steps like extracting the right paragraph or date, adress, etc. I would love to provide/support that transformation. I have been doing similar things using rasa_nlu+spacy.

Re: SpaCy 3.0

#65

Post by zburatorul »

chartpath wrote: post #49
drran wrote: post #45

Earlier quoted context omitted.

Can you help, please? I want to use AI to translate (localize) messages for free software, in my case, Ukrainian language. My plan to improve quality of automated translation is to translate from similar languages in parallel, i.e. give a same message in English, Russian, Polish, and expect message in Ukrainian as output. Where I should start? Which libraries to use? How to connect them? How to train them?

I've been using LASER from Facebook Research via https://github.com/yannvgn/laserembeddings to accept multi-lingual input in front of the the domain-specific models for recommendations and stuff (that are trained on English annotated examples).

This sounds interesting. Can you share more please? It sounds like there is some multilingual input text on the basis of which you make recommendations, but I think you would have called that a search engine rather than recommender.

Re: SpaCy 3.0

#66

Post by yewenjie »

I am not sure if SpaCy does it but is there some free and open-source framework capable of speech synthesis comparable to the level of AWS Polly or alike?

Re: SpaCy 3.0

#67

Post by mleonhard »

joshklein wrote: post #59
slow_donkey wrote: post #55

Earlier quoted context omitted.

This sounds super cool, is there a public link to your work? I'd love to check it out

I haven’t written about it or made results publicly available yet, but I do intend to. I can make a note of the email in your profile and ping you with a link when it is available if you’d like.

How about posting it to HN?

Re: SpaCy 3.0

#68

Post by SirHoki »

magicalhippo wrote: post #29

I stumbled over SpaCy when looking for something to extract key words and numbers from sentences, however it looked a bit daunting and/or overkill. Think recipes or similar, turning "take three tablespoons of sugar" into [3, 'tablespoons', 'sugar'] or similar. Should I give it another shot or are there libraries more suited for this than just plain regexp galore?

This is exactly what you want: https://github.com/facebook/duckling

Re: SpaCy 3.0

#69

Post by drran »

kthejoker2 wrote: post #58
drran wrote: post #45

Earlier quoted context omitted.

Can you help, please? I want to use AI to translate (localize) messages for free software, in my case, Ukrainian language. My plan to improve quality of automated translation is to translate from similar languages in parallel, i.e. give a same message in English, Russian, Polish, and expect message in Ukrainian as output. Where I should start? Which libraries to use? How to connect them? How to train them?

MarianMT has a lot of pre-trained models from one language to another, here is one for Polish to Ukrainian https://huggingface.co/Helsinki-NLP/opus-mt-pl-uk and English to Ukrainian https://huggingface.co/Helsinki-NLP/opus-mt-en-uk You can test the models with the form input on the right-hand side And here's a site with docs and code examples for basic usage in your app https://huggingface.co/transformers/model_doc/m…

Translation from just English language creates a lot of ambiguity. The idea is to use two or more input languages to reduce ambiguity.

Currently, volunteers manually translate from English to all other languages. It's possible to use AI to help, but quality of translation is too low. Dumb Translation Memory database is much more helpful than AI translator.

In the new system, one volunteer will translate messages from English to his native language, while all others will be able to use this translation as the additional constraint, to improve quality of translation to their native language. More input languages -> more constraints -> better quality of translation.

My best idea, so far, is to use a multilingual tokenizer (SpaCy looks good) and a linear transformer, because linear transformers are able to accept large inputs, with thousands of tokens. IMHO, I can input multiple translations of messages (they are short), and define loss as expected Ukrainian translation.

However, I'm completely new to this field. I completed just one AI project so far: recognition of animals at video. I have no idea how to start.

For example, I have no idea how to hint transformer that input messages are the same, just in 4 different languages. Should I interleave messages, like "Cannot Не можу open відкрити file файл : : ", or put them aside, via separator: "\0Cannot open file: \0Не можу відкрити файл: \0", or create few independent inputs? Or use a memory and improve quality of output message incrementally?

Any hints where to start?

Re: SpaCy 3.0

#70

Post by magicalhippo »

SirHoki wrote: post #68
magicalhippo wrote: post #29

I stumbled over SpaCy when looking for something to extract key words and numbers from sentences, however it looked a bit daunting and/or overkill. Think recipes or similar, turning "take three tablespoons of sugar" into [3, 'tablespoons', 'sugar'] or similar. Should I give it another shot or are there libraries more suited for this than just plain regexp galore?

This is exactly what you want: https://github.com/facebook/duckling

That looks great, time to dip my feet in the Haskell pond then. Thanks!
Post reply on HN