Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: DataFuel.dev – Turn websites into LLM-ready data

datafuel.dev

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#21

Post by sachou »

olup wrote: post #20
sachou wrote: post #16

Earlier quoted context omitted.

do you need to embed it directly in pinecone ? If yes then DataFuel is the right choice. Adding this feature as we speak. Please let me know :)

Interesting but we process documents before embedding them, and have specific requirements for the embedder. Having developed a couple of page to markdown myself, I think the bigger challenge is to make sense of so many pages that rely on spacial organisation of information that only makes sense to human, or even presence of images. One way to do it is to render the page as an image and extract data with a vision llm…

what would be your specific requirement?

Right now adding chunk size, model for embedding, what else?

Image is a great challenge with OCR can be solve as you mentioned

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#22

Post by olup »

sachou wrote: post #21
olup wrote: post #20

Earlier quoted context omitted.

Interesting but we process documents before embedding them, and have specific requirements for the embedder. Having developed a couple of page to markdown myself, I think the bigger challenge is to make sense of so many pages that rely on spacial organisation of information that only makes sense to human, or even presence of images. One way to do it is to render the page as an image and extract data with a vision llm…

what would be your specific requirement? Right now adding chunk size, model for embedding, what else? Image is a great challenge with OCR can be solve as you mentioned

We, as many players, have custom pipelines on embedding. We don't split docs based on chunk size but do semantic chunking and chunk augmentation. We embed everything with two embeddings services to always have a fallback if one provider is not available.

If I were in your shoes I would not think embedding and inserting in a vector store would be my responsibility, especially since there are so many different stores on the market.

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#23

Post by Onavo »

__loam wrote: post #15
tesrx wrote: post #9

Earlier quoted context omitted.

OP just graciously accepted that feedback, no need to be condescending :)

It's kind of tone deaf to launch a tool like this without considering this in the current climate. Not a popular take on hackernews but everyone outside the tech space is pretty pissed about this stuff.

And proxy farms exist solely to get around this problem. If you believe the rights of content creators is the end all be all, don't complain next time Disney tries to extend the IP expiration dates.

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#24

Post by jackienotchan »

I'm noticing a big increase in crawling activity on the sites I manage, likely from bots collecting data for LLMs. Most of them don't use proper user agents and of course don't stick to any scraping best practices that the industry has developed over the past two decades.

This trend is creating a lot of headaches for developers responsible for maintaining heavily scraped sites.

related:

- "Dear AI Companies, instead of scraping OpenStreetMap, how about a $10k donation?" - https://news.ycombinator.com/item?id=41109926

- "Multiple AI companies bypassing web standard to scrape publisher sites" https://news.ycombinator.com/item?id=40750182

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#25

Post by carpenecopinum »

sachou wrote: post #19
gregoryl wrote: post #14

Earlier quoted context omitted.

Are you concerned about making a product that does this? The legal aspect of accessing a computer system that is intending to block your use seems worrisome.

It is the responsibility of the user. Everyone should be responsible for their own actions. We still allow knives to be sold, and most people use them for good.

Now imagine that knife stabbings became so common that almost everyone started wearing body armor and you start selling body armor defeating knives explicitly. I can honestly see why most people would be upset about that.

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#27

Post by bryanrasmussen »

keyle wrote: post #6
sachou wrote: post #5

Earlier quoted context omitted.

Good point! Thanks for the feedback. Let me add that to my todos.

It boggles my mind that you would launch without that as a prime directive.

I was recently on a project and out of the 10+ devs on it I was the only one who really knew about robots.txt, or at least the only one who said hey that robots.txt needs to handle internationalized routes, the default ones we disallow are all in English.

I don't say that makes them bad, they just knew other things, so I can totally not have my mind boggled that someone launched a product like this and didn't take obeying robots.txt into consideration and then adds it to the todos when someone complains.

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#28

Post by vunderba »

tesrx wrote: post #9
keyle wrote: post #6

Earlier quoted context omitted.

It boggles my mind that you would launch without that as a prime directive.

OP just graciously accepted that feedback, no need to be condescending :)

well... while true how do you reconcile this with OP's statements on another thread?

> It has an extensive proxy IP and retry system in place to bypass bot detection.

Seems like a bit of "talking out of both sides of your mouth".

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#29

Post by numberless »

> Please make it easy for users to try your thing out, preferably without having to sign up, get a confirmation email, and other such barriers. You'll get more feedback that way, plus HN users get ornery if you make them jump through hoops. https://news.ycombinator.com/item?id=22336638

> Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead. https://news.ycombinator.com/showhn.html

This looks like quite an interesting project but Show HNs need to be usable without sign up pages

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#30

Post by freeone3000 »

bryanrasmussen wrote: post #27
keyle wrote: post #6

Earlier quoted context omitted.

It boggles my mind that you would launch without that as a prime directive.

I was recently on a project and out of the 10+ devs on it I was the only one who really knew about robots.txt, or at least the only one who said hey that robots.txt needs to handle internationalized routes, the default ones we disallow are all in English. I don't say that makes them bad, they just knew other things, so I can totally not have my mind boggled that someone launched a product like this and didn't take ob…

Programmers have no institutional memory.
Post reply on HN