Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: DataFuel.dev – Turn websites into LLM-ready data

datafuel.dev

Post reply on HN
Print view View original

31–37 of 37 posts

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#31

Post by sachou »

numberless wrote: post #29

> Please make it easy for users to try your thing out, preferably without having to sign up, get a confirmation email, and other such barriers. You'll get more feedback that way, plus HN users get ornery if you make them jump through hoops. https://news.ycombinator.com/item?id=22336638 > Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HN…

noted thank you for the nice reminder. Good I ll add more free tool and open playground

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#32

Post by bryanrasmussen »

freeone3000 wrote: post #30
bryanrasmussen wrote: post #27

Earlier quoted context omitted.

I was recently on a project and out of the 10+ devs on it I was the only one who really knew about robots.txt, or at least the only one who said hey that robots.txt needs to handle internationalized routes, the default ones we disallow are all in English. I don't say that makes them bad, they just knew other things, so I can totally not have my mind boggled that someone launched a product like this and didn't take ob…

Programmers have no institutional memory.

agreed, I had to often explain lower level simple stuff was there for them, because they didn't happen to know about that thing and were surprised.

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#33

Post by emmanueloga_ »

sachou wrote: post #18
emmanueloga_ wrote: post #11

I thought this might be interesting to share and potentially useful for the author of Datafuel as a comparison. I recently built something similar for a small app [1]. I use Bun.js's fetch to crawl pages, process them with Mozilla’s Readability (via JSDOM), and convert the cleaned content to Markdown using Turndown. I also strip href attributes from links since they’re unnecessary for my use case, and I don't recurse…

yes exactly, The main issue in scraping: - If you scrape a lot, you will be block based on you IP; You need to use PROXY - Scraping entire website need specific logic, retries and more - It becomes an heavy background job All the above takes time, so if in your business it is not your core feature, likely better to outsource it. Good job doing it tho!

Some ideas:

Highlight the advantages of your service over DIY solutions prominently on your marketing site. The site looks great! but I think it could better focus on convincing developers to adopt your product vs just listing features.

Consider reaching out to clients to quantify the time saved using your service. Emphasize how it eliminates the hassle of setting up custom background job processes, proxies, and other complexities that can snowball into a full-fledged project.

Good luck on your journey!

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#34

Post by __loam »

Onavo wrote: post #23
__loam wrote: post #15

Earlier quoted context omitted.

It's kind of tone deaf to launch a tool like this without considering this in the current climate. Not a popular take on hackernews but everyone outside the tech space is pretty pissed about this stuff.

And proxy farms exist solely to get around this problem. If you believe the rights of content creators is the end all be all, don't complain next time Disney tries to extend the IP expiration dates.

Using the behavior of one bad actor to excuse the abuse of everyone else is pretty bad.

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#35

Post by pxtail »

tesrx wrote: post #9
keyle wrote: post #6

Earlier quoted context omitted.

It boggles my mind that you would launch without that as a prime directive.

OP just graciously accepted that feedback, no need to be condescending :)

Let me translate what OP wrote:

> Good point! Thanks for the feedback.

> Nothing like this will be added to the product. Money comes from scraping content and thus content will be scrapped regardless any non-scrapping hints and we will be actively working on countering anti-scraping measures.

Re: Show HN: DataFuel.dev – Turn websites into LLM-ready data

#36

Post by gregoryl »

sachou wrote: post #19
gregoryl wrote: post #14

Earlier quoted context omitted.

Are you concerned about making a product that does this? The legal aspect of accessing a computer system that is intending to block your use seems worrisome.

It is the responsibility of the user. Everyone should be responsible for their own actions. We still allow knives to be sold, and most people use them for good.

I don't see that as a good analogy. There's very limited space for this functionality to be used legitimately / legally - anyone permitted to scrape content is likely able to access the data without the protection measures in the way.

I'm fairly sure circumvention is a (prosecuted!) crime in several countries - curious if you're across that angle, and/or have legal advice/direction you can share?

Post reply on HN