Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Scrapple – A framework for creating semi-automatic web extractors

github.com

Post reply on HN
Print view View original

1–10 of 10 posts

Re: Show HN: Scrapple – A framework for creating semi-automatic web extractors

#2

Post by alexmathew16 »

Scrapple is a framework for creating web scrapers and web crawlers according to a key-value based configuration file. It provides a CLI to run the script on a given JSON-based configuration input.

I built this along with a friend, as part of a project. I would love suggestions and feedback !

Re: Show HN: Scrapple – A framework for creating semi-automatic web extractors

#4

Post by harishb93 »

daureg wrote: post #3

Sorry it's not related, but what tool did you use to get your plagiarism report in the documentation? https://scrapple.readthedocs.org/en/latest/#plagiarism-repor...

The plagiarism report was generated using "Turnitin". It is an online plagiarism checker that can be used to generate plagiarism reports for students and educators who would want to publish research papers. It requires a paid account.

Re: Show HN: Scrapple – A framework for creating semi-automatic web extractors

#5

Post by masukomi »

alexmathew16 wrote: post #2

Scrapple is a framework for creating web scrapers and web crawlers according to a key-value based configuration file. It provides a CLI to run the script on a given JSON-based configuration input. I built this along with a friend, as part of a project. I would love suggestions and feedback !

suggestion: please add clues

* Clues such as how the hell "key-value based configuration files" have anything to do with web scraping (without making people view the source)

* Clues such as why anyone should bother using this (or even if it's worth considering for their tasks)

* Clues as to what form this "framework" takes.

* Clues as to what functionality it provides

* Clues as to what i have to do to make it scrape something...

In short. Please watch this 5 minute talk, contemplate the implications, and update the readme: https://youtu.be/23xzRCoDZf4 You do have a readme, but you may as well not, because it doesn't really tell me much.

Re: Show HN: Scrapple – A framework for creating semi-automatic web extractors

#6

Post by heyalexej »

masukomi wrote: post #5
alexmathew16 wrote: post #2

Scrapple is a framework for creating web scrapers and web crawlers according to a key-value based configuration file. It provides a CLI to run the script on a given JSON-based configuration input. I built this along with a friend, as part of a project. I would love suggestions and feedback !

suggestion: please add clues * Clues such as how the hell "key-value based configuration files" have anything to do with web scraping (without making people view the source) * Clues such as why anyone should bother using this (or even if it's worth considering for their tasks) * Clues as to what form this "framework" takes. * Clues as to what functionality it provides * Clues as to what i have to do to make it scrape…

I came here to say the same. The walls of texts look very impressive to the eye but after reading a few pages I have no idea where this is going. I am scraping on a daily basis so I am very interested to see what it does and how it will make my life and the lifes of others who might potentially need this any better.

It would be awesome if you could create a video tutorial that goes through the process of setting up a project and getting the first tiny bit of data out of it. I'd be more than willing to help you with this if you can show me how to get started. I am sure I can figure this out somehow but time is a valuable asset.

Scrapers are in huge demand as you can see on projects like Scrapy, Pyspider and the like so it would be a pity if this one goes down unnoticed.

Re: Show HN: Scrapple – A framework for creating semi-automatic web extractors

#7

Post by alexmathew16 »

masukomi wrote: post #5
alexmathew16 wrote: post #2

Scrapple is a framework for creating web scrapers and web crawlers according to a key-value based configuration file. It provides a CLI to run the script on a given JSON-based configuration input. I built this along with a friend, as part of a project. I would love suggestions and feedback !

suggestion: please add clues * Clues such as how the hell "key-value based configuration files" have anything to do with web scraping (without making people view the source) * Clues such as why anyone should bother using this (or even if it's worth considering for their tasks) * Clues as to what form this "framework" takes. * Clues as to what functionality it provides * Clues as to what i have to do to make it scrape…

Thank you for the advice. I will update the readme so it is more descriptive.

I focused more on writing the documentation for the package, and ended up putting too little time on the repository readme. I see the need for it to be more elaborate. I'll update that.

Re: Show HN: Scrapple – A framework for creating semi-automatic web extractors

#8

Post by alexmathew16 »

heyalexej wrote: post #6
masukomi wrote: post #5

Earlier quoted context omitted.

suggestion: please add clues * Clues such as how the hell "key-value based configuration files" have anything to do with web scraping (without making people view the source) * Clues such as why anyone should bother using this (or even if it's worth considering for their tasks) * Clues as to what form this "framework" takes. * Clues as to what functionality it provides * Clues as to what i have to do to make it scrape…

I came here to say the same. The walls of texts look very impressive to the eye but after reading a few pages I have no idea where this is going. I am scraping on a daily basis so I am very interested to see what it does and how it will make my life and the lifes of others who might potentially need this any better. It would be awesome if you could create a video tutorial that goes through the process of setting up a…

Thank you for your interest !

I have put a tutorial in the project documentation [http://scrapple.readthedocs.org/en/latest/#experimentation-r...]. I understand how a video tutorial would be more helpful.

It would be great if you could give some advice/suggestions on how to do the video tutorial.

Post reply on HN