Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in Python – The Complete Guide

proxiesapi.com

Post reply on HN
Print view View original

141–150 of 151 posts

Re: Web Scraping in Python – The Complete Guide

#141

Post by deanishe »

zffr wrote: post #87

Here are some tips not mentioned: 1. /robots.txt can sometimes have useful info for scraping a website. It will often include links to sitemaps that let you enumerate all pages on a site. This is a useful library for fetching/parsing a sitemap ( https://github.com/mediacloud/ultimate-sitemap-parser ) 2. Instead of parsing HTML tags, sometimes you can extract the data you need through structured metadata. This is a us…

This. A lot of modern sites can be really easy to scrape. Lots of machine-readable data.

APIs (for SPAs), OpenGraph/LD+JSON data in , and data- attributes with proper data in them (e.g. a timestamp vs "just now" in the text for the human).

Scraping is a lot easier than it used to be.

Re: Web Scraping in Python – The Complete Guide

#142

Post by deanishe »

spaniard89277 wrote: post #41
thrdbndndn wrote: post #21

Earlier quoted context omitted.

To me it's mainly the following three reasons, but take it with a grain of salt since my JS is not as fluent as Python. 1. the async nature of JS is surprisingly detrimental when writing scraping script. It's hard to describe, but it makes have a mental image of the whole code base or workflow harder. Writing mostly sync code and only use things like ThreadPoolExecutor (not even Threading directly) when necessary has…

I've had some experiences with selenium and now I'm using puppeteer, and I honestly don't see the problem with JS. It's true that I have not much experience coding but it seems to me that Pupeteer + Flask serving ML to extract data is the cake. Also, being able to play around evaluating expressions in pupeteer, etc, makes it manageable. Maybe I lack experience but I don't see JS being a barrier. I would like to know…

String processing in general is just easier in Python.

As a basic example, `lstrip()` and `rstrip()` trim whitespace by default, but can also remove any characters you'd like from the end(s) of your string.

You'd need to code that up yourself in JS. Happens quite a lot.

Re: Web Scraping in Python – The Complete Guide

#143

Post by ethbr1 »

appplication wrote: post #133
dragonwriter wrote: post #109

Earlier quoted context omitted.

I think ETL is right from the perspective where E refers to “from the source of data” and L refers to “to the ultimate store of data”. But the ETL functionality should itself lives in a (sub)system that has its own logical datastore (which may or may not be physically separate from the destination store), and things should be ELT where the L is with respect to that store. So, its E(LTE)L, in a sense.

For those confused as to whether ETL or ELT is ultimately more appropriate for you… almost everyone is really just doing ETLTLTLT or ELTLTLTL anyways. The distinction is really moot.

Maybe my understanding is incorrect, but expansion on the distinction.

Assumptions -- We're talking about two separate systems (source and destination) with non-neglible transfer time (although perhaps "quick")

ETL -- Performing the transform before/during the load, such that fields in the destination are not guaranteed to have existed in the source (i.e. 2 db model)

ELT -- Performing a 1:1 copy of source into an intermediary table/db (albeit perhaps with filtering), then performing a transform on the intermediary table/db to generate the destination table/db (either realized or materialized at query time), with the intermediary table/database history retained (i.e. 3 table/db model)

In short distinction, if regeneration or altering the destination is required, ETL relies on history being available in the upstream source.

ELT pulls control of that to the destination-owner, as they're retaining the raw data on their side.

Re: Web Scraping in Python – The Complete Guide

#144

Post by greenie_beans »

65 wrote: post #122
greenie_beans wrote: post #107

Earlier quoted context omitted.

this is what i try to do but i want to learn more about approaches like this, do you know any good resources about how to design ETL pipelines?

I built an ETL pipeline for a government client using just AWS, Node, and Snowflake. All Typescript. To cache the data I store responses in S3. If there's a cache available, use the S3 data, if not get the new data. We can also clean the old cache occasionally with a cron job. Then do transforms and put it in Snowflake. Sometimes we need to do transforms before caching the data in S3 (e.g. adding a unique ID to CSV r…

i appreciate you sharing all that, but it seems like we might be on similar levels of knowledge/experience. i've been a dev who does a lot data engineering for 5 years. i'm looking more for best practices and theory about designing the pipeline, how to arrange the order of operations, how to separate each step, logging practices, how to make it reproducible, how to restart when it fails halfway in without going back to the beginning, how many retries, what to do if a step gets stuck in failed state, how to flag that bad data, etc. so. many. questions. while i build these pipelines.

i have figured out these questions by seeing how more experienced devs do it and on my own, but i want to learn from a book or video series because you can only figure out so much yourself, eventually you need to seek out experts and sometimes the experts around you also figured it out themselves and you need to find an expert outside of your circle. unfortunately a lot of the "ETL experts" teaching stuff online are trying to sell me on prefect or airflow or snowflake etc

Re: Web Scraping in Python – The Complete Guide

#145

Post by sakisv »

deanishe wrote: post #140
sakisv wrote: post #126

Earlier quoted context omitted.

My main drive was to document the crazy price hikes that's been going on in my home country, Greece, so I'm scraping its 3 biggest supermarkets and keep track of the prices of their products* Had a lot of fun building and automating the scraping, especially in order to get around some bot catching rules that they have. For example one of them blocks all the requests originating from non-residential IPs, so I had to u…

I can't get prices on my local supermarket's website (in Germany) without selecting a specific branch. Seems kinda suspicious to me.

Yeah in this case between these 3 supermarkets there are 2 options:

1. You choose your general location or 2. You don't choose

In the first option you get one less general category to choose from (for example they may not have fresh fish)

As far as I can tell, in both cases the supermarket closest to the delivery address is responsible for filling out your order and usually what happens is that they call you to let you know that they don't have something and suggest substitutions.

Re: Web Scraping in Python – The Complete Guide

#146

Post by zffr »

deanishe wrote: post #141
zffr wrote: post #87

Here are some tips not mentioned: 1. /robots.txt can sometimes have useful info for scraping a website. It will often include links to sitemaps that let you enumerate all pages on a site. This is a useful library for fetching/parsing a sitemap ( https://github.com/mediacloud/ultimate-sitemap-parser ) 2. Instead of parsing HTML tags, sometimes you can extract the data you need through structured metadata. This is a us…

This. A lot of modern sites can be really easy to scrape. Lots of machine-readable data. APIs (for SPAs), OpenGraph/LD+JSON data in , and data- attributes with proper data in them (e.g. a timestamp vs "just now" in the text for the human). Scraping is a lot easier than it used to be.

Adding on to this, if an app uses client-side hydration (ex Next apps) sometimes you can find a big JSON object in the HTML with all the page data. In these cases you can usually write some custom code to extract and parse this JSON object. Sometimes the JSON is embedded in some JavaScript code so you need to use a little regex to extract it.

Re: Web Scraping in Python – The Complete Guide

#147

Post by jot »

nkko wrote: post #135
jot wrote: post #79

Earlier quoted context omitted.

This is how I do it. I send the URLs I want scraped to Urlbox[0] it renders the pages saves HTML (and screenshot and metadata) to my S3 bucket[1]. I get a webhook[2] when it's ready for me to process. I prefer to use Ruby so Nokogiri[3] is the tool I use for scraping step. This has been particularly useful when I've want to scrape some pages live from a web app and don't want to manage running Puppeteer or Playwright…

Does it save the whole page or just the viewport? Just checked the landing page it looks targeted to a specific case of saving “screenshots” and this is also obvious from limitations in the pricing page so it would be unfeasible for larger projects?

Urlbox will save the whole page.

It's primarily purpose is to render screenshots full-page or limited to viewport or an element. To do that well as it does the HTML has to be rendered perfectly first.

It's not as cheap as other solutions but we have customers who render millions of pages per month with us. They value the accuracy and reliability that's come from over a decade of refinements to the service.

Larger projects can request preferential pricing based on the specifics of the kinds of pages they are rendering.

Re: Web Scraping in Python – The Complete Guide

#148

Post by greenie_beans »

NortySpock wrote: post #130
greenie_beans wrote: post #107

Earlier quoted context omitted.

this is what i try to do but i want to learn more about approaches like this, do you know any good resources about how to design ETL pipelines?

Disclaimer: previous job had a lot of cases where CSVs were dropped by SFTP, your milage may vary..., JSON APIs are said to be a different flavor of crazy... Haven't heard much beyond "ask the Old Ones", but "Murphy's law strikes again", "eventually someone will want that data even though they swore it was unnecessary", "eventually someone will ask for a backfill/replay", "eventually someone will give you a duplicate…

sound advice. thank you.

Re: Web Scraping in Python – The Complete Guide

#149

Post by philippta »

simonw wrote: post #81
philippta wrote: post #14

Shameless plug: Flyscrape[0] eliminates a lot of boilerplate code that is otherwise necessary when building a scraper from scratch, while still giving you the flexibility to extract data that perfectly fit your needs. It comes as a single binary executable and runs small JavaScript files without having to deal with npm or node (or python). You can have a collection of small and isolated scraping scripts, rather than…

Does Flyscrape execute JavaScript that is on the page (e.g. by running a headless browser) or is it just parsing HTML and using CSS selectors to extract code from a static DOM?

As of right now Flyscrape just parses HTML using CSS selectors from static DOM.

But as more than enough websites there days are just an empty shell I am working on adding browser rendering support.

Re: Web Scraping in Python – The Complete Guide

#150

Post by worik »

staticautomatic wrote: post #33
danpalmer wrote: post #9

Earlier quoted context omitted.

Having done a lot of web scraping, the thing that often matters is string processing. Javascript/Node are fairly poor at this compared to Python, and lack a lot of the standard library ergonomics that Python has developed over many years. Web scraping in Node just doesn't feel productive. I'd imagine Perl is also good for those in that camp. I've also used Ruby and again it was nice and expressive in a way that JS/No…

Best web scraping guy I ever met (the type you hire when no one else can figure out how) was a Perl expert. I don’t know Perl so I don’t know why, but this is very real.

> Perl expert

Because Perl excels at text processing.

Post reply on HN