Good luck bypassing akamai
Web Scraping in Python – The Complete Guide
101–110 of 151 posts
Re: Web Scraping in Python – The Complete Guide
#102It's much simpler to get the links via pandas read_html:
import pandas as pd
tables = pd.read_html('https://commons.wikimedia.org/wiki/List_of_dog_breeds', extract_links="all")
tables[-1]
Re: Web Scraping in Python – The Complete Guide
#103This is basically an advertisement for the site's scraping proxy service.
Re: Web Scraping in Python – The Complete Guide
#104This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…
Generally it's enough to archive the retrieved HTML just in case.
Re: Web Scraping in Python – The Complete Guide
#105I recently used Playwright for Python [0] and pypandoc [1] to build a scraper that fetches a webpage and turns the content into sane markdown so that it can be passed into an AI coding chat [2].
They are both powerful yet pragmatic dependencies to add to a project. I really like that both packages contain wheels or scriptable methods to install their underlying platform-specific binary dependencies. This means you don't need to ask end users to figure out some complex, platform-specific package manager to install playwright and pandoc.
Playwright let's you scrape pages that rely on js. Pandoc is great at turning HTML into sensible markdown.
For example, below is an excerpt of the openai pricing docs [3] that have been scraped to markdown [4] in this manner.
[0] https://playwright.dev/python/docs/intro
[1] https://github.com/JessicaTegner/pypandoc
[2] https://github.com/paul-gauthier/aider
[3] https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turb...
[4] https://gist.githubusercontent.com/paul-gauthier/95a1434a28d...
## GPT-4 and GPT-4 Turbo
GPT-4 is a large multimodal model (accepting text or image inputs and
outputting text) that can solve difficult problems with greater accuracy
than any of our previous models, thanks to its broader general knowledge
and advanced reasoning capabilities. GPT-4 is available in the OpenAI
API to [paying
customers](https://help.openai.com/en/articles/7102672-how-can-i-access-gpt-4).
Like `gpt-3.5-turbo`, GPT-4 is optimized for chat but works well for
traditional completions tasks using the [Chat Completions
API](/docs/api-reference/chat). Learn how to use GPT-4 in our [text
generation guide](/docs/guides/text-generation).
+-----------------+-----------------+-----------------+-----------------+
| Model | Description | Context window | Training data |
+=================+=================+=================+=================+
| gpt | | 128,000 tokens | Up to Dec 2023 |
| -4-0125-preview | | | |
| | New | | |
| | | | |
| | | | |
| | | | |
| | **GPT-4 | | |
| | Turbo**\ | | |
| | The latest | | |
| | GPT-4 model | | |
| | intended to | | |
| | reduce cases of | | |
| | "laziness" | | |
| | where the model | | |
| | doesn't | | |
| | complete a | | |
| | task. Returns a | | |
| | maximum of | | |
| | 4,096 output | | |
| | tokens. [Learn | | |
| | more](ht | | |
| | tps://openai.co | | |
| | m/blog/new-embe | | |
| | dding-models-an | | |
| | d-api-updates). | | |
+-----------------+-----------------+-----------------+-----------------+
| gpt- | Currently | 128,000 tokens | Up to Dec 2023 |
| 4-turbo-preview | points to | | |
| | `gpt-4 | | |
| | -0125-preview`. | | |
+-----------------+-----------------+-----------------+-----------------+
...Re: Web Scraping in Python – The Complete Guide
#106This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…
this is the way.
Re: Web Scraping in Python – The Complete Guide
#107This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…
I've found this to be a good practice for ETL in general. Separate the steps, and save the raw data from "E" if you can because it makes testing and verifying "T" later much easier.
this is what i try to do but i want to learn more about approaches like this, do you know any good resources about how to design ETL pipelines?
Re: Web Scraping in Python – The Complete Guide
#108Check out the cloudscraper library if are having speed/cpu issues with sites that require js/have cloudfare defending them. That plus a proxy list plus threading allows me to make 300 requests a minute across 32 different proxies. Recently implemented it for a project: https://github.com/rezaisrad/discogs/tree/main/src/managers
Nicely written scraper, btw. Good code.
Re: Web Scraping in Python – The Complete Guide
#109Earlier quoted context omitted.
I realise from working a few places that this isn't entirely common practice, but when we built the data warehouse at a startup I worked at, we engaged with a consultancy who taught us the fundamentals of how to do it properly. One of those fundamentals was separating out the steps of landing the data vs subsequent normalisation and transformation steps.
It's unfortunate that "ETL" stuck in mindshare, as afaik almost all use cases are better with "ELT" I.e. first preserve your raw upstream via a 1:1 copy, then transform/materialize as makes sense for you, before consuming Which makes sense, as ELT models are essentially agile for data... (solution for not knowing what we don't yet know)
I think ETL is right from the perspective where E refers to “from the source of data” and L refers to “to the ultimate store of data”.
But the ETL functionality should itself lives in a (sub)system that has its own logical datastore (which may or may not be physically separate from the destination store), and things should be ELT where the L is with respect to that store. So, its E(LTE)L, in a sense.
Re: Web Scraping in Python – The Complete Guide
#110Earlier quoted context omitted.
I've found this to be a good practice for ETL in general. Separate the steps, and save the raw data from "E" if you can because it makes testing and verifying "T" later much easier.
this is what i try to do but i want to learn more about approaches like this, do you know any good resources about how to design ETL pipelines?
There's quite a bit of new tooling in this space, selecting the right one is going to depend on your needs then you can spike from there. Check out Prefect, Dagster, Windmill, Airbyte (although the latter is more ELT than ETL).