131–134 of 134 posts
Earlier quoted context omitted.
It gets more complicated when you need to leverage real browser engines (eg Chrome). I've got jobs spread across ~ 20 machines/ 140 concurrent browser instances, it's non-trivial.
How many pages can you render in a a second per vcpu core ?
One thing I notice with all blog articles, and HN comments, on scraping is that they always omit the actual use case, i.e., the specific website that someone is trying to scrape. Any examples tend to be so trivial as to be practically meaningless. They do not prove anything. If authors did name websites they wanted to scrape, or show tests on actual websites, then we might see others come forward with different solut…
I'm just curious if you have any particular use cases you can't get the existing solutions to work with
Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...
https://chrome.google.com/webstore/detail/sqanything/naejbcf...
You can export the results to Google Sheets too. One advantage of the extension is it works with JS rendered tables.