The proxy service recommendations are pretty expensive. Does anyone have alternatives they suggest to keep costs down?
Avoiding bot detection: How to scrape the web without getting blocked?
91–100 of 312 posts
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#92I am always amazed when otherwise intelligent people assert without data that the marginal cost of serving web traffic to scrapers/bots is zero. It is kind of like people who say "Why don't they put more fuel in the rocket so it can get all the way into orbit with just one stage?" It sounds great but it is a completely ignorant thing to say.
When I worked in e-commerce as a SRE, bots were doing two things: - trying to disrupt business processes (eg: false referral listings, gift card scams, etc) - trying to disrupt systems I'm sure there are folks who use bots and scrapers for home automation, but these users generate marginal traffic in comparison. The real cost, aside from successfully achieving the points above, is the bandwidth and hardware costs tha…
There's also a fair amount of scraping for things like...
- Reselling aggregated data
- Competitive pricing and inventory data
- "Sniping", like with auctions, event tickets, or things like airline check-in processes that are first-come, first-serve
- Weird SEO stuff where people scrape content in the hopes that isn't indexed yet, and they can beat you to it.
- And, sort of in the space you mentioned, searching for existing vulnerabilities by various signatures, or trying to brute-force guess things like passwords.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#93Earlier quoted context omitted.
Back when we had to scrape airline websites to get the deals they withheld for themselves, residential IP was indeed the way. Once the cottoned on to it and blocked id, you'd simply cycle the ADSL model, get a new IP, and off you'd go again. Now the best part... one division (big team) of our company worked for the (national carrier) airline , one division of our company worked for the resellers (we had a single grad…
Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.
>Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.
Can you say more about this? What do you mean by "Once you get to selenium it's usually over", and how do you manage cold starts in Selenium and emulating heavy usage?
Say your program starts right now, I assume you don't go through "adding heavy usage" to "warm-up", then get down to business, correct?
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#94Earlier quoted context omitted.
Back when we had to scrape airline websites to get the deals they withheld for themselves, residential IP was indeed the way. Once the cottoned on to it and blocked id, you'd simply cycle the ADSL model, get a new IP, and off you'd go again. Now the best part... one division (big team) of our company worked for the (national carrier) airline , one division of our company worked for the resellers (we had a single grad…
Once you get to selenium it's usually over, just had to emulate a couple of heavy users with real browsers and voila.
Yeah, but this was before Ajax when you had to hand roll XML RPC calls (i.e. long long ago).
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#95Earlier quoted context omitted.
Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.
I’ve never seen such a thing. Every locale here has a meter and you are charged monthly for exactly how much you use. Just like water.
I think he means the price per kWh changes each month.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#96By the way - is it possible to stop Google bot from scrapping without maintaining a list of IP addresses? Google doesn't publish these and it's not good to run reverse DNS as it slows down legitimate clients.
I know you can put a meta tag, but bot still has to make a request to read it.
I would like to completely cut off Google from scrapping.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#97Earlier quoted context omitted.
2FA should be a requirement on everything now. And if your site can't for some reason or you don't want to deal with it, then limit your site to external login providers only. 2FA, especially app based, has been proven to work really really well.
thats great till you're in a foreign country and your phone suddnely decides to die leaving you stranded and unable to access bank accounts or prove your identity. (happened to me)
This utility will help with that, assuming the services that use 2FA have a backup-code feature: https://github.com/sowbug/quaid
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#98Earlier quoted context omitted.
Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.
I’ve never seen such a thing. Every locale here has a meter and you are charged monthly for exactly how much you use. Just like water.
Rates are $/kWh. GP has some billing tariff where the rates change apparently.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#99Earlier quoted context omitted.
I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…
Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.
Here are providers that offer spot pricing to consumers:
https://www.flickelectric.co.nz/pricing-and-plans
https://www.pauatothepeople.co.nz/cheap-as.html
Aside from this, peak / off-peak pricing is not unusual.
Re: Avoiding bot detection: How to scrape the web without getting blocked?
#100Earlier quoted context omitted.
I wish they'd limit it to just stopping credential stuffing. Here's my scenario: My electricity provider publishes the month's electricity rates on the first of the month, I want to scrape these so that I can update the prices in Home Assistant. This is a very simple task, and it's something that Home Assistant can do with a little configuration. Unfortunately this worked exactly once, after that it started serving u…
Out of curiosity how is that you have electricity rates that change every month? Are you buying power through a third party organization? The vast majority of place I've seen have a fixed tariff for residential use that changes no more often than every 12-24 months.
In some cases it is because the consumer has opted for variable rates, essentially making a bet that net-net, variable rates will be less expensive than the fixed rates. Or that they would be able to shift usage to reduce usage during spikes. See: https://www.texastribune.org/2021/02/22/texas-pauses-electri...
Also see: https://www.ft.com/content/0e746280-e72c-4087-9c0d-df2a7af82...
Feb 2021 FT: "Bills mount in Texas power market after freeze sends prices soaring: Financial casualties emerge as grid operator Ercot requires billions in payments "