I setup an Openvpn server on my LAN, with username/password credentials so I can access my LAN from anywhere. I have multiple vpn configurations for my cell phone/laptop to either map only 192.x through the VPN or all traffic through it. It's very very useful, and seems like a much more secure option in my opinion.
A Raspberry Pi as a decent residential proxy
31–40 of 47 posts
Re: A Raspberry Pi as a decent residential proxy
#32Earlier quoted context omitted.
$ free -h total used free shared buff/cache available Mem: 432Mi 194Mi 26Mi 20Mi 211Mi 164Mi Swap: 99Mi 2.0Mi 97Mi $ top Tasks: 69 total, 1 running, 68 sleeping, 0 stopped, 0 zombie %Cpu(s): 7.3 us, 4.3 sy, 0.0 ni, 88.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 432.4 total, 26.2 free, 194.7 used, 211.5 buff/cache MiB Swap: 100.0 total, 98.0 free, 2.0 used. 164.7 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TI…
Don't forget to set gpu_mem in config.txt. It looks like you are running with the default, which is 64MB for the GPU. I'm guessing you are not using the GPU in this application and should reduce this value to the bare minimum (which I don't recall, sorry)
Thanks for the hint, I would be surprised if this old Pi actually has any GPU.
Re: A Raspberry Pi as a decent residential proxy
#33For more reliable ssh tunnels you can use autossh. https://www.harding.motd.ca/autossh/
The first time I hit an issue, autossh was the first proposed solution but I preferred to not include more dependencies. Also, I believed it should be possible to handle this with plain ssh + systemd.
Do you have any experiences with autossh/ssh to share?
Re: A Raspberry Pi as a decent residential proxy
#34I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pulled an unused Raspberry Pi from my desk drawer, installed them all onto the Pi, and now run them for near free from my desk! Granted I need to be a little bit more efficient with memory and processor cycles, but that’s a fun little challenge. I find if you re…
I did the same thing when I was running a scraper. I throttled the requests quite low to be respectful so it took a while to gather all the data, but it worked just fine. For this article, does the OP feel they need the proxy due to scraping rapidly?
The problem isn't the speed, if it were, I would certainly use something more powerful than an old Pi.
The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
Re: A Raspberry Pi as a decent residential proxy
#35I setup an Openvpn server on my LAN, with username/password credentials so I can access my LAN from anywhere. I have multiple vpn configurations for my cell phone/laptop to either map only 192.x through the VPN or all traffic through it. It's very very useful, and seems like a much more secure option in my opinion.
Can you please elaborate on why it's more secure?
Re: A Raspberry Pi as a decent residential proxy
#36Earlier quoted context omitted.
I did the same thing when I was running a scraper. I throttled the requests quite low to be respectful so it took a while to gather all the data, but it worked just fine. For this article, does the OP feel they need the proxy due to scraping rapidly?
The problem isn't the speed, if it were, I would certainly use something more powerful than an old Pi. The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?
Re: A Raspberry Pi as a decent residential proxy
#37Earlier quoted context omitted.
The problem isn't the speed, if it were, I would certainly use something more powerful than an old Pi. The actual problem is that some websites block IPs from known cloud providers, so, it doesn't matter which headers you use because what's blacklisted is the IP address.
Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?
Actually, that's what I'm doing.
In the post I detailed that I run a service (https://cazadescuentos.net) which is a browser extension to look for discounts.
Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go.
So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at home to my cloud server by using a SSH tunnel.
Re: A Raspberry Pi as a decent residential proxy
#38Earlier quoted context omitted.
Myself and the person I was replying to ran our scrapers from our homes. One benefit being the IP address doesn’t get flagged. Curious why scraping from your home network doesn’t work for you?
Actually, that's what I'm doing. In the post I detailed that I run a service ( https://cazadescuentos.net ) which is a browser extension to look for discounts. Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go. So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at hom…
Yeah I saw that in your article. I’m wondering why bother with the cloud server? Why not scrape all stores from home?
Re: A Raspberry Pi as a decent residential proxy
#39Earlier quoted context omitted.
Actually, that's what I'm doing. In the post I detailed that I run a service ( https://cazadescuentos.net ) which is a browser extension to look for discounts. Sometimes, such service needs to look for the actual discounts, it queries the actual stores from the server that's on the cloud, but some stores doesn't let it go. So, I mounted a Pi at home that resolves only those problematic queries, I expose the Pi at hom…
Yeah I saw that in your article. I’m wondering why bother with the cloud server? Why not scrape all stores from home?
The latency with the Pi is too high to handle all stores, I could easily do that if I was using a more powerful device.
Re: A Raspberry Pi as a decent residential proxy
#40Earlier quoted context omitted.
Yeah I saw that in your article. I’m wondering why bother with the cloud server? Why not scrape all stores from home?
The latency with the Pi is too high to handle all stores, I could easily do that if I was using a more powerful device.
Ah I see, cool thanks for clarifying. That does make sense.