A Raspberry Pi as a decent residential proxy
wiringbits.net
A Raspberry Pi as a decent residential proxy
1–10 of 47 posts
Re: A Raspberry Pi as a decent residential proxy
#2> Don’t expose the proxy to the world or attackers will be able to interact with your home devices
Or use it to get stuff on the internet that they don't want to get directly themselves. Once upon a time I accidentally enabled open proxying on the web server on my Linux box. This was before smart phones and tablets and such, when my Linux box was the only network client I had, and so I didn't have a NAT router. Just modem straight to Linux box.
One day I checked my Apache logs to see if my extremely low traffic web site had any visitors.
First impression: "What the hell is all this traffic? This is orders of magnitude more than I should have!".
Second impression: "Why the hell is someone trying to get horse_fucking.mov from my server?".
Third impression: "That didn't give a 404. Why is my server actually providing horse_fucking.mov?".
(No, the orders of magnitude extra traffic were not all people getting horse_fucking.mov. There was a whole plethora of bestiality titles, plus a lot of other porn that you would not want to fetch directly from your own IP address either out of embarrassment if you got caught or because it was almost certainly illegal).
Re: A Raspberry Pi as a decent residential proxy
#3> Don’t expose the proxy to the world or attackers will be able to interact with your home devices Or use it to get stuff on the internet that they don't want to get directly themselves. Once upon a time I accidentally enabled open proxying on the web server on my Linux box. This was before smart phones and tablets and such, when my Linux box was the only network client I had, and so I didn't have a NAT router. Just…
> Or use it to get stuff on the internet that they don't want to get directly themselves
That's a good point, another one is people using your proxy to attack other servers.
I find the proxy area to be interesting but something I'm unlikely to use for business, you are quoting some examples that made me laugh, and I'd certainly laugh if it happens to me, but I know I will remove the public access, if I were involved in business with this, the story would be different.
Thanks for sharing.
Re: A Raspberry Pi as a decent residential proxy
#4Why not use a normal HTTP proxy like Squid? It would be more easy to use it since almost all languages and clients support HTTP proxy built-in.
Re: A Raspberry Pi as a decent residential proxy
#5Why not use a normal HTTP proxy like Squid? It would be more easy to use it since almost all languages and clients support HTTP proxy built-in.
I evaluated Squid which was my first option, due to the existing code on the project I integrated the proxy, it wasn't that straightforward to interact with Squid.
I'm still thinking to use several Pies and it's simpler to evolve the custom-made proxy to support that, if you see the Github repo, it's pretty simple.
Last, getting a Scala project to run on the old Pi was a nice experiment worth trying.
Re: A Raspberry Pi as a decent residential proxy
#6I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pulled an unused Raspberry Pi from my desk drawer, installed them all onto the Pi, and now run them for near free from my desk! Granted I need to be a little bit more efficient with memory and processor cycles, but that’s a fun little challenge. I find if you respect the websites in terms of not hitting them multiple times a minute (or only once an hour which I do) then you’ll very seldom get banned.
Re: A Raspberry Pi as a decent residential proxy
#7Why not just use the SSH dynamic SOCKS5 proxy?
Just ssh -D 12345 raspberry and then just use localhost:12345 as the scrapers SOCKS5 proxy. No need to involve a http proxy.
EDIT: AAh, ssh port is not accessible from outside and the raspi does a reverse connect.
Re: A Raspberry Pi as a decent residential proxy
#8Why not just use the SSH dynamic SOCKS5 proxy? Just ssh -D 12345 raspberry and then just use localhost:12345 as the scrapers SOCKS5 proxy. No need to involve a http proxy. EDIT: AAh, ssh port is not accessible from outside and the raspi does a reverse connect.
That can certainly work.
A reason to writing the custom-made proxy is mostly to evolve it supporting several pies, the tunnel being the simpler workaround for now.
Also, I'm a coder and wanted to try getting Scala running on my old pi.
Re: A Raspberry Pi as a decent residential proxy
#9I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pulled an unused Raspberry Pi from my desk drawer, installed them all onto the Pi, and now run them for near free from my desk! Granted I need to be a little bit more efficient with memory and processor cycles, but that’s a fun little challenge. I find if you re…
It's interesting how simple is to get small services out of the cloud now.
I'm not that worried about websites banning the Pi as the traffic is really low, on concurrent calls it's unlikely to be a problem as I don't expect the Pi to query several websites in parallel.
Certainly it's something worth implementing if I ever start getting more traffic.
Thanks for sharing.
Re: A Raspberry Pi as a decent residential proxy
#10I still keep a server running in my basement doing WireGuard since Netflix et al. all block cloud IPs/ASNs