Live data from Hacker News

Reversing Safeway's private APIs to automate coupon collection

blog.jonlu.ca

71–80 of 148 posts

Re: Reversing Safeway's private APIs to automate coupon collection

#71

I enjoyed the article and would recommend it. Aside: At the end of the article they have a "Bonus: Speeding things up" section where they automate adding 300~ coupons via 300 HTTP connections in 5 seconds (instead of 60~ seconds). In my opinion if you're going to automate stuff like this, you should do so with the goal of minimizing disruption (and frankly, detection). They run the script automatically at midnight in…

Yeah from my experience they'll run into problems real fast making 300req/s in 5 seconds. It's so "noisy" and potentially disruptive.

I'm honestly surprised they weren't rate limited. I mean, your SPA would have to be really messed up to make more than a handful of requests a second (even then why aren't you using sockets?) - So it's super reasonable to say that if anyone is making over 100req/s then maybe they should take an hour timeout.

If you don't want to get caught doing this then you'll want a randomized delay like you said, and preferably a pool of IP addresses you proxy through. This is to prevent the automated stuff from catching you though - a human can still look at your account and wonder why you have every coupon activated 24/7.

Re: Reversing Safeway's private APIs to automate coupon collection

#72
post #50

I enjoyed the article and would recommend it. Aside: At the end of the article they have a "Bonus: Speeding things up" section where they automate adding 300~ coupons via 300 HTTP connections in 5 seconds (instead of 60~ seconds). In my opinion if you're going to automate stuff like this, you should do so with the goal of minimizing disruption (and frankly, detection). They run the script automatically at midnight in…

I agree - I actually built the thread pool stuff for work while I was testing, since I had the code from a previous project. If I was more worried about Safeway catching on I'd probably do something as you suggested (at the very least I'd add user agent headers and the other cookies expected from a real session, as right now it's trivial to detect my requests). Sneaker bots do this exceedingly well - it's a constant…

I highly recommend not making more then say 5req/s - and preferably do it at like 2am.

These projects are fun IMO, but it's best to not hammer anyone's servers if it isn't necessary.

Re: Reversing Safeway's private APIs to automate coupon collection

#73
post #65

Earlier quoted context omitted.

Its amazing to me the degree to which people will go to save 3% or less on gas once a month. If you fill up 4 times a month you're saving about $4, and all it costs is a profile of everything you buy attached to your phone number.

Given that they're probably already tracking everything via my credit card number (whether I sign up for anything or not), I don't feel like much privacy is being given up. Though as a rule I don't bother with most of these programs, mainly due to the inconvenience factor. I've lost hope of maintaining my privacy through any actions I can take individually. A legislative solution is required.

Correct me if I'm wrong, but if you're using a credit card the bank is only getting a receipt that shows the sum total of the transaction and not an itemized receipt. The club cards are consuming and tracking the data of the entire transaction.

Re: Reversing Safeway's private APIs to automate coupon collection

#74
post #17

When I see access to private API's like this, I wonder how judges would interpret these actions as they relate to the CFAA. By accessing "private" API's like this, are you knowingly accessing a computer without authorization? Are you exceeding your authorized access? The fact that Aaron's Law never went through has disturbed me...

Probably not...

There's an ongoing case Linkedin v HiQ where Linkedin said HiQ was scraping publicly available linkedin profiles but there was a robots.txt that told them not to. HiQ kept doing it until Linkedin threatened them under the CFAA. HiQ just won a preliminary injunction to get to continue where the court said it was unlikely that they were violating the CFAA but they might change their minds as the case progresses:

https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...

Re: Reversing Safeway's private APIs to automate coupon collection

#75
post #17

When I see access to private API's like this, I wonder how judges would interpret these actions as they relate to the CFAA. By accessing "private" API's like this, are you knowingly accessing a computer without authorization? Are you exceeding your authorized access? The fact that Aaron's Law never went through has disturbed me...

Probably not... There's an ongoing case Linkedin v HiQ where Linkedin said HiQ was scraping publicly available linkedin profiles but there was a robots.txt that told them not to. HiQ kept doing it until Linkedin threatened them under the CFAA. HiQ just won a preliminary injunction to get to continue where the court said it was unlikely that they were violating the CFAA but they might change their minds as the case pr…

The Safeway API in question requires a user to be logged-in, authorized, and have explicitly agreed to a TOS (clickwrap). The HiQ v. LinkedIn case was centered around publicly accessible content which did not require the user to be authenticated nor have explicitly agreed to a TOS (browsewrap is not enforceable).

I'd argue the above case does not apply here.

Re: Reversing Safeway's private APIs to automate coupon collection

#76
post #65

Earlier quoted context omitted.

Its amazing to me the degree to which people will go to save 3% or less on gas once a month. If you fill up 4 times a month you're saving about $4, and all it costs is a profile of everything you buy attached to your phone number.

The discount is markedly more substantial than 3%. If it's a Safeway brand gas station the max discount is a dollar off per gallon. The safeway gas station I used to do this at currently has regular for $2.69 a gallon.

The max discount I've ever seen is $0.20 for Chevron, but tends to be $0.10. Chevron is much less rare than a Safeway-branded gas station.

Currently west coast gasoline is ~$4.00/g, so that's about 2.5-5.0% discount, which is in line with GP said.

Re: Reversing Safeway's private APIs to automate coupon collection

#77
post #73

Earlier quoted context omitted.

Given that they're probably already tracking everything via my credit card number (whether I sign up for anything or not), I don't feel like much privacy is being given up. Though as a rule I don't bother with most of these programs, mainly due to the inconvenience factor. I've lost hope of maintaining my privacy through any actions I can take individually. A legislative solution is required.

Correct me if I'm wrong, but if you're using a credit card the bank is only getting a receipt that shows the sum total of the transaction and not an itemized receipt. The club cards are consuming and tracking the data of the entire transaction.

The club card acts as unique id to track you. The credit swiped at the machine has a unique id that they can use to track you.

The club card works across multiple credit cards and cash. That’s the main difference. And you might possibly track 2 people who control the household (wife gets one, husband gets one)

Re: Reversing Safeway's private APIs to automate coupon collection

#78
> This brought what used to be a painstaking process of clipping coupons from magazines to a slightly less painful process of having to click “Add” on all the coupons every week after logging onto their site.

This is an insane trend in supermarket usability. You want to offer low prices, but only to people who go through a practice round of online shopping before doing it in person?

Why are we doing this to ourselves?

Re: Reversing Safeway's private APIs to automate coupon collection

#79
post #73

Earlier quoted context omitted.

Given that they're probably already tracking everything via my credit card number (whether I sign up for anything or not), I don't feel like much privacy is being given up. Though as a rule I don't bother with most of these programs, mainly due to the inconvenience factor. I've lost hope of maintaining my privacy through any actions I can take individually. A legislative solution is required.

Correct me if I'm wrong, but if you're using a credit card the bank is only getting a receipt that shows the sum total of the transaction and not an itemized receipt. The club cards are consuming and tracking the data of the entire transaction.

The receipt has the last four digits on the card. Localizing that specific card number to the store and you are going to get pretty complete profiles on people. There will obviously be some duplicates but some smart algorithms could match up the common purchases. If I always buy the same boxed tea and the same deli meat, patterns can arise. The guy who buys a 1/2lb of swiss and 1/2lb of roast beef every week is probably a different person than the one that buys 1lb of cheddar and 1lb of turkey each week. Associate my other purchases with my deli meat choices and they can track when I came in for some special dinner ingredients I've never bought and a box of my usual tea bags. Even purchases that look unremarkable at first glance, like bread, eggs, and milk, have trends (bread brand, egg size/brand, 1%/2%/whole/skim/brand of milk). Having the last four digits makes it a lot easier to track customers but even without that, there are probably unique customer trends you can pull out of the seemingly anonymous data. At minimum the store could print out some coupons that it thinks you would be interested based on your latest purchase and what other people buy with the things you buy. A mom buying capri-suns and lunch snacks would probably be interested in similar kid lunch items.

Re: Reversing Safeway's private APIs to automate coupon collection

#80
post #49
post #34

Earlier quoted context omitted.

Have you considered whether doing this might be...wrong? Presumably those campsites are permitted by some government agency (NPS, BLM, the state of Hawaii, etc.), and presumably that agency designed permitting system with the assumption that people with limited time and attention would be vying for the permits by having to visit the site themselves to get one. This encodes a particular definition of fairness: that th…

> presumably that agency designed permitting system with the assumption that people with limited time and attention would be vying for the permits by having to visit the site themselves to get one. That's quite a very large assumption that I don't think we can accept as fact. > that those who register early, or are very motivated, or simply those with a lot of free time to refresh the site, will get permits. Perhaps…

I'll bite.

I think you're right the agency probably didn't sit down, write down definition of fairness, then design a permitting system around it. They probably implemented the cheapest/easiest digital analog they could find to a traditional fax-in/walk-up first-come/first-served permitting system.

However, the intentionality of the implementers was not central to my argument.

The system was created by (probably) non-technical people under a certain set of assumptions: namely, that this digital first-come first-served system would function approximately like the old paper one, but with fewer dead trees and toil. The old one was rate-limited by having to call an office and probably talk to a human, and the assumption that if you call every 5 minutes that human will probably get annoyed with you and stop answering your calls. The new one is rate-limited by the assumption that most campers simply can't spend all day refreshing a website.

The traditional first-come first-served system isn't intrinsically "ethical" or "fair" for some classes of people (as you've astutely pointed out), at best it's a crude approximation of some version of fairness. While crude, it was established by a democratically-elected government tasked with allocating a shared resource. "People who can automate HTTP API calls" and the nearby "people who can hire people to automate HTTP API calls" (as has actually happened with some outdoor permits) were almost certainly not in the groups of people the government was seeking to advantage by choosing this system, and I think most engineers are smart enough to be able to intuit that.

So the root of my comment was this: GP is using special knowledge they have (and probably worked hard for) to extract more of a public good than the public really intended to have access to.

* Is that fair to everyone who doesn't have GP's knowledge? Do people like the GP deserve more camping spots than others? This is a public resource, not sneakers, so fairness is important. * If everyone with programming knowledge acted the way GP acts, would that maximize the public good? * If everyone with programming knowledge acted the way GP acts, would the system even function at all?

My answers are basically: * No. Everyone who wants deserves an equal chance at the spots. If there's more demand than spots, it's the government's job to decide. Random programmers on the internet intentionally subverting the government's intentions is wrong regardless fairness (or lack thereof) of the original system). It would nearly-minimize the public good. Only programmers and people who can hire them would get popular camping spots. This is a real problem is popular outdoors areas around tech hubs. There's a reason NPS will only accept old-fashioned faxes provably not sent from a free online relay for the most popular Sierras routes in CA (e.g. the JMT and much of Yosemite), and it's not because they want to rock like it's the 80's or because the government is backwards. It's because assholes tried to spam the process with automation. I think some (like Half Dome) were migrated to a new lottery system on outdoor.gov this year. * The system would completely collapse, and most smart programmers could predict that. The government would have to spend more money on servers just to serve bots pinging the registration system constantly, or it would crash. Even if they did that, the people who gots slots would be a vanishingly small subset of the population (programmers) or people who can hire them. Likely, a grey market for "scalped" permits would arise.

Post reply on HN