Earlier quoted context omitted.
Exactly this. I did the same thing to find available camping spots in Hawaii because they are super difficult to get. Wrote a script that would query their "API" every 5 minutes and alert me if a spot became available anywhere.
Have you considered whether doing this might be...wrong? Presumably those campsites are permitted by some government agency (NPS, BLM, the state of Hawaii, etc.), and presumably that agency designed permitting system with the assumption that people with limited time and attention would be vying for the permits by having to visit the site themselves to get one. This encodes a particular definition of fairness: that th…
Reversing Safeway's private APIs to automate coupon collection
41–50 of 148 posts
Re: Reversing Safeway's private APIs to automate coupon collection
#42Earlier quoted context omitted.
Exactly this. I did the same thing to find available camping spots in Hawaii because they are super difficult to get. Wrote a script that would query their "API" every 5 minutes and alert me if a spot became available anywhere.
Have you considered whether doing this might be...wrong? Presumably those campsites are permitted by some government agency (NPS, BLM, the state of Hawaii, etc.), and presumably that agency designed permitting system with the assumption that people with limited time and attention would be vying for the permits by having to visit the site themselves to get one. This encodes a particular definition of fairness: that th…
Re: Reversing Safeway's private APIs to automate coupon collection
#43Earlier quoted context omitted.
Exactly this. I did the same thing to find available camping spots in Hawaii because they are super difficult to get. Wrote a script that would query their "API" every 5 minutes and alert me if a spot became available anywhere.
Have you considered whether doing this might be...wrong? Presumably those campsites are permitted by some government agency (NPS, BLM, the state of Hawaii, etc.), and presumably that agency designed permitting system with the assumption that people with limited time and attention would be vying for the permits by having to visit the site themselves to get one. This encodes a particular definition of fairness: that th…
Why is this definition changing with knowledge something that should be considered wrong? The ability to do this work isn't gated to certain people except by knowledge, and the knowledge itself isn't gated. For the longest time this has been considered a fair way of doing things to get an item in limited supply.
Re: Reversing Safeway's private APIs to automate coupon collection
#44Aside: At the end of the article they have a "Bonus: Speeding things up" section where they automate adding 300~ coupons via 300 HTTP connections in 5 seconds (instead of 60~ seconds).
In my opinion if you're going to automate stuff like this, you should do so with the goal of minimizing disruption (and frankly, detection). They run the script automatically at midnight in the background via cron, so why was going slower problematic? 300 requests in a span of 5 seconds seems much more likely to trigger an IDS[0], get flagged as unusual in the logs, or similar than 300 over 60 seconds. Particularly at midnight.
I'd be trying to look at human as possible and not set off automated security systems. Heck you could add a randomized delay (e.g. 1~2 seconds) between requests and it would still be completed inside of 10 minutes. Plus then nobody can reasonably accuse you of trying to "DoS" them/violate the CFAA.
[0] https://en.wikipedia.org/wiki/Intrusion_detection_system
Re: Reversing Safeway's private APIs to automate coupon collection
#45Earlier quoted context omitted.
Ordinary people don't use cron jobs.
This is not made for ordinary people I would say
Re: Reversing Safeway's private APIs to automate coupon collection
#46This is cool but I'm not sure the purpose. I use the Safeway app every time I go to Safeway. I scan all the offers and just add the ones I want, which builds a shopping list for me. This gives me a list of things to buy but more importantly I know what's on sale . If I just added all the coupons I'd still have to scan the list of things to find what I want to buy, but then I'd have to track them elsewhere, because th…
Coupons have two purposes, which seem contrary, but are essential to understand how coupons are a form of advertising: 1) Saves you money (versus the retail price of the item) 2) Makes you more likely to buy the product on the coupon. Your use of coupons adheres to both value #1 (saves you money, because you know what's on sale) and value #2 (advertises products on sale, increasing your chances of buying them). This…
You have a well-written and accurate comment, but I'm overwhelmingly distracted by the combination of the adorable "pay-per-clip" and the historied debate about how to handle an emoticon smiley at the end of a parenthetical. (conclusion: you're doing it wrong and are a terrible person!....but, "pay-per-clip"...tee-hee)
Re: Reversing Safeway's private APIs to automate coupon collection
#47Earlier quoted context omitted.
That is so true. Even if your algorithm works it's a human factor that breaks it.
Never underestimate the error introduced by a lack of incentive for humans to comply with the API (or the API itself being human-hostile). My favorite example is the Domino's "Your pizza is ready" signal. Since the data feeding the signal also feeds the store's performance analysis (i.e. they track how fast employees are getting pizzas ready), there's significant incentive for employees to lie to the algorithm and hi…
To add a bit of color to what's happening:
- You don't actually hit an "It's ready" button. When you knock the pizza off of the makeline screen, it transitions from "so and so is making your food" to "it's cooking".
- The "It's cooking" phase is just a simple timer. It's supposed to be adjusted to the time of the conveyor oven (which can vary from ~3-9 minutes depending on the particular oven they use). Most stores never customize that setting in the system, and it stays at the default. And other stores may have ovens running at two different speeds.
- Average make time is one of the metrics monitored by corporate audits, so you do have gaming of the system by knocking pizzas off the make screen early. But you can also have a backup at the oven during rushes, where food sit at the end of the makeline ready to go in the oven when capacity opens. At the same time, some items such as wings have to go through the oven twice (depending on the oven configuration). In both of these cases, even without gaming the system there will be dissonance between actual cook time and the cook time shown in the pizza tracker.
- For those that did try to game the performance metrics, it came with an equal headache internally beyond just that "It's cooking" timer. The scheduling system used that average make time as an input into calculating labor needs. The more you knocked stuff off the makeline early, the more optimistic the scheduler became and the more you'd have to override the suggested schedule and the more "Labor Waste" you'd create by having more people working than the system thought you needed.
It was really quite fascinating to see that system transition before I started my career. I witnessed it at two stores under two different franchises - one did the bare minimum to comply and the other one embraced the new system and it's capabilities. There were a lot of incredibly capabilities and forecasting optimizations that were made possible by the update. But they all presumed accurate data in the system at all times. But in many cases, managers were either disincentivized to do what was required to maintain that level of accuracy or transparency, or a component of the system would be designed in such an idealistic fashion that it didn't allow for the amount of pragmatic flexibility it needed. Both of which have been really valuable lessons I've taken forward with me.
Re: Reversing Safeway's private APIs to automate coupon collection
#48Another tip: (area code) 867-5309 works for Safeway club card discounts in most area codes while piping tracking of your purchases to essentially null. Yes, Jenny’s number.
At the self checkout machines at Giant there is a "Forgot my card" option. It gives you all the discounts without entering anything.
Re: Reversing Safeway's private APIs to automate coupon collection
#49Earlier quoted context omitted.
Exactly this. I did the same thing to find available camping spots in Hawaii because they are super difficult to get. Wrote a script that would query their "API" every 5 minutes and alert me if a spot became available anywhere.
Have you considered whether doing this might be...wrong? Presumably those campsites are permitted by some government agency (NPS, BLM, the state of Hawaii, etc.), and presumably that agency designed permitting system with the assumption that people with limited time and attention would be vying for the permits by having to visit the site themselves to get one. This encodes a particular definition of fairness: that th…
That's quite a very large assumption that I don't think we can accept as fact.
> that those who register early, or are very motivated, or simply those with a lot of free time to refresh the site, will get permits.
Perhaps someone who writes a script would count as "very motivated"?
> Does that make me more deserving of the camping spot?
"Deserving" has nothing to do with this in the first place. The only way that works is if you define "those who register early", "motivated", "a lot of free time" as "deserving". I could maybe see an argument for the first two classes of people as being "deserving", but I don't think you can justify "I have a lot of free time" as a reason for deserving anything, really.
Doing first-come, first-served based on availability and the random possibility of a cancellation isn't ever going to be a "fair" system. This sort of system is put into place because it requires very little coordination and work on the part of the agency that maintains the reservations. Holding that up as some sort of standard for fairness, and suggesting that anyone who thinks outside the process is wrong... is a little much.
Re: Reversing Safeway's private APIs to automate coupon collection
#50I enjoyed the article and would recommend it. Aside: At the end of the article they have a "Bonus: Speeding things up" section where they automate adding 300~ coupons via 300 HTTP connections in 5 seconds (instead of 60~ seconds). In my opinion if you're going to automate stuff like this, you should do so with the goal of minimizing disruption (and frankly, detection). They run the script automatically at midnight in…
If I was more worried about Safeway catching on I'd probably do something as you suggested (at the very least I'd add user agent headers and the other cookies expected from a real session, as right now it's trivial to detect my requests).
Sneaker bots do this exceedingly well - it's a constant cat and mouse game to make the requests look as human as possible, very interesting space right now.