Earlier quoted context omitted.
I have been thinking about that exact service for a while. One big issue is that you need the reverse engineer the pricing models of retailers, i.e. mapping out a high-dimensional space. For that you need a lot of data from the retailers . They could probably ban you quickly if you start pulling price quotes en masse. Then you need to invest a lot in cloaking techniques etc.
You don't. At least we didn't. Here's the real problem: 1. modern shopping sites are built haphazardly without structure; 2. a typical page load time is 10-20 seconds for many (most?) shops. 1. matters because you cannot build a simple crawler. This is because sometimes, the brand is in the title, the description is in the brand, and the gender is in the quantity, which itself is in the photo as a pixellated image...…
That sounds very exaggerated, are you quoting this range from actual data?
For a sanity check the first two shops that came to mind loaded in <3 seconds (guitarcenter.com, nike.com)