111–120 of 121 posts
Earlier quoted context omitted.
Use DDG. It's fine for most things. Use Google as a fallback.
>Use Google as a fallback. To do this from DDG, prepend your search using !g to search Google
Earlier quoted context omitted.
> Anyone can go build a crawler and scrape the web the way Google scrapes it so they can compete with Google. I don't think that making a scraper will make you competitive with Google. If you can make a site ranking algorithm that compete's with google, on the other hand, you might have a chance
The site ranking algorithm is a solved problem. The one reason Google is competitive is due to them taking advantage of the cheap labour that keeps track of ranking manipulation. Luckily most of the search problems have nothing to do with ranking manipulation.
A company I consulted for was using a paid API to handle search. Despite the fact that the entire site was available in an easy to scrape XML format, scrapers kept using the search feature. They were trying very hard to overcome my countermeasures--they had a seemingly limitless pool of IPs, they were rotating user agent strings, and they tried to randomize search behavior. Everytime I implemented a new countermeasur…
It may have been better to just publish the site in HTML format with an easy to find link on front page to access it.
Earlier quoted context omitted.
https://www.instagram.com/ryuji513 It looks like it just hits https://www.instagram.com/graphql/query/.. . every time you scroll down so if you scroll too fast it just hammers it and throttles your requests to that endpoint.
1. Fetch 1st page. Note id of user (e.g. 1954202703). this is the "id": value in the url. Note end_cursor. This is used for the "after": value in the url Note rhx_gis. This is used to create the "X-Instagram-GIS:" header. Looking at archive.org, it seems as recently as last year, end_cursor was once all that was needed. 2. Fetch js from ProfilePageContainer url in 1st page (e.g., https://www.instagram.com/static/bund…
Earlier quoted context omitted.
Can you elaborate more on this statement? "On the other hand I think DDG is the Sprint of search engines." I've been interested in switching to DDG for a while but as a former Sprint customer, that statement scares me but maybe some explanation from you might understand your opinion better.
I'm not sure about the comparison itself... I've tried DDG several times, I search for technical things in generic ways a lot. DDG almost never gives me what I want in the first page. Google almost always does.
The real difference is non-technical things. Google filters out unflattering results and one side of anything even remotely political. It's a nerfed world, kind of like a Disney theme park. I'm an adult and I don't need to be led with blinders to the googly viewpoint.
For example, for google.com, you can typically make only around 300 requests per day, and if you reach this limit, you will see a CAPTCHA instead of search results. 300 is pretty easy to achieve if you're "Googling hard enough" (make 5 slightly different queries, go through the 20 pages of results it's willing to show you, repeat 3 times...), and I've seen it trigger far before that if you are searching for more obsc…
Earlier quoted context omitted.
>Use Google as a fallback. To do this from DDG, prepend your search using !g to search Google
Minor fyi, it doesn't matter if the bang comes at the beginning of the query. You can save the keystroke when refining to fall back to google and just throw it to the end of the query.
Earlier quoted context omitted.
The site ranking algorithm is a solved problem. The one reason Google is competitive is due to them taking advantage of the cheap labour that keeps track of ranking manipulation. Luckily most of the search problems have nothing to do with ranking manipulation.
site ranking is not a “solved problem” - google tries to solve it all the time and yet finding anything other than trending or popular stuff still takes more than several attempts (and often doesn’t even result in best results).
From one side it's along the natural-language interface from Alexa or alike; from the other side it's an interface of search for people who generally need access to information.
If Google exposed interfaces similar to Elastic Search - the search would never be an issue anymore; but it would not be easy to use by the users.