if you do any type of webscraping. xpath is the way to go. thanks to my former co-worker Justin, for showing me that.
What Happened to XPath?
51–60 of 97 posts
Re: What Happened to XPath?
#52XPath and XSLT was the first time (despite doing Haskell at university) that I started to really understand functional programming. The first time was working on a tech stack that was basically Microsoft SHAPE queries transformed into HTML. The second was multiple projects customising Google custom search engine results. It was weird realising that these very limited primitive were actually infinitely powerful if you…
There may be need for a replacement with simpler syntax; i wonder if GraphQL might be used in that role.
Out of curiosity, what tools do you use for scraping? Is there a similar simple tool for defining queries over trees?
Re: What Happened to XPath?
#53Anyone who does scraping or automated browser work eventually comes across XPath. In some ways, XPath is like regex. It's got insane power, but comes with a relatively steep learning curve. Remember reading regex for the first time? What? But unlike regex, the number of people using it are few in comparison. I avoided XPath until I couldn't anymore. I could do a lot with CSS selectors, but eventually the DOM traversa…
I agree. I think the original author completely missed the point and conflates lack of mainstream usage with dead tech. If you never run into problems that xpath addresses, of course you’ll never use xpath. It’s not for everyday use. And certainly shouldn’t be billed as a CSS selector replacement.
I think their complaints about browser support are fair (orthogonal to whether the newer versions are any good, which most of the comments here are talking about!)
In a self-managed environment, like a PC or server, then you're right that popularity makes little difference.
Re: What Happened to XPath?
#54I used xpath last week for something
Re: What Happened to XPath?
#55XPath post 1.0 got ridiculous, like many things do. What started with a simple, elegant language morphed into one with a http client, filesystem methods, json support, functions, loops, extensions and the ability to read environment variables. I wrote a post about it a while back[1] (I regret some of the wording used there) and maintain a tool[2] that can exploit XPath injection issues. I'd recommend sticking with 1…
Damn. XPath went off the rails after v2. Though, to be fair, so did JavaScript, and look where that is today!
Re: What Happened to XPath?
#56Xpath is so powerful for web scraping I just realized recently. I'd been using css selectors for my occasional scraping needs and never bothered to learn xpath until on day on a whim decided to learn at least the basics. Man I can now write scrapers in 2 minutes that used to take me quite some time thanks to the power of xpath. Thing like ancestors, contains, the ability to chain, etc is so so powerful. I used to wri…
It is the swiss army knife of scraping indeed. I feel like I can do anything with a scraper thanks to XPath.
Re: What Happened to XPath?
#57XPath is great, and works equally well in lumbering, ceremony-heavy Enterprise Java environments; and in quick bash one-liners.
I use it in a bunch scraping scripts for Web sites which don't provide RSS feeds. It's really nice for quickly 'exploring' a document to find the needed data; it's simple to update when sites change their layout; and it can be read in from a config file, argument, env var, etc. to keep things generic and flexible.
Re: What Happened to XPath?
#58XPath and XML in general is a great example of "Death by Committee". They tried too hard to be too smart and try to solve everything, and overcomplicated it to death. This is why people largely abandoned it. This is what is happening to C++ and they are steering themselves by committee into a dead end.
I'm starting to pay more attention to technologies that are resistant to this. Maybe I'm just getting old that I'm beginning to value mature, proven technologies over fads. More importantly, is the difficult skill of being able to spot them.
Re: What Happened to XPath?
#59Xpath is so powerful for web scraping I just realized recently. I'd been using css selectors for my occasional scraping needs and never bothered to learn xpath until on day on a whim decided to learn at least the basics. Man I can now write scrapers in 2 minutes that used to take me quite some time thanks to the power of xpath. Thing like ancestors, contains, the ability to chain, etc is so so powerful. I used to wri…
I realized a couple months back that Google sheets supports using xpath to scrape web pages. So now I have a "spreadsheet" scraping a page to see when a model of laptop goes on sale. Seems to work; at least, whenever I go double check that page manually it matches the scraped result.
[deleted]
Re: What Happened to XPath?
#60Xpath is so powerful for web scraping I just realized recently. I'd been using css selectors for my occasional scraping needs and never bothered to learn xpath until on day on a whim decided to learn at least the basics. Man I can now write scrapers in 2 minutes that used to take me quite some time thanks to the power of xpath. Thing like ancestors, contains, the ability to chain, etc is so so powerful. I used to wri…
I realized a couple months back that Google sheets supports using xpath to scrape web pages. So now I have a "spreadsheet" scraping a page to see when a model of laptop goes on sale. Seems to work; at least, whenever I go double check that page manually it matches the scraped result.
The only problem with the built-in IMPORTXML() function is that it doesn't execute pages with JavaScript. If you ever run into issues give API Importer a try (where I run a headless browser to execute the JavaScript): https://gsuite.google.com/marketplace/app/api_importer/52965...