Live data from Hacker News

httpdiff – diff responses to two HTTP/HTTPS requests

github.com

31–40 of 41 posts

Re: httpdiff – diff responses to two HTTP/HTTPS requests

#31
post #27

Earlier quoted context omitted.

Came here to say similar. "Those who don't understand Unix are condemned to reinvent it, poorly." – Henry Spencer

There's this thing called ux... John's program has a nice one, diff and curl and bash (used together in the way indicated here) have a terrible one.

I said "similar", I use plan9 so my shell script would have been slight different and my interface cleaner.

Using a 1970s TTY in the 21st century is dumb.

Re: httpdiff – diff responses to two HTTP/HTTPS requests

#33
post #21
post #20

is there any reason to use md5? (apart from backward compatibility etc.)

It doesn't really make a difference in a case like this. Though certainly not an ideal algorithm, the chance of a collision for something like this is low enough to never be a concern.

counterpoint: is there any reason not to use a stronger hash function?

Why leave something dangerous lying around when /probably/ nothing is going to go wrong... until someone picks it up and decides to do something with it that was unexpected, when better alternatives abound?

Re: httpdiff – diff responses to two HTTP/HTTPS requests

#34
post #22

Perfect use case for Go.

Can you elaborate?

Not OP but I'll give it a shot. Golang has a great standard HTTP package that makes this project a breeze to implement. Combine that with cross-compilation and statically linked binaries and you get a tool that can be ran virtually anywhere a developer would want to without needing to set up a new environment.

Re: httpdiff – diff responses to two HTTP/HTTPS requests

#36
post #27

Earlier quoted context omitted.

There's this thing called ux... John's program has a nice one, diff and curl and bash (used together in the way indicated here) have a terrible one.

I said "similar", I use plan9 so my shell script would have been slight different and my interface cleaner. Using a 1970s TTY in the 21st century is dumb.

How does plan9 help make your interface cleaner?

Re: httpdiff – diff responses to two HTTP/HTTPS requests

#37
post #36

Earlier quoted context omitted.

I said "similar", I use plan9 so my shell script would have been slight different and my interface cleaner. Using a 1970s TTY in the 21st century is dumb.

How does plan9 help make your interface cleaner?

Because I can use the built in plumber to do more interesting things. The terminal scrolls, I can send commands from the mouse interface.

I would enhance the shell script a bit to output a few more commands, should I need them.

If it was a tool I used regularly I could sharpen it.

Re: httpdiff – diff responses to two HTTP/HTTPS requests

#38
post #36

Earlier quoted context omitted.

How does plan9 help make your interface cleaner?

Because I can use the built in plumber to do more interesting things. The terminal scrolls, I can send commands from the mouse interface. I would enhance the shell script a bit to output a few more commands, should I need them. If it was a tool I used regularly I could sharpen it.

If you got it sharp enough would you share it?

Re: httpdiff – diff responses to two HTTP/HTTPS requests

#39
Nice tool jgrahamc. You should consider expanding how you detect/show diffing of the response bodies, since that has a lot of applications: detecting content changes, detecting ads/malicious code, detect crawl duplicates, security audits. etc.

Years ago I found the Levenshtein distance is super helpful to determine how different the responses are, and used it as part of a black box web security scanner. You can do this just on the Raw HTML, but that's noisy and shows a number of differences. It's better to use an HTML-aware string distance function, that diffs just page content. I used that a channel for detecting blind SQL injection (in combination with some other things).

I also found that you can go a level higher, and use Levenshtein on just the HTML tag structure of different responses. By looking at page structure, and applying different weights based on the HTML tags that were added/removed you can group similar pages, which usually maps to the different functional areas/templates of a site. As in, you can say "these 5 pages are all product details pages", "these 10 pages are all blog posts", etc. Super helpful from a security scanner, since this could inform crawling/auditing choices and speed up audits. It also allowed us to say "you have a XSS vulnerability in your Blog comments form" instead of just saying "you have XSS vulnerabilities in these 100 pages".

Anyway, there is a lot of value in detected how different/similar various responses are. See some of Google's published work about detecting near duplicates for web crawling...

Re: httpdiff – diff responses to two HTTP/HTTPS requests

#40
post #38

Earlier quoted context omitted.

Because I can use the built in plumber to do more interesting things. The terminal scrolls, I can send commands from the mouse interface. I would enhance the shell script a bit to output a few more commands, should I need them. If it was a tool I used regularly I could sharpen it.

If you got it sharp enough would you share it?

Sure. But I have no interest in doing it right now.

You can look at my shell script HTTP client if you like.

http://plan9.bell-labs.com/sources/contrib/maht/rc/httplib.r...

and some of my other plan9 code

http://plan9.bell-labs.com/sources/contrib/maht/

Post reply on HN