Slightly off-topic, but this is why dev posts like this are important. I didn't know eatabit.com was a thing, it it sounds like a great service.
Dev #2 here thanks for the compliment. Where are you? Maybe we should expand to your area. :)
Post Mortem: A single whitespace character
181–190 of 209 posts
Re: Post Mortem: A single whitespace character
#182Likely "Cowboy" is a transparent proxy added by your mobile service provider. I had a similar thing happening a year ago when the mobile provider used by most of our barcode scanners decided to add a transparent proxy into the loop (without telling anybody). The solution for this problem: Use SSL. I mean: There are already many good reasons to use SSL, but whenever you need to send any kind of mission critical data o…
What's wrong with transparent proxy ? Isn't how HTTP caching is supposed to work ? I would think the cache headers are the solution rather than SSL. It feels like you are kind of throwing the baby with the bath water. IMHO, badly configured transparent proxy does not mean the concept is bad, does it ?
And before that they decided to enforce a reverse DNS lookup so all the name based virtual servers with local DNS entries stopped working - I had my staging instances setup that way.
And not only all this happens without any prior information, it is next to impossible to climb through the support layers to finally find someone who even understands what are you talking about. They just want you to restart your modem to "resolve the problem".
Re: Post Mortem: A single whitespace character
#183Likely "Cowboy" is a transparent proxy added by your mobile service provider. I had a similar thing happening a year ago when the mobile provider used by most of our barcode scanners decided to add a transparent proxy into the loop (without telling anybody). The solution for this problem: Use SSL. I mean: There are already many good reasons to use SSL, but whenever you need to send any kind of mission critical data o…
We would really like to use HTTPS but it's not supported by the Arduino chipset as I understand it. Though I'm not the hardware guy here at eatabit...
Re: Post Mortem: A single whitespace character
#184Earlier quoted context omitted.
We are at about $200 now for hardware. We have a custom PCB and 3d printed case. We also have an LCD and some control knobs in the mix. Checking out the Beaglebone stuff now...very interesting since there is basically no 3G/4G modems for Arduino right now...
I'm just curious why the default response is still to reach for an Arduino--much more powerful SoC chips are cheap these days.
Re: Post Mortem: A single whitespace character
#185Earlier quoted context omitted.
What's wrong with transparent proxy ? Isn't how HTTP caching is supposed to work ? I would think the cache headers are the solution rather than SSL. It feels like you are kind of throwing the baby with the bath water. IMHO, badly configured transparent proxy does not mean the concept is bad, does it ?
Problem is often the competence of people handling these proxies. Yesterday I discovered that my ISP is blocking all DELETE http calls (ACT fibernet in India)! And before that they decided to enforce a reverse DNS lookup so all the name based virtual servers with local DNS entries stopped working - I had my staging instances setup that way. And not only all this happens without any prior information, it is next to im…
Re: Post Mortem: A single whitespace character
#186This very example -- requests were technically illegal all the time without devs realizing, but something in the stack changed to start rejecting them -- demonstrates the fallacy of the "be liberal in what you accept, strict in what you issue" principal. If all the web servers involved had been strict in rejecting the illegal request from the start, they would have noticed the bug in development before deploying to f…
SIP takes this to the next level. http://tools.ietf.org/html/rfc4475 Is a spec for "torture tests", where the SIP authors revel in the hideously complex parsing rules they've come up with (which is basically HTTP parsing). They even suggest that code should infer the meaning of messages. So I suppose you need some sort of AI to really handle things well. Binary protocols would be a better choice. Or, a well-defined t…
Re: Post Mortem: A single whitespace character
#187Earlier quoted context omitted.
The team at Heroku (where I currently PM) is constantly trying to improve our communication and documentation. We're definitely sorry that this caused problems, and we'll work even harder to make sure that our communication calls out any potential issues. Again - thanks for reaching out to us, and let us know if we can help.
Heroku did their best here. They reached out to us (twice) advising of changes and linking to a document that describe EXACTLY the bug that we discovered (later). Honestly, I don't feel bad about his bug because even if I would have read the alert to the letter, we would not have audited the entire codebase because we don't have that luxury of time. Yes, it took our whole operation down...but we found it, fixed it an…
After all it would not be that hard to scan for which customers are going to be bitten by that particular change when it actually happens rather than using some fire-and-forget email.
Re: Post Mortem: A single whitespace character
#188Earlier quoted context omitted.
I have to agree. I developed a proprietary embedded web server using a streaming HTTP parser. Complying with the HTTP parsing rules is a headache to say the least. Variable amounts of whitespace; 2 variants of line terminators (\r\n or \n) with the provision that the latter SHOULD be accepted by the server and line continuations make complying with the whole specification a real pain if you only have 100 bytes to par…
Every appearance of SHOULD/MAY in a spec is just begging for bugs or incompatibility. We'd be better off if those words were banned. Spec writers would be less inclined (hopefully) to come up with all sorts of arbitrary behaviour that might happen and could be maybe handled.
Re: Post Mortem: A single whitespace character
#189I saw it right away - "that HTTP/1.1 looks a bit farther away than it should be..." - and confirmed it by selecting the spaces. I thought it would be a bit more subtle than that... I remember working with a server that violated the HTTP spec by not accepting allowed extra spaces in headers. According to the new HTTP/1.1 RFC 7230, it should be a single space - the previous RFC didn't specify this clearly in the wordin…
There is one called Co-Advisor [1] that can be used to test web proxies. It is commercial and pretty expensive, but the online version might be free for open source projects. Squid and Apache Traffic Server are tested with it [2][3]. There was a USENIX talk that showed some Co-Advisor results [4]
1. http://coad.measurement-factory.com/details.html
2. http://wiki.squid-cache.org/Features/HTTP11
3. http://trafficserver.apache.org/acknowledgements
4. https://www.usenix.org/conference/lisa12/rolling-d2o-choosin... (at 31:16 in to the video).
Re: Post Mortem: A single whitespace character
#190Earlier quoted context omitted.
SIP takes this to the next level. http://tools.ietf.org/html/rfc4475 Is a spec for "torture tests", where the SIP authors revel in the hideously complex parsing rules they've come up with (which is basically HTTP parsing). They even suggest that code should infer the meaning of messages. So I suppose you need some sort of AI to really handle things well. Binary protocols would be a better choice. Or, a well-defined t…
So SIP is like IRC?
This same reasoning is why mail and other messages have the Header: Value format - you can compose in plain text. This is documented at least as far back as RFC 561, in 1973. Again, that's a good excuse: Users must compose by hand. Also, the RFC is just codifying what people were already doing.
HTTP, SIP, and others do not have these excuses. HTTP headers are essentially never written by hand, and the extremely few times they are, we don't need conveniences such as comments, line folding, lenient grammar, etc. (Proof: People write XML and JSON by hand far more often, without major ordeals.)