Live data from Hacker News

Post Mortem: A single whitespace character

eatabit.com

91–100 of 209 posts

Re: Post Mortem: A single whitespace character

#91

Earlier quoted context omitted.

Probably, they tried it and it worked. This incidentally led to the bug that they're blogging about too.

Yeah, it's likely that they're all using Macs and have never checked their website on a Windows machine.

Yup

Re: Post Mortem: A single whitespace character

#92
Kudos for sorting this out quickly. Problems like this one can be really difficult to debug.

I remember one case where the coefficient table for a polyphase FIR filter we implemented in an FPGA caused huge instability problems in a design. The coefficient table, if I remember correctly, was 32 wide (32 multipliers) and 128 phases long. That's 4096 numbers. The design had about 40 of these tables that would be loaded from firmware into FPGA registers in real time as needed. We built a tool in Excel to be able to compute these tables of FIR coefficients.

We got word from a customer that things were not behaving correctly under certain circumstances. We were able to reproduce the problem in the lab but could not find anything wrong with the FPGA, microcontroller or Excel code after about three weeks of work by three engineers. This quickly became a nightmare as it threatened several lucrative contracts and failed to service our existing customer base adequately.

I had to put our other two hardware engineers back to work on their existing projects so I took on the debugging process. This was the most intense debugging I've had to do in thirty years of software and hardware development. Lots at stake. The very reputation and financial well being of my business was at stake. Enter 18 hour days, 7 days a week.

FOUR MONTHS LATER, at 2:00 AM on a fine Sunday morning without having slept for three days looking at code the bug jumped out at me. We've all had that moment but his one was well "one of those". The problem? We used "ROUND()" in instead of "ROUNDUP()" in calculation that had nothing to do with the FIR filter coefficients but rather affected the programming of counters related to them. This caused timing errors in a state machine that drove the FIR filters. If this were software this would be exactly like having the wrong count in a loop counter. Yup.

I re-calculated after making the change and everything worked as advertised. That was the best Monday I've had in years. And I took a long vacation after that.

Over four months to find a bug.

That's why sometimes it is impossible and even unreasonable to create budgets for software development. One little bug can set you back weeks, if not months.

Re: Post Mortem: A single whitespace character

#93
post #83

Earlier quoted context omitted.

That was a "privacy" mode of some personal firewall that was protecting the user's phone number from leaking out. Incidentally, this has leaked the users phone number because only that specific numer was being replaced with asterisks. Welcome to the world of very crappy "security" (-theater) end-user products.

The asterisks hide this request's BasicAuth credentials between the cellular printer and our app servers...we don't transmit customer phone numbers between the printer and our app servers. Twilio transmits them to our app servers but that is HTTPS

I think pavel_lishin was referring to a side-note in my war-store I listed as the parent comment.

Re: Post Mortem: A single whitespace character

#94

Earlier quoted context omitted.

Why don't you guys just use a Beaglebone Black ($50), M2M cape ( http://www.yantrr.com/products/m2m-cape-for-beaglebone ), and a thermal printer ( https://www.sparkfun.com/products/10438 ). BOM for that, plus project enclosure, is like $200. That's also just after a bit of quick googling...there's bound to be a much cheaper solution. What's your price point for hardware?

We are at about $200 now for hardware. We have a custom PCB and 3d printed case. We also have an LCD and some control knobs in the mix. Checking out the Beaglebone stuff now...very interesting since there is basically no 3G/4G modems for Arduino right now...

I'm just curious why the default response is still to reach for an Arduino--much more powerful SoC chips are cheap these days.

Re: Post Mortem: A single whitespace character

#96
post #50

Likely "Cowboy" is a transparent proxy added by your mobile service provider. I had a similar thing happening a year ago when the mobile provider used by most of our barcode scanners decided to add a transparent proxy into the loop (without telling anybody). The solution for this problem: Use SSL. I mean: There are already many good reasons to use SSL, but whenever you need to send any kind of mission critical data o…

We would really like to use HTTPS but it's not supported by the Arduino chipset as I understand it. Though I'm not the hardware guy here at eatabit...

Sounds right. I also found this:

http://stackoverflow.com/questions/15830333/arduino-due-http...

Re: Post Mortem: A single whitespace character

#97

      strcpy( ( char * ) commsOrderBuffer, "GET /v1/printer/");
  
      strcat( ( char * ) commsOrderBuffer, ( char * ) settings.getIMEI());
      strcat( ( char * ) commsOrderBuffer, "/orders.txt  HTTP/1.1\r\n");
      strcat( ( char * ) commsOrderBuffer, "HOST: ");
      strcat( ( char * ) commsOrderBuffer, SERVER_NAME);
      strcat( ( char * ) commsOrderBuffer, "\r\n");
      strcat( ( char * ) commsOrderBuffer, "Authorization: Basic ");
What the.... O(n) string concatenations, unnecessary pointer casts, no bounds checking... I think extra whitespace in an HTTP request is not their only problem.

Re: Post Mortem: A single whitespace character

#98
post #93

Earlier quoted context omitted.

The asterisks hide this request's BasicAuth credentials between the cellular printer and our app servers...we don't transmit customer phone numbers between the printer and our app servers. Twilio transmits them to our app servers but that is HTTPS

I think pavel_lishin was referring to a side-note in my war-store I listed as the parent comment.

Yup.

Re: Post Mortem: A single whitespace character

#99

Earlier quoted context omitted.

Who was replacing numbers with asterisks, and for what purpose?

I didn't want to hang our BasicAuth creds out to dry

Oh, I was asking pilif to comment about their experience.

Re: Post Mortem: A single whitespace character

#100

Earlier quoted context omitted.

We are at about $200 now for hardware. We have a custom PCB and 3d printed case. We also have an LCD and some control knobs in the mix. Checking out the Beaglebone stuff now...very interesting since there is basically no 3G/4G modems for Arduino right now...

I'm just curious why the default response is still to reach for an Arduino--much more powerful SoC chips are cheap these days.

For us, the reason is that my co-founder and I (both are not hardware guys) were able to build a proof of concept in my garage and Arduino seemed like the best (easiest) choice. Since then, we hired a hardware guy who designed a custom PCB etc. HTTPS would be great but we don't transmit any personal data so it's not a high priority right now.
Post reply on HN