Embedded Gurus hosts blogs by embedded engineers, including Michael Barr, on firmware bugs, C coding standards, and safety.
HN commenters cite its posts on gotofail, stack overflows, and unintended acceleration as lessons on preventable embedded and C software failures.
All the time.
If you want to be horrified, read about the ludicrous bullshit that is a breathalyzer. The fundamental problem seems to be three fold: first, as anyone not an idiot would expect, animals process alcohol differently so at best you have a model relating alcohol intake with alcohol present in the lungs. Second, the programming appears to be ludicrously bad [1]. Pigs and manufacturers have fought and fought to avoid having to share code behind these devices or present any proof that they measure what they purport to measure besides, "Trust me." When competent engineers have finally gotten their hands on code, you'll be unsurprised to discover it's full of bugs. Third, even accepting the bac to gas model arguendo, it's an open question if these devices measure what they purport to measure.
Just a few representative issues when competent software engineers have got their hands on the code:
- Of the available 12-bits of A/D precision, just 4-bits (most-significant)
are used in the actual calculation. This sorts each raw blood-alcohol reading
into one of 16 buckets. (I wonder how they biased the rounding on that.)
- Out of range A/D readings are forced to the high or low limit. This must
happen with at least 32 consecutive readings before any flags are raised.
- There is no feedback mechanism for the software to ensure that actuated
devices, such as an air pump and infrared sensor, are actually on or off
when they are supposed to be.
…
I think it means that this was the first conflict that it had in against Scuds. There was a software update made a few days before this incident that was in response to another Patriot/Scud engagement. The big issue with this time was that the missile didn't even fire.
http://embeddedgurus.com/barr-code/2014/03/lethal-software-defects-patriot-missile-failure/
It's interesting that there were 6 software updates to the Patriot missile batteries during Desert Storm, several of them were specifically to fix the clock errors.
It would be interesting to see the data for the misses and the hits compared to how long each system had been online for and what version of the software it was using.
Today's patriots are better (PAC-3, PAC-4 in testing) but still have several issues. I remember reading somewhere where a pilot got so mad that he was constantly being locked on by friendly Patriot batteries that he shot a missile at a friendly battery. http://books.google.com/books?id=7cU8Kpzu9osC&pg=PA339&lpg=PA339&dq=HARM+patriot+f16
These seem to be mostly RADAR issues, I can't imagine how difficult it would be to program a missile to intercept another missile, there's hundreds of variables involved. I'd be willing to bet that these days they can run them through a computer simulator and detect most of these issues.
Unrelated to the missile but relating to the original topic: http://www.wired.com/2012/11/google-spanner-time/
Choosing between bitfields and masks for register twiddling was something I looked into deeply. Like the article, I wanted the readability of bitfields, but after reading about all the pitfalls and how loosely bitfields specs were defined in the C89 standard, I decided to go with masks instead.
Here was a great summary I found from a comment explaining the dangers of bitfields.
Seeing it spelled out like this made it clear which to choose, lest you're a fan of nasal demons :).
"The royal mistake is to use bit fields in the first place. The following is not specified by the standard:
– whether bit field int is treated as signed or unsigned int
– the bit order (lsb is where?)
– whether the bit field can reach past the memory alignment of the CPU or not
– the alignment of non-bit field members of the struct
– the memory alignment of bit fields (to the left or to the right?)
– the endianess of bit fields larger than one byte
– whether plain int values assigned to them are interpreted as signed or unsigned
– how bit fields are promoted implicitly by the integer promotions
– whether signed bit fields are one’s compliment or two’s compliment
– padding bytes
– padding bits
– values of padding bits.
– and so on."
Source: https://embeddedgurus.com/stack-overflow/2009/10/effective-c-tip-6-creating-a-flags-variable/#comment-2390
"v later found out with more debugging that it happened because the module in question cast that struct back from memory, and then added its own stuff to it."
That's your problem right there: "cast". In C, you can cast things, but doing so breaks the type system, which can no longer help you. That's not the fault of C, that's your fault. You literally asked for it.
C is the useful idioms of assembly language made first-class. For fundamental reasons of the universe, C is the programming language. See also: https://embeddedgurus.com/barr-code/2018/02/c-the-immortal-programming-language/
"This year a portion of my analysis is focused on multi-year trends. One trend that really stands out to me is that the C programming language refuses to give up the ghost.
"A longitudinal study of survey data spanning almost a decade and a half shows that C remains the primary programming language of embedded software. Remarkably, in that time C has actually gained market share from 50% to about 70%–at the expense of assembly, C++, and Java."
"Toyota’s developers used MISRA-C and the OSEK operating system, both good choices for a safety-critical real-time system. But then they ignored, sidestepped, or circumvented many of the very safety features they are designed to enforce."
"On a cyclomatic-complexity scale, a rating of 10 is considered workable code, with 15 being the upper limit for some exceptional cases. Toyota’s code had dozens upon dozens of functions that rated higher than 50. Tellingly, the throttle-angle sensor function scored more than 100, making it completely and utterly untestable."
http://www.eejournal.com/archives/articles/20131127-toyota/
Michael Barr was the engineer that was able to sit down with the Toyota engine code during the "unintended acceleration" lawsuit and determine what NASA could not: that the code was a wreck:
http://embeddedgurus.com/barr-code/2013/10/an-update-on-toyota-and-unintended-acceleration/
Like I said. Bubble gum and duct tape.
Making a PoC is usually only the beginning. Firmware upgrades for microcontrollers may be difficult to do in the field, without UART and JTAG debugging. If clients expect that pretty demos will continue working, they might be quite disappointed. It's better to follow good design patterns (Active Objects, state machines [1]) from the start when possible.
dmitrygr is a legendary embedded hacker [2]: his Transcend WiFi SD card reverse-engineering is what led to me having a PQI Air Card in my pocket, and he's also written a bit-banged Bluetooth Low Energy driver. Great for a demo or when there's nothing else available, but far from a finished product.
Writing a MISRA-compliant BLE driver would make me MISRA BLE.
[1] https://embeddedgurus.com/state-space/2016/04/beyond-the-rtos-a-better-way-to-design-real-time-embedded-software/
[2] http://dmitry.gr/?r=05.Projects