Bugzilla – Bug 8900
Boom bricked if sent wrong firmware
Last modified: 2009-09-08 09:17:05 UTC
I somehow had the server think Boom was an SB2 and it upgraded it using SB2 firmware. It now seems to be bricked... the LEDs come on but nothing on the display and it's not possible to factory reset. I seem to recall Richard adding in some safety checks during TP development to avoid using a firmware image from another product, but maybe this didn't work properly on Boom.
Looking through the code I found an UPGRADE_APPLICATION_IDENTIFIER in the ipUpgrade package. At the moment it is set like this: - SB3: 0xdddd1111 - TR: 0xdddd2222 - SBR: 0xdddd3333 - Boom: 0xdddd1111 I guess it should be different for Boom (i.e. 0xdddd4444), but we need to come up with a strategy how to handle the switch. If I just change it now, the resulting image won't be accepted anymore. So, I guess the first step would need to be a firmware accepting both IDENTIFIERS and in a second step switch to the new one only. However we would need to wait with the second step until all Booms are on the interims fw with both IDs. Hopefully somebody comes up with a better/easier/less scary solution. Maybe SC/SN could take some precaution to make sure we hand out the correct fw?
Ouch. First: Let's harden SC and SN to be more careful about sending out fw. Second: Transitioning to the new ID is a bit scary since we will have the old FW in the field indefinitely (as a large number are coming from the factory that way). I wonder if we should leave the ID as it is and add another indicator in ipUpgrade that prevents this in new firmware. Otherwise, we'll be maintaining a two-step upgrader indefinitely.
OK, don't worry about SC/SN, it is already pretty safe unless you do something stupid like rename boom_19.bin to squeezebox2_107.bin. I just had left a bug in my SN code while trying to fix Boom SN last night where it fooled the server into thinking Boom was an SB2.
The code that checks this identifier is in upgrade_app.c, towards the end of that file. Not knowing exactly how the upgrade image is structured, I'm unclear what it would take to add a second identifier. I am very worried about adding special cases here. I suggest we upgrade everyone to a version that checks for the correct identifier. It means we would have to keep an "interim" image around indefinitely, but I think the user impact will be minimized by the fact that these first Booms will get through the sales channel relatively quickly. Those users would get upgraded the first time they connect, but they won't experience a "double update" because the next rev of the firmware after that won't come along until we release the next SS rev. Only if someone has a first-run Boom _and_ they initially install a SC version that is newer than 7.2, would they experience a back-to-back update. That number will be very small. We did this for SB1 and it didn't cause any problems.
We also need to think about the correct strategy for SC and SN if people switch back and forth and get up-/downgraded.
After thinking about this a little more I am afraid there is no way around a two step approach. One 'transitional' fw (X) still using the wrong ID (to be accepted) but itself accepting the wrong and the correct ID. Future fw (>X) are then always using the correct ID and also only accept the correct ID. SC and SN would need a switch to hand out fw (X) if a Boom connects with fw (<X) and hand out fw (X+1) if a Boom connects with fw (X). Firmware (X) would need to be around indefinitely so Booms already produced with the wrong ID can still upgrade. An additional issue I see with that approach is that QA needs a bunch of OOTB Booms with fw (<X) to test with as they cannot easily downgrade when they upgraded to fw (>X) in case they need to test again.
Can't you downgrade by going through firmware X?
exactly. We have to get the bad ones "out of circulation" as quickly as possible. Trying to somehow keep the old ones updatable directly to any version in perpetuity is a recipe for disaster. We did a two-step upgrade with SB1 and it was not a problem, and if we get this into 7.2, nearly all of the buyers of the first batch of booms will only see a single upgrade. If we wait, it gets messier. Also that an SB3 can also be bricked by loading it with a boom upgrade!
Andy wrote: Can't you downgrade by going through firmware X? Only if we design it to allow that, but I think it should enforce a one-way upgrade. Remember X is a special case, which will identify itself as the "old" magic number, but expect the "new" magic number in the next upgrade. We may be able to put some safety mechanism is version X, to confirm that the currently installed version is really X-1 before allowing the upgrade to proceed - perhaps by checksuming some region of the running flash image. Ip3k upgrade images are actually self-extracting archives, so they can be made "intelligent" like that. This would ensure that SB3s can't be bricked by accidentally loading them with X. Then only version X would have to exist forever, and we would not have to distribute any upgrade images that are dangerous to old Booms or to SB3.
Also if this is in 7.2 we don't have a downgrade problem, there won't be a version of the firmware to downgrade to.
Sean's taking a look at this. Updating status.
I came up with a routine which can detect whether the hardware is an SB3 or not (based on behavior of the audio clocks). We will put this in the upgrade.bin of "Boom Version X", and have it skip the installation if it detects that it is running on an SB3. That will make it impossible to brick an SB3 by loading "X" onto it. I haven't worked out the exact details of how to change the magic number such that X will accept post-X, while pre-X still accepts X, but I'm sure that is feasible. So my plan is: - We release 7.2 with Boom version X which installs automatically. - Because of the SB3 test, X will be safe to include in future SC or SN releases. - Version X and later versions of Boom firmware will require a new ID. - X will be the only version of Boom firmware that can be installed on top of a pre-X version - There will be no way to downgrade to a pre-X version, and no pre-X versions will be circulated outside the company.
Just a reminder: This needs to be addressed by the end of the week.
I am working on this as we speak. It is proving more difficult than anticipated.
Verified with SqueezeCenter Version: 7.2.1 - 23502
This bug has been fixed in the 7.3.0 release version of SqueezeCenter! Please download the new version from http://www.slimdevices.com/su_downloads.html if you haven't already. If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.