Bugzilla – Bug 8272
Race condition in setd pref sync logic causes inability to update firmware
Last modified: 2009-07-31 10:21:55 UTC
Support has a number of reports from customers who are installing 7.0.1 and they cannot update their player to the new firmware. The player correctly reconnects to 7.0.1, correctly asks for the user to press and hold Brightness, but nothing happens when they hold Brightness. According the customers, the player is "frozen". We will try to gather more information.
forum post: http://forums.slimdevices.com/showthread.php?t=48090
QA to investigate
Come to find out if you connect to SN and then force a frimware upgrade by holding down brightness, fixes the issue.
QA to test with release version of 7.0, 7.0.1, latest build of 7.0.x & 7.1 daily build to verify if this is a SC issue. Also verify different OS versions. Anoop: can you tell us what OS versions are reported against this bug?
I have seen this bug with Vista and XP.
I got additional reports of this from our European Support office. "Hello Dan, I have seen a few incidents on the last week (3 so far) where customers with older SB3's ... are getting stuck while updating [their firmware] or the wireless functionality become unstable after the customer updates to SC 7.0.1" So, this appears to be continuing as an issue.
Dan, in order to prioritize and target this bug, are you able to find workarounds to get these customers going, or are their units useless? Thanks for any info.
Connecting to SqueezeNetwork and performing a factory reset there has always fixed this issue, so far.
Dan, have we seen any more reports of this issue? should we keep this bug open to monitor the situation or close it.
Nope, no more reports. I'm inclined to close it and reopen if it resurfaces.
I spoke too soon. Anoop reports we are still getting report(s) of this. Do not close yet.
I have had a few customers in the past couple days that are getting this issue. Where customer is prompted to upgrade and pressing and holding the brightness button does not work. Seems as though the sb3 is frozen. With one customer connecting to SN to upgrade firmware did not do the trick. He then opted to revert to 7.0.
Incorrect rights for the FW file? Player.firmware debug would be helpful. What OS are people using?
*** Bug 8921 has been marked as a duplicate of this bug. ***
We ran into this again yesterday, but this time it was going from fw 88 to 101. (pasting Osama's description) Connecting SB3 to Rhapsody this morning, I have been prompted for an update. Once the update was finished, I was able to access Rhapsody. Switching to Squeezecenter 7.0.1-19705 the SB3 prompted me for another update which is understandable. However, the SB3 gets stuck on the update and never finishes, even when we reset the unit to factory reset. Dan was able to observe the problem, and consulted with engineering, who asked for a log file, but... When we enabled logging the problem disappeared and we could not replicate it again. It was an interesting problem we thought it should be in the record. (note from Dan) One thing I did see was when the player was unable to update, the animation on the display of the little circle spinning was choppy and erratic. Later, when suddenly the updates were going fine again, that same animation was smooth.
This bug is alive & well -- wish my SB3 was. Here's my post from the General Discussions forum: jlivesey Junior Member Join Date: Aug 2008 Posts: 2 Upgrade to 7.1 killed my SB3 I originally posted in a mac-centric thread: Quote: Originally Posted by jlivesey After upgrading to v7.1, the accompanying firmware update killed my SB3. I have tried all of the steps that other users list above, but the unit is totally unresponsive. Also have tried v 7.2, 7.3 & back to 7.01. Nothing. Support desk unmanned over weekend leaves me with a silent brick. Does anyone know how to reset a wired SB3 that won't even power on? Have since gone to 7.1.1-22254, but it makes no difference. Server works fine, but SB is dead. Was working fine with v 7.0.1 & associated firmware on my G4 PowerBook. After upgrade, all settings (favourites,etc) were ok, except for fatal firmware upgrade... progress bar went across display, then no response. Aug 4th, no reply from support yet.... I tried the following reset modes, as per JHoltz (at http://www.audiocircle.com) *while powering up: 1. Hold down the brightness key to do a firmware update. 2. Holding ADD while powering up to reset to the factory default settings. 3. Holding down the power button -> full hardware reboot 4. press 1 during reset or power on -> re-flash xilinx cpld (io ports) None of these has had any effect, because the SB doesn't power up. Maybe one of the developers who regularly post to these forums can weigh in here.... my SB is out of warranty, but since software bugs are explicitly exempted from warranty coverage anyway, it would be nice to know that someone is actually working on the problem. 2008-Aug-11 -- support dept. tells me I have to ship to California (~$30) & pay $90 for repair. A more cynical person than myself might suggest that this is a very nice business to be in: take a known bug (#8272), do not fix it, encourage users to upgrade their software, and collect repair money after their hardware is rendered useless. Guaranteed income stream. But I would never be so cynical as to suggest deliberate sabotage. It's just normal business practice. Thanks, John Last edited by jlivesey : Today at 00:45. Reason: new info
John: can you post a link to your forum tread here, as well as post a link to this bug in your thread
An update on John's player... we RMA'd it and Vinh replaced a defective CPU. (rn# 080807-001895) It's back in John's hands at this time, firmware updated successfully, and he's up and running again.
Remco brought this to my attention: "... I downloaded the latest squeezeCenter software - version 7.01 a few days ago (July 1 2008) All fine until you see the message on your SB which says "press brightness to update". When I did this, the SB freezes. No buttons work. The only option is to unplug from power, and try again. But it freezes again after setup and connection. I reboot, I unplug, I factory reset, I check firewall and virus checker, and I even delete SqueezeCenter software and do a clean reload. I try both wireless and wired connections. I reboot my pc over and over. Nothing. Every time you repower and set up the SB it freezes at the final step - ie update the software on SB by pressing brightness button. I found the solution by trial and error-- * Problem - After downloading 7.01, SB freezes on repowering and setting up - freezes at the "use brightness..." prompt. * Solution - Go to PC and start playing something in SqueezeCenter. This then unfreezes the SB and it shows the "now playing" screen. Then press and hold the brightness button - and it updates perfectly and problem goes away..."
Moving monitor bugs
I saw this again today, updating from 112 to 7.3's 120. SB3 got stuck on the "Press and hold BRIGHTNESS to begin" screen. I again note that the little swirly anim at top-right animates choppy and staggered. No IR traps works: holding LEFT or POWER does nothing. Not to mention holding BRIGHTNESS. Server log correctly shows: [08-12-12 18:09:20.9975] Slim::Player::Squeezebox::needsUpgrade (274) Reading firmware version file: /Library/PreferencePanes/SqueezeCenter.prefPane/Contents/server/Firmware/squeezebox2.version [08-12-12 18:09:20.9979] Slim::Player::Squeezebox::needsUpgrade (358) squeezebox2 v. 112 requires upgrade to 120 But no go.
Felix: what are your thoughts on this? are there any logs or settings you want Technical Support to suggest to people with this error?
Not sure, maybe what Chris proposed in comment #13.
Dan: can you try to get a few logs from customers if possible? QA will attempt to reproduce here as well.
Logging that would be useful: * slim.proto --- to see IR commands coming * player.firmware --- to see update comments Theories: * Scanner is in the middle of a big scan, slowing server * Batteries in remote are low (single button pushes take less power than pressing and holding a button) * Network traffic is increasing latency
QA Test 2x SB3 , a) old firmware b) current firmware, also, make sure the server is scanning a remote library, to simulate server delay. Make sure to turn on the logs listed to see if it's server delay or firmware upgrade
I have spent about 3 hours trying to reproduce this issue, and I am still unable to. Dan: Logs from Customers having this issue would be great. ========================================================== * Scanner is in the middle of a big scan, slowing server //733MHz system - 10MB network connection to 1TB remote music source - 7.2.1 ->> 7.3.0 - SB3 r113 preexisting - Scan started after upgrade to 7.3 - attempted FW upgrade = No Problem * Batteries in remote are low (single button pushes take less power than pressing and holding a button) // Using all models of IR Remote with batters measuring ~0.8VDC -- Network.protocol.slimproto shows IR events properly - attempted FW upgrade = No Problem * Network traffic is increasing latency // Simulated a closed 802.11 network with clogged traffic (multiple downloads / SC's updating database / many streams playing) - attempted FW upgrade = No Problem
Created attachment 4465 [details] LogSnipet Log from DEvans attempting to upgrade from 113 to 120, SB3 spinney is very slow, display does not update, Log does not show update being received or sent.
Felix: Could a corrupted Xilinx cause this?
Michael / Andy: Dan's attachment shows a lot of SETD / setd pairs. Comparing with a local SC log when changing a player's name from the web interface shows setd / SETD pairs (i.e. SC sends the new player name and the player confirms by sending it back.) In the attached log however the pairs are reversed SETD before setd, but I cannot figure out why. I've checked firmware and if I see it correctly firmware never sends a settings change (i.e. player name) on it's own. Only upon receiving a setd from SC it either sets the setting in firmware and sends it back (SETD) or only sends it back (SETD). In SC, file Squeezebox2.pm, line 888 the comment reads: "Allow the firmware to update a pref in SC" but in certain circumstances it also sends it back to the player (line 916). Why? I guess that could cause a loop? Also on line 872, the send back command (setd) is either done immediately or delayed. Could that make problems together with a pending firmware update? And is paused (i.e. turned off) included in isStopped()?
Changing target to next release
After having updated SC to 7.3.1, my SB3 would shut down (display went dark) after first very briefly displaying the 'brighness' text. Daniel's solution (in #19) fixed the problem for some reason, so that I was able to update from FW 113 to 121
Michael: Any idea about what I commented in comment #30
James to reproduce with Prefs logging, and assign back to andy
Dan: I have tried to replicate this now for an hour, can can not. Since your system is able to repo it, can you set the following logs to debug, and attach a failing log please? * slim.proto --- to see IR commands coming * player.firmware --- to see update comments * prefs - Preference Setting & Saving
Created attachment 4611 [details] Log of SB3 failing at firmware upgrade I reproduced this again-- twice. Once with a log, which is attached. Details: 1 SB3 2 servers: A) WinXP running 7.2.1 B) MacOS running 7.3.1 SB3 started out connected to server A with firmware 113. I renamed the player to "SB3_White" and disconnected it and reconnected it to server B. It failed on the first try when I connected it to server B. (but I had not yet turned logging on.) Tried several more times WITH logging but could not recreate it, until I noticed the player had inadvertently been renamed back to "Squeezebox 2" during the process. While it was connected to server A I renamed it to "SB3_White", connected to server B, and it failed again. Log attached. (with network.slim.proto, player.firmware, prefs)
This is very strange. Dan's log shows the following sequence of events: -> setd "SB3_White" <- SETD "Squeezebox 2" -> setd "Squeezebox 2" <- SETD "SB3_White" and it just repeats because the values never sync up. The player's response to setd should always contain the exact same string it received. The only way it would fail is if the netbuf_fwd_read_str failed to read the hostname. That call should probably check for errors, for example if the netbuf is too long to fit in the hostname slot. But it obviously does read the hostname, it just doesn't send it until the *next* call? This now smells like a firmware bug to me. But I can't spot any bugs in the firmware other than the lack of error checking which shouldn't matter here. Richard, any ideas?
There was a race condition where multiple setd packets could be sent to the player before a response was received. This would cause flapping. Fixed in change 24625.
Fixed - Closed Message (SC) This bug has been fixed in the 7.3.3 release version of SqueezeCenter! Please download the new version from http://www.slimdevices.com/su_downloads.html if you haven't already. If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.
Correction: SqueezeCenter version is 7.3.2
Reduce number of active targets for SC