Bugzilla – Bug 14352
Synchronization Failing
Last modified: 2010-09-29 23:18:59 UTC
Tried Synchronizing a Receiver/Boom/Baby via a controller. Constantly re-buffers. All 3 were connected to a 7.4.28660 SC running on Macintosh OS-X Tried both Local and Napster based music. Surprisingly the Napster source seems to fair a little better? But still NOT enjoyable for sure... Even with just 2 devices: Boom and Baby, experiencing periodic re-buffering and gets out of sync easy. It seems to be inordinate amount of network traffic that is jamming up my WiFi Router? As internet connection to my laptop is also affected? -Sam
Just to add... Mac Laptop running SC 7.4 also connected via Wi-Fi. -Sam
I found same thing sync'ing two Babys. Running on SN. Delay is 5 sec. Both are r7779.
Not sure it makes a diff but one baby is pb2, one is pvt2.
player.source and player.sync logs would be a good thing to look at. Also does increasing the resync time make difference?
Pat, what source were you listening to?
I would note that sync adds no additional network traffic compared with two players playing the same thing unsynced. In the case of a remote stream being synced via a local SC there are some differences: only one copy of the stream is collected from the remote source and then distributed to the multiple local players. We really need logging to see what is happening. player.source=info, player.sync=info is the starting point. Adding player.sync=debug and network.protocol.slimproto=info will be much more verbose but may be helpful.
Sam was able to reproduce the problem without using sync. Starting up Baby while Ray and Boom were playing independently would trigger rebuffering on Boom. Sam's problem looks very much like it is a networking issue. The question is: is Baby somehow provoking the problem? I guess that it is possible that a player with poor wifi reception will cause the network to be saturated because it will need a higher proportion of the total available bandwidth to get its data through. Nonetheless, given that Sam was playing 256kb/s MP3s, this would seem to be extreme. Remember, however, that at the start of a track, streaming will continue at the maximum network-capable rate until the player's buffer is full.
Matt, please use this bug for the logs that I asked for. Alan.
*** Bug 9676 has been marked as a duplicate of this bug. ***
Created attachment 5984 [details] Patch to allow resync when clock advances Brandon, can you review this please.
Looks correct to me for implementing the change discussed in email.
Richard, I don't know if you want to consider this for 7.4.1. If so then you'll need to apply the patch as I'm away for two weeks now. Note, this fix is about Pat's problem, also reported elsewhere, not Sam's (which is almost certainly a network issue).
- please consider applying this patch if it addresses sync problems with windows machines doing ntp server based time updates thanks. (Bug 14570)
Reviewing this, it is inadequate. Small changes will continue to be made each time JIFFIES_OFFSET_TRACKING_LIST_MIN is reached and then the tracking list is reset, meaning that JIFFIES_OFFSET_TRACKING_LIST_SIZE will never be reached.
== Auto-comment from SVN commit #29257 to the slim repo by ayoung == == https://svn.slimdevices.com/slim?view=revision&revision=29257 == bug 14352: Synchronization Failing Allow more rapid adjustment due to server clock reset.
So all the comments here are from people in the same building at Logitech. If there are any actual users seeing this problem, please comment with additional info so we can prioritize this bug. Thanks!
have been seeing occasionally but unrepeatable problems on 7.4.1 and beta7.5, I always run multiple players and always sync'd together. Controller, Radio and Boom (wireless), 2*SB3 AND 1*sbr (homeplug) NAS: WHS on Hp MediaSmart EX470 upgraded to 2G RAM problem has alway sbeen there - back to 6.3 but have since then upgraded router software, replaced router, moved to Homeplug, upgraded NAS memory all to some (percevived) beenfit. In parallel I have kept up with production releases and increased number of players. SBRadio replaced wired receiver - expected to use portability of Radio so left it wireless, may swap back until summer/battery available. Some changes driven by 'rebuffering' issues. I observe some players stopping and starting slightluy quicker than others, and ocasional loss of sync, usually sorted by Tonight, using IR to Boom, I stopped track, selected new and hit play. It played elsewehre but not on Boom, restarted track and Boom joined in. Have not found a repatable cause, but it seems more likely if I am doing something complex (changing to a test server) rather than just choosing and playing songs. Not sure if that will help you prioritise, if I get any useful evidence I will post it. Mitch
Thanks for that info, Mitch. I'll leave it as 'Major' but shift the priority for now. Also I added myself to the cc list.
Okay, this is happening right now, so I'll post all the info I have Summary:- stuttering started whilst added radio to sync (with Music IP scan running) stuttering continued after completion of scan and with Radio in 'off' mode. Details: Version: 7.4.2 - r30085 @ Tue Feb 9 03:06:29 PST 2010 Hostname: MusicWHS Server IP Address: 192.168.2.4 Server HTTP Port Number: 9000 Operating system: Windows Home Server - EN - cp1252 Platform Architecture: 586 Perl Version: 5.10.0 - MSWin32-x86-multi-thread MySQL Version: 5.0.22-community-nt Total Players Recognized: 4 1 SB3 wired 1 SB3 wired via homeplug 1 Reciever wired via homeplug 1 Radio wired via HOmeplug. All Synchronised together. I was listening happily with 3 players on and the Radio off (at the mains). I set a MusicIP scan running to validate some new music - no problem Then I changed rooms and switched Radio on at the mains. The radio then started to sync and all music became unstable:- playing short bursts on one player, stopping, slight continuation on distant player. Infrequent "waiting to sync" message on wired SB3 Frequent "Rebuffering Failed" on Radio. Occasional "CPU running hot message" from WHS tray icon! track is continually progressing but with the gaps taking around 3/4 times the track length to complete. album finished, so I restarted and problem persists. partway through track 1 - Music IP completed scan (no more WHS hot messages) around similar time (but was swapping from room to room - so can't say which occured first) the Radio only continued to play and both SB3's were frozen with no error message, buffer empty and elapsed time at 3:50 but NOT playing. receiver was also silent but with Bright White light. This continued until track 1 finished. When track 2 started, intermittant play, as before, until SB3's froze at 3:50 and Radio only continued intermittantly to end of track. Track 3 started with intermittant play from SB3 (no error messages) and Radio ("rebuffering Failed" messages)then stalled at 1:06 elapsed and silence everywhere - no error messages. opened web gui and noticed elapse time here was moving from 1:06 to 1:10 repeatedly. set player.source=info, player.sync=info and network.protocol.slimproto=info. used web gui to jump to track 4 which played until 1:20 before first stutters (no errors on SB3 but Radio rebuffering)and then continually stutttering used IR to skip to track 5 (at about 13:13pm to x-ref logs) via SB3 and first stutter at 0:12 then good to 00:41 and then back to few seconds on and off at 13:20, usedcontroller to skip forwrad a track, almost immediate stuttering Used Controller to choose new album, almost immediate stuttering. Switched radio to 'standby', via its front power switch, pause of about 5 seconds and then 3 remaining players continue in sync until 3:50 minor stuttering. Fine to track end (04:50) and for next track to 02:12. So still stuttering but not as bad as when radio was on. Used controller to select new album, okay til 01:47 with "waiting to sync" on SB3. Disconnected radio from power. 15 minutes stutter free.
Created attachment 6518 [details] Log file from comment 20 - stuttering in action should have added to comment 20 that all problems stopped once radio was powered off.
This is still happening for me, on two or three Receivers and one Boom. One receiver is wired, all other devices are wireless. It _seems_ as though setting all the wireless devices to transcode to 320kbps helps a bit, but still get occasional drop outs where one device (typically a Receiver) will just stop playing until the next song starts, if then.
(In reply to comment #20) one detail I forogt is that this was all stremaing FLAC.
Moving P3 and lower bugs to next release target
Just wanted to note that I have added an Asus WL-330GE wireless widget in adaptor mode (allow wired-only devices to connect to a wireless network) to my Receiver that was having problems on the wireless, and reconfigured the Receiver to use its wired connection. All equipment is in the same place, but now the Receiver thinks it has a wired connection. Notably the Asus box is sitting right next to the Receiver, and so should have a similar wireless connection back to the router. Since doing this (about 24hr ago) I have not had any drop outs, after suffering through a particularly bad week of them. I have one more Receiver on wireless that also had problems with drop outs, I guess I'll get another one of these adaptors and see if it cleans up that one as well. Will try to report back with long-term status.
Mitch, I have examined your log from comments 20/21. My guess is that 00:04:20:06:56:4a is the wired SB3, and the other 3 are via homeplug. I suspect that the homeplug network is simply being overwhelmed. To stream FLAC typically requires a bandwidth of 1Mb/s (100MB/s). It is clear that the Squeezebox Radio is having trouble getting data fast enough, although all three of 00:04:20:12:67:3b (SB3), 00:04:20:16:1b:6b (Receiver) and 00:04:20:26:10:44 (Radio) are struggling to some extent. There is a network test function available. See Settings / Advanced / Network Test on Radio and Controller or Settings / Information / Network Test for the SB3s. I suggest you try running that at 1000 or 1500 kbps, first for one player and then adding each other player. Unless you can comfortably sustain 1000kbps for all four players simultaneously then you will not be able to have them all playing FLAC at the same time, synced or not.
Alan, thanks for the reply - this is only an occasional problem for me (and often I add a wireless Boom sync'd to the other four players) You're correct on wired SB3 MAC address. My Homeplug if advertised as HD 200Mbps (netgear HDX101) and its only monitoring software shows worst performance of 55Mbps, which is theoretical overkill. Trying the network test has been problematic (but this may be me) and is slightly off-topic for the sync issue. 1) "Network Test" was missing from the radio menus until I power-cycled it (press and hold standby button)!! It then disappeared again, and also disappeared from SBC! 2) The message on radio and SBC says 'Press Right for Help' it really means press centre/click wheel. 3) The wired SB3 alone starts to drop to 98% at 4000 kbps 4) Running a network test seems to drain the SBC battery extremely quickly! 5) Initiating a network test at 1000kbps on the homeplug SB3 (00:04:20:12:67:3b) gives different results depending on whether started from SBC or SB3 (though when started from SBC - the SB3 and SBC displays seemed aligned) 6) initiating a test of wired SB3 from SBC allows you to scroll test speed up, but then it resets back to 1000kbps. The SB3 display starts switching between the network test results screen and my 'off/standby' screensaver (date/time). 7) the SBC/radio report differently from SB3 (harder to summarise SBC output) 8) the SB3 test cuts after about a minute, making it difficult to test all devices simultaneously. When you suggest "... running that at 1000 or 1500 kbps, first for one player and then adding each other player", how do you add players to a network test? Is this their presence on network, being "on" in network or being "on and sync'ed"? When you say "can comfortably sustain 1000kbps", do you mean at 100% or average above 90%, or no red bars (on SBC) thanks again Mitch
(In reply to comment #27) > My Homeplug if advertised as HD 200Mbps (netgear HDX101) and its only > monitoring software shows worst performance of 55Mbps, which is theoretical > overkill. In theory. My own experience with homeplug has been somewhat poorer. > When you suggest "... running that at 1000 or 1500 kbps, first for one player > and then adding each other player", how do you add players to a network test? > Is this their presence on network, being "on" in network or being "on and > sync'ed"? I was imply suggesting you navigate to the relevant part of the UI in each case: Settings / Information / Network Test using the IR remote for each of the SB3s, and Settings / Advanced / Network test on the Radio and on the Controller (for the Receiver). Sync has nothing to do with this test. > When you say "can comfortably sustain 1000kbps", do you mean at 100% or average > above 90%, or no red bars (on SBC) Well I was thinking of 100% at 1000kbps but I suppose 95% would be ok.
okay, I was confused because initiating any test when sync'd cut the stream to all players. If they're unsync'd then the others are unaffected. kicked off the test on 4 players at 1000kbps SB3 wired: 100% Radio: about 95% SB3 homeplug: 60% receiver: probably 60% This looks horrible and suggests FLAC would never work yet it will stream sync'd FLAC for weeks on end, and is now immediately after the test. This isn't even acheiving 4Mbps (combined) which is a factor of 10 worse than the homeplug thinks - can it really be that far out? (this is way worse than 'somewhat poorer').
What happens when you try only two or three players in parallel? It may be that it is ok with just a couple of players but things fall apart with multiple players. It would be interesting to know the maximum speed you can get for just a single player running the test: either the SB3 or Receiver on homeplug - the SB3 would be the better test, and controlled using IR, not the Controller.
repeated twice with same results 1) SBC off, other players in standby with homeplugs on 2) SBC off, other players off and homeplugs off. SB3 (Homeplug) still performing average 60% but quite variable (watched actual fluctuate between 100% and 20% before test ends) Alan, thanks for the time on this, remember that my performance is okay 95+% of the time and only occasionally do I see a sync issue. The network test results suggest I should see far worse, which leaves me scratching my head.
The trouble with all these tests is that they are not the way things actually behave when playing music. Squeezeboxes have a certain resilience to transient network problems, which often allows them to get by when a pure test would suggest that they would fail. When synced one has much smaller margins for error, as well as a significantly increased server and network load - all players start a new track at the same time, for example. So it looks like you Homeplug network is simply not up to it. Maybe you could actually get further with Wireless connections for one or more players (if you have a wireless access point), or you could try running real network cables if that is an option. I don't know what kind of electricity distribution you have in your building but here, in Switzerland, we have 3-phase supplies, with the sockets (outlets) in different rooms possibly on different phases. This can play havoc with Homeplug-type systems. But, as I alluded to earlier, my experience is that one does not get even what the devices report that one should, never mind what it says on the package. Good luck.
Thanks Alan, All my Homeplugs are on the same phase and electrical distribution board. They've even streamed video in parallel to FLAC, so I think they're okay (and significantly better than wireless). I suspect that some combination of network/player/server error increases the load beyond the network capacity and then it can't recover. If all's good it stays good - these 'events' are so intermittant they may be difficult to ever understand. Since it's 99% okay, I'm happy, I posted to try and help add data to the OP - not sure now whether any value remains in that, so let's leave it unless there's more input from elsewhere Mitch
Mitch's report was not related to the original. The original problem has not been reproduced by others and may have been related to earlier builds of Radio firmware.