Bugzilla – Bug 3161
Need more aggressive auto-retry for internet radio
Last modified: 2011-05-11 09:35:10 UTC
I don't know if the server does any auto-reconnect for internet radio streams. But I do know that streams do drop. The log says can't read file - which I assume is a read failure from the stream socket. Playing the same stations with Winamp, I had similar drops once and a while. Winamp in repeat mode retries, and this worked well for me. It would be nice if slim also retried. I think this needs some delay, some sort of back off and a give up after a while.
server settings->networking, radio station timeout not help?
Its not, as far as I can tell, a timeout. This is a typical log: 2006-03-16 10:35:04.3358 Opening connection to http://audio-mp3.ibiblio.org:8000/wcpe.mp3: [audio-mp3.ibiblio.org on port 8000 with path /wcpe.mp3 with timeout 5] 2006-03-16 10:35:04.3648 Request: GET /wcpe.mp3 HTTP/1.0 Accept: */* Cache-Control: no-cache User-Agent: iTunes/4.7.1 (Windows; N; Windows XP; 586; EN; cp1252) SlimServer/6.2.2/6583 Icy-MetaData: 1 Connection: close Host: audio-mp3.ibiblio.org:8000 2006-03-16 10:35:04.4094 Response: HTTP/1.0 404 File Not Found 2006-03-16 10:35:04.4097 Invalid response code (404) from remote stream http://audio-mp3.ibiblio.org:8000/wcpe.mp3 2006-03-16 10:35:04.4120 ERROR: Couldn't open song. 2006-03-16 10:35:04.4528 Opening connection to http://audio-mp3.ibiblio.org:8000/wcpe.mp3: [audio-mp3.ibiblio.org on port 8000 with path /wcpe.mp3 with timeout 5] ... Note that the server reponded - it just said no. Slim immediately tries a second time, and gets the same result. Or am I misinterpreting this?
error 404 is a "not found". That is supposed to indicate that the url is not available. automatically retrying would certainly have to be restricted, since a 404 IS supposed to mean "move on, nothing to see here". If it is due to the provider being temporarily overloaded, I doubt they would be very happy with being hammered by additional attempts to connect. However, just my opinion. Thanks for the log. That clears things up.
The Winamp forums warn about hammering the server. Winamp can recover from the dropped connections on the stream in this example, by the way. That's why I thought that a sequence of increasing delays would be needed. You certainly don't want to just keep retrying as fast as you can. In fact, that may guarantee you never connect. From the other side of the problem - when it happens I push play and reconnect. And mutter - why couldn't the software have done that. Note that there is a delay in this retry :-)
Right, and when the server waits for 30 seconds...you still hit play and try again saying "why is this software so slow" silly argument. agressive is the wrong choice of word for this, period. feedback in the now playing page perhaps, reporting "no response, try again later" is appropriate. similar for player
I think arguing about uzs of the word agressive is silly. Lets stick to the issue. When an internet stream drops, I need to intervene manually to get slim to try again. This behavior is not the same as, for example Winamp in repeat mode which tries again. Repeating what I said above, trying again must be done intelligently or you will swamp the server and or cause other troubles. BUT - trying to reconnect automatically for a reasonable amount of time is nicer behavior than forcing the user to press a button again and again. I agree with the useful part of the last comment - having the slim display show "trying to reconnect" when this is happening would be useful. And of course - pause/stop must stop the reconnect attempts.
what winamp does is irrelevant. I, personally, cannot support a retry of anything less than 30 seconds. even with that, slimserver got banned from slashdot a while back for simple rss feed retry violation. Even at 30 seconds, there will always be some idiot who thinks that is too slow and just keeps hitting play. And before you think I'm being harsh, I AM that idiot. However, behaving in polite way for automatic helps for every single user that isn't as impatient as myself. Let's not sink into lowest common denominator. Leave that to winamp.
I'm not following your arguement. Today, slim, I believe, tries once (does it?) and gives up. Why doesn't that encourage the user to press play again, snce the music has stopped? And again. And again. You MUST assume that the average user has NO knowledge of the retry rate issue. So why isn't it better for slim to retry in a controlled way, with appropriate back off? And with an on display indication that would encourage the user to wait rather than pushing play repeatedly? Am i missing something here? Are we talking past each other? The only reason I cite winamp is because, in my experience in practice, it works. I think it suffers from not having back off, and IMHO, it should. I would never suggest a simple retry loop.
I'm talking about having a retry after no less than 30 seconds, minimum. Users will still override (though I'd actually really like the idea of preventing manual retry aside from backing out and trying later) I think in principle, we are actually in agreement.
Andy - thoughts on this? Since you've been in radio land recently. thanks
I will give it some thought. Both mp3 and wma have direct streaming enabled for 6.5, so when a stream dies due to an internet dropout or server failure or whatever, SlimServer will get an underrun message from the player. The current response to underrun is to simply stop and/or move on to the next item in the playlist, but we could have it attempt to retry. I would suggest that this retry delay only happen if the radio station is the only item in the playlist, meaning the player would stop due to the underrun. If there are additional tracks in the playlist, the player should skip to them as soon as it gets an underrun.
Andy - going to assign this to you for now, since you've been Mr Radio
I'm very much looking forward to this enhancement!
I agreee that the retry-logic should only be run if there is only one track in the playlist. I think it would be a good idea to an exponential back-off strategy, so you would retry after 1 sec, then 2, then 4, then 8 and so on, until a maximum limit has been reached, aand then simply retry with that same interval. Maybe 4096 secs (a little over an hour). Also, any idea when this could be implemented? Listening to the same station all the time, and would love this 'self-repairing' on my SB.
*** Bug 5115 has been marked as a duplicate of this bug. ***
SC 7.3 may provide a little relief here, although not exactly what is requested. (a) if a playlist with just one entry is in repeat mode then one retry after a failure will be made; (b) failure of a radio station (any remote stream) at the end of a playlist will not defeat the repeat setting (bug 6615).
Downgrading to P4 given the enhanced retry functionality in SC 7.3
*** Bug 12287 has been marked as a duplicate of this bug. ***
I should say that this is much improved in 7.3.3. (Don't remember when it got better). For example, I just did the experiment of restarting my router while listening. Playback stopped, controller said "rebuffering", and playback resumed. It used to be that any network hiccup killed the playback. As far as I'm concerned, this is "good enough", but others apparently still want improvements. (I to NOT use any tricks like repeat mode or multiple playlist entries). I did the experiment both with an mp3 stream (direct connect, I believe) and a stream that used mplayer to convert an rtsp stream.
related bug 5115
A recent discussion in the beta forum: http://forums.slimdevices.com/showthread.php?t=64997
Change 27486 will attempt to re-establish a connection if it drops while playing. This is a single attempt: it will not continue retrying in background if the reconnection attempt fails.
How long does the single connection attempt try? When my ADSL2+ connection drops to renegotiate download rate, or requires a resync, it tends to take up to 10 seconds. I guess radio stations are buffered a little bit, but I'm wondering how short the actual interruption must be for the retry to work?
For SC, it is set by the server preference "remotestreamtimeout" which has a default of 5s. Note that a connection attempt is blocking; that is, SC does noting else while trying to connect. The player firmware has a fixed 10s TCP timeout. Note however, the the new connection attempt is only made once the player's decode buffer has been emptied, when it has about 10s left to play. Typically radio stations buffer around 15s. It depends upon the details of the scenario as to when the new connection attempt would be made: for example, does the connection get reset immediately the ADSL2+ drops or only after a 10s timeout?
*** Bug 5550 has been marked as a duplicate of this bug. ***
*** Bug 13070 has been marked as a duplicate of this bug. ***
I'd like to fan the flames on this bug. We have several customers who have either requested this feature, or this feature would solve the issue they called in about. We should continue to default to the behavior we currently use. An added setting under Settings > Network-- or something-- to allow for a more robust/aggressive retry for a radio stream would be grand. I also recommend we retarget this to 8.0, or something realistic where we could actually implement it.
this is an administrative shuffle on priority fields to help make better judgment on the top end of the priority list. P4->P5, P3->P4, and P2->P3.
== Auto-comment from SVN commit #28542 to the slim repo by ayoung == == https://svn.slimdevices.com/slim?view=revision&revision=28542 == Fixed bug 13361: Napster always repeats the same track Fixed bug 13967: On Boom: Short podcasts (and perhaps other media types?) repeat bits. Reopen bug 11888: Player should reconnect to a remote stream if the connection ends before the known duration bug 3161: Need more aggressive auto-retry for internet radio Revert change which attempts to resume fixed-length remote streams that appear to end prematurely, and which attempts to reconnect to remote streams that appear to be radio. The algorithms for detecting whether a stream has indeed ended prematurely are not sufficiently accurate in either case.
This did not happen for 7.4 and 7.5 is focussed just on Fab4 minimum requirements. Re-targeting for 8.0 and increasing priority.
Update hours
This bug has 20 votes, is support-important, is marketing-important, and I think represents poor customer experience. I'm bumping back up. We need to figure out a way to get this in 7.5.
So some of the execs at Logitech love the Radio so much, they have started using them in their offices. A couple have reported that the Internet Radio station they are listening to just "stops" after a few hours... They then hit play (or the preset button) again and it resumes, but it is a bit annoying.
as someone who runs the webstreams for WKPS as well as does other technical things for them, and uses slim to listen to them, i'd like to add an opinion: whether direct, or proxied by SBS or SN, sooner or later slim drops the stream, a lot more frequently than winamp for example does. slim's retrying of the stream only one time is, imo, very very stupid. i think the first time a stream is dropped, it should try again, as soon as is possible. i think it should wait a bit longer with each subsequent retry so as not to overburden a potentially stressed server, but these retries shouldn't be too far apart either. what i don't understand at all is this idea that it should only try once. and automatic retries should be as robust a methodology as pressing play manually is. its odd that the manual way seems to work when the auto way doesn't. perhaps adding a feature of a user configurable max stream time makes sense, so people can eg. say "5 hours after starting the stream, cut it off" if they are worried about it endlessly restarting automatically.
I've already discussed the following with the Technical Staff at Logitech many months ago. I have, in addition to my Squeezebox3, a Roku Soundbridge. I use it in an identical manner to the Squeezebox. Both are used strictly to stream radio stations. The Squeezebox will stop or will cease providing a sound stream output frequently - sometimes several times an hour. The Soundbridge has been operating *continuously* for many months without loosing output. Why can my old Roku unit work and the Squeezebox not? I have occasionally tuned the Soundbridge to the same station as the Squeezebox and the Soundbridge does not drop the radio station.
== Auto-comment from SVN commit #29370 to the slim repo by ayoung == == https://svn.slimdevices.com/slim?view=revision&revision=29370 == bug 3161: Need more agressive auto-retry for internet radio Create flag on remote stream obsolete.
== Auto-comment from SVN commit #29372 to the slim repo by ayoung == == https://svn.slimdevices.com/slim?view=revision&revision=29372 == bug 3161: Need more agressive auto-retry for internet radio HTTP Format-handler (not Protocol handler) is obsolete.
== Auto-comment from SVN commit #29380 to the slim repo by ayoung == == https://svn.slimdevices.com/slim?view=revision&revision=29380 == bug 3161: Need more agressive auto-retry for internet radio Reinstate restarting after unexpected end-of-stream on live streams.
From Alan in the bug meeting: The player doesn't know that the content is radio and not some other kind of content. It's just a URL. In many cases the URL is a playlist. We try to run the scanner on it, which expands it out into tracks. Sometimes the playlist is multiple URLs of tracks, but more commonly has fallback URLs if one playlist doesn't work. Alan says that in 7.5.0 there is a change that may improve the situation in cases where the player can tell if the stream has no definite length. This requires that at least one good connection to the stream has occurred. If the player cannot connect from the beginning, it will still not aggressively retry.
that sounds like the right way to handle it to me. so it will be part of 7.5 then? any idea when 7.5 will be released? (like, 1 month, 3 months, 6?)
Was this addressed in the 7.5.0 release? I know there are a few customers that were waiting for this to be implemented.
There was a change in 7.5.0 that means that there will be a retry if the connection fails after first connecting successfully. It is only a single retry and will occur immediately after the failure is detected. After a successful retry then a subsequent failure will cause another retry, and so on. A more-substantial change requires significant design work that was outside the scope for 7.5.0. Despite many people's assumptions, this is a difficult problem to solve within the architecture of the Squeezebox software. This work is planned for 7.6.
Alan, why does SBS or the hardware only try once? why can't it try 10 times, or infinitely? thats what most apps like winamp do... they just keep reloading, or automatically stopping and retarting on a given stream that is having a problem, (they sense the low buffer). most of the time, that works, as the problem is usually temporary, however it often takes more than one attempt to successfully reconnect. while this may not fix the underlying issue in bug 16170 it certainly couldn't hurt, (altho it may just mask it). i however don't have a SBRadio, so this bug here is important to my usage.
== Auto-comment from SVN commit #31371 to the slim repo by ayoung == == http://svn.slimdevices.com/slim?view=revision&revision=31371 == Fixed bug 3161: Need more aggressive auto-retry for internet radio If a radio station (non-finite stream) succeeds in playing for at least 10s and then subsequently fails then, after the immediate retry (existing situation), introduce a retry mechanism. This will retry the connection at successive intervals of 5s, 10s, 15s, 30s, 30s, ... until an overall timelimit is reached. This limit is 5 minutes if the station is the only (or last) item in the current playlist and 30s if there are more tracks in the current playlist (including via repeat-playlist if that is enabled), in which case playback will advance to the next track when the 30s timeout expires. The behaviour is slightly different between ip3k and SqueezePlay players, as ip3k players wait for the 10s connection timeout even when the TCP connection attempt is immediately rejected. Note: this will not attempt to retry a connection if the initial connection fails or if it does not succeed in playing for 10s. It will not retry pseudo-radio services such as as Pandora, Deezer, LastFM, etc. Introduce Client::connecting state to better track status of a player's readiness to connect to a new stream. Introduce Song:retryData, used opaquely to Song by StreamingController, to track progress of retry attempts. Introduce StreamingController::isRetrying(), mainly so that the display code can put up the right text. Slimproto: If a DSCO with a reason code (error disconnect) is received while playing and when the player (decoder) buffer is empty then this is probably because a stream attempt (strm-s) that was sent in response to an STMd (that is, either streaming the next track or retrying the current one) has failed with a connection failure. A quirk in the firmware (both ip3k and SqueezePlay) means that, when the output buffer drains, the expected STMu will not be sent in this case. So, if the player is playing when the DSCO(error) is reported, and the decoder buffer is empty, then tell the controller that the player has stopped, synthesising an STMu. StreamingController: It was already the case that _RetryOrNext(), called for a ReadyToStream event, would attempt to retry a stream that it could identify (had been identified) as radio. This now sets up Song::retryData with initial values so that subsequent events may retry again if the initial reconnect fails. The retry logic is encapsulated in _willRetry(). If the Song::open() called from _Stream() fails (only likely to happen for proxy streaming) then _willRetry() is called and, only if it returns false, is the usual failure path followed. Similarly for the two handlers called for a StreamingFailed event (_StopNextIfMore(), _SyncStopNext()), which is what one expects for direct streaming. If the current streaming Song has retryData, then _willRetry() will set up a timed retry using the usual NextTrackReady event callback. This means that any user intervention that changes the playlist state will cancel the retries because it will invalidate the NextTrackReady callback when it occurs. Successfully starting to play also resets (undef) Song::retryData().
== Auto-comment from SVN commit #31373 to the slim repo by ayoung == == http://svn.slimdevices.com/slim?view=revision&revision=31373 == bug 3161: Need more aggressive auto-retry for internet radio
== Auto-comment from SVN commit #31397 to the slim repo by ayoung == == http://svn.slimdevices.com/slim?view=revision&revision=31397 == bug 3161: Need more aggressive auto-retry for internet radio
== Auto-comment from SVN commit #31398 to the slim repo by ayoung == == http://svn.slimdevices.com/slim?view=revision&revision=31398 == bug 3161: Need more aggressive auto-retry for internet radio - wrong version
== Auto-comment from SVN commit #31399 to the slim repo by ayoung == == http://svn.slimdevices.com/slim?view=revision&revision=31399 == bug 3161: Need more aggressive auto-retry for internet radio
Closing. Re-buffering is working at set intervals when internet connection is interrupted. Using a SB Radio, setting the unit to play a streaming radio station (GotRadio Rockin' 80s), by switching the internet connections, I can break the audio when the unit queries for an IP address. This also is done when connected via Ethernet cable and unplugging the cable. The station will then have to re-buffer and will play again without any human in interaction. This is build 7.6.0 (32398) and SB Radio (7.6.0 r9432).