Bug 7935 - Audio stutter on dynamic playlists / random mix
: Audio stutter on dynamic playlists / random mix
Status: CLOSED FIXED
Product: Logitech Media Server
Classification: Unclassified
Component: Streaming From SlimServer
: 7.0.1
: PC Ubuntu Linux
: -- normal (vote)
: 7.x
Assigned To: Unassigned bug - please assign me!
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-04-22 00:56 UTC by Mike
Modified: 2009-07-31 10:19 UTC (History)
4 users (show)

See Also:
Category: ---


Attachments
Example Server Log (35.08 KB, application/octet-stream)
2008-04-22 18:58 UTC, Mike
Details
Example #2 Server Log (7.0.1 - 19047) (14.08 KB, application/octet-stream)
2008-04-22 20:29 UTC, Mike
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mike 2008-04-22 00:56:46 UTC
Alan requested I split this off from Bug 7695 :

------- Comment #17 From Mike 2008-04-21 06:31:02 [reply] ------- 
SB2 & SB3 are both wireless.  SC runs on 1.5GHz P4 with 512MB RAM running
Ubuntu 7.10 with the music on local 750GB IDE drive. The SC server is wired to
a Belkin pre N router.  I will try increasing the startup delay and report
back.  

I am suspicious about there being more to this as it also happens when only
using my SB3 (although with the Controller).  The stutter occurs when a new
song begins (a few notes and then the pause) while at the same time SC is
adding a new song to the dynamic playlist and removing the oldest of the
recently played songs (I have it set for 10 upcoming and 5 recent), loading
appropriate artwork and sending these changes (playlist & artwork) to the
Controller.  The CPU always spikes at that point regardless of whether there is
a stutter.  Since the behavior is inconsistent when I am not syncing, I can't
be certain, but it appeared to not stutter (or stutter less) when I did not
have any browsers pointing to SC (more places to reflect updates to playlist &
artwork).  When I observed this yesterday the browser was open on a wired PC,
but there have been other times when the browser is open on wireless PC.  One
thing I am pretty certain of is that since I started using the Controller it
stutters more often.  Is it fair to say that a sync'd player is another
playlist that has to be updated?  It would be interesting to test if it would
stutter more when there are more Controllers connected to SC with just one
player.

SqueezeCenter Version: 7.0.1 - 18946 @ Sun Apr 20 00:41:25 PDT 2008 - Debian -
EN - iso-8859-1
Perl Version: 5.8.8 i486-linux-gnu-thread-multi
MySQL Version: 5.0.45-Debian_1ubuntu3.3 
Platform Architecture: i686-linux
------- Comment #27 From Mike 2008-04-21 22:08:33 [reply] ------- 
(In reply to comment #19)
> I am reasonably confident that this is the result of (at least one) player in
> the sync group being starved of data immeidately after it starts to play a
> track.

Alan, what is the explanation for when this is happening when I am only playing
my SB3 by itself (no sync'd players)?  I am using a Controller.  Is that
similar to syncing with a player? It plays gapless with the Controller which
made me believe it is not similar to syncing.  
------- Comment #28 From Alan Young 2008-04-21 23:03:15 [reply] ------- 
(In reply to comment #27)
> Alan, what is the explanation for when this is happening when I am only playing
> my SB3 by itself (no sync'd players)?  I am using a Controller.  Is that
> similar to syncing with a player? It plays gapless with the Controller which
> made me believe it is not similar to syncing.  

No, I don't think that this can be the same thing. Do you get this with all
tracks or just the 'first' track? With the 'first' track I can see that
wireless network congestion / file-system delays on the server could sometimes
lead to starvation but this should not happen for subsequent tracks; SC starts
stream the new track when there are still 10s worth of the current track left
to play. I cannot see how the set of things that happen at track start could
cause that much delay - we are really talking about problems in the 100-1000ms
range here. How long is your CPU spike?
------- Comment #29 From Mike 2008-04-21 23:41:54 [reply] ------- 
(In reply to comment #28)
> No, I don't think that this can be the same thing. Do you get this with all
> tracks or just the 'first' track? With the 'first' track I can see that
> wireless network congestion / file-system delays on the server could sometimes
> lead to starvation but this should not happen for subsequent tracks; SC starts
> stream the new track when there are still 10s worth of the current track left
> to play. I cannot see how the set of things that happen at track start could
> cause that much delay - we are really talking about problems in the 100-1000ms
> range here. How long is your CPU spike?

Never the first track.  It happens on subsequent tracks when I am playing a
Dynamic Playlist.  It stutters 1-5 seconds typically a few notes after the next
song starts.  I am only looking at the CPU spike on the Ubuntu System Monitor
so it is not that precise. I would guess it is about 1-2 seconds.  The spike
occurs every time the song changes regardless of whether it stutters.  This is
the same time that tracks are being added and removed from the playlist.  When
I sync SB2 & SB3 (both wireless) and have the Controller on, the stutter
happens more often then not (more than 80%).  With one player and the
Controller on or both players sync'd and the Controller off it happens less
consistently (might do it 3 songs in a row and then be fine for several songs).
 With one player and the Controller off, I think it has stuttered, but very
infrequently (not 100% sure). 

Maybe I should look at upgrading my server.  I would have thought that 1.5GHz
P4 running Ubuntu would be plenty for SC.
Comment 1 Alan Young 2008-04-22 01:11:22 UTC
Your server should be fine.

Logging at level player.source=info would be a useful start. I have been unable to reproduce this with my local setup.
Comment 2 James Richardson 2008-04-22 08:35:05 UTC
Mike:  What format is your music in?  MP3 / WMA / FLAC ??
Comment 3 Mike 2008-04-22 18:58:35 UTC
Created attachment 3274 [details]
Example Server Log

I have 10,952 tracks and 99%+ are FLAC.  There are < 50 MP3 files.  In the attached example LOG snippet it was all FLAC.  I was playing a random dynamic playlist that is set to include 5 recently played tracks and 10 upcoming tracks.  I am theorizing that my stutter is related to the removal of oldest track as it has never stuttered until after it has completed the 6th track in a newly generated playlist. I started the snippet of the attached log at that point.  In each case when it stutters it will be about 10-15 seconds after the song begins.  I use the dynamic transition plugin to switch crossfade on for 10 seconds when subsequent songs are not from the same album.  When it stutters it is always close to after the crossfade has completed.

Here is my notes of which tracks it stuttered:
Finishing playing Caravan (Michel Petrucciani)
19:24 Starts playing For All We Know (Oscar Peterson) -- OK
19:33 Starts playing Peace (George Winston) -- OK
19:37 Starts playing There's a Boat Dat's Leavin' Soon For New York (Ella Fitzgerald & Louis Armstrong) -- Stuttered
19:41 Starts playing L'Antillaire (Rich Franks) -- OK
19:49 Starts playing Being For The Benefit Of Mr. Kite (The Beatles) -- Stuttered
19:52 Starts playing Ma Mere (Ludovico Einaudi & Ballak Sissoko) -- OK

It did not continue in a pattern of every other song.  It Stuttered on the next 3 songs and then went several without stuttering.  Let me know what else I can provide to help.  I have a much newer machine (Athlon X2 4800 4GB Ram) that I could install an Ubuntu partition and set up as an SC server to test if my hardware is the limiting factor contributing to this issue (that will have to wait to the weekend).  I have been running the same type of random dynamic playlist fine for several of the previous 6.5.x and I think even 6.3.x without any stutters.  The two things that have changed was my upgrade to 7.0 and using the new Controller (if that is even a factor).  Thanks.
Comment 4 Mike 2008-04-22 20:29:01 UTC
Created attachment 3276 [details]
Example #2 Server Log (7.0.1 - 19047)

Well, I blew my own theory after updating to 7.0.1 - 19047.  I started a new random dynamic playlist and it strated stuttering consistently with the 2nd song (see Cannonball Adderley).  So at this point no song is being removed from the playlist -- just added.  By the way the next track that started after Keith Jarrett in this LOG did not stutter and is still playing as I post this.
Comment 5 Alan Young 2008-04-24 00:14:43 UTC
Thanks for the logs and the detailed descriptions. Do you know what the dynamic transition plugin does? Does the problem occur without it?
Comment 6 Mike 2008-04-24 12:01:49 UTC
(In reply to comment #5)
> Thanks for the logs and the detailed descriptions. Do you know what the dynamic
> transition plugin does? Does the problem occur without it?

Sorry, it is actually called Transition Updater now. Here is the description from the pluglin list on the wiki:

"This plugin looks at the songs before and after the current song in the playlist and determines whether an album is playing or if random tracks are playing. After that determination, it sets the Squeezebox's transition behavior (crossfade, fade, etc) appropriately. (formerly Dynamic Transition Updater)"
http://wiki.slimdevices.com/index.cgi?PluginsAudio

I am pretty sure it was happening without the plugin, but I'll test it again to be certain.
Comment 7 Mike 2008-04-24 19:43:39 UTC
When running with without the Transition Updater plugin loaded ,I still had the stuttering problem but it happened less frequently and significantly enough so that it would appear to not be coincidence.  However it did still stutter close to 10% of the time.  I don't know if the Transition Updater plugin is trying to do something when a new song starts or more likely when a track is added to the playlist, but this all still leads me to believe that the music stutters because the server seems to have competing tasks going on at once (of course this is all coming from someone who has does not understand the how the server works or anything about the code).  Without the Transition Updater Plugin it happens less often I am assuming because it was one less item to process at the same time.  If my assumption is true is it possible to always ensure that in such situations the streaming of music has uninterrupted priority over any of these other tasks that are happening at the same time?  Pre 7.0, from my naive perspective, the server seemed to always behave this way.  Sometimes the display would lag, but the music would never be interrupted.

Is there anything else I can provide to help here? 
Comment 8 Andy Grundman 2008-04-24 20:40:30 UTC
Does this problem still occur if you remove all 3rd party plugins?
Comment 9 Mike 2008-04-26 10:21:52 UTC
It does not appear to.  I was NOT able to identify a plugin as the culprit either.  The inconsistent nature of the problem makes it impossible to be certain, but I think my testing still supports my theory that when more things are going on when a new song starts, there is a increase in the occurrence of the stutter.  As mentioned previously, on 6.5.x I was running the same setup without any issue however as you know, upgrading to 7.0 meant new versions of every plugin I use also. At this point, I am not sure how to identify the true cause of the problem. There is definitely a correlation to the CPU load at the time it happens.  Is there anything that can be done further to ensure that streaming music always has the highest priority so it will not pause despite what else the server/plugins are trying to do? Or is my only option to live with out many of the plugins my family has come accustom to?  Maybe I can see if upgrading my server hardware will help.  I am currently running Ubuntu 7.10 on a 1.5Ghz P4 with 512MB RAM.  Any thoughts on if you think upgrading my hardware would help my situation. Thanks.

Comment 10 Andy Grundman 2008-04-26 10:26:25 UTC
The server is single-threaded so a poorly written plugin or one that does heavy database things, could stall the music and everything else.
Comment 11 Mike 2008-04-26 17:59:10 UTC
The Controller is another variable here also.  With the Controller off and all of my plugins loaded it also does NOT stutter.  Turn on the Controller and it stutters.
Comment 12 Chris Owens 2008-04-28 09:41:53 UTC
QA to reproduce.
Comment 13 Mike 2008-05-02 11:51:59 UTC
Some additional observations:
1) The stutter problem appears more frquesnt the longer the server has been running since the last reboot.  As a reminder, I am running Ubuntu 7.10 (vanilla desktop install with all the latest updates).  In addition to SC, I think the only thing I installed was an NTFS-config package.  After reboot, it stuuters less frequestly play the same playlist.  I had left both FireFox and the System Monitor running for several days in additon to SC when I observed this.

2) My box is set up to dualboot to WinXP so I tried that and after installing the latest 7.0.1, I still got the stutters.  Hard to be sure given the inconsitent nature of the problem, but it did not appear to be better or worse with XP vs. Ubuntu.
Comment 14 Alan Young 2008-05-05 01:47:10 UTC
In addition to / as a complement to, I see a couple of specific targets:

1. The general increased emphasis on artwork in SC 7.9, especially with the controller;

2. The removal of forked streaming for things like artwork - but Andy, I thought that you were of the opinion that that was not working anyway.

I suspect that there are several different causes here, resulting in the same apparent behaviour. Heavy DB activity caused by a plugin and perhaps exacerbated by remote (possibly via wireless network) filestore; loading of artwork triggered by dynamic playlist updates; possibly a firmware bug (see bug 7918).
Comment 15 Alan Young 2008-05-05 08:55:11 UTC
So, you are playing FLAC, and I guess that this should be 50% or better compression => 706kb/s or 86KB/s (assuming 44100 samples/s). A (non-synced) player buffers 255KB before it starts playing, so this should be about 3s of play time. The server must be suffering quite a hefty stutter to starve the player for so long.

Unless you have a lot of metadata at the start of the FLAC file which the player has to skip before it gets to the real audio. Do you have embedded cover art?
Comment 16 Mike 2008-05-05 12:15:26 UTC
All of my cover art is stored as folder.jpg.  I use the typical text tags only.  

The stutter always occurs after you hear a second of audio from next song on the playlist, but I would say that it is probably less than 2 seconds and definitely less than 3 which I assume means the server stutters well before the buffer is filled.

When crossfade is turned on the stutter appears to happen within a beat after the end of the crossover.  In fact, if watching the Controller screen, it will stutter within the same moment that the screen is trying refresh to display the new track.  The Controller display usually gets updated with the new song info and artwork before the music resumes.   
Comment 17 Erland Isaksson 2008-05-05 23:20:09 UTC
I've made a change in the Dynamic Playlist plugin version 2.0.beta11, it now adds tracks 60 seconds after the track change instead of immediately. So if those of you that had this problem due to the Dynamic Playlist plugin could verify if this solves the problem or not it would be great. 
Comment 18 James Richardson 2008-05-15 17:32:25 UTC
E(In reply to comment #17)
> I've made a change in the Dynamic Playlist plugin version 2.0.beta11, it now
> adds tracks 60 seconds after the track change instead of immediately. So if
> those of you that had this problem due to the Dynamic Playlist plugin could
> verify if this solves the problem or not it would be great. 
> 

Thanks Erland, for the update.

Mike: can you retest with SC 7.0.1 - 19705 and Erland's updated Plug-in to see if this issue still happens on your setup?

Comment 19 Chris Owens 2008-06-19 09:25:47 UTC
Erland, it seems like the original reporter has lost interest.  Do you consider this bug fixed?  Shall I mark it as such?

Thanks.
Comment 20 Mike 2008-07-29 11:14:02 UTC
Sorry - It is not that I had lost interest, it seems I stopped getting email notices until just now when the status was changed.  I had not seen the last two comments.  I am running 7.1 now and yes Erland's change has generally resolved the problem (99%).  Although rare, I still get some strange behavior a Dynamic Playlist has been running for a long time, but rather than stutter, in the middle of one song it will just jump to the next song.  It is very inconsistent and rare and I am not even sure it is related to this same issue except that I think it is happening about the same time that the dynamic playlist is being updated.  However I have not been able to catch it enough to even be sure that is the case.  So I am OK with keeping this resolved and opening a new bug if I am able to successfully reproduce this other problem a little more consistently. Since it happens so rarely I have not been trying too hard.  If these symptoms sound familiar an elusive problem that others have reported, I am always happy to try and help.

Thanks,
--mike
Comment 21 James Richardson 2008-10-10 10:19:13 UTC
Please retest the latest version of 7.2.1 with the latest version of the plug-in.  Reopen the bug if you still see issues.
Comment 22 Mike 2008-10-13 22:27:27 UTC
(In reply to comment #21)
> Please retest the latest version of 7.2.1 with the latest version of the
> plug-in.  Reopen the bug if you still see issues.
> 

I have been running 7.2.1 (upgrading to the latest nightly every week or two or if I encounter a problem) and have not have not had any issues related to this in quite a while.  I am running SqueezeCenter Version: 7.2.1 - 23448 @ Wed Oct 8 02:06:56 PDT 2008 - Debian - EN - utf8 currently.
Comment 23 James Richardson 2008-12-15 12:33:43 UTC
This bug has been fixed in the 7.3.0 release version of SqueezeCenter!

Please download the new version from http://www.slimdevices.com/su_downloads.html if you haven't already.  

If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.
Comment 24 Chris Owens 2009-07-31 10:19:42 UTC
Reduce number of active targets for SC