Bugzilla – Bug 1469
Lost connections to clients during iTunes playlist scan
Last modified: 2009-09-08 09:19:35 UTC
Client connections lost & 100% Server CPU usage during playlist scan portion of iTunes library scan. Problem has existed since 6.0 (see bug 1093 and bug 1153 for background.) I have 6 or more playlists with around 3,000 tracks each. Will attach iTunes XML.
Created attachment 474 [details] zipped iTunes XML
if SlimServer is to support playlists with large numbers of tracks, it will probably need to be a scheduled scan through playlists. currently, to save over all time ( and to avoid another reported bug of missing/empty playlists) the playlist scan is done in burst. have I mentioned that itunes is...annoying?? more and more, I find it severely flawed, despite all its percieved features.
hey it's great if you ONLY use iTunes!
I don't see how the missing playlists bug is related to scanning the playlists in bursts. Can we try not doing this?
would love to hear what you think it is. dan is working on a parser, but that will take some time. scheduler is painfully slow, so isn't the most desired thing to do for playlists unless absolutely necessary. as for the missing playlist (which are not this bug) that has yet to have confirmation that the loss is due to rescan at all. Until something can be reproduced (or at least more visibly indicated) by those willing and able to fix code, not a whole lot can be done.
Dan to take a look at speeding up the existing parser, if it's not hard. Otherwise, this'll have to wait until we have a new itunes parser.
New iTunes parser is not going to make it for 6.1.
James - does this still happen with the new iTunes code in 6.2 and/or using the 'scan playlists only' button?
I will see. What does 'scan playlists only' do? You guys said you couldn't do that for iTunes.
Not sure where it was said that it wasn't possible. All that is mentioned here is a scheduled playlist scan (scheduled, means that it loops through one item at a time sharing with other tasks via the task scheduler). Given the pending rewrites, which are now in the trunk, it didn't make sense then to make those changes.
what I wanted to know is does 'scan playlists only' scan iTunes playlists or just those in the playlist directory. Someone said this wasn't possible because the iTunes playlists refer to the id's from the library - which is entirely reasonably.
palylists only seems to simply re-import the external playlists from the db, as oppsed to rescanning. Internal playlists are the ones in the playlistdir. This is still very new, so hard to say what the new parser has made possible as this progresses.
James - that checkbox scans both iTunes playlists and the playlist folder. The new iTunes parser has made this possible.
In a nutshell: it's worse than 6.1.2 To be more detailed; 6.1.2: start music, hit rescan. music stops immediately, server hits 100% CPU for about 10 seconds. iTunes music scan then starts, as does music. CPU still at 100%. every couple of seconds the database commit happens and CPU drops - this is very regular. Music continues. iTunes playlist scan starts. 100% CPU. Database commits are less frequent and have smaller drops in CPU. Music stops frequently. iTunes scan finishes, database cleanup runs, 100% CPU but music unaffected. 6.2: start music, hit rescan. music stops immediately, server hits 100% CPU for about 10 seconds. iTunes music scan then starts, music does not. CPU hovers 97-99% (less than 6.1.2) but music only plays for a couple of seconds after each database commit. Note database commits are less frequent and not as regular as 6.1.2 scan. iTunes playlist scan starts. From this point on same as 6.1.2 6.1.2 scan was about 30 seconds faster than 6.2 in total (including database cleanup etc)
James - I've made some tweaks to the iTunes parser in recent weeks - could you give it a go again? Thanks.
rescan performance is pretty much unchanged from my last comments. (There was a version a week or two ago that took significantly longer) about 6 minutes for the iTunes scan, however CPU remained at 100% for the whole scan and an additional 2 minutes after the scan completed. I've upgraded to a wired SB2 and this appeared to stay connected for the rescan, menu response reasonable. My wireless SBG disconnects for the whole scan. Didn't have any music playing.
I tried playlists only as well (for the first time). This scan took only 3:15 of 100% CPU time (iTunes scan itself 2:45). There weren't any drops in CPU usage for database commits here - didn't check the players. Will try again with music playing when I get a chance.
James: so this is happening with your SBG but not your SB2? That makes me think it's a networking issue. What's your network configuration?
I'm not 100% sure about the SBG, was just what I noticed once while watching TV when the rescan was scheduled. The SB2 seems to wait a lot longer before considering it's lost the server than the SBG did (the buffer is larger but that doesn't matter for the display)? I need to try the test again with the music playing though. On the network, I have an 802.11b Linksys router, server is connected wirelessly. One squeezebox (now SB2) wired to router. One squeezebox (now SBG was SB) connected wirelessly. The link between server & router is not great, I used to get dropouts quite frequently with the SBG wired. These seem gone away with the SB2's larger buffer.
an update. Wired SB2 stays connected for the whole scan. Wireless SBG suffers from short periods of 'lost connection' throughout the scan. Probably an improvement from the starting point. However I found music unplayable on the SB2 while the scan was running. Glitchy when playing and frequently stopped for seconds at a time.
ok, this is as good as we'll be able to do for 6.2, it'll be much better with some infrastructure changes in 6.2.
Please try the latest nightly from last night. iTunes should be taking less CPU and should be blocking for a shorter amount of time. Please reopen if you are still seeing issues.
is this change be in 6.5 as well?