Bugzilla – Bug 6308
No playlists, but scan for playlists running a very long time
Last modified: 2008-01-21 21:13:09 UTC
I have NO playlists in my music directory, but refer to same folder for both music scan and playlist scan. Music scan completes appropriately, but scanning progress pages shows playlist scan continuing for several minutes (not sure it has ever completed by the time I have turned off computer). Have seen this behavior in nightly builds for past week including 12/10/2007 version. Attempted to search for this bug, but did not find. Thanks
Please verify you really don't have any playlists. Some ripping program create them automatically. Do you see the same issue if you use a different folder for the playlists? As you don't have any, this might be a simple test to do. Thanks!
I can second this bug. I have four playlists. The scan of those playlists finishes in 28 seconds. I am then forced to sit through: "Merge Various Artists" (1547 of them), takes 5:04 "Database Cleanup #1" (31782), takes 1:34 "Database Cleanup #2", might have finished in 2 minutes or so, hard to tell. So basically 9 minutes to scan my four playlists, with a lot of scans I didn't ask for.
You are forced to sit through those because they are required post-scan cleanup phases. All of the various prefs make these step necessary. No post-scan organisation means no prefs means we're all using iTunes.
I'll have to take your word for it that all that stuff is necessary when doing a playlist-only scan. It is somewhat misleading, though, to call it "playlist only", if there are going to be a bunch of other housekeeping tasks that are also going to run. I'm sure you can understand why someone with four playlists -- or none -- might think such a scan would take very little time, rather than roughly ten minutes. Can you tell me what a playlist scan does that requires all of this other cleanup? Not having seen the code, I would have assumed it simply verified the existence of the files the playlist calls for, and that's about it.
(In reply to comment #4) > I'll have to take your word for it that all that stuff is necessary when doing > a playlist-only scan. It is somewhat misleading, though, to call it "playlist > only", if there are going to be a bunch of other housekeeping tasks that are > also going to run. I'm sure you can understand why someone with four playlists > -- or none -- might think such a scan would take very little time, rather than > roughly ten minutes. > Can you tell me what a playlist scan does that requires all of this other > cleanup? Not having seen the code, I would have assumed it simply verified the > existence of the files the playlist calls for, and that's about it. I see that possibly I wasn't clear. I thought the OP was talking about a playlist-only scan. He was not, but I am.
*** Bug 6370 has been marked as a duplicate of this bug. ***
I opened bug 6370 because this bug, 6308, did not seem to be discussing the same issue, as you can see from my comments in the thread above. This bug seems to be a guy complaining that the playlist scan AS PART OF A FULL SCAN takes a long time. As part of a full scan, of course all those database cleanup scans are being done, and so they should be. My complaint in 6370, which you closed as a duplicate, is that a playlist-only scan takes a long time, and seems to do a lot of unnecessary scanning, including all those database scans. Those complaints are not the same. The response I got in this thread from kdf was something about how if we didn't do all that database cleanup we couldn't have any preferences. That may be true for a full database scan but I do not see why it should be true for a playlist only scan. I repeat below my questions from bug 6370: ***************** Thanks Michael. At the risk of trying your patience, can you explain to me why doing the playlist checks you mention should require: "Merge Various Artists" (1547 of them), takes 5:04 "Database Cleanup #1" (31782), takes 1:34 "Database Cleanup #2", might have finished in 2 minutes or so, hard to tell. From the outside, I would have thought that a user-initiated playlist-only scan would have meant something like this: "I have created this playlist. Please add it to your list of playlists." It apparently also seems to mean: "Please make sure every song in the playlist is present in the database so you can play it." which is fine, but I would not have thought it _also_ meant "and while you're at it, conduct an eight minute cleanup of my database." and I'm not completely sure why it should have to mean that. Especially when those other database cleanup scans take place AFTER the playlist scan (if the messages to the user are to be believed), meaning that the playlist scan doesn't even benefit from scanning a newly clean database, if that was supposed to be the point. Any explanation appreciated. I tried to look at the source code but unfortunately my programming skills are too rusty, and I couldn't reason it out for myself.
Andy: any idea what's going on here?
Andy: can you comment?
Matt - could you provide us with some information about your system? Collection size, hardwere, OS used etc.? Those times indeed are long. Long enough to do a full scan - on a reasonable machine.
>>Matt - could you provide us with some information about your system? << Sure, I've provided that info below. However, let me mention again that for me, the issue is not why these database cleanups take so long; I have a lot of data. It is why they need to run _at all_ when the user requests a playlist-only scan. ****** 1551 albums with 31827 songs by 786 artists. Pentium 4, 2.53 GHz 1 Gb RAM Windows XP Pro, SP2 Two 250 Gb drives set up as dynamic volumes, C and D C has a shortcut over to D so that both are scanned by SqueezeCenter Machine is used pretty much as a dedicated music server. McAfee VirusScan is running, which seems to gobble a certain amount of CPU time. In Task Manager the "Commit Charge" reads 428M/2460M. Peak commit charge is 519 Mb, so I don't think adding RAM would make it faster. ****** Please let me know if there's anything else I can do to help.
rweyand - could you please provide some more information as well? I would expect long scan times if you had a large collection, spread across a large folder tree. What happens if you unset the playlist folder (as you don't use it)? Matt - as you've answered your own question yourself a bit earlier: "Please make sure every song in the playlist is present in the database so you can play it." That's the reason. Please use the forums if you have questions about the implementation, not the bug tracker. Thanks.
Will look at this post-7.0.
ok, so I don't forget this. Forum talk got sidelined by the optimise phase. I've had a look at the stages: Merge VA: not sure we want to get rid of this, as any processing of playlists is probably going to need the VA processing in order to match up all of the associated metadata. Then again, with no new metadata, I'm not sure why there is a new list of VA info to deal with, but then it may be more work to check each against the existing data in order to collect a pile of "new only" data. It seems to initially grab all albums no marked as compilations, so in theory things can be made much faster when the compilation tag specifically states 1 or 0. Still, it is better as a post process than in-scan as it avoids stalling the main data collection. Cleanup1: walks the track list to remove any tracks marked as stale. This is a needed step for garbage cleanup, but perhaps not always on a playlist only scan. It should, however, be run at some point every so often. Cleanup1: same as 1 but this time it walks the contributor, album and genre tables (usually much smaller of course than tracks). Same needs as above. As the scanner is a separated task, I'm not sure why the "time" is such an issue. It shouldn't be getting in the way of playback. Perhaps one step may be to get rid of all the browse-level warnings that suggest you cannot get a proper browse due to scanning when it is in post-scan sections. The added cpu usage is already controllable with the scanner priority settings. Having said this, I'm not sure this is what the original report is about. 9 minutes doesn't seem in line with 'never seems to finish'. However, without details from the original poster, it's hard to suggest whether it's a post-scan problem or not. Initial read seems to point the finger at the playlist section of a full scan, which would be something very different from what we are discussing now. Ideally, we need that 'always on' background scanner to do the stale entry cleanup. No, I'm not volunteering. I've finally realised that I really can't afford the time any more.
Browse Music Folder is fast so I assume it is not doing all these post scan steps. If not necessary for BMF, it would seem not strictly necessary for rescan playlists. This is especially frustrating because in the typical use case, the playlist is referring to music in your library which hasn't changed. Even if some track tags have changed it is probably not significant; otherwise you would have done a "rescan new and changed music" or a "clear and rescan" already. Since playlists just reference tracks, tag changes to a track without running the post scan steps should have minimal impact. At worst maybe the track is incorrectly shown as or not as VA. I can deal with that. Likewise if I reference tracks outside my main music library. My real hope would be to not have to tell SC to rescan playlists - it should just be done seamlessly and lazily as you browse them based on modified timestamps. But clearly to get to that point we first need to make rescanning playlists much faster.