are these tests done with or without a music folder setting?  a playlist folder
with/without playlists? 

please attach a log of d_itunes (maybe d_info as well) from a rescan.

I believe I have no music folder defined.
A playlist folder is defined but is almost empty.
Will get some logs for you.

James - what build did you run this with?

I fixed a bizzare iTunes scanning bug yesterday around 6pm PST, so it's in the
2005-03-16 build / latest subversion.

It would have been the build from the 14th.

Ok - could you please try with the build from the 16th?

Thanks.

Created attachment 343 [details]
Edited log file from iTunes rescan: --d_info --d_itunes

I started SlimServer, it did a rescan then I triggered another from the web
interface. This is the start and end of the 2nd scan.

So you had two scans running at the same time?

The attachment you sent looks like it's much faster than your original
description, yes?

hmm...If this is a rescan,should it still be updating metadata for every file? 
I wonder if we're taking the itunes xml file time and assuming that if its
recnetly changed, then all data is new.  Maybe there is room to compare filestat
and not bother updating.  I'd have to remind myself if the debug messages really
mean we're updating or just reading out.  However, 6 minutes including playlists
of 9055, 2400, 3410, 1662 and a few smaller ones.  The logs come up short on
detail, but those are parsed as well.  There may be room for a bit of
improvement.  Your 420% rescan, was that based on the log or based on watching
the web interface?  I have had a report recently of the scanning warning staying
up for a very long time.  It possible that is still not 100% bulletproof. 
Personally, I always preferred the updating count, but that was just me :)

I'm going to knock this down from critical, since it isn't really crashing the
server.  But clearly, this is a good base to look for some tweaks.  I'm curious
if you get the same results from clicking rescan as you do from an automatic
scan when the itunes file shows as changed.

I'll also add that this new version does make more extensive use of the
scheduler to keep the cpu bursts down (playlists are the exception), plus we
read more thoroughly than before.  60% overhead on an initial scan doesn't seem
to far off to me.  The rescan, maybe something there that can be done.  If you
are seeing 100% cpu (I think you mentioned this), then we might have to slow
down the playlist parsing a bit to alleviate the stress.

I just tried last night's build (2005-03-16). 
There is no significant improvement from the previous build - 16 minutes for a
full scan, 7 minutes for a rescan.
However memory consumption has dropped to 60Mb!

Agree with KDF that
1) the 'rescan' looks like it just rereads all the tags. Surely should only do
that if the file timestamp has been modified?
2) would prefer lower CPU usage and a longer scan - the initial scan is fine
(except playlists) but the whole rescan uses 100% cpu and this is bad - clients
will lose connections.

At this point I'm wondering what I'm getting from using iTunes other than the
playlists - is it actually doing more work than scanning the filesystem?
Could there be a 'playlists only' option for the iTunes scanner?

I'd also suggest ignoring the 'Library' playlist because that doesn't give you
anything useful.

re: the tests, the scans were not run at the same time. 
I took the timings from the log file. 
Web interface was not open during scans.

well, itunes does provide a way to bring in tracks from multiple paths, where
the server does not without shortcuts or links.  you should not be seeing 100%
during the track scan, since it is sceduled.  However initial scan has a huge
amount of stuff going on so it is possible that the scheduler is fully loading.
I mean, even itunes wont play a thing when you first add a folder of 9000 songs.

playlists tend to load in burst.  this can present a problem for really large
playlists. (ie Library).  I saw that Dan recently changed the itunes import to
skip this one.  its redundant since any of the browse modes have an "all" item,
or pressing play on any browse category in the player UI will play all, just as
Library would. I'll look into the rescan merging, but I'm not sure I'll be able
to get to it before 6.0 release.  That aspect, thankfully, isn't as critical.

same results from rescan/timed scan.

Some suggestions:
1) Don't check for the readability of a file if it's in iTunes. 

2) use the 'Date Modified' key in iTunes xml to determine if a track has changed
(must admit I don't know how reliable this is) 

3) in updateOrCreateTrack, don't set attributes if they've not changed

4) if no attributes have changed don't do an update at all
 (maybe the last 2 are already done internally, I couldn't work out where
set/update methods come from)

Here are my timings for last night's version:
Full scan: 943 seconds (pretty much unchanged)
Rescan: 136 seconds!
So this is back to the rescan times for 5.4. 
Congratulations!

Slim.exe is using 36Mb after the 2 scans.

My only comment would be that the rescan uses almost 100% CPU for the duration -
could this be slowed down a bit to reduce it? 

Great. I'm closing this bug - and will refer your last comment to bug 1153