Bug 17450 - MusicIP import fails due to upper/lower case differences in the file path (Windows)
: MusicIP import fails due to upper/lower case differences in the file path (Wi...
Status: RESOLVED FIXED
Product: Logitech Media Server
Classification: Unclassified
Component: MusicIP
: 7.6.0
: PC Windows Server 2003
: P2 normal with 1 vote (vote)
: 7.6.1
Assigned To: Michael Herger
:
Depends on: 17457
Blocks:
  Show dependency treegraph
 
Reported: 2011-08-14 07:34 UTC by Bob Koure
Modified: 2011-08-19 06:32 UTC (History)
1 user (show)

See Also:
Category: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Bob Koure 2011-08-14 07:34:39 UTC
Once the MusicIP importer has been run (or MusicIP flags are all set to true) - see http://forums.slimdevices.com/showthread.php?t=89462 ) you can click an 'M' icon in the track list.
The left side panel redisplays and you get:
at the top: Music IP Mix
next line: Home > XXX MusicIP Mix (where XXX is whatever you mixed on)
next line: All Songs
...And the rest of the left panel (where the playlist should be displayed) is blank - or occasionally a few tracks will be listed.

I *think* this might be a timing issue.
No errors are generated in the server.log file when creating a MusicIP mix, but when I hover over "All Songs" and press either the 'play' or 'add to playlist' icon, there's a long pause before the tracks are added to the "now playing" list (right hand side) - and errors in the following format appear in the server log
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
[11-08-14 08:43:54.9481] Slim::Formats::Playlists::M3U::write (262) Error: Couldn't retrieve objectForUrl: [file:///d:/[music]/lossless-music-1/Avishai%20Cohen%20Trio%20-%20Gently%20Disturbed/02%20-%20Chutzpan.flac] - skipping!
[11-08-14 08:43:54.9527] Slim::Formats::Playlists::M3U::write (262) Error: Couldn't retrieve objectForUrl: [file:///e:/[music]/lossless-music-2/Stefon%20Harris%20&%20Jacky%20Terrasson%20-%20Kindred/13%20-%20Body%20&%20Soul.flac] - skipping!
[11-08-14 08:43:54.9571] Slim::Formats::Playlists::M3U::write (262) Error: Couldn't retrieve objectForUrl: [file:///d:/[music]/lossless-music-1/Acoustic%20Alchemy%20-%20This%20Way/11%20-%20Now%20I%27m%20on%20My%20Way.flac] - skipping!
[11-08-14 08:43:54.9617] Slim::Formats::Playlists::M3U::write (262) Error: Couldn't retrieve objectForUrl: [file:///d:/[music]/lossless-music-1/Habib%20Koite%20&%20Bamada%20-%20Ma%20Ya/03%20-%20Ma%20Ya.flac] - skipping!
[11-08-14 08:43:54.9662] Slim::Formats::Playlists::M3U::write (262) Error: Couldn't retrieve objectForUrl: [file:///e:/[music]/mp3-collections/[Putumayo]/Putumayo%20-%20Blues%20Lounge/Putumayo%20-%20Blues%20Lounge%20-%2006%20-%20Moby%20-%20Run%20On.mp3] - skipping!
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

I have verified that these files exist - and even the ones reported as skipped seem to be added to the playlist.

Finally, items added to the playlist do not have 'M' (mixable) icons, even though they are flagged as mixable in the database.

From 'Information' in the webGUI 
Version: 7.6.1 - r33044 @ Thu Aug 11 02:05:41 PDT 2011
Hostname: server4
Server IP Address: 172.16.51.16
Server HTTP Port Number: 9000
Operating system: Windows Server 2003 - EN - cp1252
Platform Architecture: 586
Perl Version: 5.10.0 - MSWin32-x86-multi-thread
Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1)
Total Players Recognized: 2
 
Library Statistics
Total Tracks: 72,901
Total Albums: 5,669
Total Artists: 1,962
Total Genres: 261
Total Playing Time: 8205:34:19


I'd be glad to reproduce this with other logging on - please let me know what to turn on.

Thanks!
Comment 1 Michael Herger 2011-08-15 22:19:44 UTC
I've encountered an issue where MIP would fail on Windows because it sometimes uses uppercase drive letters where SBS would have stored lowercase and vice versa. Could you check the track information of one of these files, and see what the path SBS is reporting is spelled like?
Comment 2 Bob Koure 2011-08-16 07:33:23 UTC
(In reply to comment #1)
>Could you check the track information of one of these files, and see
> what the path SBS is reporting is spelled like?
I've just dumped all track info (sqlite3 squeezebox.db "select url from tracks;")
Yes there's a case mismatch in volume letters.
Further, looking at all the URLs, there's a mix of uppercase and lowercase volume letters
I'm not sure why there's a mix. I haven't done any "Look for new and changed music" scans. For these last 6.1 dailies, I've installed the app, set to start when system starts, stopped the service, deleted the cache, and restarted (so nothing left over from previous). Last done with b33044
Do you want the list?
Do you want me to do this with the latest build?

It appears that MusicIP is always using lower case drive letters - is that the actual case?
Comment 3 Bob Koure 2011-08-16 20:16:16 UTC
I wrote a bit of Python to just set all those uppercase volume letters to lowercase, and MusicIP works (pretty much)
I get 1 error:
[11-08-16 23:04:23.6519] Slim::Plugin::MusicMagic::Plugin::getMix (1008) MIP attempted to mix in a song at e:\[music]\lossless-new-2\Ottmar Liebert & Luna Negra - Up Close (binaural) (24.96 downsampled)\03 - Cocteau.flac that can't be found at that location
So maybe I borked something in the db - or I need to refresh the musicIP db (started that now)

It seems slower than with 7.5 (40 tracks), and, if a track is playing, output goes silent for about 20 sec, then resumes - seems to pick up that much later in the track rather then just stop.

Here's thePython, on the off chance it might help...
import sqlite3
conn = sqlite3.connect('test.db')
c = conn.cursor()

# get all id:url tuples into the list 'allURLs'
c.execute ("""select id,url from tracks""")
#c.execute ("""select id,url from tracks where id<10""")
allURLs=[]
for row in c:
  if(len(row[1])) > 0:
    allURLs.append(row)

updated=0
for row in allURLs:
  if row[1][8].isupper():
    newURL=row[1][0:8]+row[1][8].lower()+row[1][9:]
    #print(row[1])
    #print(newURL)
    update="update tracks set url='"+newURL+"' where id="+str(row[0])
    #print(update)
    c.execute (update)
    updated=updated+1

conn.commit()
c.close()

print('updated '+str(updated)+' URLs')
Comment 4 Michael Herger 2011-08-16 21:27:45 UTC
Great feedback! Thanks Bobo. I think I've identified the underlying issue: in MySQL the url column in the db was case insensitive, while in SQLite it is. I'll run a few more tests today, but this finding might help us fix several other issue like "all my playlists are empty" too.
Comment 5 Bob Koure 2011-08-17 05:57:22 UTC
Ran MusicIP refresh.
Got a lot of 
[11-08-17 08:08:35.9680] Slim::Utils::Scanner::Local::deleted (545) Handling deleted track file:///d:/[music]/blah blah blah
messages in the server.log

Now pressing the 'M'ix button on a track gets a mix pretty quickly, but (minor issue) only returns 20 tracks (system is set for 40).

No errors in the server.log after a mix.

Michael - kudos on spotting a fundamental issue. *Somebody* ought to be buying you a beer - and if nobody at SB central is doing that, I'd be glad to.
Comment 6 Bob Koure 2011-08-17 07:55:41 UTC
OK - that minor issue 
When mixing, and system is set to make a mix of 40 tracks, only 20 tracks are shown in the left Win GUI window. No album art is displayed.

BUT if the 'play' icon next to 'All Songs' is clicked, 40 tracks get added to the playlist.
The left (shorter) list isn't a truncation of the right hand one, but appears to be every other one

When I 'save' now playing, I get an m3u file.
In that m3u file, some files have uppercase volume letters and some have lowercase.
The ones with uppercase were the ones that were displayed in the leftmost window when the mix was created.

Also, clicking 'play' causes a number of errors in the server.log. e.g.
[11-08-17 08:57:13.0408] Slim::Formats::Playlists::M3U::write (262) Error: Couldn't retrieve objectForUrl: [file:///d:/[music]/lossless-music-1/Fourplay%20-%20Yes,%20Please!/07%20-%20Fortress.flac] - skipping!

Note that the errors were for URLs with lower cased volume letters - and I know that the URLs in the db all have lower case volume letters.

So, I'm not sure what exactly is going on, but, yeah, a change in case enforcement looks like it's causing at least some of those playlist issues
Comment 7 Bob Koure 2011-08-17 07:57:48 UTC
Actually, I don't think it's clicking 'play' that caused those errors but clicking 'save'.
I'll have to experiment, but can't get to it for a bit as I really have to get some work done.
Comment 8 Michael Herger 2011-08-17 08:34:43 UTC
I checked in a schema change which should fix this. Unfortunately this will trigger a full re-scan. Please give it a try and let me know how it works for you. Thanks!
Comment 9 Bob Koure 2011-08-18 04:46:35 UTC
Is that in a nightly, or somewhere else?
I'm asking because I don't see anything in the bug fixes.
Are there experimental builds I haven't been clever enough to discover around somewhere?
Comment 10 Michael Herger 2011-08-18 05:03:48 UTC
Yes, it's in the release candidate: http://downloads.slimdevices.com/SqueezeboxServer_v7.6.1_rc3/

I somehow managed to check the change in without leaving a note here - I'm sorry about that. r33105 should fix this.
Comment 11 Bob Koure 2011-08-19 05:05:01 UTC
Found your comment in 17457 that the changes had been checked in, just nothing to the bug report, so I tried 7.6.1-33110.
I deleted my entire cache and ran a 'find everything' scan.
- Did a select on tracks.url: all volume letters are capitalized (yeah, that was the point - just looking)
- MusicIP mix generation, works great, and no errors to the server.log
- saving / retrieving playlists (m3us) works great and no errors in the server log
Also - these latter two are dramatically faster.

What I did not test: upgrading one tracks table to another to see that all volume letters got upcased. Sorry. I'd already upcased all the volume letters manually, decided deleting the cache was the right thing to do, and afterwards realized that I had an unmodified copy of squeezebox.db I could have copied back in. My bad.
Comment 12 Michael Herger 2011-08-19 05:09:42 UTC
Thanks for the testing!

> What I did not test: upgrading one tracks table to another to see that all
> volume letters got upcased. 

That's not important any more, as the important change was to make the comparison of those paths case insensitive.

> Sorry. I'd already upcased all the volume letters
> manually, decided deleting the cache was the right thing to do, and afterwards
> realized that I had an unmodified copy of squeezebox.db I could have copied
> back in. My bad.

No problem! Thanks!
Comment 13 Bob Koure 2011-08-19 06:08:31 UTC
Oh right - LIKE in SQLie is case insensitive. Well, so long as the characters are not international.
I was wondering about the mismatch between all MusicIP lists having lower case volume letters and the db having all uppercase letters and figured you went with something like that.
I don't know that there are any windows users with above-ascii-127 volume letters. bit maybe it's worth adding to the list of known limitations for the support folks? With millions of users, there's (by definition) a couple of "one in a million" types in there.
Comment 14 Michael Herger 2011-08-19 06:32:32 UTC
> Oh right - LIKE in SQLie is case insensitive. Well, so long as the characters
> are not international.

We've added the ICU libraries to overcome this issue (http://www.sqlite.org/faq.html#q18)