Bug 15486 - Songs in MusicIP Mixes with accented/international characters fail to play
: Songs in MusicIP Mixes with accented/international characters fail to play
Status: NEW
Product: Logitech Media Server
Classification: Unclassified
Component: MusicIP
: 7.4.1
: PC Windows XP
: P3 normal with 3 votes (vote)
: Investigating
Assigned To: Squeezebox QA Team email alias
: community_fix
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-01-18 10:04 UTC by treble
Modified: 2011-10-17 02:20 UTC (History)
4 users (show)

See Also:
Category: Bug


Attachments
MusicIP server log, Version 7.4 (22.40 KB, application/octet-stream)
2010-01-18 10:04 UTC, treble
Details
MusicIP server log, Version 7.3 (20.68 KB, application/octet-stream)
2010-01-18 10:05 UTC, treble
Details
Server Log with MusicIP debugging enabled. SqueezeServer 7.4.2 (47.94 KB, application/octet-stream)
2010-01-19 10:57 UTC, treble
Details
Server Log with MusicIP debugging enabled. SqueezeServer 7.5.0 - r30155 (104.61 KB, text/plain)
2010-02-22 16:52 UTC, treble
Details
Possible patch (473 bytes, patch)
2011-01-27 04:17 UTC, Alan Young
Details | Diff
scanner.log file (117.98 KB, text/plain)
2011-10-16 17:33 UTC, Dieter
Details
server.log file (117.98 KB, text/plain)
2011-10-16 17:34 UTC, Dieter
Details
scanner.log file (2.59 MB, text/plain)
2011-10-16 17:38 UTC, Dieter
Details

Note You need to log in before you can comment on or make changes to this bug.
Description treble 2010-01-18 10:04:25 UTC
Created attachment 6442 [details]
MusicIP server log, Version 7.4

Mixes with international characters (like umlaut, accents, etc.) are not
mixable and cause the playlist to stop.  This problem started when I upgraded
to 7.4.1.  It doesn't work in 7.4.2 either.  It was fine in 7.3.

It seems to be something to do with the encoding.

Attached two debug MusicIP log files.  One with a mix created in 7.3, where
accented characters work fine, and one in 7.4.1 where it doesn't work.

Errors are like:
[10-01-17 09:02:33.4491] Slim::Formats::Playlists::M3U::write (220) Error:
Couldn't retrieve objectForUrl:
[file:///E:/Music/All%20Music/Music/%27%20A%20B%20C%20D%20E/Cachao/1995%20-%20Master%20Sessions,%20Vol.%202/07%20-%20Rom%C3%A1ntica%20Mujer.mp3]
- skipping!


[10-01-17 09:02:34.9869] Slim::Web::Pages::Playlist::playlist (166) Error:
Couldn't retrieve objectForUrl: [] - skipping!
[10-01-17 09:02:34.9907] Slim::Formats::readTags (159) File missing:
E:\Music\All Music\Music\' A B C D E\Cachao\1995 - Master Sessions, Vol. 2\07 -
Romántica Mujer.mp3
[10-01-17 09:02:34.9910] Slim::Schema::objectForUrl (808) Error: Null track
request! Returning undef.
[10-01-17 09:02:34.9914] Slim::Schema::objectForUrl (808) Backtrace:


See also forum thread: http://forums.slimdevices.com/showthread.php?t=71450
Comment 1 treble 2010-01-18 10:05:15 UTC
Created attachment 6443 [details]
MusicIP server log, Version 7.3

Server log when creating mix in 7.3
Comment 2 Michael Herger 2010-01-18 13:43:15 UTC
what are your MIP import settings?
Comment 3 treble 2010-01-18 19:24:31 UTC
(In reply to comment #2)
> what are your MIP import settings?

Under SB Settings -> Plugins -> MusicIP it says:

Use MusicIP:  Only read mixable status from MusicIP (faster)

It that what you mean?   Thanks,
Comment 4 Michael Herger 2010-01-19 03:48:56 UTC
I'm sorry to say, but this is working fine for me (7.5 on XP). I selected a track with an umlaut in it, and it mixed just fine.

The errors you have in the log aren't directly related to MusicIP, but they seem to arise when a playlist is to be written(?). Please enable logging for plugin.musicip and try again.
Comment 5 treble 2010-01-19 10:56:57 UTC
(In reply to comment #4)
> I'm sorry to say, but this is working fine for me (7.5 on XP). I selected a
> track with an umlaut in it, and it mixed just fine.
> 
> The errors you have in the log aren't directly related to MusicIP, but they
> seem to arise when a playlist is to be written(?). Please enable logging for
> plugin.musicip and try again.

Thanks for the feedback.  Sorry for the confusion.  Yes, a MusicIP can create a mix from a song with an umlaut, but you can not play them.  For example, if there is a song with an umlaut present in the playlist as song number 5, on the webinterface it will not be listed in the playlist.  If you play song 4, and skip to the next song, it will play song 6 (there is no song 5 in the playlist).  On the squeezebox the display will show totally black when you browse to song 5, and when you hit play, it will display: "Problem: can't open file for:".  On the Boom it will also show black.

If I just search for the trouble song and play it directly it plays fine.  

In the logs I see that the character encoding in 7.4 is very different from 7.3, so I suspect that to be the problem.  

Uploaded a new log, with MusicIP debug enabled.  Near time 10:44 you will not the errors in the log when I browse to a problem song (where the SB display is black) and try to play it.

Thanks.
Comment 6 treble 2010-01-19 10:57:49 UTC
Created attachment 6444 [details]
Server Log with MusicIP debugging enabled.  SqueezeServer 7.4.2
Comment 7 Chris Owens 2010-02-01 08:09:02 UTC
QA to get solid repro instructions
Comment 8 KDF 2010-02-15 12:42:19 UTC
*** Bug 15701 has been marked as a duplicate of this bug. ***
Comment 9 treble 2010-02-22 16:52:49 UTC
Created attachment 6555 [details]
Server Log with MusicIP debugging enabled.  SqueezeServer 7.5.0 - r30155

Bug is also in Version 7.5.  Attached server log, trying to make a mix based on:
Location: E:\Maho Backup\Music\All Music\Music\' P Q R S T\Rokia Traoré\2003 - Bowmboï\03 - Kôté Don.mp3

In scanner log imported from MusicIP as (no errors):
[10-02-22 12:43:08.0334] Slim::Plugin::MusicMagic::Importer::setMixable (352) track: E:\Maho Backup\Music\All Music\Music\' P Q R S T\Rokia Traoré\2003 - Bowmboï\03 - Kôté Don.mp3 is mixable

Creating mix as (no errors):
[10-02-22 16:19:36.2499] Slim::Plugin::MusicMagic::Plugin::getMix (940) Creating mix for: song using: E:\Maho Backup\Music\All Music\Music\' P Q R S T\Rokia Traoré\2003 - Bowmboï\03 - Kôté Don.mp3 as seed.
[10-02-22 16:19:36.2513] Slim::Plugin::MusicMagic::Plugin::getMix (951) Request http://localhost:8082/api/mix?song=E%3A%5CMaho%20Backup%5CMusic%5CAll%20Music%5CMusic%5C'%20P%20Q%20R%20S%20T%5CRokia%20Traor%E9%5C2003%20-%20Bowmbo%EF%5C03%20-%20K%F4t%E9%20Don.mp3&variety=2&rejectsize=0&mixgenre=0&style=15&sizetype=tracks&size=50

Tries to play as (error):
[10-02-22 16:32:14.5006] Slim::Formats::readTags (157) File missing: E:\Maho Backup\Music\All Music\Music\' P Q R S T\Rokia Traoré\2003 - Bowmboï\03 - Kôté Don.mp3

Other songs in the mix with 'strange' characters fail to play as well.
Comment 10 Dieter 2010-05-16 14:05:53 UTC
I also have problems with creating MIP mixes for files with German umlauts in the file name. I am using SC 7.5.1 on Ubuntu.

For all files with German umlauts in the file name the "M" symbol is not shown when browsing through the web interface. On the Squeezebox, the menu item "Create MusicIP Mix" does not exist for such songs. For all other songs the "M" is shown in the web interface and the menu item exists on the Squeezebox.

I am sure that this worked in the past but unfortunately do not know when it has stopped to work.

Creating mixes for such songs directly in the MusicIP programm works without problems.

Can I provide any log to help solving this bug?
Comment 11 Alan Young 2011-01-27 04:17:55 UTC
Created attachment 7090 [details]
Possible patch

It is not clear what the actual API to Music IP is, especially with respect to character-set encodings. It has been suggested that it is always UTF-8, even when the local file-system used a different character-set encoding.

This patch would help if the encoding is indeed UTF-8 but there are several other places in the code where character-set encoding is probably not handled correctly.

Resolution of these issues would require someone with access to all the necessary systems to go through these aspects of the code carefully. In particular, look at places where pathnames are encoded or decoded or where URI-escaping is used. Places where Slim::Utils::Misc::pathFromFileURL() and Slim::Utils::Misc::fileURLFromPath() would deserve inspection.
Comment 12 Dieter 2011-10-09 13:37:33 UTC
I am now on SC 7.6.1 - r33110 on Ubuntu Linux and it is still not possible to use a track as seed for a MusicIP mix if there is a German umlaut (or another "irregular" character) within the file path (track name or directory name).

For these tracks/albums the MusicIP "m" is neither shwon in the web interface nor in the player display. Further, on the player there is no item "Create MusicIP mix" in the context menu of a track (when presseing the RIGHT button on the remote) as it is for other tracks.

I love MusicIP and would really appreciate if this very old bug could be solved. The problem is only related to file/directory names not to tags with umlauts. I know that I could rename all files/directories but I cannot go throuh 15000 songs to look for file/directory names comprising umlauts. Further, when ripping CDs the file names and directory structure is automatically derived from the tags (which can comprise umlauts). So I had to manually correct all file/directory names after each ripping to avoid umlauts in file/directory names.

Is there a chance that this problem will ever be solved?
Comment 13 Michael Herger 2011-10-10 02:30:47 UTC
Dieter - would Alan's patch fix the issue for you?
Comment 14 Dieter 2011-10-10 02:38:23 UTC
Can I simply apply the patch or do I have to compile something? I remember that with the precompiled Windows version of squeezecenter it was not possible to use amended pm files. This may be different since I am using now the linux version.
Comment 15 Michael Herger 2011-10-10 02:44:05 UTC
(In reply to comment #14)
> Can I simply apply the patch or do I have to compile something?

Correct, most of the code on Windows is built in to the binary, but in this case you're lucky: plugins are still there as source code. You can edit them and the change should be picked up when you restart the server.
Comment 16 Dieter 2011-10-10 02:47:34 UTC
I will test it this evening and then report back.
Comment 17 Dieter 2011-10-10 14:35:19 UTC
I have applied the patch to /usr/share/perl5/Slim/Plugin/MusicMagic/Common.pm but unfortuntately it did not help. It is still not possible to create MusicIP mixes for files which have an umlaut in the file name or path. For those files the "context menu" item "Create MusicIP mix" is not shown and long pressing PLAY on the remote, which usually creates a mix based on the selected item, does also not work. 

I previously mentioned that the "m" (indicating that the selected item can be used as seed for a MusicIP mix) is not shown for these files neither in the web interface nor in the player display. I just recognized that in the player display the "m" is not shown for ANY items AT ALL as long as one use the standard browse menus. I use CustomBrowse and in the CustomBrowse Artist and Album browse menus the "m" is shown but only for items which have no umlaut in the file name or path.

I also checked the web interface (which I hardly ever use) and I found that the standard Artist and Album browse menus do also behave different from the CustomBrowse Artist and Album browse menus. While the CB menus in the web interace behave exactly like on the player (showing the "m" only for mixable items) the standard menus show the "m" for ALL items (that's the opposite to the player standard browse menus). However, although the "m" is shown in the standard browse menus, items with an umlaut in the file name or path are not mixable even through the web interface. So it seems that CB does a more correct job with showing the MusicIP indicator.

BTW, when I talk of my player I always mean a squeezebox 2 or 3, which may behave different from a squeezebox touch (which I don't have).

Is there anything else I can test. I would really appreciate to see this bug being fixed and can provide you with log files or anything else you need.
Comment 18 Michael Herger 2011-10-11 04:20:28 UTC
> I have applied the patch to /usr/share/perl5/Slim/Plugin/MusicMagic/Common.pm

I thought you were on Windows?

> but unfortuntately it did not help. It is still not possible to create MusicIP
> mixes for files which have an umlaut in the file name or path. For those files

Did you run a wipe & rescan of your collection?
Comment 19 Dieter 2011-10-11 04:31:09 UTC
(In reply to comment #18)
> I thought you were on Windows?
No, in the past (long time ago) I was on Windows, but now I am on Ubuntu Linux (as stated in comments 12 and 14 ;).
 
> Did you run a wipe & rescan of your collection?
No, should I have done that? I restarted the server (several times) and there was always an automatic scan after the restart (that's new, perhaps since I turned on the new automatic detection of new music feature?).

But I assume that this automatic scan (which takes only 1,5 minutes for about 13000 tracks) does not make a wipe. So should I repeat the test with a wipe?
Comment 20 Dieter 2011-10-11 14:50:12 UTC
I again applied Alans's patch, restarted the server and made a complete wipe and rescan afterwards but it did not change anything. Tracks with umlauts in the file/directory name can still not be used to make a MusicIP mix.

Anything else I can test?
Comment 21 Dieter 2011-10-14 05:39:46 UTC
Michael, would it help if I provided a log file? Or can I do anything else?
Comment 22 Michael Herger 2011-10-14 06:47:07 UTC
logs always are helpful, but I doubt we'll soon have time to further investigate this issue
Comment 23 Dieter 2011-10-16 17:32:50 UTC
I enclose my server.log and scanner.log files. I have made a wipe and rescan and you see from the scanner.log file that for each track with an umlaut in the filename/path during the MusicIP-import there is an error entry like 

Slim::Plugin::MusicMagic::Importer::setMixable (352) Couldn't get track for file:///media/Daten/Musik/Herbert%20Gr%F6nemeyer/4630%20Bochum/01-Herbert%20Gr%F6nemeyer%20-%20Bochum.ogg

So it seems that there is (at least) a problem in the MusicIP-import.

Can you point me to the source code which must be changed that the MusicIP-import is done correctly? If we are lucky, that's all what must be corrected.
Comment 24 Dieter 2011-10-16 17:33:47 UTC
Created attachment 7517 [details]
scanner.log file
Comment 25 Dieter 2011-10-16 17:34:10 UTC
Created attachment 7518 [details]
server.log file
Comment 26 Dieter 2011-10-16 17:38:42 UTC
Created attachment 7519 [details]
scanner.log file
Comment 27 Michael Herger 2011-10-17 02:20:10 UTC
Slim::Plugin::MusicMagic::Importer is the starting point. It's calling some methods in Slim::Plugin::MusicMagic::Common too, so make sure to follow the flow.