Bug 9878 - Favorites with non-latin characters in their url don't work
: Favorites with non-latin characters in their url don't work
Status: VERIFIED FIXED
Product: Logitech Media Server
Classification: Unclassified
Component: Localization
: 7.3.0
: PC Ubuntu Linux
: P4 normal (vote)
: 7.4.1
Assigned To: Squeezebox QA Team email alias
http://forums.slimdevices.com/showthr...
: charset_issues
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-11-02 06:57 UTC by Michael Herger
Modified: 2009-10-21 09:45 UTC (History)
3 users (show)

See Also:
Category: Bug


Attachments
Patches for Bug 9878 (1.58 KB, application/x-zip-compressed)
2008-12-28 08:58 UTC, chrismuller@gmx.net
Details
Patch for Bug 9878 (reopened) (592 bytes, patch)
2009-08-11 01:14 UTC, chrismuller@gmx.net
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Herger 2008-11-02 06:57:15 UTC
An album by Herbert Grönemeyer:

Titel:  Zwölf
URL: db:album.titlesearch=ZW%C3%96LF

Wouldn't work. We'd probably have to save the titlesort string in these cases.
Comment 1 Michael Herger 2008-11-05 05:23:27 UTC
non-latin characters seem to be double-encoded:

[08-11-05 14:22:21.3592] Slim::Player::Song::new (76) Error: Could not find handler for db:contributor.namesearch=ALI%20FARKA%20TOUR%C3%83%C2%89!
[08-11-05 14:22:21.3600] Slim::Control::Request::execute (1887) Error: While trying to run function coderef [Slim::Control::Commands::playlistJumpCommand]: [Can't call method "blessed" without a package or object reference at /Users/mh/Documents/workspace/7.3/server/Slim/Player/StreamingController.pm line 644.
]
[08-11-05 14:22:40.6425] Slim::Player::Song::new (76) Error: Could not find handler for db:album.titlesearch=F%C3%83%C2%9CR%20USSZEGSCHNIGGE!
[08-11-05 14:22:40.6431] Slim::Control::Request::execute (1887) Error: While trying to run function coderef [Slim::Control::Commands::playlistJumpCommand]: [Can't call method "blessed" without a package or object reference at /Users/mh/Documents/workspace/7.3/server/Slim/Player/StreamingController.pm line 644.
]

Comment 2 Michael Herger 2008-11-05 05:26:53 UTC
Alan - do you have an idea what's wrong with the db: url handler?

[08-11-05 14:22:21.3592] Slim::Player::Song::new (76) Error: Could not find handler for db:contributor.namesearch=ALI%20FARKA%20TOUR%C3%83%C2%89!
[08-11-05 14:22:21.3600] Slim::Control::Request::execute (1887) Error: While trying to run function coderef [Slim::Control::Commands::playlistJumpCommand]: [Can't call method "blessed" without a package or object reference at /Users/mh/Documents/workspace/7.3/server/Slim/Player/StreamingController.pm line 644.
]
[08-11-05 14:22:40.6425] Slim::Player::Song::new (76) Error: Could not find handler for db:album.titlesearch=F%C3%83%C2%9CR%20USSZEGSCHNIGGE!
[08-11-05 14:22:40.6431] Slim::Control::Request::execute (1887) Error: While trying to run function coderef [Slim::Control::Commands::playlistJumpCommand]: [Can't call method "blessed" without a package or object reference at /Users/mh/Documents/workspace/7.3/server/Slim/Player/StreamingController.pm line 644.
]

Comment 3 chrismuller@gmx.net 2008-11-28 10:30:41 UTC
Not sure whether this is the same bug: I have albums with utf8 characters in the file name (url). When such an album is saved as favorite, playing the favorite does not work. E.g.

URL="db:album.titlesearch=COMPTINES%20%C3%80%20CHANTER%20VOLUME%202"

When the favorite is played, the player shows "Now Playing: Nothing" (empty playlist). No errors in "server.log". 

(SqueezeCenter 7.2.1 on Ubuntu Feisty)
Comment 4 chrismuller@gmx.net 2008-11-28 10:49:36 UTC
(In reply to comment #3)
> Not sure whether this is the same bug: I have albums with utf8 characters in
> the file name (url). When such an album is saved as favorite, playing the
> favorite does not work. E.g.
> 
> URL="db:album.titlesearch=COMPTINES%20%C3%80%20CHANTER%20VOLUME%202"
> 
> When the favorite is played, the player shows "Now Playing: Nothing" (empty
> playlist). No errors in "server.log". 
> 
> (SqueezeCenter 7.2.1 on Ubuntu Feisty)
> 

Just upgraded to 7.2.2 and the problem persists. 
Comment 5 chrismuller@gmx.net 2008-12-28 08:58:00 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > Not sure whether this is the same bug: I have albums with utf8 characters in
> > the file name (url). When such an album is saved as favorite, playing the
> > favorite does not work. E.g.
> > 
> > URL="db:album.titlesearch=COMPTINES%20%C3%80%20CHANTER%20VOLUME%202"
> > 
> > When the favorite is played, the player shows "Now Playing: Nothing" (empty
> > playlist). No errors in "server.log". 
> > 
> > (SqueezeCenter 7.2.1 on Ubuntu Feisty)
> > 
> 
> Just upgraded to 7.2.2 and the problem persists. 
> 

There seem to be two issues, for which I include suggested patches:

- An album/genre/contributor with latin characters, if saved as a favorite, is utf8-encoded twice before being uri-encoded. Thus, the resulting URL contains garbage --> see suggested patches for Schema/Album.pm, Schema/Genre.pm and Schema/Contributor.pm attached.

- A favorite with URI-encoded UTF-8-encoded latin characters is needs to be not only URI-decoded, but also UTF-8-decoded --> see suggested patch for Control/Commands.pm attached.

Comment 6 chrismuller@gmx.net 2008-12-28 08:58:29 UTC
Created attachment 4519 [details]
Patches for Bug 9878
Comment 7 Michael Herger 2008-12-28 10:12:29 UTC
Thanks Chris. Do you have an idea why you need to utf8decode in one case, but not the others?

I'll take another look at this issue in the new year. But I fear your patch will not work on all platforms (first of all not on Windows). It'll need some more work.
Comment 8 Michael Herger 2009-01-06 09:03:36 UTC
Chris - do you still see this issue with the latest 7.3.x revisions? And could you please give me some details about your system (as found in Settings/Information)?
Comment 9 Michael Herger 2009-01-15 00:40:20 UTC
Chris?
Comment 10 Markus Schiegl 2009-01-16 05:17:11 UTC
although i'm not Chris...

i've checked out the latest 7.3r24680 and created two favorites, one "artist" and one "album", each containing an umlaut. favorites.opml looks like:

<?xml version="1.0" encoding="UTF-8"?>
<opml version="1.0">
  <head title="Favorites">
    <expansionState></expansionState>
  </head>
  <body>
    <outline URL="db:contributor.namesearch=ALIZ%C3%83%C2%89E" icon="html/images/artists.png" text="Alizée" type="audio" />
    <outline URL="db:album.titlesearch=%C3%83%C2%87A%20IRA" icon="html/images/albums.png" text="Ça Ira" type="audio" />
  </body>
</opml>

Both entries are displayed by SC. Trying to playback those results in errors logged to server.log:

[09-01-16 13:59:45.3989] Slim::Player::Song::new (76) Error: Could not find handler for db:contributor.namesearch=ALIZ%C3%83%C2%89E!
[09-01-16 13:59:53.7025] Slim::Player::Song::new (76) Error: Could not find handler for db:album.titlesearch=%C3%83%C2%87A%20IRA!

system:

Version: 7.3.3 - TRUNK @ UNKNOWN
Hostname: vserver1
Server IP Address: <IP>
Server HTTP Port Number: 9000
Operating system: Linux - EN - utf8
Platform Architecture: x86_64-linux
Perl Version: 5.8.8 - x86_64-linux-thread-multi
MySQL Version: 5.0.70-log
Total Players Recognized: 5

btw. Chris' patch works here...those two lines in favorites.opml are now stored as:

<outline URL="db:contributor.namesearch=ALIZ%C3%89E" icon="html/images/artists.png" text="Alizée" type="audio" />
<outline URL="db:album.titlesearch=%C3%87A%20IRA" icon="music/4604/cover.png" text="Ça Ira" type="audio" />
Comment 11 Michael Herger 2009-01-21 06:22:05 UTC
change 24729 - Thanks for the patch - seems to be working fine on Windows too.

use uri_encode instead of uri_encode_utf8 to store URLs
Comment 12 Markus Schiegl 2009-01-21 10:09:51 UTC
works fine - once more: thanks!
Comment 13 James Richardson 2009-06-17 09:35:41 UTC
This bug has been fixed in the 7.3.3 release version of SqueezeCenter!

If you haven't already. please download the new version from http://www.logitechsqueezebox.com/support/download-squeezecenter.html 

If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.
Comment 14 chrismuller@gmx.net 2009-08-11 01:13:12 UTC
(In reply to comment #13)
> This bug has been fixed in the 7.3.3 release version of SqueezeCenter!
> 
> If you haven't already. please download the new version from
> http://www.logitechsqueezebox.com/support/download-squeezecenter.html 
> 
> If you are still experiencing this problem, feel free to reopen the bug with
> your new comments and we'll have another look.

Whoever incorporated my patch into the 7.3 tree has changed my patch for Commands.pm from

Encode::decode_utf8

to

Slims::Utils::Unicode::utf8decode

But this gives wrong results (at least for me). URI::Escape::uri_unescape returns an octet stream, not a string. Thus, it cannot be fed to utf8decode, but the octet streams must be converted into utf8 by using Encode::decode_utf8. See patch below:

---------------------------------------------------------
--- Commands.pm.orig	2009-06-16 00:12:38.000000000 +0200
+++ Commands.pm	2009-08-11 01:33:07.000000000 +0200
@@ -3124,8 +3124,8 @@
 			if ($term =~ /^(\w+)\.(\w+)=(.*)$/) {
 
 				my $key   = URI::Escape::uri_unescape($2);
-				my $value = Slim::Utils::Unicode::utf8decode( URI::Escape::uri_unescape($3) );
-
+				my $value = Encode::decode_utf8(URI::Escape::uri_unescape($3));
+# And not Slim::Utils::Unicode::utf8decode as URI::Escape::uri_unescape returns an octet stream, not a string !!!
 				$class = ucfirst($1);
 				$obj   = Slim::Schema->single( $class, { $key => $value } );
Comment 15 chrismuller@gmx.net 2009-08-11 01:14:00 UTC
Created attachment 5594 [details]
Patch for Bug 9878 (reopened)

See my earlier comment
Comment 16 James Richardson 2009-10-05 16:53:37 UTC
Michael: Can you comment on this?
Comment 17 Michael Herger 2009-10-05 21:19:33 UTC
Chris - is this patch against 7.4? And what platforms did you test it with?
Comment 18 chrismuller@gmx.net 2009-10-07 05:38:05 UTC
(In reply to comment #17)
> Chris - is this patch against 7.4? And what platforms did you test it with?

It is against 7.3.3, where I have tested it, but should work also against 7.4, as the relevant part of Commands.pm was not changed in 7.4. 

It was actually already suggested in my original patch (Comment 6). When the original patch was incorporated into the 7.3.3 release, someone changed 

Encode::decode_utf8

to

Slims::Utils::Unicode::utf8decode

But as explained, that seems wrong. 

Best regards,
Chris
Comment 19 Michael Herger 2009-10-13 05:53:53 UTC
Chris - this is working fine for me from the web UI anc Controller: I've added artist "Ang�lique Kidjo" and album "F�r usszeschnigge" to my favorites, and they play fine. It does fail from the player UI. Can you confirm this?
Comment 20 Michael Herger 2009-10-13 06:13:47 UTC
QA - could you please try to reproduce with 7.4.1?

Please note: album/artist favorites are indeed broken. But that's a different issue. Only test with single tracks.
Comment 21 James Richardson 2009-10-13 08:51:42 UTC
Tested with 7.4.1 r28825, appears to be working for me now.
Comment 22 James Richardson 2009-10-13 13:10:21 UTC
Verified fixed in 7.4.1 r28825
Comment 23 chrismuller@gmx.net 2009-10-14 01:49:08 UTC
(In reply to comment #22)
> Verified fixed in 7.4.1 r28825

Michael mentioned that album/artist favorites are still broken (which is also my perception) - so is this a different bug? 

I still think that in Commands.pm, line 3251, URI::Escape::uri_unescape returns an octet stream, not a string. Thus, it cannot be fed to utf8decode, but the octet streams must be converted into utf8 by using Encode::decode_utf8

For me my suggest change to this line made album/artist favorites work. Can you please verify!

Regards,
Chris