Bug 6621 - Can't store non-latin characters in preference file
: Can't store non-latin characters in preference file
Status: CLOSED FIXED
Product: Logitech Media Server
Classification: Unclassified
Component: Web Interface
: 7.0
: PC Linux (other)
: P1 major (vote)
: 7.x
Assigned To: Michael Herger
:
Depends on:
Blocks: 6789 6914 7429
  Show dependency treegraph
 
Reported: 2008-01-15 07:14 UTC by Oscar Carlsson
Modified: 2009-07-31 10:15 UTC (History)
6 users (show)

See Also:
Category: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Oscar Carlsson 2008-01-15 07:14:23 UTC
I entered a name for a player via the web interface using Firefox with a swedish character, which ended up encoded incorrectly in some way. "Kök" (kitchen) became "Kök".

SqueezeCenter-version: 7.0 - 16284 - Debian - SV - utf8
Perl-version: 5.8.8 i486-linux-gnu-thread-multi
MySQL-version: 5.0.45-Debian_1ubuntu3.1-log
Platform Architecture: i686-linux
Comment 1 Michael Herger 2008-01-15 07:59:44 UTC
Works for me. What are your browser's language settings?
Comment 2 Michael Herger 2008-01-15 08:08:36 UTC
Ok, now I see it. It only changes once the screen has been left and re-entered.
Comment 3 Michael Herger 2008-01-15 09:27:52 UTC
Adrian - didn't we have this case already? Looks as if values aren't correctly written or read as utf8 encoded. Any idea?
Comment 4 Oscar Carlsson 2008-01-15 09:30:48 UTC
Yeah, exactly, it needs a refresh to be visible for some reason. Haven't had time to check the database yet, so I'm not sure where the problem is.
Comment 5 Adrian Smith 2008-01-15 10:56:49 UTC
Michael - see comment 5, 13 and 14 in bug 5173.

I believe we may need to set the ImplictUnicode flag.  Previously I was arguing (possibly on campfire) for a newer version of YAML::Syck and to turn on escaping of binary as well with that.  I think Andy stores binary prefs for SN?  Anyway Chris did not want to do it "this close to 7.0"...
Comment 6 Michael Herger 2008-01-15 13:53:17 UTC
> Michael - see comment 5, 13 and 14 in bug 5173.

Thanks. I knew we've had this one already. I should not have closed that other bug, as it did (as you mentioned in your comment) cover various issues. We would have had enough time since september to fix it :-(.

> I believe we may need to set the ImplictUnicode flag.  Previously I was arguing
> (possibly on campfire) for a newer version of YAML::Syck and to turn on
> escaping of binary as well with that.  I think Andy stores binary prefs for SN?

I'll ping him. Thanks!
Comment 7 Michael Herger 2008-01-16 02:00:05 UTC
change 16333 - enabled ImplicitUnicode, working for me now. Please test. Thanks Adrian!
Comment 8 Oscar Carlsson 2008-01-16 02:02:36 UTC
Cool, thanks. Will test as soon as it hits the Debian repo.
Comment 9 Michael Herger 2008-01-16 02:19:06 UTC
If you don't mind, the change is fairly simple: just add "$YAML::Syck::ImplicitUnicode = 1;" around line 29 in Slim/Utils/Prefs/Namespace.pm:

--- trunk/server/Slim/Utils/Prefs/Namespace.pm (original)
+++ trunk/server/Slim/Utils/Prefs/Namespace.pm Wed Jan 16 01:58:35 2008
@@ -28,6 +28,8 @@
 
 use Slim::Utils::Prefs::Client;
 use Slim::Utils::Log;
+
+$YAML::Syck::ImplicitUnicode = 1;
 
 my $log = logger('prefs');
Comment 10 Oscar Carlsson 2008-01-16 02:44:10 UTC
Hmm, Version: 7.0-16329, made the change you said, and now I can't save anything in the settings pages. This is the same even when I reverted the change. When clicking Apply I just get an empty maincontent and the change isn't saved.. I did, however, upgrade the version as well, so it's probably something between r16284 and r16329.
Comment 11 Michael Herger 2008-01-16 03:30:40 UTC
Thanks for the hint regarding the save problem. Can you confirm that you didn't encounter it before 16284? Are you accessing your server from the same machine?
Comment 12 Oscar Carlsson 2008-01-16 03:58:34 UTC
Hmm, tried the computer at home now, which is on the same network, and it works fine there. With Firefox under Windows here at work, it doesn't work (saving that is). IE7 does work though. Can't remember if I've tried to save settings remotely with FF earlier, sorry.

The name problem is still there though, after saving it appears fine, but is displayed incorrectly once I click "Players" again. This is with the changes from r16333 made to the file /usr/share/perl5/Slim/Utils/Prefs/Namespace.pm.
Comment 13 Michael Herger 2008-01-17 02:41:58 UTC
Did you update your installation to the latest nightly?
Comment 14 Oscar Carlsson 2008-01-17 03:52:40 UTC
Just upgraded to:
SqueezeCenter-version: 7.0 - 16373 - Debian - SV - utf8

Still not able to name it "Kök" though :/  But no problem saving now.

It appears as "Kök" in the webinterface and "K\xC3\xB6k" in /etc/squeezecenter/server.prefs.

Maybe it's a missing meta charset tag?
Comment 15 Michael Herger 2008-01-17 04:01:00 UTC
> It appears as "Kök" in the webinterface and "K\xC3\xB6k" in
> /etc/squeezecenter/server.prefs.

Is your browser set to display utf8 (or auto)? What you're seeing is what I get if I force eg. iso8859-1.
Comment 16 Oscar Carlsson 2008-01-17 04:13:30 UTC
Hmmmm..

My browser automatically selects UTF8.. If I force it to iso8859-1 it displays as "Kök".. I'm using the swedish translation, and it displays everything as it should when using UTF-8, but also displays errors when using iso8895-1, it displays the letter "ö" the same way as when I save it (ö instead of ö)..
Comment 17 Michael Herger 2008-01-17 04:39:02 UTC
What does the following tell you:
perl -MYAML::Syck -e 'print "$YAML::Syck::VERSION\n"'
Comment 18 Oscar Carlsson 2008-01-17 04:41:45 UTC
oscar@dragon:/etc/squeezecenter$ perl -MYAML::Syck -e 'print "$YAML::Syck::VERSION\n"'
Can't locate YAML/Syck.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .).
BEGIN failed--compilation aborted.
Comment 19 Michael Herger 2008-01-17 05:00:50 UTC
Ok, let's make this:

perl -I/usr/share/squeezecenter/CPAN -MYAML::Syck -e 'print "$YAML::Syck::VERSION\n"'

Please replace the include path with wherever SC has its own CPAN folder. Thanks!
Comment 20 Oscar Carlsson 2008-01-17 05:20:56 UTC
oscar@dragon:/etc/squeezecenter$ perl -I/usr/share/squeezecenter/CPAN -MYAML::Syck -e 'print "$YAML::Syck::VERSION\n"'
0.64
Comment 21 Michael Herger 2008-01-17 06:51:09 UTC
change 16386 - reverting change 16333 as it didn't fix your issue, but caused negative side-effects. Steinar wasn't able to scan his collection any more after this change. Why ever this would influence the scanner...

This won't be fixed in 7.0.
Comment 22 Oscar Carlsson 2008-01-17 07:15:45 UTC
:/

Okay. Is it possible to manually enter the name in the correct format in the config file?
Comment 23 Michael Herger 2008-01-17 07:29:27 UTC
> Okay. Is it possible to manually enter the name in the correct format in the
> config file?

I doubt it. It would be overwritten the next time _any_ change to the file is done.

But I'll try to find a solution post 7.0. Thanks for testing!
Comment 24 sbjaerum 2008-01-25 23:06:27 UTC
Some additional information using WinXP. I change a player's name to æøå. This is rendered correctly until SC is restarted. After a restart, the player's name is displayed as æøå. It is the same behavior both in the web interface and on the player display. This is on 7.0 r16779.
Comment 25 Michael Herger 2008-01-25 23:14:05 UTC
The problem here is that the preferences aren't correctly encoded when saved to disk. As long as SC is running, it keeps them in the memory. Thus you only see the issue after a restart.
Comment 26 Michael Herger 2008-02-04 04:12:25 UTC
changing summary to reflect the root cause of the issue reported. We're seeing more and more related problems.
Comment 27 Mickey Gee 2008-03-05 09:19:14 UTC
*** Bug 7390 has been marked as a duplicate of this bug. ***
Comment 28 Michael Herger 2008-03-06 17:19:22 UTC
we seem to be getting variations of this issue on a daily basis - increasing severity
Comment 29 Michael Herger 2008-03-11 04:59:17 UTC
change 17846

I'm sorry Steinar. Checking this in again. We need some more testing. But I successfully tested the change once again, using FF2 (Win, OSX, Ubuntu), IE6/7, Safari as clients, and Windows XP, Ubuntu, OSX 10.5, and a ReadyNAS as servers, restarting SC after the changes. It didn't miss one non-latin character. I even used a Hebrew Windows system to change the setting on the ReadyNAS.

Please test again one of the next 7.0.1 nightlies so we can track this down. Report back as soon as possible, with information about your client/server configuration etc.
Comment 30 Michael Herger 2008-03-11 05:28:45 UTC
QA - could you please go through the list of related bugs to see whether this patch fixes them, too?
Comment 31 sbjaerum 2008-03-11 09:49:08 UTC
I see the same side effect of the line checked in at r17846 as the last time it was checked in. Files with accented characters (at least the Norwegian æøå characters) are not picked up by the scanner. To me this side effect is much more severe than the bug this line is supposed to fix. Try to see if you are able to scan a file with a filename containing one of the characters æøå on Windows.

My system information is:
SqueezeCenter Version: 7.0.1 - 16718 - Windows XP - EN - cp1252
Server IP address: 10.0.0.4
Perl Version: 5.8.8 MSWin32-x86-multi-thread
MySQL Version: 5.0.22-community-nt
Platform Architecture: 586
Comment 32 Adrian Smith 2008-03-11 10:06:27 UTC
Michael - there's another bug here related to the player name.

Squeezebox.pm will send the player name to the player for saving as a pack Z* which is an Ascii string.  This means once we support synching with the player again it doesn't actually save non ascii strings correctly on the player.
Comment 33 Michael Herger 2008-03-11 13:23:39 UTC
Looks as if this will need a bit more work. The patch seems to work for _some_ non-latin characters only while completely breaking writing the file in other cases.

Additionally Triode just made me understand that the player name must _not_ contain non-ascii characters. There's a lot more work to do. Thanks for testing.
Comment 34 Michael Herger 2008-03-12 08:16:49 UTC
change 17864 - one more step... this will make sure a player's name is utf8 when it's read from the player. 

Steinar - the issue you mention (file name) is a different issue. The problem we're seeing here is not being able to store any preference with non-latin characters. Player name is a special case, but you should now be able to call it "æøå". Can you confirm this?
Comment 35 sbjaerum 2008-03-12 13:45:50 UTC
I can confirm that I now can save a player name of æøå. But I don't like the fix. On my system, a direct consequence of adding the line
$YAML::Syck::ImplicitUnicode = 1;
to Namespace.pm, is that the scanner does not include files with accented characters into the library. This is weird, but it is a fact.
So the change introduces a more severe bug than it fixes.
Comment 36 Michael Herger 2008-03-12 15:07:02 UTC
change 17866 - don't use File::Slurp to write the file, as it can't handle utf8 encoded data

Steinar - I'll do some more testing with file names tomorrow. Thanks for specifying the issue.
Comment 37 James Richardson 2008-05-07 10:49:17 UTC
Verified fixed in 7.0.1 - 19422 

If anyone else see strange behavior with UTF-8 please open a new bug with details of the errors.
Comment 38 James Richardson 2008-05-15 12:27:04 UTC
This bug has recently been fixed in the latest release of SqueezeCenter 7.0.1

Please try that version, if you still see the error, then reopen this bug.

To download this version, please navigate to: http://www.slimdevices.com/su_downloads.html
Comment 39 Chris Owens 2009-07-31 10:15:59 UTC
Reduce number of active targets for SC