Bug 16180 - Files with UTF-8 filenames not showing up.
: Files with UTF-8 filenames not showing up.
Status: CLOSED FIXED
Product: Logitech Media Server
Classification: Unclassified
Component: Scanner
: 7.6.0
: PC Ubuntu Linux
: -- normal with 5 votes (vote)
: 7.6.0
Assigned To: Alan Young
: charset_issues
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-05-04 20:20 UTC by Dave Cook
Modified: 2011-04-27 15:42 UTC (History)
4 users (show)

See Also:
Category: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dave Cook 2010-05-04 20:20:00 UTC
This occurs both in the "Music Folder" menu and any listing by ALBUM.  Tags with UTF-8 characters seem to be OK.  An example UTF-8 filename is

'2-03 Chabrier_ Haban\xcc\x83era.flac'

Version: 7.6.0 - r30722 @ Mon May 3 01:02:08 PDT 2010
Server HTTP Port Number: 9000
Operating system: Debian - EN - utf8 (Ubuntu 9.04)
Platform Architecture: i686-linux
Perl Version: 5.10.0 - i486-linux-gnu-thread-multi
Database Version: DBD::SQLite 1.29 (sqlite 3.6.22)
Comment 1 Mikael Nyberg 2010-05-04 21:20:35 UTC
Yep it seems broken again.

If I use "Browse Music Folder" my Björk and Röyksopp albums are gone.

I it is so that if you have  2 folders in a row with a "foreign" character.
Then it the second folder is gone.

Example

/media/music/files/Lossles Music/Röyksopp/Röyksopp - Junior 2009

Or If you also have song with a foreign character in the name
That song dissapears in a folder with "foreign" character in the name.

As I can see the Björk folder but it is empty.

So the first "foreign" character is breaking the decoding of the following ones.

We have had *exactly* this once before ? I emptied the mailbox of my replies cant remember the bug nr, apparently these character bugs don't stay fixed.

I checked the file names from the shell they all look ok.
No mystery smb file transfer bug this time around.

My files still shows in the artist list, but I have not yet done a scan since i installed 7.6.0-30722

My system:

Version: 7.6.0 - r30722 @ Sat May 1 02:03:51 MDT 2010
Hostname: hal.home.lan
IP: 192.168.1.5
HTTP Port: 9000
OS: Red Hat - EN - utf8
Platform: i686-linux
Perl Version: 5.8.8 - i686-linux-thread-multi
Database Version: DBD::SQLite 1.29 (sqlite 3.6.22)
Total Players Recognized: 3

And to verify my locale:

[root@hal ~]# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Comment 2 Dave Cook 2010-05-05 18:21:59 UTC
Example error from server.log:

[10-05-05 18:18:46.5776] Slim::Formats::readTags (157) File missing: /usr/local/share/Music/Music/CDs/Tortelier/Introuvables/cd3/3-11 Prelude in E flat minor (extr. du _Clavier bien tempéré_).flac
[10-05-05 18:18:46.5779] Slim::Utils::Scanner::Local::__ANON__ (619) ERROR SCANNING file:///usr/local/share/Music/Music/CDs/Tortelier/Introuvables/cd3/3-11%20Prelude%20in%20E%20flat%20minor%20(extr.%20du%20_Clavier%20bien%20temp%C3%A9r%C3%A9_).flac: Unable to read tags from file
Comment 3 Mikael Nyberg 2010-06-29 04:05:16 UTC
More logs for you:

[10-06-29 12:54:08.6389] Slim::Utils::Misc::fileURLFromPath (314) Warning: fileURLFromPath got decoded UTF-8 path: "/media/music/files/CD/Bj\xF6rk/Bj\xF6rk - Vespertine"
[10-06-29 12:54:08.6409] Slim::Utils::Misc::msg (1179) Warning: [12:54:08.6395] Backtrace:

   frame 0: Slim::Utils::Misc::fileURLFromPath (/usr/local/bin/myperl/lib/site_perl/Slim/Utils/Misc.pm line 602)
   frame 1: Slim::Utils::Misc::fixPath (/usr/local/bin/myperl/lib/site_perl/Slim/Utils/Misc.pm line 561)
   frame 2: Slim::Utils::Misc::fixPath (/usr/local/bin/myperl/lib/site_perl/Slim/Web/Pages/BrowseTree.pm line 104)
   frame 3: Slim::Web::Pages::BrowseTree::browsetree (/usr/local/bin/myperl/lib/site_perl/Slim/Web/HTTP.pm line 1097)
   frame 4: Slim::Web::HTTP::generateHTTPResponse (/usr/local/bin/myperl/lib/site_perl/Slim/Web/HTTP.pm line 924)
   frame 5: Slim::Web::HTTP::processURL (/usr/local/bin/myperl/lib/site_perl/Slim/Web/HTTP.pm line 735)
   frame 6: Slim::Web::HTTP::processHTTP (/usr/local/bin/myperl/lib/site_perl/Slim/Networking/IO/Select.pm line 139)
   frame 7: (eval) (/usr/local/bin/myperl/lib/site_perl/Slim/Networking/IO/Select.pm line 123)
   frame 8: Slim::Networking::IO::Select::__ANON__ (/usr/local/bin/myperl/lib/site_perl/Slim/Networking/IO/Select.pm line 183)
   frame 9: (eval) (/usr/local/bin/myperl/lib/site_perl/Slim/Networking/IO/Select.pm line 183)
   frame 10: Slim::Networking::IO::Select::loop (/usr/libexec/squeezeboxserver line 682)
   frame 11: main::idle (/usr/libexec/squeezeboxserver line 632)
   frame 12: main::main (/usr/libexec/squeezeboxserver line 1125)

[10-06-29 12:54:13.2175] Slim::Utils::Scanner::Local::rescan (143) Discovering files in /media/music/files/CD/Björk/Björk - Debut
[10-06-29 12:54:18.1186] Slim::Utils::Scanner::Local::__ANON__ (214) Removing deleted files (0)
[10-06-29 12:54:18.1196] Slim::Utils::Scanner::Local::__ANON__ (267) Scanning new files (0)
[10-06-29 12:54:18.1206] Slim::Utils::Scanner::Local::__ANON__ (320) Rescanning changed files (0)
Comment 4 Mikael Nyberg 2010-06-29 04:09:11 UTC
Another thing related or not, using BMF seems to trigger a "look for new and changed" scan in all levels above where there is any file's ?

So by choosing "music" in BMF it scans all sublevels for new and changed directly ??

Is this another known bug ?

[10-06-29 12:16:39.9616] Slim::Utils::Scanner::Local::rescan (143) Discovering files in /media/music
[10-06-29 12:16:47.1532] Slim::Utils::Scanner::Local::__ANON__ (214) Removing deleted files (0)
[10-06-29 12:16:47.1543] Slim::Utils::Scanner::Local::__ANON__ (267) Scanning new files (30)
[10-06-29 12:16:48.7600] Slim::Utils::Scanner::Local::__ANON__ (320) Rescanning changed files (0)
[10-06-29 12:24:40.7414] Slim::Utils::Scanner::Local::rescan (143) Discovering files in /media/music/files
[10-06-29 12:24:46.4651] Slim::Utils::Scanner::Local::__ANON__ (214) Removing deleted files (0)
[10-06-29 12:24:46.4662] Slim::Utils::Scanner::Local::__ANON__ (267) Scanning new files (30)
[10-06-29 12:24:48.0642] Slim::Utils::Scanner::Local::__ANON__ (320) Rescanning changed files (0)
[10-06-29 12:33:26.1044] Slim::Utils::Scanner::Local::rescan (143) Discovering files in /media/music/files/CD
[10-06-29 12:33:31.4643] Slim::Utils::Scanner::Local::__ANON__ (214) Removing deleted files (0)
[10-06-29 12:33:31.4654] Slim::Utils::Scanner::Local::__ANON__ (267) Scanning new files (0)
[10-06-29 12:33:31.4664] Slim::Utils::Scanner::Local::__ANON__ (320) Rescanning changed files (0)
[10-06-29 12:33:31.8196] Slim::Utils::Misc::fileURLFromPath (314) Warning: fileURLFromPath got decoded UTF-8 path: "/media/music/files/CD/Anna-Lotta Larsson - V\xE4dur"
[10-06-29 12:33:31.8215] Slim::Utils::Misc::msg (1179) Warning: [12:33:31.8204] Backtrace:
Comment 5 Alan Young 2010-10-25 08:49:57 UTC
I have a bunch of related changes that are not yet ready for checkin. They should resolve these issues.
Comment 6 Alan Young 2010-12-23 05:32:33 UTC
Mikael, Have you had a chance to try this issue with the latest 7.6?
Comment 7 Mikael Nyberg 2010-12-23 10:46:02 UTC
It works ok for me right now , so it appears to be fixed for this time around .
Comment 8 Paul Chandler 2011-04-26 17:00:30 UTC
I can still see the issue in:
SqueezeboxServer-7.6.0-32302.exe   (Windows)

Browse Music Folder ---shows empty folders if the folder has an Umlaut and the file contained in that folder also has a UTC character
Comment 9 Paul Chandler 2011-04-27 11:14:55 UTC
(In reply to comment #8)
> I can still see the issue in:
> SqueezeboxServer-7.6.0-32302.exe   (Windows)
> 
> Browse Music Folder ---shows empty folders if the folder has an Umlaut and the
> file contained in that folder also has a UTC character

OK, it doesn't matter if the FILE name has any odd characters or not.
If the any of the folders has extended asciii, the FILE will not appear in the Music Folder on either the Radio or the Touch.  
There are 3 bugs that are about the same: 16180, 16113 and 15460
[tested with 7.6.0 32364  (fw 9430 for touch, 9432 for radio)
Comment 10 Paul Chandler 2011-04-27 15:42:36 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > I can still see the issue in:
> > SqueezeboxServer-7.6.0-32302.exe   (Windows)
> > 
> > Browse Music Folder ---shows empty folders if the folder has an Umlaut and the
> > file contained in that folder also has a UTC character
> 
> OK, it doesn't matter if the FILE name has any odd characters or not.
> If the any of the folders has extended asciii, the FILE will not appear in the
> Music Folder on either the Radio or the Touch.  
> There are 3 bugs that are about the same: 16180, 16113 and 15460
> [tested with 7.6.0 32364  (fw 9430 for touch, 9432 for radio)

Seems to be working correctly now after a lean install of r32364
on both Mac 10.6.7, XP32 and Win764. Will test on Ubuntu