Bugzilla – Bug 3252
UTF-8 encoded characters in path confuse "Browse Music Folder"
Last modified: 2011-03-16 04:18:47 UTC
I noticed last night that when I choose "Browse Music Folder" in the Web UI, none of the files with non-ASCII characters in the path were showing up. (In this case, SlimServer is running on an OS X server accessing files stored on a NAS mounted via NFS.) I checked the encoding of the file names by byte analyzing the inode, and the encoding is correct. I have a bunch of files under a folder that contains an "O" with an umlaud, and the UTF-8 encoding is showing up as C396, which is correct for Unicode character 00D6, "Latin Capital Letter O With Diaeresis". The log shows a bunch of errors about a bad file name. I poked through the code, and found the problem was in Slim::Utils::Misc::readDirectory. I added a bunch of debug messages, and found the directory scan was working correctly up until the point where relative names were being converted to absolute paths via my $fullpath = catdir($dirname, $item); I found that while item was encoded correctly, the resulting fullpath is not - it mapped O with umlaud to the sequence "O\314\210". That translates to a 'O' followed by the UTF-8 character cc88, or Unicode 0308, "Combining Diaeresis". Then it tries to access the file by the full path, with the encoding changed, which fails. As a workaround, I replace the call above with "my $fullpath = $dirname . "/" . $item;" which works fine on OS X. I'm not sure if this is a generic bug in the Perl catdir command or unique to the version of Perl that is shipped with OS X.
Have you tried the latest nightly builds (6.2.2 for stable)? See the link in note 1 on the bugzilla home page.
I'll try it tonight if I get the time.
I just tested with 6.2.2 and the same problem occurs, and the same change to readDirectory fixed it. To make it clear, I don't think the bug is actually in the slim server but in perl - it appears that the catdir function is for some reason re-encoding the names, and using a different way of representing the same character. (The joys of unicode combining symbols...) Does seem like something worth putting in a workaround for. The perl I'm using: perl -version This is perl, v5.8.1-RC3 built for darwin-thread-multi-2level (with 1 registered patch, see perl -V for more detail) The patch is the RC3 patch.
This seems like the same bug that prompted a work around in Misc::pathFromFileURL() - see that function for comments.
Dan, I think you are right - I've read the bug entry refered to in the code, and it sounds like the same issue. In fact, it explains why it would work if I browsed the Music Folder right after starting the server - the recycled buffers hadn't been set to ASCII yet. I'll test using the same fix you pointed to instead of my hack and let you know what I find. If it works, what's the proper method for submitting a proposed patch?
attach a diff -upB, or svn diff to this report and someone with commit access will merge it in for you. If you are interested in more long-term development, talk to dan about getting your own access to svn.
I've been playing around with this some more. The change I mentioned earlier fixes the browse folder problem, but I turned up a number of other issues, all related to Perl 3.8.1's handling of UTF-8. I've never liked working around other people's bugs.... I finally gave up and upgraded the version of Perl on my machine to 5.8.6 (via Fink). A complete rescan of my entire library looks perfect, and browsing the music folder works too. I'm going to close out this bug with a workaround of updating to a non-broken version of Perl. I see on the Wiki that you are thinking of including a newer version of Perl for OS X with the next release of SlimServer. OS X 10.4 comes with 5.8.6, so you should only consider it for 10.3 and earlier. Perhaps pointing people to either the ActiveState download or the Fink download would make sense. Both distro's install in a different place than the built-in perl, so the start up script needs to be modified to use the later version of Perl. I've run into a few UI related issues with Unicode and I'll post those when I have isolated them more.