Bugzilla – Bug 2475
Problems with filenames containing non-current-codepage (foreign language, double byte) characters
Last modified: 2009-05-29 09:10:51 UTC
I'm using SB2 with a windows xp machine running slimserver. I manually deleted the slimserver database. The Arialuni.ttf file is in the graphics directory. After a full scan I can't find albums with unicode titles (like "Studio Ghibli Songs 原声大碟 － 专辑"). I tried doing a search with sqlite database browser in the "tracks" table for records containing "脱" in the url field and found none. Actually I did copy&paste of the "脱" from a song file name. Using the web interface (default2), I also noticed that if I search for "原声大 碟" the search text input area contains "原声大碟" after the search. I checked this issue starting from version 20/10 up to 03/11 and the error is still there. this is a log of the search : 2005-10-24 19:54:36.4157 Used previous results for findKey 2005-10-24 19:54:36.4193 Used previous results for findKey 2005-10-24 19:54:36.4481 Used previous results for findKey 2005-10-24 19:54:45.2247 Start and End node: [contributor:contributor] 2005-10-24 19:54:45.2248 Field query. Need additional join. start and End node: [contributor:contributor] 2005-10-24 19:54:45.2249 Start and End node: [contributor:contributor] 2005-10-24 19:54:45.2249 Field query. Need additional join. start and End node: [contributor:contributor] 2005-10-24 19:54:45.2250 Start and End node: [album:contributor] 2005-10-24 19:54:45.2250 Field query. Need additional join. start and End node: [album:album] 2005-10-24 19:54:45.2254 Running SQL query: [SELECT COUNT(*) FROM (SELECT DISTINCT contributors.id AS id,contributors.name AS name,contributors.namesort AS namesort,contributors.moodlogic_id AS moodlogic_id,contributors.moodlogic_mixable AS moodlogic_mixable,contributors.musicmagic_mixable AS musicmagic_mixable FROM contributor_track, tracks, contributors, albums WHERE contributor_track.track = tracks.id AND contributors.id = contributor_track.contributor AND albums.id = tracks.album AND ( contributor_track.role IN ( ?, ? ) AND contributors.namesearch LIKE ? ))] 2005-10-24 19:54:45.2255 Bind arguments: [1, 5, %A�DA��%] 2005-10-24 19:54:45.3270 Start and End node: [album:album] 2005-10-24 19:54:45.3271 Field query. Need additional join. start and End node: [album:album] 2005-10-24 19:54:45.3278 Running SQL query: [SELECT COUNT(*) FROM (SELECT DISTINCT albums.id AS id,albums.title AS title,albums.titlesort AS titlesort,albums.contributor AS contributor,albums.compilation AS compilation,albums.year AS year,albums.artwork_path AS artwork_path,albums.disc AS disc,albums.discc AS discc,albums.musicmagic_mixable AS musicmagic_mixable FROM tracks, albums WHERE albums.id = tracks.album AND ( albums.titlesearch LIKE ? ))] 2005-10-24 19:54:45.3278 Bind arguments: [%A�DA��%] 2005-10-24 19:54:45.3906 Start and End node: [default:default] 2005-10-24 19:54:45.3907 Start and End node: [default:default] 2005-10-24 19:54:45.3907 Field query. Need additional join. start and End node: [default:default] 2005-10-24 19:54:45.3910 Running SQL query: [SELECT COUNT(*) FROM (SELECT DISTINCT tracks.id AS id,tracks.multialbumsortkey AS multialbumsortkey,tracks.thumb AS thumb,tracks.age AS age,tracks.remote AS remote,tracks.ct AS ct,tracks.audio AS audio,tracks.titlesort AS titlesort,tracks.album AS album,tracks.tracknum AS tracknum,tracks.url AS url,tracks.tag AS tag,tracks.title AS title,tracks.disc AS disc,tracks.fs AS fs FROM tracks WHERE ( tracks.audio = ? AND tracks.titlesearch LIKE ? ))] 2005-10-24 19:54:45.3910 Bind arguments: [1, %A�DA��%] thanks giuliano
I'm changing the title to what I think you are saying the problem is.
I can see the japanese characters are changed in the first message to something like "åå£°å¤§ç¢ ï¼ ä¸è¾"
Dan, can you comment?
I'm looking at this bug right now.
This works just fine for me - using tracks from Katamari Damacy to search for Namco(三宅 優) Could you please attach a zipped copy of your slimserversql.db file to this bug? Thanks.
Created attachment 991 [details] my library file.zip
Looking at your database, those chars don't exist in the album title: Loading resources from /Users/dsully/.sqliterc id title ------ ------------------- 959 Studio Ghibli Songs 960 Studio Ghibli SOngs (and it looks like one of the tracks has a typo of it's Album title) Please check that the song you are searching for actually contains the metadata you think it does. If this is still an issue, please attach one of the songs in question to this bug. Also - what does the SlimServer version line say at the bottom of the Settings page? Thanks.
Created attachment 1008 [details] a.rar unicode file sample. use winrar to extract
I Also noticed that I can't play a file using winamp if it's in a unicoded directory that contains latin characters (like "アニメ三銃士 OST 音楽篇")
it seems that the file is not in the database if the filename/dir contains unicode characters. renaming to 01.mp3 instead the original name seems to work.
Giuliano - what does the bottom line of your SlimServer -> Settings page say? cp1252? Or something else?
SlimServer -> Settings page : slimserver version 6.5b1 - 5310 - windows xp - en - cp1252
Unfortunately this is a low-level bug in Perl on Windows. I'm not sure if we're going to be able to fix it.. or it will take a while.
I've found a similar problem with winamp. the poster refers to multibyte chars (they are not unicode as far as I know). I googled a little and I found a MSVC library to convert from multibye to widechar (unicode ?), but nothing ready to use. Any idea on how can I convert filenames to something useable by slimserver/windows without losing the song name in the filename ? I wish to avoid renaming files to 01.mp3, 02.mp3 and so on :) thanks giuliano
Hi, any news about this one ? I'm googling around in my sare time and I've found I can play those files in media player classic. I also switched to mysql as the backend db without anything changes (I have the directory in the tracks db but no "songs" records. Att the files have the same attributes and have complete permissions for every user (win xp pro sp2) thanks giuliano
Unfortunately not. There is still a low-level Perl bug that we don't have a good solution for. My best suggestion right now is the work-around of renaming your files. Sorry.
Not getting much/any traction from the perl dev folks.. again, a work around is to rename your files. Pushing off past 6.5
*** Bug 3739 has been marked as a duplicate of this bug. ***
*** Bug 4263 has been marked as a duplicate of this bug. ***
*** Bug 4267 has been marked as a duplicate of this bug. ***
Could you post a bit more info on what the perl bug is and where in the code? I'm familiar enough with perl/win32/unicode that i'd like to take a look and see if i can get something working? Leverage the fact your open source :)
I may have stumbled onto the root cause, confirmation would be nice. The short brief: "I don't think this is possible from Perl code right now. You need to call CreateFileW() to open a file with a Unicode name. If you want to hack something, then I would suggest to write a little XS module that just swaps out the file handle in a PerlIO* structure. Look at PerlIOWin32_open() in win32/win32io.c to see how Perl currently opens a file. Another quick-and-dirty "solution" would be to build a custom Perl by hacking win32/win32.h. If you change the USING_WIDE definition to "1" then you end up with a version of Perl that has the old "-C" behavior hardcoded. Remember that this is not really compatible with Perl's Unicode handling." source: http://www.mhonarc.org/archive/html/perl-unicode/2004-06/msg00005.html Since i would like this working sooner than perl is likely to add unicode across their API's, it might be possible to get a intermediate solution in place. I'll dig through the source but if you could point me to a location i'd appreciate it. Hopefully i can help.
*** Bug 4470 has been marked as a duplicate of this bug. ***
The issue at hand is not just Japanese. SlimServer on Windows is sensitive to the codepage that is set in ControlPanel->RegionalAndLanguageOptions->Advanced->LanguageForNon-UnicodePrograms Any file name with a character that is not in the codepage is currently ignored by scanner. For example: When codepage is set to Hebrew (cp1255), the scanner ignores all files & folders with names that have accented characters e.g. the character á which does not exist in the cp1255 codepage If codepage is set to English (cp1252), the scanner ignores all files & folders with names that have Hebrew characters The following link points to a bug fix in ActiveState build 820 which may provide a possible solution: http://bugs.activestate.com/show_bug.cgi?id=60967
We need to fix this bug. Although it's an Activestate issue, it's been affecting our customers for years. The workaround y360 points to may be usable, or we may have to roll our own somehow.
From what I can tell the "bug fix" linked in Comment #24 would require SlimServer to be able to use the Windows 8.3 character name equivalents to get around the non current codepage issue. Can this change be implemented in SlimServer for Windows? How difficult would it be? What would be the down sides?
It looks like build 820 of ActivePerl does fix this issue, returning an 8.3 path for this file. However, we call Win32::GetLongPathName on all paths, and I'm not quite sure why we do that, as all the paths are already in long file format if possible. With that line commented out (line 571 in Slim::Utils::Misc), the file is scanned and displayed properly.
I commented out that line in trunk, and so we will need to test tonight's nightly build of 7.0 and see if the bug is fixed. We may need to update ActivePerl on our win32 build box as well.
Andy, that small change along with ActivePerl 820 works great! The build box was updated a couple of weeks ago so no problem there. Is this a safe change for 6.5.4? I did have to edit 'types.conf' since my flac files were not showing up. Turns out 'flac' gets shortened to 'fla' which I had to add. Any risk in adding this change as well? Thanks!
I'm not sure of the implications of removing that line of code, I need to look at where it's used more carefully, which I'll do in a week or so. Good point on types.conf needing .fla for 8.3 FLAC files.
One thing to know is that 8.3 filename creation can be disabled in Windows. I guess disabling them offers some degree of performance gain. System Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem Value Name: NtfsDisable8dot3NameCreation Data Type: REG_DWORD (DWORD Value) Value Data: 0=disable, 1=enable
I am going to backport this fix to 6.5.4. I looked around and it does not seem that the extra call to GetLongPathName is necessary. There is a mention in this area of the code regarding cue sheets, so if you guys could do some extra testing with cue sheets that reference both long filenames as well as 8.3 names, that would be helpful. Please mark this bug fixed if you don't see any problems.
That sounds good, Andy. It will give users an opportunity to try the fix on their wide, weird variety of PCs, too.
Just got a positive feedback by a Japanese user (SS6.5.4 nightly): http://forums.slimdevices.com/showthread.php?t=36958#18
QA: can you guys test folders with foreign characters, see this post: http://forums.slimdevices.com/showthread.php?p=219097#post219097
I haven't tested thoroughly but I did see that this is successfully working now. Finally!
(In reply to comment #35) > QA: can you guys test folders with foreign characters, see this post: > http://forums.slimdevices.com/showthread.php?p=219097#post219097 Andy, I did test it with folders with foreign characters. For example C:\audio_test\Юрий Визбор - Солнышко лесное [MP3](128)\01 - Солнышко Лесное.mp3 becomes C:\audio_test\-_MP3_~1\01-~1.MP3 in SlimServer. I suspect that the problem that Yiannis is seeing may have something to do with using external USB drives.
Several people are calling this bug fixed now after Andy's changes, so I'll go ahead and mark it fixed. Please let us know how it works for you! Also a note: this bug is really not fixed in the 'right way' since it is an Activestate bug. Our 'fix' is a workaround for their bug.
Hello, I can't get the scanner to find files like Z:\dischi\anime soundtracks\death note\16.死神界.mp3 Z:\dischi\JPop\w-inds - 单曲\03.夏祭り.mp3 (this folder is not recognized too) is there something I can try ? I'm using the latest nighty build disk Z is a local disk (sata) NTFS formatted if that matters thanks giuliano
Giuliano, I renamed some of my mp3s using the names you provided and SlimServer has no problem seeing them. Using the examples you provided: Z:\dischi\anime soundtracks\death note\16.死神界.mp3 Z:\dischi\JPop\w-inds - 单曲\03.夏祭り.mp3 became the following in SlimServer for me: C:\dischi\anime soundtracks\death note\16MP3~1.MP3 C:\dischi\JPop\W-INDS~1\03MP3~1.MP3 Is it possible that Windows is somehow disabling or not providing the 8.3 name equivalents to SlimServer? Perhaps the 8.3 equivalents are disabled. Is there an easy way to check? I'm at a loss here.
http://support.microsoft.com/?scid=kb%3Ben-us%3B121007&x=17&y=20 to check the 8.3 filenames in zp use nfsutil.exe behavior query disable8dot3 to modify the setting use nfsutil.exe behavior set disable8dot3 1|0 (1 disabled 8.3 filename creation, 0 enable 8.3 filename creation I'll reboot and check if something changes (mine was set to 1)
nothing happened after the reboot with 8.3 filenames enabled :( any other idea ?
Enabling 8.3 filenames won't create them for already existing files. Maybe there's a utility available somewhere that can go through your file system and create them on files where they're missing. I would guess that if you rename a file, Windows will create a 8.3 filename, so maybe a simple script that would rename every file then name it back to the original name would do this for you.
moved the files to another partition and moved back and now the files are recognized :) however filenames shown in the interface are in 8.3 format, but at last they're working :)
Hi, I updated from slimserver 6.5.3 to 6.5.4 in the process of moving my slimserver to an different machine and after the initial scan to discover my music files, found that 25 of my albums had gone missing. This is the same number that have foreign accented characters in their file names. e.g.: Acadamy of Ancient Music, Richard Egarr - Händel Concerti grossi Op.3, Sonata a 5.cue Beethoven - Beethoven Symphonies Nos.3 & 8 - Minnesota Orchestra - Vänskä.cue Dvorák, Antonín - Cello Concerto, Dumky Trio.cue Kölner Kammerorchester, Helmut Müller-Brühl - J.S. Bach - Messe in h-Moll, BWV 232 (CD2).cue etc. etc. My library is one flat directory of EAC-generated CUE sheets with a single WAVE file corresponding to each CUE sheet. Real simple. Foobar2000 can find them all and play them all (yes, I checked). So I tried "clear library and rescan everything" several times with various debug options turned on and found no useful error messages. In the end, I found this thread and reverted back to 6.5.3 and the initial scan found all my music again! After reverting slimsever the following is my system (was 6.5.4 when the problem showed) and the code page is and was cp1252. All music is stored on an NTFS formated drive intialized from within Windows XP. SlimServer Version: 6.5.3 - 12376 - Windows XP - EN - cp1252 Server IP address: 192.168.1.111 Perl Version: 5.8.8 MSWin32-x86-multi-thread MySQL Version: 5.0.22-community-nt Regards, Charles Razzell
Charles, this sounds like a different problem. I've asked our support team to contact you to try and work out your specific issue.
Changed milestone to keep this bug from interfering with current tasks while leaving it open to resolution.
Charles, did anyone from our support group contact you? Were they able to get it working?
This bug originally was about any music containing non-ascii characters not working, and all the previous comments are about that work. With regard specifically to the CUE sheet problem, I'd like to move your comments over to bug 4276 and re-close this one (unless, of course, all music with non-ascii characters stops working again).
This bug appears to have been fixed in the latest release! If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look. Make sure to include the version number of the software you are seeing the error with.