Bugzilla – Bug 3727
Unicode genres are duplicated between MP3 and FLACs
Last modified: 2009-07-31 10:13:50 UTC
A problem exists with unicode (Cyrillic) genre listing in web UI. If I select "browse Genres", the english genre list appears in alphnumeric order, followed by the similary sorted cyrillic genre list. However, the same cyrillic genres also appear in the middle of the english genres. Attachments illustrate this. It appears that MP3 tracks are associated with the "correctly" placed cyrillic genres, while FLACs are listed under cyrillic genres that are placed in the middle of the english list. SlimServer Version: 6.3.0 - 8148 - Windows Server 2003 - EN - cp1251
Created attachment 1332 [details] MP3 genres are listed in the end of the genre list
Created attachment 1333 [details] FLAC genres are listed in the middle of the genre list
Yuly - does this problem exist in the latest 6.5 nightly? Thanks
Subject: Re: Unicode genres are duplicated between MP3 and FLACs Hi Dan, I haven't tried the 6.5 yet. I suppose I can maintain a separate install of 6.5 in another directory, and it will create its own database, right? If yes, I'm going to try - it's just that the latest build does not (yet?) have a windown build. Regards, Yuly On 7/14/06, Slim Devices Bugzilla <bugs@bugs.slimdevices.com> wrote: > > https://bugs-archive.lyrion.org/show_bug.cgi?id=3727 > > > dan@slimdevices.com changed: > > What |Removed |Added > > ---------------------------------------------------------------------------- > Status|NEW |ASSIGNED > > > > > ------- Comment #3 from dan@slimdevices.com 2006-07-13 23:37 ------- > Yuly - does this problem exist in the latest 6.5 nightly? > > Thanks > > > > > ------- You are receiving this mail because: ------- > You reported the bug, or are watching the reporter. > <div>Hi Dan,</div> <div> </div> <div>I haven't tried the 6.5 yet. I suppose I can maintain a separate install of 6.5 in another directory, and it will create its own database, right?</div> <div>If yes, I'm going to try - it's just that the latest build does not (yet?) have a windown build.</div> <div> </div> <div>Regards,</div> <div> </div> <div>Yuly<br><br> </div> <div><span class="gmail_quote">On 7/14/06, <b class="gmail_sendername">Slim Devices Bugzilla</b> <<a href="mailto:bugs@bugs.slimdevices.com">bugs@bugs.slimdevices.com</a>> wrote:</span> <blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><a href="https://bugs-archive.lyrion.org/show_bug.cgi?id=3727">https://bugs-archive.lyrion.org/show_bug.cgi?id=3727 </a><br><br><br><a href="mailto:dan@slimdevices.com">dan@slimdevices.com</a> changed:<br><br> What |Removed |Added<br>---------------------------------------------------------------------------- <br> Status|NEW |ASSIGNED<br><br><br><br><br>------- Comment #3 from <a href="mailto:dan@slimdevices.com">dan@slimdevices.com</a> 2006-07-13 23:37 -------<br>Yuly - does this problem exist in the latest 6.5 nightly?<br><br>Thanks<br><br><br><br><br>------- You are receiving this mail because: -------<br>You reported the bug, or are watching the reporter.<br></blockquote></div><br>
The Windows build is up. I would recommend backing up your slimserver.pref file - but other than that, you can uninstall and reinstall pretty easily.
I have made a separate install of 6.5, created a test directory with 3 albums (US English FLAC, Cyrillic MP3 and Cyrillic FLAC, latter 2 having the same genre). 6.5 have correctly picked just 2 genres(1 for US English album, and 1 for both Cyrillic albums). When I tried 6.3 on the same test directory, it created a category list of 4, with Cyrillic genre being listed twice. So I'd say 6.5 does not have this problem.
Ok.. let's keep you on 6.5 then. There are too many issues with 6.3 to try and fix it there, without breaking other things.
Well, while I certainly agree this is not a major issue and one could live with it, 6.5 is still far from being stable. My short experience with it showed some font issues, things became case-sensitive all over the genre list, etc. Cheers, Yuly
Could you please file another bug regarding the Genre issue? 6.5 is becoming more stable every day - and it allows much faster bug fixes for any issues. Thanks
This problem has returned some time ago in 6.5.1. I currently have the yesterday's nightly build on Windows, and my cyrillic genres are again duplicated between Mp3 and FLAC tracks. My previous image from 1/12/2006 also had this problem. This is what is running now: SlimServer Version: 6.5.1 - 11040 - Windows Server 2003 - EN - cp1251 Server IP address: 192.168.11.6 Perl Version: 5.8.8 MSWin32-x86-multi-thread MySQL Version: 5.0.22-community-nt
I have uploaded a sample directory structure with 3 albums and 3 tracks in bug 4485, which does illustrate this bug too. Here us the link: https://bugs-archive.lyrion.org/attachment.cgi?id=1773&action=view
Any news about this one?
Yuly, was this fixed for a while in the 6.5.2 beta? That's what it sounds like if I read between the lines. But now the problem is back?
It was fixed in 6.5 beta, and returned back with 6.5.1. It's present now in 6.5.2 current
seems to be a windows codepage problem. Running 6.5.2 11686, I'm seeing the genre as one with two tracks. I'm also seeing the artwork (bug 4485). In the songinfo, the path isn't displaying with cyrillic however, so I can certainly believe there is a problem somewhere. I may have some time to test with windows tomorrow. If anyone else is into it, try d_scan, d_info for logging.
I have started the server with a test audio directory, using the sample music library. Cache was empty. d_scan and d_info were used. After scan, the library shows as having 3 genres, with cyrllic one duplicated, exactly as per this bug description. C:\Program Files\SlimServer\server>Slim --audiodir=g:\audio_test --d_scan --d_in fo 2007-05-02 16:29:32.3097 utf8 "\xC1" does not map to Unicode at /PerlApp/Slim/Ut ils/Strings.pm line 128, <STRINGS> chunk 1. 2007-05-02 16:29:32.4772 utf8 "\xC1" does not map to Unicode at /PerlApp/Slim/Ut ils/Strings.pm line 128, <STRINGS> chunk 1. 2007-05-02 16:29:35.1066 Executing SQL file C:\Program Files\SlimServer\server\M ySQL\system.sql 2007-05-02 16:29:35.4750 ERROR: Running service shutdown failed! 2007-05-02 16:29:42.0754 Warning: Migrating from 6.3.x used with MySQL! 2007-05-02 16:29:42.3157 Connected to database dbi:mysql:hostname=127.0.0.1;port =9092;database=slimserver - schema version: [3] 2007-05-02 16:29:42.3184 Migrated database from schema version: 0 to version: 3 2007-05-02 16:29:43.2629 loading types config file... 2007-05-02 16:29:46.6066 UPnP: Failed to initialize multicast socket, disabling UPnP. 2007-05-02 16:29:49.1847 checkDataSource - updated schema or no tracks in the da tabase, initiating scan. 2007-05-02 16:30:20.8376
Yuli, that log doesn't look right. Did you stop the server before running the command line? There should be a full set of info regarding tag data and references to the files scanned. Once you start the server from command line, trigger a wipe and rescan from the web ui. Then please add the log as an attachment. It will make it a lot more readable than pasting here since the paste wraps lines in awkward places. thanks for taking the time. Hopefully the info will point to some section of code. Unfortunately, I've not had much time to do any windows tests. Ross, any chance you might be able to run similar tests?
Created attachment 1940 [details] test with audio_test folder Just tried this with 6.5.2 (svn rev 11931) SlimServer Version: 6.5.2 - TRUNK - Windows XP - EN - cp1252 Scan with audio_test folder shows two genres, with cyrillic as a single genre. Not sure what significance there might e between cp1252 and cp1251
Created attachment 1941 [details] another view
adding musicmagic to the mix, DOES mess things up.
If you are using MusicMagic, I have a possible solution: In Plugins/MusicMagic/Importer.pm, change line 290 from: $songInfo{$key} = Slim::Utils::Unicode::utf8decode_guess($songInfo{$key}, $enc); to: $songInfo{$key} = Slim::Utils::Unicode::utf8decode($songInfo{$key}, 'utf8'); This seems to get rid of the dupes for my case, and some messed up artist/album info at the songinfo level. I'm not sure if it is officially safe to assume utf8 for all musicmagic cases, but I'm looking into it. As it is a plugin, you should be avle to make the edit and restart the server to test changes without needing a patched build.
I'm not sure if my log is helpful, SlimServer doesn't seem to be indexing these files at all for some reason. SlimServer Version: 6.5.1 - 11206 - Windows Server 2003 - EN - cp1251
Created attachment 1944 [details] ross' log
I have tried this again, with latest 6.5.2. The log and screenshoot of the genre list is attached. The problem is still there. BTW, with this nightly I slim tray no longer gives me right-click menu:(. Is it only me?
Forgot the verion info: VERSION INFO SlimServer Version: 6.5.2 - 11968 - Windows Server 2003 - EN - cp1251 Server IP address: 192.168.11.6 Perl Version: 5.8.8 MSWin32-x86-multi-thread MySQL Version: 5.0.22-community-nt
Created attachment 1945 [details] Log file Server was started from the command line with --d_scan and --d_info. After that, full rescan was triggered from WebUI
Created attachment 1946 [details] Genre list
(In reply to comment #24) >BTW, with this nightly I > slim tray no longer gives me right-click menu:(. Is it only me? bug 4990
With 20 other songs (13 different genres) and cp1252 or cp1251 on windows 2003 server I'm not able to reproduce this. However I'm noticing some odd behavior. With cp1251 SlimServer will not index the test files, with cp1252 it indexes them just fine. Let me know if theres any other testing I can do to help.
Yuly, are you using musicmagic/MusicIP, moodlogic or iTunes along with a music folder setting? I can't seem to get scanner data working from the main server to the log, so logs really arent' helping. Might try with scanner.exe instead: scanner.exe --d_scan --d_import --d_info --wipe --logfile c:\log.txt --musicmagic --itunes --moodlogic each of the last 3 options are only needed if you are using one of the above importers. Unfortunately, it doesn't look like this can be handled in time for 6.5.2
I have stopped the slimserver, deleted the cache directory and started the scanner as follows: C:\Program Files\SlimServer\server>scanner.exe --d_scan --d_import --d_info --wipe --logfile c:\log.txt g:\audio_test The resulting log.txt is attached
Created attachment 2006 [details] Log from the manual scanner run on the test dir
call me stumped on this one. Other than the database connection problem at the start, that log looks fine. Based on that, I'd have expected the genres to mix together perfectly. I can't see any difference in them from the log. Maybe adding d_sql and d_mysql might show something, at least as far as what is being inserted into the tables. Unfortunately, I'm not good at reading sql so I may not be much help interpreting.
Marking as fix for 7.0 since we're out of time for 6.5.2
(In reply to comment #33) > call me stumped on this one. Other than the database connection problem at the > start, that log looks fine. Based on that, I'd have expected the genres to mix > together perfectly. I can't see any difference in them from the log. Maybe > adding d_sql and d_mysql might show something, at least as far as what is being > inserted into the tables. Unfortunately, I'm not good at reading sql so I may > not be much help interpreting. I have run the same scanner command with addition of d_sql (d_mysql is not valid) The log follows. It's pretty hard to read, since the cyrillic tags are presented in ugly form here. In particular, the cyrillic genre tag of wuth a value of Барды is shown in the log as 'Барды' - but as far as I can see, this string is identical for both mp3 and flac tracks.
Created attachment 2012 [details] Another scanner log, with addition of d_sql
d_mysql should be valid, so that's a bit odd. However, the log is finally showing us something to target. Not the lines: 2007-05-16 11:48:52.8896 INSERT INTO genres (name, namesearch, namesort) VALUES (?, ?, ?): 'Барды', 'ÐаÑÐ¥Ñ', 'БарХы' 2007-05-16 11:48:53.1132 INSERT INTO genres (name, namesearch, namesort) VALUES (?, ?, ?): 'Барды', 'БАРДЫ', 'БАРДЫ' The latter is the mp3 track, former is flac. for some reason the namesearch and namesort values are not matching up in the same way as the mp3. I don't know exactly where this is being handled (I would have expected them to both be handled in the same bit of code, however).
(In reply to comment #37) > d_mysql should be valid, so that's a bit odd. > However, the log is finally showing us something to target. Not the lines: > 2007-05-16 11:48:52.8896 INSERT INTO genres (name, namesearch, namesort) VALUES > (?, ?, ?): 'Барды', 'ÐаÑÐ¥Ñ', 'БарХы' > 2007-05-16 11:48:53.1132 INSERT INTO genres (name, namesearch, namesort) VALUES > (?, ?, ?): 'Барды', 'БАРДЫ', 'БАРДЫ' > The latter is the mp3 track, former is flac. for some reason the namesearch > and namesort values are not matching up in the same way as the mp3. I don't > know exactly where this is being handled (I would have expected them to both be > handled in the same bit of code, however). I'm beginning to see the light! :) hope ant last this small annoyance can be dealt with!
The annoyance is just starting for me. I still cannot see this on my system. Yes, there are inconsistencies in the namesearch and namesort values, but the second file gets a match on the genre anyway. I've even switched my codepage to 1251 SlimServer Version: 6.5.2 - TRUNK - Windows XP - EN - cp1251 Still no joy. I get just the one genre. Ross, if you want to try this out again, I had the same problem with indexing the files, but extracting the zip again while set to cp1251 seems to be ok.
upgraded to Perl 5.8.8 and no change. not sure what else I can suggest here. The code for adding genres tracks down to one location in the code Slim::Schema::Genre::add(), so I really don't know how a file format can make any difference. I can only guess that somehow the locale is doing something strange in the OS, or it's being set incorrectly in perl due to something in startup. random guesses. I'm afraid.
Any ideas about it? Anyone?
Ross if you would follow up I would appreciate it.
Yuly - do you still see this issue in 7.0?
I'm still on 6.5. I can try 7.0; any quick pointers to how can I keep my 6.5 and run 7.0?
After installing the latest 7.0, I can confirm the issue is still there. SqueezeCenter InformationSqueezeCenter Version: 7.0 - 16051 - Windows Server 2003 - EN - cp1251 Server IP address: 192.168.11.6 Perl Version: 5.8.8 MSWin32-x86-multi-thread MySQL Version: 5.0.22-community-nt Platform Architecture: 586 Hostname: yuly-home-new Server Port Number: 9000 Total Players Recognized: 3 Cache Folder: C:\Documents and Settings\All Users\Application Data\SqueezeCenter\Cache Plugin Folders: C:\Program Files\SqueezeCenter\server\Slim\Plugin,C:\Program Files\SqueezeCenter\server\Plugins Player InformationName: bathroom Model: squeezebox2
What ever happened to the test files for this bug? I'd be glad to try to reproduce again and attach a log.
(In reply to comment #46) > What ever happened to the test files for this bug? I'd be glad to try to > reproduce again and attach a log. Not sure If I got you correctly, but the test library I uploaded some time ago seems to be still there: https://bugs-archive.lyrion.org/attachment.cgi?id=1773&action=view I have seen the problem with my full library (~3K albums), but I see no reason the problem will not show itself with the test library too.
(In reply to comment #46) > What ever happened to the test files for this bug? I'd be glad to try to > reproduce again and attach a log. Have you looked at it lately? Do you need any assistance from my side? Thanks!
Ross to review post 7.0. Sorry for the delay, Yuly.
Yuly - as you might know I've added quite a few fixes to unicode handling in 7.0.1. Do you still see this issue with the latest nightly builds?
(In reply to comment #50) > Yuly - as you might know I've added quite a few fixes to unicode handling in > 7.0.1. Do you still see this issue with the latest nightly builds? Thanks for the update, Michael. I will try the latest 7.0.1. I will report in a week or so (it usually takes about that time to see the bug).
(In reply to comment #51) > (In reply to comment #50) > > Yuly - as you might know I've added quite a few fixes to unicode handling in > > 7.0.1. Do you still see this issue with the latest nightly builds? > Thanks for the update, Michael. I will try the latest 7.0.1. I will report in a > week or so (it usually takes about that time to see the bug). Sorry, mixed up with my other bug). The result will be evident immediately... stand by.
The bug is still there with the latest nightly: SqueezeCenter Version: 7.0.1 - 17981 - Windows Server 2003 - EN - cp1251 Server IP address: 192.168.11.6 Perl Version: 5.8.8 MSWin32-x86-multi-thread MySQL Version: 5.0.22-community-nt Platform Architecture: 586 Hostname: yuly-home-new Server Port Number: 9000 Total Players Recognized: 2 Cache Folder: C:\Documents and Settings\All Users\Application Data\SqueezeCenter\Cache Preferences Folder: C:\Documents and Settings\All Users\Application Data\SqueezeCenter\prefs Plugin Folders: C:\Program Files\SqueezeCenter\server\Slim\Plugin, C:\Program Files\SqueezeCenter\server\Plugins Player InformationName: bathroom Model: squeezebox2 Firmware: 86 The IP address for this player is: 192.168.11.3:19901 The ethernet MAC address for this player is: 00:04:20:05:c3:12 Name: salon Model: Squeezebox v3 Firmware: 86 The IP address for this player is: 192.168.11.5:23462 The ethernet MAC address for this player is: 00:04:20:06:34:22 SqueezeCenter Log FileSqueezeCenter keeps a log file for all application related activities (Audio Streaming, Infrared, etc) here: C:\Documents and Settings\All Users\Application Data\SqueezeCenter\Logs\server.log Scanner Log FileSqueezeCenter keeps a log file for all scanning related activities, including iTunes & MusicIP here: C:\Documents and Settings\All Users\Application Data\SqueezeCenter\Logs\scanner.log
Michael, do you need more support from QA to investigate this bug?
Well, I guess I'm at the same point as kdf - whatever I do, I can't reproduce the issue with these files.
I'm able to reproduce this with windows server 2003, SC 7.0.1 - 18232, CP 1251. Note, I had to set both my regional option as well as the language pack (under advanced). I noticed some strange things, the little language selection toolbar indicated the system was switching back to EN after I chose MO (Mongolian). I rebooted and this behavior seemed to stop, and then this bug is easily reproduced I see the duplicate genre. I'll try reproducing from the command line, having some trouble getting trunk to work with server 2003.
Ross - would you mind providing me with that W2003 VM so I can play with it? How big is the image?
Sending to Michael via FTP now.
Ross: Please follow up with Michale to make sure he gets the image, or clear repro instructions
At first it seemed the transfer went fine, but Michael got an error about cannot open, Failed to lock the file. I'm not sure what that means. Michael let me know if you'd like to give transferring another try, or if there is anything else you'd like from QA regarding this bug.
Yuli (or Ross) - if you are running the perl version of SC instead of the compiled binary, please give the following simple change a try: Index: /Users/mh/Documents/workspace/Teststoff/Slim/Formats/FLAC.pm =================================================================== --- /Users/mh/Documents/workspace/Teststoff/Slim/Formats/FLAC.pm (revision 18669) +++ /Users/mh/Documents/workspace/Teststoff/Slim/Formats/FLAC.pm (working copy) @@ -58,7 +58,7 @@ 'DISC #' => 'DISC', ); -my @tagNames = (Slim::Schema::Contributor->contributorRoles, qw(ALBUM DISCNUMBER TITLE TRACKNUMBER DATE)); +my @tagNames = (Slim::Schema::Contributor->contributorRoles, qw(ALBUM DISCNUMBER TITLE TRACKNUMBER DATE GENRE)); # peem id (http://flac.sf.net/id.html http://peem.iconoclast.net/) my $PEEM = 1885693293; This basically adds the GENRE to the tags which should utf8 decoded before storing them in the DB. I'm sorry, I wasn't able to reproduce. Will have to do an install from scratch. My Hungarian box uses cp1250...
Looks like it did the trick!
Wow! Great news! Ross - please give change 18740 some thorough testing. Thanks everyone.
Not sure if this is connected, however 1 day after I'm running the Perl version with the fix in FLAC.pm, I once again got hit by 4851 (https://bugs-archive.lyrion.org/show_bug.cgi?id=4851). Shall I re-open it?
Ross: can (In reply to comment #64) > Not sure if this is connected, however 1 day after I'm running the Perl version > with the fix in FLAC.pm, I once again got hit by 4851 > (https://bugs-archive.lyrion.org/show_bug.cgi?id=4851). > Shall I re-open it? > Ross: Can you please check this out? then change the status appropriately
(In reply to comment #65) > Ross: can (In reply to comment #64) > > Not sure if this is connected, however 1 day after I'm running the Perl version > > with the fix in FLAC.pm, I once again got hit by 4851 > > (https://bugs-archive.lyrion.org/show_bug.cgi?id=4851). > > Shall I re-open it? > > > Ross: Can you please check this out? then change the status appropriately This (3727) bug is really fixed; at least I do not experience the original symptoms after the change 18740 was implemented. 4851 is correctly re-opened.
Okay then! If anyone's still seeing this problem feel free to reopen.
This bug has now been fixed in the 7.1 release version of SqueezeCenter! Please download the new version from http://www.slimdevices.com if you haven't already. If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.
Reduce number of active targets for SC