Bugzilla – Bug 8637
Searching for files with non-ASCII characters on linux works intermittently
Last modified: 2009-08-10 02:19:46 UTC
The following is an excerpt from a customer that has reported this issue. I have installed SqueezeCenter on my Ubuntu/Linux box. I am running Ubuntu 7.04. SqueezeCenter information: SqueezeCenter Version: 7.0.1 - 19705 @ Wed May 14 19:53:57 PDT 2008 - Debian - EN - utf8 Perl Version: 5.8.8 i486-linux-gnu-thread-multi MySQL Version: 5.0.38-Ubuntu_0ubuntu1.4 Platform Architecture: i686-linux Everything seems to work fine except when searching from the SqueezeCenter web interface. When searching for something containing non-ASCII chars (e.g. the Danish characters "æøåÆØÅ") like e.g. "røde" I would expect to find 4 songs, but finds nothing. I can browse my way to the songs and all characters are displayed correctly in SqueezeCenter and in my players. When I restart the SqueezeCenter service like this: sudo /etc/init.d/squeezecenter restart the problem disappears -- searching for "røde" finds the 4 songs I would expect. Then everything works fine for a day or two and then suddenly the problem reappears.
Ross, could you try to reproduce this for a fix in the 7.2 timeframe?
Hello, I'm the customer that reported the problem. Currently, my SqueezeCenter is having the problem. If I can in any way be helpful by fetching some logs or maybe enabling some debugging information then please let me know. Currently searching for "røde" or "Røde" returns nothing whereas searching for "staterne" returns: Song titles matching "staterne": 1 15. Røde i Staterne by Shu-bi-dua from Shu-bi-dua 200 (CD 07) So as you can see, everything is displayed correctly, but I don't find what I expect when searching for non-ASCII chars.
Very strange; I reproduced this one time but cannot again since. MGD do you ever reboot this system? I ask because I reproduced this and then rebooted, and haven't been able to reproduce since. I tagged an artist with røde and searched for that.
When I first encountered the problem, I rebooted the system after a while (for other reasons) and discovered that the problem disappeared. When I later encountered the problem, I simply restarted the SqueezeCenter service which cured the problem for a while. Right now, my system is in a state where it does not work and I left it there on purpose if it could in some way help debugging the problem. Please let me know if there is some debug info I can extract from the system which would help you diagnose the problem.
Michael would you mind looking at this? As mentioned I was able to reproduce this first try, maybe you'll be as lucky.
Hmm... seems to be working fine for me. But then I've already restarted my SC several times since I've rebooted that machine (updates). Do you see a different locale setting in SC's version string when you run SC right after reboot or after the service's restart?
No it's the same (Debian - EN - iso-8859-1). I'm still able to reproduce this what seems like randomly. I restart my virtual image and search for 'røde' and about 1 in 3 times there are no results.
> No it's the same (Debian - EN - iso-8859-1). I'm still able to reproduce this Could you please try with 7.1 and add "--charset=utf8" to your startup parameters? > what seems like randomly. I restart my virtual image and search for 'røde' and > about 1 in 3 times there are no results. What search field are you using? Always the same?
I'm always clicking search, not using the mouse over search. Thanks for your help (via email) regarding the parameter, I'll try looking at this again after 7.1.
No longer able to reproduce this with 7.2. I have tried Michael's suggestion and added CHARSET=utf8 to /etc/init.d/squeezecenter. SC7 shows iso-8859-1, so it's possible this isn't the right place still. I'll continue to investigate this. MGD, care to try out a nightly?
I know it hasn't been long but, I think I found a way to reproduce this much more easily Michael. I was using a music library of only a few tracks before, now that I'm over 1000 tracks this is reproducible almost every time for me.
Ross, (In reply to comment #10) > I'll continue to investigate this. MGD, care to try out a nightly? I guess this is no longer necessary since you are able to reproduce yourself. Otherwise, let me know what to do.
(In reply to comment #12) > I guess this is no longer necessary since you are able to reproduce yourself. > Otherwise, let me know what to do. Please hang in there it may take time to better understand this issue and find a solution. Thank you MGD.
recently i ran into the same problem (7.3r23017) on my gentoo linux utf-8 system. I've excluded browser & mySQL from the problem (through various tests) and inserted some debug code into Slim/Web/Pages/Search.pm to better pin-point the problem. For me the best way to reproduce the problem (so far) seems to be a long running slimserver process. A restart always cured the problem. I'll report back if/when i find anything...
Created attachment 3948 [details] Patch to Slim/Web/Pages/Search.pm after applying the patch and if everything is okay (searching for "männer"): [08-09-04 20:06:14.1986] Slim::Web::Pages::Search::basicSearch (78) type: contributor search: %MÄNNER% q: männer erg: 0 [08-09-04 20:06:14.2037] Slim::Web::Pages::Search::basicSearch (78) type: album search: %MÄNNER% q: männer erg: 0 [08-09-04 20:06:14.2067] Slim::Web::Pages::Search::basicSearch (78) type: genre search: %MÄNNER% q: männer erg: 0 [08-09-04 20:06:14.3501] Slim::Web::Pages::Search::basicSearch (78) type: track search: %MÄNNER% q: männer erg: 3 without any changes (no restart, no configuration changes) a few hours later (erg is 0 for all queries): [08-09-05 19:15:41.0735] Slim::Web::Pages::Search::basicSearch (78) type: contributor search: %MÄNNER% q: männer erg: 0 [08-09-05 19:15:41.0782] Slim::Web::Pages::Search::basicSearch (78) type: album search: %MÄNNER% q: männer erg: 0 [08-09-05 19:15:41.0810] Slim::Web::Pages::Search::basicSearch (78) type: genre search: %MÄNNER% q: männer erg: 0 [08-09-05 19:15:41.2398] Slim::Web::Pages::Search::basicSearch (78) type: track search: %MÄNNER% q: männer erg: 0 result: although nothing obvious has changed Slim::Schema doesn't return any results anymore - with the exactly same input parameters! any ideas? next time i'll strace the communication between slimserver.pl and mysql. unfortunately i had to restart SC so the error is gone again. kind regards, Markus
I've strace'd the communication (SELECT COUNT...) between slimserver.pl and mysqld. This confirmed although in both cases the query is identical ("Ä" is utf8-encoded to "c3 84") i receive different results. As written above i had previously verified that the mysql-db itself is still working ok - using the external mysql-client - so this lead me to a dead end (sort of). after some thinking i came up with a new assumption: mysqld does - as many servers/daemons - some houskeeping with it's worker/child processes. what happens if the tcp-connection (normally to localhost:9092) is closed, e.g. after a long idle time (one night?). could even be the operating system itself... I've now killed all mysqld processes (NOT slimserver.pl) and restarted the DB with the same parameter, locales, environment and user account. As expected DBI/DBD does the reconnect and SC continues to work without problems, BUT: Searching with non-ASCII characters fails! It seems i've found an easy way to reproduce the error...someone else with an external mysqld (not the bundled one) could confirm this behaviour? I'd suspect something odd within the DBD module concerning reconnects. Maybe it's time to update DBD (from 3.x to 4.x)? I tried, but failed due to various dependencies problems. Certainly someone else with more perl/dbi/dbd knowledge is needed here... over and out, Markus
Andy - did we plan to update DBD::mysql for 7.3? Or any other idea what might be going wrong here? (see comment #15 and comment #16)
(In reply to comment #16) > after some thinking i came up with a new assumption: > > mysqld does - as many servers/daemons - some houskeeping with it's worker/child > processes. what happens if the tcp-connection (normally to localhost:9092) is > closed, e.g. after a long idle time (one night?). could even be the operating > system itself... > My long-time test confirms this. I've repeatedly called the following command: date >> /tmp/mysql-session ; netstat -an | grep 9092 >> /tmp/mysql-session and without touching SC at all, many hours later the tcp-source port changed indicating the tcp connection to mysql has been reestablished. the result is as expected: searching for files with non-ASCII characters fails. kind regards, Markus
(In reply to comment #16) > It seems i've found an easy way to reproduce the error...someone else with an > external mysqld (not the bundled one) could confirm this behaviour? Because of other reasons i've switched to an external mysqld (5.0.60) a few weeks ago and can confirm my guess. After restarting mysql searching vor non-ASCII characters fails immediately (it's now a 5 seconds test). Everything else works fine.
Michael, did you need more from me on this bug?
To be honest - I have no idea what exactly is going wrong here. See my comment #17. It's unlikely I'll be able to fix this in 7.3. Andy - any idea?
No target without assignees
Andy to make a patch to try setting the ASCII string on reconnect.
Might be related to SET NAMES UTF8 not being run on reconnect.
Hey Andy! you hit the bull's eye :-) I hacked up the following patch to basic search: Index: Slim/Web/Pages/Search.pm =================================================================== --- Slim/Web/Pages/Search.pm (revision 23955) +++ Slim/Web/Pages/Search.pm (working copy) @@ -37,6 +37,9 @@ my $player = $params->{'player'}; my $query = $params->{'query'}; + # POC-hack for bug 8637 + Slim::Schema->storage->dbh->do('SET NAMES UTF8;'); + # set some defaults for the template $params->{'browse_list'} = " "; $params->{'numresults'} = -1; and can confirm that the problem's gone! I can restart mysql many times and after each reconnect the search returns successful. Leaves the question where to put this command in a no hackish way... I'll now run a long time test (need to hold off myself doing few svn-updates ;-) to confirm my original findings...
Created attachment 4292 [details] Use on_connect_do Can you try this patch? It should cause SET NAMES UTF8 to be run on connect and reconnect.
This patch works - and is a clean solution! good job...
Good. This would mean breaking support for 4.0 MySQL versions. KDF: There is a comment in here about you running a really old MySQL version, is that still the case? I don't feel too bad about not supporting it anymore.
mysql 4 is long gone, thanks.
Hold on... MySQL 4.0 or MySQL 4.x would be incompatible? I'm running a 4.1 here...
4.1 is fine, it's < 4.1 that don't support SET NAMES UTF8.
Andy to commit his patch.
Fixed in change 24046.
long time test confirms this bug is fixed (for me)...thanks!
This bug has been fixed in the 7.3.0 release version of SqueezeCenter! Please download the new version from http://www.slimdevices.com/su_downloads.html if you haven't already. If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.
Reduce number of active targets for SC