Steven, can you look into this one

Michael notes this works for him.  Is there some additional complexity here?  Is the music library on a network drive?

No added complexity.  I'm using WinXP, with a standard drive letter.  Files with ASCII names work, foreign characters in path/filename don't.

What exact MIP version are you running? I realised this seems to only be the case when doing "search for new and changed" type of scan. On a full wipe/rescan, import is fine. Can you confirm this?

This imho is due to a bug I've reported to MIP more than 2yrs ago:
http://forums.musicip.com/index.php?showtopic=2194

MIP 1.8 still fails to mix tracks based on a filename with umlauts etc. in it. Can you create mixes based on such a track using MIP's web UI (http://localhost:10002)?

I am running the latest beta version of MIP: 1.9 beta 6 (build date: 2008-07-18).

I have done a full rescan and examined the tracks table:

SELECT * FROM tracks t where musicmagic_mixable is null and url like 'file:%'

It hasn't found 578 track rows; the majority of which have "file:///" urls containing accented characters, such as:

"file:///M:/Music/Phil%27s%20Music/Pop/Jos%E9%20Gonzal%E9s/In%20Our%20Nature/01%20-%20How%20Low.mp3"

Moonbase posted a patch in the ripping/tagging forum that works under windows.  I confirmed that this worked (only in Windows) when patched to SC 7.3/trunk SVN 24296.

i.e. with the patch, the scanner determines that the correct tracks are mixable by MIP, and after scanning I was able to produce a mix from a track that I could not previously mix (the one in the original bug report description).

Needs testing under different OS's.

Created attachment 4429 [details]
Patch that works for Windows

Yes, needs testing on Linux (LC UTF-8, LC ISO, atc.) and MacOS.
Since I was checking through Importer.pm (and Plugin.pm) anyway, I’ve also fixed the issues with Playlist and Mood names.

See bugs 10314, 10315, 10316.
(Can anyone tell me how to make links here?)

Maybe we can put all this together.

I guess it might also be wise to test against MIP versions 1.6 .. 1.9beta6, since many are still using those. (I tested against 1.8 and 1.8.1b, both headless & GUI API, all Windows.)

Moonbase - thanks again for your various patches. I'm working on integrating them right now.

Quick Q. regarding https://bugs-archive.lyrion.org/attachment.cgi?id=4429 - isn't there more than we actually need to do? I've been successful using the following one line change:

-		$pathEnc = Slim::Utils::Misc::escape($path);
+		$pathEnc = $isWin ? URI::Escape::uri_escape($path) : Slim::Utils::Misc::escape($path);

(the full patch will be a bit bigger, but that's basically the change)

No need to mess with utf8 stuff

*** Bug 10314 has been marked as a duplicate of this bug. ***

change 24304 - I've checked in a bunch of changes for MIP import/mix/mood/playlist handling related to non-latin stuff. Please give it a try. Thanks!

Generally, the patch has worked well.

I reverted the patch, and synced up.  I then did a full wipe/rescan.  Most songs were detected as mixable - big improvement on the previous release.

However, some characters are not working - apostrophe (’) and elipses (…) are not matching.

eg. I have a file with a proper apostrophe in it: "06 - Don’t Hate Me.flac"

SC holds the url as: "file:///M:/Music/Live%20Concerts/Porcupine%20Tree/2005-08-10/06%20-%20Don%92t%20Hate%20Me.flac"

This file is mixable in MusicIP, but the url doesn't match.

Similarly, "05 - Je T'aime… Moi Non Plus.mp3"

I haven't got many, and I can change to use "'" and "..." instead, but I thought I'd mention that it's not fully working anyway.

can you mix these songs manually using MIP service interface (http://localhost:10002/api)? You can get the path from the extended song list (http://localhost:10002/api/songs?extended). How are those characters displayed in the browser's URL field after you've created a mix based on them?

I renamed a file using the above apostrophe - and MIP fails using its own interface. I copy/pasted the file's path from it's own extended query result to the input field. The playlist button returned an "API error", and the status returned 0. I guess this is a problem in MIP. Using a straight apostrophe (') does work fine though ("S'Blauä vom Himmel").

Would still like to know whether it's working fine for you or not. What exact MIP version are you running?

I've been in touch with Wendell, and he told me that the next version of MIP will have some more character set handling on their side.

http://localhost:10002/api/mix?song=M:/Music/Live%20Concerts/Porcupine%20Tree/2005-08-10/06%20-%20Don%92t%20Hate%20Me.flac

finds the song in MusicIP, and returns a valid mix:

M:\Music\Live Concerts\Porcupine Tree\2005-08-10\06 - Don’t Hate Me.flac
M:\Music\Phil's Music\Pop\Sarabeth Tucek\Sarabeth Tucek\02 - Stillborn.mp3
M:\Music\Phil's Music\Electronic\Code Indigo\Chill\06 - Back With The Weather (Calm Front).mp3
M:\Music\Phil's Music\Rock\Sheryl Crow\Unreleased Album\04 - Hundreds Of Tears.flac
M:\Music\Phil's Music\Electronic\Djivan Gasparyan & Michael Brook\Black Rock\05 - Freedom.mp3
M:\Music\Phil's Music\Folk Rock\Kathryn Williams\Over Fly Over\10 - Untilt The Dark.mp3
M:\Music\Phil's Music\Electronic\Paul Nagle\Red Book - Blue Book\02 - 01 - Power Haus.mp3
M:\Music\Live Concerts\Radiohead\2006-05-09\18 - There There.flac


The results show both curly apostrophes ("Phil's"), and straight apostrophes ("Don’t").

Michael: Thanks for integrating that stuff. I hope that I’ll be able to check the latest nightly maybe tomorrow or the day after … so much to do here, sorry I couldn’t be faster.

Also, I’ll give this apostrophe problem a tryout. I *reckon* it’s MIP’s fault again since they do some odd conversions internally. (Phil: My MIP 1.8.1b is really converting »Guns N' Roses« to »Guns 'n' Roses«, tags were okay! So I assume maybe the apostrophe thing has its origin there too … they just *might* convert all kinds of accents to the »'«, who knows?)

Thanks for all your work and all talking and feedback going on!

Quick Feedback re apostrophe (still using 7.3-24282/Win with my patches applied):

1. Mp3tag 2.42 won’t even allow the apostrophe »’« to be saved in ID3v2.3/ISO-8859-1 (which I normally use). It changes the apostrophe proper to a »'«. (Which is correct so far, since ISO-8859-1 doesn’t allow for characters in the range 0x80..0x9f. So no apostrophe, endash, emdash, ellipsis, dagger, etc. Windows CP-1252 is NOT ISO-8859-1, but a superset.)

2. Entering, saving and putting it in the filename works when using ID3v2.3/UTF-16.

3. Using this file as mix seed manually works fine:
http://localhost:10002/api/mix?song=D:/Temp/Testdaten/MusicIP/Porcupine%20Tree%20-%20Don%92t%20Hate%20Me.mp3
returns (abbrev. list):
D:\Temp\Testdaten\MusicIP\Porcupine Tree - Don’t Hate Me.mp3
M:\MP3\Tagged\Pentangle, The\Pentangle, The - The Pentangle\Pentangle, The - Hear My Call.mp3
M:\MP3\Music\O\OMD\OMD - Secret.mp3
M:\MP3\Tagged\Kunze, Heinz Rudolf\Kunze, Heinz Rudolf - Wunderkinder\Kunze, Heinz Rudolf - Finden Sie Mabel.mp3
…

4. Almost immediately after validating this song in MMM, SC started a rescan new. Intelligence or coincidence? *g

5. The song »Porcupine Tree - Don’t Hate Me.mp3« appeared mixable in SC after a rescan new.
Log excerpt:
[08-12-18 08:03:31.1469] Slim::Plugin::MusicMagic::Importer::exportSongs (292) trackurl: file:///D:/Temp/Testdaten/MusicIP/Porcupine%20Tree%20-%20Don%92t%20Hate%20Me.mp3
[08-12-18 08:03:31.1776] Slim::Plugin::MusicMagic::Importer::exportSongs (314) track: D:\Temp\Testdaten\MusicIP\Porcupine Tree - Don’t Hate Me.mp3 is mixable

(The »track« message in the log actually contains character code 0092 instead of the block.)

[OFFTOPIC]
When trying to do mixes, I often get empty mix lists (like on the first try now):
[08-12-18 08:24:11.8201] Slim::Plugin::MusicMagic::Plugin::getMix (790) Creating mix for: artist using: Porcupine Tree as seed.
[08-12-18 08:24:11.8568] Slim::Plugin::MusicMagic::Plugin::getMix (806) Request http://localhost:10002/api/mix?artist%3DPorcupine%20Tree&variety=0&rejectsize=30&mixgenre=0&style=20&sizetype=min&size=60
[08-12-18 08:24:17.1884] Slim::Plugin::MusicMagic::Plugin::getMix (827) Warning: Couldn't get mix: artist%3DPorcupine%20Tree&variety=0&rejectsize=30&mixgenre=0&style=20&sizetype=min&size=60
[08-12-18 08:24:17.1905] Slim::Plugin::MusicMagic::Plugin::getMix (828) 500 read timeout
Content-Type: text/plain
Client-Date: Thu, 18 Dec 2008 07:24:17 GMT
Client-Warning: Internal response

500 read timeout

We might have to increase some timeouts here. Please.
[/OFFTOPIC]

OOPS. The apostrophe 0x92 gets converted to Unicode 0x2019! Which *would* be correct as far as Unicode goes, but …
Maybe we’re actually better off with giving only
$pathEnc = $isWin ? URI::Escape::uri_escape($path) : Slim::Utils::Misc::escape($path);
a try, Michael. (?!)

[08-12-18 08:27:41.4810] Slim::Plugin::MusicMagic::Plugin::getMix (790) Creating mix for: song using: D:\Temp\Testdaten\MusicIP\Porcupine Tree - Don’t Hate Me.mp3 as seed.
[08-12-18 08:27:41.4844] Slim::Networking::IO::Select::select (271) Error: Select task failed: Can't escape \x{2019}, try uri_escape_utf8() instead at C:/PROGRA~1/SQUEEZ~1/server/Slim/Plugin/MusicMagic/Plugin.pm line 804

When this happens, the Web GUI gets unresponsive (reloading forever), so you have to start at http://localhost:9000/ again.

Nevertheless, I checked stuff the whole night, gotta get a nap now.
Hopefully, this helps, and we stay in touch …

No, Michael, it’s the other way round:

(My) Plugin.pm:
803 # url encode the request, but not the argstring
804 $mixArgs = $isWin ? URI::Escape::uri_escape($mixArgs) : Slim::Utils::Misc::escape($mixArgs);

So maybe instead we DO need to do the UTF-8 stuff …

Laters…

MIP isn't converting apostrophes for me - I really have those curly apostrophes in my tags.  MIP is displaying what is in my tags correctly (eg "Don’t Hate Me").

The SC MySQL database holds the correct character too.  If I display the track name using "select CONVERT(title USING UTF8) from tracks", I see the apostrophe ("Don’t Hate Me").

In SC Web UI, it also displays correctly (i.e. not a single quote ' symbol ).

The problem is just in the encoding of the url, and matching the url with MIP.

My "Don’t Hate Me" example I've been using is a FLAC file.

I usually store mp3id3v2.3 tags using UTF-16 when using Mp3Tag.  Don't know what the default is (what do other taggers write?).

Interestingly, I checked in MySQL DB, and the track is musicmagic_mixable=NULL.  

I navigated to the album (the album is mixable, because there are other songs on the album that are mixable), and I get a MusicIP Mix icon next to each song, including the one that is not mixable.

It successfully returns a mix.

The WebUI displaying a mix icon incorrectly.  I guess it is assuming all tracks on a mixable album are mixable songs.

Okay, no re-conversion for windows paths. Got it working, please verify for MacOS and Linux! Attaching complete Plugin.pm diff (unified, 3 lines context) against SC 7.3-24282.

Actually simple. (My) Plugin.pm line 794:
- $id = Slim::Utils::Unicode::utf8decode_locale($id);
+ $id = $isWin ? $id : Slim::Utils::Unicode::utf8decode_locale($id);

Quick-tested against above-said Porcupine file with Artist Mix, Album Mix, Song Mix. Please someone test against other files using a) NO,  b) accented/foreign characters in the range 0x80..0x9f, c) foreign characters in the range 0xa0..$fe in the path! — Thanks.

Now REALLY going to sleep.

Created attachment 4469 [details]
Updated MusicMagic Plugin.pm diff against SC 7.3-24282

Includes fix for non-functioning song/album mixes with files that contain characters in range 0x80..0x9f on Windows.

Phil: Could you check again with my new patch? Also what the SQL table says?

I created a one-track »Test Bug 9772« album for the test, have no other Porcupine artists, and artist, album and track mix work now.

ID3v2.3/UTF-16 is nice to have, only I can’t currently use it since other (broadcasting) software down the road doesn’t understand it. *sigh* Not to talk about ID3v2.4 … almost NO one understands THAT (yet). That’s why I’m (unfortunately) stuck with ID3v2.3/ISO-8859-1.

I think we may have a double problem here (it’s NEVER just one, is it?): I think you might be running a newer MusicIP version that (hopefully) has fixed the »artist renaming« problem. Also, the miscellaneous headless versions differ, even if showing the same version (they always forget to change the version resource, I complained about that already).

Thirdly, I’m using MMM’s built-in API from the GUI, since the 1.8.1b headless was too buggy to be used here. The headless versions unfortunately DIFFER from what the GUI versions do. *sigh again*

So we might want to compare version stuff here and probably give Wendell some feedback.

I think with the above new patch (or, what Michael makes of it, I’m quite hopeful ;-) we should be "clean" regarding the Windows end. Crash-testing NEEDS to be done on all platforms, though.

It’s already too many places in the code where OS-specific decisions and odd conversions are made. In the long run, I guess all this should be cleaned up, towards a more general encapsulated OS or filesystem layer, so that the internal workings only use one character set and the code can be cleaned up.

(I wonder if I’ll get some sleep still … oh what the heck! It’s fun.)

Created attachment 4470 [details]
Server Log after applying new patch, mixing MusicIP

Here’s the server log after doing some mixes.

Please someone check the »wide character« warnings in Screen.pm — it might not yet be ENTIRELY clean.

Thanks.

If watching the attached server log in a browser, please set the browser’s character encoding to ISO-8859-1 in order to see what’s REALLY meant. This site will assume UTF-8 encoding which would display some (?) instead.

Moonbase - thanks again for your efforts. Getting closer here! I'll check them out (though they probably have to wait for the new year - sorry for the delay)

About the timeout: the problem here is we're using synchronous HTTP lookups in this case. Blocking SC for more than 5 seconds might lead to hiccup on some systems. Rewriting the plugin to use async code would be the best solution, but requires quite a bit of work. What hardware is your system running on and how large is your collection? We assumed MIP would only be run on pretty decent hardware, capable of returning within 5 seconds ;-)

You could always increase the value in _syncHTTPRequest()

Thanks for the hint at _syncHTTPRequest()!

Timeout I must investigate, seems maybe a MusicIP problem (was actually a crash test: a non-binary mood file with 23.058 files in the .m3u): MMM GUI handles this in roughly 1-2 minutes whereas the API (even if used with a browser) doesn’t return anything even after 55 minutes.

Btw, I understand the need of not letting the user wait at his remote—it’s a consumer product, after all. So it must be USABLE, mainly. Maybe your way is really the better one. Alternatively, one might fill a buffer in the background using a longer timeout, then somehow come back with a list to be used later. Sounds more complicated and is probably not worth the effort, web GUI users could benefit from it. Maybe I’ll to increase t/o a *little* since with "only" 1 GB RAM the system tends to get hogged pretty fast using SC, which in turn leads to some sluggishness on both SC and the MIP API due to swapping. This is why I’ve seen "standard" MIP mixes time out, not only this wild test case.

Still there seems to be a little problem somewhere when this happens: MIP API (from the GUI app) won’t return any more mixes to SC, and when exiting the MIP app, it briefly shows as many error messages as "bad" mixes you have requested. Those are not there before, so the error message is still hidden from me. Quitting and restarting MIP fast enough lets one continue doing mixes.

Should I have some time left, I’ll investigate this further, maybe even use the standalone "headless" version again for testing.

Testing unit here:
1.8 GHz P4, 1 GB RAM, 120 GB internal (partitions 40/70/10), 500 GiB external USB2 (finally!), Win/XP+SP2 Home German, Firefox 3.0.5. Currently still SC 7.3-24282 (will test current nightly next days), MusicIP Mixer 1.8.1b (reg'd; using its API since headless too buggy).
Collection some 23,700 songs, mostly MP3 still, newer albums FLAC. MusicIP states 23,717 songs, 198 genres, 4,121 artists, 3,537 albums. SC states 6168 albums with 23720 songs by 4139 artists (probably due to about 10-15% of the files not yet being "finally" tagged).

Hopefully in January, I’ll get another machine for testing. Can’t break too much here, since it’s real-life data and also used for broadcasting. So I won’t be able to test with one of the MusicIP 1.9 betas until then, since at least 1.9beta6 was still so buggy that it broke a lot … took me over 2 days to revert everything.

Most probably, the new machine will run kubuntu so I’ll have the chance to do more debugging/developing using the Linux software, and also testing the SC/MIP mixed Win/Linux environment. My music collection is mirrored between the Win and Linux machines already, so I could as well use the Linux mirror as "test collection" then.

Well thanks again anyone so far, especially Michael and Philip, and looking forward to more feedback on this. Don’t forget having a happy holiday season, you and your beloved ones!

How did this get in the change log as fixed? I can reproduce with a mixable track, but I'm not testing with the patch. 

> How did this get in the change log as fixed? 

I added it because I thought it was fixed. But character set issues obviously are never fixed... I'll pull it.

We all thought. A while ;-) Did you test Linux/MacOS? I’m currently testing against SC7.3.1-24367/Win.

When checking Plugin.pm and Importer.pm, I find Michael’s code much cleaner than mine. Looks like we can finally expect SC running under non-UTF8 Linux, too. One of the days. Seems using Slim::Utils::Unicode::utf8encode_locale() doesn’t break too much, have to check the code still. Good thing.

Expect (temporary) Plugin.pm diff against SC7.3.1-24367 later.

Created attachment 4476 [details]
MusicMagic Plugin.pm diff against SC7.3.1-24367

MusicMagic Plugin.pm diff against SC7.3.1-24367.
Verified working using original tests above plus Phil’s »Porcupine apostrophe« test case above, testing artist/album/song mix on (German) Windows/XP+SP2.

Slightly off-topic maybe, but I still experience empty mixes ever so often, on a pretty idle system here. Setting the timout in _syncHTTPRequest() to 10 instead of 5 seconds keeps getting me mixes EVERYTIME (approx. 50x checked now).

I’d propose to make it settable in the MusicIP plugin GUI part, and still use a default of 5. Guess this would help keeping the end user’s remote experience smooth, but still allowing for the fact that MusicIP doesn’t always return mix lists THAT fast on all machinery (NAS, slow Win, etc.).

(Should I open an extra bug for this?)

Created attachment 4504 [details]
Server Log after applying new patch to SC7.3.2-24401

Just quick info on SC7.3.2-24401: (Non-)Functionality unchanged.

Plus above patch: Creates another problem apparently, see attached server log snippet.

Phil - did you have time to test Moonbase's latest change? (see comment #22)

Thanks Moonbase, looking good with Phil's apostrophs as well as with Björk and the other usual suspects. 

Change 24528 - don't utf8decode file name on Windows

Sure, I'll test.  I'm back to work now and busy again, but will try to get round to testing in the next day or so.

Is there an urgency for this to go in a patch release?  The bug is targetted for 7.3, not 7.3.?, so I'm not sure what that means this latest fix will go into.

Is it committed to 7.3/trunc (7.3.2), or 7.4.  I can't run 7.4 at the moment due to mySQL instance issue.

> Is it committed to 7.3/trunc (7.3.2), or 7.4.

It's in 7.3.2

> I can't run 7.4 at the moment due to mySQL instance issue.

Give it another try. I hopefully fixed this issue yesterday (if it was the "7.4 doesn't use my own MySQL instance on Windows" issue, aka bug. 10430)

Cheers, Michael. I'm currently on SC7.3.2-24523/Win, will check again with latest nightly tomorrow.

Btw, not [i]one[/i] "Empty Mix" anymore since changing $http->timeout(5); to 10s in _syncHTTPRequest ... ;-)

My test track now shows:

[09-01-08 07:10:47.4564] Slim::Plugin::MusicMagic::Importer::setMixable (352) Couldn't get track for file:///D:/Temp/Testdaten/MusicIP/Porcupine%20Tree%20-%20Don%E2

with SC7.3.2-24541/Windows after clear & rescan. Track not mixable.
Will investigate further tomorrow.

Error Located: The previously existing Windows patch to Slim/Plugin/MusicMagic/Importer.pm is gone. (Attachments: "Patch that works for Windows".)

Could you check please?

> Error Located: The previously existing Windows patch to
> Slim/Plugin/MusicMagic/Importer.pm is gone.

Well, that's what your latest patch did: disable utf8decode in Windows.

Well the latest change you made to Plugin.pm looks sane (and seems to work), I only wonder why the "old" Importer.pm problem crept up again. Maybe I didn’t test hard enough with special cases because I was so happy that you made the changes.

On a first glance, I see that you didn’t patch up Importer.pm as suggested but instead made a (more logical) change in Common.pm to be able to re-use code. To me, this makes sense, and it seems to work on the "usual suspects" (i.e., Latin-1/ISO-8859-1 encoded filenames) BUT NOT on the special "Phil Test case" (which contains the non-ISO-8859-1 apostrophe in the filename "Porcupine Tree - Don’t Hate Me.mp3").

The %E2 (instead of a %92) in the error message and the truncation of the filename let me suspect that Importer.pm/Common.pm still has a fault in decoding to what we would like to see:
[09-01-08 07:10:47.4564] Slim::Plugin::MusicMagic::Importer::setMixable (352) Couldn't get track for file:///D:/Temp/Testdaten/MusicIP/Porcupine%20Tree%20-%20Don%E2

The filename SHOULD be
file:///D:/Temp/Testdaten/MusicIP/Porcupine%20Tree%20-%20Don%92t%20Hate%20Me.mp3
as earlier scanner logs show.

So what I would guess at a first glance is that the "guess" Perl makes on the encoding of the string is wrong, or somehow doesn’t work with the current logic. Remember my comment #25 where Screen.pm complains about a "wide character". I would assume that *maybe* Perl "guesses" ISO-encoding while the encoding is actually CP-1252, or the like. See also my comment #17 where I explained a bit about characters in the range 0x80..0x9f (non-ISO use in Windows Character Pages).

Maybe we need to "help Perl along" by somehow stating the current Windows CP instead of letting it determine the encoding? Perl should know about current locale and charset, shouldn’t it? (Just in case someone uses, say, a Russian codepage instead of CP-1252, we need to consider that.)

We might need to re-compare what the differences between my original patch and your Common.pm change were (since it worked okay using my clumsy patch over two releases, until you integrated the changes).

Again, I’d consider the filename, playlists, moods bugs fixed FOR ISO-8859-1 FILENAMES in Windows BUT NOT for filenames that have characters in the range 0x80..0x9f. Character Encoding Wars, indeed ...

Created attachment 4576 [details]
Test Suite: Pseudo-Album with 6 tracks, includes all Windows filenaming variants

I’ve prepared a "test suite" (pseudo-album with 6 MP3 tracks) that (hopefully) includes all oddities that can happen with Windows file names, both on Unicode- and non-Unicode file systems. I think this is the best way for all of us to have the same test data and check on various setups.

The archive is a WinRAR file that contains Windows Unicode filenames, so please use an appropriate extractor that can handle these, otherwise test files #5 and #6 won’t extract! Please check the included README.txt file (UTF-8+BOM, CR+LF line endings).

For legal reasons, the audio data in the files is just 30s of pink noise each.

In a working SC, the test suite should display as one album »SqueezeCenter Bug 9772« by »Alfred E. Moonbase«, with 6 tracks.

I’ll now go and check these example files myself against SC7.3.2-24541/Windows on a XP+SP2 machine.

Thanks for all your help!

Moonbase - tracks 5-8 don't mix. Agreed? I'm getting MIP internal errors:

[09-01-09 15:41:27.7198] Slim::Plugin::MusicMagic::Plugin::getMix (776) Creating mix for: album using: D:\Private\mherge
r\My Documents\My Music\9772\MO1CE1~1.MP3 as seed.
[09-01-09 15:41:27.7211] Slim::Plugin::MusicMagic::Plugin::getMix (789) Request http://localhost:10002/api/mix?album%3DD
%3A%5CPrivate%5Cmherger%5CMy%20Documents%5CMy%20Music%5C9772%5CMO1CE1~1.MP3&variety=0&rejectsize=12&mixgenre=0&style=20&
sizetype=tracks&size=12
[09-01-09 15:41:27.7335] Slim::Plugin::MusicMagic::Plugin::getMix (810) Warning: Couldn't get mix: album%3DD%3A%5CPrivat
e%5Cmherger%5CMy%20Documents%5CMy%20Music%5C9772%5CMO1CE1~1.MP3&variety=0&rejectsize=12&mixgenre=0&style=20&sizetype=tra
cks&size=12
[09-01-09 15:41:27.7355] Slim::Plugin::MusicMagic::Plugin::getMix (811) HTTP/1.0 500 Internal Server Error
Client-Date: Fri, 09 Jan 2009 14:41:27 GMT
Client-Peer: 127.0.0.1:10002
Client-Response-Num: 1

MusicIP API error - invalid request or internal error.

Created attachment 4591 [details]
scanner.log snippets: Test Suite - Scan for new and changed

Here’s the result (scanner.log snippets) of the automatic "Look for new and changed music" scan SC did after I copied the 6-track test suite into the folder structure.

Result: Test files #1..#4 were seen "mixable", #5..#6 "not mixable".

Created attachment 4592 [details]
scanner.log snippets: Test Suite - Manual Clear library and rescan everything

Here’s the results (relevant scanner.log snippets) of doing a full wipe and rescan, the files from the 6-track test suite included.

Result: Files #1..#2 seen as "mixable", #3..#6 "not mixable" (and actually no found).

The interesting part is here:
[09-01-08 22:16:56.2808] Slim::Plugin::MusicMagic::Importer::setMixable (352) Couldn't get track for file:///D:/Temp/Testdaten/MusicIP/Moonbase,%20Alfred%20E.%20-%20Test%20%233_%20Non-ISO%20_%20Philip%E2
[09-01-08 22:16:56.3076] Slim::Plugin::MusicMagic::Importer::setMixable (352) Couldn't get track for file:///D:/Temp/Testdaten/MusicIP/Moonbase,%20Alfred%20E.%20-%20Test%20%234_%20Non-ISO%20_%20Dashes%20%E2
[09-01-08 22:16:56.3264] Slim::Plugin::MusicMagic::Importer::setMixable (352) Couldn't get track for file:///D:/Temp/Testdaten/MusicIP/Moonbase,%20Alfred%20E.%20-%20Test%20%235_%20Russian%20_%20%D0%A0%D1
[09-01-08 22:16:56.3488] Slim::Plugin::MusicMagic::Importer::setMixable (352) Couldn't get track for file:///D:/Temp/Testdaten/MusicIP/Moonbase,%20Alfred%20E.%20-%20Test%20%236_%20Thai%20_%20%E0%B9
...
[09-01-08 22:20:24.5071] Slim::Plugin::MusicMagic::Importer::setMixable (352) Couldn't get track for file:///D:/Temp/Testdaten/MusicIP/Porcupine%20Tree%20-%20Don%E2

where it truncates the filenames and thus cannot see the tracks.

In SC7.3.2-24541, ALL 6 tracks were (correctly) shown as an album having the correct characters. The WebUI, SoftSqueeze, SqueezePlay and Winamp were able to show the titles correctly, though SqueezePlay (2009-01-07 version) displayed the Thai characters as "blocks" (a font issue, no major problem).

They *appeared* playable (I should have put numbers into the audio instead of just 30s of pink noise, to better distinguish them).

So I reckon our changes work in principle, except that we still have to find the issue with the "Couldn't get track for ... (truncated filename)".

Also suspicious is this:

[09-01-08 22:38:55.3959] Slim::Formats::readTags (161) File missing: D:\Temp\Testdaten\MusicIP\Moonbase, Alfred E. - Test #3_ Non-ISO _ Philips Apostrophe.mp3
[09-01-08 22:38:55.4176] Slim::Formats::readTags (161) File missing: D:\Temp\Testdaten\MusicIP\Moonbase, Alfred E. - Test #4_ Non-ISO _ Dashes  .mp3
[09-01-08 22:38:55.4411] Slim::Formats::readTags (161) File missing: D:\Temp\Testdaten\MusicIP\Moonbase, Alfred E. - Test #5_ Russian _ .mp3
[09-01-08 22:38:55.4648] Slim::Formats::readTags (161) File missing: D:\Temp\Testdaten\MusicIP\Moonbase, Alfred E. - Test #6_ Thai _ .mp3
[09-01-08 22:45:44.9141] Slim::Formats::readTags (161) File missing: D:\Temp\Testdaten\MusicIP\Porcupine Tree - Dont Hate Me.mp3

(I have set Logging to "Debug" for "plugin.musicip", "scan.import" and "scan.scanner".)

> So I reckon our changes work in principle, except that we still have to find
> the issue with the "Couldn't get track for ... (truncated filename)".

Perl has serious issues converting utf8 to cp125x/unicode filenames. According to what I've read so far on the internet is simply isn't possible. In these cases I've started falling back to using the short name (Win32::GetShortFileName()). We might need to do this here as well (when reading the tags). But this still won't fix the mixing issue, where the file name sent to MIP to create a mix seems to be in a format MIP doesn't like.

Created attachment 4600 [details]
New Test Suite for Windows Unicode filenames - 10 tracks

New Test Suite with 10 tracks, contains more examples from "rami69", "romanr", and "Moonbase":

Moonbase, Alfred E. - Test #01_ All ASCII _ Only 0x20-0x7f.mp3
Moonbase, Alfred E. - Test #02_ All ISO-8859-1 _ ÄÖÜäöüßÉéØøÆæÅå.mp3
Moonbase, Alfred E. - Test #03_ Non-ISO _ Philip’s Apostrophe.mp3
Moonbase, Alfred E. - Test #04_ Non-ISO _ Dashes – —.mp3
Moonbase, Alfred E. - Test #05_ Russian _ Русский.mp3
Moonbase, Alfred E. - Test #06_ Thai _ ไทย.mp3
Moonbase, Alfred E. - Test #07_ Hebrew _ עברית.mp3
Moonbase, Alfred E. - Test #08_ HebRus _ עברית _ Русский.mp3
Moonbase, Alfred E. - Test #09_ Japanese (Nihongo) _ 日本語.mp3
Moonbase, Alfred E. - Test #10_ Chinese (Zhōngwén) _ 中文.mp3

See also forum thread "Unicode filenames" at
http://forums.slimdevices.com/showthread.php?t=57900

I tried a full rescan with 7.4.  I still have problems with backtick symbols in urls.

The file "01 - 02 - Relativity [Transeau’s Excursion] - Brian Transeau.mp3" is mixable in MusicIP.

SC does not find it in MusicIP.  SC holds the track url as:

file:///M:/Music/Phil%27s%20Music/Electronica/BT/10%20Years%20In%20The%20Life/01%20-%2002%20-%20Relativity%20[Transeau%92s%20Excursion]%20-%20Brian%20Transeau.mp3

Michael, sorry I missed Comment #45 — Agreed, tracks 5-8 don't mix (5-10 with new test suite).

All these are the ones trying to mix using the short filename: Errors (811) followed by (810).

Couldn't we use CPAN's Win32::GetLongPathName() to convert the short name back for MIP API purposes? Only I couldn't find it in what SC/Win installs in the server/CPAN folder.

Would need to be re-encoded properly for Win, of course.

(Sometimes I hate having to do all these kludges for Windows *sigh*)

Ok, Win32::GetLongPathName() would work in theory (it's in the Core anyway, I missed that), BUT there is no encoding I can find that works with the MusicIP HTTP API (using GUI v1.8.1b), entering the request manually. (For the filenames containing Windows Unicode characters, i.e. test case #08.)

We might have to talk to Wendell. Seems to be a headless problem on their end, I'd think. I actually opened a new thread in MusicIP's forum:
http://forums.musicip.com/index.php?showtopic=4085&hl=

Created attachment 4620 [details]
New Test Suite for Windows Unicode filenames - 12 tracks

Updated Windows Unicode Filenames Test Suite to include test cases #11 and #12 for reverse-checking against broken short/long filename logic (these should display as MOONBA~1.MP3 and MOONBS~B.MP3 and NOT be converted). Can also be used to check for uppercase file extension handling.

I also put the new version on MY server, just in case (and for linking):
http://www.kaufen-ist-toll.de/download/radio/Moonbase%20-%20Windows%20Unicode%20Filenames%20Test%20Suite.rar

Users of earlier versions of the test suite: Please replace all files, not just the two new ones!

(Test suite requires an unpacker that handles Windows Unicode filenames correctly, i.e. WinRAR.)

Non-mixability of test cases #5..#10 filed as "case 1494" with MusicIP.

We might have to wait for some fix until this can be re-verified.

Philip, is it possible for you to verify against MusicIP version 1.9, latest beta?

Fixed - Closed Message (SC)

This bug has been fixed in the 7.3.3 release version of SqueezeCenter!

Please download the new version from http://www.slimdevices.com/su_downloads.html if you haven't already.  

If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.

Correction: SqueezeCenter version is 7.3.2

Reduce number of active targets for SC