After hours of troubleshooting with 7.6.1, I finaly downgraded to 7.5.6 and iTunes integration works perfectly again, even with non-ASCII and non-ISO characters. So definitely the problems seems to be related to version 7.6.x and the switch to SQLite.

Please fix it, since scanning times under 7.5.6 are very long.

*** This bug has been confirmed by popular vote. ***

Could you please give us some information about your server, in particular about its locale configuration?

My server is a Sheevaplug running Debian from Squeezeplug V3.50 (www.squeezeplug.de).

Default locale is set to en_US.UTF8

SBS info screen reads: Debian-EN-utf8

iTunes library (xml) file and media folder are on external usb hard disk, ext3 formatted, mounted as /share

library file: /share/Media/iTunes1/iTunes Music Library.xml
media folder: /share/Media/iTunes1/iTunes%20Media/

Just checked my Squeezeplug, Debian and Perl versions:

-> Linux SqueezePlug 2.6.37 #3 PREEMPT Sat Jan 22 22:39:36 MST 2011 armv5tel
-> Debian 6.0.2
-> Perl v5.10.1 built for arm-linux-gnueabi-thread-multi

Hope this helps
Klaus

Problem occurs on Synology NAS DS210j, firmware 3.2, official Synology SBS package 7.6.1.
Was exactly the same with 7.5.1.

Same Problem: Files with german characters not found while scanning iTunes Library.xml.

Synology Disk Station 710+, Mac with OSX 10.6 an iTunes, Music Library and iTunes Library.xml via AFP on Synology.

Two nights with testing and frustration: Squeezebox 7.5.4 and now Version: 7.6.1 - r33110. 
The Folder Scanner find these Files but the iTunes Scanner has a simple encoding BUG!

Klaus: I think we're on the wrong track here, with UTF-8 vs. ISO-8859-1, etc. Looking at your initial report, the file that fails has an "M:/Media" path when it should be /share. The log line right before it has the correct /share path, so something is going wrong there.

I was unable to reproduce this, although I tested with an XML file from my Mac. I will do another test with Windows drive letters.

 == Auto-comment from SVN commit #33433 to the slim repo by agrundman ==
 == http://svn.slimdevices.com/slim?view=revision&revision=33433 ==

Bug 17530, log failed iTunes filenames using Data::Dump

Andy: Thanks for looking into this.

I realised the wrong path substitution "M:/Media". It seems that the scanner falls back to M:/Media since it can't find the file with special characters under the /share path.

There is a comment in the importer.pm program which descibes that rule:

# Bug 3402 
# If the file can't be found using itunes_library_music_path,
# we want to fall back to the real file path from the XML file

Klaus: Try clearing your setting for "iTunes Media Folder". This value is not needed when running on Linux.

OK. I'll try that when I'm back home on Friday.

OK I was wrong, "iTunes Media Folder" is needed because it it used to search/replace the media path in the URLs.

And I was finally able to reproduce this!

 == Auto-comment from SVN commit #33449 to the slim repo by agrundman ==
 == http://svn.slimdevices.com/slim?view=revision&revision=33449 ==

Fixed bug 17530, UTF-8 paths with the Perl UTF8 flag enabled don't work when passed to stat(), so we need to remove the UTF8 flag in pathFromFileURL

I fixed this in 7.6.2 by the way, so you can test it without possibly running into any 7.7 issues as well.

Thanks Andy!
7.6.2 works for me. All songs with international characters are found.
Scanning times (approx. 5000 titles) are 4 times faster than with 7.5.6.
Klaus

I have replaced this Misc.pm (from http://svn.slimdevices.com/slim?view=revision&revision=33449) in   the today's Synology Package (7.6.1) but unfortunately it does not help and a 7.6.2 Update from Synology will take months...

This bug seems to be back in LMS 7.7.1~33681.
I upgraded today and the iTunes Scanner does not find Unicode paths.
Went back to 7.6.2 and everything works fine.
Andy, can you please check it again.
Klaus

Same situation here. Was fixed, is broken again at least since LMS 7.7.1 - r33641

This bug appeared once more with the latest version of LMS 7.7.1 -33747-i386-readynas.bin : (

For your information, all my playlist with songs, artist or album who have international character like é,è,à,ù, etc. doesn't appear.

I have a ReadyNas Ultra 2 Plus. It think it was OK before this nightly beta release.

Upgraded to 7.7.2. Did a "delete and rescan" right after the upgrade. Tracks with international characters appeared in the library as they should. Hurray!

BUT: I did a reboot of my server (ubuntu) afterwards and now it's broken again. Did a "delete and rescan". And every other options available. Tracks with international characters are missing again.

SO: Itunes importer is broken for non-english users for almost a year!!!!!

Yup, upgraded to 7.7.2 from 7.7.0, this issue resurfaced. Anyone care to comment about this?

This seems to affect my Synology but not my Mac OS X, both scanning the same music folder and the xml file.

I see a failure in Importer.pm file in the following code block:

if (Slim::Music::Info::isFileURL($url)) {
    if ( !$file || !-r $file ) {
        // I followed the execution to here.
        # Use Data::Dump to log exactly what the wrong file path is, avoiding UTF-8 output issues
          require Data::Dump;
          $log->warn("File not found: " . Data::Dump::dump($file));
    }
}

I will spend more time tracking this. Does anyone have any hints or tips for me?

Thanks,

Baskaran

BTW, I am running LMS 7.9.

Baskaran

(In reply to comment #24)
> This seems to affect my Synology but not my Mac OS X, both scanning the same
> music folder and the xml file.
> 
> I see a failure in Importer.pm file in the following code block:
> 
> if (Slim::Music::Info::isFileURL($url)) {
>     if ( !$file || !-r $file ) {
>         // I followed the execution to here.
>         # Use Data::Dump to log exactly what the wrong file path is,
> avoiding UTF-8 output issues
>           require Data::Dump;
>           $log->warn("File not found: " . Data::Dump::dump($file));
>     }
> }
> 
> I will spend more time tracking this. Does anyone have any hints or tips for
> me?
> 
> Thanks,
> 
> Baskaran

I have identified two possible work-arounds so far:

Work-around 1:

Change code at Import.pm:246 from:

	my $url = $class->normalize_location($location);

to:

	my $url = $class->normalize_location($location, 'fallback');

Before the change I see this logged:

[15-07-18 17:40:18.0859] Slim::Plugin::iTunes::Common::normalize_location (366) Normalized file:///Volumes/music/iTunes/iTunes%20Media/Music/Anu%CC%81na/Anu%CC%81na/01%20Media%20Vita.m4a to file:///volume1/music/iTunes/iTunes%20Media/Musicffile:///volume1/music/iTunes/iTunes%20Media/Musicifile:///volume1/music/iTunes/iTunes%20Media/Musiclfile:///volume1/music/iTunes/iTunes%20Media/Musicefile:///volume1/music/iTunes/iTunes%20Media/Music:file:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/MusicVfile:///volume1/music/iTunes/iTunes%20Media/Musicofile:///volume1/music/iTunes/iTunes%20Media/Musiclfile:///volume1/music/iTunes/iTunes%20Media/Musicufile:///volume1/music/iTunes/iTunes%20Media/Musicmfile:///volume1/music/iTunes/iTunes%20Media/Musicefile:///volume1/music/iTunes/iTunes%20Media/Musicsfile:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/Musicmfile:///volume1/music/iTunes/iTunes%20Media/Musicufile:///volume1/music/iTunes/iTunes%20Media/Musicsfile:///volume1/music/iTunes/iTunes%20Media/Musicifile:///volume1/music/iTunes/iTunes%20Media/Musiccfile:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/Musicifile:///volume1/music/iTunes/iTunes%20Media/MusicTfile:///volume1/music/iTunes/iTunes%20Media/Musicufile:///volume1/music/iTunes/iTunes%20Media/Musicnfile:///volume1/music/iTunes/iTunes%20Media/Musicefile:///volume1/music/iTunes/iTunes%20Media/Musicsfile:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/Musicifile:///volume1/music/iTunes/iTunes%20Media/MusicTfile:///volume1/music/iTunes/iTunes%20Media/Musicufile:///volume1/music/iTunes/iTunes%20Media/Musicnfile:///volume1/music/iTunes/iTunes%20Media/Musicefile:///volume1/music/iTunes/iTunes%20Media/Musicsfile:///volume1/music/iTunes/iTunes%20Media/Music%file:///volume1/music/iTunes/iTunes%20Media/Music2file:///volume1/music/iTunes/iTunes%20Media/Music0file:///volume1/music/iTunes/iTunes%20Media/MusicMfile:///volume1/music/iTunes/iTunes%20Media/Musicefile:///volume1/music/iTunes/iTunes%20Media/Musicdfile:///volume1/music/iTunes/iTunes%20Media/Musicifile:///volume1/music/iTunes/iTunes%20Media/Musicafile:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/MusicMfile:///volume1/music/iTunes/iTunes%20Media/Musicufile:///volume1/music/iTunes/iTunes%20Media/Musicsfile:///volume1/music/iTunes/iTunes%20Media/Musicifile:///volume1/music/iTunes/iTunes%20Media/Musiccfile:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/MusicAfile:///volume1/music/iTunes/iTunes%20Media/Musicnfile:///volume1/music/iTunes/iTunes%20Media/Musicufile:///volume1/music/iTunes/iTunes%20Media/Music%file:///volume1/music/iTunes/iTunes%20Media/MusicCfile:///volume1/music/iTunes/iTunes%20Media/MusicCfile:///volume1/music/iTunes/iTunes%20Media/Music%file:///volume1/music/iTunes/iTunes%20Media/Music8file:///volume1/music/iTunes/iTunes%20Media/Music1file:///volume1/music/iTunes/iTunes%20Media/Musicnfile:///volume1/music/iTunes/iTunes%20Media/Musicafile:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/MusicAfile:///volume1/music/iTunes/iTunes%20Media/Musicnfile:///volume1/music/iTunes/iTunes%20Media/Musicufile:///volume1/music/iTunes/iTunes%20Media/Music%file:///volume1/music/iTunes/iTunes%20Media/MusicCfile:///volume1/music/iTunes/iTunes%20Media/MusicCfile:///volume1/music/iTunes/iTunes%20Media/Music%file:///volume1/music/iTunes/iTunes%20Media/Music8file:///volume1/music/iTunes/iTunes%20Media/Music1file:///volume1/music/iTunes/iTunes%20Media/Musicnfile:///volume1/music/iTunes/iTunes%20Media/Musicafile:///volume1/music/iTunes/iTunes%20Media/Music/file:///volume1/music/iTunes/iTunes%20Media/Music0file:///volume1/music/iTunes/iTunes%20Media/Music1file:///volume1/music/iTunes/iTunes%20Media/Music%file:///volume1/music/iTunes/iTunes%20Media/Music2file:///volume1/music/iTunes/iTunes%20Media/Music0file:///volume1/music/iTunes/iTunes%20Media/MusicMfile:///volume1/music/iTunes/iTunes%20Media/Musicefile:///volume1/music/iTunes/iTunes%20Media/Musicdfile:///volume1/music/iTunes/iTunes%20Media/Musicifile:///volume1/music/iTunes/iTunes%20Media/Musicafile:///volume1/music/iTunes/iTunes%20Media/Music%file:///volume1/music/iTunes/iTunes%20Media/Music2file:///volume1/music/iTunes/iTunes%20Media/Music0file:///volume1/music/iTunes/iTunes%20Media/MusicVfile:///volume1/music/iTunes/iTunes%20Media/Musicifile:///volume1/music/iTunes/iTunes%20Media/Musictfile:///volume1/music/iTunes/iTunes%20Media/Musicafile:///volume1/music/iTunes/iTunes%20Media/Music.file:///volume1/music/iTunes/iTunes%20Media/Musicmfile:///volume1/music/iTunes/iTunes%20Media/Music4file:///volume1/music/iTunes/iTunes%20Media/Musicafile:///volume1/music/iTunes/iTunes%20Media/Music

After the change I see this logged:

[15-07-18 17:46:34.3806] Slim::Plugin::iTunes::Common::normalize_location (366) Normalized file:///Volumes/music/iTunes/iTunes%20Media/Music/Anu%CC%81na/Anu%CC%81na/01%20Media%20Vita.m4a to file:///Volumes/music/iTunes/iTunes%20Media/Music/Anu%CC%81na/Anu%CC%81na/01%20Media%20Vita.m4a

The first log looks very strange!

The following check fails with the strange value for $file:

			if (!-e $file && -e Slim::Utils::Unicode::recomposeUnicode( $file ))
			
In addition the path "/Volumes/music/iTunes/iTunes Media/Music/Anúna/Anúna/01 Media Vita.m4a" needs to be recomposed in order to get a valid file path.

If the above work-around is risky, here is another work-around.

Work-around 2:

Here is the diff:

diff Importer.pm Importer_orig.pm 
274a275,282
> 		# Bug 5339
> 		# iTunes uses decomposed utf-8 (on a MAC), which corresponds to the decomposed UTF-8 file path on an HFS+ volume.
> 		# if the file is moved to a different path on an NFS or SMB file system, the path is automagically
> 		# converted into composed utf-8.
> 		if (!-e $file && -e Slim::Utils::Unicode::recomposeUnicode( $file )) {
> 			$file = Slim::Utils::Unicode::recomposeUnicode( $file );
> 		}
> 
280c288
< 		if (!-e $file && $prefs->get('music_path')) {
---
> 		elsif (!-e $file && $prefs->get('music_path')) {
285,292d292
< 		# Bug 5339
< 		# iTunes uses decomposed utf-8 (on a MAC), which corresponds to the decomposed UTF-8 file path on an HFS+ volume.
< 		# if the file is moved to a different path on an NFS or SMB file system, the path is automagically
< 		# converted into composed utf-8.
< 		if (!-e $file && -e Slim::Utils::Unicode::recomposeUnicode( $file )) {
< 			$file = Slim::Utils::Unicode::recomposeUnicode( $file );
< 		}
< 

With either work-arounds, the value of $file at Import:pm:299 is: /Volumes/music/iTunes/iTunes Media/Music/Anúna/Anúna/01 Media Vita.m4a	

Without one of these work-arounds, the value of $file at Import:pm:299 is: /Volumes/music/iTunes/iTunes Media/Music/Anúna/Anúna/01 Media Vita.m4a, which is invalid.

It looks like the current implementation is sensitive to order of key-value pairs in the iTunes Library.xml. It expects to find "Music Folder" key at the beginning. iTunes 12.x updates moved that key to the bottom of the plist.

I believe the root-cause of problem with importing iTunes playlists with tracks that contain UTF-8 is the failure to get the "Music Folder" path from "iTunes Library.xml" file before invoking normalize_location() function.

The XML parsing code that retrieves the value of "Music Folder" key is sensitive to where the key is located. With iTunes 12.0 this key has moved to bottom of the file. So the parser code gets to it only after parsing all the playlists and that is too late.

I have experimented with a fix that gets this key before parsing the xml file for playlists. The following diff to Importer.pm takes should work regardless of the order of the keys in the xml file.

diff Importer.pm Importer_orig.pm 
16d15
< use XML::Simple;
118,148d116
< sub getMusicFolder {
< 	my $class = shift;
< 	my $file  = shift || $class->findMusicLibraryFile;
< 	my $type = 'Music Folder';
< 	my $myfolder = undef;
< 
< 	open(XML, $file) or do {
< 		print("Couldn't open [$file]: [$@]");
< 	};
< 
< 	while(<XML>) {
< 		if (/<key>$type/) {
< 			$myfolder = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><dict>" .$_."</dict>";
< 		}
< 	}
< 
< 	close(XML);
< 
< 	if ($myfolder) {
< 		my $parser = XML::Simple->new;
< 		my $data = $parser->XMLin($myfolder);
< 		
< 		undef $myfolder;
< 		$myfolder = ${$data}{'string'};
< 		$class->iTunesLibraryBasePath( $class->strip_automounter($myfolder) );
< 		if ( main::INFOLOG && $log->is_info ) {
< 			$log->info("Found the music folder: ", $class->iTunesLibraryBasePath);
< 		}
< 	}
< }
< 
204,205d171
< 	$class->getMusicFolder;
< 	
353d318
< 		#$file  = Slim::Utils::Misc::pathFromFileURL($url);
570c535
< 		#$class->iTunesLibraryBasePath( $class->strip_automounter($value) );
---
> 		$class->iTunesLibraryBasePath( $class->strip_automounter($value) );
572,574c537,539
< 		#if ( main::INFOLOG && $log->is_info ) {
< 		#	$log->info("Found the music folder: ", $class->iTunesLibraryBasePath);
< 		#}
---
> 		if ( main::INFOLOG && $log->is_info ) {
> 			$log->info("Found the music folder: ", $class->iTunesLibraryBasePath);
> 		}

Created attachment 7746 [details]
Fixes problems in scanning iTunes playlist

Please see my last comment in this bug for more details.