Bugzilla – Bug 5637
MusicIP takes hours to complete
Last modified: 2009-07-31 10:14:52 UTC
Once Scanning is complete, it will scan all over again, taking hours to complete the musicIP import. Dir Structure: Music, with 3 sub-directories -Audio(not music), Media, and Playlists (empty). Music library: approx. 17000 tracks MusicIP mixer enabled (analysis archived in the tracks). Approx 85% of the tracks are analysed/mixable. The music scan completes almost instantly, as there is no new music. Afterwards, the MusicIP import takes hours to complete, and seems to run from start.
BTW: SqueezeCentre from 30 Sept 2007 Cheers
Works for me. I have only about 12k tracks and it takes about 45 minutes to do a complete scan. I do not use MusicIP itself, so the data is not updated often. If you are updating MusicIP data, this will trigger rescans. Look in SqueezeCenter settings->MusicIP to adjust the value. 0 disables the rescan interval, which means you'd have to do a manual rescan if you change anything on the MusicIP side (this may include track play counts, etc). shortcuts/links from one part of the library to another can often cause problems with folder scan, but I've never heard of this with just the MusicIP scan, so not sure what else to suggest. plugin.musicip and scan.import set to DEBUG under SC Settings->debugging will show the process more clearly in the logs.
Created attachment 2206 [details] scanner log file
Thanks for the tip in setting the refresh. It was set at 3600, and I have increased it to 86400. Still, the scanning process seems to be in a loop. I do not use shortcuts in the music directories. I have added the log file as attachement. I Cheers Matt
So it takes hours, but it does eventually complete?
It does complete. I am watching it closely, and it seems that the scan has stopped after completing for a while. I will monitor the behaviour, and report back. Also, as a daily routine, I will install the latest version of SqueezeCenter. Cheers Matt
How many tracks are in your library, Matt?
Around 17000. I have placed MusicIP on a daily rescan (only about 15000 tracks there), and I am comfortable with the outcome (the rescan happens at night, so I am not confronted with it during operation of the server). Cheers Matt
I'll change the bug description to reflect that it does eventually finish, and reduce the severity slightly.
I just realised that we may be talking about "new and changed" scan only? I think there is a basic problem here: MusicIP import is always a full import. Perhaps what we should do on a 'new and changed' is simply check isMusicLibraryFileChanged and skip the import if nothing has changed. If there is a change, we are probably still stuck with a full import as there is no way to currently tell which musicip song info has changed.
So perhaps the bug is that 'new and changed' scan doesn't know about MusicIP. Steven to run a benchmark on this to see how slow it is, and be able to verify any improvements.
Hi! Well.. if MusicIP automatically saves the validation info to the mp3 file (there is an option in MIP to do that if I recall it right), then why can't we have a third scanning option in SC that make it possible to scan just a specific folder. Then if one adds one album, one can just scan that and not the whole database (which for me with 54 000 tracks takes ~2 hours). That would go so much faster and then one would get the tags, the covers and the MIP info very rapidly Zevs
When I add a new album and do a rescan for new and changed files, it takes ~30 minutes. Music IP takes the vast majority of this time (close to 29 minutes) - about 95% of the total time. An example: Directory Scan (250 of 250) Complete 00:01:33 MusicIP Import (19981 of 19981) Complete 00:28:47 Playlist Scan (10 of 10) Complete 00:00:05 Merge Various Artists (1730 of 1730) Complete 00:00:42 Artwork Scan (17 of 17) Complete 00:00:01 Database Cleanup #1 (20783 of 20783) Complete 00:00:34 Database Cleanup #2 (0 of ) Complete 00:00:25 Database Optimize (0 of ) Complete 00:00:11 SqueezeCenter has finished scanning your music collection.00:32:18 The MusicIP plugin fetches a list of all songs from MusicIP, and then tries to match each song to a song in the SqueezeCenter library, such that it can set the is_mixable property. If the song isn't in the SqueezeCenter library, then it creates it. For my use case, the scanning phase only needs to determine for each new or changed song whether the song is mixable in MusicIP. If it is, then the artist, album and genre must also be mixable. I'm not bothered personally about MusicIP being a source for adding songs into SqueezeCenter library - my songs are always scanned into SC library before musicIP scanning is started. In fact I wouldn't want that to happen - MusicIP is a feature provider, not an alternative source of music. The MusicIP api allows a song url to be queried to see if it is mixable or not, so when the SC scanner finds new/changed songs, it could determine mixable status only for each of those new/changed songs.
I have created a patch to the MusicIP plugin to provide an option to turn off MusicIP as a source for song tag information, such that it will only determine the mixable status for songs, rather than fetch and process all song data from MusicIP each time a scan is performed. With this patch, for new/changed files the MusicIP import is now much faster: Directory Scan (596 of 596) Complete 00:02:00 MusicIP Import (596 of 596) Complete 00:00:17 Playlist Scan (10 of 10) Complete 00:00:06 Merge Various Artists (1732 of 1732) Complete 00:00:44 Artwork Scan (7 of 7) Complete 00:00:00 Database Cleanup #1 (20813 of 20813) Complete 00:00:29 Database Cleanup #2 ( of ) Complete 00:00:24 Database Optimize ( of ) Complete 00:00:11 SqueezeCenter has finished scanning your music collection. 00:04:11 17 seconds to process 596 changed songs in MusicIP. A full rescan using my new processing method of only looking for mixable status is three times faster: MusicIP Import (20760 of 20760) Complete 00:10:00
Created attachment 3713 [details] Patch for MusicIP plugin for 7.2/trunk branch
If MusicIP analysis is performed after a SqueezeCenter scan, SC will not know that the songs are mixable. If file timestamps are updated after the analysis is complete (eg. using archive analysis within MusicIP without preserving timestamps option ticked), the next SC scan for new/changed files will determine the mixable status. Alternatively, a full rescan will always determine the mixable status of all songs. Another idea I had is for a new/changed rescan to check MusicIP mixable status for all songs that were previously not mixable, even if they have not changed in SC. Dan Sully suggested an alternative/additional scanning mechanism: "I think the best way to go about this is still pulling down the mmm-data / extended, and then scanning through that, turning it into a hash of filenames to MiP active files. This could even be persistent via DB_File. That way, even for a wipe, if the user is not using MiP as their primary database (setting option?), it can just be an O(1) lookup, and there should be a callback available for the main music folder importer to call and set the setSongMixable()."
Do any of the others on the cc list for this bug have an opinion about Philip's patch? I'm concerned that MM users might be confused or irritated that we're asking them to choose between fast scanning and rigorously checked tags. Or maybe all SC & MM users really only care about mixing and don't care about MM's metadata?
I'll answer from my point of view: I don't depend upon MusicIP to correct my tags - I correct them with Jaikoz and MusicBrainz prior to loading them in MIP. I currently have SC setup to only pull music from MusicIP to prevent duplicates and make scanning faster - if there is a better way then I would certainly change it. I run MusicIP headless on my server and don't use the features on MIP except for the mixing (from SC's UIs). I add music to MIP by using the web interface to the server service. Currently, I notice that SC always starts rescanning the Music in from MIP prior to me even starting the verify process - let alone finishing it (I believe that the verify process is the one that does this, please correct me if I'm wrong). If SC didn't report the newly added albums as mixable that wouldn't work very well and I'd have to do a rescan manually? How about this: Could SC trigger the MIP Add and Validate processes from inside of SC itself and then know when they are finished so it could then start the SC portion of the rescan? That would be much more convienent in any case - only 1 place to go. Even nicer if there was a "Scan for New Music" button somewhere on the main web UI screen... In the Music Library group as the last entry would certainly make sense. Hopefully this gives you some information to make a decision on - I've got to think that I'm not the only use case, however.
>I'll answer from my point of view: I don't depend upon MusicIP to correct my >tags - I correct them with Jaikoz and MusicBrainz prior to loading them in MIP. > Tag content of files is not the issue. When the tag info is corrected with an editor such as MusicBrainz, they will be written back to the files, so either SqueezeCenter (SC) or MusicIP (MIP) will read the corrected content. MIP song tag data is interpreted differently to SC. For example, a song with no album in SC appears under an album called "No Album", whilst in MusicIP it appears under an album called "Miscellaneous". If you have songs by different artists without an album tag, in SC you'd get one "No Album" album, whereas in MIP you'll have one "Miscellaneous" album per music folder. So, if you have SC configured to point at your music files, and also import music from MIP, there are several conflicts going on, as well as taking twice as long because it scans all songs twice. >I currently have SC setup to only pull music from MusicIP to prevent >duplicates and make scanning faster. > In my opinion, it's better to only retrieve tag data from the SC scanner, and not from MIP. This is faster, and allow more control over the information. >Currently, I notice that SC always starts rescanning the Music in from MIP >prior to me even starting the verify process > When the MIP cache changes, SC can recognise there has been a change and start scanning the information. MIP will be updating the cache constantly as it is verifying new songs, so SC will end up rescanning a lot whilst this is happening. >If SC didn't report the newly added albums as mixable that wouldn't work very >well and I'd have to do a rescan manually? > As each file becomes verified, I guess it updates the cache, and the MusicIP plugin in SC may be configured to rescan if it detects a change. If the patch is used, and the "Read Song Data" setting is turned off, only files that SC detects during a rescan will be interrogated in MIP to see if they are mixable. What I do is load new songs in MIP and analyse/verify them. Once complete, I then to a rescan in SC for new/changed files, which picks up the mixable status. This is a lot faster. If a rescan is done before the analysis, the songs will appear in SC, but not mixable. If analysis is then done, the files will still not be mixable in SC until a full rescan is done, or if the timestamps on the files have changed, they would be detected as mixable when a scan for new/changed files is performed. >How about this: Could SC trigger the MIP Add and Validate processes from >inside of SC itself and then know when they are finished so it could then start >the SC portion of the rescan? > I don't want to do this within SC (eg. the MusicIP plugin in SC), as a normally fast rescan for new/changed music could take ages waiting for MIP analysis to complete. A better approach would be to write a separate tool that added the music in MIP, performed analysis, and then started the SC rescan. i.e. to automate what users can do manually. I don't know how feasible that is; fairly trivial, except each persons personal circumstances could change what it is required to do. Alternatively, the enhancement to my patch could be made such that it always reassess songs in SC that are not mixable to see if they are now mixable.
I would rather get all of the data from the SC scan as well - just was told not to do it that way in the past. Perhaps this can change now. I don't want MIP to change my files, either, so it sounds like I would still have some timing problems and have to do a full rescan a lot. I was thinking more along the lines of an MIP scan started from SC, not blocking SC, I guess. "Ideal" scenario: 1) SC is configured to scan the library files directly and have a link to MIP. 2) A scan is started from SC and 2 things occur - the SC scanner starts loading the files and MIP starts adding music and verifying. 3) When the MIP verify is done then SC pulls in the updated mix information. The hope is that the files would be available quickly from the SC scan and then flip over to mixable when MIP has finished its analysis. If the MIP status could be used to determine when its analysis is complete (maybe scanner.exe waits for it or something) then it could check for the mixable changes "at the end" of the SC scan. I don't think that anyone would care that the SC "scan" process takes longer if it waits for MIP to finish to get the additional data as long as the status is indicated (i.e. "Waiting for MusicIP Analysis") and the files are playable, etc. right away. I was thinking that perhaps scanner.exe was the "separate tool", I guess. Certainly might not be possible in any kind of quick timeframe - mainly trying to articulate what I would like (in the hope that it is generally useful). Getting an integrated "Add Music" concept into SC would certainly increase its ease of use for new users, etc. Right now I'm the only one who will add music because I'm the only one that knows the process, etc. Out of 6,000 files I have 5 that are not mixable. All due to duration, I believe. I think that duration is also stored in SC - you wouldn't have to check for any files that are too short all of the time. Aren't some genres (like "spoken") automatically not mixable as well? Just thinking of some ways to cut down on the thrash calling if you recheck the unmixables.
I support Philip's point of view regarding MIP as "provider and mixer". In the past i had many problems with MIP as source because of tagging problems (ID3v1/ID3v2, no album vs. miscellaneous, changing tags without updating the modification time and so on). I've tested the patch (more tests and feedback will follow) and the MIP-behaviour is so much better - in my use case at least. MIP-Scanning time is twice as fast as before, too. I know implementing the "perfect solution" is best, but this patch is at least a big improvement (for me again :-) so i'd like to see it merged back into trunk. Maybe leave the default settings as is, that's fine with me. my 2 cents, Markus
I've done some further tests with update scans (including utf-8 characters in file names) and the patch has passed each test. So no regressions with this patch and my own use cases.
That's good to know Markus. Thanks
It would be good to have updated instructions on the preferred way to access MIP mixing with this change - it sounds like it is different than before?
Creating mixes is exactly the same; all that is different is the speed of scanning (esp. a scan for new/changed files) if you are happy for MusicIP to NOT be a source for adding music into the SqueezeCenter library ("Read Song Data" setting unticked). If "Read Song Data" is ticked, everything will be exactly the same as before.
I've got to think that there may be a bit more to it than that. I've got SC's file locations set to blank so that the only source is MIP. I think that I've got to change that to the file location as well - at a minimum. So will this fix also prevent the double listing problems that were occuring with earlier? And what is the new process for scanning? Scan in SC, add in MIP, verify in MIP, then rescan in SC (to get the verified ones marked properly) or something else? Thanks!
Yes - if you want SC to be the source for your music data, then you need to set the music folder path in SC. I don't know what your duplicate songs problem was. You should add new songs in MusicIP, perform analysis on them in MusicIP, and then rescan in SC.
Andy says he'll review this patch for 7.3.
Created attachment 3877 [details] Only scan songs that are not mixable This is an alternative approach. Instead of finding only new/changed music and asking MIP if each of those songs is mixable, it now finds all songs that do not have mixable status in SC, and asks MIP if each of those songs is mixable. This avoids issues with analysing in MIP after scanning in SC.
Phil - thanks for the continued update. As I'm trying to clean up some other MIP issues as well, are these patches based on the special 7.2 branch or the public 7.2/7.3?
Based on the special 7.2 branch.
OK, just to give two more data points: I tested Phil's patch on a 7.1 distribution on Mac and on August 25's 7.2, special branch on Ubuntu. Both work fine, especially the ubuntu test was very favorable since that was on my C7 machine and takes almost an hour out of the scanning process. :-)
How much faster is it on the C7 (running a C6 myself, that's interesting to me :-))
OK, ran it again. This time with an idle server (did nothing else but scanning the library). Yesterday I just installed and kicked it off and looked at it later - I think there were quite a few other things going on on the server at that time. Now it doesn't look THAT impressive anymore but it's still taking out 17min from the scan process (20% overall) or speeding up the MIP scan by a factor of 3: Durchsuche Musik-Verzeichnis (7014 von 7014) Beendet 00:19:32 MusicIP-Import (6961 von 6961) Beendet 00:24:04 Suche Wiedergabelisten (33 von 33) Beendet 00:01:27 Gruppiere "Diverse Artisten" (640 von 640) Beendet 00:01:01 Suche Plattenhüllen (706 von 706) Beendet 00:21:03 Datenbank Optimierung (0 von ) Beendet 00:01:45 SqueezeCenter hat das Durchsuchen ihrer Musiksammlung abgeschlossen. Laufzeit: 01:08:52 Durchsuche Musik-Verzeichnis (7014 von 7014) Beendet 00:18:45 MusicIP-Import (6961 von 6961) Beendet 00:07:42 Suche Wiedergabelisten (33 von 33) Beendet 00:01:25 Gruppiere "Diverse Artisten" (638 von 638) Beendet 00:01:01 Suche Plattenhüllen (704 von 704) Beendet 00:21:28 Datenbank Optimierung (0 von ) Beendet 00:01:45 SqueezeCenter hat das Durchsuchen ihrer Musiksammlung abgeschlossen. Laufzeit: 00:52:06 This is comparing a scan for a 7.000 track library on a 1GHz, 1GB RAM, 1TB SATA HD Via C7 machine, 7.2 as of Aug 25th, Ubuntu server 7.10, Phil's patched plugin, full rescan. The first scan was with the option to use the "old" code enabled, the second one disabled.
Joerg, what performance are you getting with a scan for new/changed files? I imagine that the MusicIP scan would take a few seconds.
ONE second, to be exact. It only scans the files that are not yet mixable, doesn't it? Durchsuche Musik-Verzeichnis (0 von ) Beendet 00:00:29 MusicIP-Import (77 von 77) Beendet 00:00:01 Suche Wiedergabelisten (33 von 33) Beendet 00:01:14 Gruppiere "Diverse Artisten" (628 von 628) Beendet 00:00:36 Datenbankbereinigung 1. Durchgang (7014 von 7014) Beendet 00:00:18 Datenbankbereinigung 2. Durchgang (0 von ) Beendet 00:00:28 Datenbank Optimierung (0 von ) Beendet 00:01:08 SqueezeCenter hat das Durchsuchen ihrer Musiksammlung abgeschlossen. Laufzeit: 00:04:14 vs. the same as the full rescan with "Read Song Data" set to "ON": Durchsuche Musik-Verzeichnis ( von ) Beendet 00:00:34 MusicIP-Import (6961 von 6961) Beendet 00:21:45 Suche Wiedergabelisten (33 von 33) Beendet 00:01:27 Gruppiere "Diverse Artisten" (640 von 640) Beendet 00:01:04 Datenbankbereinigung 1. Durchgang (7014 von 7014) Beendet 00:00:29 Datenbankbereinigung 2. Durchgang (0 von ) Beendet 00:00:43 Datenbank Optimierung (0 von ) Beendet 00:01:21 SqueezeCenter hat das Durchsuchen ihrer Musiksammlung abgeschlossen. Laufzeit: 00:27:23 So yes, this is where it really makes a difference, 4 min. vs. 27...
just a quick "thumbs up" for this new patch. Applies cleanly to public 7.2 and it passed all my tests again (full-scan, update-scan, utf-8 file/tracknames). old way fullscan: 6700 tracks, Directory Scan 4,5mins, MIP-Scan 5mins new way fullscan: 6700 tracks, Directory Scan 4,5mins, MIP-Scan 1,5mins although the new way is two times faster i like the "set mixable only"-feature even more - not to mention those really fast update scans... thanks! Markus
Created attachment 3970 [details] Slightly updated patch I've made the musicip preference a 3-value pref, instead of defining the type of integration in an additional preference. It's now "no integration", "full" or "mixable info only". Diff against 7.3. Please test. Thanks Phil!
checked in at change 23145 - please test! We might want to add some more "intelligence" in eg. the wizard to set the scanning mode automatically if there's another local music source defined or not. Thanks a lot Phil!
Can a quick couple of notes be added on how to configure everything with this new setting? I think, from previous conversations, that the SC file patch should be setup now, etc. but I'm a bit unsure. Thanks!
> Can a quick couple of notes be added on how to configure everything with this > new setting? The only user visible change is that you now have one more option instead of just "use itunes - on/off". When enabling it you can choose whether SC should read all metadata from MIP (names, genres, duration... etc.), or only whether a track is mixable or not. The first option should imho only be used if you don't want to use a music folder or iTunes. > I think, from previous conversations, that the SC file patch > should be setup now, etc. but I'm a bit unsure. Thanks! What file patch?
I have compared my patch to the changes you have committed. Largely the same, but you've tidied up a few things for the better. Thanks! One thing that I noticed was the path encoding has been simplified. I had a bit of trouble with the path encoding, and had to call Slim::Utils::Unicode::utf8encode_locale($pathEnc), which has now been removed. I never really understood it fully - someone gave me some example code that I copied. Was this redundant, or is now redundant because character encoding utils has been updated recently? I'll have a play to see if I can break this, but I'm only running SC on Windows. Another thing that examining the differences has reminded me about is MIP supported file formats. @MIPSupportedFormats was declared and set to the format that MIP supports on windows. I believe Linux MIP file format support is different. May not matter if Linux has a smaller set of supported formats - if SC finds unmixable songs and makes the request it will return 0, but it will always check and never return mixable status, so it could be improved to setting the supported formats based on OS type. I haven't tested the new version yet - will have a play later tonight.
> One thing that I noticed was the path encoding has been simplified. I had a > bit of trouble with the path encoding, and had to call It's an area of dark magic. > I never really understood it fully Neither do I. > copied. Was this redundant, or is now redundant because character encoding > utils has been updated recently? I'm not sure, really. I thought it was overly complicated, plus it crashed on one of my (admittedly bad) files. (I ran it against a collection of files which had been uploaded to bugs. Thus they are expected to cause problems, but they shouldn't crash SC). We'll have to thoroughly test this. Please let me know if you encounter issues on Windows. I should be able to do more tests on Mac/Linux next week. > supported file formats. @MIPSupportedFormats was declared and set to the > format that MIP supports on windows. I believe Linux MIP file format support > is different. May not matter if Linux has a smaller set of supported formats Thanks for the heads up. I'll verify this with MIP.
I can confirm there's something odd concerning encoding/decoding non-ascii path names...will need to investigate further...
I can't test this on SC 7.3 at the moment - I can't get the default skin to load in the WebUI. I started a thread in the beta forum; no responses there yet. Look here for supported MIP file types per OS: https://secure.musicip.com/mixer/mixerfaq.jsp#1 Non-ascii path names worked in my original patch on Windows. Several people also commented that they had no problems after applying my patch.
just a quick update before someone gets on the wrong track: it seems "musicmagic_mixable" is correctly set for those special (i.e. non-ASCII) files, but i cannot create mixes with such a track as seed - don't know yet if this is related to this patch (i'm in doubt about it). At least i know this has worked a few weeks ago (including Philip's patch). system as always: linux, utf-8 :-) I'll collect some logs later.
(In reply to comment #41) > > Can a quick couple of notes be added on how to configure everything with this > > new setting? > The only user visible change is that you now have one more option instead of > just "use itunes - on/off". When enabling it you can choose whether SC should > read all metadata from MIP (names, genres, duration... etc.), or only whether a > track is mixable or not. The first option should imho only be used if you don't > want to use a music folder or iTunes. > > I think, from previous conversations, that the SC file patch > > should be setup now, etc. but I'm a bit unsure. Thanks! > What file patch? If I could only type. I meant "file path". Sorry. Basic Settings, Music Folder - to be specific. I've always had to have that be blank in the past. Now, it should be set, correct, or doesn't it matter. The reason it was blank was to avoid the double scan time and prevent duplicate entries in SC. To switch over I figure that I have to setup the music folder and then do a clear/rescan. Anything else?
after some more testing i can confirm that the updated patch works as good as Philip's one, especially regarding utf8-encoded directory/file names. Therefore almost all tracks do get the "mixable" flag. Philip, Michael thank you for the patch and merging it into SC. unfortunatey a regresssion crept in between 7.2 and 7.3 because of the switch to LWP (in the broadest sense). - it's not possible to use an utf8-encoded file/a file form an utf8 encoded directoy as seed (results in an empty playlist) - creating mixes from other songs will never contain tracks with utf8-encoded files i think the problem is caused by calling $reponse->decoded_content (Plugin.pm, line 796) whereas in 7.2 it used $http->content only. The following patch fixed the problem for me. Index: Slim/Plugin/MusicMagic/Plugin.pm =================================================================== --- Slim/Plugin/MusicMagic/Plugin.pm (revision 23162) +++ Slim/Plugin/MusicMagic/Plugin.pm (working copy) @@ -793,7 +793,7 @@ } } - my @songs = split(/\n/, $response->decoded_content); + my @songs = split(/\n/, $response->content); my $count = scalar @songs; for (my $j = 0; $j < $count; $j++) { Some more eyes are probably recommended (as almost always on such charset/encoding issues). btw. in the moods section decoded_content is used, too. But i don't have moods so i can't test it. kind regards, Markus
(In reply to comment #47) > To switch over I figure that I have to setup the music folder and then do a > clear/rescan. Anything else? short version: nothing else :-)
Created attachment 3985 [details] screenshot of scan library statistics As you can see... the results are stunning! 3 sec to scan MIP (only checking if tracks are mixable). Great work!
As a "garden variety" user I downloaded and installed Philip Meyer's revised Plugin (as per http://forums.slimdevices.com/showthread.php?t=48422), added a Music Folder entry into SC's Basic Settings as it was blank and ran the various rescans to record great improvements in scanning speeds. Philip's reply #13 sums up perfectly - "For my use case, the scanning phase only needs to determine for each new or changed song whether the song is mixable in MusicIP. If it is, then the artist, album and genre must also be mixable. I'm not bothered personally about MusicIP being a source for adding songs into SqueezeCenter library - my songs are always scanned into SC library before musicIP scanning is started. In fact I wouldn't want that to happen - MusicIP is a feature provider, not an alternative source of music." When I get a new CD, I rip it to an appropriate directory (determined by the location where the actual CD is stored), wait for MIP to complete its analysis and run a new music scan in SC. The "Look for new or changed music" scan being reduced to 3 minutes from over an hour is a great bonus. Question : Is Michael Herger's "Slightly updated patch" available as a zip file download to replace the MIP plugin in SC 7.2 ? Will it be available in SC 7.2.1 ?
The change is in SC 7.3 only. Thanks everybody for the testing!
didn't want to close, have to check path encoding issues.
Markus - what's your exact OS configuration (locale, distro etc.)? My CentOS 4.7 based system is mixing fine tracks/albums/folders with umlauts in their name.
I also applied the Sunday morning build and did some tests. The clearing and rescanning worked fine and tracks were mixable afterwards. I then added some additional tracks to the library and MIP and did a change scan - very quick!! I did have to clear out the browser cache, though, because it was displaying incorrect album artwork in the album list until I did so. I would think that that would be more related to the clear rather than anything else, though. Just including it for completness.
(In reply to comment #54) > Markus - what's your exact OS configuration (locale, distro etc.)? My CentOS > 4.7 based system is mixing fine tracks/albums/folders with umlauts in their > name. > Michael, before delivering a full set of logs and system information i had to admit i got caught in my own trap (by a misconfiguration of MIP). Sorry. I'm now able to create mixes from those tracks! - but my second point is still valid: Mixes never contain Tracks with umlauts/special characters. This should be easy to verify, because the first track of the mix is not the seed song (which usually is the case). Do you get mixes with non-ascii chars file/directory names?
> Do you get mixes with non-ascii chars file/directory names? Yes, I do. On Linux (CentOS), OSX and Windows.
Created attachment 3990 [details] 2 logs (plugin.musicip=debug) + screenshot okay, here it is: - system: Gentoo Linux 64bit, stable - SqueezeCenter Version: 7.3r23183 - TRUNK @ UNKNOWN - Linux - EN - utf8 - using --charset=utf8 (which i normal don't) doesn't make any difference $ locale LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=en_US.UTF-8 Logs (compressed to preserve each bit) server-content-decode.log (default): contains the log of a 5-track mix, but only three are displayed (screenshot-decode.png). server-content-only.log (my patch from #48, mimics the codings + behaviour from SC 7.2): contains the log of a 5-track mix, all files are displayed in the SC. Using the non modified Plugin.pm the first line from "Slim::Plugin::MusicMagic::Plugin::getMix (811) Original" looks different than Slim::Plugin::MusicMagic::Plugin::getMix (761) Creating mix for:" Both should look the same, don't they? Applying the Patch at least matches both codings to be equal - not necessarily correct (although i hoped) ... i'll do a few more tests.
I blew away my database instance today to rescan everything with a clean database. I seem to have character encoding issues with the patch. I created my database instance with the following statement: create database squeezecenter character set utf8; For all files that have accented characters, I get warnings in the log trying to retrieve MusicIP mixable status. [09:41:58.5854] Slim::Plugin::MusicMagic::Importer::setMixable (353) Couldn't get track for file:///M:/Music/Phil%27s%20Music/Progressive%20Rock/Amon%20D%C3%BC%C3%BCl%20II/Lemmingmania.mp3 The url for the song in the database is: file:///M:/Music/Phil%27s%20Music/Progressive%20Rock/Amon%20D%FC%FCl%20II/Lemmingmania.mp3 So somewhere along the lines, the url from the track is being translated: "Amon%20D%FC%FCl%20II" -> "Amon%20D%C3%BC%C3%BCl%20II". The tracks table collation is utf8_general_ci, whilst the url column collation is utf8_unicode_ci. Is that expected?
Hmmm, it appears that actually its not a collation/character encoding problem. I tried issuing a MusicIP request in a browser using: http://localhost:10002/api/status?song=M:/Music/Phil%27s%20Music/Progressive%20Rock/Amon%20D%C3%BC%C3%BCl%20II/Lemmingmania.mp3 and http://localhost:10002/api/status?song=M:/Music/Phil%27s%20Music/Progressive%20Rock/Amon%20D%FC%FCl%20II/Lemmingmania.mp3 Both fail to find the item in MusicIP. Then I pasted the actual path from windows explorer into the query, and Firefox translated this into: http://localhost:10002/api/status?song=M:\Music\Phil%27s%20Music\Progressive%20Rock\Amon%20D%FC%FCl%20II\Lemmingmania.mp3 which worked. The only difference is the path has "\" characters not "/". In fact, MusicIP appears to be really weird in that the last path divider is accepted as either "/" or "\", but not any other path divider. When I enter: http://localhost:10002/api/status?song=M:\Music\Phil%27s%20Music\Progressive%20Rock\Amon%20D%C3%BC%C3%BCl%20II\Lemmingmania.mp3 firefox first converts this path into "M:\Music\Phil's Music\Progressive Rock\Amon Düül II", and the lookup fails, but then refreshing the page converts into "http://localhost:10002/api/status?song=M:\Music\Phil%27s%20Music\Progressive%20Rock\Amon%20D%FC%FCl%20II\Lemmingmania.mp3" which then works. So, essentially I think the url in the tracks database is correct, and this needs to be passed in the request to MusicIP without any translation, other than replace "/" with "\".
> http://localhost:10002/api/status?song=M:/Music/Phil%27s%20Music/Progressive%20Rock/Amon%20D%C3%BC%C3%BCl%20II/Lemmingmania.mp3 In http forward slashes are used to separate folders. These should be encoded before being sent to MIS. Do you see these being sent as shown above?
I'm closing this bug though there seem to be some (related at all?) outstanding issues. Could you please open new, specific bugs for them? Filtering the relevant information out of these >60 comments is getting very cumbersome. Thanks for your understanding.
This bug has been fixed in the 7.3.0 release version of SqueezeCenter! Please download the new version from http://www.slimdevices.com/su_downloads.html if you haven't already. If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.
Reduce number of active targets for SC