Bugzilla – Bug 5067
Offline storage causes database corruption
Last modified: 2008-06-17 13:44:49 UTC
We've been experiencing bizarre corruptions in our library where, over time, more and more albums would appear twice or more in the library, with a reduced number of tracks showing. I've tracked down the problem, and it relates to the availability or otherwise of NAS storage. OUR SETUP ********* We have a standard setup, the only thing which is slightly unusual is that all our music files are stored on a NAS rather than on the machine running the slimserver. The only other unusual thing (but becoming more common these days) is that we power off all our electronics kit at night and when out, to save on bills and to save the planet (!) SCENARIOS ********* There are two scenarios where corruption repeatably occurs: 1) Typically, in the morning, we'll just listen to the radio through the squeezebox, so there's no need to have the NAS on. However, if, the previous evening, we were listening to an album on that squeezebox, then we notice that the album appears when you press the power button on the remote, though it won't play anything because our NAS is turned off. The album that was being played gets corrupted (I'll explain the nature of this in a moment) 2) If we choose and try to play an album, but we forgot to switch the NAS on, then nothing plays as you'd expect. We switch the NAS on, and boom... it's screwed. THE CORRUPTION ************** There are two effects. 1) Additional album entries are created for an album that gets screwed. 2) Tracks go missing. The best example I have is what happened this morning. Previous evening: we were listening to a 27 track album titled: 'Handel: Dixit Dominus'. It was playing track 19 (we think), when we put the machine running the slimserver into hibernation because we were going out. We did not switch off the squeezebox playing the album first. Next morning: We turned on the machine running the slimserver (but NOT the NAS). We pressed the power button on the squeezebox remote to turn it on. Saw the album from last night but chose internet radio instead and listened to the radio all morning. Fine. Later that day, we noticed the following: 1) The original album 'Handel:Dixit Dominus' was still there, but the tracks 19 to 27 were missing from the slimserver database. 2) A new album had appeared, titled 'Händel DIXIT DOMINUS' (in capitals). This has one track in it - track 19. Choosing to 'add new or missing tracks' doesn't fix things, but rebuilding the libary from scratch does. ------- Note that the NAS hasn't lost any files. If we rebuild the library from scratch, then all is fine again. WHY THIS IS BAD *************** NAS is increasingly common in the home, and there are real energy savings to be had by turning off equipment when not using it. Offline storage shouldn't cause a library screw up like this. If the slimserver cannot find a file, then it shouldn't try to change the library entries (or at the very least, there should be an option to stop it changing the libary entries if some files are currently offline). Even if the slimserver *is* trying to keep in sync with the files, it's failing to do so, as it adds multiple albums and removes tracks but does not reinstate them. Hope this helps. Email me if you need more info.
I think this behavior was in response to bug 2315 and bug 3262. Slimserver noticed the tracks were gone, and removed them as best it could from its database. There may be some additional minor bug in the fact that it apparently created the new album you notice. If we undertake this, we will need to consider how to indicate to the user that the content is not currently mounted. Perhaps displaying an appropriate message on the player for a few seconds, and trying the next track in the playlist might be good enough?
(In reply to comment #1) > I think this behavior was in response to bug 2315 and bug 3262. Slimserver > noticed the tracks were gone, and removed them as best it could from its > database. There may be some additional minor bug in the fact that it > apparently created the new album you notice. > If we undertake this, we will need to consider how to indicate to the user that > the content is not currently mounted. Perhaps displaying an appropriate > message on the player for a few seconds, and trying the next track in the > playlist might be good enough? Yes, that would be fine on the squeezebox... but I think it's very important to be able to tell the server whether to expect media to be offline or not. I guess there might be some people who will want the server to remove from the database tracks which have been deleted from the filesystem manually, and others - like us - who want the server to leave the database intact even if it can't access the music files for any reason (i.e. we'd only want the database changed when we explicitly do a rescan or a 'look for new and changed music). This is the only problem we've found with the software - we think it's great btw.
Perhaps the right thing is to just skip missing tracks, display an error message, and not delete them from the DB. I'm always looking for ways to not include more options we need to test before a release. :)
I really wish SlimServer wouldn't try to be so smart, too often with screwy results. First, it should gracefully handle deleted, moved, renamed and offline files and folders. If a file referenced by the library database isn't available, then just skip over it and either display or log a warning message. Don't attempt to remove it from the database. This is where we get odd results. This would apply to Browse Music Folder as well. Add content, but don't remove anything. Require an explicit scan run by the user before removing anything from the library (it would be nice if a new & changed rescan were finally made robust enough to correctly add and remove content, so that a clear/rescan is seldom necessary). If the root music folder is unreachable when a user attempts to run a scan, don't even begin the scan, and do not clear the database if doing a clear/rescan. Instead, display a warning message in big bold letters. (One scenario this won't protect against is if a user runs a scan and there is storage that is offline and linked to from either the main music folder or a subfolder.) A more complete solution would be something in conjunction with a feature that people have been requesting forever. Permit the designation of multiple music folders. And for each of those folders have a property (set by a checkbox in the web ui) for whether the folder can be expected to be offline. Then, when a new & changed rescan is run, if that particular music folder is unavailable, none of the files within the folder tree should be removed from the library database.
(In reply to comment #4) > I really wish SlimServer wouldn't try to be so smart, too often with screwy > results. First, it should gracefully handle deleted, moved, renamed and > offline files and folders. If a file referenced by the library database isn't > available, then just skip over it and either display or log a warning message. > Don't attempt to remove it from the database. This is where we get odd > results. This would apply to Browse Music Folder as well. Add content, but > don't remove anything. That sounds good to me. I'm all for smart thing as long as a) They don't actually fight against you some of the time b) They work properly! > Require an explicit scan run by the user before removing anything from the > library (it would be nice if a new & changed rescan were finally made robust > enough to correctly add and remove content, so that a clear/rescan is seldom > necessary). Yes that sounds like the best thing - you could change the web UI so that when people press the 'go' button for a rescan, the UI comes up in big letters with "Make sure all your music storage is online, then press 'start'" or something like that. If the root music folder is unreachable when a user attempts to > run a scan, don't even begin the scan, and do not clear the database if doing a > clear/rescan. Instead, display a warning message in big bold letters. Sounds good > A more complete solution would be something in conjunction with a feature that > people have been requesting forever. Permit the designation of multiple music > folders. And for each of those folders have a property (set by a checkbox in > the web ui) for whether the folder can be expected to be offline. Then, when a > new & changed rescan is run, if that particular music folder is unavailable, > none of the files within the folder tree should be removed from the library > database. That all sounds good!
*** Bug 5252 has been marked as a duplicate of this bug. ***
We'll look at this post-7.0.
*** This bug has been marked as a duplicate of bug 4402 ***