Bug 15405 - TinySBS should gracefully recover from "malformed db image" problem
: TinySBS should gracefully recover from "malformed db image" problem
Status: CLOSED FIXED
Product: SB Touch
Classification: Unclassified
Component: TinySC
: unspecified
: Macintosh Debian Linux
: P1 major (vote)
: 7.5.0
Assigned To: Ben Klaas
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-01-04 02:56 UTC by Michael Herger
Modified: 2010-04-08 17:25 UTC (History)
4 users (show)

See Also:
Category: Bug


Attachments
try to remove corrupted image and start from scratch (889 bytes, patch)
2010-01-04 10:34 UTC, Michael Herger
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Herger 2010-01-04 02:56:24 UTC
Sometimes TinySC wouldn't start any more due to the DB image file being corrupted (after unplugging without unmounting etc.):

Jan  4 11:46:46 slimserver.pl: [10-01-04 11:46:46.7283] main::init (341) Starting Squeezebox Server (v7.5.0, r29714, Sat Jan  2 02:10:50 PST 2010) perl 5.010000
Jan  4 11:46:56 slimserver.pl: [10-01-04 11:46:56.0789] Slim::Control::Request::execute (1923) Request in error, returning
Jan  4 11:46:59 slimserver.pl: [10-01-04 11:46:59.0590] Slim::Utils::Scanner::Local::rescan (129) Discovering files in /media/sda1
Jan  4 11:47:00 slimserver.pl: [10-01-04 11:47:00.8527] Slim::Schema::Storage::throw_exception (82) Error: DBI Exception: DBD::SQLite::db do failed: database disk image is malformed [for Statement "DELETE FRO
M scanned_files WHERE url LIKE 'file:///med
Jan  4 11:47:00 slimserver.pl: [10-01-04 11:47:00.8680] Slim::Schema::Storage::throw_exception (82) Backtrace:     frame 0: Slim::Utils::Log::logBacktrace (/usr/squeezecenter/Slim/Schema/Storage.pm line 82)  
  frame 1: Slim::Schema::Storage::throw_exc
Jan  4 11:47:00 slimserver.pl: [10-01-04 11:47:00.8825] Carp::Clan::__ANON__ (214) Warning: Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db do failed: database disk image is malformed [for Statement "D
ELETE FROM scanned_files WHERE url LIKE 'fi


(Unfortunately the messages in /var/log/messages are truncated)

In the UI there's no feedback at all. The "Stop SBS" menu item changes to "Start SBS" after a few seconds. That's it.

We should have some feature in the SBS applet to discover and recover from this situation. Eg. parse the log file for the above message if SBS is not running while a USB stick is mounted, then offer the user to start from scratch.
Comment 1 Michael Herger 2010-01-04 10:12:45 UTC
would we add this kind of trigger to SBS or SqueezeOS (SOS :-))?
Comment 2 Andy Grundman 2010-01-04 10:27:17 UTC
Should be in SBS because it's been seen on other systems too.
Comment 3 Michael Herger 2010-01-04 10:32:12 UTC
Looks like a similar job that's already in Slim::Schema::Storage. I'll try to come up with something.
Comment 4 Michael Herger 2010-01-04 10:34:59 UTC
Created attachment 6418 [details]
try to remove corrupted image and start from scratch

completely untested q'n'd hack
Comment 5 Andy Grundman 2010-01-04 10:51:00 UTC
Looks good, worth a try.
Comment 6 SVN Bot 2010-01-04 22:05:54 UTC
 == Auto-comment from SVN commit #29722 to the slim repo by michael ==
 == https://svn.slimdevices.com/slim?view=revision&revision=29722 ==

Fixed Bug: 15405
Description: if SQLite initialization fails due to corrupt db image, remove file and try to rebuild from scratch
Comment 7 Michael Herger 2010-01-14 08:55:20 UTC
TinySC can still get into a state where it wouldn't start due to some corrupt file.

Should we add a "start from scratch" item to fab4's menu?
Comment 8 SVN Bot 2010-01-14 08:57:55 UTC
 == Auto-comment from SVN commit #29808 to the slim repo by michael ==
 == https://svn.slimdevices.com/slim?view=revision&revision=29808 ==

Bug: 15405
Description: try harder to recover from corrupt db file. Check for the squeezebox-persistent.db too.
Comment 9 Ben Klaas 2010-01-18 11:33:53 UTC
Discussed this in bug meeting, and the consensus was that another item should be added under the squeezebox server menu when the server is not running

Wipe, Rescan and Start Squeezebox Server

I will add the necessary bits for a menu item and callback but we'll also need to consider strings for this.

further notes:
"wipe" will only remove the .Squeezebox directory from the drive currently configured in /etc/squeezecenter/prefs.json
Comment 10 SVN Bot 2010-01-18 14:20:55 UTC
 == Auto-comment from SVN commit #8367 to the jive repo by bklaas ==
 == https://svn.slimdevices.com/jive?view=revision&revision=8367 ==

Fixed Bug: 15405
Description: add wipe and rescan option to SBS settings menu when tinySC is not running
Comment 11 Ben Klaas 2010-01-18 14:37:10 UTC
Reopening because I had to add strings for this.
Comment 12 Michael Herger 2010-01-26 02:42:01 UTC
Ben - tried this today. And SB Touch crashed during the "rm -rf /media/sda1/.Squeezebox" phase. I don't know who's currently getting the crash reports, but there might be some information about this. A second try (the folder had not been completely removed) crashed, and monitoring processes I wonder whether it somehow run out of memory. Last glimpse I got before it crashed again memory usage without slimserver.pl running was around 105MB. Would rm use large amounts of memory?
Comment 13 Michael Herger 2010-01-26 02:53:58 UTC
As I can't imagine rm is using so much memory fab4 would crash, I rather think the watchdog triggered the reboot, as removing thousands of artwork files from the cache blocked the jive process for too long? Maybe we should make this kind of async. Mickey already mentioned the lack of visual feedback.
Comment 14 Michael Herger 2010-01-26 03:33:07 UTC
Looking at the SBS applet code it seems as if os.execute() was expected to be running in the background: a spinny should be shown while the files are being removed. But as os.execute() is blocking that popup is never shown (which explains the lack of feedback), and jive can time out. Would this small change do the job for us?

Index: src/squeezeplay_fab4/share/applets/SqueezeCenter/SqueezeCenterApplet.lua
===================================================================
--- src/squeezeplay_fab4/share/applets/SqueezeCenter/SqueezeCenterApplet.lua	(revision 8398)
+++ src/squeezeplay_fab4/share/applets/SqueezeCenter/SqueezeCenterApplet.lua	(working copy)
@@ -766,7 +766,7 @@
 	end
 
 	-- wipe the .Squeezebox directory forcefully through linux
-	local command = "rm -rf " .. scDrive .. "/.Squeezebox"
+	local command = "rm -rf " .. scDrive .. "/.Squeezebox &"
 	os.execute(command)
 
 	self.wipeTimeout = 0

Send the "rm -rf" command to the background? Seems to be working fine in my quick test.
Comment 15 Ben Klaas 2010-01-26 06:54:44 UTC
If you are backgrounding the command then it isn't blocking to the Squeezeplay UI, is it? That is, the rescan will start while the rm -rf is still in progress, because os.execute() returns after backgrounding the command?
Comment 16 SVN Bot 2010-01-26 08:06:02 UTC
 == Auto-comment from SVN commit #8400 to the jive repo by michael ==
 == https://svn.slimdevices.com/jive?view=revision&revision=8400 ==

Fixed Bug: 15405
Description: background removal of folders, the spinny will wait for the folder to be removed before it launches SBS. 

Increase the timeout for the folder removal. 15 seconds is too optimistic with larger collections on slow disks.
Comment 17 Chris Owens 2010-04-08 17:25:40 UTC
This bug has been marked fixed in a released version of Squeezebox Server or the accompanying firmware or mysqueezebox.com release.

If you are still seeing this issue, please let us know!