Bugzilla – Bug 17471
TMPDIR environment variable isn't passed to external scanner when --user switch is used
Last modified: 2012-03-05 09:29:01 UTC
Several users encountered scanner failures with big collections when SBS 7.6 runs on a QNAP NAS: http://forums.slimdevices.com/showthread.php?t=88524 http://forums.slimdevices.com/showthread.php?p=650697&postcount=623 Message shown in scanner.log is like this one: main::main (346) Error: Failed when running scan post-process: [Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db selectrow_array failed: disk I/O error [for Statement (...) It appears that on QNAP systems, /tmp is a 32MB tmpfs filesystem (which /var/tmp links to), and when the I/O error occurs, it's exhausted by SQLite temporary tables and indices. To work around this problem, it's necessary to either extend the /tmp partition or to use the "High" database memory config profile (which has a "temp_store = MEMORY" pragma), but both solutions suppose the target system has plenty memory to deal with, which is not the case with most entry level NAS (a 50K+ collection required 96MB in /tmp for the scan to complete). Another (working) solution is to add the SQLite's temp_store_directory pragma to the "Normal" database memory config profile, and make it point to SBS' cache directory (for example), but this pragma is deprecated, and hence shouldn't be used. Finally, on *NIX systems, SQLite is supposed to use the directory pointed by the TMPDIR environment variable before failing over to /var/tmp (which is symlinked to /tmp on the QNAP) or /usr/tmp. And that's where something is wrong with SBS, because further investigation has shown that even if you export the TMPDIR environment variable to the slimserver.pl script, it will see it (and SQLite *will* use it), but when SBS calls the scanner.pl script, the TMPDIR environment variable isn't defined any more (although it's correctly passed to the script by the system, 9th post in the first forum thread linked shows it clearly). There must be something in the scanner's code making it undefine the exported variable, and thus preventing SQLite from using the directory pointed by the TMPDIR environment variable.
Created attachment 7403 [details] Temporary workaround Attached a temporary workaround based on the use of the deprecated temp_store_directory pragma, to force SQLite storing its temporary files in the Library Cache directory.
Sébastien - if you want to give it another try: Slim::Music::Import->launchScan() is where we start the scanner process. You might be able to pass along the environment variable's value or something. Andy - is there no way but the environment variable to configure this behaviour at execution/runtime? I could imagine this could become a major issue with many NAS devices which are running a small /tmp folder in RAM.
No problem on ReadyNAS: its /tmp is on a 2GB partition of the disk. Sébastien: one more thing: if SSOTS (or whatever you're running) is using a Slim/Utils/OS/Custom.pm, you could modify it to use your code. Doesn't make it more valid, but it's probably more likely to get some deprecated call to fix a particular issue on one particular platform into some 3rd party's file than into the main code...
SQLite can use a lot of temporary disk space, this is to avoid using actual memory. Don't know what to tell you, get a bigger /tmp disk?
(In reply to comment #4) > SQLite can use a lot of temporary disk space, this is to avoid using actual > memory. Don't know what to tell you, get a bigger /tmp disk? Are you really saying that QNAP (and probably Synogy, too) customers, average Joe users who aren't even able to troubleshoot their own computer, will have to hack their NAS units from an obscure Linux command shell so that /tmp is not mounted over tmpfs but some real disk volume? Sheesh, I'm not even sure it's possible without hacking the NAS firmware itself (for which I don't have source code)! Come on... SQLite will follow $TMPDIR, and SSOxS already sets this environment variable accordingly and pass it to slimserver.pl at launch time. The problem here, is that once forked from slimserver.pl, $TMPDIR will be masked to scanner.pl (although it's been exported by slimserver.pl itself). That's where the bug is, and that's what needs to be corrected. If you don't want to get your hands on it now or ever (which I could understand), could you at least offer an alternative option to "Low" and "High" database memory config profiles, that would set the temp_store_directory pragma pointing to the cache directory? As long as the pragma works (it does for now), it would: 1. save support calls (which won't bother you) 2. let the community some time to find the bug and offer a proper patch Thanks.
Do you know whether the QNAP and/or Synology distros are using a Slim::Utils::OS::Custom class?
Created attachment 7415 [details] Optional Custom.pm file bundled with SSODS (In reply to comment #6) > Do you know whether the QNAP and/or Synology distros are using a > Slim::Utils::OS::Custom class? There's a Custom.pm file bundled with SSOxS, but its installation is optional. I've just discovered this, so it's not installed on my QNAP unit, though. And there's nothing really fancy there in, it barely defines some specific ignored items and defines a restartServer method that will use SSOxS' own startup script. How could that help?
Same problem on the eTRAYz NAS from Xtreamer. Thats i support to fix the bug. Hopefully before official 7.61 is out. Other solution please make possible that my squeezebox not allways run firmware updates. There are times i change with my Squeezebox betwheen my Etrayz NAS what now runs SBS 7.54 and my EEE Box what runs Windows 7 7.61 after each song i change libary. Allways the Firmware of my Logtech players need a Update of the Firmware. I love it :-)
(In reply to comment #8) > Other solution please make possible that my squeezebox not allways run firmware > updates. There are times i change with my Squeezebox betwheen my Etrayz NAS > what now runs SBS 7.54 and my EEE Box what runs Windows 7 7.61 after each song > i change libary. Allways the Firmware of my Logtech players need a Update of > the Firmware. This has nothing to do with this bug. You could submit a request enhancement for this, but I already know the answer: WONTFIX. Run the same SBS version on both machines, and it'll be solved.
> There's a Custom.pm file bundled with SSOxS, but its installation is optional. > I've just discovered this, so it's not installed on my QNAP unit, though. How is it made optional? > How could that help? See my comment #3. Is SSOTS still based on SSODS, or is there a different maintainer for that file?
(In reply to comment #10) > How is it made optional? Manual installation from shell, so not very user friendly. > > How could that help? > See my comment #3. I have yet to figure out how this works. Would I be able to overload Slim::Utils::SQLiteHelper.pm's on_connect_do() from there? > Is SSOTS still based on SSODS, or is there a different maintainer for that > file? SSOTS is SSODS.
(In reply to comment #10) > > There's a Custom.pm file bundled with SSOxS, but its installation is optional. > > I've just discovered this, so it's not installed on my QNAP unit, though. > > How is it made optional? > > > How could that help? > > See my comment #3. > > Is SSOTS still based on SSODS, or is there a different maintainer for that > file? For Etrayz we have some private project and someone make a installer. Is in general the same and Flip who did make the SSODS told in january he have no time for several months. see inside the ssods thread his last posting. J-R did compile based on SSODS a version that will work. but to be compatible for Etrayz we only give final versions. If this bug is not solved inside 7.61 official all Etrayz users like me also can not update our Etrayz because noone make a package. For me this means because i use 2 servers with my SB3 allways the SB3 download a new firmware.
> Would I be able to overload Slim::Utils::SQLiteHelper.pm's on_connect_do() > from there? Checked that again, and there's indeed no obvious hook to do this. It might be possible to add it somewhere in a call that's executed before the first scan is run, but after the schema has been initialized. Eg. S::U::OS->getOS()->scanner(), which is called in order to figure out how to call the scanner.
sorry, screwed up.
> And that's where something is wrong with SBS, because further investigation has > shown that even if you export the TMPDIR environment variable to the > slimserver.pl script, it will see it (and SQLite *will* use it), but when SBS > calls the scanner.pl script, the TMPDIR environment variable isn't defined any > more (although it's correctly passed to the script by the system, 9th post in > the first forum thread linked shows it clearly). How do you export that variable? I just run a test: TEST=yo slimserver.pl The value for TEST would be seen by the server as well as the scanner.
Created attachment 7447 [details] Patch to SQLiteHelper.pm for debuging purpose (In reply to comment #15) > How do you export that variable? I just run a test: > > TEST=yo slimserver.pl > > The value for TEST would be seen by the server as well as the scanner. The way SSOxS sets environment variables is very basic, the script starting up SBS sources a ssods.conf file containing UNIX shell "export" commands. Here's an excerpt from mine: # directory for temporary files SSODSTMP=$SSODSDIR/var/tmp export TMPDIR=$SSODSTMP export TMP=$SSODSTMP export yo=$SSODSTMP As you can see, it already exports TMPDIR and TMP, and I've added a "yo" environment variable exported the same way for the tests. With my debuging patch applied to r33287, when starting SBS on the NAS, you can see that in server.log: [11-09-02 19:37:55.3973] Slim::Utils::SQLiteHelper::on_connect_do (120) BEGIN === bug 17471 === [11-09-02 19:37:55.3982] Slim::Utils::SQLiteHelper::on_connect_do (121) $ENV{'yo'}='/opt/ssods4/var/tmp' [11-09-02 19:37:55.3991] Slim::Utils::SQLiteHelper::on_connect_do (122) $ENV{'TMPDIR'}='/opt/ssods4/var/tmp' [11-09-02 19:37:55.3999] Slim::Utils::SQLiteHelper::on_connect_do (123) END === bug 17471 === So, it's clear that both "TMPDIR" and "yo" are defined at this point. Now, launching a wipe scan, here's what I'll have in scanner.log: [11-09-02 19:38:51.1850] main::main (244) Cache init... [11-09-02 19:38:51.2035] Slim::Utils::SQLiteHelper::on_connect_do (120) BEGIN === bug 17471 === [11-09-02 19:38:51.2044] Slim::Utils::SQLiteHelper::on_connect_do (121) $ENV{'yo'}='/opt/ssods4/var/tmp' [11-09-02 19:38:51.2053] Slim::Utils::SQLiteHelper::on_connect_do (122) $ENV{'TMPDIR'}=<undefined> [11-09-02 19:38:51.2061] Slim::Utils::SQLiteHelper::on_connect_do (123) END === bug 17471 === [11-09-02 19:38:52.1508] main::main (265) Squeezebox Server Scanner done init... As you can see, "TMPDIR" is not defined any more, although "yo" is still there. But now, to be perfectly honest, I've made the same test on my Mac machine and Linux laptop out of curiosity, and I wasn't able to reproduce this problem on none of them, "TMPDIR" is still defined in scanner.log. My first thought was that neither the Mac nor the Linux laptop were using the CPAN libraries bundled with the SBS archive, but forcing their use on the Linux laptop didn't show the problem. At this time, I can only suppose the perl 5.10 distribution coming with SSODS is buggy...
Closing this bug as it's clearly not related to Squeezebox Server.
Reopening this bug, cause I first thought the bug only occurred when SBS ran from SSODS, but I could reproduce it on my Debian GNU/Linux laptop today: that's the --user switch that causes the TMPDIR environment variable to vanish.
There seems to be something fishy going on here. I've done several small test programs in c/Perl/shell running other c/Perl/shell test programs that print the TMPDIR env variable. All combinations of caller/calles and variants of exec(3) used and with no/single/double fork() resulted in what we expect (the TMPDIR env variable beeing there). However, in SBS, that doesn't work somehow. It uses Proc::Background, which uses (CORE::)exec to run the scanner.pl program. But for some reason the TMPDIR variable (and as it seems only this) is not preserved. As far as I could see Perl uses the execve() system call. And all is fine with that in my test programs. My guess is that one of the libraries/packages used in SBS overrides the CORE::exec routine (or some other routine involved), which specifically removes TMPDIR from the environment passed to the process. (Maybe that's a useful attack vector and removing it is a security measure?) It would be interesting knowing what exactly is going on. A work-around would be to pass a --tmpdir=/path/to/some/tmp/dir to scanner.pl and in there set the TMPDIR env variable accordingly. The cache dir seems to be a sensible default. I'll do some more testing when I have new ideas on what to test.
Dropping privileges is the culprit: Try this: --------------- print "ENV{TMPDIR} = $ENV{TMPDIR}\n"; print "real TMPDIR = " . `echo \$TMPDIR` . "\n"; $) = "1000"; # GID $> = "1000"; # UID print "ENV{TMPDIR} = $ENV{TMPDIR}\n"; print "real TMPDIR = " . `echo \$TMPDIR` . "\n"; ---------------- Calling "sudo TMPDIR=/foo/bar ./test.pl" prints: --------------- ENV{TMPDIR} = /foo/bar real TMPDIR = /foo/bar ENV{TMPDIR} = /foo/bar real TMPDIR = --------------- %ENV still has the TMPDIR, but commands run from Perl do not have it in the env. Interestingly, /proc/$PID/env _does_ have the TMPDIR still listed after setting the new uid/gid. So I'm thinking that removing the TMPDIR from the environment is a system (glibc) feature of exec*() functions. I've not been able to find any documentation on that, or even the responsible code. So I'm suggesting that scanner.pl sets the TMPDIR env variable if it has a --tmpdir command line switch, which is to be used by the slimserver.pl process if it has $ENV{TMPDIR}.
Created attachment 7469 [details] Suggested patch to fix the problem Seems to work here. Needs further testing, I guess.
Andy - this patch looks reasonable to me. Should we include it?
Sure, looks fine to me.
Philippe/Sébastien - we don't feel like adding this at this stage. We don't want to break some of the working installations by fiddling with the environment. But we should follow up on this post 7.7.0 release. I still believe using Custom.pm for such cases would be best. What about this plan: - we add the --tmpdir parameter to scanner.pl - we add some line to Import.pm checking whether Slim::Utils::OS->getTmpDir() would return something, and add this to the params for scanner.pl This way you could add the TMPDIR value reading the environment or using SBS' dirsFor('cache') or similar. Thoughts?
All my ideas from comment 20/21 do not work. I've now actually tested it now and it seems you cannot modify TMPDIR from within Perl if the effective UID is not the real UID (see perlsec docu and the Perl changelog for some vague references). Maybe one could work-around or fix this in main::daemonize() and/or main::changeEffectiveUserAndGroup() (double-fork?). I guess you don't want to mess with that. Hence I'd close this bug as WONTFIX. I'll patch the SQLite stuff in SSODS to use /opt/ssods4/var/tmp instead of /tmp.
(In reply to comment #25) > All my ideas from comment 20/21 do not work. I was just giving them a try through a simpler test file, and went to the same conclusion. > I'll patch the SQLite stuff in SSODS to use /opt/ssods4/var/tmp instead of > /tmp. Why don't you rather run SBS as a simple user? You wouldn't need to use the --user switch, and wouldn't be faced to this problem, then.
Andy, Michael, before closing this bug for good, I'd suggest some warning info about this problem being added somewhere in SBS/LMS doc. Time for a KNOWN-PROBLEMS.txt file, maybe?