Bugzilla – Bug 10917
multiple instances of socketwrapper and lame
Last modified: 2009-10-05 14:29:42 UTC
Does anyone know why I'd have multiple instances of socketwrapper.exe and lame.exe running in my windows task manager? This occurs on my XP server after listening to music (FLAC's and MP3's down sampled with lame to 96k) streamed via http with Squeezeplay. Currently I have over 30 instances of lame and 10 or so instances of socketwrapper... each consuming approximately a 1,600K to 1,700K. Is that behavior normal? Thanks! Currently running: Squeezecenter: 7.4 - 24777 @ Wed Jan 28 01:03:55 PST 2009 Operating system: Windows XP - EN - cp1252 Platform Architecture: 586 Perl Version: 5.8.8 - MSWin32-x86-multi-thread MySQL Version: 5.0.22-community-nt And the latest Squeezeplay packaged with 7.4 (version 7.3r0?)
*** Bug 10999 has been marked as a duplicate of this bug. ***
hi there, this problem exsists already in rel. 7.3.2! playing mp3 files via server located decoders like lame or mad will open a new socketwrapper plus lame, mad and/or flac process each time a forward or back command is entered in the squeezecenter gui or a new mp3 file is played. when letting the squeezecenter play number of tracks by itself, the previous socketwrapper (plus lame or mad) process is closed automatically and new one starts. according to my observations, this behaviour seems new from rel. 7.3.2. rel.7.3.1 did close these extra processes. maybe a setting to support the sox was changed??? br duesentrieb
Observing on SC 7.4-25017/Windows: These seem to occur when transcoding breaks, i.e. by the remote player aborting or losing contact. Would there be any means to mark these »orphaned« after a while and kill them?
Can QA please verify exactly which version this was first seen in, in particular if it is seen in 7.2.1.
Just to update: Behaviour as per comment #2 is reproducable in SC 7.4-25416 (svn/Win).
Alan: The first version of SC with this error is 7.3.0 I tested 7.0.0 - 7.2.1 as OK.. I.E. no extra socketwrappers or CPU Max load. Starting with 7.3.0 I see the extra socketwrappers and the CPU max's the more I FF/RW tracks in a playlist.
Yesterday I was testing transcoding from ogg to both aiff and mp3, streamed to an SB1. In (Windows Server 2003) Task Manager I just noticed seven instances each of socketwrapper.exe and sox.exe. Nothing unusual - pausing, stopping, starting, skipping tracks.
Michael, would you be able to take this as I do not have a suitable test environment? Alan.
Ping Michael.
Is this specifically for local file transcoding rather than remote streaming?
Yes. It relates to when Pipeline is being used with both input and output going/coming from/to SC. Here is is my theory, although it is obviously flawed because my fix does not work. 7.3 changed the implementation of Pipeline.pm to stuff the input pipe more aggressively. Another change means that the output will probably be emptied more quickly too. (1) Streaming starts up and there is a network connection from SC to socketwrapper, and another one from socketwrapper back to SC. socketwrapper has two threads (plus its main thread), and a child transcoding process. Pipes connect the inputs and outputs of the child process to socketwrapper. The threads are used to shuffle data from the input network connection to the child process input pipe (the 'input thread'), and similarly on the output side (the 'output thread'). (1.1) The main thread is blocked in WaitForMultipleObjects, waiting for the child process to die. (2) The user skips to the next track and so Pipeline::close() is called, which closes both the network connections to the socketwrapper process. (2a) Either the output thread was already blocked in send(), or it calls it soon after. It notices the error because the socket has been closed. It tries to flush its end of the network connection and then the thread exits, without closing its hInput handle. (2b) The input thread is blocked trying to write to the child input pipe. Therefore it does not notice that the input network connection has been closed. (2c) The child process is blocked in write on its output pipe. (3) Nothing is left that will read from the output pipe of the child process, so it remains blocked. The theory looks good but must be flawed. I tried adding a 'CloseHandle(pS->hInput)' before _endthreadex in MoveDataThreadProc but it appeared to make no difference. I assumed that closing the output pipe for the child process would cause the write from the child process to fail and that process to exit. Somehow, that does not seem to happen. My next attempt would be kill the child process when the output thread dies because its output connection has an error. Other suggestions welcome.
I'll have a look at this issue but it'll take a day or so to create a suitable setup to reproduce.. Does it happen on Vista as well or just XP ? I "inherited" socketwrapper and tidied it up to into its current version. IIRC The "_endthreadex|" call is special. I tried a lot of variants of "closes" and process killing to get a reasonably robust version as Windows documentation is woeful in this area. I think I have a version of socketwrapper which kills orphans on exit because I think there was a similar problem with alac. Eventually I found the real problem. I think the fix required a mod to alac as its handling of piped input/output left child process as a zombie when killed. I wonder whether lame and sox have the same issue
Debugging help. If you run SC from a command prompt with player.source set to DEBUG - you will get debug info from socketwrapper on command prompt console window. Socketwrapper messages are prefixed with "SW:"
Created attachment 5330 [details] output Were you asking for the debugging output? This is a capture of the screen output while skipping tracks. Each skip left a socketwrapper and sox instance running. - SC 7.4 r27032 SQLite branch - Windows 2k3 Server - Run from the command line - standard convert.conf - player.source set to DEBUG, all other categories to ERROR - Streaming Ogg transcoded to AIFF to a Squeezebox 1
It's been about 2 years so memory maybe faulty but I don't believe I tested socketwrapper in the way it is being used now. With AlienBBC and other users of socketwrapper at the time - the source of data was an application and not a pipe.
If I use the same setup as above, but I stream Ogg -> Mp3 (sox | lame) I see the following behavior: - Stop always kills socketwrapper, sox and lame. - Pause always leaves socketwrapper, sox and lame running. - FWD (skip) _sometimes_ leave them running, especially if you hit multiple FWDs in quick succession.
(In reply to comment #15) > It's been about 2 years so memory maybe faulty but I don't believe I tested > socketwrapper in the way it is being used now. > > With AlienBBC and other users of socketwrapper at the time - the source of > data was an application and not a pipe. If my theory is in any way right, then an application that has unlimited access to data (a local file) would suffer the same problem.
(In reply to comment #16) > If I use the same setup as above, but I stream Ogg -> Mp3 (sox | lame) I see > the following behavior: > > - Stop always kills socketwrapper, sox and lame. > > - Pause always leaves socketwrapper, sox and lame running. > > - FWD (skip) _sometimes_ leave them running, especially if you hit multiple > FWDs in quick succession. Does Stop always kill socketwrapper if you invoke it very soon after track start?
>If my theory is in any way right, then an application that >has unlimited access to data (a local file) would suffer the same problem. I don't understand. I never tested input from a pipe because in those situation SC didn't need or use socketwrapper to work. Why is socketwrapper needed in this case ? Are the processes that are left behind orphans ? (i.e. the parent process id is an id of a dead process). One of my plugins (CDplayer) uses the cddawav.exe app and this leaves orphans v. easily - as a result I have a Perl routine which can tidy Windows orphan processes. This may be useful to run as part of the "close".
(In reply to comment #18) > > Does Stop always kill socketwrapper if you invoke it very soon after track > start? No, I spoke too soon. Usually, but not always. This might also be interesting... I killed the server and all the instances of socketwrapper died, all the instances of lame died, but all of the sox remained running.
(In reply to comment #20) > This might also be interesting... I killed the server and all the instances of > socketwrapper died, all the instances of lame died, but all of the sox remained > running. That would be consistent with my theory.
(In reply to comment #19) > >If my theory is in any way right, then an application that > >has unlimited access to data (a local file) would suffer the same problem. > > I don't understand. I never tested input from a pipe because in those > situation SC didn't need or use socketwrapper to work. > > Why is socketwrapper needed in this case ? Pipeline.pm always uses socketwrapper on Windows. Pipeline.pm is always used when the transcoding pipeline is being fed by SC (as well as being consumed by it). > Are the processes that are left behind orphans ? (i.e. the parent process id is > an id of a dead process). One of my plugins (CDplayer) uses the cddawav.exe > app and this leaves orphans v. easily - as a result I have a Perl routine which > can tidy Windows orphan processes. This may be useful to run as part of the > "close". No, the processes are not orphans.
(In reply to comment #12) > I think I have a version of socketwrapper which kills orphans on exit because I > think there was a similar problem with alac. Eventually I found the real > problem. I think the fix required a mod to alac as its handling of piped > input/output left child process as a zombie when killed. I wonder whether lame > and sox have the same issue I wonder if this is a generic problem with writing to pipes on Windows.
Are you sure - I thought some older version of SC didn't use socketwrapper for all cases. It sounds like the sox processes are orphans. They have no live parent and so stay around after SC dies. The best way to see this is with Process Explorer - when processes become orphans they change position in the process hierarchy away from SC branch.
I have to go now but it sounds like I should reproduce the problem here. > I wonder if this is a generic problem with writing to pipes on Windows. I think it is common for applications which create a child process to handle input and parent does output - typically direct ports from Linux. The child process doesn't die when the parent is killed say due to broken pipe as child is waiting for input when parent dies.
I've reproduced the problem on Vista and compared 7.2.1 and 7.3.3 using bit rate limiting on MP3 files. With 7.2.1 no problem as only lame is started - socketwrapper is not used. With 7.3.3 both socketwrapper and lame stay around when another file is played. Basic question - why is socketwrapper being used in 7.3.3 when it was not used in 7.2.1 ? You're right - no orphan processes. I didn't appreciate socketwrapper stays around - this prevent orphans as parent is alive. socketwrapper will only die when the output socket is closed by SC. If socketwrapper closes the output pipe the last few bits in the output pipe will be lost and SC user will not get full transcoding. This was the "last few secs are cut off" bug of 2 yrs ago. What needs to be checked out is whether or not SC is closing the port which it using to receive transcoded data from socketwrapper. Once that port is closed then socketwrapper will stop.
(In reply to comment #26) > With 7.2.1 no problem as only lame is started - socketwrapper is not used. > With 7.3.3 both socketwrapper and lame stay around when another file is played. > > Basic question - why is socketwrapper being used in 7.3.3 when it was not used > in 7.2.1 ? So that seeking (song scanner) can be used in conjunction with such transcoding. You are right, this is a change in 7.3, in that double-ended pipelines are used more often. > You're right - no orphan processes. I didn't appreciate socketwrapper stays > around - this prevent orphans as parent is alive. > > socketwrapper will only die when the output socket is closed by SC. If > socketwrapper closes the output pipe the last few bits in the output pipe will > be lost and SC user will not get full transcoding. This was the "last few secs > are cut off" bug of 2 yrs ago. > > What needs to be checked out is whether or not SC is closing the port which it > using to receive transcoded data from socketwrapper. Once that port is closed > then socketwrapper will stop. Yes, SC is closing both the input and output sockets. I have checked all this with Process Explorer. Do you not agree with my analysis that says that the problem is with the output thread not closing its input file handle when it detects that the output socket has been closed? My feeling is that it would be safe to add a ProcessTerminate of the child here when the worker thread detects that its output connection has been closed (not the input connection)- do you aggree?
After seeing the problem and using debugger - I think I agree with your analysis. I need to think about the TerminateProcess - as socketwrapper has to be careful about not flushing data from pipes as otherwise you get secs chopped off songs.
(In reply to comment #28) > I need to think about the TerminateProcess - as socketwrapper has to be careful > about not flushing data from pipes as otherwise you get secs chopped off songs. Yes, that is why I suggested only killing a worker thread's input child process when that worker thread's output has been closed. If the output connection or pipe has gone then there is no more chance to flush data.
Not sure but I think the tidy up routine may need to be rewritten to suit the way socketwrapper is invoked here. With AlienBBC and other uses of socketwrapper the process list was 0 - Process such as mplayer, alac etc. no input actually from input socket 1 - thread with named piped (i.e. #PIPE#) 2 - process - such as flac /lame 3 - Thread - output socket With the usage of socketwrapper here - the process list looks like 0 - thread - input taken from input socket 1 - process - lame, sox 2 - thread - output socket. The issue is whether the handling of a process or a thread as the 1st process needs to be different.
(In reply to comment #30) > The issue is whether the handling of a process or a thread as the 1st process > needs to be different. I don't think so. I think that if a thread's output pipe or socket is closed, then the preceding process (if any) should be killed.
I have been comparing the difference in termination between the ALienBBC use of socketwrapper (i.e. 0 is a process) and current one (i.e. -0 is a thread). When "0" is a process the code will do timed WaitForSingleObject and if process does not terminated within timeout it is terminated. When "0" is a thread - then depending on a command line flag ( bWatchdogEnabled ) the WaitForSingleObject is either timed or INFINITE. At time of writing the code, SC never used this branch and so I never tested this branch of the code. The command line flag was put in to timeout jammed streams but it doesn't make sense to use it in the tidyup part of the code. I think it is a case of code that is unnecessary but never removed as not tested. If the bWatchdogEnabled test condition is removed then threads also have a timed WaitforSingleObject and then socketwrapper shutsdown as expected. I think this is the proper solution as it makes treatment of threads and processes similar. I'll do some testing.
I have built a version (V1.11beta) of socketwrapper which can be found here. This version has fixed the problem on my systems. http://homepage.eircom.net/~altondsl/slim/socketwrapperv111beta.zip To install 1. save current version opf socketwrapper in case you need to revert. 2. Download,unzip and replace current socketwrapper.exe. No need to restart SC. I have made the following changes: 1. On tidy up, the Wait on the threads is now timed and not Infinite. This means threads are treated the same as processes. The infinite wait is the cause of the socketwrapper not terminating. I have tested it and I don't think the timeout will result in any truncation of songs. 2. To speed up shutdown - the timeout value is reduced when the output pipe is closed. There is no need to wait for processes/threads to flush data since output has been cutoff. If no problems are found I'll post the patch file. During testing I found some problem with short songs typically 4 secs or less when transcoding for bandwidth limitaton is used. The problem seems to be in transcoding handling with some formats and not socketwrapper.
my comment regarding: Comment #33 from Bryan Alton <baltontwo@eircom.net> 2009-06-17 01:53:30 --- I have built a version (V1.11beta) of socketwrapper which can be found here. This version has fixed the problem on my systems. great work, brian! exchanged the socketwrapper in my setup with SC rel. 7.3.2. the excessive socketwrapper processes are now closing down automatically, under the conditions as described in comment #2. will let you know if sthg stays open in the further practical operation... thanks for your effort and best regards duesentrieb
Created attachment 5362 [details] Patch file to socketwrapper v1.10 to fix failure of socketwrapper to terminate Since no more remarks about the socketwrapper update - I attach the patch file for the V1.11beta of socketwrapper
Matt, How and where does socketwrapper get built? Get you apply this patch and make sure the new binary makes it into the builds. Thanks, Alan.
It needs VisualC 2005 to be built so the build should be just like faad.exe.
Hmm I've never buitl socketwrapper, nor have I seen that binary. Perhaps this has been built by hand in the past. Where's the source for it in SVn?
slim/7.4/trunk/tools/socketwrapper/
Created attachment 5363 [details] Patched socketwrapper.exe for testing...
:Passing back to QA to test the change before I check it in.
Steven, could you have a look at this?
*** Bug 13493 has been marked as a duplicate of this bug. ***
this is an administrative shuffle on priority fields to help make better judgment on the top end of the priority list. P4->P5, P3->P4, and P2->P3.
Matt: your patch is working please check it in.
== Auto-comment from SVN commit #28389 to the slim repo by mwise == == https://svn.slimdevices.com/slim?view=revision&revision=28389 == Fixed Bug 10917: Revision 27233 of socketwrapper.exe.
This bug has been marked as fixed in the 7.4.0 release version of SqueezeBox Server! * SqueezeCenter: 28672 * Squeezebox 2 and 3: 130 * Transporter: 80 * Receiver: 65 * Boom: 50 * Controller: 7790 * Radio: 7790 Please see the Release Notes for all the details: http://wiki.slimdevices.com/index.php/Release_Notes If you haven't already, please download and install the new version from http://www.logitechsqueezebox.com/support/download-squeezebox-server.html If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.