Bug 16874 - SBS Crashes (CPU spin) after update to 7.5.3
: SBS Crashes (CPU spin) after update to 7.5.3
Status: CLOSED FIXED
Product: Logitech Media Server
Classification: Unclassified
Component: Mac Preference Pane
: 7.5.x
: Macintosh MacOS X 10.6
: P1 critical with 5 votes (vote)
: 7.5.x
Assigned To: Andy Grundman
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-02-01 20:51 UTC by Tony
Modified: 2011-12-16 02:12 UTC (History)
7 users (show)

See Also:
Category: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tony 2011-02-01 20:51:36 UTC
I'm getting crashes since updating to 7.5.3 (same problem with 7.5.4 beta).
When playing a radio stream (i.e. 1010 WINS), after a few minutes, SBS crashes with 100% cpu.
(problem is resolved after re-installting 7.5.2).

Here's TOP output:

PID COMMAND %CPU TIME #TH #WQ #POR #MRE RPRVT RSHRD RSIZE VPRVT
184 perl5.10.0 99.8 01:38.94 1/1 0 18 227 97M 264K 100M 113M
Comment 1 Michael Herger 2011-02-01 21:33:04 UTC
could you please give us the stream's URL or instructions how we can find it? If I search for "1010 WINS" I get an entry "1010 WINS - Not Supported". I've never seen this...
Comment 2 Michael Herger 2011-02-01 21:33:49 UTC
Ah... "Stream is Country restricted". But it would play that message.

Do you have any other stream causing your issue?
Comment 3 Tony 2011-02-02 05:45:17 UTC
(In reply to comment #2)
> Ah... "Stream is Country restricted". But it would play that message.
> 
> Do you have any other stream causing your issue?

Sure here's the stream.

http://opml.radiotime.com/Tune.ashx?id=s29616&formats=aac,ogg,mp3,wma,wmvoice&partnerId=16&serial=81fc0e417ebefb6df89c6fa969205fee

But please note that this is only an example.  I've had crashes with other streams.
Comment 4 Tony 2011-02-02 05:46:30 UTC
(In reply to comment #2)
> Ah... "Stream is Country restricted". But it would play that message.
> 
> Do you have any other stream causing your issue?

I've also had it with:

http://opml.radiotime.com/Tune.ashx?id=s2588&formats=aac,ogg,mp3,wma,wmvoice&partnerId=16&serial=9c51827a80620dddb98a24154208d0f3
Comment 5 Tony 2011-02-02 06:10:06 UTC
Here's the log at the time of the crash:

[11-02-01 13:30:20.0386] Slim::Utils::Scanner::Remote::__ANON__ (237) Error: Can't connect to remote server to retrieve playlist for, [url]http://1382.live.streamtheworld.com/WINSAMDIALUPCMP3:[/url] Timed out waiting for data.
[11-02-01 13:30:20.9822] Slim::Utils::Scanner::Remote::__ANON__ (237) Error: Can't connect to remote server to retrieve playlist for, [url]http://1382.live.streamtheworld.com:3690/WINSAMDIALUPCMP3:[/url] Timed out waiting for data.
[11-02-01 13:30:21.9807] Slim::Utils::Scanner::Remote::__ANON__ (237) Error: Can't connect to remote server to retrieve playlist for, [url]http://1382.live.streamtheworld.com:443/WINSAMDIALUPCMP3:[/url] Timed out waiting for data.

I force-quit perl at this point, looks like its stuck in a loop
Comment 6 Tony 2011-02-02 06:12:15 UTC
I've also had this problem with 7.5.4 beta as well as 7.5.3.
I re-installed 7.5.2 last night -- ,no crashes so far
Comment 7 Michael Herger 2011-02-02 06:29:10 UTC
no problem here playing the second stream (the first one gives me the "not in your country" message)
Comment 8 Tony 2011-02-02 07:25:09 UTC
(In reply to comment #7)
> no problem here playing the second stream (the first one gives me the "not in
> your country" message)

What about the output from my log?  Does it look like it's stuck in a loop?
(still playing with 7.5.2, only crashes with 7.5.3)
Comment 9 Tony 2011-02-03 06:06:41 UTC
(In reply to comment #8)
> (In reply to comment #7)
> > no problem here playing the second stream (the first one gives me the "not in
> > your country" message)
> 
> What about the output from my log?  Does it look like it's stuck in a loop?
> (still playing with 7.5.2, only crashes with 7.5.3)

Well, so far, it appears that shortening the stream addresses on my stations fixed the problem.

For example, changing:
http://opml.radiotime.com/Tune.ashx?id=s2588&formats=aac,ogg,mp3,wma,wmvoice&partnerId=16&serial=9c51827a80620dddb98a24154208d0f3

to:

http://opml.radiotime.com/Tune.ashx?id=s2588

seems to work.
Comment 11 Tony 2011-02-19 08:57:00 UTC
Seems the problem is that SBS gets stuck in a "Timed out waiting for data" loop.  This problem was introduced with 7.5.3.

Unfortunately, this bug has not yet been assigned (even though the problem has been identified).
Fortunately (for me), I was able to identify and work-around the problem.

Tony
Comment 13 David Potts 2011-02-25 09:19:57 UTC
I have exactly the same problem/symptoms as Tony describes. I believe you will find, if you check, that all the urls that cause the problem are streaming AAC files. Urls that stream MP3s don't cause don't cause this problem. The problem has to do with the FAAD conversion of AAC to FLAC. See here:

http://forums.slimdevices.com/showthread.php?t=85779

So, besides rolling back from 7.5.3 to 7.5.2, an alternative way to fix the problem is to disable AAC decoding in SBS.
Comment 14 David Potts 2011-02-25 09:20:22 UTC
*** This bug has been confirmed by popular vote. ***
Comment 15 Tony 2011-02-25 09:29:34 UTC
(In reply to comment #14)
> *** This bug has been confirmed by popular vote. ***

This bug hasn't even been assigned yet!

Tony
Comment 16 David Potts 2011-02-25 10:11:41 UTC
(In reply to comment #15)
> (In reply to comment #14)
> > *** This bug has been confirmed by popular vote. ***
> This bug hasn't even been assigned yet!
> Tony

I didn't write Comment #14. I have no idea how it got there. (But I did vote for this bug.)
Comment 17 David Potts 2011-02-25 10:19:40 UTC
Here is another datum that I take to indicate that this bug is related to AAC decoding. Yesterday, after SBS crashed when listening to Internet radio, when I restarted SBS 7.5.3, I got the following message from my Windows Home Server: "faad.exe encountered a problem and needed to close."
Comment 18 Sean Echevarria 2011-02-25 10:40:12 UTC
The changelist for 7.5.3 indicates that faad was updated for aac->pcm transcoding (bug 16379).
Comment 19 p_lemonde 2011-02-28 11:17:26 UTC
Can we get this assigned and have someone working on it please.

The platform should be widened as it is definitely affecting WHS (in this bug report) and Debian users as well (me).

AAC internet radio streams are quite common, meaning a problem for those of us that want to listen to them. And downgrading to 7.5.2 means I can't simply use apt-get to keep my system up-to-date without having to change my configuration now as well, as it would default to upgrading Squeezebox Server.
Comment 20 UliMy 2011-03-03 12:33:06 UTC
With the latest "nightly" "SqueezeboxServer-7.5.4-31979.exe" the bug seems to
be fixed. With squeezebox.com (31910) the bug is still there...
Comment 21 Andy Grundman 2011-03-08 19:13:39 UTC
I cannot reproduce this problem.

Can someone provide a log with player.source set to INFO and scan.scanner set to DEBUG?
Comment 22 p_lemonde 2011-03-08 19:29:28 UTC
Andy

I've gone back to 7.5.2 (again) today after swapping out the faad executable with the 7.5.2 one still wasn't enough to solve the problem on 7.5.3, so I can't debug it at the moment.

I've encountered the problem repeatedly by basically doing the following:
Start listening to an AAC station on a Receiver (so faad/flac is launched to transcode)
(it may race at this point)
Synchronize a second Receiver to the stream

I've not needed to do anything more than that before it hits 100% CPU and I end up having to kill the squeezeboxserver process.
Comment 23 UliMy 2011-03-09 10:26:14 UTC
Due to lack of technical knowledge I am not able to provide a log with player.source set to INFO and scan.scanner set to DEBUG. Sorry!

I think all this could be a firmware issue and comes with FW 132 or 52 (Boom),
effecting SBS and also mySB, playing either when playing radio streams . On my systems it happens every couple of minutes, all the time.

I also recognized that my SB resets completely about once in a day.

Andy, please, try to solve all this asap. Meanwhile I am really pissed off...
Comment 24 UliMy 2011-03-09 10:28:47 UTC
I think all this could be a firmware issue and comes with FW 132 or 52 (Boom),
effecting SBS and also mySB, playing either radio streams or mp3-files, kept locally on my computer.
Comment 25 SVN Bot 2011-03-10 07:49:53 UTC
 == Auto-comment from SVN commit #32060 to the slim repo by ayoung ==
 == http://svn.slimdevices.com/slim?view=revision&revision=32060 ==

bug 16874: SBS Crashes after update to 7.5.3 
Enable generation of stack trace upon signal USR2.
Comment 26 Alan Young 2011-03-10 22:39:32 UTC
Anyone who is getting this regularly this and running on Linux or Mac could use the latest 7.5.4 nightly (r32061 or above), or they could apply the patch from comment 25. If the spin happens then sending signal USR2 to the perl process will generate a stack trace in the logfile. Something like "kill -USR2 nnnn" where nnnn is the process-id (PID) of the SqueezeboxServer perl process. Doing it a few times in succession is probably a good idea. Attach the resulting stack traces to this bug report please.
Comment 27 Andy Grundman 2011-03-21 10:29:07 UTC
I found it, it's a bug in the AAC ADTS header parser.  I need to reproduce it one more time in order to fix it.
Comment 28 Tony 2011-03-21 10:43:47 UTC
(In reply to comment #27)
> I found it, it's a bug in the AAC ADTS header parser.  I need to reproduce it
> one more time in order to fix it.

Thank You!!!

Tony
Comment 29 SVN Bot 2011-03-21 10:58:05 UTC
 == Auto-comment from SVN commit #856 to the opensource repo by agrundman ==
 == http://svn.slimdevices.com/opensource?view=revision&revision=856 ==

Bug 16874, fixed infinite loop that could occur when reading a truncated ADTS stream
Comment 30 David Potts 2011-03-21 11:36:41 UTC
(In reply to comment #29)
> == Auto-comment from SVN commit #856 to the opensource repo by agrundman ==
>  == http://svn.slimdevices.com/opensource?view=revision&revision=856 ==
> 
> Bug 16874, fixed infinite loop that could occur when reading a truncated ADTS
> stream

Bravo!
Comment 31 SVN Bot 2011-03-21 12:41:57 UTC
 == Auto-comment from SVN commit #32150 to the slim repo by agrundman ==
 == http://svn.slimdevices.com/slim?view=revision&revision=32150 ==

Fixed bug 16874, updated to Audio::Scan 0.87
Comment 32 Paul Chandler 2011-05-02 09:10:54 UTC
Fix verified in 7.6. 0 r 32383
Comment 33 Roger66 2011-12-16 02:11:01 UTC
Bug still exists in my LMS installation on Windows XP connected to a ethernet wired Touch and a Boom. Tried every build up to latest 7.7.2 nightly (12-15-2011). Last build really working correctly in this point was 7.5.2.

When playing a radio stream with AAC format as f.e. Radio Paradise the server always crashes. Sometimes in the first 10 seconds, sometimes after a few minutes, but more likely after heavy changing the radio streams and final going back to an AAC playing station. This happens whether changing the station on the server's web interface or local at the Touch's panel itself.

Last message in logfile is
[11-12-14 20:27:06.9689] Slim::Utils::Scanner::Remote::__ANON__ (240) Error: Can't connect to remote server to retrieve playlist for, http://scfire-dtc-aa06.stream.aol.com/stream/1049: Connect timed out: Bad file descriptor.
[11-12-14 20:29:24.0453] Slim::Utils::Scanner::Remote::__ANON__ (240) Error: Can't connect to remote server to retrieve playlist for, http://87.118.90.178:8003/live: Connect timed out: .

This always goes together with a significant increase of cpu load. On a Intel CoreDuo 1.5 GHz it goes up to 60-70% for a 10 seconds or so, before the crash message from server.exe appears.

Seems to me as still a problem with FAAD, as MP3 streams are always playing fine.

I have found 2 ways to "fix" this (except using 7.5.2):

- edit the stream URL by deleting everything after "&formats=", which forces playing the stream as MP3
- edit the AAC file preferences in LMS to force native AAC decoding by Touch disabling all AAC options containing FAAD

Both workarounds are no real option for me since I have a Boom which can't do native AAC decoding and I still want to enjoy AAC streams regularly sounding better than 128k MP3 streams.

A complete clean LMS reinstall didn't help.

Can somebody please investigate what's wrong with FAAD and AAC ?