Bugzilla – Bug 1307
RSS Reader being banned from Slashdot
Last modified: 2009-09-08 09:32:12 UTC
Seems some people are getting banned from the Slashdot RSS feed. "Your RSS reader is abusing the Slashdot server." <title>You May Only Load Headlines Every 30 Minutes</title>
are these people hacking the $refresh_sec variable in the module? it is set to 30 minutes. maybe it needs to be set to 40 * 60 instead.
I have the Slashdot RSS feed up all the time and have never seen this. Is it possible that servers are being restarted too often? Or that multiple players are causing retriggers? To be safe, let's up the timeout to 40 minutes as suggested. Dave: can you make this change?
Its easy to up the timeout. It might even be worthwhile build a more sophisticated cache that inspects the feed for a recommended time to live. But frankly I suspect either server restarts, or multiple servers on a subnet are the issue. Do we know who the people are with this error. Did I miss this on the discussion group?
I don't think this is a simple timeout issue. I first encountered this with a stock 6.0b1 install on OS X with no modifications whatsoever. Robert Rozeboom of Slash.dot emailed me saying I had hit their RSS feeds almost 250 times in an hour.
slashdotting slashdot....rather poetic really :) I haven't seen this either, and I would have expected that multiple clients would still be sharing the same cache. if not, it would help to make it so that the server is connecting once for all players.
I experienced this today it seems, while I was at work. Slimserver was not restarted during that time. The $refresh var was not changed. The unit was on/idle and my settings specify that the rss screen saver is used in that mode. I ran a sniffer for about 2 hours tonight and only saw a request every 30min, and I tried restarting slimserver during that time to force an update, but it still never requested an update more than every 30min. Over the weekend I restarted slimserver several times, but the unit was "off" during that time, and the 'off' screen saver was a modification of the date/time screen saver that I was working on. Would slimserver try to do anything in relation to the rss screen saver if it is never activated? Does it still try to refresh the feeds before it is used?
Thanks for such a detailed report. slashdot isn't known for being flexible or, for that matter, accurate ;) I expect that they may have an issue with even 30:01 or perhaps their version of 30 mins is 29:59. If so, the 40 minute update rate would be sufficient. The more intelligent cache is nicer becuase it would allow updates more frequently from feeds that allow it.
I'm unable to reproduce this bug. I left a server running all day with the plugin running as screensaver, and refreshing about every 5 minutes. But slashdot always returns with the feed. It's not complaining that I'm hitting the server too frequently. Can anyone reproduce this consistently? I changed the default feed refresh to every hour, just in case.
I just had this happen to me. I was out for dinner, while running from the trunk. When i got back, the rss feed showed that it was banned. annoyingly, no debugging was on at the time :( however, when I restarted d_remotestream showed the server Getting the slashdot feed again. Given that I was testing the server before I left, it may have grabbed too many times. we probably need some sort of caching across restarts.
So, when one gets this message from slashdot, one gets an actual RSS feed in which the text is something like "you are abusing..." correct? This means as far as the reader is concerned, it got a feed, and it won't refresh that feed for another half hour, or whatever the refresh time is set to. Could it be that slashdot is fooled into thinking that a great many requests are coming from your machine, when in reality a great many request are coming from all the machines in you subnet, office, or perhaps even ISP? I have a favor to ask of anyone who can reproduce.... Next time it happens, go to a web browser and request the feed <http://slashdot.org/index.rss>. Save the source and send it to me. (If your browser is on the same machine as your slimserver, I'm assuming it will also get the "you are abusing..." message) Thanks.
People who can reproduce this bug: Are you connecting through a proxy server?
I run the server on Mandrake 10.0, connecting to the 'net through a wrt54G with wifi-box firmware. Slashdot should be seeing the IP on teh router, which isn't shared through any proxy on the provider that I know of.
using d_perf, I got this. might this have something to do with this?? 005-04-16 20:58:10.6123 Idle loop speed: 0 iterations per second Deep recursion on subroutine "Plugins::RssNews::retrieveNews" at /usr/local/slimserver/trunk/Plugins/RssNews.pm line 655. Deep recursion on subroutine "Plugins::RssNews::getFeedXml" at /usr/local/slimserver/trunk/Plugins/RssNews.pm line 485. Deep recursion on subroutine "Slim::Player::Protocols::HTTP::content" at /usr/local/slimserver/trunk/Plugins/RssNews.pm line 429. Deep recursion on subroutine "main::idleStreams" at /usr/local/slimserver/trunk/Slim/Player/Protocols/HTTP.pm line 347. Deep recursion on subroutine "Slim::Networking::Select::select" at ./slimserver.pl line 654. Deep recursion on subroutine "Slim::Networking::Slimproto::client_readable" at /usr/local/slimserver/trunk/Slim/Networking/Select.pm line 115. Deep recursion on subroutine "Slim::Networking::Slimproto::process_slimproto_frame" at /usr/local/slimserver/trunk/Slim/Networking/Slimproto.pm line 236. Deep recursion on subroutine "Slim::Player::Squeezebox::reconnect" at /usr/local/slimserver/trunk/Slim/Networking/Slimproto.pm line 340. Deep recursion on subroutine "Slim::Player::Player::update" at /usr/local/slimserver/trunk/Slim/Player/Player.pm line 429. Deep recursion on subroutine "Slim::Player::Player::update" at /usr/local/slimserver/trunk/Slim/Player/Squeezebox.pm line 92. Deep recursion on subroutine "Slim::Display::Display::curLines" at /usr/local/slimserver/trunk/Slim/Player/Player.pm line 172. Deep recursion on subroutine "Plugins::RssNews::lines" at /usr/local/slimserver/trunk/Slim/Display/Display.pm line 121.
Dave, do you know how to reproduce? (My regular slashdot feed is fine.) I that stack trace even related?
I can't say if it was related or not, I was already banned at the point where I was doing this debugging.
The stack trace may be related. I don't know how to reproduce. I suspect converting the RSS screensaver to use async http might solve this particular problem. But I'm hesistant to do that because this recursion could affect anything that uses Slim::Player::Protocols::HTTP. It would be nice to understand this better and I will look into it. Dan, any thoughts?
Here's my theory... From KDF's d_perf output, we see deep recursion, and that his client is reconnecting. During the reconnect, the player is put into Rss screensaver mode, which attempts to download a feed. I believe for some reason this is taking a long time, long enough for the client to send yet another reconnect message while the feed is being downloaded. The feed download recursively calls idleStreams (so that the server does not completely freeze during the download), and idleStream calls select. Yet another reconnect is sent from the client, which again causes the stream to download. According to this theory, the bug will happen whenever there is a substantial delay in downloading the feed. Perhaps a dns resolve that takes 10 seconds? So for anyone whos seen this bug in the wild, do you know what your network was doing at the time? Does your player show the "Lost contact with SlimServer" message? I added the following immediately before the call to $http->content() in getFeedXml, and it caused the recursion to happen, but I haven't let it run long enoungh for slashdot to complain. # bug 1307 # sleep long enough to trigger reconnect sleep 20; ::idleStreams(0.1); From Fazal's comment that his server pinged slashdot 250 times an hour, that's once every 14 seconds, pretty consistent with this theory. Regarding the solution: RssNews can detect and avoid this recursion. But that solves the problem only for RssNews. Perhaps some change could be made to the reconnect code that catches this problem for all modes. Any thoughts?
Created attachment 441 [details] possible fix for SqueezeboxG and Slimp3 players Ok, here's an attempt to kill the recursion in update(). Seems reasonable to not allow recursion there, right? KDF et al, I'm not sure how easily you can reproduce the bug, but please try with attached patch. And let us know what happens.
KDF, can you try with the recently attached patch?
I've applied locally and will put slashdot back in the list of feeds. I'll report if anything comes up.
tried the patch and I seem to be banned again (or still). maybe their 72 hours is really permanent :(
I've checked in changes localized to RssNews.pm that should fix the slashdot issue. Its unlikely this sort of recursion occurs in other modes. However changes to idleStreams are in the works that should prevent it from happening elsewhere. But to solve it rapidly, I checked in revisions 3041 (6_0_X) and 3042 (trunk). If slashdot has you banned, you may have to wait for the ban to expire, but afterwards this should not occur again. Please let me know if it does.