Bug 11227 - wireless bandwidth utilization is woeful
: wireless bandwidth utilization is woeful
Status: RESOLVED WORKSFORME
Product: SB Controller
Classification: Unclassified
Component: SB Server
: unspecified
: PC Fedora
: -- normal (vote)
: Investigating
Assigned To: Squeezebox QA Team email alias
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-01 19:09 UTC by Paul Davis
Modified: 2009-10-30 13:59 UTC (History)
6 users (show)

See Also:
Category: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Davis 2009-03-01 19:09:11 UTC
I've had an SB3 (or is it an SB2) for at least 2 years, and a Duet since last spring. My family and I were happy users of these systems - one installed in a bedroom and used for hours every night; the other connected to our main house music system and used for hours every day. The devices worked flawlessly (mostly) and I was thrilled to have purchased them when I did. 

At the end of August, we moved to Berlin for 6 months, and took the Duet system with us. It took a bit of work to set it up, and the wifi router there seemed to do nasty things to the wireless network every time the local DSL provider dropped the DSL link, but overall, it was still a fairly satisfactory experience.

I returned to the USA (with the Duet) 2 weeks ago. In that time, I have been absolutely unable to use either of the two systems with the *exact* same router and server as were in place before we left, at least not via wifi. I can use a wired connection, and things are fine. Via wifi, the devices are essentially unusable, even when in use within a few feet of the router (and, coincidentally, the server, which is on a wired connection to the router). 

I have done a lot of reading on the forums about similar issues, and as a result, have done several things.

1) tested the wired behaviour - flawless, as expected
2) rolled back to 7.2.1 and 7.0.1, and forward to the nightly build of 7.3.3 - no change in unusable wifi behaviour
3) run the network test often and in many different locations around the house
4) tested 2 wifi-equipped laptops in similar locations, using scp(1) to copy large files to/from the server
5) tested every wifi channel, using the network test on the controller. no channel makes any notable difference to successful bandwidth.
6) made sure to disable every wifi device in the house, and requested that my neighbours disabled their wifi systems for testing purposes.
7) verified that the controller reports my wireless network operating at 11Mbps

My observations at this point are as follows:

a) wifi-equipped laptops can get in excess of 4Mbps out of the existing wifi network, from almost any location in our house
b) neither the duet or the squeezebox can reliably get more than 128Kbps, even when positioned next the router *and* each other.
c) when the squeezebox is next to the router (signal strength approx 80-100%), the "fullness" level of the receive buffer rises slowly, preventing dropouts and rebuffering. It takes several minutes but it will generally rise monotonically, and thus offers usable operation *in this location*
d) when moved to the bedroom where it used to work flawlessly, the signal strength drops to 60%, and the fullness level never reaches more than 5-6%, frequently hitting zero again, and thus causing rebuffering
e) this behaviour is identical with 7.0.1, 7.2.1, 7.3.2 and the nightly of 7.3.3

As it stands right now, I have about US$700 of equipment which I purchased to allow wireless reproduction of music in our home. This equipment worked more than adequately a few months ago - with an apparently identical h/w setup, and a wireless network provably capable of delivering megabits per second, my slimdevices hardware is barely capable of 256kilobits per second. The Duet system also worked very well until just a few weeks ago in a different wireless configuration with reduced signal strength. I am not prepared to run cat5 cable around the (old) house to facilitate playback. 

I am my wits end. I have no idea what do next. The laptop tests seem to prove that there is absolutely nothing wrong with the house wireless network. I have no idea what might have changed with the h/w configuration since last summer, but my belief is that absolutely nothing has changed. At this time last year, I was excited to be ordering a Duet. At this point in time, I am wondering what the trade-in value might be. 

I would appreciate suggestions. Everything I have found in your forums and other bug reports seems to blame wireless networking, but I believe I have demonstrated reasonably well that if there is a problem with wifi bandwidth, it is a property of the slimdevices equipment, not my network h/w or configuration.
Comment 1 Marc Auslander 2009-03-02 09:50:05 UTC
You didn't by chance switch the wireless region to europe rather than US channels while in Berlin and not switch back?
Comment 2 Paul Davis 2009-03-02 09:56:14 UTC
No, not only did I not do that, the SB2 stayed here in the USA and has the same behavioural pattern as the Duet. If you prefer, we can focus solely on the SB2 if you want to eliminate the uncertainty over the "travels" of the Duet.
Comment 3 Keith Briscoe 2009-03-03 19:33:11 UTC
(I'm not a SlimDevices employee)

Considering that it worked before and no longer works, that the hardware on both ends of the connection is the same, that the method of connection is the same, and assuming there's no hardware defect on either side, then the answer seems to be pretty well narrowed down already--new interference is pretty much the only thing left.

In the past six months, have you or anyone who lives within a few hundred yards of you set up: a wireless router, an older cordless phone, a microwave oven, a bluetooth device, or anything of that nature?

If it is avoidable interference (i.e. a neighbor's access point on an overlapping channel), then just change your wireless channel.  NetStumbler is a helpful tool for this.  If it is unavoidable interference (can't identify the source), then I'd recommend using SqueezeCenter's bitrate limiting feature.  256kbps is still a decent of bandwidth if you're not opposed to lossy transcoding.  The fact that you are still able to achieve 256kbps is an indicator you don't have faulty hardware.

Wireless tests on laptops frequently produce rosy results, for many reasons: different network protocols (such as HTTP) that are more forgiving of sporadic network drops (but are unsuitable for streaming) and possible proprietary "pre-N" wireless tech being the two big ones.  For what it's worth, I've encountered wireless networks with severe problems that were completely unnoticeable if you were using a laptop to browse the web.  Streaming is a different and more demanding beast.  It's really worth re-assessing your network.
Comment 4 Andrew Rose 2009-03-04 05:41:38 UTC
Paul,

(I'm not a SlimDevices employee either)

I just added a Duet to my system last week.  After being totally thrilled with the SB3's performance for years I found the Duet to be a total waste.  Despite the fact that everything else in the house worked fine I changed to a different router and the system appears to work perfectly now.

YMMV but if you have a different router to test with you may find it worthwhile to test.  My old router was Netgear that has bugs listed here.  The new one was D-Link.

Good luck,
Comment 5 Blackketter Dean 2009-03-26 10:15:15 UTC
Paul: when you say the "Duet" do you mean the receiver or controller?

Also, is the controller under battery or charging when you do these tests?  The controller's power management will reduce bandwidth to save battery power.

Finally, can you do some ping testing against the devices and see if you are seeing packet loss, latency or some other issue.
Comment 6 Paul Davis 2009-03-26 10:28:01 UTC
I was using the term "Duet" to refer to the combination of the receiver and the controller. 

Ping testing shows trip times all under 10msecs, with the average at about 7-8msec. Packet loss is 14-18%. This covers the controller (in its charging cradle), the receiver and a the older squeezebox.

Do you want info on comparable data for 2 laptops? It occured to me that I can also use SqueezePlay on at least one of the laptops for somewhat more comparable testing.
Comment 7 Marc Auslander 2009-03-26 10:39:05 UTC
%14 packet loss is massive.  IMHO, something is very wrong in your setup.

By comparison, I just did 100 pings of my controller with NO packet loss.  Times similar to yours - 4 min, 7 avg, 20 max.
Comment 8 Chris Owens 2009-04-13 09:39:07 UTC
So, just to verify, the problem you are reporting is that there is packet loss?

This is not typical or expected (or generally reproducible) with either component of the Duet product.  Does this seem to be related to wireless range?  That is, if you move the device physically closer to the router does it get better?  

If so, then the action becomes more clear; that is, we can strive for improved wireless range in our products.

Unfortunately, due to the small physical size and low-profile shape of our devices (and the lack of a second (also called 'diversity') antenna in the controller, it's not going to be possible to achieve the same range as a laptop, but we can strive to get as close as possible.
Comment 9 Paul Davis 2009-04-23 16:50:38 UTC
Sorry for the delays in responding - very busy with software work (including hacking on a modified version of squeezeplay, not for my own use though).

I've just revisited the tests I described in the original bug report. I have stacked the Receiver on top, next to, and generally within 1 meter of the 802.11b router. The signal strength (SNR) is reported at 99-100%. 

A ping test from the server system (wired) to the Receiver still loses 5% of the packets with this configuration. Using the SqueezeController Network Test shows that rates above 192kbps are still not viable, and even that rate is unstable. Dropping to 128kbps becomes more or less a stable network rate. This is with a transmission range of a few centimeters and an SNR of 100%. How can this device operate so poorly?

With the original physical configuration, SNR varied between 60% and 85%. I've watched the logs on the server, and trying to ignore everything I know about the situation and just analyzing it as a software issue, I'd say that the buffering algorithm appears totally broken. One alternative is that the logging is misleading. What I can see happen repeatedly is the "fullness" figure for buffering on the Receiver will hover around 0-4% (which is OK in and of itself, though obviously it doesn't really provide much buffering). Occasionally it will rise to 20% and then drop again. Even more occasionally, it will hit 0% for long enough that the music stops and the rebuffering starts. But ... rather than a steady continuous rebuffering, the fullness figure instead yoyo's up and down (staying in very small numbers of bytes, and always at 0%). It appears that rather than stop data consumption on the Receiver while this buffering is taking place, instead the server is trying to get it rebuffered and the Receiver is still eating the data. It therefore takes a *long* time to get the buffer back up to a level where playback can resume. 

I also note that the fullness level stays *relatively* constant when compared with the buffer size. This suggests the following taking place on the Receiver: there is a buffer (about 3MB if memory serves correctly), and playback does not start until the buffer reaches a certain "fullness" level. After this, continued streaming keeps refilling the buffer. If the streaming rate drops below the playback rate for any sustained period of time, the buffer empties and playback stops. The problem is that playback appears to start a very very long time before the buffer is even remotely full, and therefore with a relatively low streaming rate (still adequate for playback, but low), it remains close to 0% - so close that degradations in the streaming rate caused by network interference will cause it to empty completely because there effectively is very *little* buffering going on. If the buffer was required to hit 85% full before playback started, it would probably hover between 75% and 90% full, occasionally dipping down to 50% when the network "fails" for a short time. Playback wouldn't stop. When the network resumes "normal" operation, the buffer would likely refill over time (though it may not, and might instead settle at the 45%-55% level - still better than what happens now).

But to return to the basic issue: with the Receiver  within a few centimeters of the router, packets are still being lost, and the network streaming rate (via the Network Test) is massively below what I can achieve with a wifi enabled laptop anywhere in the house. This is really problematic - I can understand the neither the Reciever, SB3 or Controller have antenna's that are the same size as the one in my laptop lid, but surely the difference cannot be this great?

BTW: to the commentator who noted that "streaming is totally different": streaming is not totally different, it just involves adequate buffering. I write audio software for a living, and have done lots of work with streaming audio. If the network supports a laptop getting 2Mbps rates on average (measured using, say, scp), then adequate buffering will make that rate appear sustainable, more or less, even as the actual rate fluctuates quite dramatically. It isn't necessary to move 2Mb across the network every second - its necessary to have a data buffer that can be consumed at 2Mbps on the receiving end. This is accomplished by buffering to hide the variability in the actual transmission rate.
Comment 10 Paul Davis 2009-04-23 17:27:04 UTC
another quick update: i redid the Network Test from the Controller to the Receiver, but with the Controller in the charging stand. Under these conditions, 256Kbps appears stable, but not 320Kbps.
Comment 11 Chris Owens 2009-04-27 15:42:47 UTC
I think perhaps there is some confusion about the purpose of Bugzilla here at the SMS team at Logitech.  Bugzilla is a service operated by the Quality Assurance group, where we gather information about (and coordinate fixing of) bugs.  A bug is a problem that affects a large number of users, and which requires engineering work to fix.  It's very unusual for commercial products to have bug databases which are open to the public as ours is, and this causes confusion from time to time.

Specifically, Bugzilla is not part of our Technical Support effort, which involves agents trained to solve user problems, diagnose hardware and software issues, recommend changes to the user's network and server, and issue replacement products if they deem it warranted.

The problem you're describing seems to be related to your network.  You note that your laptops do not suffer packet loss, but you also note that using the same hardware in the past, your products were working acceptably.

This problem does not appear to affect a large number of users and is not reproducible by the QA team here.  I don't think it falls into the prerogative of Quality Assurance, but instead belongs to Technical Support.  Especially if you are angling for an RMA, which QA cannot provide.

Would you like me to have someone from our support team contact you?
Comment 12 Paul Davis 2009-04-27 16:40:49 UTC
I believe that I fully understand the purpose of bugzilla (I develop professional audio software and have a similar bug tracker/feature tracker in addition to paid support contracts). I am attempting to do whatever I can to help everyone involved (including myself) fix whatever problem(s) are manifesting themselves in my setup. I am not asking slimdevices/logitech to "just fix this" - I understand from my own work that this is likely to be a complex problem that is not amenable to traditional "tech support" approaches.

When I encountered this situation on my return from Germany, a google search revealed what appeared to be many users all having problems with recent upgrades to squeezecenter that manifested as music stoppage and then rebuffering (sometimes with a rebuffering message, sometimes without). Reading the forums appeared to show a quite a lot of vocal discontent with the way people's systems had gone from operational to at best irritating and at worst non-functional. I concluded that I was not the only user with this problem, and from the reading I did, and my own testing of various releases of squeezecenter, it appeared that this problem/these problems were not yet fixed.

The title of this bug report reflects the most obvious behavioural property of my slimdevices hardware when I started to investigate what was happening. At this point in time, I am divided over which of (a) the lack of utilization of apparently available network bandwidth or (b) the buffering behaviour I am seeing in the logs is playing the larger role in the behaviour. 

In addition, by an odd coincidence, just after I started digging into this problem, I started doing some contract work that involves a developing a customized version of SqueezePlay. This led me to become a lot more familiar with the details of SlimProto and the overall design of your devices from a software and networking perspective. 

I am (currently) absolutely NOT seeking an RMA for my devices. I would like to bring whatever information, technical insight and so forth I can to the situation I am facing, which I believe is faced by other users of your devices who may not necessarily have the same technical skill set as myself. It appears to me that your bugzilla system is the most appropriate place to do this. If not, please suggest a better one. If I should just back off because there's no way to integrate anything I might bring to the problem (including red herrings), then let me know, and I'll switch to RMA mode and get the hell out of here. 

That would be a shame, because overall I am a huge fan of your devices and overall system design. I do not believe that there is any reason why they should not function on our house network, and that the fact that there are currently severe problems reflects some subtle details of the device's software implementation. I could be very wrong about this.
Comment 13 Alan Young 2009-04-29 00:08:43 UTC
My view is that the poor throughput and consequent poor maintenance of buffer fullness is almost certainly a direct consequence of the high rate of packet loss. 

From your description, you appear to have done all the right things to try to diagnose the problem. None of your tests seem to give a good indication of what is going on. But to be sure, the high rate of packet loss (anything over 2% really) is a clear indicator of a significant problem.

The fact that you have tried the full range of wireless channels, including while having asked you neighbours to disable their WiFi devices, is really puzzling.
Comment 14 Alan Young 2009-04-29 00:17:19 UTC
In comment #9 you make some observations about the behaviour of the buffer fullness while rebuffering. Were these observations made with the Receiver or SB2, or with SqueezePlay (either within the Controller or on a PC)?

SqueezePlay does indeed consume the decode buffer while paused (rebuffering). In this case you should see the fullness of the output buffer rise, while the decode buffer fullness is fluctuating. The rebuffering management algorithm in SqueezeCenter has not yet been adjusted to take this into account. Overall however, the effect should be simply to increase the period of audio that has been buffered before playback resumes.
Comment 15 Paul Davis 2009-04-29 06:08:33 UTC
the buffering observations were made using a Receiver as the playback device. 

i notice from the server log that the (playback) threshold set by SC is 255kB, whereas the buffer size is reported as 3MB. is there a way to change this threshold from the SC settings interface? its not clear to me why you'd only fill the buffer to this level before starting playback. it also seems unaffected by the network latency setting, which i would have thought might play a role in determining its value.
Comment 16 Paul Davis 2009-04-30 19:50:48 UTC
This evening, I discovered how to set the playback bufferThreshold via the server.conf file. I increased it from 255 to 2867 (units of kB) and got a DRAMATIC improvement in the quality of service from my Receiver. There are still occasional dropouts, but they are now rare. I have sat watching the server log and the behaviour is what I previously guessed it would be, though with more variation.

The decode/input buffer fullness now fluctuates between 100% and 40%, rather than 0% and 4%. The output buffer fullness now climbs rapidly to between 90% and 100^ and then stays there almost all the time. 

I notice major variations in signal strength seen by the Receiver. It will drop to 10%, and when this happens the decode/input buffer will slowly empty. But then, most of the time, it rises back up to about 45-50% and the buffer will refill back toward 95% or higher. Its not perfect - I've seen two instances where communication between SC and the receiver appeared to simply halt for a substantial period of time, and eventually the "Rebuffering ..." message shows up.

Another interesting observation - not sure what it means at all - the (duet) Controller has reset at least twice in the last couple of minutes (while music continues to play and buffer levels stay excellent. 

Tomorrow I will see if this behaviour applies to the SB3, and try experimenting with the bufferThreshold value to see how low it can go before reversing the basically-working behaviour I have now.
Comment 17 Alan Young 2009-04-30 22:28:15 UTC
I cannot see how that can real have made the difference. The protocol field that carries this element is an unsigned byte with a maximum value of 255. Trying to set it to 2867 will result in an actual value of 51.
Comment 18 Paul Davis 2009-05-01 04:18:08 UTC
I don't know whether this message comes from SC or is a status message back from the player:

[09-05-01 07:12:52.6983] Slim::Player::Squeezebox::stream_s (821) Starting with decoder with format: m autostart: 1 threshold: 2867 samplesize: ? samplerate: ? endian: ? channels: ?

But, sadly, I lied :) I had forgotten that I had earlier in the day set the bitrate to 128kbps. It appears that neither of these two steps makes enough difference - unlimiting the bitrate causes things to fall apart again; just limiting it isn't enough to get stable playback. I managed to use a limit of 160kbps successfully.
Comment 19 Paul Davis 2009-05-01 04:31:30 UTC
i should have said 'yes, i can see that', regarding its 8-bit limit. hmm. i guess the bitrate limiting must be the only factor. sigh.
Comment 20 Paul Davis 2009-06-26 19:47:36 UTC
sigh. nothing ever resolved. this week, i picked up a new wifi router. problems all "solved". of course, this doesn't address why our laptops and mac mini could get adequate bandwidth via the old router and the s'boxen couldn't but .. at this point its all a bit academic. i guess that while we were in germany, the wifi performance of our router degraded and the replacement router is working correctly.

thanks for paying attention.
Comment 21 James Richardson 2009-10-30 13:59:24 UTC
Closing this bug based on latest customer feedback