Bug 17791 - Large libraries have severe performance problem.
: Large libraries have severe performance problem.
Status: RESOLVED FIXED
Product: Logitech Media Server
Classification: Unclassified
Component: Streaming From SlimServer
: 7.7.1
: PC Linux (other)
: -- major with 3 votes (vote)
: 7.9.x
Assigned To: Michael Herger
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-25 16:29 UTC by msux
Modified: 2014-10-13 21:51 UTC (History)
1 user (show)

See Also:
Category: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description msux 2011-11-25 16:29:27 UTC
I have a very large library, 350K+ tracks. Ever since 7.6, the server freezes under some usages:
  - My Music-->New Music (almost all the time)
  - My Music-->Albums, Artists. (seen sometimes).

The web page will attempt to connect and wait (2-3 minutes). During this time the connected player will continue to play until emptying their buffers, then go silent. The display will say 'Can't connect to server. Left to go back, right to try again.' I leave it be, and it will reconnect in 2-3 minutes and be ok after that.

This was working perfectly in 7.5.X, and was broken in 7.6 and 7.7. (When Mysql support was dropped.....)


Logitech Media Server Version: 7.7.0 - r33614 @ Tue Oct 18 11:07:25 PDT 2011
Hostname: XXXXredactedXXXX
Server IP Address: 192.168.1.110
Server HTTP Port Number: 9000
Operating system: Debian - EN - utf8
Platform Architecture: x86_64-linux
Perl Version: 5.10.1 - x86_64-linux-gnu-thread-multi
Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1)
Total Players Recognized: 6

No images, No videos.

 

Ubuntu 11.10 - 64bit.
Comment 1 Jim McAtee 2011-11-25 22:50:32 UTC
Are you already using the 'High' setting under Settings > Performance > Database Memory Config? Maybe an enhancement request for additional 'Very High' memory settings would be worthwhile, although I'm not sure how much difference the existing High setting makes.

At the current time you can still run MySQL in 7.7, although it's not officially supported. You must run your own instance of MySQL server and create an empty database before doing the following:

1. Update your LMS installation to a recent 7.7.1 nightly. It contains some missing database definition scripts for MySQL.

2. Manually install DBD::mysql in your Perl installation.

3. With the server stopped, edit the server.prefs file, changing the database settings to include the following:

dbpassword: <your_MySQL_LMS_user_password>
dbsource: dbi:mysql:hostname=127.0.0.1;port=3306;database=<your_MySQL_LMS_database>
dbtype: MySQL
dbusername: <your_MySQL_LMS_user_name>

If the MySQL server is running on another computer, enter that computer's IP address instead of 127.0.0.1. The 'dbtype' setting is new to 7.6/7.7 and must be capitalized exactly as shown.

4. Restart the server.
Comment 2 Jim McAtee 2011-11-25 23:07:22 UTC
(In reply to comment #1)
> dbpassword: <your_MySQL_LMS_user_password>
> dbsource:
> dbi:mysql:hostname=127.0.0.1;port=3306;database=<your_MySQL_LMS_database>
> dbtype: MySQL
> dbusername: <your_MySQL_LMS_user_name>

Note that the "dbsouce: dbi:mysql:hostname=..." entry should on a single line in the server.prefs file.
Comment 3 msux 2011-11-28 15:48:51 UTC
Thanks Jim,

Yes I am using the 'High' setting. 

I'll try the running MySQL, but would much prefer an official solution.
Comment 4 Dennis Mutsaers 2011-12-06 12:14:23 UTC
I experience the same behaviour, but on windows with a 45K+ library. Sometimes it happens after a search, sometimes when changing tracks. Can't find the cause.
Comment 5 msux 2012-01-01 01:54:47 UTC
I ran strace when the hang occurs (This is just a small sample, it will go on for 2-3 minutes:

read(16, "\r\0\0\0\2\1 \0\2\221\1 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(16, 4810752, SEEK_SET)            = 4810752
read(16, "\r\0\0\0\2\1*\0\2\211\1*\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(16, 269996032, SEEK_SET)          = 269996032
read(16, "\r\0\0\0\2\0\360\0\2s\0\360\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(16, 192765952, SEEK_SET)          = 192765952
read(16, "\r\0\0\0\2\0\366\0\2}\0\366\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(16, 457278464, SEEK_SET)          = 457278464
read(16, "\r\0\0\0\2\0\340\0\2n\0\340\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(16, 286321664, SEEK_SET)          = 286321664
read(16, "\r\0\0\0\2\0|\0\2L\0|\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(16, 298785792, SEEK_SET)          = 298785792
read(16, "\r\0\0\0\2\0\322\0\2J\0\322\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024
lseek(16, 297192448, SEEK_SET)          = 297192448

root@MusicServer:~# ls -l /proc/23649/fd/16
lrwx------ 1 squeezeboxserver nogroup 64 2012-01-01 01:49 /proc/23649/fd/16 -> /var/lib/squeezeboxserver/cache/library.db
Comment 6 msux 2012-01-30 17:03:27 UTC
I finally got a chance to try this.

I'm running Suse 12.1 - 64-bit

/etc/squeezeboxserver/server.conf 
is a symbolic link to 
/var/lib/squeezeboxserver/prefs/server.prefs

I stop server (/etc/init.d/squeezeboxserver stop), and edit the prefs file, changing:
bpassword: ''
dbsource: dbi:SQLite:dbname=/var/lib/squeezeboxserver/cache/library.db
dbusername: slimserver

To:
dbpassword: ___MYPASSEWORD____
dbsource:
dbi:mysql:hostname=127.0.0.1;port=3306;database=slimserver
dbtype: MySQL
dbusername: slimserver

I've created a MySQL db:
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| slimserver         |
| test               |
+--------------------+
5 rows in set (0.00 sec)

mysql> use slimserver;
Database changed
mysql> show tables;
Empty set (0.00 sec)
---------------------------------------------
It's there & empty.

Now I start server...and wait a bit.
when I look at the prefs its overwritten the db* items with the original sqlite versions:
dbpassword: ''
dbsource: dbi:SQLite:dbname=/var/lib/squeezeboxserver/cache/library.db
dbusername: slimserver

Back on the webpage, after the normal setup (mysqueezebox.com, entering music & playlist dirs, All tracks are there immediately, I'm assuming its using the old (sqlite db).

I ran strace and the server writes to server.prefs.tmp and then move server.prefs.tmp to server.prefs. 

Any idea what I'm doing wrong??
Comment 7 msux 2012-01-30 17:11:39 UTC
One more thing:
I have perl-DBD-mysql installed.
Comment 8 msux 2012-01-30 17:18:51 UTC
Never mind...

I missed the comment about dbi: not being on a separate line. Fixed this & now I'm getting:
 12-01-30 17:16:31.8822] Carp::Clan::__ANON__ (214) Warning: Carp::Clan::__ANON__(): Carp::Clan::__ANON__(): DBI Connection failed: DBI connect('hostname=127.0.0.1;port=3306;database=slimserver','slimserver',...) failed: Access denied for user 'slimserver'@'localhost' (using password: YES) at /usr/share/squeezeboxserver/CPAN/DBIx/Class/Storage/DBI.pm line 999

pretty sure I can track this down...
Comment 9 msux 2012-02-06 17:42:11 UTC
Well I got the server working with mysql & hit 2 problems.

1) Scan took 50 hours (vs ~10 for 7.5.x).
2) 'look for new & changed' rescan causes each track to be listed twice.

I'm going back to 7.5.6 for now.

I am requesting this serious regression is fixed.
Comment 10 Yuly Milner 2012-06-02 00:22:30 UTC
*** This bug has been confirmed by popular vote. ***
Comment 11 Michael Herger 2014-08-06 10:40:13 UTC
Performance with large collection should be vastly improved in 7.9. Make sure you're setting the performance/memory preference to maximum. 

The bottle neck with large collections often was disk I/O, as sqlite had to read indices from disk during queries. Giving it more memory to keep that data in RAM greatly improves performance. Please give it a try.
Comment 12 msux 2014-10-13 21:51:45 UTC
Works like a dream. Squeezboxes are a pleasure to use again.

Thanks for all your hard work.