Bugzilla – Bug 7455
Get existing stats working on new racks
Last modified: 2008-12-15 12:16:43 UTC
The basic plan is: (1) Get rid of event_log_archive (unused) (2) Move event_stats to a separate (unreplicated) database on dbslave (3) Alter the stats script to not modify event_log, and to use the moved event_stats (4) Get the web UI for stats up on dbslave (either that, or put a model in the normal admin UI that knows to access the data via dbslave - we have the tunnels for this). (5) Cron job on the regular racks to periodically delete outdated event_log entries. (need to process the backlogged events first).
New plan: (1) Get rid of event_log_archive (2) Have the stats script make two connections, one to the local slave, and another over the inter-site tunnels to one of the prod datacenters (should fail down the list until it gets a live connection). (3) Process data from the local slave, and do the event_stats and the deletions on the remote db.
The plan was further simplified to just run the thing on meta.sv with only minimal changes to port it to that environment. Right now we're stalled waiting on a restoration of old stats data from files that existed on db02.sfo before it was pulled from the SFO datacenter. Matt says he hasn't overwritten anything on those yet, just needs to get them powered up so we can pull the data.
The old data was finally imported (nearly 30GB), and the backlogged stats for Feb 29 -> now were run last night. The cron has been set up again to keep things up to date once an hour. We'll need to keep an eye on the disk situation. We'll probably run out of room under the current scheme sometime in the early part of 2009. We could compress the logfiles (and upgrade the script to use compressed logfiles) to solve this problem for a few more years, but odds are high we'll migrate away from this stats solution by then anyways.
This bug has been fixed in the latest release of SqueezeNetwork! If you are still experiencing this problem, feel free to reopen the bug with your new comments and we'll have another look.