Scaling
Had a bit of a nightmare the last couple of days! When we moved the MSG Block over to the ‘live’ server, we started to find that things weren’t working very well – the Moodle site was incredibly slow, this obviously(!) turned out to be a problem with the MSG block slowing things down with the number of requests being made keeping peoples presence status up to date. And this was with relatively few users…
So I’ve had to make fairly big changes over the last day or so to the MSG block to reduce the amount of communication between the Moodle and MSG servers. The way I’ve got around this is to create a database table for the MSG block to cache users presence for a minute or so and only to update the presence when the cache has expired. The obvious downside of this is that user’s presence doesn’t appear quite so ‘live’ on the browser page. There’s also a kind of natural limit to how long the cache should be set to – as if it’s too long it defeats the point of having the live updates via AJAX and the presences only refreshed when the whole page is refreshed (and no AJAX). At the moment I have the cache set to 1 minute, which seems ok for now, but we may have to make this even longer if performance drops again.
I would have preferred to have kept this cache in memory rather than database table (which keeps needing to be read and updated), but apparently this is a bit tricky to do in PHP – see the posting I made on the Moodle forums at: http://moodle.org/mod/forum/discuss.php?d=54994
So if I understand this right, you popped MSG on the Moodle server, opened up a few clients, and it got upset?
How many concurrent MSG users were you testing against? how many hits per second?
Is Moodle going to scale *at all*?
Ho hum – I just got an amazon ec2 account if that’ll help?
Though I suppose not – I imagine the problem is with the Moodle code, rather than the server…?
tony
Hi Tony, you’re kinda right! We’ve not actually got MSG running on the same server as Moodle (only the integration block is on Moodle server), so all requests from browser to update the presence go from browser -> Moodle -> MSG -> Moodle -> browser – all due to the fact AJAX can’t communicate with service on different domain
So in the MSG-Moodle block we can’t point the browser directly at MSG.
Obviously means that lots of traffic going through Moodle that it didn’t really need to handle.
On my dev environment I’d got it set up to update the user presence every few seconds (to give the illusion of it being virtually instant!) and seemed to work fine with 7 or 8 users simulataneously on my dev machine. My plan was to use these setting on the staging server and then load test to see how much it could handle and adjust the update timings accordingly. However with these initial settings the staging server threw a wobbler with only 9 or 10 users (I’m slightly guessing at exact numbers here – but it certainly wasn’t many). So I’ve not bothered to load test yet until I had the caching on Moodle implemented – I guess they’re hoping for more than 10 simulatenous users on LabSpace website!!!
I’ve a feeling there might have been other issues at play – eg due to KeepAlive requests nad there were issues with virtual IP addresses and machines refering back to each other – didn’t quite understand it all myself (not being a networky person). But anyway I think having the presences cached is the probably the ‘right’ thing to do – even if it means the presences appear a bit slow to update.
Alex
Alex
“all due to the fact AJAX can’t communicate with service on different”
you could json it?
Or get a relay on the moodle domain? They wouldn’t begrudge you that, surely?
I was idling tonight drving home on the way to do the school run about whether comet (http://en.wikipedia.org/wiki/Comet_(programming)) would be better than ajax for keeping the presence indicator current?
I haven;t tried comet style things at all yet, so I’m still hazy to know pros/cons etc, though i do have a friend who I believe has used the athena library as part of the app that’s going naked at http://blog.sleevenotez.com/ …
Cheers Tony, the reason we didn’t go for JSON was that we’d have to make quite a lot of changes to the MSG service and given the time we’d available to get the first release out it wasn’t really possible – but once we’ve got the first release out of the way it’ll give us chance to look at some of these other methods. Effectively Moodle is acting as the relay, but it’s still an increased load on that web server (forwarding the requests back and forth).
The comet one you mention looks interesting and yes, as you say, looks like what we need – though the statement ‘some potential concerns regarding how well a web server implementing Comet can scale’ didn’t fill me with confidence that it’ll be the one for us!
I think with some of these things that it’s a case of just implementing it and testing out – I think it’s hard to predict which would actually be the best route.
Thanks again for your input – always good to get other suggestions & views
Alex