KeepAlive
Eventually tracked down the issue with the headers – they were being automatically set by Apache and had been set to have KeepAlive off (apparently that’s the default for Apache on RedHat, but default for Apache generally is KeepAlive on). Once we’d set this up correctly there aren’t loads of open connections left open on the server (the same connection is used for multiple requests), the performance has significantly improved.
The other area I’m looking at too try and speed things up is to replace the proxy library that I’m using which downloads the data from the MSG server. I’m currently using fsockopen and fread to download the data, but I’m looking to replace this with cURL – which is apparently much quicker….
