How can I reduce the TCP connection time for web resource requests at the OS level?

I'm experiencing an issue where almost every request for a resource on a web application has a 100ms - 500ms delay in the "connecting" phase as reported by firebug, which is the time it takes to establish a TCP connection.

The server, Nginx, is configured to use keep-alive, and the response headers indicate that keep-alive is on. Yet, every resource that loads on the page has a long "connecting" delay.

This system is not in production and I am the only user making requests against it. Are there any OS variables related to TCP optimization that I should check to find out why each request has a TCP connection delay, even though keep-alive is enabled?

The server is running Ubuntu 14.04 LTS.