I believe all ISPs do provide the MRTG chart for customer. If yes, it will be able to tell whether the bandwidth usage of CS at 10pm - 12am is saturated.
If it is saturated, it'll be clearly a bandwidth issues. If not, the CPU and Memory resource usage at the server will indicate whether it is a hardware limitation. I believe you've already checked since you mentioned that Server is not the cause.
Traceroute to your site from various ISPs is pretty healthy, with that I think it is left with only the Application that I think it may need more fine tunning. Thanks.