There are many potential factors that may slow down the response time of your server. Some causes could be slow application logic, slow database queries, slow routing, frameworks, libraries, resource CPU starvation or memory starvation.
If you believe your slow response time is due to CPU or memory, you may think about upgrading your web server. It is important to reduce your server response time to less than 200ms.