Ready for Cloud Migration (As-Is) or Scaling out on cloud .. beware it is not High Speed Motorway ..
Recently , I have been tasked with application’s operational architecture simplification initiative . The application was already in cloud but running on single large instance and follows a typical 3 tier architecture pattern.
One of the motivations to remove the slowness and stop server restart issues and overall making the application scalable .
We approach the problem and got certain assurances that application is stateless and uses RDS capability to persist any transaction and meta-data storage needs.
OK!!! So we started our steps as follows
- We introduced load balancer and number of smaller instances .
- Application deployed to smaller instances without any change.
- Very simple changes (related to configuration as well)
After couple of hours of migration , we could see CPU utilization reaches 100 % on RDS side .
We investigated and found that there is a huge traffic demand hence the reason of that spike..
In production we change the RDS size .. to the higher and more powerful CPU & Memory utilization.
After couple of hours … the cpu utilization again reach 100 % on RDS side. … This time we were serious that there is something wrong ..
We dig deeper into the RDS logs (Enhanced Monitoring) and found that each thread is taking almost 5- 12 % of CPU . This is alarming and lead us to slow memory logs .
We could see there are Spring JPA based queries being run and most of the costly queries taking lot of time to release the threads .. hence the utilization…
Ultimately , we have to choke the funnel and stop 2 servers to find the sweet spot between performance and resource utilization.
Lesson learned … autoscaling or loadbalancer may do there bits but can’t provide the real benefits unless your application and each component of your application is scalable on their own ..