Ready for Cloud Migration (As-Is) or Scaling out on cloud .. beware it is not High Speed Motorway ..

Recently , I have been tasked with  application’s operational architecture simplification initiative . The application was already in cloud but running on single large instance and follows a typical 3 tier architecture pattern.

One of the motivations to remove the slowness and stop server restart issues and overall making the application scalable .

We approach the problem and got certain assurances that application is stateless and uses RDS capability to persist any transaction and meta-data storage needs.

OK!!! So we started our steps as follows

  1. We introduced load balancer and number of smaller instances .
  2. Application deployed to smaller instances without any change.
  3. Very simple changes (related to configuration as well)

 

After couple of hours of migration , we could see CPU utilization reaches 100 % on RDS side .

 

We investigated and found that there is a huge traffic demand hence the reason of that spike..

 

In production we change the RDS size .. to the higher and more powerful CPU & Memory utilization.

 

After couple of hours … the cpu utilization again reach 100 % on RDS side. … This time we were serious that there is something wrong ..

 

We dig deeper into the RDS logs (Enhanced Monitoring) and found that each thread is taking almost 5- 12 % of CPU . This is alarming and lead us to slow memory logs .

 

We could see there are Spring JPA based queries being run and most of the costly queries taking lot of time to release the threads .. hence the utilization…

 

Ultimately , we have to choke the funnel and stop 2 servers to find the sweet spot between performance and resource utilization.

 

Lesson learned … autoscaling or loadbalancer may do there bits but can’t provide the real benefits unless your application and each component of your application is scalable on their own ..