Empowering Backend Engineering Team

I am writing this blog to share the ideas and concepts which will empower the backend developer, devops engineer and QAs.

Terraform Presentation Slide

These ideas are not tied or limited to particular framework or libraries or cloud providers, instead these can be in many different ways using different tool sets which the team might already be familiar with.

To achieve these end goals, the backend application and devops toolsets should be complementing each other in various aspects.

I am going to list out the desired features I would like to have in my project / team under various categories.

I have also created the following sample repositories on github to showcase these ideas in a simplified way.

Sample Java Spring Boot application with Maven:

https://github.com/harishkannarao/MySpringBoot

Sample QA Acceptance tests with Gradle:

https://github.com/harishkannarao/gradle-qa-acceptance-tests

Sample Infrastructure as Code in AWS with Terraform:

https://github.com/harishkannarao/github-actions-aws-terraform

Application Development and Deployment Stage:

Application Pipelines:
  1. Generated or updated based on configuration files
  2. Run on isolated containers for consistent integration test runs
  3. Ability to install required dependency service (eg: database) to run isolated integration tests
  4. Deploy the application to different environments from development to production





Infrastructure Provisioning Stage:

Infrastructure as Code:
  1. Provision and maintain infrastructure through configuration files (eg: Terraform, Cloudformation)
  2. Parameterise the variable values and reuse the configuration templates

Infrastructure Pipelines:
  1. Generated or updated based on configuration files





QA Acceptance Tests Stage:

QA Pipelines:
  1. Generated or updated based on configuration files
  2. Run on isolated containers
  3. Run tests against different environments using properties files
  4. Capture screenshots during browser testing
  5. Capture request and response during api testing



Production Deployment Stage:

  1. Ability to quickly rollback during deployment failure

Post Production or Operational Stage:

Dashboard and Alerts:
  1. Total requests
  2. Success (2XX and 3XX) and error (4XX and 5XX) rate graphs
  3. Response time (average and 95th percentile) graph
  4. Application Cpu Utilisation graph
  5. Application Memory Utilisation graph
  6. Application running instances graph
  7. Database Cpu Utilisation graph
  8. Database Storage space availability graph




 Logging and Tracing:

  1. Distributed log tracing between micro-services (eg: Spring Cloud Sleuth or Zipkin)
  2. Correlation key between application access logs and error logs


 Connecting to instances and database:
  1. Ability to remotely monitor JVM
  2. Ability to SSH into the running instance
  3. Ability to connect to database

Comments

Post a Comment

Popular posts from this blog

JSON with curl and jq

This blog post captures some quick commands to deal with JSON from servers (Http APIs) using curl and jq command line tools About Jq: https://stedolan.github.io/jq/ Install jq: Mac: brew install jq Ubuntu or Debian: sudo apt-get install jq Sample JSON content: Assume the following JSON content will be returned from  URL http://api.example.com/some-json-content {     "user_name": "User 123",     "some_other_field": {         "sub_field": "sub_value"     },     "array_field": [         {             "field1": "value1",             "field2": "value2"         },         {             "field1": "value3",             "field2": "value4"         }     ] } UR...
Read more

Import self signed in Linux for Chrome / Chromium headless testing

Chrome / Chromium browser offers headless mode, which is quite useful to run tests via Selenium WebDriver in CI servers. However under some circumstance your test will hit a server with self signed certificate or invalid certificate or expired certificate. This may happen for developers trying to write tests against a server started on localhost with self signed certificates or testers trying to write tests against non-production servers with expired certificates (invalid certificates). Chrome / Chromium browser ignores the SSL errors when running in non-headless mode. However in CI servers, then it should be running in headless mode and currently we cannot ignore the SSL errors for tests accessing non-localhost websites. This blogs explains the steps for import a certificate into Linux's trust store, which will make the Chrome / Chromium browser to trust the self signed certificate.  Please do not apply these steps in production servers as it will make server vulnerable to ...
Read more

Colima - Drop In replacement for Docker Desktop for Mac and Linux

Colima ( Co ntainer on Li nux Ma chines) is an open source alternative and drop in replacement for Docker Desktop on Mac and Linux. https://github.com/abiosoft/colima It is simple to setup and run containers without the need for sudo or root access. Images can be pulled form both Docker hub ( https://hub.docker.com ) or Amazon's public registry ( https://gallery.ecr.aws ) Amazon's registry is more permissive or friendly with pull rates for both clients and non-clients of AWS. This article show the setup and basics of running containers on mac. Installation command for Linux can vary based on the distribution type, hence please refer to official documentation for up-to-date steps at  https://github.com/abiosoft/colima/blob/main/docs/INSTALL.md Installation on mac: brew install docker  brew install docker-buildx brew install docker-compose brew install jq brew install colima Setup docker compose as docker plugin mkdir -p $HOME/.docker/cli-plugins ln -sfn $(which docker-compose) ...
Read more