Restructuring deploy flow

Why is restructuring needed?

We need to increase the visibility of configuration and keep it more consolidated in a service directory, rather than having all pipelines managed from a huge file.

Configurable from files in service dir

Using a single config file to configure the pipelines, role and service variables for example: we could use multiple files, like pipelines.yaml, role.json and variables.yaml.

This can then on the PR be planned and applied thru something like Atlantis. (need to verify if atlantis can be configured for our use case) And be applied before merging the code that would then be built and deployed. Alternatively we could work with an automatically applied terraform.

Example if we were to combine all into 1 file.

pipelines:
  regions:
    eu-central-1:
      environments:
        - dev
        - qa
        - preprod
        - production
    us-east-1:
      environments:
        - dev
        - preprod
        - production

role:
  permissions:
    - Sid: ListBuckets
      Effect: Allow
      Action:
        - s3:GetBucketAcl
        - s3:GetBucketLocation
        - s3:ListBucket
      Resource:
        - arn:aws:s3:${region}::quinyx-data-access-layer-${env}-*
    
    - Sid: ReadWriteDeleteObjects
      Effect: Allow
      Action:
        - s3:DeleteObject
        - s3:GetObject
        - s3:PutObject
      Resource:
        - arn:aws:s3:${region}::quinyx-data-access-layer-${env}-*/*/*

variables:
  - regions:
      - eu-central-1
    environments:
      - dev
    key: API_KEY
    value: <<-GPG
      -----BEGIN PGP MESSAGE-----
    
      hQIMA9uaVQNQNfofAQ//YqJ4EQ+RTDTgAq+WrRqlPkVFO+SQZN4MNYUK65G9JCaB
      505OcTMe2k9RySidpIEQAHDGimcMsLDpN73de5RzqNtu90JFYbcNbmn7cIyfJljS
      9H4xVCUKAN+PCzYPPhH/LM8H4KzVUNCjgZE4tZK+u2qR
      =106i
      -----END PGP MESSAGE-----
      GPG
    

Use the advantages of Helm charts

Instead of needing a manifest file for every service containing a lot of configuration, we should use sensible defaults and allow teams to tweak variables fed into those templates instead.

For a lot of service that means we could limit it to mostly the env vars and resource allocations.

          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 512Mi