Feature/GitHub AKS Provisioning

documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc

@@ -1,89 +1,8 @@
-= Setting up a Azure AKS provisioning pipeline on Azure DevOps
-
-In this section we will create a pipeline which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix to retrieve the DNS name of the Ingress controller independently.
-
-The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the pipeline and the terraform files for creating the cluster will be pushed to it.
-
-Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag.
-
-The script located at `/scripts/pipelines/azure-devops/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch.
-
-== Prerequisites
-
-* Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.
-* Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.
-* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command:
-
-```
-az group create -n <resource group name> -l <location>
-```
-
-* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command:
-
-```
-az storage account create -n <storage account name> -g <resource group name> -l <location>
-```
-
-* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command:
-
-```
-az storage container create -n <storage container name> --account-name <storage account name>
-```
-
-* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`).
-
-== Creating the pipeline using provided script
-
-Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example:
-
-```
-./set-terraform-variables.sh --cluster_name <cluster name> --location <location> --resource_group_name <resource group name> --instance_type <worker instance type> --worker_node_count <number of worker nodes> --dns_prefix <dns prefix>
-```
-
-=== Usage
-```
-pipeline_generator.sh \
-  -c <config file path> \
-  -n <pipeline name> \
-  -d <project local path> \
-  --resource-group <resource group name> \
-  --storage-account <storage account name> \
-  --storage-container <storage container name> \
-  [-b <branch>] \
-  [-w]
-```
-
-NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg`.
-
-=== Flags
-```
--c, --config-file        [Required] Configuration file containing pipeline definition.
--n, --pipeline-name      [Required] Name that will be set to the pipeline.
--d, --local-directory    [Required] Local directory of your project (the path should always be using '/' and not '\').
-    --resource-group     [Required] Name of the resource group for the cluster.
-    --storage--account   [Required] Name of the storage account for the cluster.
-    --storage-container  [Required] Name of the storage container where the Terraform state of the cluster will be stored.
--b, --target-branch                 Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided.
--w                                  Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag.
-```
-
-=== Example
-
-```
-./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state -b develop -w
-```
-
-=== Appendix: Interacting with the cluster
-
-NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed.
-
-In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either:
-
-```
-# via environment variable (you can add this on your profile)
-export KUBECONFIG=<path to downloaded kubeconfig>
-kubectl <command>
-
-# via command-line flag
-kubectl <command> --kubeconfig=<path to downloaded kubeconfig>
-```
+:provider: Azure Devops
+:pipeline_type: Pipeline
+:trigger_sentence:  * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.
+:pipeline_type2: pipeline
+:path_provider: azure-devops
+:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w
+:aks_variables: Azure DevOps > Pipelines > Library > `aks-variables`
+Unresolved directive in setup-aks-provisioning-pipeline.asciidoc - include::../common_templates/setup-build-pipeline.asciidoc[]

documentation/github/setup-aks-provisioning-pipeline.asciidoc

@@ -0,0 +1,8 @@
+:provider: Github
+:pipeline_type: workflow
+:trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly.
+:pipeline_type2: Github action
+:path_provider: github
+:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w
+:aks_variables: .github > vars > `aks-variables.env`
+Unresolved directive in setup-aks-provisioning-pipeline.asciidoc - include::../common_templates/setup-build-pipeline.asciidoc[]

documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc

@@ -0,0 +1,8 @@
+:provider: Azure Devops
+:pipeline_type: Pipeline
+:trigger_sentence:  * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.
+:pipeline_type2: pipeline
+:path_provider: azure-devops
+:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w
+:aks_variables: Azure DevOps > Pipelines > Library > `aks-variables`
+include::../common_templates/setup-build-pipeline.asciidoc[]

documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc

@@ -0,0 +1,105 @@
+= Setting up a Azure AKS provisioning {pipeline_type} on {provider} 
+
+In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details.
+
+The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it.
+
+Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag.
+
+The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch.
+
+== Prerequisites
+
+{trigger_sentence}
+
+* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command:
+
+```
+az group create -n <resource group name> -l <location>
+```
+
+* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command:
+
+```
+az storage account create -n <storage account name> -g <resource group name> -l <location>
+```
+
+* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command:
+
+```
+az storage container create -n <storage container name> --account-name <storage account name>
+```
+
+* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`).
+
+== Creating the {pipeline_type} using provided script
+
+Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example:
+
+```
+./set-terraform-variables.sh --cluster_name <cluster name> --location <location> --resource_group_name <resource group name> --instance_type <worker instance type> --worker_node_count <number of worker nodes> --dns_prefix <dns prefix>
+```
+
+=== Usage
+```
+pipeline_generator.sh \
+  -c <config file path> \
+  -n <pipeline name> \
+  -d <project local path> \
+  --resource-group <resource group name> \
+  --storage-account <storage account name> \
+  --storage-container <storage container name> \
+  [--rancher] \
+  [-b <branch>] \
+  [-w]
+```
+
+NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`.
+
+=== Flags
+```
+-c, --config-file        [Required] Configuration file containing pipeline definition.
+-n, --pipeline-name      [Required] Name that will be set to the pipeline.
+-d, --local-directory    [Required] Local directory of your project (the path should always be using '/' and not '\').
+    --resource-group     [Required] Name of the resource group for the cluster.
+    --storage--account   [Required] Name of the storage account for the cluster.
+    --storage-container  [Required] Name of the storage container where the Terraform state of the cluster will be stored.
+    --rancher                       Install Rancher to manage the cluster.
+-b, --target-branch                 Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided.
+-w                                  Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag.
+```
+
+=== Example
+
+```
+{cmd_add}
+```
+
+NOTE: Rancher is installed on the cluster after provisioning when using the above command.
+
+=== Appendix: Interacting with the cluster
+
+NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed.
+
+In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either:
+
+```
+# via environment variable (you can add this on your profile)
+export KUBECONFIG=<path to downloaded kubeconfig>
+kubectl <command>
+
+# via command-line flag
+kubectl <command> --kubeconfig=<path to downloaded kubeconfig>
+```
+
+To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}.
+
+Rancher, if installed, will be available on `https://<ingress controller domain>/dashboard`. You will be asked for an initial password, which can be retrieved with:
+
+```
+kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'
+```
+
+=== Appendix: Destroying the cluster
+
+To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline.

documentation/src/github/setup-aks-provisioning-pipeline.asciidoc

@@ -0,0 +1,8 @@
+:provider: Github
+:pipeline_type: workflow
+:trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly.
+:pipeline_type2: Github action
+:path_provider: github
+:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w
+:aks_variables: .github > vars > `aks-variables.env`
+include::../common_templates/setup-build-pipeline.asciidoc[]  

scripts/environment-provisioning/azure/aks/terraform.tfvars

@@ -1,8 +1,8 @@
-cluster_name = "hangar-k8s"
+#cluster_name = "hangar-k8s"
 
 location = "westeurope"
 
-resource_group_name = "hangar-aks-rg"
+resource_group_name = "az-devops-rg"
 
 instance_type = "Standard_DS2_v2"
 

scripts/pipelines/github/pipeline_generator.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 set -e
-FLAGS=$(getopt -a --options c:n:d:a:b:l:i:u:p:hw --long "config-file:,pipeline-name:,local-directory:,artifact-path:,target-branch:,language:,build-pipeline-name:,sonar-url:,sonar-token:,image-name:,registry-user:,registry-password:,resource-group:,storage-account:,storage-container:,cluster-name:,s3-bucket:,s3-key-path:,quality-pipeline-name:,dockerfile:,test-pipeline-name:,aws-access-key:,aws-secret-access-key:,aws-region:,help" -- "$@")
+FLAGS=$(getopt -a --options c:n:d:a:b:l:i:u:p:hw --long "config-file:,pipeline-name:,local-directory:,artifact-path:,target-branch:,language:,build-pipeline-name:,sonar-url:,sonar-token:,image-name:,registry-user:,registry-password:,resource-group:,storage-account:,storage-container:,cluster-name:,s3-bucket:,s3-key-path:,quality-pipeline-name:,dockerfile:,test-pipeline-name:,aws-access-key:,aws-secret-access-key:,aws-region:,help,rancher" -- "$@")
 
 eval set -- "$FLAGS"
 while true; do
@@ -20,6 +20,7 @@ while true; do
         --resource-group)         resourceGroupName=$2; shift 2;;
         --storage-account)        storageAccountName=$2; shift 2;;
         --storage-container)      storageContainerName=$2; shift 2;;
+        --rancher)                installRancher="true"; shift 1;;
         --cluster-name)           clusterName=$2; shift 2;;
         --s3-bucket)              s3Bucket=$2; shift 2;;
         --s3-key-path)            s3KeyPath=$2; shift 2;;

scripts/pipelines/github/templates/aks/aks-pipeline.cfg

@@ -0,0 +1,61 @@
+# Mandatory flags.
+mandatoryFalgs="$pipelineName,$localDirectory,$clusterName,$resourceGroupName,$storageAccountName,$storageContainerName,"
+# Path to the templates.
+templatesPath="scripts/pipelines/github/templates/aks"
+#Path to common kubernetes templates.
+commonKubernetesPath="scripts/pipelines/github/templates/common/kubernetes"
+# YAML file name.
+yamlFile="aks-provisioning.yml"
+# Source branch.
+sourceBranch="feature/aks-provisioning"
+# Path to terraform templates.
+terraformTemplatesPath="scripts/environment-provisioning/azure/aks"
+# Path to terraform scripts.
+terraformPath=".terraform/aks"
+# Default cluster operation.
+operation="create"
+# Install Rancher on AKS cluster.
+if test -z $installRancher
+then
+    installRancher=false
+fi
+
+# Function that copies the necessary scripts into the directory.
+function copyScript {
+    # Create .terraform/aks folder if it does not exist.
+    mkdir -p "${localDirectory}/${terraformPath}"
+
+    # Copy the terraform files.
+    cd "${hangarPath}/${terraformTemplatesPath}"
+    cp * "${localDirectory}/${terraformPath}"
+
+    # Copy the script for the DNS name into the directory.
+    cp "${hangarPath}/${templatesPath}/obtain-dns.sh" "${localDirectory}/${scriptFilePath}/obtain-dns.sh"
+
+    # Copy the common files for kubernetes
+    cp "${hangarPath}/${commonKubernetesPath}"/*.sh "${localDirectory}/${scriptFilePath}"
+}
+
+function commitFiles {
+    # Add the terraform files.
+    git add .terraform -f
+
+    # Changing all files to be executable.
+    find .terraform -type f -name '*.sh' -exec git update-index --chmod=+x {} \;
+
+    # Git commit and push it into the repository.
+    git commit -m "Adding the terraform files"
+    git push -u origin ${sourceBranch}
+}
+
+# Function that adds the variables to be used in the pipeline.
+function addPipelineVariables {
+
+    export clusterName
+    export resourceGroupName
+    export storageAccountName
+    export storageContainerName
+    export installRancher
+    export operation
+    specificEnvSubstList='${clusterName} ${resourceGroupName} ${storageAccountName} ${storageContainerName} ${installRancher} ${operation}'
+}