Enable NGINX App Protect as Ingress for Azure Kubernetes Services (AKS)

Table of Contents

• Enable NGINX App Protect as Ingress for Azure Kubernetes Services (AKS)
• Introduction
  • Use Case
  • Benefit
  • Architecture
    • Functional view
    • Product
    • Network view
    • Kubernetes view
    • WAF policy structure
  • Demo
    • A) [SecOps] Deploy Ingress Controller
    • B) [DevOps] Publish an Application
    • C) [DevOps] Update App's Security level
    • D) [SecOps] Update WAF policy attached to a security level
    • E) [SecOps] Fix false positive
    • F) [DevOps] Secure published API
• Pre-requisites
  • Ansible Tower
    • virtualenv
    • Helm
    • Project
    • Credential
  • Ansible role structure
• 0) [DevOps] Deploy AKS infrastructure
• A) [SecOps] Deploy Ingress Controller
  • Pre-requisites
    • NGINX licence
    • AKS - kubeconfig
    • ACR - token
  • Workflow
• B) [DevOps] Publish an Application
  • Pre-requisites
    • ACR - token
  • Workflow
• C) [DevOps] Update App's Security level
• D) [SecOps] Update WAF policy attached to a security level
  • Workflow
    • Webhook
• E) [SecOps] Fix false positive
• F) [DevOps] Secure published API
• G) [SecOps] Update WAF signatures
• Troubleshoot
• Reference

Introduction

Use Case

• Security -- Publish and Secure your applications with a Web Application Firewall (WAF)
• Organization -- Delegate App publication to DevOps and WAF policies to SecOps
• Automation -- Automate use cases:

1. [DevOps/SecOps] Deploy Ingress Controller -- DevOps deploy K8S Ingress with an embedded WAF and wildcard certificate
2. [DevOps] Publish an Application -- DevOps deploy Applications on Ingress (Cache, Content Routing, LB with monitoring) and by selecting a pre-defined security level
3. [DevOps] Update App's Security level -- Regarding risk analysis and new App's features, Product Owner requests an update of App's Security level
4. [SecOps] Update WAF policy attached to a security level -- Threats evolve, SecOps adapt WAF strategy to protect capital assets: applications
5. [SecOps] Fix false positive -- SecOps modify an application's WAF policy to fix False Positives that impact User Experience
6. [DevOps] Secure published API -- Limit the attack surface by allow access to only compliant API call. DevOps regularly update App's OpenAPI specifications.
7. [SecOps] Update WAF signatures -- SecOps do rolling upgrade of NGINX Ingress Controller images with up to date protection engine and signatures.

Benefit

• Security -- protect globally from threats and manage False Positive per Application to reach Service Level Objective and so satisfy User Experience
• Organization -- In a blameless culture, clear responsibility between DevOps and SecOps creates strong collaboration
• Security Service Catalog -- Managed Security objects by SecOps are ready to be consumed by DevOps
• Automation -- Asynchronous deployment operations between DevOps and SecOps does not delayed app's time to market

Architecture

Functional view

In this demo, functional components in the data path between Application code and consumers are:

• GSLB -- Make fastest DNS resolution and redirect users to closest App's Point of Presence. DNS anti-DDoS included.
• WAF -- Adapt asset (App's component or micro-service) protection regarding its value and threat risk. Protection features are:
• +-- Reduce attack surface: Publish the strict necessary of API apps using up to date openAPI spec file (swagger)
• +-- Virtual Patching: block exploitation of vulnerabilities (CVE) on underlying App's technologies / framework (Apache, Django, Postgre SQL...)
• +-- Weakness in code: Following awareness of OWASP foundation for Web app and API app, raise protection to prevent from hacking actions
• +-- Threat Campaign: Because patient zero are honey pots, obtain associated attack signatures against 0-day from Application Threat Intelligence
• K8S Ingress / Content routing -- Cache HTTP objects, micro-cache, route based on URI, monitor service, load balance traffic to PODs and send metrics/security events to remote log collectors

Product

Products used for this demo:

• GSLB: F5 Cloud Services
• WAF: NGINX App Protect
• K8S Ingress / Content routing: NGINX Ingress Controller
• K8S: Azure Kubernetes Services (AKS)
• Image Registry: Azure Container Registry (ACR)
• WAF policy repository: GitHub
• Automation tool: Ansible Tower

Network view

• Azure External Load Balancer: L3 router that Load Balances and Destination NAT (Public > Private) traffic to Ingress
• NGINX Ingress Controller: L7 reverse-proxy that redirect + Load Balances traffic to PODs regarding its Content Routing policy

Kubernetes view

Administrative segmentation between DevOps and SecOps is done by associating k8s object to related owner namespace

WAF policy structure

• Security level: During Risk Analysis, Product Owner defines Security level needed for an App component, with SecOps or following a decision tree.
• WAF policy: Each App have a WAF policy that includes:
• +-- Core: protection required by Security level
• +-- Modification: deviation from Security level
• Core: A core policy includes:
• +-- Protection properties
• +-- External references: external file that specifies protection properties
• Modification: Contains a list of changes to express exceptions to the intended policy. These exceptions are usually the result of fixing false positive incidents and failures in tests applied to those policies. These changes are more frequent than the Core policy.

More details here.

Demo

A) [SecOps] Deploy Ingress Controller

B) [DevOps] Publish an Application

C) [DevOps] Update App's Security level

D) [SecOps] Update WAF policy attached to a security level

E) [SecOps] Fix false positive

F) [DevOps] Secure published API

Pre-requisites

Ansible Tower

virtualenv

• Create a virtualenv following this guide
• In virtualenv, as a prerequisite for Azure collection, install Azure SDK following this guide
• In virtualenv, as a prerequisite for K8S collection, install openshift==0.11.2 following this guide
• In virtualenv, fix an issue during openshift installation google package dependency:

$ vi /var/lib/awx/venv/myVirtualEnv/lib/python2.7/site-packages/google/__init__.py
$ <copy paste https://raw.githubusercontent.com/googleapis/google-auth-library-python/master/google/__init__.py>

Helm

Install Helm following this guide

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

Project

• Clone this repository to a private repo. A private repo is needed because a kubeconfig file will be store in playbooks/roles/poc-k8s/files
• Create a project following this guide

Credential

• Create a Service Principal on Azure following this guide
• Create a Microsoft Azure Resource Manager following this guide
• Create Credentials cred_jumphost for Jumphost tasks following this guide

REDENTIAL TYPE	USERNAME	SSH PRIVATE KEY	SIGNED SSH CERTIFICATE	PRIVILEGE ESCALATION METHOD
Machine	my_VM_admin_user	my_VM_admin_user_key	my_VM_admin_user_CRT	sudo

Ansible role structure

• Deployment is based on workflow template. Example: workflow template = wf-create_create_edge_security_inbound
• workflow template includes multiple job template. Example: job template = poc-azure_create_hub_edge_security_inbound
• job template have an associated playbook. Example: playbook = playbooks/poc-azure.yaml
• playbook launch a play in a role. Example: role = poc-azure

- hosts: localhost
  gather_facts: no
  roles:
    - role: poc-azure

• play is an extra variable named activity and set in each job template. Example: create_hub_edge_security_inbound
• The specified play (or activity) is launched by the main.yaml task located in the role tasks/main.yaml

- name: Run specified activity
  include_tasks: "{{ activity }}.yaml"
  when: activity is defined

• The specified play contains tasks to execute. Example: play=``create_hub_edge_security_inbound.yaml``

0) [DevOps] Deploy AKS infrastructure

Create and launch a workflow template wf-aks-create-infra that includes those Job templates in this order:

Job template	objective	playbook	activity	inventory	limit	credential
poc-azure_create-spoke-aks	Create Ressource Group and vNet	playbooks/poc-azure.yaml	create-spoke-aks			my_azure_credential
poc-aks_create-registry	Create ACR	playbooks/poc-aks.yaml	create-registry			my_azure_credential
poc-aks_create-cluster	Create AKS	playbooks/poc-aks.yaml	create-cluster			my_azure_credential
poc-azure_create-vm-jumphost	Create Jumphost	playbooks/poc-azure.yaml	create-vm-jumphost			my_azure_credential

Extra variable	Description	Example
extra_platform_name	name used for resource group, vNet...	aksdistrict
extra_location	Azure region	eastus2
extra_platform_tags	Object tags	environment=DMO project=CloudBuilderf5
extra_hub_name	used to create vNet peering with a HUB	HubInbound
extra_vnet_address_prefixes	vNet CIDR	10.13.0.0/16
extra_management_subnet_address_prefix	Management subnet that hosts juphost	10.13.0.0/24
extra_zone_subnet_address_prefix	K8S Nodes and PODs subnet ; Azure CNI used	10.13.1.0/24
extra_zone_name	K8S Nodes and PODs subnet ; Azure CNI used	cni-nodesandpods
extra_service_cidr	K8S internal service subnet	10.200.0.0/24
extra_dns_service_ip	K8S internal DNS service subnet	10.200.0.10
extra_k8s_version	K8S version	1.19.0
extra_admin_username	K8S admin user of jumphost	PawnedAdmin
extra_admin_ssh_crt	K8S public key of admin user	ssh-rsa ...
extra_app_vm_size	K8S VMSS / node VM size	Standard_DS1_v2
extra_sp_client_id	Service Principal / client ID	<UUID>>
extra_sp_client_secret	Service Principal / client Secret	...
extra_jumphost	properties of jumphost	dict, see below

extra_jumphost:
  name: jumphost
  vm_size: Standard_DS1_v2
  private_ip: 10.13.0.10
  acl_src_ips:
    - '10.0.0.0/8'
  ssh_crt: "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----"

A) [SecOps] Deploy Ingress Controller

Pre-requisites

NGINX licence

Download your NGINX+ licence files nginx-repo.crt and nginx-repo.key to your private repository /playbooks/roles/poc-k8s/files/

AKS - kubeconfig

• Connect to Azure console

$ az aks get-credentials --resource-group rg-<platform_name> --name CloudBuilder

• Download your kubeconfig file ~/.kube/config to your private repository /playbooks/roles/poc-k8s/files/config.yaml

ACR - token

• Connect to Azure console

$ az acr login --name cloudbuilder.azurecr.io --expose-token

• Get a repository accessToken to be authorized to push NGINX Controller image to ACR

Workflow

Create and launch a workflow template wf-k8s-create-ingress-controller that includes those Job templates in this order:

Job template	objective	playbook	activity	inventory	limit	credential
poc-aks_get-registry_info	Get login_server info	playbooks/poc-aks.yaml	get-registry_info			my_azure_credential
poc-azure_get-vm-jumphost	Get FQDN jumphost info	playbooks/poc-azure.yaml	get-vm-jumphost			my_azure_credential
poc-k8s-create_nginx_ic_image	Build and push NGINX IC + App Protect	playbooks/poc-k8s_jumphost.yaml	create_nginx_ic_image	localhost		cred_jumphost
poc-k8s-deploy_nginx_ic	Create or update Ingress container instances	playbooks/poc-k8s.yaml	deploy_nginx_ic	localhost

Extra variable	Description	Example
extra_platform_name	name used for resource group, vNet...	aksdistrict
extra_nginx_ic_version	NGINX Ingress Controller version	1.9.0
extra_ilb_ingress_ip	Azure ILB VIP for Internal Ingress	eastus2
extra_jumphost	properties of jumphost	dict, see below
extra_acr_token	ACR token	survey entry, text type
extra_wildcard_tls_crt	Default wildcard certificate	survey entry, textarea type
extra_wildcard_tls_key	Default wildcard private key	survey entry, textarea type

extra_jumphost:
  name: jumphost

B) [DevOps] Publish an Application

Pre-requisites

ACR - token

• Connect to Azure console

$ az acr login --name cloudbuilder.azurecr.io --expose-token

• Get a repository accessToken to be authorized to push NGINX Controller image to ACR

Workflow

Create and launch a workflow template wf-k8s-publish-app that includes those Job templates in this order:

Job template	objective	playbook	activity	inventory	limit	credential
poc-aks_get-registry_info	Get login_server info	playbooks/poc-aks.yaml	get-registry_info			my_azure_credential
poc-azure_get-vm-jumphost	Get FQDN jumphost info	playbooks/poc-azure.yaml	get-vm-jumphost			my_azure_credential
poc-k8s-create_app_image	Build and push micro-services images	playbooks/poc-k8s_jumphost.yaml	create_app_image	localhost		cred_jumphost
poc-k8s-deploy_app	Deploy App, Services and Ingress	playbooks/poc-k8s.yaml	deploy_app	localhost
poc-k8s-deploy_gslb	Deploy GSLB	playbooks/poc-k8s.yaml	deploy_gslb	localhost

Extra variable	Description	Example
extra_platform_name	name used for resource group, vNet...	aksdistrict
extra_elk	Security log collector	10.13.0.10
extra_app	App properties	dict, see below
extra_cs	F5 Cloud Services credentials	dict, see below
extra_jumphost	properties of jumphost	dict, see below
extra_acr_token	ACR token	survey, text type
extra_app_swagger_url	swagger file repo URI	survey, text type; 'none' == no API Security
extra_waf_policy_level	Security level	survey, multiple choice type: low, medium, high
extra_app_tls_crt	App SSL certificate	survey, textarea type
extra_app_tls_key	App SSL private key	survey, textarea type

extra_app:
  name: arcadia
  domain: f5app.dev
  gslb_location:
    - eu
  components:
    - name: main
      location: /
      source_image: 'https://gitlab.com/arcadia-application/main-app.git'
    - name: app2
      location: /api
      source_image: 'https://gitlab.com/arcadia-application/app2.git'
    - name: app3
      location: /app3
      source_image: 'https://gitlab.com/arcadia-application/app3.git'
    - name: backend
      location: /files
      source_image: 'https://gitlab.com/arcadia-application/back-end.git'

extra_cs:
  username: [email protected]
  password: ...
  hostname: api.cloudservices.f5.com
  api_version: v1

extra_jumphost:
  name: jumphost

C) [DevOps] Update App's Security level

Create and launch a workflow template wf-k8s-update_app_security that includes those Job templates in this order:

Job template	objective	playbook	activity	inventory	limit	credential
Confirm Security level?	Request approval from SecOps	playbooks/poc-k8s.yaml	deploy_app	localhost
poc-k8s-update_security	Update SSL Certificate and WAF policy level	playbooks/poc-k8s.yaml	update_security	localhost

Extra variable	Description	Example
extra_app	App properties	dict, see below
extra_app_swagger_url	swagger file repo URI	survey, text type; 'none' == no API Security
extra_waf_policy_level	Security level	survey, multiple choice type: low, medium, high

extra_app:
  name: arcadia
  domain: f5app.dev

D) [SecOps] Update WAF policy attached to a security level

Raise webhook after a pull request is done on WAF policies repository and launch automatically step (B).

Workflow

Create and launch a workflow template wf-k8s-fetch-waf-policies that includes those Job templates in this order:

Job template	objective	playbook	activity	inventory	limit	credential
poc-k8s-reload_ingress	Reload NGINX Ingress Controller	playbooks/poc-k8s.yaml	deploy_app	localhost

Webhook

• Clone WAF policies repository to a new repo
• Create a Webhook following this guide

E) [SecOps] Fix false positive

Raise webhook after a pull request is done on WAF policies repository, that's launched automatically step (D).

F) [DevOps] Secure published API

Execute step (C) setting extra_app_swagger_url value with https://raw.githubusercontent.com/nergalex/f5-nap-policies/master/policy/open-api-files/arcadia.f5app.dev.yaml

G) [SecOps] Update WAF signatures

Execute step (A). In real life, when a notification of update is received from F5 mailing list

Troubleshoot

Get External Ingress Controller PODs:

kubectl describe pod --namespace external-ingress-controller

View Ingress Controller status (Cache, Zones, Upstream servers) from Jumphost:

http://Pod_IP:8080/dashboard.html

Get error logs from an External Ingress Controller POD:

kubectl logs --namespace external-ingress-controller POD_name

Launch a shell on an External Ingress Controller POD:

kubectl exec --namespace external-ingress-controller -it POD_name sh

View WAF policies for App 'arcadia':

kubectl describe --namespace external-ingress-controller --selector 'app==arcadia' APPolicy

View App's Service:

kubectl get svc --namespace arcadia -owide

View App's Ingress:

kubectl get ingress --namespace arcadia -owide

Reference

• AWS NGINX eXperience
• F5 attack signature info
• F5 mailing list
• NGINX Ingress Controller - HELM
• NGINX Ingress Controller - snippet
• NGINX Ingress Controller - App Protect annotation
• NGINX Ingress Controller - Minions
• WAF policies repository