Secure DevOps Kit (AzSK) CICD Extensions - AzSKARMTemplateChecker task fails if template does not contain supported resources

[warner-aemo]

Title

Secure DevOps Kit (AzSK) CICD Extensions - AzSKARMTemplateChecker task fails if template does not contain supported resources

Description

The AzSKARMTemplateChecker does not have controls for every Azure resource and it is possible to create an ARM template consisting of resources for which the AzSKARMTemplateChecker does not support.

When using the AzSKARMTemplateChecker task in Azure Pipelines, the task will fail when it encounters such a template.

Environment

Azure Pipelines with Secure DevOps Kit (AzSK) CICD Extensions for Azure v3.1.7 (Latest)

Steps to reproduce

Given the following ARM template:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
		"accountName": {
			"type": "string",
			"metadata": {
				"description": "Cosmos DB account name, max length 44 characters"
			}
        },
		"databaseName": {
			"type": "string",
			"metadata": {
				"description": "The name for the database"
			}
		},
		"throughput": {
			"type": "int",
			"defaultValue": 400,
			"minValue": 400,
			"maxValue": 1000000,
			"metadata": {
				"description": "The throughput for the container"
			}			
        }
    },
    "variables": {
		"containerName": "my-app-data"
    },
    "resources": [
		{
			"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases",
			"name": "[concat(parameters('accountName'), '/', parameters('databaseName'))]",
			"apiVersion": "2020-03-01",
			"properties":{
				"resource":{
					"id": "[parameters('databaseName')]"
				}
			}
        },
        {
            "type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers",
            "apiVersion": "2020-03-01",
            "name": "[concat(parameters('accountName'), '/', parameters('databaseName'), '/', variables('containerName'))]",
            "dependsOn": [
                "[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', parameters('accountName'), parameters('databaseName'))]"
            ],
            "properties": {
                "resource": {
                    "id": "[variables('containerName')]",
                    "indexingPolicy": {
                        "indexingMode": "consistent",
                        "automatic": true,
                        "includedPaths": [
                            {
                                "path": "/*"
                            }
                        ],
                        "excludedPaths": [
                            {
                                "path": "/\"_etag\"/?"
                            }
                        ]
                    },
                    "partitionKey": {
                        "paths": [
                            "/id"
                        ],
                        "kind": "Hash"
                    },
                    "uniqueKeyPolicy": {
                        "uniqueKeys": []
                    },
                    "conflictResolutionPolicy": {
                        "mode": "LastWriterWins",
                        "conflictResolutionPath": "/_ts"
                    },
                    "defaultTtl": 86400
                },
				"options": { "throughput": "[parameters('throughput')]" }
            }
        }
    ]
}

Create an Azure pipeline with the following job:

- job: AzSK
    pool:
      vmImage: 'windows-latest'
    steps:
      - task: AzSKARMTemplateChecker@4
        inputs:
          ARMTemplateFilePath: 'arm-templates/cosmosdb-container.json'
          ARMTemplateParameterFilePath: 'arm-templates/cosmosdb-container-params.json'
          Recurse: false

Run the pipeline

Expected behavior

The AzSK ARM Template Checker task will pass and output warnings if the template contains resources that are not supported or for which there are no controls.

I expect this behaviour because the Get-AzSKARMTemplateSecurityStatus cmdlet returns sucessfully when run against exactly the same template:

PS C:\Users\jbloe\arm-templates> Get-AzSKARMTemplateSecurityStatus -ARMTemplatePath .\cosmosdb-container.json -ParameterFilePath .\cosmosdb-container-params.json
================================================================================
AzSK Version: 4.10.0
================================================================================
Method Name: Get-AzSKARMTemplateSecurityStatus
Input Parameters:
Key               Value
---               -----
ARMTemplatePath   .\cosmosdb-container.json
ParameterFilePath .\cosmosdb-container-params.json
================================================================================
================================================================================
Starting analysis: [FileName: C:\Users\jbloe\arm-templates\cosmosdb-container.json]
--------------------------------------------------------------------------------
No controls have been evaluated for file: C:\Users\jbloe\arm-templates\cosmosdb-container.json
================================================================================
Skipped file(s): 1
C:\Users\jbloe\arm-templates\cosmosdb-container.json
--------------------------------------------------------------------------------
One or more files were skipped during the scan.
Either the files are invalid as ARM templates or those resource types are currently not supported by this command.
Please verify the files and re-run the command.
For files that should not be included in the scan, you can use the '-ExcludeFiles' parameter.
--------------------------------------------------------------------------------
No controls have been evaluated for ARM Template(s).
--------------------------------------------------------------------------------
** Next steps **
Look at the individual control evaluation status in the CSV file.
        a) If the control has passed, no action is necessary.
        b) If the control has failed, look at the control evaluation detail in the CSV file (LineNumber, ExpectedValue, CurrentValue, etc.) and fix the issue.
        c) If the control status says 'Skipped', it means that you have chosen to skip certain controls using the '-SkipControlsFromFile' parameter.
For further details, refer: https://aka.ms/devopskit/cicd
--------------------------------------------------------------------------------
Status and detailed logs have been exported to: C:\Users\jbloe\AppData\Local\Microsoft\AzSKLogs\ARMChecker\20200626_180811
================================================================================
C:\Users\jbloe\AppData\Local\Microsoft\AzSKLogs\ARMChecker\20200626_180811
PS C:\Users\jbloe\arm-templates> $?
True

Actual behavior

The AzSK ARM Template Checker task fails with the following output:

2020-06-05T07:39:46.9218270Z ##[section]Starting: AzSKARMTemplateChecker
2020-06-05T07:39:46.9344996Z ==============================================================================
2020-06-05T07:39:46.9345303Z Task         : AzSK ARM Template Checker
2020-06-05T07:39:46.9345568Z Description  : Scan ARM templates for security issues using AzSK.
2020-06-05T07:39:46.9345793Z Version      : 4.0.3
2020-06-05T07:39:46.9346005Z Author       : Microsoft Corporation
2020-06-05T07:39:46.9346278Z Help         : [More Information](http://aka.ms/azskossdocs)
2020-06-05T07:39:46.9346580Z ==============================================================================
2020-06-05T07:39:59.2550089Z Installing Module AzSK...
2020-06-05T07:41:56.7820112Z WARNING: Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the 
2020-06-05T07:41:56.7844826Z same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can
2020-06-05T07:41:56.7845713Z  use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure 
2020-06-05T07:41:56.7846641Z Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found 
2020-06-05T07:41:56.7847240Z here: https://aka.ms/azps-migration-guide
2020-06-05T07:42:07.4726032Z Importing Az modules. This may take a while...
2020-06-05T07:42:12.2041118Z Successfully updated privacy settings.
2020-06-05T07:42:13.6929858Z Clearing AzSK session state...
2020-06-05T07:42:13.7379647Z Session state cleared.
2020-06-05T07:42:15.5509324Z ================================================================================
2020-06-05T07:42:15.5510114Z AzSK Version: 4.9.0 
2020-06-05T07:42:15.5511095Z ================================================================================
2020-06-05T07:42:15.5615097Z Method Name: Get-AzSKARMTemplateSecurityStatus 
2020-06-05T07:42:15.5615532Z Input Parameters: 
2020-06-05T07:42:15.5616414Z Key               Value                                                
2020-06-05T07:42:15.5616889Z ---               -----                                                
2020-06-05T07:42:15.5617412Z ARMTemplatePath   d:\a\1\s\arm-templates\cosmosdb-container.json       
2020-06-05T07:42:15.5617819Z ParameterFilePath d:\a\1\s\arm-templates\cosmosdb-container-params.json 
2020-06-05T07:42:15.5618384Z ================================================================================
2020-06-05T07:42:16.0363865Z ================================================================================
2020-06-05T07:42:16.0364534Z Starting analysis: [FileName: D:\a\1\s\arm-templates\cosmosdb-container.json] 
2020-06-05T07:42:16.0364966Z --------------------------------------------------------------------------------
2020-06-05T07:42:16.0467256Z No controls have been evaluated for file: D:\a\1\s\arm-templates\cosmosdb-container.json
2020-06-05T07:42:16.0550124Z ================================================================================
2020-06-05T07:42:16.0574430Z Skipped file(s): 1
2020-06-05T07:42:16.0623527Z D:\a\1\s\arm-templates\cosmosdb-container.json
2020-06-05T07:42:16.0739812Z --------------------------------------------------------------------------------
2020-06-05T07:42:16.0768478Z One or more files were skipped during the scan. 
2020-06-05T07:42:16.0773543Z Either the files are invalid as ARM templates or those resource types are currently not supported by this command.
2020-06-05T07:42:16.0777334Z Please verify the files and re-run the command. 
2020-06-05T07:42:16.0951626Z For files that should not be included in the scan, you can use the '-ExcludeFiles' parameter.
2020-06-05T07:42:16.0952282Z --------------------------------------------------------------------------------
2020-06-05T07:42:16.0952927Z No controls have been evaluated for ARM Template(s).
2020-06-05T07:42:16.0953482Z --------------------------------------------------------------------------------
2020-06-05T07:42:16.0959929Z ** Next steps **
2020-06-05T07:42:16.0962072Z Look at the individual control evaluation status in the CSV file.
2020-06-05T07:42:16.0962642Z         a) If the control has passed, no action is necessary.
2020-06-05T07:42:16.0963206Z         b) If the control has failed, look at the control evaluation detail in the CSV file (LineNumber, ExpectedValue, CurrentValue, etc.) and fix the issue.
2020-06-05T07:42:16.0963912Z         c) If the control status says 'Skipped', it means that you have chosen to skip certain controls using the '-SkipControlsFromFile' parameter.
2020-06-05T07:42:16.0988067Z For further details, refer: https://aka.ms/devopskit/cicd
2020-06-05T07:42:16.1014941Z --------------------------------------------------------------------------------
2020-06-05T07:42:16.1373135Z Status and detailed logs have been exported to: C:\Users\VssAdministrator\AppData\Local\Microsoft\AzSKLogs\ARMChecker\20200605_074215
2020-06-05T07:42:16.1379877Z ================================================================================
2020-06-05T07:42:19.4844802Z Cleaning logs from temp directory...
2020-06-05T07:42:19.6704859Z ##[error]No controls have been evaluated for ARM Template(s).
2020-06-05T07:42:19.7579710Z ##[section]Finishing: AzSKARMTemplateChecker

[ankrause]

This looks like a duplicate of #678 created about a year ago now and which hasn't received a response.

This issue of considering it a failure when evaluating 0 controls is present in both the ARM template checker as well as the SVT task in ADO. Both of them cause issues when trying to enforce the appropriate controls on Azure resources, both prior to creation (template) and post-creation (SVTs).

Even recently, a change in which policies were enforced caused at least one of my release pipelines to go from passing to failing because the single control evaluated by SVTs for Key Vault was removed (no longer evaluated) resulting in no controls to evaluate and a failing release task, despite no changes in either the template or resource.

[ganesh-msft]

You can use and set variable FailTaskIfNoControlsScanned to 'false' for the task to not fail even if no controls were scanned. Please refer https://github.com/azsk/DevOpsKit-docs/blob/master/03-Security-In-CICD/Readme.md#advanced-cicd-scanning-capabilities-1 for more details.