Multipart/form-data with File upload does not work with Pulumi

[frank-bee]

Describe what happened

How to reproduce

1. single-step API test with multipart/form-data manually with the datadog GUI ( with file upload !)
2. import it to a pulumi stack
3. run pulumi up
   --> see error below

Sample program

Log output

Error:

• error updating synthetics API test from /api/v1/synthetics/tests/api/7z6-tpu-dcr: 400 Bad Request: {"errors":["'request' value '{'allow_insecure': True, 'body': '', 'bodyType': 'multipart/form-data', 'follow_redirects': False, 'headers': {'Authorization': '*** BPC_PROD_API_TETRLB_KEY }}', 'content-type': 'multipart/form-data; boundary="DatadogSyntheticsFiles"'}, 'httpVersion': 'any', 'method': 'POST', 'noSavingResponseBody': False, 'persistCookies': False, 'timeout': 0, 'url': '{{ BASEURL }}/namespaces/{{ NAMESPACE_ID }}/configs/{{ CONFIG_ID }}/files'}' is invalid"]}

Affected Resource(s)

No response

Output of pulumi about

Additional context

Datadog support wrote

However, when testing with Pulumi, both our team and Pulumi's engineers encountered the same issue you described, where request_file was not recognized. This issue requires further investigation.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[frank-bee]

One more info: in terraform this seems to work ( according to datadog support )

[VenelinMartinov]

Thanks for the report here @frank-bee and sorry for the trouble.

Can you please provider a pulumi program which reproduces this? Given that you've also tested in Terraform, can you please provide the terraform program too?

To verify, did you also import the resource in terraform and then run apply with the generated code or did you import the state without generating code?

[frank-bee]

@VenelinMartinov
As I said ,according to datadog support this issue was already reproduced by "the pulumi team". Are you part of that team?

Otherwise I can try to repoduce it again or at least send you some sample code.
But the relevant part - using the file upload button in datadog for multipart/formdata in synth tests - cannot be done in Pulumi at all at the moment. At least via pulumi import, I did not see the file / file content in the pulumi preview. Also in the pulumi documentation for synth tests I do not find this feature.

[frank-bee]

Hi @VenelinMartinov again,

here some sample code, but I didn't run it after reducing it to the critical part.
To actually reproduce it , it might be need to

1. run the code ( little modified ) with pulumi up and
2. afterwards modify the actual synth test in datadog using the feature "file upload" as request body (multipart form data) plus
3. import the test into pulumi stack

import * as pulumi from "@pulumi/pulumi";
import * as datadog from "@pulumi/datadog";

const config = new pulumi.Config();
const alertChannelCritical = config.require('alertChannelCritical');
const alertChannelWarning = config.require('alertChannelWarning');


    const multiStepTest = new datadog.SyntheticsTest("service_monitor_test", {
        name: "Service Monitor (BCN Prod)",
        type: "api",
        status: "paused",
        subtype: "multi",
        request: {
            method: "GET",
            url: "https://example.com/api/step1",
        },
        apiSteps: [
            {
                allowFailure: false,
                assertions: [
                    {
                        operator: "is",
                        property: "",
                        target: "201",
                        targetjsonpath: null,
                        targetxpath: null,
                        type: "statusCode"
                    }
                ],
                extractedValues: [],
                isCritical: true,
                name: "upload config",
                requestBasicauth: null,
                requestClientCertificate: null,
                requestDefinition: {
                    allowInsecure: true,
                    bodyType: "multipart/form-data",
                    body: "??", //TODO fix this
                    certificateDomains: [],
                    dnsServer: "",
                    dnsServerPort: 0,
                    followRedirects: false,
                    host: "",
                    httpVersion: "any",
                    message: "",
                    method: "POST",
                    noSavingResponseBody: false,
                    numberOfPackets: 0,
                    persistCookies: false,
                    plainProtoFile: "",
                    port: 0,
                    servername: "",
                    service: "",
                    shouldTrackHops: false,
                    timeout: 0,
                    url: "{{ BASEURL }}/namespaces/{{ NAMESPACE_ID }}/configs/{{ CONFIG_ID }}/files"
                },
                requestHeaders: {
                    //"content-type": "application/json",
                    "content-type": "multipart/form-data; boundary=\"DatadogSyntheticsFiles\"", //TODO fix this
                    Authorization: "Bearer {{ BPC_PROD_API_TETRLB_KEY }}",
                },
                requestProxy: null,
                requestQuery: {},
                retry: {
                    count: 0,
                    interval: 300
                },
                subtype: "http"
            }
        ],
        locations: ["aws:us-east-1", "aws:eu-west-1"],
        message: "foo bar",
        optionsList: {
            acceptSelfSigned: false,
            allowInsecure: false,
            checkCertificateRevocation: false,

            disableCors: false,
            disableCsp: false,
            followRedirects: false,
            ignoreServerCertificateError: false,
            initialNavigationTimeout: 0,
            minFailureDuration: 0,
            minLocationFailed: 1,
            monitorName: "",
            monitorOptions: {
                renotifyInterval: 0
            },
            monitorPriority: 2,
            noScreenshot: false,
            restrictedRoles: [],
            retry: {
                count: 0,
                interval: 300
            },
            scheduling: {
                timeframes: [
                    {
                        day: 4,
                        from: "00:00",
                        to: "23:59"
                    },
                    {
                        day: 5,
                        from: "00:00",
                        to: "23:59"
                    },
                    {
                        day: 3,
                        from: "00:00",
                        to: "23:59"
                    },
                    {
                        day: 2,
                        from: "00:00",
                        to: "23:59"
                    },
                    {
                        day: 1,
                        from: "00:00",
                        to: "23:59"
                    }
                ],
                timezone: "Europe/Berlin"
            },
            tickEvery: 600
        },
        tags: TEST_TAGS,
        configVariables: [
            {
                example: "",
                id: "",
                name: "TOKEN",
                pattern: "",
                secure: true,
                type: "text"
            },
            {
                example: "https://example.com/api/v1",
                id: "",
                name: "BASEURL",
                pattern: "https://example.com/api/v1",
                secure: false,
                type: "text"
            },
            {
                example: "1",
                id: "",
                name: "NAMESPACE_ID",
                pattern: "1",
                secure: false,
                type: "text"
            },
            {
                example: "",
                id: "3c4d6fda-...",
                name: "BPC_PROD_API_TETRLB_KEY",
                pattern: "",
                secure: false,
                type: "global",
            }
        ],
        locations: [
            "pl:bcn-dev-..."
        ]
    }, /*{
       import: "iau-usz-svk"
    }*/);

[frank-bee]

this is the UI in datadog I talked about

[VenelinMartinov]

Thanks for providing the details @frank-bee. I've been unable to run that code as it seems to depend on other external bits, however I managed to reproduce the issue with the following:

import * as pulumi from "@pulumi/pulumi";
import * as datadog from "@pulumi/datadog";

const testApi = new datadog.SyntheticsTest("test_api", {
    name: "An API test on example.org",
    type: "api",
    subtype: "http",
    status: "live",
    message: "Notify @pagerduty",
    locations: ["aws:eu-central-1"],
    tags: [
        "foo:bar",
        "foo",
        "env:test",
    ],
    requestDefinition: {
        bodyType: "multipart/form-data", // added after first pulumi up
        method: "GET",
        url: "https://www.example.org",
    },
    requestHeaders: {
        "Content-Type": "application/json",
        Authentication: "Token: 1234566789",
    },
    assertions: [{
        type: "statusCode",
        operator: "is",
        target: "200",
    }],
    optionsList: {
        tickEvery: 900,
        retry: {
            count: 2,
            interval: 300,
        },
        monitorOptions: {
            renotifyInterval: 120,
        },
    },
},{
    import: "ate-p9x-735" // added after first pulumi up
});

[VenelinMartinov]

However running the pulumi import CLI command after modifying the test in the dashboard seems to work, can you try that:

1. provision test.
2. modify in the dashboard
3. run pulumi import datadog:index/syntheticsTest:SyntheticsTest test mytest-id

This generates code for the test which works with some slight modifications.

Does that work around the issue for you?

If not, can you please explain what you are trying to achieve here? Why are you modifying the test in the dashboard? Can you use the pulumi program to achieve the same?

[frank-bee]

Hi @VenelinMartinov ,

did you try a file Upload in the body of the multipart form data and afterwards import? yaml file ideally ,this is what we use
This is the part the does not work.
To which attribute in datadog.SyntheticsTest is the content of the file written?

[frank-bee]

I tried to create a test with your code above getting

Do you want to perform this update? yes
Updating (datadog-local):
     Type                             Name                            Status                       Info
 +   pulumi:pulumi:Stack              datadog-monitors-datadog-local  **creating failed (1s)**     1 er
 +   └─ datadog:index:SyntheticsTest  test_api                        **creating failed**          2 er

Diagnostics:
  pulumi:pulumi:Stack (datadog-monitors-datadog-local):
    error: update failed

  datadog:index:SyntheticsTest (test_api):
    error:   sdk-v2/provider2.go:385: sdk.helper_schema: error creating synthetics API test from /api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["'request' value '{'bodyType': 'multipart/form-data', 'headers': {'Authentication': 'Token: 1234566789', 'Content-Type': 'application/json'}, 'method': 'GET', 'url': 'https://www.example.org'}' is invalid"]}: [email protected]
    error: 1 error occurred:
        * error creating synthetics API test from /api/v1/synthetics/tests/api: 400 Bad Request: {"errors":["'request' value '{'bodyType': 'multipart/form-data', 'headers': {'Authentication': 'Token: 1234566789', 'Content-Type': 'application/json'}, 'method': 'GET', 'url': 'https://www.example.org'}' is invalid"]}

Resources:
    + 1 created

Duration: 3s

But there is no resource created , so how should I import something?

[VenelinMartinov]

Hey @frank-bee, the code above was for confirming that the issue is indeed present. It should provide the same error message.

For working around the issue can you try:

1. Create the test without the import resource option and without the body type:

const testApi = new datadog.SyntheticsTest("test_api", {
    name: "An API test on example.org",
    type: "api",
    subtype: "http",
    status: "live",
    message: "Notify @pagerduty",
    locations: ["aws:eu-central-1"],
    tags: [
        "foo:bar",
        "foo",
        "env:test",
    ],
    requestDefinition: {
        method: "GET",
        url: "https://www.example.org",
    },
    requestHeaders: {
        "Content-Type": "application/json",
        Authentication: "Token: 1234566789",
    },
    assertions: [{
        type: "statusCode",
        operator: "is",
        target: "200",
    }],
    optionsList: {
        tickEvery: 900,
        retry: {
            count: 2,
            interval: 300,
        },
        monitorOptions: {
            renotifyInterval: 120,
        },
    },
}

2. Change as required in the datadog console
3. Run the pulumi import on the command line - that should generate the required code for you: https://www.pulumi.com/registry/packages/datadog/api-docs/syntheticstest/#import
4. Paste the code in your program, replacing the previous Test code
5. Try pulumi up again and make sure there are no changes.