REST API: POST requests to Compute API fail with 411 (Length Required) #121

TmLev · 2024-01-25T15:23:26Z

I'm trying to start a stopped/terminated compute instance:

let compute_config = google_rest_client.create_google_compute_v1_config().await?;
let request = gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodStartParams {
    project: "<PROJECT>".into(),
    instance: "<INSTANCE>".into(),
    zone: "<ZONE>".into(),
    ..Default::default()
};
let response =
    gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_start(
        &compute_config,
        request,
    )
    .await?;

Here's the response I get:

Error: ResponseError(ResponseContent { status: 411, content: "<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n  <title>Error 411 (Length Required)!!1</title>\n  <style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n  </style>\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n  <p><b>411.</b> <ins>That’s an error.</ins>\n  <p>POST requests require a <code>Content-length</code> header.  <ins>That’s all we know.</ins>\n", entity: None })

Stripped down:

status: 411
content: 411. That’s an error. POST requests require a Content-length header. That’s all we know.

The text was updated successfully, but these errors were encountered:

abdolence · 2024-01-25T19:52:11Z

Hey,

I have almost the same working example here:

gcloud-sdk-rs/examples/gcs-rest-client/src/main.rs

Line 70 in d0320ae

    
           let google_project_id = gcloud_sdk::GoogleEnvironment::detect_google_project_id().await

and it works without issues. In fact, I just copied your code and filled in my parameters and it seems to be working also as I expected.

So, something different in your environment and looking at your response it a bit weird, it returns HTML instead of more REST-friendly responses.

So, there are possible issues:

Please check if you authenticated correctly. For example using gcloud auth application-default login first
Verify if you had enabled appropriate features, e.g. features = ["rest", "tls-roots", "google-rest-compute-v1"] in your Cargo.toml
You don't have any unusual proxy/DNS settings (https://compute.googleapis.com/compute/v1 this URL must be available without any disruptions)
Using the latest version
You didn't fill in some unexpected characters in the parameters

abdolence · 2024-01-25T20:00:59Z

Most likely this is related to the authentication so you have something off in your generated token.

TmLev · 2024-01-25T20:14:32Z

Hey! Thanks for your response :)

Please check if you authenticated correctly

I'm using service account saved as JSON. This service account should have the necessary permissions, but I'll double check.

Verify if you had enabled appropriate features

Using the latest version

Here's my Cargo.toml:

gcloud-sdk = { version = "0.24.2", features = ["google-rest-storage-v1", "google-rest-compute-v1"] }

You don't have any unusual proxy/DNS settings (https://compute.googleapis.com/compute/v1 this URL must be available without any disruptions)

Hmm, this exact URL is returning 404.

You didn't fill in some unexpected characters in the parameters

In the parameters of the ComputePeriodInstancesPeriodStartParams?

abdolence · 2024-01-25T21:00:40Z

Your Cargo.toml is fine.

I'm using service account saved as JSON. This service account should have the necessary permissions, but I'll double check.

I'm quite sure this is related to the way your app is authenticated. How did you generate that JSON file? Is there anything unusual about it?

Hmm, this exact URL is returning 404.

Yes, this should return 404 since this is only the base URL for the API.

In the parameters of the ComputePeriodInstancesPeriodStartParams?

Yes, something off in instance name for example?

abdolence · 2024-01-25T21:02:17Z

Try to check using:

gcloud auth application-default login

and compare those JSON files - one generated by gcloud tool, and another yours.
PS. gcloud cli generates usually a file here: $HOME/.config/gcloud/application_default_credentials.json

abdolence · 2024-01-25T21:06:26Z

I checked even when you don't permissions you are supposed to get something like this:

ResponseError(ResponseContent { status: 403, content: "{\n  \"error\": {\n    \"code\": 403,\n    \"message\": \"Required 'compute.instances.list' permission for 'projects/'...

please check if you provided appropriate PROJECT_ID as well.

TmLev · 2024-01-26T15:35:34Z

I'm quite sure this is related to the way your app is authenticated. How did you generate that JSON file? Is there anything unusual about it?

I downloaded this JSON file from the Google Cloud UI for creating service accounts here:

I've been using this service account for uploading images to Cloud Storage and for getting the instance details (gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_get) -- both scenarios work just fine.

Can't spot anything unusual about it, here's its structure:

{
  "type": "service_account",
  "project_id": "<REDACTED>",
  "private_key_id": "<REDACTED>",
  "private_key": "<REDACTED>",
  "client_email": "<REDACTED>",
  "client_id": "<REDACTED>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "<REDACTED>",
  "universe_domain": "googleapis.com"
}

Yes, something off in instance name for example?

The same name most definitely works for ComputePeriodInstancesPeriodGetParams, so seems like everything is okay here.

and compare those JSON files - one generated by gcloud tool, and another yours.

gcloud CLI generates a different JSON:

{
  "client_id": "<REDACTED>",
  "client_secret": "<REDACTED>",
  "quota_project_id": "<REDACTED>",
  "refresh_token": "<REDACTED>",
  "type": "authorized_user"
}

please check if you provided appropriate PROJECT_ID as well.

The same project id works for ComputePeriodInstancesPeriodGetParams, I even put all three parameters in const definitions just to be sure.

TmLev · 2024-01-26T15:38:39Z

Here's the initialization bit:

    let google_rest_client = Arc::new(
        gcloud_sdk::GoogleRestApi::with_token_source(
            TokenSourceType::File("./google-service-account.json".into()),
            GCP_DEFAULT_SCOPES.clone(),
        )
        .await?,
    );

I assume GCP_DEFAULT_SCOPES should work since the error message differs from the one you get when you lack permissions?..

abdolence · 2024-01-26T16:33:38Z

This is interesting, so you're saying the same setup for the same project ID and instance names you have actually other methods working and only compute_instances_start fails? Then I was wrong and it is not related to authentication. I didn't know that it works for other methods before.

abdolence · 2024-01-26T17:01:21Z

let compute_config = google_rest_client.create_google_compute_v1_config().await

Can you elaborate how are using those configs? Are you storing them for long time and reusing? If so, please create them for each requests separately, since they contain short lived tokens.

abdolence · 2024-01-26T17:06:18Z

I tested this example with my service account and test project:

let google_project_id = gcloud_sdk::GoogleEnvironment::detect_google_project_id().await
        .expect("No Google Project ID detected. Please specify it explicitly using env variable: PROJECT_ID");

    let google_rest_client = gcloud_sdk::GoogleRestApi::new().await.unwrap();
    let compute_config = google_rest_client.create_google_compute_v1_config().await.unwrap();

    let response = gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_list(
        &compute_config,
        gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodListParams {
            project: google_project_id.to_string(),
            zone: "europe-north1-a".to_string(),
            ..Default::default()
        }
    ).await.unwrap();

    println!("{:?}", response.items.map(|xs| xs.iter().map(|x| x.name.clone()).collect::<Vec<_>>()));


    let request = gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodStartParams {
        project: google_project_id.to_string(),
        instance: "lb-mini-balancer-node".into(),
        zone: "europe-north1-a".into(),
        ..Default::default()
    };
    let response =
        gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_start(
            &compute_config,
            request,
        )
            .await.unwrap();
            
    println!("{:?}", response);

    let request = gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodStopParams {
        project: google_project_id.to_string(),
        instance: "lb-mini-balancer-node".into(),
        zone: "europe-north1-a".into(),
        ..Default::default()
    };
    let response =
        gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_stop(
            &compute_config,
            request,
        )
            .await.unwrap();

    println!("{:?}", response);

and it works. Something different with our environments or parameters.

TmLev · 2024-01-26T17:11:55Z

This is interesting, so you're saying the same setup for the same project ID and instance names you have actually other methods working and only compute_instances_start fails?

If we're talking about compute.instances API, I haven't tested any "mutating" requests (meaning non-GET) apart from the compute.instances.start. But yeah, GET requests work.

Can you elaborate how are using those configs?

I'm creating a new config before every request and I'm not storing them anywhere.

and it works

I will triple check everything and will try a different service account

abdolence · 2024-01-26T17:20:55Z

I think service accounts are irrelevant now - if at least one method is working.
Something with either network/proxy or your parameters (structure fields for params).

TmLev · 2024-01-31T13:14:54Z

Something with either network/proxy

I've tried sending the same request from a Google Cloud instance located in the same zone/region, but it returned the same response.

Something with ... your parameters (structure fields for params).

They are the same as the ones I use for ComputePeriodInstancesPeriodGetParams.

abdolence · 2024-01-31T13:24:55Z

It is hard me to help here since this is not reproducible on my infrastructure. As one possible option to debug you can try to check HTTP request printing it out before it sent to Google. Just add some print inside locally cloned in google cloud sdk crate when the request is built. It may show something unexpected. One more option is to try to use some kind of HTTP to HTTPS simple proxy and check the traffic, but it maybe more complicated.

TmLev · 2024-05-21T11:11:34Z

I took another try at this.

I forked this repo, added it via path = "..." to Cargo.toml and made the following changes:

diff --git a/gcloud-sdk/src/rest_apis/google_rest_apis/compute_v1/apis/instances_api.rs b/gcloud-sdk/src/rest_apis/google_rest_apis/compute_v1/apis/instances_api.rs
index 989d988a2..2d4833810 100644
--- a/gcloud-sdk/src/rest_apis/google_rest_apis/compute_v1/apis/instances_api.rs
+++ b/gcloud-sdk/src/rest_apis/google_rest_apis/compute_v1/apis/instances_api.rs
@@ -7023,6 +7023,7 @@ pub async fn compute_instances_start(
         local_var_req_builder = local_var_req_builder.bearer_auth(local_var_token.to_owned());
     };

+    local_var_req_builder = local_var_req_builder.header(reqwest::header::CONTENT_LENGTH, 0);
     let local_var_req = local_var_req_builder.build()?;
     let local_var_resp = local_var_client.execute(local_var_req).await?;

It works with the explicit CONTENT_LENGTH set to 0 and fails without it. Just to be clear, I made zero changes to the code from OP or to the service account/auth stuff.

I really don't know why it works on your infrastructure.

TmLev · 2024-05-21T11:16:53Z

This may be relevant: seanmonstar/reqwest#838

TmLev · 2024-05-21T11:28:22Z

I can also confirm that you can replace the default client of GoogleRestApi with your own, setting CONTENT_LENGTH to 0 by default for all request headers:

let mut headers = reqwest::header::HeaderMap::new();
headers.insert(CONTENT_LENGTH, 0.into());
let http_client = reqwest::ClientBuilder::new()
    .default_headers(headers)
    .build()
    .unwrap();

let google_rest_client = gcloud_sdk::GoogleRestApi::with_client_token_source(
    http_client,
    token_source_type,
    token_scopes,
).await?;

Although I'm not sure whether it will break requests with non-empty body...

abdolence · 2024-05-21T11:28:24Z

Hey, hm, interesting.

Maybe this is related that I had different reqwest version when I tested it.
Right now I have "0.11.20" in my lock file. Which one do you have in yours?

abdolence · 2024-05-21T11:29:06Z

The problem with this it can be 0 only when body is actually empty. And that's not true for all requests, but only for some of them.

TmLev · 2024-05-21T11:29:41Z

Maybe this is related that I had different reqwest version when I tested it.
Right now I have "0.11.20" in my lock file. Which one do you have in yours?

Same:

[[package]]
name = "reqwest"
version = "0.11.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e9ad3fe7488d7e34558a2033d45a0c90b72d97b4f80705666fea71472e2e6a1"

abdolence · 2024-05-21T11:30:13Z

I'll test it again and come back to you. Thanks for the update!

TmLev · 2024-05-21T11:30:15Z

The problem with this it can be 0 only when body is actually empty. And that's not true for all requests, but only for some of them.

I assume setting the body explicitly results in implicit update of CONTENT_LENGTH header by reqwest itself (don't quote me on that)

abdolence · 2024-05-21T16:29:54Z

I just tested it again, and it still works for me with no issues 🤔
I even added some println!() to print out whole content of that local_var_req.

You sure you don't have any proxy between your application and Google Cloud?

!!!!! Request { method: POST, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("compute.googleapis.com")), port: None, path: "/compute/v1/projects/latestbit/zones/europe-north1-a/instances/lb-mini-balancer-node/start", query: None, fragment: None }, headers: {"user-agent": "gcloud-sdk-rs/v0.24.6", "authorization": Sensitive, "authorization": Sensitive} }

Response:
Operation { client_operation_id: None, creation_timestamp: None, description: None, end_time: Some("2024-05-21T09:27:12.162-07:00"), error: None, http_error_message: None, http_error_status_code: None, id: Some("1961906089995892111"), insert_time: Some("2024-05-21T09:27:12.156-07:00"), instances_bulk_insert_operation_metadata: None, kind: Some("compute#operation"), name: Some("operation-1716308831374-618f94a380c9b-bd60f52f-e5366792"), operation_group_id: None, operation_type: Some("start"), progress: Some(100), region: None, self_link: Some("https://www.googleapis.com/compute/v1/projects/latestbit/zones/europe-north1-a/operations/operation-1716308831374-618f94a380c9b-bd60f52f-e5366792"), set_common_instance_metadata_operation_metadata: None, start_time: Some("2024-05-21T09:27:12.162-07:00"), status: Some(Done), status_message: None, target_id: Some("8330157900564331422"), target_link: Some("https://www.googleapis.com/compute/v1/projects/latestbit/zones/europe-north1-a/instances/lb-mini-balancer-node"), ...}

abdolence · 2024-05-21T16:30:39Z

Also tested it with the latest 0.11.27 with the same results.

abdolence · 2024-05-21T16:31:52Z

Another theory, maybe this is also different in different GCP regions. Which GCP region/zone are you trying to work with?

abdolence · 2024-05-21T16:33:31Z

Which OS are you using as well?

TmLev · 2024-05-23T11:25:10Z

You sure you don't have any proxy between your application and Google Cloud?

I'm sure I don't -- I even tried to start an instance in the same zone and send a request from there but it still didn't work

Which GCP region/zone are you trying to work with?

us-central1-a

Which OS are you using as well?

Locally macOS 13.4. For production, I deploy via Docker with debian:bookworm-slim as the base image.

abdolence · 2024-05-23T13:17:29Z

Just tested it on Mac OS + us-central1. Works again for me :/

This is frustrating. I tested this on different laptops now, and I just update the example gcs-rest-client with just this code:

    let compute_config = google_rest_client.create_google_compute_v1_config().await.unwrap();
    let request = gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodStartParams {
        project: google_project_id.to_string(),
        instance: "abd-test-micro".into(),
        zone: "us-central1-a".into(),
        ..Default::default()
    };
    let response =
        gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_start(
            &compute_config,
            request,
        )
            .await.unwrap();

and it works without any issue.

Do you a complete simple example that doesn't work for you? (obviously, don't include any sensitive info there, like service accounts etc).

abdolence added the question Further information is requested label Jan 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REST API: POST requests to Compute API fail with 411 (Length Required) #121

REST API: POST requests to Compute API fail with 411 (Length Required) #121

TmLev commented Jan 25, 2024

abdolence commented Jan 25, 2024 •

edited

Loading

abdolence commented Jan 25, 2024 •

edited

Loading

TmLev commented Jan 25, 2024 •

edited

Loading

abdolence commented Jan 25, 2024

abdolence commented Jan 25, 2024

abdolence commented Jan 25, 2024

TmLev commented Jan 26, 2024

TmLev commented Jan 26, 2024

abdolence commented Jan 26, 2024

abdolence commented Jan 26, 2024

abdolence commented Jan 26, 2024

TmLev commented Jan 26, 2024

abdolence commented Jan 26, 2024

TmLev commented Jan 31, 2024 •

edited

Loading

abdolence commented Jan 31, 2024

TmLev commented May 21, 2024 •

edited

Loading

TmLev commented May 21, 2024

TmLev commented May 21, 2024

abdolence commented May 21, 2024

abdolence commented May 21, 2024

TmLev commented May 21, 2024

abdolence commented May 21, 2024

TmLev commented May 21, 2024 •

edited

Loading

abdolence commented May 21, 2024

abdolence commented May 21, 2024

abdolence commented May 21, 2024

abdolence commented May 21, 2024

TmLev commented May 23, 2024

abdolence commented May 23, 2024 •

edited

Loading

REST API: POST requests to Compute API fail with 411 (Length Required) #121

REST API: POST requests to Compute API fail with 411 (Length Required) #121

Comments

TmLev commented Jan 25, 2024

abdolence commented Jan 25, 2024 • edited Loading

abdolence commented Jan 25, 2024 • edited Loading

TmLev commented Jan 25, 2024 • edited Loading

abdolence commented Jan 25, 2024

abdolence commented Jan 25, 2024

abdolence commented Jan 25, 2024

TmLev commented Jan 26, 2024

TmLev commented Jan 26, 2024

abdolence commented Jan 26, 2024

abdolence commented Jan 26, 2024

abdolence commented Jan 26, 2024

TmLev commented Jan 26, 2024

abdolence commented Jan 26, 2024

TmLev commented Jan 31, 2024 • edited Loading

abdolence commented Jan 31, 2024

TmLev commented May 21, 2024 • edited Loading

TmLev commented May 21, 2024

TmLev commented May 21, 2024

abdolence commented May 21, 2024

abdolence commented May 21, 2024

TmLev commented May 21, 2024

abdolence commented May 21, 2024

TmLev commented May 21, 2024 • edited Loading

abdolence commented May 21, 2024

abdolence commented May 21, 2024

abdolence commented May 21, 2024

abdolence commented May 21, 2024

TmLev commented May 23, 2024

abdolence commented May 23, 2024 • edited Loading

abdolence commented Jan 25, 2024 •

edited

Loading

abdolence commented Jan 25, 2024 •

edited

Loading

TmLev commented Jan 25, 2024 •

edited

Loading

TmLev commented Jan 31, 2024 •

edited

Loading

TmLev commented May 21, 2024 •

edited

Loading

TmLev commented May 21, 2024 •

edited

Loading

abdolence commented May 23, 2024 •

edited

Loading