ddtrace/tracer: report datadog.tracer.api.errors health metric

[mtoffl01]

What does this PR do?

Reports a datadogtracer.api.errors health metric when httpClient request to submit a trace payload fails or has status code >= 400, with a tag statusCode whose value is populated by the response status code

Motivation

More data to correlated dropped traces (datadog.tracer.traces_dropped) with failed submission attempt.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[datadog-datadog-prod-us1]

Datadog Report

Branch report: mtoff/health-metrics-2
Commit report: f736c44
Test service: dd-trace-go

❌ 5 Failed (0 Known Flaky), 5046 Passed, 67 Skipped, 2m 25.94s Total Time

❌ Failed Tests (5)

• TestIssue2050 - gopkg.in/DataDog/dd-trace-go.v1/contrib/google.golang.org/grpc - Details

  Expand for error

   Failed
   
   === RUN   TestIssue2050
   panic: runtime error: invalid memory address or nil pointer dereference
   [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1243414]
   
   goroutine 1164 [running]:
   gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*httpTransport).send(0xc00016a8a0, 0xc0000b5860)
   	/home/runner/work/dd-trace-go/dd-trace-go/ddtrace/tracer/transport.go:173 +0x1054
   gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*agentTraceWriter).flush.func1(0xc0000b5860)
   ...
  

• TestSpanAddEvent - gopkg.in/DataDog/dd-trace-go.v1/ddtrace/opentelemetry - Details

  Expand for error

   Failed
   
   === RUN   TestSpanAddEvent
   === RUN   TestSpanAddEvent/event_with_attributes
   === RUN   TestSpanAddEvent/event_with_timestamp
   === RUN   TestSpanAddEvent/mulitple_events
   panic: runtime error: invalid memory address or nil pointer dereference
   [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12aeeb4]
   
   goroutine 258 [running]:
   ...
  

• TestSpanAddEvent/event_with_attributes - gopkg.in/DataDog/dd-trace-go.v1/ddtrace/opentelemetry - Details

  Expand for error

   Failed
   
   === RUN   TestSpanAddEvent/event_with_attributes
  

• TestSpanAddEvent/event_with_timestamp - gopkg.in/DataDog/dd-trace-go.v1/ddtrace/opentelemetry - Details

  Expand for error

   Failed
   
   === RUN   TestSpanAddEvent/event_with_timestamp
  

• TestSpanAddEvent/mulitple_events - gopkg.in/DataDog/dd-trace-go.v1/ddtrace/opentelemetry - Details

  Expand for error

   Failed
   
   === RUN   TestSpanAddEvent/mulitple_events
   panic: runtime error: invalid memory address or nil pointer dereference
   [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12aeeb4]
   
   goroutine 258 [running]:
   gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*httpTransport).send(0xc000af5f50, 0xc00033b1a0)
   	/home/runner/work/dd-trace-go/dd-trace-go/ddtrace/tracer/transport.go:173 +0x1054
   gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.(*agentTraceWriter).flush.func1(0xc00033b1a0)
   ...