Fix issue with loadbalancer failover to default server

[brandond]

Proposed Changes

The loadbalancer should only fail over to the default server if all other server have failed, and it should force fail-back to a preferred server as soon as one passes health checks.

The loadbalancer tests have been improved to ensure that this occurs.

Types of Changes

bugfix

Verification

See linked issue

Testing

yes

Linked Issues

• Secondary etcd-only nodes do not reconnect to apiserver after outage if joined against an etcd-only node #11311
• Add tests to pkg/agent/loadbalancer #10505
• Nodes may fail to reconnect after K8s upgrade for K3s clusters with multiple etcd-only nodes rancher/rancher#48050

User-Facing Change

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 73.91304% with 6 lines in your changes missing coverage. Please review.

Project coverage is 13.82%. Comparing base (71fdc4a) to head (3fb8207).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/agent/loadbalancer/servers.go	78.94%	4 Missing ⚠️
pkg/util/apierrors.go	0.00%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (71fdc4a) and HEAD (3fb8207). Click for more details.

HEAD has 18 uploads less than BASE

Flag	BASE (71fdc4a)	HEAD (3fb8207)
inttests	11	0
e2etests	7	0

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #11319       +/-   ##
===========================================
- Coverage   46.87%   13.82%   -33.05%     
===========================================
  Files         179      176        -3     
  Lines       18587    18526       -61     
===========================================
- Hits         8712     2562     -6150     
- Misses       8516    15665     +7149     
+ Partials     1359      299     -1060     

Flag	Coverage Δ
e2etests	?
inttests	?
unittests	13.82% <73.91%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.