More than +/- 1600 upstreams causes "Error when reloading NGINX"

[jppitout]

How can we tune Nginx OSS IC to better handle config reloads with large number of upstreams?

Would NginxPlus config reloads via API present a similar issue?

[brianehlert]

Can you provide more details around the error?
Is it only the number of upstreams or could it be related to other combinations of things?

The NGINX Plus version is the same configuration, but the project takes advantage of the N+ API under the hood for the upstream group management, thus there is no reload when changes happen to the upstream services, only when configuration objects like Ingress, VirtualServer, TransportServer, etc. change.

[jppitout]

@brianehlert We spent most of last week troubleshooting but as far as we can tell it is only when the number of upstream pods is greater than +/- 1600 pods. Number of pods were scaled by HPA's but we later made this static for all deployment when we discovered the "upstream limit".
Once the "limit" is reached we see loads of these events for a newly created ingress:

  Warning  AddedOrUpdatedWithError  3m40s  nginx-ingress-controller  Configuration for test1/ingress-test3 was added or updated ; but was not applied: Error when reloading NGINX when updating resources: could not get newest config version: could not get expected version: 117 after 1m0s
  Warning  AddedOrUpdatedWithError  3m39s  nginx-ingress-controller  Configuration for test1/ingress-test3 was added or updated ; but was not applied: Error when reloading NGINX when updating resources: could not get newest config version: could not get expected version: 119 after 1m0s
  Warning  AddedOrUpdatedWithError  3m39s  nginx-ingress-controller  Configuration for test1/ingress-test3 was added or updated ; but was not applied: Error when reloading NGINX when updating resources: could not get newest config version: could not get expected version: 118 after 1m0s
  Warning  AddedOrUpdatedWithError  3m35s  nginx-ingress-controller  Configuration for test1/ingress-test3 was added or updated ; but was not applied: Error when reloading NGINX when updating resources: could not get newest config version: could not get expected version: 118 after 1m0s
  Warning  AddedOrUpdatedWithError  3m34s  nginx-ingress-controller  Configuration for test1/ingress-test3 was added or updated ; but was not applied: Error when reloading NGINX when updating resources: could not get newest config version: could not get expected version: 118 after 1m0s
  Warning  AddedOrUpdatedWithError  3m28s  nginx-ingress-controller  Configuration for test1/ingress-test3 was added or updated ; but was not applied: Error when reloading NGINX when updating resources: could not get newest config version: could not get expected version: 119 after 1m0s

For existing ingresses the upstreams stop receiving traffic.

Another thing that might be worth noting is the total number of pods on the cluster is around 3600 when this occurs.

We're currently deploying NginxPlus IC hoping that it will resolve the issue.

[jppitout]

Thanks to NginxPlus metrics it seems very likely that the upstream-zone-size was the cause but we are still monitoring.

[brianehlert]

I was in the process of typing... ;-)

Are you increasing the upstream-zone-size for these larger service scales?
https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#backend-services-upstreams

[jppitout]

We've set upstream-zone-size to 1024k

[brianehlert]

I do have a roadmap item to automatically grow this as upstream services scale.
Would that be useful?
Would something that conflict with pod limits?

[jppitout]

That would definitely be useful although it could affect pod resource requests/limits it would be easier to know what to do in the case of the nginx-ingress pods being OOM killed rather than config reload errors on the ingress resources.

I really like NginxPlus because it allowed us to find the error a whole lot quicker.