Tech debt: I'm unable to convert NGINX Ingress resources with rewrite annotations to proper HTTPRoute resources

[ferdinandosimonetti]

Description:

I have some Ingress resources to translate in HTTPRoute format, during the transition from NGINX Ingress Controller to Envoy Gateway, and I'm unable to have them right.

The most problematic Ingress was this one (extract from kubectl describe ingress):

Rules:
  Host                         Path  Backends
  ----                         ----  --------
  ms-dev.xxx.yyy.com  
                               /(benchmark|estimation|predictive)/api/(.*)   ms-context-integration-service:80 (10.100.66.62:80)
Annotations:                   nginx.ingress.kubernetes.io/rewrite-target: /$1/$2
                               nginx.ingress.kubernetes.io/use-regex: true

I tried this:

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ms-context-integration-redirect
  namespace: apis-dev
spec:
  parentRefs:
    - name: gateway-private
      namespace: balancers-dev
      sectionName: https
  hostnames:
    - "ms-dev.xxx.yyy.com"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: "/benchmark/api"
      filters:
        - type: ExtensionRef
          extensionRef:
            group: gateway.envoyproxy.io
            kind: HTTPRouteFilter
            name: ms-context-integration-redirect
      backendRefs:
        - name: ms-context-integration-service
          port: 80
    - matches:
        - path:
            type: PathPrefix
            value: "/estimation/api"
      filters:
        - type: ExtensionRef
          extensionRef:
            group: gateway.envoyproxy.io
            kind: HTTPRouteFilter
            name: ms-context-integration-redirect
      backendRefs:
        - name: ms-context-integration-service
          port: 80
    - matches:
        - path:
            type: PathPrefix
            value: "/predictive/api"
      filters:
        - type: ExtensionRef
          extensionRef:
            group: gateway.envoyproxy.io
            kind: HTTPRouteFilter
            name: ms-context-integration-redirect
      backendRefs:
        - name: ms-context-integration-service
          port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRouteFilter
metadata:
  name: ms-context-integration-redirect
  namespace: apis-dev
spec:
  urlRewrite:
    path:
      type: ReplaceRegexMatch
      replaceRegexMatch:
        pattern: '^/(benchmark|estimation|predictive)/api/(.*)'
        substitution: '/\1/\2'

But when I tried calling it,

curl -v https://ms-dev.xxx.yyy.com/benchmark/api/swagger/index.html

the logs said route not found

[2024-11-15 17:37:35.419][14][debug][http] [source/common/http/conn_manager_impl.cc:1183] [Tags: "ConnectionId":"19940","StreamId":"3177253553529166521"] request headers complete (end_stream=true):
':method', 'GET'
':scheme', 'https'
':authority', 'ms-dev.xxx.yyy.com'
':path', '/benchmark/api/swagger/index.html'
'user-agent', 'curl/8.7.1'
'accept', '*/*'

[2024-11-15 17:37:35.419][14][debug][http] [source/common/http/conn_manager_impl.cc:1166] [Tags: "ConnectionId":"19940","StreamId":"3177253553529166521"] request end stream timestamp recorded
--
[2024-11-15 17:37:41.777][15][debug][connection] [source/common/network/connection_impl.cc:150] [Tags: "ConnectionId":"19942"] closing data_to_write=162 type=2
[2024-11-15 17:37:41.777][15][debug][connection] [source/common/network/connection_impl.cc:788] [Tags: "ConnectionId":"19942"] write flush complete
[2024-11-15 17:37:41.778][15][debug][connection] [source/common/network/connection_impl.cc:636] [Tags: "ConnectionId":"19942"] remote early close
[2024-11-15 17:37:41.778][15][debug][connection] [source/common/network/connection_impl.cc:276] [Tags: "ConnectionId":"19942"] closing socket: 0
[2024-11-15 17:37:41.778][15][debug][conn_handler] [source/common/listener_manager/active_stream_listener_base.cc:136] [Tags: "ConnectionId":"19942"] adding to cleanup list
{"start_time":"2024-11-15T17:37:35.419Z","method":"GET","x-envoy-origin-path":"/benchmark/api/swagger/index.html","protocol":"HTTP/2","response_code":"404","response_flags":"NR","response_code_details":"route_not_found","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"0","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.100.244.222","user-agent":"curl/8.7.1","x-request-id":"ff7ba8e3-f4ee-4078-b01a-6d8d41ffb76c",":authority":"ms-dev.farmhub.nadara.com","upstream_host":"-","upstream_cluster":"-","upstream_local_address":"-","downstream_local_address":"10.100.60.158:10443","downstream_remote_address":"10.100.244.222:53319","requested_server_name":"-","route_name":"-"}
[2024-11-15 17:37:42.164][13][debug][conn_handler] [source/common/listener_manager/active_tcp_listener.cc:160] [Tags: "ConnectionId":"20533"] new connection from 10.100.60.145:39864
[2024-11-15 17:37:42.164][13][debug][http] [source/common/http/conn_manager_impl.cc:393] [Tags: "ConnectionId":"20533"] new stream
[2024-11-15 17:37:42.164][13][debug][http] [source/common/http/conn_manager_impl.cc:1183] [Tags: "ConnectionId":"20533","StreamId":"14028152975819975829"] request headers complete (end_stream=true):
':authority', '10.100.60.16:19001'
':path', '/ready'

What am I doing wrong?

[optional Relevant Links:]

I tried to follow the suggestions here

[ferdinandosimonetti]

Ok, now it works. See the reference below

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ms-context-integration-redirect
  namespace: apis-dev
spec:
  parentRefs:
    - name: gateway-private
      namespace: balancers-dev
      sectionName: https
  hostnames:
    - "ms-dev.xxx.yyy.com"
  rules:
    - matches:
        - path:
            type: RegularExpression
            value: '^/(benchmark|estimation|predictive)/api/(.*)'
      filters:
        - type: ExtensionRef
          extensionRef:
            group: gateway.envoyproxy.io
            kind: HTTPRouteFilter
            name: ms-context-integration-redirect
      backendRefs:
        - name: ms-context-integration-service
          port: 80
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: HTTPRouteFilter
metadata:
  name: ms-context-integration-redirect
  namespace: apis-dev
spec:
  urlRewrite:
    path:
      type: ReplaceRegexMatch
      replaceRegexMatch:
        pattern: '^/(benchmark|estimation|predictive)/api/(.*)'
        substitution: '/\1/\2'

But I have another problem. Given this other HTTPRoute

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ms-v1-ingress
  namespace: apis-dev
spec:
  parentRefs:
    - name: gateway-private
      namespace: balancers-dev
      sectionName: https
  hostnames:
    - ms-dev.xxx.yyy.com
  rules:
    - filters:
      - type: URLRewrite
        urlRewrite:
          path:
            type: ReplacePrefixMatch
            replacePrefixMatch: /
    - matches:
      - path:
          type: PathPrefix
          value: "/api/catalog"
      backendRefs:
      - name: ms-catalog-service
        port: 80
    - matches:
      - path:
          type: PathPrefix
          value: "/api/coordinator"
      backendRefs:
      - name: ms-coordinator-service
        port: 80
    - matches:
      - path:
          type: PathPrefix
          value: "/api/field"
      backendRefs:
      - name: ms-context-service
        port: 80
    - matches:
      - path:
          type: PathPrefix
          value: "/api/ingestion"
      backendRefs:
      - name: ms-ingestion-service
        port: 80
    - matches:
      - path:
          type: PathPrefix
          value: "/api/integration"
      backendRefs:
      - name: ms-context-integration-service
        port: 80
    - matches:
      - path:
          type: PathPrefix
          value: "/api/logs"
      backendRefs:
      - name: ms-logs-service
        port: 80
    - matches:
      - path:
          type: PathPrefix
          value: "/historical"
      backendRefs:
      - name: ms-historical-service
        port: 80
    - matches:
      - path:
          type: PathPrefix
          value: "/historicalv2"
      backendRefs:
      - name: ms-historicalv2-service
        port: 80
    - matches:
      - path:
          type: PathPrefix
          value: "/recovery"
      backendRefs:
      - name: ms-recovery-service
        port: 80

I was expecting to see the URL Rewrite in action, substituting /recovery/whatever with /whatever and routing the request toward ms-recovery-service, or substituting /api/ingestion/whatelse with /whatelse and routing it to ms-ingestion-service.

However, this doesn't happen.

[2024-11-18 12:54:44.183][13][debug][http] [source/common/http/conn_manager_impl.cc:1183] [Tags: "ConnectionId":"69002","StreamId":"4692612904423442305"] request headers complete (end_stream=true):
':method', 'GET'
':scheme', 'https'
':authority', 'ms-dev.xxx.yyy.com'
':path', '/api/ingestion/swagger/index.html'
'user-agent', 'curl/8.7.1'
'accept', '*/*'

[2024-11-18 12:54:44.183][13][debug][http] [source/common/http/conn_manager_impl.cc:1166] [Tags: "ConnectionId":"69002","StreamId":"4692612904423442305"] request end stream timestamp recorded
[2024-11-18 12:54:44.183][13][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"69002"] current connecting state: false
[2024-11-18 12:54:44.183][13][debug][router] [source/common/router/router.cc:527] [Tags: "ConnectionId":"69002","StreamId":"4692612904423442305"] cluster 'httproute/apis-dev/ms-v1-ingress/rule/4' match for URL '/api/ingestion/swagger/index.html'
[2024-11-18 12:54:44.183][13][debug][router] [source/common/router/router.cc:756] [Tags: "ConnectionId":"69002","StreamId":"4692612904423442305"] router decoding headers:
':method', 'GET'
':scheme', 'https'
':authority', 'ms-dev.xxx.yyy.com'
':path', '/api/ingestion/swagger/index.html'
'user-agent', 'curl/8.7.1'
'accept', '*/*'
'x-forwarded-for', '10.100.244.222'
'x-forwarded-proto', 'https'
--
[2024-11-18 12:54:49.740][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:173] dns resolution without records for envoy-gateway
[2024-11-18 12:54:49.741][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:308] dns resolution for envoy-gateway completed with status 0
[2024-11-18 12:54:49.741][1][debug][upstream] [source/common/upstream/upstream_impl.cc:484] transport socket match, socket default selected for host with address 10.0.214.65:18002
[2024-11-18 12:54:49.741][1][debug][upstream] [source/extensions/clusters/strict_dns/strict_dns_cluster.cc:201] DNS refresh rate reset for envoy-gateway, refresh rate 5000 ms
[2024-11-18 12:54:51.006][1][debug][main] [source/server/server.cc:237] flushing stats
{"start_time":"2024-11-18T12:54:44.183Z","method":"GET","x-envoy-origin-path":"/api/ingestion/swagger/index.html","protocol":"HTTP/2","response_code":"404","response_flags":"-","response_code_details":"via_upstream","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"3","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.100.244.222","user-agent":"curl/8.7.1","x-request-id":"b237c762-eca4-4ed7-91f6-80c10384873d",":authority":"ms-dev.farmhub.nadara.com","upstream_host":"10.0.106.202:80","upstream_cluster":"httproute/apis-dev/ms-v1-ingress/rule/4","upstream_local_address":"10.100.60.16:41890","downstream_local_address":"10.100.60.16:10443","downstream_remote_address":"10.100.244.222:54793","requested_server_name":"-","route_name":"httproute/apis-dev/ms-v1-ingress/rule/4/match/0/ms-dev_farmhub_nadara_com"}

[arkodg]

cc @guydc

[guydc]

@arkodg , @ferdinandosimonetti - shouldn't the filters be scoped to the routes that you want this rewrite to apply to? It looks like they are currently part of a match-less route rule.