Connection refused when using nginx-proxy and podman environment

[fenandosr]

I have tried so many times with subtle differences that I have actually learned from this, but I can't make it work.

I'm trying to configure nginx-proxy and kratos running on podman containers. This implementation is working fine for other services also running on containers. This is relevant nginx config for kratos:

# 4433
upstream accounts.example.org-4433 {
    server 10.89.0.15:4433;
}
# admin at 4434
upstream accounts.example.org-4434 {
    server 10.89.0.15:4434;
}
server {
    server_name accounts.example.org;
    access_log /var/log/nginx/access.log vhost;
    listen 80 ;
    # [...] Let's Encrypt [...]
}
server {
    server_name accounts.example.org;
    access_log /var/log/nginx/access.log vhost;
    http2 on;
    listen 443 ssl ;
    # [...] Let's Encrypt [...]
    location / {
        proxy_pass http://accounts.mcps-epcm.org-4433;
        set $upstream_keepalive false;
    }
    location /admin {
        proxy_pass http://accounts.mcps-epcm.org-4434;
        set $upstream_keepalive false;
    }
}

kratos is also running from a podman container, in the same network. This container exposes 4433 and 4434 as expected. When the container runs, everything seems to be ok:

time=## level=info msg=No tracer configured - skipping tracing setup audience=application service_name=Ory Kratos service_version=v1.0.0
time=## level=debug msg=Connecting to SQL Database audience=application connMaxLifetime=0s idlePool=4 pool=20 service_name=Ory Kratos service_version=v1.0.0
time=## level=info msg=Software quality assurance features are enabled. Learn more at: https://www.ory.sh/docs/ecosystem/sqa audience=application service_name=Ory Kratos service_version=v1.0.0
time=## level=info msg=TLS has not been configured for admin, skipping audience=application service_name=Ory Kratos service_version=v1.0.0
time=## level=info msg=Starting the admin httpd on: 127.0.0.1:4434 audience=application service_name=Ory Kratos service_version=v1.0.0
time=## level=info msg=TLS has not been configured for public, skipping audience=application service_name=Ory Kratos service_version=v1.0.0
time=## level=info msg=Starting the public httpd on: 127.0.0.1:4433 audience=application service_name=Ory Kratos service_version=v1.0.0

When trying to make a request such as curl -vs https://accounts.example.org/sessions/whoami I got 502 Bad Gateway on the console, nginx logs says that the connection refused and nothing gets logged on the kratos end.

pinging from nginx container to kratos container gets a successful connection.

What is wrong?!

[fenandosr]

UPDATE

I forgot to post the kratos config, in extenso:

version: v0.13.0

serve:
  public:
    base_url: http://127.0.0.1:4433/
    host: 127.0.0.1
    cors:
      enabled: true
  admin:
    base_url: http://mcps_kratos:4434/
    host: 127.0.0.1

selfservice:
  default_browser_return_url: http://127.0.0.1:4433/
  allowed_return_urls:
    - http://127.0.0.1:4433

  methods:
    password:
      enabled: true
    totp:
      config:
        issuer: Kratos
      enabled: true
    lookup_secret:
      enabled: true
    link:
      enabled: true
    code:
      enabled: true

  flows:
    error:
      ui_url: http://127.0.0.1:4433/error

    settings:
      ui_url: http://127.0.0.1:4433/settings
      privileged_session_max_age: 15m
      required_aal: highest_available

    recovery:
      enabled: true
      ui_url: http://127.0.0.1:4433/recovery
      use: code

    verification:
      enabled: true
      ui_url: http://127.0.0.1:4433/verification
      use: code
      after:
        default_browser_return_url: http://127.0.0.1:4433/

    logout:
      after:
        default_browser_return_url: http://127.0.0.1:4433/login

    login:
      ui_url: http://127.0.0.1:4433/login
      lifespan: 10m

    registration:
      lifespan: 10m
      ui_url: http://127.0.0.1:4433/registration
      after:
        password:
          hooks:
            - hook: session
            - hook: show_verification_ui

log:
  level: debug
  format: text
  leak_sensitive_values: true

secrets:
  cookie:
    - [...]
  cipher:
    - [...]

ciphers:
  algorithm: xchacha20-poly1305

hashers:
  algorithm: bcrypt
  bcrypt:
    cost: 8

identity:
  default_schema_id: default
  schemas:
    - id: default
      url: file:///home/ory/identity.schema.json

[...]

[mtillberg]

The problem is that you're setting serve.public.host and serve.admin.host to 127.0.0.1 in the kratos config. That limits kratos to listening only on localhost, you won't be able to access it via the container's IP. If you remove the host: lines, your log should have something like:

level=info msg=Starting the public httpd on: 0.0.0.0:4433 func=github.com/ory/x/logrusx.(*Logger).Logf file=/go/pkg/mod/github.com/ory/[email protected]/logrusx/helper.go:125 audience=application service_name=Ory Kratos service_version=v1.1.0
level=info msg=Starting the admin httpd on: 0.0.0.0:4434 func=github.com/ory/x/logrusx.(*Logger).Logf file=/go/pkg/mod/github.com/ory/[email protected]/logrusx/helper.go:125 audience=application service_name=Ory Kratos service_version=v1.1.0

listening on 0.0.0.0 means listening on all available interfaces.

Additionally, all the *_url values are going to be used to create urls handled by the browser. All the http://127.0.0.1:4433 references should be https://accounts.example.org.