env_prefix is ignored if pydantic_settings finds an env var without prefix

[hofrob]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

When using env_prefix in a model_config, pydantic_settings will still load values from env vars that do not use the given prefix. Values from env vars without the prefix even have higher priority.

Example repo: https://github.com/hofrob/pydantic-settings-env-prefix

• clone
• run uv run hello

HELLO=ABC should just be ignored in my opinion (can't find anything definitive in the docs).

Example Code

import pydantic_settings


class Settings(pydantic_settings.BaseSettings):
    model_config = pydantic_settings.SettingsConfigDict(
        env_file=".env",
        extra="ignore",
        env_prefix="some_prefix_",
    )

    hello: str | None = None


def hello() -> None:
    print(Settings())

# this should only the show the value of `SOME_PREFIX_HELLO`
# output: hello='ABC'

Python, Pydantic & OS Version

pydantic version: 2.9.2
        pydantic-core version: 2.23.4
          pydantic-core build: profile=release pgo=false
                 install path: /path/to/pydantic-settings-env-prefix/.venv/lib/python3.12/site-packages/pydantic
               python version: 3.12.6 (main, Sep  9 2024, 22:11:19) [Clang 18.1.8 ]
                     platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.39
             related packages: typing_extensions-4.12.2 pydantic-settings-2.5.2
                       commit: unknown

[hramezani]

Thanks @hofrob for reporting this bug. I will prepare a fix for it.

[hramezani]

@hofrob I created #440 to fix the bug. Could you please confirm the fix?

[hofrob]

I checked the following:

• re-order env vars in .env
• run HELLO=321 uv run hello
• SOME_PREFIX_HELLO=321 uv run hello

All show the expected results!

bug? remove the prefixed variable

Removing the prefixed variable falls back to the other variable which I think is a problem. Potentially even a security risk.

bug? .env var is prioritized over env var

If the .env file only contains:

HELLO=ABC

and I run HELLO=321 uv run hello the result is: hello='ABC'

I think this is still a bug because either it has no value (because no prefixed variable exists which I would prefer), or it has the value of the env var.

Thank you and sorry for creating this issue in the wrong project!

[hramezani]

Thanks @hofrob for quick feedback!

[hofrob]

Should I create separate issues for the other ones? Are they not considered bugs? Just using a different env var when it's explicitly configured to use a prefix sounds like an actual problem to me.

[hramezani]

This happens because the dotenv settings source must consider the extra envs provided in the dotenv file because dotenv settings sources cares about extra config and has to raise error in case of extra env in file.

So, pydantic-settings has to collect all the values in dotenv file. please create a new issue.