NICs being configured twice on Rocky Linux 9

[bab5470]

Describe the bug

I have a terraform script that performs a clone operation on a VMware template created with packer. In the terraform script I add two NICs to the existing VM (for a total of 3) and then configure IP information on all three.

What I see is that the new NICs in Rocky Linux are configured but duplicated:

ens256
VMware customization ens256
ens224
VMware customization ens224
ens192
VMware customization ens192

I am running openvmtools 12.3.5 on the system. It seems like network connections are getting created in two places:

The ens connections are created in /etc/sysconfig/network-scripts, and the "Vmware Customizations" connections are created in /etc/Network-Manager/system-connections

I don't think connections are supposed to be created in both locations and obviously, we don't want our NIC/connections named "VMware customization xxxx"

Is this a bug in the openvmtools customization process?

Reproduction steps

I am running a terraform script which in turn is calling open-vm-tools to configure networking. The logs are attached below. The scripts can be provided if that would help but since this is an openvmtools bug tracker I'm not sure if you care to see those or not.

Expected behavior

A single set of interfaces are connected and configured.

Additional context

toolsDeployPkg.log

[PengpengSun]

Hi @bab5470,
Since vSphere 8.0 U3 release. Guest OS Customization will write network configuration to /etc/NetworkManager/system-connections/VMware-customization-{NAME}.connection and /etc/sysconfig/network-scripts/ifcfg-{NAME}. {NAME} is the connection name. Please check KB article: https://knowledge.broadcom.com/external/article/311956.

In your case:
NetworkManager loads keyfiles from: (btw: connection names are still eth192,eth224, eth256)
/etc/NetworkManager/system-connections/VMware-customization-eth192.nmconnection
/etc/NetworkManager/system-connections/VMware-customization-eth224.nmconnection
/etc/NetworkManager/system-connections/VMware-customization-eth256.nmconnection

NetworkManager loads ifcfg-rh from:
/etc/sysconfig/network-scripts/ifcfg-eth192
/etc/sysconfig/network-scripts/ifcfg-eth224
/etc/sysconfig/network-scripts/ifcfg-eth256

So you will see 6 connections by running command nmcli connection show, while NetworkManager only activates either keyfile or ifcfg-rh, you can check the current active connections by running command nmcli connection show --active

This change is due to NetworkManager default plugin is keyfile[1] since RHEL9 and RHEL9-like Linux, while ifcfg-rh plugin is still supported although RedHat is deprecating it. If you check NetworkManager configuration file /etc/NetworkManager/NetworkManager.conf, you will see plugins=keyfile,ifcfg-rh. When multiple plugins are specified, the connections are read from all listed plugins. When writing connections, the plugins will be asked to save the connection in the order listed. On startup, NetworkManager always tries to activate the most suitable persistent connection (the one with highest autoconnect-priority or, in case of a tie, the one activated most recently). [2]

[1] https://networkmanager.dev/docs/api/latest/nm-settings-keyfile.html
[2] https://linux.die.net/man/5/networkmanager.conf

[bab5470]

Thank you for the response. I appreciate it.

From an end user perspective, this is a mess. Why would I want my nics prefaced with "VMware-customization" and doubled up? That's not a very user-friendly name to say or type. And who wants to deal with two different systems simultaneously?

I ended up doing this convoluted mess to clean things up which works but took me days to figure out:

  provisioner "remote-exec" {
    inline = [
      "nmcli connection migrate",
      "rm -f /etc/NetworkManager/system-connections/VMware-customization-ens*",
      "PRIMARY_NIC=$(ip route | grep default | awk '{print $5}')",
      "nmcli con mod \"$PRIMARY_NIC\" ipv4.dns \"${var.dns_server_list[0]} ${var.dns_server_list[1]}\"",
      "nmcli con mod \"$PRIMARY_NIC\" ipv4.dns-search \"${var.dns_domain}\"",
      "reboot"
    ]
  }

This ends up migrating everything to Network Manager, removing the "VMware customization NICs" and sets the DNS servers and DNS domains - but what a convoluted work around!

I assume RedHat is to blame for this half-baked migration but what a frustrating mess to deal with. This is not intuitive at all!

[PengpengSun]

Hi @bab5470

The 'VMware-customization-ethXXX.nmconnection' is the keyfile name, the "VMware customization ethXXX" is the connection id, the nic name is still ethXXX, the nic name is configurated in keyfile at "interface-name=ethXXX".

And Yes, although RedHat is deprecating ifcfg-rh plugin, RHEL9 does support both plugins by default, and I do see customers are confused about this state. nmcli connection migrate works if ifcfg-rh is still there.

If direct to NetworkManager keyfile, how about removing ifcfg-rh plugin by modifying /etc/NetworkManager.conf as below, this will make sure NetworkManager only loads keyfile, then ip configuration, DNS server and search domain configured in keyfile will take effect after clone and customize your template.
plugins = keyfile