vCenter gets unresponsive after adding a DVS Port Group

During the creation of a new VLAN for my nested environment I encounter a really strange behavior. When I tried to add a Distributed Port Group, my vCenter became unresponsive. After some seconds it was back again. I checked the port group and it wasn’t created. I analysed the vpxd.log and found the following error messages:

LError was thrown: “ODBC error: (23505) – ERROR: duplicate key value violates unique constraint “pk_vpx_dvport_membership”
–> Key (dvs_id, dvport_key)=(11, 0) already exists.;
–> Error while executing the query” is returned when executing SQL statement “INSERT INTO VPX_DVPORT_MEMBERSHIP (DVS_ID, DVPORT_KEY, DVPORTGROUP_ID, HOST_ID, LAG_KEY) VALUES (?, ?, ?, ?, ?)”
Received unrecoverable VmRootError. Generating minidump …
An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) “ODBC error: (23505) – ERROR: duplicate key value violates unique constraint “pk_vpx_dvport_membership”
–> Key (dvs_id, dvport_key)=(11, 0) already exists.;
–> Error while executing the query” is returned when executing SQL statement “INSERT INTO VPX_DVPORT_MEMBERSHIP (DVS_ID, DVPORT_KEY, DVPORTGROUP_ID, HOST_ID, LAG_KEY) VALUES (?, ?, ?, ?, ?)”
–> Panic: Unrecoverable VmRootError. Panic!

It took some time until I found this KB article describing the problem and having a solution for a vCenter for Windows and a SQL Server database. The problem now was that I’m running vCenter Server Appliance and I needed a solution for a vPostgres SQL database. The following steps describes how to fix the issue on a vCenter Server Appliance with embedded vPostgres database.
As the KB article describes this can happen when you backup and restore a DVS configuration to a new vCenter Server instance.

• Login to the affected Appliance
• Use shell.set –enable true and shell to enter the OS level file system.
• The password for the vPostgres instance can be found under /etc/vmware-vpx/embedded_db.cfg (PGUSER_PASSWORD). Copy the password string without the single quotation marks.
• To open the vPostgres vCenter database enter the following command – sudo -u postgres /opt/vmware/vpostgres/9.3/bin/psql -d VCDB postgres and enter the previous copied password.
• \dt – shows all tables
• The following command will give you information about the existing Distributed Switches select id, name, uuid, upgrade_mode,port_counter from VPX_DVS;. The most important thing when using sql statements is that you shouldn’t forget the semicolon at the end of the commando.

• As you can see the value for upgrade_mode is 0 and the port_counter is 16. Ths should be changed after creating new port groups. The problem is that there is already an existing row with the port_counter value in the VPX_DVPORT_MEMBERSHIP table. To fix the issue the following command must be used:

update VPX_DVS set upgrade_mode=1 where id = 11;

NOTE: The command you would be using can vary a little bit.

• After a vCenter Service restart (service vmware-vpxd restart) creating new port groups will work again. Also after creating an additional port group the port_counter value will be replace with a correct value.