VMware Cloud Foundations 4.0.1: Problems with SDDC Manager refreshing

I’ve been doing some studying on VMware Cloud Foundations 4.0.1 and have it running in my lab. It seems a bit finicky at times I’ve noticed. One of the issues I’ve run into so far is that when I added 3 more hosts, everything seemed to be fine. I then wanted to add a third NIC to the hosts in order to access ISCSI storage on them. When I create the NIC though (while the nested host was turned on) it locked up my physical host and ended up needing to reboot it. Not nice….

Anyways I got that sorted. The next issue I ended up with was SDDC manager didn’t want to refresh or connect to vCenter since it wasn’t shut down properly. I started doing some research and HUGE shoutout to vSAM.pro for figuring out what was going on. I ended up having to do a bit more though. So here is what I ended up doing (and his blogs made much more sense after I figured it out weirdly enough 🙂 )

The Issue:

First, I Checked logs. These logs are located at

/var/log/vmware/vcf/

Underneath there is a bunch of service folders with logs in them. In this particular case I checked through most of the logs and found the following issue

root@sddc-manager [ /var/log/vmware/vcf/operationsmanager ]# tail operationsmanager.log

2020-08-03T22:57:50.268+0000 INFO [0000000000000000,0000] [liquibase.executor.jvm.JdbcExecutor,main] SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1
2020-08-03T22:57:50.270+0000 INFO [0000000000000000,0000] [l.lockservice.StandardLockService,main] Waiting for changelog lock….
2020-08-03T22:58:00.270+0000 INFO [0000000000000000,0000] [liquibase.executor.jvm.JdbcExecutor,main] SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1
2020-08-03T22:58:00.273+0000 INFO [0000000000000000,0000] [l.lockservice.StandardLockService,main] Waiting for changelog lock….
2020-08-03T22:58:10.273+0000 INFO [0000000000000000,0000] [liquibase.executor.jvm.JdbcExecutor,main] SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1
2020-08-03T22:58:10.276+0000 INFO [0000000000000000,0000] [l.lockservice.StandardLockService,main] Waiting for changelog lock….
2020-08-03T22:58:20.277+0000 INFO [0000000000000000,0000] [liquibase.executor.jvm.JdbcExecutor,main] SELECT LOCKED FROM public.databasechangeloglock WHERE ID=
2020-08-03T22:58:20.278+0000 INFO [0000000000000000,0000] [l.lockservice.StandardLockService,main] Waiting for changelog lock….
2020-08-03T22:58:30.279+0000 INFO [0000000000000000,0000] [liquibase.executor.jvm.JdbcExecutor,main] SELECT LOCKED FROM public.databasechangeloglock WHERE ID=1
2020-08-03T22:58:30.281+0000 INFO [0000000000000000,0000] [l.lockservice.StandardLockService,main] Waiting for changelog lock….

It looked like there was an issue with a locked file somewhere.

In order to check the database, you need to go to the DB command line

psql –host=localhost -U postgres

Then you need to change to the database that is locked. You can list the DBs by typing

\l (That is an L)

To change databases – type the following

\c [database_name] without the brackets

This will allow you to run the command

\dt

Which shows tables for that DB in this case of the OperationsManager it looked like this:

Next you need to see if there is something in that table so type the following:

select * from databasechangeloglock

And it will return the following:

You can now delete this table by doing typing the following

delete from databasechangeloglock;

And it will kill the lock. Give the SDDC manager a few minutes and it should start working again.