VMware vCenter Appliance 6.7 (VCSA) SSL Certificates using Let’s Encrypt

By | 2019-02-03

From time to time I got nerved about the missing trust of my demo VMware vCenter Server. Since I’m using Let’s Encrypt (LE) for years for my blog and other sites, I thought about also using this nice service for my vCenter Server:

After playing around a bit with the vSphere 6.7 Certificate Manager, I found a nice (but dirty) solution:

The following is a very quick & dirty solution and I will not explain what the Certificate Manager does exactly. If you are interested in all of its features, please just read the documentation first 🙂

Let’s start. First of all, you need two requirements: you cannot run the LE stuff on your vCenter Server Appliance, so you need a linux host for doing this. Because you are running LE on a different ip than your vCenter Server, you need a DNS record of type TXT for verification. Let’s start on our linux host:

linux:/root # certbot certonly --manual --preferred-challenges=dns -d vcenter.ich-halt.net

While using this wizard, you need to create your TXT DNS record, for example:

linux:/root # dig TXT _acme-challenge.vcenter.ich-halt.net +short
"xxxxxxxx"

After the LE wizard ran successfully, you got the three needed files for your vCenter Server:

linux:/etc/letsencrypt/live/vcenter.ich-halt.net # ls
(...) cert.pem  fullchain.pem  privkey.pem

Just copy these files to your vCenter Server Appliance, switch to its shell (login via ssh) and run the Certificate Manager:

root@vcenter [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager
  1. Replace Machine SSL certificate with Custom Certificate

Option[1 to 8]: 1

2. Import custom certificate(s) and key(s) to replace existing Machine SSL certificate

Option [1 or 2]: 2

Please provide valid custom certificate for Machine SSL.
File : /root/cert.pem


Please provide valid custom key for Machine SSL.
File : /root/privkey.pem


Please provide the signing certificate of the Machine SSL certificate
File : /root/fullchain.pem


You are going to replace Machine SSL cert using custom cert
Continue operation : Option[Y/N] ? : y

For vCenter Server Appliance 6.5 this works fine. But (in my setup) with version 6.7 it doesn’t. You get some errors and the Certificate Manager will perform a revert of your changes:

Updated 0 service(s)
Status : 85% Completed [starting services...]                  
Error while starting services, please see service-control log for more details
Status : 0% Completed [Operation failed, performing automatic rollback]
                 
Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

Performing rollback of Machine SSL Cert...

To prevent from this revert, you need to press CTRL + C instant after this “Updated 0 service(s)” and reboot your VCSA with “reboot”.

(...)
Get service 86b7419e-7b41-4aa1-9bc6-5eb01cb860ad
Don't update service 86b7419e-7b41-4aa1-9bc6-5eb01cb860ad
Updated 0 service(s)

After coming up again, all my services started correct and are using the LE certifiate:

Next step will be the challenge to automate the LE certificate renewal 🙂

Appendix: maybe it isn’t a generell vSphere 6.7 problem – now I have found a VCSA 6.7 which worked like 6.5 without problems …

8 thoughts on “VMware vCenter Appliance 6.7 (VCSA) SSL Certificates using Let’s Encrypt

  1. Joseph

    Did you ever manage to figure out why the service startup failed at 85% and a rollback was initiated? I’m having the same problem on vCenter 6.7.

    Reply
  2. W

    I had similar issue,
    rollback starts, becouse service vpxd won’t start.
    cause?
    > ExpectedPeerName: localhost
    > Host name does not match the subject name(s) in certificate.)

    but why it expect ‘localhost’ SSL certificate?! I don’t know.

    Reply
  3. Laubi

    I had similar issue,
    for me, the error also occurred in 6.7, meanwhile I believe in this post:
    https://www.reddit.com/r/vmware/comments/8dk1y3/as_vcenter_67_requires_a_valid_ssl_certificate_i/
    There a:
    /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish –chain –cert /tmp/LE-root-CA.pem
    solved my problem for second call after this-.

    @alex: Warum schreibst Du auch englisch, and please excuse my bad English. Its not my, its from google 😉

    PS:

    Meanwhile, it works automatically for me.
    Unfortunately, the implementation is not very fancy, so I have to work out the documentation a bit.
    If I have documented it I like to report to you;)

    Reply
  4. Callan Christensen

    I came across this article while I was having issues with replacing my Machine SSL Cert. I wanted some clear examples for comparison as a sanity check.

    Here was my error:
    ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
    2020-01-01T22:03:29.653Z ERROR certificate-manager {
    “problemId”: null,
    “resolution”: null,
    “detail”: [
    {
    “localized”: “An error occurred while invoking external command : ‘None'”,
    “id”: “install.ciscommon.command.errinvoke”,
    “args”: [
    “None”
    ],
    “translatable”: “An error occurred while invoking external command : ‘%(0)s'”
    },
    “Error while starting services, please see service-control log for more details”
    ],
    “componentKey”: null
    }
    2020-01-01T22:03:29.654Z INFO certificate-manager Performing rollback of Machine SSL Cert…

    Cause: I had old intermediate root certificates in the VMWare trusted certificate store

    Solution: I had to get VMWare support on the line and they removed each of the expired root certs. Then we added the new machine and intermediate certs. After that, they also had to use a python script from their internal KB to repair a bunch of VMWare services that needed to be re-registered to the new cert.

    I unfortunately don’t have these commands, but maybe this info would be helpful for someone.

    Reply
  5. Brian Dang

    thank you! this works great! I had issues with 6.7 replacing machine ssl stopping at 85% and rolling back. And even the rolling back action wasn’t even successful. Janky software!

    Reply
  6. Paquis

    I followed the procedure on my vcenter 6.7 but the let’s encrypt certificate did not replace the machine certificate, it just added to the list of trusted certificates. I specify that I did not get an error message during the installation

    Reply
  7. Alex

    Hello Alex. My name is Mia. Your solution helped me. I was compelled to write to you because you saved me from so many hours I have already spent in trying to fix this issue and I was ultimately going to give up and create a new vCenter server. I’m using 6.7. I had a snapshot before I troubleshot anything. I don’t use Let’s Encrypt. I was using option 8 where I needed to replace the machine and user solution SSL from the vCetner Server manager. I entered the admin username and password, selected Y to start the process, N to reconfigure the cert file, Y to config all of the SSL, and Y to run the operation. I hit Ctrl +C when it was at 85% of stating the services. I rebooted the server thinking that I made the changes at the wrong time and will need to do it again, To my surprise I can get to the website and the cert is up to date.

    I was actually really skeptical about trying your method but I had nothing else to lose. I had tried so many things… I actually wrote out all of my troubleshooting steps and this was my 14th attempt and it worked. Techs should learn about this solution as an option. It would save so many people.

    Thanks again for this resource and your blog.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.