VCF9 – Offline Depot – Every day is a learning day

VMware Cloud Foundation is designed so that you will have the same experience if you’re online, or in a air-gapped bunker. Well, as close to as the same experience – There are tools available to make this as friction free as it can possibly be. This will be a complete guide on how to setup a Photon VM with all the bits.
Official documentation that this has been based on is availble here: https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/deployment/deploying-a-new-vmware-cloud-foundation-or-vmware-vsphere-foundation-private-cloud-/preparing-your-environment/downloading-binaries-to-the-vcf-installer-appliance/connect-to-an-offline-depot-to-download-binaries/set-up-an-offline-depot-web-server-for-vmware-cloud-foundation.html

Setup of Virtual Machine

First, download the Photon OVA that supports the latest virtual hardware from here: https://github.com/vmware/photon/wiki/Downloading-Photon-OS

Once downloaded, go ahead and deploy the OVA, preferrably in your Management/Consolidated workload domain. The deployment is rather simple since it doesn’t support any customization like static ip, or passwords.
Don’t select to power on the VM in the wizard, we need to add a drive first.
In my example, i am deploying the vm as depot-dev (depot-dev.rainpole.io)

Once the Photon VM has been deployed, edit the hardware and add another harddrive and add another harddisk with the recommended size of 1TB.
Now power on the VM.

Once the VM has powered on, use the following credentials to login to the virtual console of the VM.
username: root
password: changeme
You will be asked to enter a new secure password.

First step is to sort the networking and dns and hostname.
Create the file /etc/systemd/network/10-static-en.network in your editor of choice and enter.

[Match]
Name=eth0

[Network]
Address=172.16.10.14/24
Gateway=172.16.10.1
DNS=172.16.10.4 172.16.10.5

Update the hostname

hostnamectl set-hostname depot-dev.rainpole.io

Restart the network and dns lookup services.

systemctl restart systemd-networkd
systemctl restart systemd-resolved

The VM should now be able to communicate with your network, and dns servers. Now we’ll start by downloading some dependencies

tdnf install httpd tar jq --assumeyes

Well now format the disk that was added before starting the VM.

mkdir -p /var/www/html
mkfs.ext4 /dev/sdb
echo "/dev/sdb /var/www/html ext4 defaults 1 1" >> /etc/fstab
mount -a

Now we need to create the ssl certificate to be used by apache (httpd) to server all the files securely. We will start by creating a directory, creating the private key and a ssl configuration template, please edit bold values in the template to reflect your own values.

mkdir /root/http-certificates
openssl genpkey -out /root/http-certificates/server.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048

cat << 'EOF' > /root/http-certificates/ssl.cfg
[req]
distinguished_name = req_distinguished_name
req_extensions = req_ext
prompt = no

[req_distinguished_name]
C = SE
ST = Stockholm
L = Stockholm
O = Rainpole
OU = IT
CN = depot.rainpole.io

[req_ext]
subjectAltName = @alt_names

[alt_names]
IP.1 = 172.16.10.11
DNS.1 = depot.rainpole.io
EOF

Now we need to create a .csr using the .key and ssl template.

openssl req -new -key server.key -out server.csr -config ssl.cfg

we now have a .csr, use cat server.csr to get the base64 formated csr that can be used to obtain a certificate from your CA. The expected format to recieve back is a base64 formated full chain, often in a p7b format (Microsoft CA). If you get a base64 .crt back you can skip the below, otherwise convert it.

openssl pkcs7 -print_certs -in certnew.p7b -out server.crt

Now lets move the .key and .crt to a secure location that apache can use. For security we will alter filesystem permissions and owner.

mv server.key server.crt /etc/httpd/conf/

chmod 0400 /etc/httpd/conf/server.key /etc/httpd/conf/server.crt
chown root:root /etc/httpd/conf/server.key /etc/httpd/conf/server.crt

Now we need to make some modifications to the configuration files used by apache. Replace the values in bold with your values.

sed -i 's|#LoadModule ssl_module|LoadModule ssl_module|' /etc/httpd/conf/httpd.conf
sed -i 's|#LoadModule socache_shmcb_module|LoadModule socache_shmcb_module|' /etc/httpd/conf/httpd.conf
sed -i 's|#Include conf/extra/httpd-ssl.conf|Include conf/extra/httpd-ssl.conf|' /etc/httpd/conf/httpd.conf

sed -i 's|DocumentRoot "/etc/httpd/html"|DocumentRoot "/var/www/html"|' /etc/httpd/conf/extra/httpd-ssl.conf
sed -i 's|ServerAdmin you@example.com|ServerAdmin kim.johansson@rainpole.com|' /etc/httpd/conf/extra/httpd-ssl.conf
sed -i 's|ServerName www.example.com:443|ServerName depot-dev.rainpole.io:443|' /etc/httpd/conf/extra/httpd-ssl.conf

Now read this part closely, edit the file /etc/httpd/conf/extra/httpd-ssl.conf
At the very bottom of the file you will find </VirtualHost>. Just above this line, paste the following block of configuration.

<Directory /var/www/html/PROD/COMP>
        AuthType Basic
        AuthName "Basic Authentication"
        AuthUserFile /etc/httpd/conf/.htpasswd
        require valid-user
</Directory>
<Directory /var/www/html/PROD/metadata>
        AuthType Basic
    AuthName "Basic Authentication"
    AuthUserFile /etc/httpd/conf/.htpasswd
    require valid-user
</Directory>
<Directory "/var/www/html/PROD/COMP/Compatibility/VxrailCompatibilityData.json">
        # VxRail VVS Cookie Validation (VCF 5.0)
        <If "%{HTTP:Cookie} == 'ngssosession=ngsso-token' ">
        Require all granted
        </If>
</Directory>
<Directory /var/www/html/PROD/vsan/hcl>
        Require all granted
</Directory>
        # Those Alias statements are needed only for VCF 5.1.0.0.
        Alias /products/v1/bundles/lastupdatedtime /var/www/html/PROD/vsan/hcl/lastupdatedtime.json
        Alias /products/v1/bundles/all /var/www/html/PROD/vsan/hcl/all.json
        # Needed only if UMDS downloads are presented
<Directory /var/www/html/umds-patch-store>
        Require all granted
</Directory>

Now we need to create the Basic Authentication user and password that will be used. replace username with whatever you want the user to be called.

htpasswd -c /etc/httpd/conf/.htpasswd username
chown apache /etc/httpd/conf/.htpasswd
chmod 0400 /etc/httpd/conf/.htpasswd

Now let’s test the configuration, enable the service, and start the service.

httpd -t
systemctl start httpd
systemctl enable httpd

Now we need to fix the firewall, iptables is o by default. Edit the following file /etc/systemd/scripts/ip4save and make the bold changes.

# init
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
# Allow local-only connections
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
#keep commented till upgrade issues are sorted
#-A INPUT -j LOG --log-prefix "FIREWALL:INPUT "
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A OUTPUT -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
COMMIT

Now we lets restart service to make them take affect.

systemctl restart iptables

Some cleanup

rm -rf /root/http-certificates
rm -f /var/www/html/index.html

VMware Download Tool

Config is now basically done, now we need to download the vmware-download-tool, download the bits, and set some permissions.

This is where things will differ, you can be “semi-online” and use the vmware-download-tool on this VM:
– Because your security team don’t want VCF to have direct access to the internet
– Because you have multiple VCF instances and want to “cache” the files locally instead of wasting internet bandwidth

Or you are actually airgapped, and then you will be required to do the following on a VM/computer that does have internet access.

First download vmware-download-tool and upload it the machine where you intend to run it, the following documentation will be written to run on the depot vm, but should be easy to adapt.

mkdir -p /opt/vmware-download-tool
tar -zxvf vcf-download-tool-9.0.0.0.24703747.tar.gz -C /opt/vmware-download-tool/
echo "MYSUPERSECRETTOKEN OBTAINED IN THE SUPPORT PORTAL" >> /opt/vmware-download-tool/token

We’ve now setup the tool, depending on what bits you want to download, refer to the complete documentation here. Here’s an example to download the install packages for both infrastructure, and supporting features such as Operations, Automation, Logs, NetOps.

/opt/vmware-download-tool/bin/vcf-download-tool binaries download --depot-store=/var/www/html/ --depot-download-token-file=/opt/vmware-download-tool/token --vcf-version="9.0.0.0" --lifecycle-managed-by=SDDC_MANAGER_VCF --type=INSTALL

/opt/vmware-download-tool/bin/vcf-download-tool binaries download --depot-store=/var/www/html/ --depot-download-token-file=/opt/vmware-download-tool/token --vcf-version="9.0.0.0" --lifecycle-managed-by=VRSLCM --type=INSTALL

/opt/vmware-download-tool/bin/vcf-download-tool binaries download --depot-store=/var/www/html/ --depot-download-token-file=/root/vcf-download/token --vcf-version="9.0.0.0" --automated-install

If you’ve downloaded the bits on a different machine, transfer the files to to /var/www/html on the depot using scp.

The very last step of the depot configuration is to create the correct permissions. The last two commands should be run everytime you update the depot with new bits.

chown apache -R /var/www
find /var/www/html -type d -exec chmod 0500 {} \;
find /var/www/html -type f -exec chmod 0400 {} \;

Whats next? Well, we need to configure Operations and SDDC-Manager to use the new depot.

Login to your Operations console and navigate to Fleet Management -> Lifecycle.
Under VCF Management select Depot Configuration.
In the Offline Depot section, select Configure.
Leave OfflineDepotType: Webserver selected.
in Repository URL enter your depot (https://depot-dev.rainpole.io)
in Username enter the username you chose when creating your httpasswd.
in Password, create a new password, and select it.
Since the function that Operations uses to communicate with the depot doesn’t trust your CA, you will need to select the checkbox I accept the imported certificate.
Before clicking Ok, select VIEW CERTIFICATE DETAILS to verify that the certificate displays correctly.
Now click OK.

If you now navigate to the Binary Management -> Install Binaries menu, you can now select to download the binaries for Operations, Automation, Logs and NetOps.

Now we need to configure the infrastructure repository. To do this we first need to get the sddc-manager appliance to trust the certificate. Currently, this isn’t as easy.

Open the sddc-manager in your browser and login.
Select Developer Center in the left menu.
In the Filter option to the right, enter trust and hit enter.
There should be 3 items displayed, expand the POST one.

Now, ssh to your depot VM (depot-dev.rainpole.), we’re going to grab the public key to trust by running the following command.

echo '{ "certificate" : '$(jq -sR . /etc/httpd/conf/server.crt)',
  "certificateUsageType" : "TRUSTED_FOR_OUTBOUND"
}'

copy the entire output, but be careful not to also grab the three rows you just entered. Example below.

{ "certificate" : "subject=C = SE, ST = Stockholm, L = Stockholm, O = Rainpole, OU = IT, CN = depot-dev.rainpole.io\nissuer=DC = io, DC = rainpole, CN = rainpole-RPL-AD01-CA\n-----BEGIN CERTIFICATE-----\nMIIFnDCCBISgAw......MpEkV22x6geVze1RvRzCm/3EfOvkeoreg==\n-----END CERTIFICATE-----\n\n",
  "certificateUsageType" : "TRUSTED_FOR_OUTBOUND"
}

Paste the text into the the body input in the sddc-manager, and select Execute.
Within a second, you should get a response without an error.
There should be a PageOfTrustedCertificate, and at least two TrustedCertificate in the tree below it. You can select the TrustedCertificate to expand them, and see that it’s registered your CA, any intermediates, and depot as trusted endpoints.

Now go back to Operations, under Fleet Management -> Lifecycle, expand the VCF Instances menu and select the instance you want to configure, and then select Depot Settings.
Under Offline Depot select Set Up.

Enter the required information, this time you dont need to enter “https://”, just the hostname.
Select Save.

You’ve now successfully configured both the Infrastructure, and fleet management part of Cloud Foundation 9.

Troubleshooting

To write this, i had to do some troubleshooting, here’s some tips.

Edit the verbose level of apache by editing /etc/httpd/httpd.conf
modify the following.

LogLevel debug

now restart the service

systemctl restart httpd

now tail this file to check requests.

tail -f /var/log/httpd/ssl_request_log

If you get permission denied because of filesystem rights, this is a good tool. Replace with the file that is trying to be accessed but denied. This will alert you to if a dir is owned by the incorrect user, or a rwx permission doesn’t look right.

namei -mol /var/www/html/PROD/metadata/manifest/v1/vcfManifest.json

f: /var/www/html/PROD/metadata/manifest/v1/vcfManifest.json
drwxr-xr-x root   root /
drwxr-xr-x root   root var
drwxr-x--- apache root www
dr-x------ apache root html
dr-x------ apache root PROD
dr-x------ apache root metadata
dr-x------ apache root manifest
dr-x------ apache root v1
-r-------- apache root vcfManifest.json

Setup of Virtual Machine

VMware Download Tool

Troubleshooting

Share this:

Related

Published by Kim Johansson

Leave a comment Cancel reply