NSX Application Platform Part 2: Harbor Image Registry

Harbor Image Registry Configuration

The Harbor image registry is critical to the successful deployment of the NSX Application Platform (NAPP). It holds all the images and binaries required for the application platform, which are pulled as pods are being deployed.

There are many ways to deploy Harbor on-premises including, but not limited to:

• Using Helm to deploy a HA instance of Harbor on kubernetes
• Tanzu Application Service with Harbor Integration
• Deploying the Harbor image registry on a virtual machine

This guide will walkthrough deploying Harbor on a Ubuntu virtual machine.

I have recently put together a video that provides clear guidance on deploying NAPP, it can be seen here.

42:12

NSX Application Platform (NAPP) // NSX Advanced Load Balancer (NSX ALB)

Understand NSX Application Platform (NAPP) architecture and how to deploy it using NSX Advanced Load...

Virtual Machine Configuration

This section details the specifications of the Ubuntu virtual machine that I have deployed for use as my Harbor image registry.

• Ubuntu 20.0.4.3 Focal Fossa
• 2 vCPU/8GB RAM
• 2 Disks attached, one for the OS (40GB) and one for data (150GB)
• Single network interface on my management network (IP address 192.168.63.100)
• VM has internet access to generate certificates and pull files for install

Note: This article will not cover the Ubuntu deployment process.

Prerequisites – Installing Docker

Prior to installing Harbor, the Ubuntu VM must meet the prerequisites, all of which can be found here.

The official steps from Docker can be found here. The commands I ran are provided in the output below.

root@harbor:/# apt-get update

root@harbor:/# apt-get install \
>     ca-certificates \
>     curl \
>     gnupg \
>     lsb-release

root@harbor:/#  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

root@harbor:/#  echo \
>   "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
>   $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

root@harbor:/# apt-get update  <--------- DO NOT SKIP THIS
root@harbor:/# apt-get install docker-ce docker-ce-cli containerd.io

Next, check that the service is running.

Finally, I complete the prerequisite installation by installing Docker Compose.

root@harbor:/mnt/data/harbor# sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   664  100   664    0     0  14434      0 --:--:-- --:--:-- --:--:-- 14434
100 11.6M  100 11.6M    0     0  14.6M      0 --:--:-- --:--:-- --:--:-- 25.8M
root@harbor:/mnt/data/harbor# sudo chmod +x /usr/local/bin/docker-compose
root@harbor:/mnt/data/harbor# docker-compose --version
docker-compose version 1.27.4, build 40524192

Harbor Repository Certificate Configuration

In this article I utilize a Lets Encrypt certificate for my Harbor registry.

Note: as of NSX-T 3.2 release (Impactor), self-signed certificates, including Active Directory issued certificates are not supported.

The certificates used for the registry must be signed by a trusted certificate authority. If you do not utilize a trusted certificate, when attempting to deploy NAPP, you will be faced with an x509 certificate issue, similar to the below output.

  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  11m                   default-scheduler  Successfully assigned cert-manager/cert-manager-69cc999bb5-khjws to impactorlab-workers-snmxl-dc89f6748-s9p4p
  Normal   Pulling    10m (x4 over 11m)     kubelet            Pulling image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763"
  Warning  Failed     10m (x4 over 11m)     kubelet            Failed to pull image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763": rpc error: code = Unknown desc = failed to pull and unpack image "harbor.shank.com/impactor/clustering/third-part
y/cert-manager-controller:19067763": failed to resolve reference "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763": failed to do request: Head "https://harbor.shank.com/v2/impactor/clustering/third-party/cert-manager-controller/manifests/19067763":
 x509: certificate signed by unknown authority
  Warning  Failed     10m (x4 over 11m)     kubelet            Error: ErrImagePull
  Warning  Failed     6m54s (x21 over 11m)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    108s (x44 over 11m)   kubelet            Back-off pulling image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763"

You will still be presented with this issue if you utilize the workaround here. This is because the underlying docker / containerd still doesn’t trust the certificate and chain. There is potential to work around this issue, however, I will not get into that here. If you absolutely require the use of a self-signed certificate, get in contact via Twitter, LinkedIn, or email.

Generating a Lets Encrypt Certificate

First, I install Lets Encrypt on my virtual appliance. I have listed the commands I ran in the output below. Remember, this command is being run on Ubuntu, you will need to edit the command to suit your operating system.

### Install the package and all its dependencies 
root@harbor:/home/harbor# apt install letsencrypt

### Check to ensure it is running
root@harbor:/home/harbor# systemctl status certbot.timer

Now that Lets Encrypt is installed, I generate the certificate for the appliance. Keep in mind that this virtual appliance has internet connectivity, and I have created a DNS A record with my hosting provider for my appliance.

Issue the command below on your Harbor appliance, and follow the prompts on screen.

## Change the value after -d to match the hostname of your Harbor appliance.
root@harbor:/home/harbor# certbot certonly --standalone -d harbor.lab2prod.com.au

As per the prompt, the certificates have been generated and are available in /etc/letsencrypt/live/harbor.lab2prod.com.au/.

Setup and configure Harbor

Download the offline Harbor installer package located here, and transfer it to the Harbor appliance.

Extract the archive using the command below.

root@harbor:/mnt/data# tar xzvf harbor-offline-installer-v2.4.1.tgz

Before continuing, I realised that the Docker data was being stored in /var/lib/docker, this wasn’t going to work for me as it was running on the smaller 40GB drive. I had to move the Docker data to the larger partition, to do so I followed this guide.

Now, copy the Harbor configuration file that is in the uncompressed Harbor installer folder or rename it.

root@harbor:/mnt/data/harbor# cp harbor.yml.tmpl harbor.yml

Edit the yml file to reflect your environment details, the output below are the lines that need to be changed.

root@harbor:/mnt/data/harbor# vi harbor.yml
#### The main options that I changed are
hostname: reg.mydomain.com
  certificate: /your/certificate/path
  private_key: /your/private/key/path
data_volume: /data

If you would like to see my harbor.yml it is located here.

Next, I install Harbor by running the command in the output below.

Note: Do not skip installing chartmuseum, it is required for NAPP.

root@harbor:/mnt/data/harbor# ./install.sh --with-chartmuseum

It will go through 5 steps to install and configure harbor and you should see something similar to the screen below once it is complete.

You should now be able to access the Harbor registry UI, it is also worthwhile to check that the certificate was properly installed and the site is secure.

Uploading NSX Application Platform Binaries to the Harbor Registry

The final part of this article is to push the images to the Harbor registry.

First I created a new project, to do so, login to Harbor and click on create a new project.

Once the project is created, the images can be uploaded. In my environment I chose to utilize a jumpbox which has all the tools I required pre-installed, as well as the NAPP binaries transferred and uncompressed. Details and configuration of this jump box can be found in the first article of this series.

Navigate to the folder that contains the uncompressed NAPP binaries, there should be a file called upload_artifacts_to_private_harbor.sh. Open this file in a text editor, in my case I used vim.

Change the first three lines to suit your environment, instructions on the VMware website for this can be found here.

DOCKER_REPO=harbor.lab2prod.com.au/impactor
DOCKER_USERNAME=admin
DOCKER_PASSWORD=Harbor12345

Once you have made those changes, run the script to start the upload process.

Ignore the errors you see regarding a connection, this is specific to my environment. At this point you can leave the images to be pushed to your Harbor registry, the time this takes can vary depending on your environment.

Once this process completes, you should see something similar on your jumpbox or whatever you chose to upload the images from.

The repository should list 80 repositories and 18 charts.

And that’s it for the Harbor repository deployment!

Next Steps

NSX Application Platform Part 3: NSX-T, NSX-ALB (Avi), and Tanzu

The next part in this series focusses on NSX-T, NSX-ALB, and Tanzu.

NSX Application Platform Part 4: Deploying the Application Platform

The final part of the series demonstrates the deployment process for NSX Application Platform and its security features (NSX Intelligence, Network Detection and Response, and Malware Prevention.

Previous Article

NSX Application Platform Part 1: Environment Overview

This was the first article in the series, it provides an overview to the environment.