This is the last part in my Getting Started series for Enterprise PKS. In this article I am going to be looking at how we deploy and utilize the VMware Harbor registry with PKS to store container images, enabling users to push to and pull from Harbor as required.
Other articles in this series include:
- Part 1 Overview
- Part 2 Ops Manager and Bosh
- Part 3 NSX-T Prep
- Part 4 Installing PKS
- Part 5 Granting PKS Access
- Part 6 Cluster Operations
- Part 7 Cluster Access
- Part 8 Harbor Registry
What Is It?
Before I get to installing and configuring Harbor lets just take a minute to cover what it actually is. As I stated in the opening paragraph Harbor is a container image store that allows users of the PKS environment (and the Kubernetes clusters within) to push and pull images to and from it as required. These container images are the basis of any pods and deployments that users deploy onto Kubernetes clusters.
Harbor itself is actually a containerized application made up of multiple services. The following diagram from the Harbor github site can explain things far more eloquently than I can.
The registry is sub-divided into admin defined projects with each project having specific members and access permissions via RBAC. Users can then user the docker CLI to push or pull their images to or from specific projects within the registry assuming they have the appropriate access permissions on the project to do so.
In addition Harbor can be configured to replicate its contents to other registries that may also be on-prem or in the cloud somewhere.
As you will no doubt be aware now, anything covered within Enterprise PKS is deployed from Pivotal Ops Manager. This means I need a pivotal image for Harbor and a stemcell to run the application. Both can be obtained from the Pivotal network (network.pivotal.io) with the release details of the harbor image linking me to the correct stemcell version.
I have already uploaded the required stemcell version previously to my Ops Manager so I’m going to move straight to installing the Harbor tile into Ops Manager.
Although the tile is installed it still needs to be added to the dashboard using the plus icon from the left menu bar.
Once added to the dashboard Harbor shows up as orange signifying that it is not configured. In my screenshot there is no stemcell hyperlink on the tile as I have already uploaded the stemcell required however if you have not done this then a hyperlink will display linking you to the stemcell library within Ops Manager.
The sections within the Harbor tile now allow me to configure the specifics for Harbor to be deployed with. I am placing Harbor on the same NSX-T logical switch as the other PKS control components (Ops Man, Bosh Director and PKS). Harbor is also a singleton (i.e. it’s a one VM platform) so the AZ I select is the same for singleton and balance.
Harbor needs to have a FQDN and IP address for users to be able to interact with it. The FQDN is applied to the Harbor VM. The IP I am assigning is one that is valid for my PKS-MGMT logical switch and not reserved for anything else.
I have the option here to customize the networking for the containers that make up the Harbor application. I have no requirement to do this so I am leaving the container networking as default.
I could use a self-signed SSL certificate at this point generated by Ops Manager however I am choosing to do things properly with the PEM keys below being issued from my MS CA along with the CA that represents my CA.
Note that my public key entry in the top window also includes the full issuing chain however I am unclear whether this is actually required or not given that the issuer cert is also provided in a separate box. It seems to do no harm though!
Harbor needs a default administrator account (called admin) so a password for this account is provided on the next page.
There are several options for authentication. If you look at the reference diagram for Harbor at the top of this page you will see there are 2 authentication options, AD/LDAP or OIDC. However, with the PKS integration the selections are slightly different allowing me to select to use Internal, LDAP or UAA from either PKS or PAS. I don’t have Pivotal PAS here and Internal/LDAP seems a bit ordinary so I’m going to use UAA for PKS. This means I am going to be relying on UAA within the PKS control VM to grant me access (i.e I need to be in a group that is defined as a cluster admin or manager). There is an implication to this decision which you will see further on.
The container images need to be stored somewhere from local storage through to some form of cloud provided storage. My implementation is simple so I am going to use the local storage of the Harbor VM.
One of the big parts of an image registry is to scan images that are checked into the registry for vulnerabilities before they are made available for other people to use to make sure there are no known security flaws that other users of the image could fall fowl of. This is optional but a nice feature so I’m turning this on.
Harbor can be monitored by VMware Wavefront if desired however I’m keep things simple. Remember you can come back and update any of this configuration at a later date (note this will re-deploy the VM but keep the persistent data).
As with all BOSH deployed VMs the Harbor VM can be smoke tested as it is deployed to verify it is functioning as expected. This will include test push and pull operations to and from the registry.
I’m sticking with the recommended VM size for Harbor. All of my images will be kept in the persistent disk defined here.
Once the changes have been reviewed they are applied in the same way as Bosh Director and PKS.
Harbor is accessed (for administration purposes) from the FQDN provided within the tile configuration. In my deployment that is https://harbor.corp.local.
The only user that can login to Harbor at this point is the local “admin” user with the password defined within the Harbor tile.
The interface itself is relatively simple. There is a public project created by default that any user can access. You do not have to login to Harbor to push and pull to/from any public projects.
I want to test out my authentication so I am going to create a private project. This means I will need to define who has access to it and with what rights. First things first, I need to provide a name for my project.
Once created my project will be listed on the main page allowing me the admin to then decide who will have access to it.
The “members” tab shows all my current users and their roles. As expected it just shows me the admin account. Also note that the “groups” button is greyed out, most likely because Harbor has no direct AD/LDAP access because I am using PKS UAA.
Lets add an account I mapped to the “pks.clusters.admin” role when I granted access to PKS in a previous article. When I did this I mapped the “pks-admin” AD group which contained two users (Administrator and Backup Admin) so I should be able to add both of those users individually here.
Administrator adds fine.
When adding my second user I get an error as if Harbor has no knowledge of the user. This is because even though my user has the appropriate permissions within UAA the user has never actually logged in to PKS before. As soon as I perform a PKS login operation from the command line then my second user can be added. Keep this in mind when you are deciding what authentication options to use!
Any user that is added to a project will then be displayed under the “Administration > Users” section allowing the account to be designated a Harbor Admin if required.
Logging Into The Registry
Now that I have a project defined with a user account I should be able to use the docker cli to push something into the registry. Lets give it a try!
I’m going to use a CentOS 7.7 VM as my client machine. This has docker-ce installed on it as well as kubectl and the PKS CLI. I should be good to go to do all my PKS operations and tasks from this VM.
The first thing I am going to try and do is log into the Harbor registry using the docker client. I should be able to authenticate with a user that is defined in Harbor via UAA in PKS.
Looks like there might be a bit of work to do. From the error I can see that my client machine doesn’t trust the authority that issued Harbors certificate. That’s my MS CA so I need to get that sorted.
I’ve downloaded my CA certificate in pem format and copied it to the folder for CA trusts (/etc/pki/ca-trust/source/anchors/). Once there the certificate is trusted using “update-ca-trust”. Now when I try to login to Harbor using docker the login succeeds!
Note: remember to add your user to the docker group on your client machine if you have started the docker engine on that machine. If you don’t do this then you will not have sufficient permissions to use the docker command.
Pushing an Image
In this simple example I am going to push the latest nginx image available on the internet from my client machine into Harbor, specifically the “vmware-pks” project I defined earlier (note pushing something to Harbor is the same no matter whether it is a container image created on your local machine or one downloaded from the internet). To do this I have pulled down the nginx image using a “docker pull”. Without specifying the registry the image comes from the docker.io public registry by default.
Now the nginx image is sitting on my client machine I can push it into my private Harbor registry. Before the image can be pushed I need to tag it with the registry it needs to be pushed to together with the project name and image name to use. Once that is done the push is done using the tag.
Now when I look at the “vmware-pks” project within the Harbor admin console I can see the nginx image listed as well as the specific versions available.
In my example I used “latest” as the version.
As I have vulnerability scanning enabled on this project the container image is then rated for security by examining the packages that are within the container for any known security issues. This requires the Harbor VM to have internet access (either direct or via proxy) so it can reference online databases.
Pulling an Image
As this article (and series) relates to Enterprise PKS I wanted to show how my image can be pulled to a Kubernetes cluster, not just a standalone docker instance. This process therefore requires me to define a yaml manifest for Kubernetes to execute against as well as some additional configuration to access my private project in Harbor.
The first thing I need to do is to define how Kubernetes will authenticate to the “vmware-pks” project. Doing basic pulls using docker is easy as I can just type my username and password in to authenticate however Kubernetes does things differently as the manifest cannot contain my credentials.
What I am going to do is define a Kubernetes “secret” resource on my Kubernetes cluster which I can then reference within a manifest. I need to get some Kubernetes authentication credentials for the PKS cluster first using the PKS CLI. In this example I am using the “administrator” account from my directory.
Once authenticated I am able to create a “secret” which I have named “harborcred” using the docker config.json file (created when I performed a “docker login” operation and contains my authentication details). Note you can also create the secret by supplying the username and password on the command line rather than using the docker json file.
If get the secret from the cluster then it should show me the base64 encoded version of my docker file. Now I have the secret created I can use it in a Kubernetes manifest yaml file.
Here is a simple manifest to create a Kubernetes pod. This is a very basic manifest. I’m not defining any deployments, replica sets, services etc. Instead of just using “nginx:latest” as my image name I instead use the harbor registry path. I also add an “imagePullSecrets” section which references the secret name I am going to use.
apiVersion: v1 kind: Pod metadata: name: nginx-demo namespace: default spec: containers: - name: nginx-container image: harbor.corp.local/vmware-pks/nginx:latest imagePullSecrets: - name: harborcred
With this image applied I can see that my pod and associated container within is created successfully and started.
That completes my simple example of using the Harbor registry with PKS and Kubernetes. I hope you have enjoyed this series and learned something!