GlobalSign Blog

Simplify PKI in Kubernetes Environments and Accelerate DevOps

Simplify PKI in Kubernetes Environments and Accelerate DevOps

I recently worked with a client on their public key infrastructure (PKI). After a few short discussions, it became evident that they were ill-prepared. They had significant challenges related to managing SSL/TLS certificates, which serve as machine identities in their diverse infrastructure. Like most organizations, they had started their digital transformation journey. They were grappling with changes like moving from on-premises to the cloud and rearchitecting applications from the ground up. In addition, operational pain and quality of talent had long plagued the organization, which resulted in a confusing amalgamation of technology across various on-premises and cloud providers. 

Sound familiar?

Furthermore, as we zeroed in on the PKI environment it too was in disarray. After several rounds of discussions, the root cause behind this became clear. They had built several layers of complex PKI infrastructure at all points of the infrastructure - both on-premises and in the cloud. There were no standardized controls or approaches in use. This was costing developer and DevOps engineering time. In addition, security teams felt things were getting out of control since many certificates were not controlled or made visible to InfoSec and PKI teams. And, security teams couldn’t inspect traffic since they didn’t have access to the private keys. Information security was concerned that an attacker could go undetected by using encrypted tunnels.

This is how their PKI was set up:

venafi kubernetes_image1_pki setup

As you can see, they were using many different sources on-premises, ranging from Let’s Encrypt to GlobalSign, in addition to Amazon Certificate manager and Azure Key Vault. They also had many instances of AWS and Azure.

My goal for this client was to remove unnecessary complexity and provide building blocks for developers to request certificates quickly via automation, while also addressing security concerns.

Here is how I tackled the challenge. 

Streamlining Certificate Management to Address DevOps Security Challenges

Rather than reinvent the wheel, I turned to NIST for a blueprint that we could practically follow. Sparing you all the details, I read through several documents surrounding cryptography and landed a new guidance document: NIST 1800-16B: Securing Web Transactions: TLS Server Certificate Management.

After studying this detailed guidance, I extrapolated several clear, immediate recommendations around these themes: 

  • Simplify processes by offering a certificate service to developers
  • Provide security teams control and visibility over certificates in use

Current State

Challenges

Private trust certificates were coming from a variety of sources:

  1. Microsoft Certification Authority (CA)
  2. HashiCorp Vault
  3. Homegrown OpenSSL CA

- Developers had to code their applications to work with each source

- Developers couldn’t easily migrate applications across environments due to PKI considerations

- Developers were spending time maintaining homegrown CAs

- Security teams only had visibility into certificates issued by Microsoft CA

- Security teams didn’t know what traffic could be trusted due to rogue CAs

Publicly-trusted certificates were also non-standardized and coming from a variety of sources

- Let’s Encrypt was non-compliant with security policy but it was convenient, so developers often used it

- Private keys were not centralized so security teams could not inspect traffic

- Cloud providers did not provide sufficient options for their certificates to comply with policy

My recommendations were to:
1. Standardize CAs across on-premises and cloud providers
2. Use Jetstack cert-manager to automate certificate life cycles for Kubernetes clusters
3. Use Venafi as a Service to connect certificate authorities to DevOps workflows via native integrations, enforce policy (e.g. certificate attributes and sources), and provide visibility for audits and compliance

Ultimately, these choices helped to streamline certificate management workflows in multicloud Kubernetes environments and also acted as a unified API that would allow interactions with different CAs. In the section that follows, I’ll provide detailed step-by-step instructions for how to address points #2 and #3 above, using GlobalSign as a CA.

How to Connect Kubernetes to Venafi as a Service and GlobalSign

Technologies Used

  • Jetstack cert-manager for Kubernetes (and OpenShift)
  • Venafi as a Service

Step 1. Get a free Venafi as a Service Account: https://www.venafi.com/cloud

Step 2. Configure CA Accounts 

On the left-hand menu select “CA Accounts” and set up your CA account(s).  

venafi kubernetes_image2_CA account configuration screencap

Select add new account for each provider.

venafi kubernetes_image3_add CA account GlobalSign screencap

The end result should look like this:

venafi kubernetes_image4_external CA screencap

Step 3. Set Up Issuing Templates
Issuing templates connect a certificate authority and certificate attributes together – aka security policy. You’ll likely have multiple issuing templates since you will need publicly-trusted certificates with longer validity periods as well as private certificates for back-end infrastructure, such as containers and service meshes.
 

venafi kubernetes_image5_set up issuing templates

Note: Make sure you use .*.yourdomain.com to match your domain if you want to make sure you validate domains before issuance (Recommended). Example below:

venafi kubernetes_image6_new issuing template screencap


Step 4. Configure Projects and Zones

In this example we are going to create a Cloud Project and grant the team the ability to issue both public and private certificates.

In order to issue certificates we first create a Project, which allows us to segment certificate issuance for an application.
 

venafi kubernetes_image7_create first project

Select “Create A New Project” and give the project a name, e.g. “Cloud.”

Then, add a Zone. A Zone represents an environment, e.g., development, staging, production. 

venafi kubernetes_image8_add a zone

Since this is the first time using Venafi as a Service, I’ll create zones and tie them to the issuing templates. This connects an application’s environment to the certificate source and attributes as defined in the issuing template (aka security policy).

Step 5. Grab Zone IDs to Use for Issuing

Once you’ve finished the previous step, your project should be listed. Click on the project. In this case our project name is “Cloud” so I clicked on the Project named “Cloud.”
 

venafi kubernetes_image9_zone id for project cloud

Once opened, you will see each Zone that is tied to our issuing templates (this defines the CA and certificate attributes). Click on each tab – Development, Production, Staging – and copy the Zone ID’s. These will be used in our yaml files to create the issuers in cert-manager. An example is highlighted in the subsequent screenshot:

venafi kubernetes_image10_cloud project details


Step 6. Grab the API key and Store It for Use Later

This is what we are going to need to make the certificate request to the Venafi as a Service API broker.

venafi kubernetes_image11_grab api key

Now that we have the information we need to get started issuing certificates., lets iInstall Jetstack cert-manager so we can start requesting certificates from the various CA providers to use in our Kubernetes environment

Step 7. Install Jetstack cert-manager

First, let’s create a kubernetes namespace specifically for cert-manager.

kubectl create namespace cert-manager
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml

If all went well, you should see something like this:.
 

venafi kubernetes_image12_kubernetes namespace

Run the following to check on status, running means we are ready to move on the next step:

 kubectl get pods --namespace cert-manager
 

venafi kubernetes_image13_kubernetes namespace status check

Step 8. Set Up API Key as a Secret Within Kubernetes

This is the API that you grabbed in Step 6 above.

If you are using a separate namespace, just change the name from default.

kubectl create secret generic cloud-secrets --namespace='default' --from-literal=apikey='1a637c2b-125f-0000-82ea-c7b60ed00000'

Should return:

venafi kubernetes_image14_cloud secrets created

Step 9. Set Up the Issuers That You Want to Configure

In our case we have one issuer, so we are going to need to create the issuer.
Please note the name and zone id in the file.

kubectl create -f globalsign-cloud-issuer.yaml

venafi kubernetes_image15_globalsign cloud issuer yaml

kubectl describe issuer globalsign-venafi-issuer --namespace=default

venafi kubernetes_image16_globalsign venafi issuer namespace default part 1

venafi kubernetes_image17_globalsign venafi issuer namespace default part 2

kubectl describe issuer globalsign-venafi-issuer --namespace=default

Step 10. Try it Out!

Now that we have installed cert-manager, we can start requesting certificates. Note also that cert-manager automatically attempts to renew certificates before they expire which is very handy.

In my example, I had to request a certificate from our internal CA and one from our external to be used. Example request format is below.

venafi kubernetes_image18_certficiate request format example

Running the request:
The below command will run the request to get a certificate.

kubectl apply -f examplecert.yml

venafi kubernetes_image19_command for running cert request

View your request here:
https://ui.venafi.cloud/certificates/inventory

Step 11. Troubleshooting
The first step to troubleshooting is to get a status on your existing certificates. So we will run our kubectl get certificates command. This will allow us to see what is currently in inventory and the status.

kubectl get certificates

venafi kubernetes_image20_troubleshooting kybectl get certificates

Since the flag is false (expected to be true) we will dive a little deeper and get the appropriate status messages using:

kubectl describe certificates globalsign-devops.amplisight.io

venafi kubernetes_image21_kubectl describe certificates globalsign-devops.amplisight.io part 1

venafi kubernetes_image22_kubectl describe certificates globalsign-devops.amplisight.io part 2

Once the command is run, you will see there is a listing for the certificate request in the message and the current status.  Above you will see that each certificate request gets assigned a unique id which is in the format of <certificate name>-Unique ID.  

In order to see more details on potential errors on issuance you must run the following command:

kubectl describe certificaterequest globalsign-devops.amplisight.io-23434489

venafi kubernetes_image23_see more details on potential errors on issuance

In our case, we were just a little (or a lot!) impatient and needed to allow the issuer time to issue the certificate.

Take the Next Step

To learn more about GlobalSign’s PKI for DevOps or Venafi as a Service, we invite you to watch this short webinar demo and visit: 
https://www.globalsign.com/en/lp/venafi
https://www.venafi.com/cloud 

Share this Post

Related Blogs