Automating Palo Alto Certificate renewal using Let's Encrypt

11 August, 2023

Automating Palo Alto Certificate renewal using Let's Encrypt

Updated August 2023: Now that the lego client supports Azure DNS with Managed Identities, I've updated this post to use that instead of the janky scripts.

I'm now responsible for managing a lab Palo Alto firewall. And the first thing I noticed was how cumbersome the certificate renewal process was, especially if you use 90-day Let's Encrypt certificates.

Automation is possible! All we have to do is:

Setup a scripting host. Sorry, can't do this on the firewall directly
Automate DNS-based Let's Encrypt renewals
Create an Admin user on the firewall, with SSH authentication
Create an Expect script to import the certificate and private key over SSH
Tie it all together

Automate Azure DNS-based Let's Encrypt renewals

In this example, I'm using Azure DNS, with a Managed Identity provided by the Azure Arc agent.

As I've already got my Managed Identity created, in an admin cloud shell, all I need to do is assign both the Reader role at the zone level, and the DNS Zone Contributor role for the necessary TXT record, to the Service Principal:

#!/bin/bash

AZURE_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
AZURE_RESOURCE_GROUP="rg1"
AZ_ZONE="lab.example.com"
AZ_HOSTNAME="fw"
AZ_RECORD_SET="_acme-challenge.${AZ_HOSTNAME}"

SERVICE_PRINCIPAL="00000000-0000-0000-0000-000000000000"

az role assignment create \
--assignee "${SERVICE_PRINCIPAL}" \
--role "Reader" \
--scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${AZURE_RESOURCE_GROUP}/providers/Microsoft.Network/dnszones/${AZ_ZONE}"

az role assignment create \
--assignee "${SERVICE_PRINCIPAL}" \
--role "DNS Zone Contributor" \
--scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${AZURE_RESOURCE_GROUP}/providers/Microsoft.Network/dnszones/${AZ_ZONE}/TXT/${AZ_RECORD_SET}"

Certbot doesn't have a native Azure DNS plugin, but the Let's Encrypt/ACME client and library written in Go does through the azuredns provider. This client also has the benefit of being a single Go binary (no more fluffing around with snaps! no more root!), so we're going to use that instead.

Couple notes about the below:

I'm using Azure Arc, so I need to set the IMDS_ENDPOINT and IDENTITY_ENDPOINT parameters in the below - you don't need to do this if you're using a native Azure VM.
If you're using Azure Arc, don't forget to set sudo usermod -G himds -a automation_user, otherwise your user won't be able to read keys from /var/opt/azcmagent/tokens/ and you'll get file errors.
I also like having extra debug output, so I've left AZURE_SDK_GO_LOGGING turned on.

#!/bin/bash

AZURE_TENANT_ID="00000000-0000-0000-0000-000000000000" \
AZURE_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000" \
AZURE_RESOURCE_GROUP="rg1" \
IMDS_ENDPOINT=http://localhost:40342 \
IDENTITY_ENDPOINT=http://localhost:40342/metadata/identity/oauth2/token \
AZURE_SDK_GO_LOGGING=all \
./lego --key-type rsa2048 --email "[email protected]" --dns azuredns --domains "fw.lab.example.com" run

If that works, congratulations! Change run to renew, save that as renew-certificate.sh, and renew the certificate via the same script.

Create an Expect script to import the certificate and private key over SSH

PAN-OS expects us to import certificates with a multi-line string... which is tricky.

Thanks to a Reddit post, we know that we can send multi-line input when SSH'd to the firewall by using Ctrl-V Ctrl-M (Ctrl-V being SYN, Hex 16) - which we can do via Expect.

Expect is a great tool for automating interactive sessions. As below, this script will pause for the device, and check that the config has committed successfully (adding error handling is an exercise for the reader).

Note: we do NOT want to set cli scripting-mode on - this will prevent the multi-line escape sequence from working.

The reason for using -- "$line" is because each line will contain Expect control sequences, like - and [. The -- tells send to ignore those. We use sleeps and send -s (slow), because fast scripting hosts will exhaust smaller Palo's buffers, and missed control sequences will result in "Unknown command" errors.

So, create an update-certificate.exp file with the below:

#!/usr/bin/expect

set timeout 60
set send_slow {1 .001}

set f [open ".lego/certificates/fw.lab.example.com.crt"]
set fullchain [split [read $f] "\n"]
close $f

set f [open ".lego/certificates/fw.lab.example.com.key"]
set privkey [split [read $f] "\n"]
close $f

spawn ssh fw01

expect "Password:" {
    stty -echo
    expect_user -re "(.*)\n"
    send_user "\n"
    stty echo
    set pass $expect_out(1,string)

    send -- "$pass\r"
}

expect "fw01>" {
    send "configure\r"
}

expect "fw01#" {
    send "edit shared certificate fw_lab_example_com\r"
}

sleep 2

expect "\[edit shared certificate fw_lab_example_com\]" {
    send "set public-key \""
    foreach line $fullchain {
        set line_trimmed [string trim $line]
        send -s -- "$line_trimmed"
        send -s "\x16\r"
    }
    send "\"\r"
}

sleep 2

expect "\[edit shared certificate fw_lab_example_com\]" {
    send "set private-key \""
    foreach line $privkey {
        set line_trimmed [string trim $line]
        send -s -- "$line_trimmed"
        send -s "\x16\r"
    }
    send "\"\r"
}

sleep 2

expect "\[edit shared certificate fw_lab_example_com\]" {
    send "commit\r"
}

expect "Configuration committed successfully" {
    exit
}

Tie it all together

So, the process to renew the certificates is now:

./renew-certificate.sh
./update-certificate.exp

OLD CONTENT: Scripts to use certbot with Azure DNS

Note: Below are the original scripts to use Azure DNS with certbot. I don't recommend doing this.

Certbot doesn't have a native Azure DNS plugin, and while I could have used certbot-dns-azure, I instead decided to use this example to show how Certbot hook scripts work.

Unfortunately, the Azure CLI doesn't support az login --identity with Azure Arc Azure/azure-cli#16573, so we have to use curl to send a PUT request instead:

#!/bin/bash

AZ_SUBSCRIPTION="00000000-0000-0000-0000-000000000000"
AZ_RESOURCE_GROUP="rg1"
AZ_ZONE="lab.example.com"
AZ_HOSTNAME="fw"
AZ_RECORD_SET="_acme-challenge.${AZ_HOSTNAME}"

# az login --identity
# echo "running: az network dns record-set txt add-record -g ${AZ_RESOURCE_GROUP} -z ${AZ_ZONE} --record-set-name ${AZ_RECORD_SET} -v=${CERTBOT_VALIDATION}"
# az network dns record-set txt add-record -g "${AZ_RESOURCE_GROUP}" -z "${AZ_ZONE}" --record-set-name "${AZ_RECORD_SET}" -v="${CERTBOT_VALIDATION}"

# Challenge token
challengeTokenPath=$(curl -s -D - -H Metadata:true "http://127.0.0.1:40342/metadata/identity/oauth2/token?api-version=2019-11-01&resource=https%3A%2F%2Fmanagement.azure.com" | grep Www-Authenticate | cut -d "=" -f 2 | tr -d "[:cntrl:]")
challengeToken=$(sudo cat $challengeTokenPath)

# Resource token
token=$(curl -s -H Metadata:true -H "Authorization: Basic $challengeToken" "http://127.0.0.1:40342/metadata/identity/oauth2/token?api-version=2019-11-01&resource=https%3A%2F%2Fmanagement.azure.com" | jq -r .access_token)

cat << EOF > request.json
{
  "properties": {
    "TTL": 5,
    "TXTRecords": [
      {
        "value": [
          "${CERTBOT_VALIDATION}"
        ]
      }
    ]
  }
}
EOF

curl -d @request.json -sSL -X PUT -H "Authorization: Bearer $token" -H "Content-Type: application/json" "https://management.azure.com/subscriptions/${AZ_SUBSCRIPTION}/resourceGroups/${AZ_RESOURCE_GROUP}/providers/Microsoft.Network/dnsZones/${AZ_ZONE}/TXT/${AZ_RECORD_SET}?api-version=2018-05-01"

rm request.json

echo "waiting 10 seconds for DNS..."
sleep 10

Now we can test this and run through the initial setup process via:

sudo certbot certonly --key-type rsa --manual --preferred-challenges dns --manual-auth-hook $(pwd)/azure-dns-hook.sh --debug-challenges -d fw.lab.example.com