Renewing your Gitlab Pages certificate the Hard Way

I got a nice and polite notice a month ago that my certificate on my blog is expiring. Thankfully i have documented the process on my blog, i thought. Sadly, i can’t remember on which of my computers i created the certificate. Also, i need that computer to renew the cert, and in the end, none of the computers i tend to use actually seemed to be able to renew the cert. So it looks like i’m having to renew the cert the Hard Way. From scratch.

As prerequisites, i assume you, like me, have a web site on Gitlab Pages, it’s running with a custom domain, and that the site is a static site generated with Hugo. This’ll probably work with any (static) site running on Gitlab pages, but this is how my stuff is set up.

I’m also assuming you’ve tried your damndest using the certbot renew command and that nothing useful has come out of it.

If this all seems kind of what you need but not really, maybe you should read my earlier post on how to add certificates to Gitlab pages. In all honesty, it tells more or less the same as this post, but from a slightly different angle.

Step one: Say goodbye to your old certificate

This may very well be the wrong thing to do, but first, go to your Gitlab pages project, go Settings > Pages, regard the lonely URL with the expired certificate, where it says Domains (and Expired). Take a melancholic breath and click the Remove button. This will remove the expired certificate (the URL), not the pages. That’s in the text block below (“Remove pages”). Don’t click it. Removing the cert is bad enough for your mental well-being.

Step two: A new domain

You should now be back on the Pages Settings page. For the next step to work, you’l first need make sure the setting Force domains with SSL certificates to use HTTPS is unchecked. Click Save if you just unchecked that setting.

Now click the green New Domain button on the top right of page. Enter the “custom” domain name, which really is the one you just deleted. Leave the Certificate and Key text areas blank for now.

Step three: DNS verification

Your domain is not verified, it says, so let’s do that now, by DNS. Fire up your DNS service – how you do this really depends on your DNS – and edit the TXT record _gitlab-pages-verification-code.your.custom.domain. Note that the value of the record is not just a long string of hexadecimals but actually includes the bit gitlab-pages-verification-code=.

Free plug: I use ZoneEdit, which gives me one free DNS domain (with a seemingly unlimited number of subdomains, and any other kinds of records that i can think of). And no, not the domain itself, but the DNS records. But yeah. Plug out.

Take a short break for the DNS record to propagate, then head back to the Gitlab setting page that complains that your page isn’t verified and click the Retry Verification button. The red Unverified field should eventually turn green and say Verified instead. Happy times.

Step four and Step five, in parallel

Open up a terminal and cd to the directory containing your site, and thereunder (is there such a word?) to static/.well-known/acme-challenge. This is the Hugo-dependent bit. If use another web generator, you’ll have to cd to your static directory and to .well-known/acme-challenge therein.

Now open up another terminal tab (or window, if you’re that kind of guy/gal) and cd to the root of your site. I’m here going to diverge a bit from my previous post about certs on gitlab sites. Create a directory “ssl” for your certs and make sure it doesn’t end up in your repo by adding the line ssl/* into your .gitignore file, then git commit -a to make sure your .gitignore file is updated. Your future self will happy.

Exclaim the following chant: certbot certonly --manual --config-dir ssl --logs-dir ssl --work-dir ssl -d your.custom.domain (replacing your.custom.domain with, you know, your custom domain and ssl with tls or just certs, because ssl is just legacy). If it turns out you didn’t have certbot installed, enter brew install certbot and try again ;)

Certbot will ask for your email address. Carefully read the Certbot Terms of Service (yes?) and Agree. Agree to have your IP address publicly logged (‘cuz it won’t work without – duh).

Now the user interface seems to break, but don’t panic. Create a file in that other tab you have open in static/.well-known/acme-challenge named the long line of characters, with the content of that and even more seemingly random characters. The UI suggests you use a command akin to printf "%s" n1_FOODOOBEjU-kbL545TM.5tXt2-fBurpHtI_Lshus30pRA > n1_FOODOOBEjU-kbL545TM but your mileage may vary. I just used echo. If this seems really obscure to you, it might be that you didn’t create the original certificate request :)

Now git commit -am "Add new acme challenge" and git push. Wait a mo, then verify that the file ended up on the web using curl or a browser. Remember to use the http protocol as https isn’t up and running just yet.

Go back to your certbot terminal tab and press Enter. Lean back in your comfy chair and watch the magic happen, then feel the satisfaction as Certbot congratulates you that you now have your certificates dun and dusted.

Or as in my case, repeat the process a few times because something inexplicable just happened (or more to the point, did not happen). It all worked out the firth or fifth time.

Step, is it six now: Upload your certificate and key

Back to your Gitlab settings page. Assuming you still are on the page where you verified the domain ownership using DNS, click the green Edit button. Otherwise you’ll find it on Settings > Pages > Domains > Details, then Edit.

From your computer, copy contents of the file ssl/live/your.custom.domain/fullchain.pem into the Certificate text area and the contents of ssl/live/your.custom.domain/privkey.pem into the Key text area. Press Save Changes.

Then check the setting Force domains with SSL certificates to use HTTPS and press Save.

Step seven: Trust but verify

Give your settings a few minutes to brew and propagate on the Gitlab servers, then check your cert using a browser (https://your.custom.domain) or curl -vI https://your.custom.domain (which, of course, is way cooler). Your site may still give a cert error if you’re too hasty, but eventually, things should turn out green.

Step future: Renewing

Make a note, preferrably in a file within your repo, that the command to renew your certificate will be certbot renew --config-dir ssl --logs-dir ssl --work-dir ssl. And make some sort of note what computer you used to create the certificate, as you’ll need it again to renew your certificate.

Step improvement: GPG

If you want to be really nifty, encrypt your privkey.pem and allow your ssl directory to be uploaded to your repo anyway :)