Hey, Docker experts! Need some help, and I know literally nothing about Docker.
The Toot! notifications relay runs in a using a "distroless" image. Apparently there has been some change in the SSL certificate for the the APNS service, and the right root certificates are now not available in this image. How do I fix this?
Here's one report of the issue: https://github.com/GoogleContainerTools/distroless/issues/545
Here's the Apple announcement: https://github.com/GoogleContainerTools/distroless/issues/545
And here's my setup: https://github.com/DagAgren/toot-relay/blob/master/Dockerfile
Sorry, got the wrong link to the Apple announcement there. This is the one: https://developer.apple.com/news/?id=7gx0a2lp
@WAHa_06x36 The link to the Apple announcement goes to github.com. However, from your toot I gather the root CA with which Apple's server certificate is signed is not available in your image? I see two options.
Option 1: Fiddle with the files that the distroless image provides so that the correct root CA is installed in the system certificate store (/etc/ssl/ something), this can be done with some additional commands or running scripts in the Dockerfile.
(option 2 follows in a separate toot)
@WAHa_06x36 Option 2: Tell the HTTP client inside your Go application to use a custom CA. This can be done by passing a http.Client with a custom Transport setting, let me see if a code snippet fits in a toot...
@WAHa_06x36 Option 2, contd: As I gather from https://github.com/DagAgren/toot-relay/blob/b1921965f1dbf633965201fcd40c8be9af0883fd/vendor/github.com/sideshow/apns2/client.go#L75, you could just do
// Let filename be a .pem file with the correct root CA.
caBytes, err := ioutil.ReadFile(filename)
if err != nil {
// handle
}
rootCAs := x509.NewCertPool()
rootCAs.AppendCertsFromPEM(caBytes)
productionClient.HTTPClient.Transport.TLSClientConfig.RootCAs = rootCAs
somewhere around https://github.com/DagAgren/toot-relay/blob/master/toot-relay.go#L56.
@Xjs Interesting! Ideally I would like to not put anything extra in the code that might just need changing again some time in the future, so it would be nice to update the distro certificates, but as I have zero clues about Docker I have no idea how to do it.
@WAHa_06x36 This can of course be done as well, I would have to read up a little on how one is supposed to do this the right(tm) way. I'll get back to you after lunch :)
@Xjs Thanks, it'd be much appreciated!
@WAHa_06x36 Hey, I'm back.
It looks to me as though the best way would be to add a second build stage (we could use the golang base image since it is incidentally based upon Debian, but I would not consider this an API guarantee) with a debian base image, copy AAACertificateServices.crt inside and run update-ca-certificates, and then copy /etc/ssl/certs from there to the final image. I'll provide a PR doing this in a minute.
Btw, I suggest you update to Go 1.14 for you build environment. :)
@WAHa_06x36 I believe it should be done like this: https://github.com/DagAgren/toot-relay/pull/3
@Xjs Annoyingly it does not seem to work:
2021-02-11T16:36:29.389442+00:00 app[web.1]: 2021/02/11 16:36:29 Push error: Post https://api.push.apple.com/3/device/3881c33666f8265119910771ab999d90534bad3ce6dac8ed77ebd1da60142586: x509: certificate signed by unknown authority
@Xjs I tried connecting to the dyno on heroku, and It seems the certificate does make it into the image correctly, but somehow still isn't picked up.
@WAHa_06x36 The Go implementation reads the first found of those files: https://go.googlesource.com/go/+/refs/tags/go1.11.13/src/crypto/x509/root_linux.go (unchanged through Go 1.15 btw), can you confirm both root certificates (GeoTrust and AAA-thingy) are contained in /etc/ssl/certs/ca-certificates.crt?
@WAHa_06x36 FTR, when I build the container, both certificates (GeoTrust and AAA Certificate Services) are contained and *should* be read by the Go application. I tested by running a simple http.Get("https://api.push.apple.com") from within the container as first line of main() and it connects without problem from my machine in Frankfurt.
@WAHa_06x36 Unfortunately, I have no Apple Developer account and thus no client certificate, so I can't test with an actual notification. I'm happy to make some time in the next days to help you debug this, synchronously if you like.
@WAHa_06x36 Also do you have access to `openssl` on the dyno and can you verify which certificate you are presented when doing `openssl s_client -connect api.push.apple.com:443`?
@WAHa_06x36 I keep wondering why my phone wouldn’t buzz, but obviously I’m using Toot!, haha.
This is still signed by the old, GeoTrust, CA. However, you’re getting verification errors on the node as well (»couldn’t get local issuer certificate«). Can you share how the container is started? Does it maybe read /etc/ssl from the node instead of the container contents, or something like that?
@Xjs It's just running on Heroku, but I have no idea how that really works either. I think it should just use the container? And it used to work up until yesterday or so, which is the very confusing part. That issue linked at the start had something about that I think...
@WAHa_06x36 Ah. I have an idea. You could just copy the GeoTrust CA into the Docker image as well (hang on, I'll push a commit that does that in a sec, since I appear to have the GeoTrust CA on my Debian...). I'd say let's build an image with both CAs added manually and see if it works?
@Xjs Sounds good!
@WAHa_06x36 https://github.com/DagAgren/toot-relay/pull/4
(I can't really explain why my base image does contain the GeoTrust CA and yours apparently doesn't, but I assume there might be caching involved or something.)
@Xjs ...it still doesn't work. Looking at the output from the deploy command, maybe it IS in fact running it with its own go environment and ignoring the rest of the image? Maybe we need to try the other approach and load the certificate directly in the Go code.
@WAHa_06x36 I have a PR ready that would do that, but it builds on the state before our two commits. (https://github.com/DagAgren/toot-relay/pull/5)
Also I just noticed it contains some `go fmt` artefacts, sorry about that.
If you like, I can rebase it so that it contains the reverts, or you can just revert before merging. Or you check out my branch and build an image with it, and try if it works, and if it does, we can think about what we do with git.
@Xjs I'll revert them.
@WAHa_06x36 Note that if Heroku doesn't just run the Docker container, you'd have to set the CA_FILENAME environment variable and supply the cas.crt file in some other way. (You should be providing the toot-relay.p12 file and its password in some way, so there should already be a place where you can specify files & environment variables, right?)
@WAHa_06x36 Wenn I try to connect to api.push.apple.com, I am presented a certificate thich is signed by the (old) GeoTrust CA. Can you verify whether that one is included in the certificate store?
@WAHa_06x36 (Docs: https://pkg.go.dev/crypto/tls#Config)
This would mean that *only* the Root CAs in the given file will be used for server certificate validation, but as far as I understand the announcement, this would be fine. You could make the file configurable and would only have to input and change the CA file in the future. I'd happy to provide a PR if you'd like.