Handling of TLS cert formats: "appending provided certificate to pool failed"

[scaleoutsean]

Go's AppendCertsFromPEM() function returns false when it can't parse any valid PEM-encoded certificates from the provided data.

The error "appending provided certificate to pool failed" comes from certPool.AppendCertsFromPEM(config.TLSCaCert) returning false in grpc.go

The problem may be in poor docs, or the BeeGFS init script's base64 decoding or the certificate format isn't what Go's crypto/x509 library expects.

Yesterday I (again) spent hours trying to figure out TLS, and couldn't. I think there may be a problem in here, or the docs.

[iamjoemccormick]

Was this trying to run a CTL command or use CTL as a library? What flags and/or environment variables did you specify?

In particular were you specifying the path to a TLS cert file (--tls-cert-file=/etc/beegfs/cert.pem) or wanting to use the system certificate pool for TLS (--tls-cert-file="")?

Most likely we need to make some improvements at https://doc.beegfs.io/latest/advanced_topics/tls.html but perhaps there is also an issue with how we handle TLS.

[scaleoutsean]

I was trying to get TLS in that all-in-one stack of mine, so I was using the BeeGFS containers (including an extra client container).
I couldn't get it to work so I removed my TLS config later and don't have the logs or config samples from that time, but I think tried both as strings and as file paths, and neither worked.

I opened issue here because that seemed to be where the issue is as mentioned above, but also in the TLS-related docs, maybe some improvements would be welcome.

For example, is it documented anywhere what the right format of those files is?
What is $TLS_CERT_FILE_DATA? And where's the accompanying env.example?

Or, rather than let the people who choose wrong have it conveniently documented, why not do the opposite and make it convenient for those who choose correctly (TLS)? It should be reversed. All docs should be TLS only with a "here are 3 hints - figure out it if you can" Appendix for TLS-less gotchas at the end...

[scaleoutsean]

As to which TLS I'm talking about, it's as per the initial comment:

        // If the user explicitly disabled TLS that should take precedence. Note a mismatch between the
        // client and server TLS configuration will likely return a vague `error reading server preface:
        // EOF`. It is up to the caller to intercept this error and display a meaningful error/hint.
        if config.TLSDisable {
                opts = append(opts, grpc.WithTransportCredentials(insecure.NewCredentials()))
        } else {
                certPool, err := x509.SystemCertPool()
                if err != nil {
                        return nil, fmt.Errorf("couldn't load system cert pool: %w", err)
                }
                // Append custom ca certificate if provided
                if config.TLSCaCert != nil {
                        if !certPool.AppendCertsFromPEM(config.TLSCaCert) {
                                return nil, fmt.Errorf("appending provided certificate to pool failed") // <===== HERE
                        }
                }
                creds := credentials.NewTLS(&tls.Config{
                        RootCAs:            certPool,
                        InsecureSkipVerify: config.TLSDisableVerification,
                })
                opts = append(opts, grpc.WithTransportCredentials(creds))
        }