Background

AWS deprecated path-style access for S3 buckets for virtual-host accessing method. Ever since, it is generally advisable to use virtual-host style accessing in applications.

Path-style access model

https://s3.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg
https://s3.amazonaws.com/jsb-public/classic_amazon_door_desk.png

Virtual-host style access model

https://jbarr-public.s3.amazonaws.com/images/ritchie_and_thompson_pdp11.jpeg
https://jsb-public.s3.amazonaws.com/classic_amazon_door_desk.png

Rook’s CephObjectStore CR uses path-style addressing by default. In this blog post we’ll explore how to enable virtual-host style addressing for Ceph RGW.

Steps

  1. First, we need to ensure a wildcard certificate exists for the S3 endpoint.

    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: public-rgw-internal-nanibot-net-cert
      namespace: rook-ceph
    spec:
      secretName: public-rgw-internal-nanibot-net-cert-secret
      duration: 2160h
      renewBefore: 720h
      privateKey:
        algorithm: RSA
        encoding: PKCS1
        size: 2048
        rotationPolicy: Always
      dnsNames:
        - "*.ceph-objectstore.apps.openshift.internal.nanibot.net"
      issuerRef:
        name: nanibot-net-issuer
        kind: ClusterIssuer
    
  2. Modify the default IngressController to allow wildcard routes by setting spec.routeAdmission.wildcardPolicy.

    apiVersion: operator.openshift.io/v1
    kind: IngressController
    metadata:
      name: default
      namespace: openshift-ingress-operator
    spec:
      ...
      routeAdmission:
        wildcardPolicy: WildcardsAllowed
      ...
    
  3. Set the spec.hosting.dnsNames field for the CephObjectStore to include the wildcard addressable domain name (Note: Don’t include the * character).

    apiVersion: ceph.rook.io/v1
    kind: CephObjectStore
    metadata:
      name: ceph-objectstore
      namespace: rook-ceph
    spec:
      ...
      hosting:
        dnsNames:
          - ceph-objectstore.apps.openshift.internal.nanibot.net
    
  4. Finally, create an ingress for the S3 endpoint. OpenShift’s Route Controller Manager will convert the ingress to an appropriate route.

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: ceph-objectstore-ingress
      namespace: rook-ceph
      annotations:
        route.openshift.io/termination: "reencrypt"
        route.openshift.io/destination-ca-certificate-secret: pki-production-ca
    spec:
      ingressClassName: openshift-default
      tls:
      - secretName: public-rgw-internal-nanibot-net-cert-secret
        hosts:
          - '*.ceph-objectstore.apps.openshift.internal.nanibot.net'
      rules:
      - host: '*.ceph-objectstore.apps.openshift.internal.nanibot.net'
        http:
          paths:
          - path: '/'
            pathType: Prefix
            backend:
              service:
                name: rook-ceph-rgw-ceph-objectstore
                port:
                  name: https
    

Note: The above will generate a route of type “reencrypt”. The annotation “route.openshift.io/destination-ca-certificate-secret” points to the secret containing the CA certificate of the CephObjectStore’s internal certificate (the one that’s specified under spec.gateway.sslCertificateRef).