I’ve been a big fan of Rook for simplifying storage on Kubernetes. Sure, CEPH is complex but the Rook operator makes it very easy to get a CEPH cluster up and running on Kubernetes. However, it’s common knowledge that CEPH is usually meant for large clusters and is resource intensive.

I’ve been wanting to try out Linstor for quite some time now. In this blog post, we’ll use the Piraeus Operator to deploy a Linstor cluster and use it to provision volumes on our Kubernetes cluster.

We’ll also ensure that TLS is configured for communication between the components.

DRBD kernel module#

Linstor depends on the DRBD kernel module. You will need to ensure that the kernel module is loaded before you attempt to install the operator and configure a cluster.

On Talos, this is how you’d do it:

Use https://factory.talos.dev to get a Talos Linux image with the DRBD module built-in.
Use the following patches to load the kernel module(s).

machine:
  kernel:
    modules:
      - name: drbd
        parameters:
          - usermode_helper=disabled
      - name: drbd_transport_tcp
      - name: dm-thin-pool

Verify that the module(s) are loaded:

talosctl --talosconfig talosconfig -n 192.168.16.12 get modules

Sample output:

NODE            NAMESPACE   TYPE               ID                   VERSION
192.168.16.12   runtime     KernelModuleSpec   dm-thin-pool         1
192.168.16.12   runtime     KernelModuleSpec   drbd                 1
192.168.16.12   runtime     KernelModuleSpec   drbd_transport_tcp   1

Installing the Operator#

Follow the official instructions here: Link

Create the Linstor Cluster#

Generate the required certificates using cert-manager (we’ll use these certificates to enable TLS communication between the various components)

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-bootstrapper
  namespace: piraeus-datastore
spec:
  selfSigned: { }
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linstor-internal-ca
  namespace: piraeus-datastore
spec:
  commonName: linstor-internal-ca
  secretName: linstor-internal-ca
  duration: 87600h
  isCA: true
  usages:
    - signing
    - key encipherment
    - cert sign
  issuerRef:
    name: ca-bootstrapper
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linstor-internal-ca
  namespace: piraeus-datastore
spec:
  ca:
    secretName: linstor-internal-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: linstor-api-ca
  namespace: piraeus-datastore
spec:
  commonName: linstor-api-ca
  secretName: linstor-api-ca
  duration: 87600h # 10 years
  isCA: true
  usages:
    - signing
    - key encipherment
    - cert sign
  issuerRef:
    name: ca-bootstrapper
    kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: linstor-api-ca
  namespace: piraeus-datastore
spec:
  ca:
    secretName: linstor-api-ca

Create the Linstor cluster

apiVersion: piraeus.io/v1
kind: LinstorCluster
metadata:
  name: linstor-cluster
spec:
  apiTLS:
    certManager:
      name: linstor-api-ca
      kind: Issuer
  internalTLS:
    certManager:
      name: linstor-internal-ca
      kind: Issuer
  properties:
    - name: DrbdOptions/Net/tls
      value: "yes"
  nodeAffinity:
    nodeSelectorTerms:
      - matchExpressions:
          - key: node-role.kubernetes.io/control-plane
            operator: DoesNotExist

Create the LinstorSatelliteConfiguration

apiVersion: piraeus.io/v1
kind: LinstorSatelliteConfiguration
metadata:
  name: linstor-satellite-configuration
spec:
  nodeAffinity:
    nodeSelectorTerms:
      - matchExpressions:
          - key: node-role.kubernetes.io/control-plane
            operator: DoesNotExist
  internalTLS:
    tlsHandshakeDaemon: true
    certManager:
      name: linstor-internal-ca
      kind: Issuer
  storagePools:
    - name: volume-group-1
      lvmThinPool:
        volumeGroup: vg1
      source:
        hostDevices:
          - /dev/vdb
  podTemplate:
    spec:
      initContainers:
        - name: drbd-shutdown-guard
          $patch: delete
        - name: drbd-module-loader
          $patch: delete
      volumes:
        - name: run-systemd-system
          $patch: delete
        - name: run-drbd-shutdown-guard
          $patch: delete
        - name: systemd-bus-socket
          $patch: delete
        - name: lib-modules
          $patch: delete
        - name: usr-src
          $patch: delete
        - name: etc-lvm-backup
          hostPath:
            path: /var/etc/lvm/backup
            type: DirectoryOrCreate
        - name: etc-lvm-archive
          hostPath:
            path: /var/etc/lvm/archive
            type: DirectoryOrCreate

Note: The podTemplate field is required only if you're running Talos

Create the storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: piraeus-storage-replicated
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  linstor.csi.linbit.com/storagePool: volume-group-1
  linstor.csi.linbit.com/placementCount: "2"

Verify storage and TLS

kubectl exec -n piraeus-datastore deploy/linstor-controller -- linstor storage-pool list

kubectl exec -n piraeus-datastore deploy/linstor-controller -- linstor node list

kubectl exec -n piraeus-datastore deploy/linstor-controller -- curl --key /etc/linstor/client/tls.key \
--cert /etc/linstor/client/tls.crt --cacert /etc/linstor/client/ca.crt \
https://linstor-controller.piraeus-datastore.svc:3371/v1/controller/version

kubectl logs -l app.kubernetes.io/component=linstor-satellite -c ktls-utils

Piraeus on Kubernetes

DRBD kernel module#

Installing the Operator#

Create the Linstor Cluster#