GlusterFS

It is a very common problem that you need a filesystem that is synced between several computers and stays available, even if some servers are down. Here is a nice overview of those filesystems https://en.wikipedia.org/wiki/Comparison_of_distributed_file_systems

https://en.wikipedia.org/wiki/Gluster#GlusterFS is one of those filesystems.

Install

https://www.osradar.com/install-and-configure-glusterfs-debian-10/

On each server:

apt install glusterfs-server
systemctl start glusterd
systemctl status glusterd
[$[Get Code]]

On any server probe all the other servers

gluster peer probe thorsten-gluster-test-2.example.com
gluster peer probe thorsten-gluster-test-3.example.com
...
[$[Get Code]]

Plus one server also needs to add to first one

gluster peer probe thorsten-gluster-test-1.example.com
[$[Get Code]]

This lists all the servers but yourself

gluster peer status
[$[Get Code]]

This lists all the server

gluster pool list
[$[Get Code]]

Volumes

Create a volume, list all the servers and where on the server the data should be stored

gluster volume create my-glusterfsvolume replica 3 \
  thorsten-gluster-test-1.example.com:/mnt/my_gulsterfs_test \
  thorsten-gluster-test-2.example.com:/mnt/my_gulsterfs_test \
  thorsten-gluster-test-3.example.com:/mnt/my_gulsterfs_test
[$[Get Code]]
gluster volume start my-glusterfsvolume
[$[Get Code]]

Mount a volume (this is a stupid way as it only works when this server is online while you mount)

mount -t glusterfs thorsten-gluster-test-1.example.com:/my-glusterfsvolume /mnt/my_gulsterfs_test/ -o backupvolfile-server=thorsten-gluster-test-2.example.com
[$[Get Code]]

If a node is down you can force remove it from all volumes that have it and then detach it (but you need to lower the replica the the number of servers that will remain)

gluster volume | grep bad-server.example.com
gluster volume remove-brick my_volume replica 2 bad-server.example.com:/data1/glusterfs_volumes/my_volume/brick force
gluster peer detach bad-server.example.com
[$[Get Code]]

Later you can add a new server and increase the replica again

gluster volume add-brick my_volume replica 3 new-server.example.com:/data1/glusterfs_volumes/my_volume/brick
[$[Get Code]]

Kubernetes PVC based on

For on premise Kubernetes solution you often need a storage solution that does not have a single point of failure. Glusterfs is nice for this.

apiVersion: v1
kind: PersistentVolume
metadata:
  # The name of the PV, which is referenced in pod definitions or displayed in various oc volume commands.
  name: my-glusterfs-pvc
spec:
  capacity:
    # The amount of storage allocated to this volume.
    storage: 8Gi    
  accessModes:
    # labels to match a PV and a PVC. They currently do not define any form of access control.
  - ReadWriteOnce    
  # The glusterfs plug-in defines the volume type being used
  glusterfs:        
    endpoints: glusterfs-cluster
    # Gluster volume name, preceded by /
    path: /my_volume
    readOnly: false
  # volume reclaim policy indicates that the volume will be preserved after the pods accessing it terminate.
  # Accepted values include Retain, Delete, and Recycle.
  persistentVolumeReclaimPolicy: Delete
[$[Get Code]]