Leverage containers to distribute NFS DRBD storage among pacemaker cluster nodes
a project by zzhou
Updated 6 months ago. 2 hacker ♥️. 1 follower.

Current Situation [bsc#1201271]

SUSE HA NFS Storage Guide [1] provides a resilient NFS implementation to the clients even if the NS server node fails over within the cluster. However, it is Active-Passive of the two node cluster.

Motivation

Could be possible to extend more NFS server instances on both nodes? In the end, any NFS server instance can run either node in parallel within the pacemaker cluster.

Challenges and Possibilities:

  • NFS server configuration file and state isolation for its own exportfs eg. /var/lib/nfs. Could be helpful from the container technology?
  • How to bundle pacemaker RA service inside container to run nfs-server?
  • How to manage IP address inside container while failover between nodes?

[1] https://documentation.suse.com/en-us/sle-ha/15-SP5/single-html/SLE-HA-nfs-storage/

Join this project

Looking for hackers with the skills:

nfs cluster drbd ha

This project is part of:

Hack Week 23

Activity

  • 6 months ago: roseswe liked this project.
  • 6 months ago: zzhou started this project.
  • 6 months ago: sthackarajan liked this project.
  • 6 months ago: zzhou removed keyword pacemakercluster from this project.
  • 6 months ago: zzhou added keyword "ha" to this project.
  • 6 months ago: zzhou added keyword "nfs" to this project.
  • 6 months ago: zzhou added keyword "cluster" to this project.
  • 6 months ago: zzhou added keyword "drbd" to this project.
  • 6 months ago: zzhou added keyword "pacemakercluster" to this project.
  • 6 months ago: zzhou originated this project.


    • Comments

    • zzhou
      6 months ago by zzhou | Reply 

      * Exercise-1: Lunch multiple NFS docker instances directly by systemd inside containers

  1. 
  tumbleweed status
  target   : 20231101

  Dockerfile: 

  FROM opensuse/tumbleweed
  RUN zypper -n install systemd nfs-kernel-server vim iproute2 iputils pacemaker-remote gawk which
  RUN systemctl enable nfs-server
  RUN echo "/srv/nfs/share  *(rw)" > /etc/exports
  CMD ["/usr/lib/systemd/systemd", "--system"]


  2. 
  docker build -t nfsserver .

  run_nfsserver_docker () {
    i=$1  # eg. i=1; 
    N=nfsserver; h=$N-$i; \
    docker run -v /srv/nfs${i}/state:/var/lib/nfs \
    -v /srv/nfs${i}/share:/srv/nfs/share \
    -it --privileged --name=$h -h=$h  $N &
  }
  run_nfsserver_docker 1
  run_nfsserver_docker 2

  3. verify two nfsserver docker instances
  ip_nfsserver_1=`docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' nfsserver-1`; echo $ip_1
  ip_nfsserver_2=`docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' nfsserver-2`; echo $ip_2
  showmount -e $ip_nfsserver_1
  showmount -e $ip_nfsserver_2

  tws-1:~ # showmount -e $ip_nfsserver_1
  Export list for 172.17.0.3:
  /srv/nfs/share *
  tws-1:~ #   showmount -e $ip_nfsserver_2
  Export list for 172.17.0.4:
  /srv/nfs/share *


* Exercise-2: Lunch NFS server docker directly by pacemaker-remote inside containers  


  4. Add pacemaker docker bundle into CIB.xml
    TODO: FEAT: crmsh does not support container yet

    <bundle id="nfs"> 
      <docker image="nfsserver"/> 
      <network ip-range-start="172.17.0.100" host-netmask="16"/>
      <storage> 
        <storage-mapping id="nfs-state" 
                          source-dir-root="/srv/nfs-state" 
                          target-dir="/var/lib/nfs" 
                          options="rw"/> 
        <storage-mapping id="nfs-share" 
                          source-dir-root="/srv/nfs-share" 
                          target-dir="/srv/nfs/share" 
                          options="rw"/> 
      </storage> 
      <primitive id="nfsserver001" class="ocf" provider="heartbeat" type="nfsserver"/> 
    </bundle>

*  Summary:  

   Unfortunately in the end, this reveals a significant issue as the "show stopper" because pcmk-init for pacemaker-remote inside the container conflicts with systemd, both of them require PID=1.

  Open questions:
  - what's the purpose of pcmk-init in the bundle container? 
  - If possible let the pacemaker bundle container still run systemd?
  - Is there any solid/stable approach to run nfsserver containers without systemd?

    • zzhou
      3 months ago by zzhou | Reply

      Back on this topic and update some major progress since hackweek 23.

      In summary, the show-stopper issue in the past has been addressed. As a result, nfsserver can now operate within the Pacemaker bundle containers and can be distributed across the cluster nodes. The successful mounting of nfsserver exports on various cluster nodes using NFSv3 and v4 protocols has been confirmed.

      The ongoing challenge lies with the showmount operation, specifically in the context of the NFS protocol GETADDR operation, which is currently not functioning correctly. Unfortunately, a resolution for this issue has not been identified yet.

      The sample configuration is provided below. Some refinement may still be necessary, and adjustments might be required to enhance certain subtle NFS functionalities. [1] for Dockerfile, [2] for crm configure show

      [1] ``` Dockerfile FROM opensuse/tumbleweed RUN zypper -n install systemd nfs-kernel-server vim iproute2 iputils pacemaker-remote gawk which

      RUN echo -e "[Unit]\nRequires=pacemakerremote.service\nAfter=pacemakerremote.service\nRequires=nfs-server.service\nAfter=nfs-server.service" > /usr/lib/systemd/system/runpcmkremoteandnfs_server.target RUN echo -e "[Service]\nExecStartPre=/usr/bin/mkdir -p /var/lib/nfs/sm /var/lib/nfs/sm.bak" > /usr/lib/systemd/system/nfs-server.service.d/10-prepare-dirs.conf

      RUN sed -e 's/STATDPORT=.*/STATDPORT="662"/' -i /etc/sysconfig/nfs RUN sed -e 's/LOCKDTCPPORT=.*/LOCKDTCPPORT="32768"/' -i /etc/sysconfig/nfs RUN sed -e 's/LOCKDUDPPORT=.*/LOCKDUDPPORT="32768"/' -i /etc/sysconfig/nfs

      CMD ["/usr/lib/systemd/systemd", "--system"] ```

      [2] ``` crm configure show

      primitive drbd1 ocf:linbit:drbd \ params drbdresource=nfsserver1 \ op monitor interval=15 role=Promoted timeout=20 \ op monitor interval=30 role=Unpromoted timeout=20 \ op start timeout=240 interval=0s \ op promote timeout=90 interval=0s \ op demote timeout=90 interval=0s \ op stop timeout=100 interval=0s primitive exportfs1 exportfs \ params directory="/srv/nfs/share" options="rw,mountpoint" clientspec="*" fsid=0 \ op monitor interval=30s timeout=40s \ op start timeout=60s interval=0s \ op stop timeout=120s interval=0s primitive fs1 Filesystem \ params device="/dev/drbd1" directory="/srv/nfs1" fstype=ext4 \ op monitor interval=30s timeout=40s \ op start timeout=60s interval=0s \ op stop timeout=60s interval=0s bundle nfsserver1 \ docker image=nfsserver options="--privileged --stop-signal SIGRTMIN+3" run-command="/usr/lib/systemd/systemd --system --unit=runpcmkremoteandnfsserver.target" \ network ip-range-start=192.168.1.200 port-mapping id=nfs1portsunrpc port=111 port-mapping id=nfs1portdata port=2049 port-mapping id=nfs1portrpcmount port=20048 port-mapping id=nfs1portstatd port=662 port-mapping id=nfs1portlockd-tcpudp port=32768 \ storage storage-mapping id=nfs1-state source-dir="/srv/nfs1/state" target-dir="/var/lib/nfs" options=rw storage-mapping id=nfs1-share source-dir="/srv/nfs1/share" target-dir="/srv/nfs/share" options=rw \ meta target-role=Started \ primitive exportfs1 clone drbd-nfs1 drbd1 \ meta promotable=true promoted-max=1 promoted-node-max=1 clone-max=2 clone-node-max=1 notify=true interleave=true colocation col-nfs1-on-drbd inf: nfsserver1 fs1 drbd-nfs1:Promoted order o-drbd-before-nfs1 Mandatory: drbd-nfs1:promote fs1:start nfsserver1

    Similar Projects

    This project is one of its kind!