Use JuiceFS in Kubernetes

Compared to other cloud storage services (like AWS EBS), JuiceFS better suits the cloud environment:

• As a distributed shared storage, JuiceFS can be mounted into multiple pods, achieving concurrent reading / writing.
• Mount is not limited by availability zones, JuiceFS can be mounted in both cloud and local hosts.
• JuiceFS metadata service is by default deployed across two AZs, achieving high availability.
• Virtually unlimited storage capacity (depending on the object storage of your choice), no need to plan ahead for scaling.

You can use JuiceFS in Kubernetes with the following methods:

• hostPath, for all Kubernetes versions.
• CSI, for Kubernetes v1.13 and above (CSI support is promoted to GA in v1.13).

hostPath Volume

If you simply need to use JuiceFS inside Kubernetes pods, without any special requirements (e.g. isolation, permission control), then hostPath can be a good practice, which is also really easy to setup:

1. Install and mount JuiceFS on all Kubernetes worker nodes, Automated Deployment is recommended for this type of work.

2. Use hostPath volume inside pod definition, and mount a JuiceFS sub-directory to container:

   apiVersion: v1
   kind: Pod
   metadata:
     name: juicefs-app
   spec:
     containers:
       - ...
         volumeMounts:
           - name: jfs-data
             mountPath: /opt/app-data
     volumes:
       - name: jfs-data
         hostPath:
           # Assuming JuiceFS is mounted on /jfs
           path: "/jfs/myapp/"
           type: Directory

In comparison to using JuiceFS CSI Driver, hostPath is a much more simple practice, and easier to debug when things go wrong, but notice that:

• For ease of management, generally all pods use the same host mount point. Lack of isolation may lead to data security issues, and obviously, you won't be able to adjust JuiceFS mount parameters separately for each application. Please evaluate carefully.

• All worker nodes should mount JuiceFS in advance, so when adding a new node to the cluster, JuiceFS needs to be installed and mounted during the initialization process, otherwise the new node does not have a JuiceFS mount point, and the container will not be created.

• The system resources (such as CPU, memory, etc.) occupied by the JuiceFS mounting process on the host are not controlled by Kubernetes, and may occupy too many host resources. You can consider using system-reserved to properly adjust the system resource reservation of Kubernetes, to reserve more resources for the JuiceFS mount process.

• If the JuiceFS mount process on the host exits unexpectedly, the application pod will not be able to access the mount point normally. In this case, the JuiceFS file system needs to be remounted and the application pod must be rebuilt. However, JuiceFS CSI Driver solves this problem well by providing the "Automatic Mount Point Recovery" mechanism.

• The lifecycle of the JuiceFS mount point is bound to the host. If you need to remount the file system, you can use the seamless remount feature of the JuiceFS client to ensure that the remount will not affect the access of the application pod to the mount point. Refer to "Upgrade & Seamless remount" for details.

• If you're using Docker as Kubernetes container runtime, it's best to start JuiceFS mount prior to Docker in startup order, to avoid containers being created before JuiceFS is properly mounted. For systemd, you can use below unit file to manually control startup order:

  /etc/systemd/system/docker.service.d/override.conf

  [Unit]
  # Use below command to obtain JuiceFS mount service name
  # systemctl list-units | grep "\.mount"
  After=network-online.target firewalld.service containerd.service jfs.mount

CSI Volume

If flexibility and observability is a must for your scenario, we recommend using CSI Volume to integrate JuiceFS into Kubernetes. JuiceFS CSI comes with its own documentation, see JuiceFS CSI Driver.