How to Use JuiceFS in Kubernetes

JuiceFS can be mounted as Volume in Kubernetes Pod for persistent storage.

Comparing to some other volume types such as awsElasticBlockStore, JuiceFS provides more cloud native features:

  • Support ReadWriteMany access mode, allowing shared storage among multiple pods.

  • Your file system can be accessed across multiple Availability Zones (AZs) for high availability and durability.

  • Elastic capacity up to 10 PiB, reducing the operation overhead

  • Support fast copy to duplicate existing volume on dynamic provisioning

There are various way to use JuiceFS in kubernetes:

  • CSI (recommended for Kubernetes v1.13+)

  • flexVolume (available since Kubernetes v1.2)

  • hostPath (no version restriction)

CSI

The JuiceFS Container Storage Interface (CSI) Driver implements the CSI specification for container orchestrators to manage the lifecycle of JuiceFS filesystems.

Installation

Deploy the driver:

kubectl apply -f https://raw.githubusercontent.com/juicedata/juicefs-csi-driver/master/deploy/k8s.yaml

You may now use CSI volumes through the familiar Kubernetes storage API objects: PersistentVolumeClaims, PersistentVolumes, and StorageClasses.

Upgrading

We have two conponents in JuiceFS CSI Driver to upgrade:

  • JuiceFS CSI Driver itself

  • JuiceFS Client in JuiceFS CSI Driver image

Upgrade JuiceFS CSI Driver

  1. Stop all Pods using JuiceFS CSI Driver

  2. Upgrade containers of JuiceFS CSI Driver

    • If you are using JuiceFS CSI Driver image with latest tag, then simply run

      kubectl rollout restart -f k8s.yaml
      

      and make sure juicefs-csi-controller and juicefs-csi-node Pods are restarted.

    • If you have pinned JuiceFS CSI Driver to a specific version, then modify your k8s.yaml, to a newer version, then

      kubectl apply -f k8s.yaml
      

      For example if you upgrade driver from 0.3.0 to 0.4.0:

      $ sed -i 's+juicedata/juicefs-csi-driver:0.3.0+juicedata/juicefs-csi-driver:0.4.0+g' k8s.yaml
      $ cat k8s.yaml
      ...
      ---
      apiVersion: apps/v1
      kind: StatefulSet
      ...
          spec:
            containers:
            - args:
              - --endpoint=$(CSI_ENDPOINT)
              - --logtostderr
              - --nodeid=$(NODE_ID)
              - --v=5
              env:
              - name: CSI_ENDPOINT
                value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
              image: juicedata/juicefs-csi-driver:0.4.0
      ...
      $ kubectl apply -f k8s.yaml
      

Upgrade JuiceFS Client in JuiceFS CSI Driver

Automatically upgrade of JuiceFS Client in JuiceFS CSI Driver image is now supported, and can be disabled. JuiceFS Client will upgrade automatically everytime before mounting a PersistentVolume if enabled. We also offer different ways to control this function:

  • If you are using JuiceFS CSI Driver image with latest tag, then auto-upgrade is enabled by default. JuiceFS Client will try to upgrade itself everytime before mounting, a very short check time will be taken if no upgrade.

  • If you have pinned JuiceFS CSI Driver to a specific version, then auto-upgrade is disabled by default. You can still enable this through environment variables:

    • two environment variables are supported to configure auto-upgrade:

      • JFS_AUTO_UPGRADE: enable auto-upgrade if set to none empty value, otherwise disable

      • JFS_AUTO_UPGRADE_TIMEOUT: the timeout for auto-upgrade in seconds, default 10

    • configure the above environment variables when deploy JuiceFS CSI Driver:

      $ cat k8s.yaml
      ...
      ---
      apiVersion: apps/v1
      kind: StatefulSet
      ...
          spec:
            containers:
            - args:
              - --endpoint=$(CSI_ENDPOINT)
              - --logtostderr
              - --nodeid=$(NODE_ID)
              - --v=5
              env:
              - name: CSI_ENDPOINT
                value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
              - name: JFS_AUTO_UPGRADE
                value: "true"
              - name: JFS_AUTO_UPGRADE_TIMEOUT
                value: "15"
              image: juicedata/juicefs-csi-driver:0.4.0
      ...
      $ kubectl apply -f k8s.yaml
      

      Notice JuiceFS CSI Driver need to be re-deployed to ensure the environment variables are applied to controller and node driver

Dynamic Provisioning

You can enable automatic creation/deletion of JuiceFS volumes by creating a StorageClass pointing to the CSI plugin, e.g.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs-sc
  namespace: default
provisioner: csi.juicefs.com
parameters:
  csi.storage.k8s.io/provisioner-secret-name: juicefs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-publish-secret-name: juicefs-secret
  csi.storage.k8s.io/node-publish-secret-namespace: default

juicefs-secret is the name of Kubernetes Secret containing credentials for JuiceFS and object storage access, including name, token, accesskey, secretkey. You should create the filesystem manually, to meet the requirement of StorageClass. Every persistent volume is a subdirectory of the JuiceFS volume. An example command to create Secret is shown below:

kubectl create secret generic juicefs-secret \
--from-literal=name=${JUICEFS_NAME} \
--from-literal=token=${JUICEFS_TOKEN} \
--from-literal=accesskey=${JUICEFS_ACCESSKEY} \
--from-literal=secretkey=${JUICEFS_SECRETKEY}

Dynamic provisioning is triggered by the creation of a PersistentVolumeClaim object. The following PersistentVolumeClaim, for example, triggers dynamic provisioning using the StorageClass above.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: juicefs-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Pi
  storageClassName: juicefs-sc

When volume provisioning is invoked, the parameters and all referenced secrets in StorageClass are passed to the CSI plugin “csi.juicefs.com” via a CreateVolume call. In response, the JuiceFS CSI driver provisions a new volume and then automatically create a PersistentVolume object to represent the new volume. Kubernetes then binds the new PersistentVolume object to the PersistentVolumeClaim, making it ready to use.

Dynamic provisioning making it splendidly easy to create/delete persistent storage for applications. In case you want to access contents in an existing JuiceFS file system, pre-provisioned volume will be the right solution.

Note

Since JuiceFS is an elastic filesystem it doesn’t really enforce any filesystem capacity. The actual storage capacity value in persistence volume and persistence volume claim is not used when creating the filesystem. However, since the storage capacity is a required field by Kubernetes, you must specify the value and you can use any valid value e.g. 10Pi for the capacity.

Pre-Provisioned Volumes

You can always create a PersistenVolume to mount a pre-existing JuiceFS file system. and exposed in Kubernetes. The following PersistentVolume, for example, exposes the JuiceFs file system “juicefs-name” to the CSI driver with the name “juicefs-pv”.

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: juicefs-pv
  labels:
    juicefs-name: ten-pb-fs
spec:
  capacity:
    storage: 10Pi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: csi.juicefs.com
    volumeHandle: juicefs-name
    fsType: juicefs
    nodePublishSecretRef:
      name: juicefs-secret
      namespace: default

The value of volumeHandle should be the name of JuiceFS file system while juicefs-secret should contain required credentials for the access such as name, token, accesskey and secretkey.

You may then bind the PersistentVolume above by specifying matchLabels in PersistentVolumeClaim as below:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: juicefs-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Pi
  selector:
    matchLabels:
      juicefs-name: ten-pb-fs

Using JuiceFS in applications

You can reference a PersistentVolumeClaim that is bound to a CSI volume in any pod or pod template, e.g.

---
apiVersion: v1
kind: Pod
metadata:
  name: juicefs-app
  namespace: default
spec:
  containers:
  - args:
    - -c
    - while true; do echo $(date -u) >> /data/out.txt; sleep 5; done
    command:
    - /bin/sh
    image: centos
    name: app
    volumeMounts:
    - mountPath: /data
      name: juicefs-pv
  volumes:
  - name: juicefs-pv
    persistentVolumeClaim:
      claimName: juicefs-pvc

The complete code and steps can be found in basic and static provisioning examples.

Mount options

Mount options for JuiceFS can be provided in csi/volumeAttributes/mountOptions to fine tune for various scenarios. Example patch as below:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: juicefs-with-metacache
spec:
  csi:
    volumeAttributes:
      mountOptions: "metacache,cache-size=100,cache-dir=/var/foo"

Find more in mount options example.

Sub Path

You may create PersistentVolume to represent a directory in JuiceFS instead of the whole file system by providing the attribute csi/volumeAttributes/subPath e.g.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: juicefs-name-subpath
spec:
  csi:
    volumeAttributes:
      subPath: sub-path-in-juicefs

Find more in sub path example.

ReadWriteMany

JuiceFS supports ReadWriteMany access mode to be shared by multiple pods, e.g.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name:  scaling-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: centos
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo $(date -u) >> /data/out-$(POD).txt; sleep 5; done"]
        env:
        - name: POD
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: data
          mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: juicefs-shared

All the replicas in this Deployment shares the same volume in this example.

Find more in read write many example.

Version Compatibility

The Kubernetes implementation of the Container Storage Interface (CSI) has been promoted to GA in the Kubernetes v1.13 release. Support for CSI was introduced as alpha in Kubernetes v1.9 release, and promoted to beta in the Kubernetes v1.10 release.

To use JuiceFS in earlier versions of Kubernetes, try flexVolume or hostPath.

flexVolume

The client juicefs can also serve as a plugin to flex volume, which support the init, mount and unmount call-outs.

You should place juicefs at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/juicedata~juicefs/juicefs on all the nodes, and make sure that it’s executable. juicefs requires Python2.6+ or Python3.

$ wget https://juicefs.com/static/juicefs
$ chmod +x juicefs
$ cp juicefs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/juicedata~juicefs/juicefs

Plain Secrets

Then you could specify a volume as flexVolume and use juicedata/juicefs as driver:

volumes:
- name: test
  flexVolume:
    driver: "juicedata/juicefs"
    options:
      name: "my-jfs"
      token: "TOKEN"
      accesskey: "ACCESSKEY"
      secretkey: "SECRETKEY"

You could also specify other command arguments inside the options, for example, cacheSize and cacheDir, see juicefs mount -h for the full list.

A dedicated mount point will be created to each Pod, which has the same life cycle with the Pod.

Kubernetes Secrets

You may not want to put all the secrets in the Pod config, so you could create secrets for them by:

apiVersion: v1
kind: Secret
metadata:
  name: my-jfs
type: juicedata/juicefs
data:
  token: BASE64(TOKEN)
  accesskey: BASE64(ACCESSKEY)
  secretkey: BASE64(SECRETKEY)

Here is the way to generate base64 encoding for your secrets:

$ echo -n TOKEN | base64

Then you could reference the secrets for JuiceFS:

volumes:
- name: test
  flexVolume:
    driver: "juicedata/juicefs"
    secretRef:
      name: “my-jfs”
    options:
      name: "my-jfs"

JuiceFS configs

The Flex volume driver will pass these options (including secrets) to juicefs using command line arguments, for example:

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/juicedata~juicefs/juicefs mount /var/lib/kubelet/pods/aa8ef19f-ba1f-11e7-ab55-0800279245af/volumes/juicedata~juicefs/test '{"kubernetes.io/fsType":"","kubernetes.io/pod.name":"test","kubernetes.io/pod.namespace":"default","kubernetes.io/pod.uid":"aa8ef19f-ba1f-11e7-ab55-0800279245af","kubernetes.io/pvOrVolumeName":"test","kubernetes.io/readwrite":"rw","kubernetes.io/secret/accesskey":"","kubernetes.io/secret/secretkey":"","kubernetes.io/secret/token":"TOKEN","kubernetes.io/serviceAccount.name":"default","name":"NAME"}'

So there is another way to hide these secrets from command line arguments by provide a config file for JuiceFS at /root/.juicefs/my-jfs.conf. which is a JSON file containing secrets in it.

{"token": "TOKEN", "accesskey”: "ACCESSKEY", "secretkey": "SECRETKEY"}

Then you don’t need to put all these secrets in Pod config or create secrets, just have this:

volumes:
- name: test
  flexVolume:
    driver: "juicedata/juicefs"
    options:
      name: "NAME"

hostPath

We can mount JuiceFS to a fixed mount point (for example /jfs) on all the hosts, then use the hostPath driver to link it (or event part of it) into container.

volumes:
- name: test-volume
  hostPath:
    path: /jfs
    # this field is optional
    type: Directory

All the containers in the same node will share the same mount point (FUSE client).

Note: HostPath will be deprecated by Kubernetes in the future. Local volumes will be the preferred approach for most use cases going forward.

Road Map

More features for Kubernetes are under development.

Feel free to contact us by clicking the conversation icon in bottom right.