Skip to main content

How to Use JuiceFS in Kubernetes

JuiceFS can be mounted as Volume in Kubernetes Pod for persistent storage.

Comparing to some other volume types such as awsElasticBlockStore, JuiceFS provides more cloud native features:

  • Support ReadWriteMany access mode, allowing shared storage among multiple pods.
  • Support ReadOnlyMany access mode, allowing shared readonly storage among multiple pods.
  • Your file system can be accessed across multiple Availability Zones (AZs) for high availability and durability.
  • Elastic capacity up to 10 PiB, reducing the operation overhead
  • Support fast copy to duplicate existing volume on dynamic provisioning

There are various way to use JuiceFS in kubernetes:

  • CSI (recommended for Kubernetes v1.13+)
  • flexVolume (available since Kubernetes v1.2)
  • hostPath (no version restriction)

CSI

The JuiceFS Container Storage Interface (CSI) Driver implements the CSI specification for container orchestrators to manage the lifecycle of JuiceFS filesystems.

Installation

Deploy the driver:

kubectl apply -f https://raw.githubusercontent.com/juicedata/juicefs-csi-driver/master/deploy/k8s.yaml

You may now use CSI volumes through the familiar Kubernetes storage API objects: PersistentVolumeClaims, PersistentVolumes, and StorageClasses.

Upgrading

We have two conponents in JuiceFS CSI Driver to upgrade:

  • JuiceFS CSI Driver itself
  • JuiceFS Client in JuiceFS CSI Driver image

Upgrade JuiceFS CSI Driver

CSI Driver version >= v0.10

Juicefs CSI Driver separated JuiceFS client from CSI Driver since v0.10.0, CSI Driver upgrade will not interrupt existing PVs. If CSI Driver version >= v0.10.0, do operations below:

  • Modify your k8s.yaml to a newer version, then run kubectl apply -f k8s.yaml.
  • Alternatively, if JuiceFS CSI driver is installed using Helm, you can also use Helm to upgrade it.
CSI Driver version < v0.10
  • Minor version upgrade

    Upgrade of CSI Driver requires restart the DaemonSet, which has all the JuiceFS client running inside. The restart will cause all PVs become unavailable, so we need to stop all the application pod first.

    • Stop all pods using this driver.

    • Upgrade driver:

      • If you\'re using latest tag, simple run kubectl rollout restart -f k8s.yaml and make sure juicefs-csi-controller and juicefs-csi-node pods are restarted.

      • If you have pinned JuiceFS CSI Driver to a specific version, then modify your k8s.yaml, to a newer version, then

        kubectl apply -f k8s.yaml

        For example if you upgrade driver from 0.3.0 to 0.4.0:

        $ sed -i 's+juicedata/juicefs-csi-driver:0.3.0+juicedata/juicefs-csi-driver:0.4.0+g' k8s.yaml
        $ cat k8s.yaml
        ...
        ---
        apiVersion: apps/v1
        kind: StatefulSet
        ...
        spec:
        containers:
        - args:
        - --endpoint=$(CSI_ENDPOINT)
        - --logtostderr
        - --nodeid=$(NODE_ID)
        - --v=5
        env:
        - name: CSI_ENDPOINT
        value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
        image: juicedata/juicefs-csi-driver:0.4.0
        ...
        $ kubectl apply -f k8s.yaml
    • Alternatively, if JuiceFS CSI driver is installed using Helm, you can also use Helm to upgrade it.

    • Start all the application pods.

  • Cross-version upgrade

    If you want to upgrade CSI Driver from v0.9.0 to v0.10.0+, follow How to upgrade CSI Driver from v0.9.0 to v0.10.0+.

Upgrade JuiceFS Client in JuiceFS CSI Driver

Automatically upgrade of JuiceFS Client in JuiceFS CSI Driver image is now supported, and can be disabled. JuiceFS Client will upgrade automatically everytime before mounting a PersistentVolume if enabled.

Auto-upgrade in JuiceFS CSI Driver image is disabled by default. You can still enable this through environment variables:

  • two environment variables are supported to configure auto-upgrade:

    • JFS_AUTO_UPGRADE: enable auto-upgrade if set to none empty value, otherwise disable
    • JFS_AUTO_UPGRADE_TIMEOUT: the timeout for auto-upgrade in seconds, default 10
  • configure the above environment variables when deploy JuiceFS CSI Driver:

    $ cat k8s.yaml
    ...
    ---
    apiVersion: apps/v1
    kind: StatefulSet
    ...
    spec:
    containers:
    - args:
    - --endpoint=$(CSI_ENDPOINT)
    - --logtostderr
    - --nodeid=$(NODE_ID)
    - --v=5
    env:
    - name: CSI_ENDPOINT
    value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
    - name: JFS_AUTO_UPGRADE
    value: "true"
    - name: JFS_AUTO_UPGRADE_TIMEOUT
    value: "15"
    image: juicedata/juicefs-csi-driver:0.4.0
    ...
    $ kubectl apply -f k8s.yaml

    Notice JuiceFS CSI Driver need to be re-deployed to ensure the environment variables are applied to controller and node driver

Dynamic Provisioning

You can enable automatic creation/deletion of JuiceFS volumes by creating a StorageClass pointing to the CSI plugin, e.g.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: juicefs-sc
provisioner: csi.juicefs.com
parameters:
csi.storage.k8s.io/provisioner-secret-name: juicefs-secret
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/node-publish-secret-name: juicefs-secret
csi.storage.k8s.io/node-publish-secret-namespace: default

juicefs-secret is the name of Kubernetes [Secret]{.title-ref} containing credentials for JuiceFS and object storage access, including name, token, accesskey, secretkey. You should create the filesystem manually, to meet the requirement of StorageClass. Every persistent volume is a subdirectory of the JuiceFS volume. An example command to create Secret is shown below:

kubectl create secret generic juicefs-secret \
--from-literal=name=${JUICEFS_NAME} \
--from-literal=token=${JUICEFS_TOKEN} \
--from-literal=accesskey=${JUICEFS_ACCESSKEY} \
--from-literal=secretkey=${JUICEFS_SECRETKEY}

Dynamic provisioning is triggered by the creation of a PersistentVolumeClaim object. The following PersistentVolumeClaim, for example, triggers dynamic provisioning using the StorageClass above.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: juicefs-pvc
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Pi
storageClassName: juicefs-sc

When volume provisioning is invoked, the parameters and all referenced secrets in StorageClass are passed to the CSI plugin \"csi.juicefs.com\" via a CreateVolume call. In response, the JuiceFS CSI driver provisions a new volume and then automatically create a PersistentVolume object to represent the new volume. Kubernetes then binds the new PersistentVolume object to the PersistentVolumeClaim, making it ready to use.

Dynamic provisioning making it splendidly easy to create/delete persistent storage for applications. In case you want to access contents in an existing JuiceFS file system, pre-provisioned volume will be the right solution.

Note: Since JuiceFS is an elastic filesystem it doesn\'t really enforce any filesystem capacity. The actual storage capacity value in persistence volume and persistence volume claim is not used when creating the filesystem. However, since the storage capacity is a required field by Kubernetes, you must specify the value and you can use any valid value e.g. 10Pi for the capacity.

Pre-Provisioned Volumes

You can always create a PersistenVolume to mount a pre-existing JuiceFS file system. and exposed in Kubernetes. The following PersistentVolume, for example, exposes the JuiceFs file system \"juicefs-name\" to the CSI driver with the name \"juicefs-pv\".

---
apiVersion: v1
kind: PersistentVolume
metadata:
name: juicefs-pv
labels:
juicefs-name: ten-pb-fs
spec:
capacity:
storage: 10Pi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
csi:
driver: csi.juicefs.com
volumeHandle: juicefs-name
fsType: juicefs
nodePublishSecretRef:
name: juicefs-secret
namespace: default

The value of volumeHandle should be the name of JuiceFS file system while juicefs-secret should contain required credentials for the access such as name, token, accesskey and secretkey.

You may then bind the PersistentVolume above by specifying matchLabels in PersistentVolumeClaim as below:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: juicefs-pvc
namespace: default
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 10Pi
selector:
matchLabels:
juicefs-name: ten-pb-fs

Using JuiceFS in applications

You can reference a PersistentVolumeClaim that is bound to a CSI volume in any pod or pod template, e.g.

---
apiVersion: v1
kind: Pod
metadata:
name: juicefs-app
namespace: default
spec:
containers:
- args:
- -c
- while true; do echo $(date -u) >> /data/out.txt; sleep 5; done
command:
- /bin/sh
image: centos
name: app
volumeMounts:
- mountPath: /data
name: juicefs-pv
volumes:
- name: juicefs-pv
persistentVolumeClaim:
claimName: juicefs-pvc

The complete code and steps can be found in basic and static provisioning examples.

Mount options

Mount options for JuiceFS can be provided in csi/volumeAttributes/mountOptions to fine tune for various scenarios. Example patch as below:

apiVersion: v1
kind: PersistentVolume
metadata:
name: juicefs-with-metacache
spec:
csi:
volumeAttributes:
mountOptions: "metacache,cache-size=100,cache-dir=/var/foo"

Find more in mount options example.

Sub Path

You may create PersistentVolume to represent a directory in JuiceFS instead of the whole file system by providing the attribute csi/volumeAttributes/subPath e.g.

apiVersion: v1
kind: PersistentVolume
metadata:
name: juicefs-name-subpath
spec:
csi:
volumeAttributes:
subPath: sub-path-in-juicefs

Find more in sub path example.

ReadWriteMany

JuiceFS supports ReadWriteMany access mode to be shared by multiple pods, e.g.

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: scaling-app
spec:
template:
spec:
containers:
- name: app
image: centos
command: ["/bin/sh"]
args: ["-c", "while true; do echo $(date -u) >> /data/out-$(POD).txt; sleep 5; done"]
env:
- name: POD
valueFrom:
fieldRef:
fieldPath: metadata.name
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: juicefs-shared

All the replicas in this Deployment shares the same volume in this example.

Find more in read write many example.

Version Compatibility

The Kubernetes implementation of the Container Storage Interface (CSI) has been promoted to GA in the Kubernetes v1.13 release. Support for CSI was introduced as alpha in Kubernetes v1.9 release, and promoted to beta in the Kubernetes v1.10 release.

To use JuiceFS in earlier versions of Kubernetes, try flexVolume or hostPath.

flexVolume

The client juicefs can also serve as a plugin to flex volume, which support the init, mount and unmount call-outs.

You should place juicefs at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/juicedata~juicefs/juicefs on all the nodes, and make sure that it's executable. juicefs requires Python2.6+ or Python3.

$ wget https://juicefs.com/static/juicefs
$ chmod +x juicefs
$ cp juicefs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/juicedata~juicefs/juicefs

Plain Secrets

Then you could specify a volume as flexVolume and use juicedata/juicefs as driver:

volumes:
- name: test
flexVolume:
driver: "juicedata/juicefs"
options:
name: "my-jfs"
token: "TOKEN"
accesskey: "ACCESSKEY"
secretkey: "SECRETKEY"

You could also specify other command arguments inside the options, for example, cacheSize and cacheDir, see juicefs mount -h for the full list.

A dedicated mount point will be created to each Pod, which has the same life cycle with the Pod.

Kubernetes Secrets

You may not want to put all the secrets in the Pod config, so you could create secrets for them by:

apiVersion: v1
kind: Secret
metadata:
name: my-jfs
type: juicedata/juicefs
data:
token: BASE64(TOKEN)
accesskey: BASE64(ACCESSKEY)
secretkey: BASE64(SECRETKEY)

Here is the way to generate base64 encoding for your secrets:

$ echo -n TOKEN | base64

Then you could reference the secrets for JuiceFS:

volumes:
- name: test
flexVolume:
driver: "juicedata/juicefs"
secretRef:
name: β€œmy-jfs”
options:
name: "my-jfs"

JuiceFS configs

The Flex volume driver will pass these options (including secrets) to juicefs using command line arguments, for example:

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/juicedata~juicefs/juicefs mount /var/lib/kubelet/pods/aa8ef19f-ba1f-11e7-ab55-0800279245af/volumes/juicedata~juicefs/test '{"kubernetes.io/fsType":"","kubernetes.io/pod.name":"test","kubernetes.io/pod.namespace":"default","kubernetes.io/pod.uid":"aa8ef19f-ba1f-11e7-ab55-0800279245af","kubernetes.io/pvOrVolumeName":"test","kubernetes.io/readwrite":"rw","kubernetes.io/secret/accesskey":"","kubernetes.io/secret/secretkey":"","kubernetes.io/secret/token":"TOKEN","kubernetes.io/serviceAccount.name":"default","name":"NAME"}'

So there is another way to hide these secrets from command line arguments by provide a config file for JuiceFS at /root/.juicefs/my-jfs.conf. which is a JSON file containing secrets in it.

{"token": "TOKEN", "accesskey”: "ACCESSKEY", "secretkey": "SECRETKEY"}

Then you don't need to put all these secrets in Pod config or create secrets, just have this:

volumes:
- name: test
flexVolume:
driver: "juicedata/juicefs"
options:
name: "NAME"

hostPath

We can mount JuiceFS to a fixed mount point (for example /jfs) on all the hosts, then use the hostPath driver to link it (or event part of it) into container.

volumes:
- name: test-volume
hostPath:
path: /jfs
# this field is optional
type: Directory

All the containers in the same node will share the same mount point (FUSE client).

Note: HostPath will be deprecated by Kubernetes in the future. Local volumes will be the preferred approach for most use cases going forward.

Road Map

More features for Kubernetes are under development.

Feel free to contact us by clicking the conversation icon in bottom right.