Skip to main content

Use JuiceFS on Kubernetes

JuiceFS is very suitable as a storage layer for Kubernetes clusters, and there are currently two common usages.

JuiceFS CSI Driver

JuiceFS CSI Driver follows CSI specification, which implements the interface between the container orchestration system and the JuiceFS file system, and supports the dynamic configuration of JuiceFS volumes for use by Pod.

Prerequisites

  • Kubernetes 1.14+

Installation

JuiceFS CSI Driver has the following two installation methods.

Install with Helm

Helm is the package manager of Kubernetes, and Chart is the package managed by Helm. You can think of it as the equivalent of Homebrew formula, APT dpkg, or YUM RPM in Kubernetes.

This installation method requires Helm 3.1.0 and above. For the specific installation method, please refer to "Helm Installation Guide".

  1. Prepare a configuration file for setting the basic information of the storage class, for example: values.yaml, copy and complete the following configuration information. Among them, the backend part is JuiceFS file system related information, you can refer to JuiceFS Quick Start Guide for related content. If you are using a JuiceFS volume that has been created in advance, you only need to fill in the two items name and metaurl. The mountPod part can set the resource configuration of CPU and memory for the Pod using this driver. Unneeded items can be deleted, or its value can be left blank.
storageClasses:
- name: juicefs-sc
enabled: true
reclaimPolicy: Retain
backend:
name: "test"
metaurl: "redis://juicefs.afyq4z.0001.use1.cache.amazonaws.com/3"
storage: "s3"
accessKey: ""
secretKey: ""
bucket: "https://juicefs-test.s3.us-east-1.amazonaws.com"
mountPod:
resources:
limits:
cpu: "<cpu-limit>"
memory: "<memory-limit>"
requests:
cpu: "<cpu-request>"
memory: "<memory-request>"

On cloud platforms that support "role management", you can assign "service role" to Kubernetes nodes to achieve key-free access to the object storage API. In this case, there is no need to set the accessKey and secretKey in the configuration file.

  1. Execute the following three commands in sequence to deploy JuiceFS CSI Driver through Helm.
$ helm repo add juicefs-csi-driver https://juicedata.github.io/juicefs-csi-driver/
$ helm repo update
$ helm upgrade juicefs-csi-driver juicefs-csi-driver/juicefs-csi-driver --install -f ./values.yaml
  1. Check the deployment
  • Check Pods: the deployment will launch a StatefulSet named juicefs-csi-controller with replica 1 and a DaemonSet named juicefs-csi-node, so run kubectl -n kube-system get pods -l app.kubernetes.io/name=juicefs-csi-driver should see n+1 (where n is the number of worker nodes of the Kubernetes cluster) pods is running. For example:
$ kubectl -n kube-system get pods -l app.kubernetes.io/name=juicefs-csi-driver
NAME READY STATUS RESTARTS AGE
juicefs-csi-controller-0 3/3 Running 0 22m
juicefs-csi-node-v9tzb 3/3 Running 0 14m
  • Check secret: kubectl -n kube-system describe secret juicefs-sc-secret will show the secret with above backend fields in values.yaml:
$ kubectl -n kube-system describe secret juicefs-sc-secret
Name: juicefs-sc-secret
Namespace: kube-system
Labels: app.kubernetes.io/instance=juicefs-csi-driver
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=juicefs-csi-driver
app.kubernetes.io/version=0.7.0
helm.sh/chart=juicefs-csi-driver-0.1.0
Annotations: meta.helm.sh/release-name: juicefs-csi-driver
meta.helm.sh/release-namespace: default

Type: Opaque

Data
====
access-key: 0 bytes
bucket: 47 bytes
metaurl: 54 bytes
name: 4 bytes
secret-key: 0 bytes
storage: 2 bytes
  • Check storage class: kubectl get sc juicefs-sc will show the storage class like this:
$ kubectl get sc juicefs-sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
juicefs-sc csi.juicefs.com Retain Immediate false 69m

Install with kubectl

Since Kubernetes will discard some old APIs during the version change process, you need to select the applicable deployment file according to the version of Kubernetes you are using:

Kubernetes v1.18 and above

$ kubectl apply -f https://raw.githubusercontent.com/juicedata/juicefs-csi-driver/master/deploy/k8s.yaml

Version below Kubernetes v1.18

$ kubectl apply -f https://raw.githubusercontent.com/juicedata/juicefs-csi-driver/master/deploy/k8s_before_v1_18.yaml

Create storage class

Create a configuration file with reference to the following content, for example: juicefs-sc.yaml, fill in the configuration information of the JuiceFS file system in the stringData section:

apiVersion: v1
kind: Secret
metadata:
name: juicefs-sc-secret
namespace: kube-system
type: Opaque
stringData:
name: "test"
metaurl: "redis://juicefs.afyq4z.0001.use1.cache.amazonaws.com/3"
storage: "s3"
bucket: "https://juicefs-test.s3.us-east-1.amazonaws.com"
access-key: ""
secret-key: ""
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: juicefs-sc
provisioner: csi.juicefs.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:
csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system

Execute the command to deploy the storage class:

$ kubectl apply -f ./juicefs-sc.yaml

In addition, you can also extract the Secret part of the above configuration file and create it on the command line through kubectl:

$ kubectl -n kube-system create secret generic juicefs-sc-secret \
--from-literal=name=test \
--from-literal=metaurl=redis://juicefs.afyq4z.0001.use1.cache.amazonaws.com/3 \
--from-literal=storage=s3 \
--from-literal=bucket=https://juicefs-test.s3.us-east-1.amazonaws.com \
--from-literal=access-key="" \
--from-literal=secret-key=""

In this way, the storage class configuration file juicefs-sc.yaml should look like the following:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: juicefs-sc
provisioner: csi.juicefs.com
reclaimPolicy: Retain
parameters:
csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
csi.storage.k8s.io/node-publish-secret-namespace: kube-system
csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
csi.storage.k8s.io/provisioner-secret-namespace: kube-system

Then deploy the storage class through kubectl apply:

$ kubectl apply -f ./juicefs-sc.yaml

Use JuiceFS

JuiceFS CSI Driver supports both static and dynamic PV. You can either manually assign the PV created in advance to Pods, or you can dynamically create volumes through PVC when Pods are deployed.

For example, you can use the following configuration to create a configuration file named development.yaml, which creates a persistent volume for the Nginx container through PVC and mounts it to the container's /config directory:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: web-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Pi
storageClassName: juicefs-sc
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-run
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: linuxserver/nginx
ports:
- containerPort: 80
volumeMounts:
- mountPath: /config
name: web-data
volumes:
- name: web-data
persistentVolumeClaim:
claimName: web-pvc

Deploy Pods via kubectl apply:

$ kubectl apply -f ./development.yaml

After the deployment is successful, check the pods status:

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-run-7d6fb7d6df-cfsvp 1/1 Running 0 21m

We can simply use the kubectl exec command to view the file system mount status in the container:

$ kubectl exec nginx-run-7d6fb7d6df-cfsvp -- df -Th
Filesystem Type Size Used Avail Use% Mounted on
overlay overlay 40G 7.0G 34G 18% /
tmpfs tmpfs 64M 0 64M 0% /dev
tmpfs tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
JuiceFS:jfs fuse.juicefs 1.0P 180M 1.0P 1% /config
...

From the results returned from the container, we can see that it is in full compliance with expectations, and the JuiceFS volume has been mounted to the /config directory we specified.

When a PV is dynamically created through PVC as above, JuiceFS will create a directory with the same name as the PV in the root directory of the file system and mount it to the container. Execute the following command to view all PVs in the cluster:

$ kubectl get pv -A
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-b670c8a1-2962-497c-afa2-33bc8b8bb05d 10Pi RWX Retain Bound default/web-pvc juicefs-sc 34m

Mount the same JuiceFS storage through an external host, and you can see the PVs currently in use and the PVs that have been created.

For more details about JuiceFS CSI Driver please refer to project homepage.

Create more JuiceFS storage classes

You can repeat the previous steps according to actual needs to create any number of storage classes through JuiceFS CSI Driver. But pay attention to modifying the name of the storage class and the configuration information of the JuiceFS file system to avoid conflicts with the created storage class. For example, when using Helm, you can create a configuration file named jfs2.yaml:

storageClasses:
- name: jfs-sc2
enabled: true
reclaimPolicy: Retain
backend:
name: "jfs-2"
metaurl: "redis://example.abc.0001.use1.cache.amazonaws.com/3"
storage: "s3"
accessKey: ""
secretKey: ""
bucket: "https://jfs2.s3.us-east-1.amazonaws.com"

Execute the Helm command to deploy:

$ helm repo add juicefs-csi-driver https://juicedata.github.io/juicefs-csi-driver/
$ helm repo update
$ helm upgrade juicefs-csi-driver juicefs-csi-driver/juicefs-csi-driver --install -f ./jfs2.yaml

View the storage classes in the cluster:

$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
juicefs-sc csi.juicefs.com Retain Immediate false 88m
juicefs-sc2 csi.juicefs.com Retain Immediate false 13m
standard (default) k8s.io/minikube-hostpath Delete Immediate false 128m

Monitoring

JuiceFS CSI Driver can export Prometheus metrics at port 9567. For a description of all monitoring metrics, please refer to JuiceFS Metrics.

Configure Prometheus server

Add a job to prometheus.yml:

scrape_configs:
- job_name: 'juicefs'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
action: keep
regex: juicefs-mount
- source_labels: [__address__]
action: replace
regex: ([^:]+)(:\d+)?
replacement: $1:9567
target_label: __address__
- source_labels: [__meta_kubernetes_pod_node_name]
target_label: node
action: replace

Here we assume the Prometheus server is running inside Kubernetes cluster, if your Prometheus server is running outside Kubernetes cluster, make sure Kubernetes cluster nodes are reachable from Prometheus server, refer to this issue to add the api_server and tls_config client auth to the above configuration like this:

scrape_configs:
- job_name: 'juicefs'
kubernetes_sd_configs:
- api_server: <Kubernetes API Server>
role: pod
tls_config:
ca_file: <...>
cert_file: <...>
key_file: <...>
insecure_skip_verify: false
relabel_configs:
...
...

Configure Grafana dashboard

JuiceFS provides a dashboard template for Grafana, which can be imported to show the collected metrics in Prometheus.

Mount JuiceFS in the container

In some cases, you may need to mount JuiceFS volume directly in the container, which requires the use of the JuiceFS client in the container. You can refer to the following Dockerfile sample to integrate the JuiceFS client into the application image:

FROM alpine:latest
LABEL maintainer="Juicedata <https://juicefs.com>"

# Install JuiceFS client
RUN apk add --no-cache curl && \
JFS_LATEST_TAG=$(curl -s https://api.github.com/repos/juicedata/juicefs/releases/latest | grep 'tag_name' | cut -d '"' -f 4 | tr -d 'v') && \
wget "https://github.com/juicedata/juicefs/releases/download/v${JFS_LATEST_TAG}/juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \
tar -zxf "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \
install juicefs /usr/bin && \
rm juicefs "juicefs-${JFS_LATEST_TAG}-linux-amd64.tar.gz" && \
rm -rf /var/cache/apk/* && \
apk del curl

ENTRYPOINT ["/usr/bin/juicefs", "mount"]

Since JuiceFS needs to use the FUSE device to mount the file system, it is necessary to allow the container to run in a privileged mode when creating a Pod:

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-run
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: linuxserver/nginx
ports:
- containerPort: 80
securityContext:
privileged: true

⚠️ Risk Warning: After enabling the privileged mode of privileged: true, the container has access to all devices of the host, that is, it has full control of the host's kernel. Improper use will bring serious safety hazards, please conduct a sufficient safety assessment before using this method.