Static Provisioning Of JuiceFS Using in Kubernetes

This document shows how to make a static provisioned JuiceFS PersistentVolume (PV) mounted inside container.

Prerequisite

To create the CSI Driver Secret in Kubernetes, the required fields for the community edition and the cloud service edition are different, as follows:

Community edition

Take Amazon S3 as an example:

apiVersion: v1
kind: Secret
metadata:
  name: juicefs-secret
type: Opaque
stringData:
  name: <NAME>
  metaurl: redis://[:<PASSWORD>]@<HOST>:6379[/<DB>]
  storage: s3
  bucket: https://<BUCKET>.s3.<REGION>.amazonaws.com
  access-key: <ACCESS_KEY>
  secret-key: <SECRET_KEY>

• name: The JuiceFS file system name.
• metaurl: Connection URL for metadata engine (e.g. Redis). Read this document for more information.
• storage: Object storage type, such as s3, gs, oss. Read this document for the full supported list.
• bucket: Bucket URL. Read this document to learn how to setup different object storage.
• access-key: Access key.
• secret-key: Secret key.

Replace fields enclosed by <> with your own environment variables. The fields enclosed [] is optional which related your deployment environment.

You should ensure:

1. The access-key, secret-key pair has GetObject, PutObject, DeleteObject permission for the object bucket
2. The Redis DB is clean and the password (if provided) is right

You can execute the juicefs format command to ensure the Secret is OK:

juicefs format --storage=s3 --bucket=https://<BUCKET>.s3.<REGION>.amazonaws.com \
    --access-key=<ACCESS_KEY> --secret-key=<SECRET_KEY> \
    redis://[:<PASSWORD>]@<HOST>:6379[/<DB>] <NAME>

Cloud service edition

apiVersion: v1
kind: Secret
metadata:
  name: juicefs-secret
type: Opaque
stringData:
  name: ${JUICEFS_NAME}
  token: ${JUICEFS_TOKEN}
  access-key: ${JUICEFS_ACCESSKEY}
  secret-key: ${JUICEFS_SECRETKEY}

• name: JuiceFS file system name
• token: JuiceFS managed token. Read this document for more details.
• access-key: Object storage access key
• secret-key: Object storage secret key

You should ensure access-key and secret-key pair has GetObject, PutObject, DeleteObject permission for the object bucket.

Deploy

Create PersistentVolume (PV), PersistentVolumeClaim (PVC) and sample pod:

note

The PV volumeHandle needs to be unique within the cluster, just use the PV name.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: juicefs-pv
  labels:
    juicefs-name: ten-pb-fs
spec:
  capacity:
    storage: 10Pi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  csi:
    driver: csi.juicefs.com
    volumeHandle: juicefs-pv
    fsType: juicefs
    nodePublishSecretRef:
      name: juicefs-secret
      namespace: default
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: juicefs-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  storageClassName: ""
  resources:
    requests:
      storage: 10Pi
  selector:
    matchLabels:
      juicefs-name: ten-pb-fs
---
apiVersion: v1
kind: Pod
metadata:
  name: juicefs-app
  namespace: default
spec:
  containers:
  - args:
    - -c
    - while true; do echo $(date -u) >> /data/out.txt; sleep 5; done
    command:
    - /bin/sh
    image: centos
    name: app
    volumeMounts:
    - mountPath: /data
      name: data
    resources:
      requests:
        cpu: 10m
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: juicefs-pvc

Check JuiceFS file system is used

After all objects are created, verify that a 10 Pi PV is created:

kubectl get pv

Verify the pod is running:

kubectl get pods

Verify that data is written onto JuiceFS file system:

kubectl exec -ti juicefs-app -- tail -f /data/out.txt

Verify the directory created as PV in JuiceFS file system by mounting it in a host:

juicefs mount -d redis://[:<PASSWORD>]@<HOST>:6379[/<DB>] /jfs