Skip to main content

Using JuiceFS on ACK in Serverless Container via Fluid

Alibaba Cloud ECI (Elastic Container Instance) is the implementation of Serverless Container on Alibaba Cloud. Similar Kubernetes products based on Serverless Container include AWS Fargate and Azure AKS ACI. Fluid is an open source Kubernetes-native Distributed Dataset Orchestrator and Accelerator for data-intensive applications, such as big data and AI. Fluid has been included as ACK-AI suit in ACK, users can directly install and use JuiceFSRuntime.

Architecture

Fluid injects FUSE client (different runtime are supported, including JuiceFS) as a sidecar container into serverless pod, by detecting Fluid PVC (PersistentVolumeClaim) used in the application pod. JuiceFS Client runs independently inside the sidecar container, sharing lifecycle with the application pod.

Fluid-arch

In order to improve data accelerate capabilities, a dedicated cache cluster is recommended, use ECS nodes to run worker pods and provide distributed cache for application pods. The cache cluster runs on ECS nodes as a StatefulSet, while the JuiceFS clients used by application pods runs as sidecar containers on ECI nodes.

Fluid-JuiceFS-arch

Prepare environment

Fluid is available on ACK Pro clusters (Kubernetes >= 1.18). Before proceeding, you should first create a cluster.

Install Fluid

From the ACK console, select "Applications → Cloud-native AI Suite", and click "Deploy" to install the Cloud-native AI Suite.

Fluid-install

Select "Fluid Data Acceleration":

Fluid-install-choose

Install virtual node

In Kubernetes cluster, ECI applications are scheduled to a virtual node and actually run on a temporary ECS node. In the "Add-ons" of "Operations", install "ACK Virtual Node":

Fluid-vk

Use JuiceFS

JuiceFS is an architecture that separates metadata and actual file data. Metadata will be stored in the Metadata Service, and data will be uploaded to an object storage service of your choice. Since distributed cache is a Cloud Service exclusive feature, we'll use JuiceFS Cloud Service to demonstrate.

Before starting, make sure you have created a JuiceFS file system, and have successfully mounted and verified to be able to read and write normally.

Create Secret

Create a secret in the ACK cluster, and provide the JuiceFS Volume Token and object storage credentials.

Fluid-create-secret

Create JuiceFSRuntime

In "Custom Resources", click "Create from YAML" button to create JuiceFSRuntime, and specify information such as replicas and cache directories.

Fluid-create-runtime

The JuiceFSRuntime example is as follows:

apiVersion: data.fluid.io/v1alpha1
kind: JuiceFSRuntime
metadata:
name: jfsdemo
spec:
# The number of nodes in the cache cluster
replicas: 5
podMetadata:
labels:
# Use this label to run cache cluster within ECI
# alibabacloud.com/eci: "true"
tieredstore:
levels:
- mediumtype: MEM
path: /dev/shm
quota: 40Gi
low: "0.1"
volumeType: emptyDir

Use temporary storage space as cache disk for Worker Pod

By default, the Worker Pod uses the path defined in tieredstore as the cache directory (that is, the cache-dir option). For example, in the above example, the cache-dir of the Worker Pod is /dev/shm. In order to allow the Worker Pod to have a larger cache storage, you can use ECI temporary storage space as the Worker Pod's cache disk.

The ECI provides 30GiB of temporary storage space by default. In order to expand the capacity of the temporary storage space, it can be achieved by adding annotations to the Worker Pod:

apiVersion: data.fluid.io/v1alpha1
kind: JuiceFSRuntime
metadata:
name: jfsdemo
spec:
# The number of nodes in the cache cluster
replicas: 5
podMetadata:
labels:
# Run cache cluster within ECI
alibabacloud.com/eci: "true"
# Worker Pod config
worker:
podMetadata:
annotations:
# Set the capacity of ECI temporary storage space
k8s.aliyun.com/eci-extra-ephemeral-storage: "200Gi"
# Mount options
options:
# The cache disk path
"cache-dir": "/var/jfsCache"
# Cache disk size, in MiB. Needs to match the capacity of temporary storage space set above.
"cache-size": "204800"
# The minimum remaining space ratio of the cache disk
"free-space-ratio": "0.1"
tieredstore:
levels:
- mediumtype: MEM
path: /dev/shm
quota: 40Gi
low: "0.1"
volumeType: emptyDir

Create Dataset

In "Custom Resources", create a Dataset using raw YAML. Fill in the object storage bucket in the options, and reference previously created Secret in encryptOptions:

The Dataset example is as follows:

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: jfsdemo
spec:
mounts:
- name: <vol-name> # Get volume name in JuiceFS console
mountPoint: "juicefs:///"
options:
bucket: <bucket>
encryptOptions:
- name: token
valueFrom:
secretKeyRef:
name: jfs-secret
key: token
- name: access-key
valueFrom:
secretKeyRef:
name: jfs-secret
key: access-key
- name: secret-key
valueFrom:
secretKeyRef:
name: jfs-secret
key: secret-key

Navigate to the "Pods" page, verify that the worker pods have been created, these five worker pods form a dedicated cache cluster, providing distributed cache for JuiceFS clients:

Fluid-worker

Cache acceleration

11G of test data (110k files of 1Mi size) have been prepared in the JuiceFS Volume in advance. After the cache cluster is created, warm up the cache by creating a DataLoad, referencing the Dataset created previously:

apiVersion: data.fluid.io/v1alpha1
kind: DataLoad
metadata:
name: jfs-load
spec:
dataset:
name: jfsdemo
namespace: default

Head to the "Task" page, verify that the jfs-load-loader-job has completed, which means data has been warmed up.

Create application

Create application pod and use the PVC with the same name as the above Dataset. We'll create a job to run a simple file copy test:

Fluid-app

Set the following two labels for the pod:

    alibabacloud.com/fluid-sidecar-target: eci
alibabacloud.com/eci: "true"

Verify that there is a fluid-fuse container injected into the pod, and it runs on the virtual node:

Fluid-pod

Now, the pod is running on the ECI node. Pod log will show the data copy result:

Fluid-result-with-cache

Cache acceleration benefits

Running the same job, this time without cache acceleration:

Fluid-result-without-cache

It can be seen that without distributed cache, the time to copy the same data is 13m19.096s. Compared with the distributed cache acceleration above, the copy time is shortened to 1m17.445s, and the speed is increased by 13 times.