JuiceFS Benchmark Results

I/O benchmark results

Sequential read

Test configuration:

• Specifications: 48 CPU cores | 192 GiB memory | 36 Gbps network
• Software: Ubuntu 22.04, fio 3.7, and JuiceFS 5.2

fio --name=seqread_bigfile     --filename=/jfs/bigfile-10G     --rw=read     --bs=1M     --size=10G   --iodepth=1     --numjobs={thread_num}       --group_reporting  --direct=1 --time_based --runtime=120

Test results:

• A single JuiceFS thread can achieve 2.4 GiB/s of throughput.
• Throughput scales nearly linearly as the number of threads increases, peaking at 25.1 GiB/s with 20 threads.
• CPU usage is approximately 1–1.5 cores per GiB/s. At 25 GiB/s, total CPU usage ranges from 30 to 40 cores.

Note: Local memory is used for caching to prevent disk throughput from limiting peak performance.

Random read

Test configuration:

• JuiceFS test specifications: 32 CPU cores | 64 GiB memory | 40 Gbps network
• Software: Ubuntu 22.04, fio 3.7, and JuiceFS 5.4
• GPFS comparison test specifications: 96 CPU cores | 384 GiB memory | 2 × 400 Gbps RoCE network, executed on an I/O node with all-flash storage.

Single process

fio --name=jfs-randread --filename=/{mnt}/readfile --rw=randread --bs=4k --size=10g --time_based --runtime=20s --time_based --ramp_time=2s --verify=0  --group_reporting=1  --ioengine=libaio  --direct=1 --numjobs=1 --iodepth={x}

Test results:

• With a single process, JuiceFS peaks at 68K IOPS with iodepth=8. Performance begins to decline at iodepth=16 and gradually stabilizes at iodepth=64.
• GPFS performs best at iodepth=16, reaching 83K IOPS.

Note: In this test, the underlying storage for tmpfs is a cloud disk. Performance would be better with local NVMe storage. The tmpfs result is included only as a local file system reference for comparison with random reads on distributed and parallel file systems.

Multiple processes with iodepth fixed at 8

fio --name=jfs-randread --filename=/{mnt}/readfile --rw=randread --bs=4k --size=10g --time_based --runtime=20s --time_based --ramp_time=2s --verify=0  --group_reporting=1  --ioengine=libaio  --direct=1 --iodepth=8 --numjobs={x}

Test results:

• JuiceFS performs well when multiple processes randomly read a single file. It reaches a bottleneck at 12 concurrent processes, peaking at 258K IOPS, while latency remains approximately 0.4 ms.
• GPFS incurs significant lock contention when multiple processes read the same file, and its performance begins to decline noticeably with more than two processes. An additional test was therefore run in which each fio process read a different file, avoiding lock contention and metadata token contention. JuiceFS performs consistently in both test scenarios, so no adjustment was necessary.
• tmpfs appears to reach the cloud disk throughput limit of 200 MB/s and remains at approximately 50K IOPS.

Sequential write

Test configuration:

• Specifications: 48 CPU cores | 192 GiB memory | 36 Gbps network × 5 nodes
• Software: CentOS 7.9, fio 3.7, and JuiceFS 5.2
• Maximum object storage write bandwidth: 200 Gbps

Test results:

• JuiceFS writes directly to object storage, so write throughput depends on the bandwidth provided by the object storage service. With sufficient concurrency, JuiceFS can easily saturate the network bandwidth.
• Aggregate throughput scales linearly across multiple nodes, reaching 21.4 GiB/s of aggregate write throughput across five nodes.

Random write

Test configuration:

• Specifications: 64 CPU cores | 128 GiB memory | 40 Gbps network
• Software: Ubuntu 22.04 and JuiceFS 5.2
• GPFS test specifications: 96 CPU cores | 384 GiB memory | 2 × 400 Gbps RoCE network, executed on an I/O node with all-flash storage.

To avoid lock contention caused by multiple processes randomly writing to the same file, each fio process writes to a separate file.

fio --name=randwrite    --directory=/{mnt}    --filename_format=file.\$jobnum    --size=512M    --bs=4k    --rw=randwrite     --iodepth=1   --ioengine=libaio      --direct=1    --time_based    --runtime=20    --group_reporting --numjobs={x}

Test results:

• JuiceFS achieves up to 57K IOPS for 4 KiB random writes.
• JuiceFS is mounted with writeback, and cache-dir is set to a path on a local NVMe drive.
• GPFS performance is low with a single process, but scales linearly and catches up with and surpasses JuiceFS at 12 concurrent processes.
• After random writes, JuiceFS requires additional time and bandwidth to compact fragments in object storage.

Metadata benchmark results

JuiceFS Enterprise Edition uses a proprietary metadata engine. Its lock-free, in-memory implementation is based on Raft and uses multiple zones to deliver extremely low request latency, linear capacity scalability, and near-linear performance scalability.

Test configuration:

• Service node specifications: 30 CPU cores | 120 GiB memory | 25 Gbps network
• Software: Ubuntu 22.04 and JuiceFS 5.3 Metadata Service

Test results:

Operation	QPS (k/s)
getattr	90
lookup	18.4
setattr	29
mkdir	60
readdir	3.7
create	54
write	48
read	182
rename	40
hardlink	37.5

Note:

• By default, JuiceFS Enterprise Edition uses a single Raft zone to provide metadata services. The preceding results show the metadata performance of a single zone.
• Multiple zones can be deployed across multiple metadata nodes, providing near-linear performance scalability for all operations except readdir and lookup.