Skip to main content

Metadata Engines Benchmark

Conclusion first:

  • For pure metadata operations, MySQL costs about 2 ~ 4x times of Redis; TiKV has similar performance to MySQL, and in most cases it costs a bit less
  • For small I/O (~100 KiB) workloads, total time costs with MySQL are about 1 ~ 3x of those with Redis; TiKV performs similarly to MySQL
  • For large I/O (~4 MiB) workloads, total time costs with different metadata engines show no obvious difference (object storage becomes the bottleneck)

Note:

  1. By changing appendfsync from always to everysec, Redis gains performance boost but loses a bit of data reliability; more information can be found here
  2. Both Redis and MySQL store only one replica locally, while TiKV stores three replicas in three different hosts using Raft protocol

Details are provided below. Please note all the tests are run with the same object storage (to save data), client and metadata hosts; only metadata engines differ.

Environment

JuiceFS Version

juicefs version 0.16-dev (2021-07-20 9efa870)

Object Storage

Amazon S3

Client Hosts

  • Amazon c5.xlarge: 4 vCPUs, 8 GiB Memory, Up to 10 Gigabit Network
  • Ubuntu 18.04.4 LTS

Meta Hosts

  • Amazon c5d.xlarge: 4 vCPUs, 8 GiB Memory, Up to 10 Gigabit Network, 100 GB SSD (local storage for metadata engines)
  • Ubuntu 18.04.4 LTS
  • SSD is formated as ext4 and mounted on /data

Meta Engines

Redis

  • Version: 6.2.3
  • Configuration:
    • appendonly: yes
    • appendfsync: always or everysec
    • dir: /data/redis

MySQL

  • Version: 8.0.25
  • /var/lib/mysql is bind mounted on /data/mysql

TiKV

  • Version: 5.1.0
  • Configuration:
    • deploy_dir: /data/tikv-deploy
    • data_dir: /data/tikv-data

Tools

All the following tests are run for each metadata engine.

Golang Benchmark

Simple benchmarks within the source code: pkg/meta/benchmarks_test.go.

JuiceFS Bench

JuiceFS provides a basic benchmark command:

$ ./juicefs bench /mnt/jfs

mdtest

  • Version: mdtest-3.4.0+dev

Run parallel tests on 3 client nodes:

$ cat myhost
client1 slots=4
client2 slots=4
client3 slots=4

Test commands:

# metadata only
$ mpirun --use-hwthread-cpus --allow-run-as-root -np 12 --hostfile myhost --map-by slot /root/mdtest -b 3 -z 1 -I 100 -u -d /mnt/jfs

# 12000 * 100KiB files
$ mpirun --use-hwthread-cpus --allow-run-as-root -np 12 --hostfile myhost --map-by slot /root/mdtest -F -w 102400 -I 1000 -z 0 -u -d /mnt/jfs

fio

  • Version: fio-3.1
fio --name=big-write --directory=/mnt/jfs --rw=write --refill_buffers --bs=4M --size=4G --numjobs=4 --end_fsync=1 --group_reporting

Results

Golang Benchmark

  • Shows time cost (us/op), smaller is better
  • Number in parentheses is the multiple of Redis-Always cost (always and everysec are candidates for Redis configuration appendfsync)
  • Because of metadata cache, the results of Read are all less than 1us, which are not comparable for now
Redis-AlwaysRedis-EverysecMySQLTiKV
mkdir986700 (0.7)2274 (2.3)1961 (2.0)
mvdir1116940 (0.8)3690 (3.3)2145 (1.9)
rmdir981817 (0.8)2980 (3.0)2300 (2.3)
readdir_10376378 (1.0)1365 (3.6)965 (2.6)
readdir_1k18041819 (1.0)15449 (8.6)6776 (3.8)
mknod968665 (0.7)2325 (2.4)1997 (2.1)
create957703 (0.7)2291 (2.4)1971 (2.1)
rename10821040 (1.0)3701 (3.4)2162 (2.0)
unlink842710 (0.8)3293 (3.9)2217 (2.6)
lookup118127 (1.1)409 (3.5)571 (4.8)
getattr108120 (1.1)358 (3.3)285 (2.6)
setattr568490 (0.9)1239 (2.2)1720 (3.0)
access109116 (1.1)354 (3.2)283 (2.6)
setxattr237113 (0.5)1197 (5.1)1508 (6.4)
getxattr110108 (1.0)326 (3.0)279 (2.5)
removexattr244116 (0.5)847 (3.5)1856 (7.6)
listxattr_1111106 (1.0)336 (3.0)286 (2.6)
listxattr_10112111 (1.0)376 (3.4)303 (2.7)
link715574 (0.8)2610 (3.7)1949 (2.7)
symlink952702 (0.7)2583 (2.7)1960 (2.1)
newchunk235113 (0.5)1 (0.0)1 (0.0)
write816564 (0.7)2788 (3.4)2138 (2.6)
read_100 (0.0)0 (0.0)0 (0.0)
read_1000 (0.0)0 (0.0)0 (0.0)

JuiceFS Bench

Redis-AlwaysRedis-EverysecMySQLTiKV
Write big312.81 MiB/s303.45 MiB/s310.26 MiB/s310.90 MiB/s
Read big348.06 MiB/s525.78 MiB/s493.45 MiB/s477.78 MiB/s
Write small26.0 files/s27.5 files/s22.7 files/s24.2 files/s
Read small1431.6 files/s1113.4 files/s608.0 files/s415.7 files/s
Stat file6713.7 files/s6885.8 files/s2144.9 files/s1164.5 files/s
FUSE operation0.45 ms0.32 ms0.41 ms0.40 ms
Update meta1.04 ms0.79 ms3.36 ms1.74 ms

mdtest

  • Shows rate (ops/sec), bigger is better
Redis-AlwaysRedis-EverysecMySQLTiKV
EMPTY FILES
Directory creation4149.6459261.1901603.2982023.177
Directory stat172665.701243307.52715678.64315029.717
Directory removal4687.0279575.7061420.1241772.861
File creation4257.3678994.2321632.2252119.616
File stat158793.214287425.36815598.03114466.477
File read38872.11647938.79214004.08317149.941
File removal3831.42110538.675983.3381497.474
Tree creation100.403108.65744.15415.615
Tree removal127.257143.62551.80421.005
SMALL FILES
File creation317.999317.925272.272280.493
File stat54063.61757798.96313882.94010984.141
File read56891.01057548.88916038.7167155.426
File removal3638.8098490.490837.5101184.253
Tree creation54.523119.31723.3365.233
Tree removal73.36682.19522.7834.918

fio

Redis-AlwaysRedis-EverysecMySQLTiKV
Write bandwidth350 MiB/s360 MiB/s360 MiB/s358 MiB/s