IORPC Protocol for Distributed Cache
Starting from JuiceFS Enterprise Edition 5.2, IORPC provides an RPC framework for data transfer between cache group peers. TCP Pool is the default communication mode and is suitable for most scenarios. IORPC should only be used when excessive connections cause performance degradation — IORPC's connection multiplexing can reduce CPU overhead from goroutine scheduling under such conditions. IORPC is disabled by default; contact Juicedata technical support to evaluate feasibility before enabling it via --use-iorpc.
IORPC vs. TCP Pool
The distributed cache system in JuiceFS supports two network communication modes for peer-to-peer data transfer:
| Feature | IORPC | TCP Pool |
|---|---|---|
| Multiplexing | One connection handles multiple concurrent requests | Each connection handles one request at a time |
| I/O path (default) | Direct I/O read + write to socket (two user-space copies per request) | sendfile (zero-copy via kernel) |
| Connection count | Requires fewer connections under high concurrency | Requires maintaining a large number of connections under high concurrency |
| Transport | TCP (default) or RDMA | TCP only |
| Cache read modes | dio (default), pagecache, splice | sendfile (default) or dio (with --cache-try-dio) |
| Use case | When excessive connections cause performance degradation | Most scenarios |
TCP Pool is the legacy mode available since earlier versions and remains the default mode in all versions. It uses sendfile for zero-copy data transfer from cache disk to network by default.
IORPC supports multiplexing: a single connection can process multiple requests concurrently over TCP, significantly reducing the number of connections needed in high-concurrency scenarios. In addition, IORPC can work with RDMA to achieve higher throughput and lower CPU utilization.
Enabling IORPC
IORPC has two independent components: the server and the client. Both must be enabled for IORPC communication to take effect.
Server side
The IORPC server runs on cache group member nodes (that is, nodes mounted without --no-sharing). It is enabled by default starting from 5.2.
To disable the IORPC server on a node, use --iorpc-mode=disable.
Client side
The IORPC client is disabled by default. To enable it on any JuiceFS client (including --no-sharing nodes), use --use-iorpc.
When you set --rdma-network, IORPC is automatically enabled on both the server and client sides, using RDMA transport. You do not need to set --use-iorpc separately. See Distributed Cache RDMA Acceleration for details.
IORPC cache access modes
The --iorpc-mode option controls how the IORPC server reads cache data from disk. This setting applies to cache group member nodes only.
| Mode | Description |
|---|---|
dio (default) | Direct I/O. Bypasses the kernel page cache. Reduces memory copies and system memory usage. |
pagecache | Buffered read through the kernel page cache. Uses the standard file system read path. |
splice | Zero-copy mode. Uses a kernel pipe (splice/vmsplice) to transfer data directly between disk and network without user-space copying. Requires system pipe max size ≥ 8 MiB. Automatically falls back to dio when RDMA is enabled. |
disable | Disables the IORPC server on this node. Cache requests will be served via TCP Pool instead. |
Configuring splice mode
To use the splice mode, set --iorpc-mode=splice.
Before enabling splice, increase the system's maximum pipe size to at least 8 MiB:
echo 8388608 > /proc/sys/fs/pipe-max-size
Check the current pipe max size with:
sysctl fs.pipe-max-size
If the pipe size is less than 8 MiB or RDMA is enabled, the server automatically falls back to dio mode and logs a warning.
Connection management
IORPC automatically adjusts the number of connections based on bandwidth and QPS, evaluated every second for adding connections and every minute for removing connections. TCP connections are added when bandwidth exceeds 300 MiB/s or QPS exceeds 1,000, and removed when both bandwidth and QPS fall below 100 MiB/s and 500 respectively. RDMA connections use higher thresholds for both scaling up and down (the increase threshold is 1 GiB/s or 5,000 QPS, and the decrease threshold is 512 MiB/s and 2,000 QPS). At least one connection is always kept.
Verifying IORPC
On the server side
Check the JuiceFS client log on cache group member nodes. The log line indicates the current IORPC mode and TCP pool mode:
max pipe size: 8388608, iorpc: dio, tcp pool: sendfile
The iorpc field may show:
dio: Direct I/O (default)pagecache: Buffered readsplice: Zero-copy modedisable: IORPC server disabled
The tcp pool field may show:
sendfile: Default zero-copy modedio: Direct I/O enabled (via--cache-try-dio)
On the client side
When IORPC is successfully established between a client and a cache server, the client log shows:
iorpc client started, server address X.X.X.X:X
Where X.X.X.X:X is the IP address and port of the connected cache server.
Monitoring
Key log messages
| Log | Source | Description |
|---|---|---|
max pipe size: ..., iorpc: dio, tcp pool: sendfile | Server startup | Displays the current IORPC mode and TCP Pool disk read method. See Verifying on the server for field values. |
iorpc client started, server X.X.X.X:X | Client | IORPC connection successfully established with the cache server. |
get iorpc address from ... failed: rpc disabled | Client | The peer does not support IORPC (older version), so it falls back to TCP Pool. |
splice iorpc is disabled since max pipe size is ... or RDMA is enabled, use dio instead | Server | The conditions for splice are not met, so it automatically falls back to dio. |
iorpc-mode should be one of ..., but got ..., set to dio | Server | An invalid --iorpc-mode value is specified, so it falls back to the default dio. |
Prometheus metrics
The following are core IORPC metrics with the mount_ prefix. Each latency metric provides _tp50, _tp90, _tp99 quantiles and _max.
| Metric | Description |
|---|---|
mount_iorpc_processRespDur | Server-side request processing latency (from receiving the request to serializing the response). |
mount_iorpc_writeBodyDur | Latency of writing the response body to the network. Monitor this metric when debugging network congestion. |
mount_iorpc_decodeBodyDur | Latency of decoding request/response bodies. |
mount_iorpc_inflight | Current number of concurrent IORPC requests being processed by the server. When reaching the default concurrency limit of 2,048, new requests will queue and wait until a slot is freed up, which may cause increased latency and client-side timeouts. Consider scaling out the cache group when this metric remains near 2,048. |
juicefs stats real-time output
Use juicefs stats with the r schema character and -l 1 to view IORPC internal latencies in real time:
juicefs stats --schema=ufmcro -l 1 /jfs
In the remotecache section, three additional columns appear:
db_c/lat: Number of decode operations for request / response bodies and average latency.pr_c/lat: Number of RPC request processing operations on the server and average latency.wb_c/lat: Number of response body write operations to the network and average latency.
Without -l 1, the r schema only shows application-level remotecache metrics. These IORPC columns always appear with -l 1 but only contain actual values when IORPC is enabled.
Tuning recommendations
When to use IORPC
Contact Juicedata technical support to evaluate feasibility before enabling IORPC.
When excessive TCP connections cause performance degradation, IORPC may help. Typical indicators include:
- The cache server process has thousands of goroutines (check with
cat /jfs/.stats | grep goroutines), but the remotecache throughput shown injuicefs statsdoes not scale with increased concurrency. This indicates that the one-goroutine-per-connection model of TCP Pool consumes CPU resources without improving throughput. juicefs statsshows high CPU usage, while remotecache throughput is far below the NIC's rated bandwidth. This indicates that CPU resources are consumed by goroutine scheduling rather than data transfer.- A large number of TCP connections exist between clients and cache group nodes (check with
ss -tan | wc -l,ESTABLISHEDandTIME_WAITconnections. A highTIME_WAITcount directly indicates frequent connection creation and closure, a typical characteristic of TCP Pool). Combined with the CPU or throughput issues above, this can further confirm that the bottleneck comes from the TCP Pool connection model.
IORPC addresses this by multiplexing: a single connection can carry multiple concurrent requests simultaneously by matching message IDs. One goroutine can serve multiple requests, reducing the number of connections from thousands or tens of thousands to dozens. The number of goroutines is significantly reduced, and the overhead caused by frequent goroutine context switching in the Go scheduler is greatly minimized. CPU resources are released from connection management and returned to cache disk I/O and network data transfer.
If IORPC alone does not fully utilize available bandwidth, try the splice mode.
When to use Direct I/O (dio)
The default dio mode works well for most scenarios. It is especially useful when top shows high CPU usage by the kswapd process. This indicates that memory fragmentation is slowing down page cache allocation. In this case, direct I/O bypasses the kernel page cache to avoid the issue. However, direct I/O performance is limited by disk performance.
When to use splice
If dio mode does not provide sufficient throughput, switch to splice mode for zero-copy transfer via kernel pipes. This mode is especially beneficial when you want to maximize throughput while minimizing CPU overhead, particularly on older kernels (below 5.12) that lack TCP stack improvements.
Before enabling splice, make sure to increase the maximum pipe size to at least 8 MiB.
IORPC and RDMA
When using RDMA acceleration via --rdma-network, the --iorpc-mode option still controls how the server accesses cache data. However, splice mode automatically falls back to dio when RDMA is active. For details on how RDMA automatically enables IORPC, see the previous section. For complete configuration instructions, see Distributed Cache RDMA Acceleration.
