Metrics
JuiceFS provided a Prometheus API for each file system, this page only contains some of the mostly used metrics. For the full metrics list, visit JuiceFS console, navigate to the "Monitor" page, and click "Prometheus API".
Some of the metrics come with multiple subnodes, and are omitted in this section. e.g. juicefs_fuse_ops_lookup、juicefs_fuse_ops_open are all included in juicefs_fuse_ops_<name>.
On the other hand, metrics introduced in this chapter all start with the juicefs_ prefix, this is different in on-prem Grafana data source, for example:
juicefs_trash_size is called jfs_stat_trash_sizejuicefs_operationDuration is called mount_operationDuration
So when you encounter trouble searching for the desired metrics, strip the prefix or suffix and try again.
File system
Labels
| Name | Description |
|---|
name | file system name |
Metrics
| Name | Description | Unit |
|---|
juicefs_size | size of files | byte |
juicefs_inodes | number of inodes | |
juicefs_trash_size | size of files in Trash | byte |
juicefs_trash_files | number of files in Trash | |
Client
Labels
| Name | Description |
|---|
name | file system name |
host | client hostname |
ip | client IP address |
mountpoint | path of mount point |
Metrics
Operating system
| Name | Description | Unit |
|---|
juicefs_uptime | uptime | second |
juicefs_cpuusage | overall CPU usage time | microsecond |
juicefs_memusage | current RSS memory usage | byte |
juicefs_heapSys | Total memory allocated to the client process, same as Sys | byte |
juicefs_heapInuse | Same as HeapInuse | byte |
juicefs_handles | Number of file handlers (descriptors) | |
juicefs_threads | Number of threads in the client process, see ThreadCreateProfile | |
juicefs_goroutines | Number of goroutines in the client process | |
juicefs_gcPause | Same as PauseTotalNs | nanosecond |
| Name | Description | Unit |
|---|
juicefs_metaDuration | Duration of Metadata requests | microsecond |
juicefs_metaRequest | Number of Metadata requests | |
juicefs_meta | Sum of all meta metrics, not very useful | |
juicefs_meta_operations | Number of metadata operations | |
juicefs_meta_bytes_sent | Metadata requests sent traffic | byte |
juicefs_meta_bytes_received | Metadata requests received traffic | byte |
juicefs_meta_packets_sent | Metadata requests sent packets | |
juicefs_meta_packets_received | Metadata requests received packets | |
juicefs_meta_reconnects | Number of metadata service reconnects | |
juicefs_meta_usec_ping | Ping latency between client and metadata service | microsecond |
juicefs_meta_dircache | Number of client memory metadata cache operations | |
juicefs_meta_dircache_<name> | Number of different types of client memory metadata cache operations | |
juicefs_meta_dircache0_dirs | Number of directories in the client memory metadata cache | |
juicefs_meta_dircache0_inodes | Number of inodes in the client memory metadata cache | |
File access
| Name | Description | Unit |
|---|
juicefs_read_bytes | Total read bytes, different from juicefs_get_bytes, this is counted in the file system level | byte |
juicefs_write_bytes | Total write bytes, different from juicefs_put_bytes, this is counted in the file system level | byte |
juicefs_operations | Number of file operations | |
juicefs_operationDuration | File operation latency | |
juicefs_fuse_ops | Number of file operations, roughly the same as juicefs_operations, but counted by summing up all different operation types | |
juicefs_fuse_ops_<name> | Number of single file operations, e.g. getattr、lookup、open | |
juicefs_openfiles | Number of opened files | |
juicefs_bgjobs | Number of currently running background jobs | |
juicefs_bgjobs_compact | Number of currently running compaction in background jobs | |
juicefs_bgjobs_delete | Number of currently running Trash deletion in background jobs | |
juicefs_compacts | Number of successful compactions | |
juicefs_compact_bytes | Compaction traffic | byte |
Buffer
| Name | Description | Unit |
|---|
juicefs_totalBufferUsed | Used buffer size | byte |
juicefs_readBufferUsed | Used read buffer size | byte |
Local cache
| Name | Description | Unit |
|---|
juicefs_blockcache_blocks | Number of cache blocks | |
juicefs_blockcache_bytes | Size of cache blocks | byte |
juicefs_blockcache_hits | Number of cache block hits | |
juicefs_blockcache_hitBytes | Bytes of cache block hits | byte |
juicefs_blockcache_miss | Number of missed cache block hits | |
juicefs_blockcache_missBytes | Size of missed cache block hits | byte |
juicefs_blockcache_evict | Number of cache block evictions | |
juicefs_blockcache_evictBytes | Size of cache block evictions | byte |
juicefs_blockcache_evictDur | Duration of cache block evictions | microsecond |
juicefs_blockcache_readDuration | Duration of cache block reads | microsecond |
juicefs_blockcache_write | Number of cache block writes | |
juicefs_blockcache_writeBytes | Size of cache block writes | byte |
juicefs_blockcache_writeDuration | Duration of cache block writes | microsecond |
juicefs_symlink_cache | Sum of symbolic link cache related metrics, not very useful | |
juicefs_symlink_cache_<name> | Metrics of different types of symbolic link cache operations | |
Distributed cache
In the cache group, there will be corresponding metrics for "sender" and "receiver" for reading data and writing data, also called "server" and "client" in later descriptions, note that the phrase "server" in this context refers to the party serving the cache blocks in the cache group, and the phrase "client" stands for the party receiving the cache blocks. In a cache group, servers and clients can actually be the same set of JuiceFS clients, but if the cache group is deployed as a dedicated cache cluster, then in this architecture, clients do not share their cache, hence servers and clients are two separate sets of JuiceFS clients.
For example, when reading files through a cache group, member nodes will not only receive data from other members, but also share and send their own cache data to other members.
There are two sets of mutually contrasting metrics in distributed cache metrics, corresponding to different application scenarios:
remotecache_get and remotecache_send: correspond to the scenario where the client reads data from the distributed cacheremotecache_receive and remotecache_put: correspond to the scenario where the client writes data to the distributed cache (the --fill-group-cache option needs to be enabled), or the scenario of balancing data in the distributed cache.
Therefore, under normal circumstances, if you sum up metrics of all cache group members (including --no-sharing clients), the size of remotecache_sendBytes should be roughly equal to remotecache_getBytes. If it is found that remotecache_sendBytes is much larger that remotecache_getBytes, it means that there are errors during data transfer, check remotecache_errors metric and client logs for causes.
| Name | Description | Unit |
|---|
juicefs_remotecache_errors | Number of all failed distributed cache requests | |
juicefs_remotecache_get | Number of requests for cache group client to read data | |
juicefs_remotecache_getBytes | Size of cache group client read request | byte |
juicefs_remotecache_getDuration | Latency of cache group client read request | microsecond |
juicefs_remotecache_send | Number of requests for data sent by the cache group server | |
juicefs_remotecache_sendBytes | Size of the data sent by the cache group server | byte |
juicefs_remotecache_sendDuration | Latency of the cache group server send requests, including the latency in reading the local cache (juicefs_blockcache_readDuration). If there is no hit in the local cache, it also includes the request time of the object storage (juicefs_objectDuration_get). If the object storage response is slow or there is a speed limit, this metric will increase. | microsecond |
juicefs_remotecache_put | Number of requests for cache group client to write data | |
juicefs_remotecache_putBytes | Size of cache group client write request | byte |
juicefs_remotecache_putDuration | Latency of cache group client write request | microsecond |
juicefs_remotecache_receive | Number of received write requests for the cache group server | |
juicefs_remotecache_receiveBytes | Size of the data received by the cache group server | byte |
juicefs_remotecache_recvDuration | Latency of the write requests received for cache group server | microsecond |
Object storage
| Name | Description | Unit |
|---|
juicefs_object | Number of total object storage requests | |
juicefs_object_get | Number of GetObject requests | |
juicefs_object_put | Number of PutObject requests | |
juicefs_object_delete | Number of DeleteObject requests | |
juicefs_object_copy | Number of CopyObject requests | |
juicefs_object_head | Number of HeadObject requests | |
juicefs_object_list | Number of ListObjects requests | |
juicefs_objectDuration | Duration of all object storage requests | microsecond |
juicefs_objectDuration_get | Duration of GetObject requests | microsecond |
juicefs_objectDuration_getDelay | GetObject requests wait, wait is caused by client token speed control or congestion caused by low --max-downloads | microsecond |
juicefs_objectDuration_put | Duration of PutObject requests | microsecond |
juicefs_objectDuration_putDelay | PutObject requests wait, wait is caused by client token speed control or congestion caused by low --max-uploads | microsecond |
juicefs_objectDuration_delete | Duration of DeleteObject requests | microsecond |
juicefs_object_error | Number of object storage requests failures | |
juicefs_get_bytes | Size of GET requests | byte |
juicefs_put_bytes | Size of PUT requests | byte |
Data synchronization
| Name | Description | Unit |
|---|
juicefs_sync_scanned | Number of all objects scanned from the source | |
juicefs_sync_handled | Number of objects from the source that have been processed | |
juicefs_sync_pending | Number of objects waiting to be synchronized | |
juicefs_sync_copied | Number of objects that have been synchronized | |
juicefs_sync_copied_bytes | Total size of data that has been synchronized | byte |
juicefs_sync_skipped | Number of objects that skipped during synchronization | |
juicefs_sync_failed | Number of objects that failed during synchronization | |
juicefs_sync_deleted | Number of objects that deleted during synchronization | |
juicefs_sync_checked | Number of objects that have been verified by checksum during synchronization | |
juicefs_sync_checked_bytes | Total size of data that has been verified by checksum during synchronization | byte |
