Quota Design in Distributed Architectures: Implementation and Use Cases in JuiceFS

In distributed storage environments, storage resources are typically shared across multiple users, projects, and applications. Without effective constraint mechanisms, abnormal writes or erroneous operations from a single tenant can quickly consume large amounts of space or inodes, impacting system stability and cost control. Quota management provides a way to establish predictable resource boundaries in shared environments.

In distributed systems, quota management is far more than just "setting a limit." The system must balance concurrent writes from multiple clients, asynchronous metadata updates, and overall throughput. At the same time, quota rules must be enforced at different levels of control. To address this, JuiceFS provides multi-level quota capabilities covering the entire file system, directories, and users, supporting scenarios ranging from overall capacity control to individual and team-level constraints.

In this article, we’ll introduce the design and implementation of JuiceFS' quota mechanism, including its core data structures, synchronization model, and the validation and accounting logic in write and delete processes. We’ll also include typical use cases that highlight common issues around quota changes, space reclamation, and over-limit writes.

Quota types and resource dimensions supported by JuiceFS

JuiceFS quotas support two resource dimensions:

• Space: Used storage capacity. Statistics are based on the file system's usage perspective and are aligned to block granularity. The write path section later will explain how incremental usage is estimated under 4 KiB alignment.
• Inodes: Number of used inodes. For workloads with a large number of small files, inodes often become the constraint bottleneck earlier than space. Therefore, inode quotas must also be part of the management strategy.

Based on these two resource dimensions, JuiceFS currently supports four types of quotas.

User quotas and user group quotas are expected to be released in JuiceFS Community Edition 1.4.
In practice, a common and effective strategy combines the following:

• Total file system quota as a safety net.
• Directory quotas to address individual abuse and small‑file storms.
• User/group quotas for multi‑tenant management.

This layered approach controls overall resource limits while preventing abnormal growth of a single entity from affecting other workloads.

Quota implementation mechanism

Synchronization model and data structures

The main challenge of implementing quotas is how to perform checking, accounting, and convergence at an acceptable cost under concurrent writes from multiple clients. JuiceFS clients run on various nodes and continuously issue resource‑changing operations such as creation, writing, truncation, and deletion. If every operation required a strongly consistent server‑side check and update, the write path would incur unacceptable overhead.
Therefore, the quota mechanism must satisfy two goals:

• Performance: Avoid a strongly consistent server‑side update on every write.
• Consistency: Ensure that system usage eventually converges under concurrent writes from multiple clients and prevent over‑limit operations before they happen, as much as possible.

Based on this trade‑off, JuiceFS adopts a synchronization model that works as "local accumulation, periodic flush, and periodic refresh." Clients first accumulate resource deltas in local memory, with background tasks periodically persisting them to the metadata engine in batches. At the same time, each client periodically pulls the latest quota configuration and baseline usage from the server, gradually aligning its own global view. Clients do not communicate directly with each other; instead, the metadata engine serves as the central coordination point.

In other words, JuiceFS quotas do not pursue strong consistency on each operation but achieve eventually consistent resource control through periodic synchronization.

In the current implementation, quota deltas are flushed every 3 seconds (flushQuotas). Clients reload the latest quota configuration and baseline usage from the backend approximately every 12 seconds (via a refresh call triggered by the mount heartbeat). This means that under extreme conditions, the global views seen by different clients may diverge by up to about 12 seconds, but they will gradually converge in subsequent sync cycles.

Quota information is managed uniformly by the quota structure. It represents a single quota entity and can adapt to different types of managed objects such as directories, users, and user groups. Its core design decouples baseline usage from incremental usage:

• UsedSpace/UsedInodes: Represents the baseline usage already persisted in the backend.
• newSpace/newInodes: Represents the locally accumulated deltas on this client that have not yet been flushed to the backend.

type Quota struct {  
    MaxSpace, MaxInodes   int64  // Maximum space and inode limits  
    UsedSpace, UsedInodes int64  // Used space and inodes  
    newSpace, newInodes   int64  // Pending usage deltas to be synced  
}  

For inode accounting, hard links require special attention. Different quota types have different counting semantics for hard links. For directory quotas, counting is based on directory entries: when a hard link is created under a directory, both space and inode usage of that directory increase by 1, and they decrease accordingly when the hard link is removed. For user quotas and user group quotas, counting is deduplicated by the file object (inode). Even if a file has multiple hard links, it’s counted only once per UID/GID dimension. Therefore, creating or deleting hard links does not change the usage for the associated user or user group.

Quota storage

Regarding the quota storage mechanism, the total file system quota (the global "red line") has its capacity and inode limits directly persisted in the metadata engine. Clients load this configuration during mount and enforce hard limits, ensuring the underlying resources are not exceeded.

In contrast, checks and delta accumulation for directory, user, and user group quotas rely more on the client side. Clients maintain in‑memory indexing structures keyed by inode, UID, and GID, and periodically synchronize the corresponding quota information from the backend. This keeps lookup overhead low in high‑frequency I/O scenarios. It’s important to emphasise that the client in‑memory state is only a runtime cache and incremental view; the authoritative source for quota configuration and baseline usage remains the metadata backend.

Quota checks

A synchronization model and data structures alone are not sufficient, and quota logic must also be embedded into the specific resource‑changing paths. A single write operation may not be a simple data append; it can simultaneously involve inode creation, block allocation, directory entry changes, and parent‑directory statistics updates. Under multi‑client concurrency, these changes collectively affect the same set of quota constraints. Therefore, only by placing checks and statistics updates directly into the operation paths (write, create, truncate, and delete) can we avoid out‑of‑limit writes and statistical inaccuracies.

Pre‑write: incremental estimation and multi‑dimensional quota check

When a user initiates an operation that may change resource usage (such as write, create, and truncate), the client first estimates the expected resource delta, including both space and inode changes.

Space delta is estimated based on the actual allocation granularity of underlying data blocks (for example, 4 KiB alignment), therefore block‑aligned calculation is required. Inode deltas primarily occur in creation operations (such as creating a new file or directory).

After obtaining the resource delta for the operation, the client performs a quota check before actually writing. The check covers multiple dimensions: user and user group quotas, total file system quota, and directory quotas for the target directory tree. If any dimension would exceed its limit after this operation, the request is rejected with an error such as quota exceeded or out of space.

By placing the check in the write path before the resource change, the system can block risky operations before they happen, avoiding complex cleanup or rollback afterwards.

Post‑write: local delta accumulation and background batched sync

After a successful write, the resource delta generated by the operation is incorporated into the corresponding usage statistics and gradually aligns with the global state according to the defined convergence mechanism. Specifically, three categories of statistics are affected:

• Global level: The overall file system usage increases (or decreases).
• Directory level: The usage of the relevant directory subtree changes accordingly.
• User / user group level: The usage of the corresponding subject also accumulates.

These updates are first reflected in the client’s local accumulated deltas and are not immediately written back to the backend in a strongly consistent way. Later, background tasks flush them in batches, and periodic refresh operations gradually align them with other clients, achieving global convergence.

Usage statistics (stats): foundation for the quota system

For quotas to work effectively, the system must be able to track current resource usage with low overhead. Whether for large directory trees or many users and user groups, if every check requires a real‑time full scan, the performance cost will be unacceptable. Therefore, an efficient and reliable usage statistics mechanism is a prerequisite for implementing quotas.

Directory statistics

Directory quotas constrain the total space and inode usage of an entire directory subtree, not the size of individual files. Consequently, they rely on directory‑level usage statistics.

It’s important to note that directory statistics (DirStats) and quota statistics have different scopes: DirStats only sums up the usage of immediate children (files and subdirectories) under a given directory – a single‑level statistic. In contrast, directory quotas recursively sum up the entire subtree. This design allows DirStats to be maintained with lower overhead, while directory quotas provide a full subtree view.

The key to implementing such statistics is maintaining low overhead and high availability for large directory trees. JuiceFS follows the same approach as the quota mechanism: high‑frequency local updates and batched background persistence. Clients maintain directory usage deltas in memory; when operations such as writes or deletions occur, the changes are first recorded locally and then periodically synced in batches to the metadata engine by background tasks.

In addition, the system does not load all directory statistics at mount time. For large directory trees, a full load would cause significant latency and memory overhead. Therefore, directory statistics adopt an on‑demand fetch strategy: only when precise usage is required (such as quota checks, usage summarisation, and administrative queries) does the system load the statistics of the corresponding directory from the backend.

When users query usage information via df or an application calls statfs, JuiceFS makes a trade‑off between performance and accuracy:

• It first uses locally cached used space and inodes for fast calculation.
• If the local baseline is incomplete (for example, just after startup) or higher real‑time accuracy is needed, it fetches the latest global counters from the backend for calibration.
• Finally, it adds locally accumulated (not yet synced) deltas to make the result more accurate for the current node’s write state.

After obtaining the used amounts, the client calculates total and avail based on whether a total capacity limit is configured:

• If a limit is configured, total capacity equals that limit, and available capacity is "limit minus used."
• If no limit is configured, it returns a dynamically estimated total capacity so that tools like df can display normally.

Moreover, when querying quotas from the root directory, the system displays the maximum space and inode limits, allowing administrators to see the global resource boundaries.

In addition, JuiceFS will support real‑time updates of directory statistics for the trash in version 1.4. When files are deleted (moved to the trash), restored from the trash, or permanently cleaned up, the system updates the trash directory’s statistics immediately. This enables administrators to accurately track space usage of the trash.

User and user group statistics

User and user group statistics are collected only after the corresponding quota feature is enabled. Before enabling, the updateUserGroupStat call in the kernel path returns directly without generating any statistics. After enabling, clients maintain usage data in an in‑memory map keyed by uid and gid and update the relevant statistics on all paths that may cause usage changes.

A special note: when setting a quota for a user or user group for the first time via juicefs quota set --uid or juicefs quota set --gid, the system immediately performs a full scan of existing files to initialise the baseline usage. After this initialisation, subsequent writes and deletions become incremental updates, and no further full scan is required.

Common scenarios

1. A file has been deleted, why hasn’t the total file system quota decreased? Why hasn’t the object storage billing changed?

This is usually not a statistics error, but a result of file system semantics combined with the statistical model.

For example, after enabling the trash in JuiceFS, a deletion operation does not immediately free space. The file is first moved to the trash for possible recovery. Therefore, files in the trash are still counted in the total file system quota and user / user group quotas, but are no longer counted in the original directory quota.

Another common reason is the time lag between file system statistics and object storage billing. JuiceFS quota statistics use a local accumulation + periodic background sync model, so it’s possible that different clients or different statistical interfaces have not yet converged in a short time. At the same time, object storage may not have completed garbage collection or lifecycle cleanup. Therefore, temporarily seeing inconsistency between file system usage, quota statistics, and object storage billing is generally expected. This is not considered a system anomaly as long as they gradually converge over time.

In addition, note that quota and statfs show the file system perspective of space usage and availability, while object storage billing is based on the underlying object storage model – affected by factors such as chunking, merging, delayed reclamation, and lifecycle rules.

The two are not required to be the same.

2. Quota is full, but appending to an existing file did not report an error immediately.
This is often related to the asynchronous commit path in some JuiceFS writes. From the application’s perspective, the write system call may return success early, while the actual data commit and corresponding quota check happen later. Thus, appending may appear to "succeed," but the data may not be fully persisted; if the later commit stage determines that the quota would be exceeded, the write may still fail.

In other words, a successful write return does not guarantee that the write has been finally committed. In scenarios involving quota limits, a safer approach is to check the return status on close, the final file size, and handle possible errors accordingly.

3. Quota is not yet full, but file creation fails.
This phenomenon is usually related to temporary view divergence under the eventual‑consistent statistical model.

Example: a volume has a total quota of 2,000 inodes, and there are currently 1,999 files. One more file should be creatable. However, in extreme concurrency or unusual refresh timing, the client’s local cache may diverge briefly from the backend baseline count. This may cause the in‑memory used inode count to be temporarily too high, thus rejecting a legitimate creation request.

This type of problem inherently stems from the local accumulation + periodic sync convergence model. It avoids the high overhead of strong‑consistent backend updates on every operation, but in extreme cases the system may have short‑term false positives.

Typically, such false positives disappear with the next sync cycle, and retries can mitigate the issue.

This also illustrates that, in a distributed environment, quotas are best understood as an efficient, near‑real‑time constraint mechanism, not a fully synchronous, strongly consistent judgement for every concurrent operation.

4. After a write exceeds the quota, why does the "failed" file remain in the directory?

This is not unique to JuiceFS; it’s not uncommon in file systems that follow POSIX semantics.

For example, a user sets a 1 GiB quota on a directory and then tries to write a 2 GiB file using dd. The file system first allows the first 1 GiB of valid writes; only when the subsequent write exceeds the quota does it return “Disk quota exceeded.” Consequently, a "partial file" of about 1 GiB is left behind. This does not indicate abnormal behaviour. It simply means the first part of the data was written successfully, while the remainder failed due to the quota.

The file system's responsibility is to report the error, not to decide whether to delete the successfully written data. Whether to clean up such an incomplete file is left to the application. This follows standard POSIX semantics: the file system returns the error, and the application handles subsequent cleanup and recovery.

Summary

In a distributed file system, quotas are not a simple "counter" feature, but a system design that must balance performance, consistency, and management granularity. Through pre‑write checks, local accumulation, and periodic background synchronization, JuiceFS minimizes overhead on the write path while allowing various usage statistics to gradually converge under an eventual consistency model. Based on this mechanism, quota control covers not only total file system capacity, but also multiple levels such as directories, users, and user groups, thereby meeting the needs of typical scenarios including multi‑tenant isolation, individual constraints, and team‑level resource management.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.