JuiceFS Blog

5.5× Faster Small-File Writes: Tuhu Built a Unified AI Storage Platform with JuiceFS + Ceph RADOS

Thu, 30 Jul 2026 09:02:00 +0000

TL;DR:
Tuhu leverages JuiceFS, TiKV, and Ceph RADOS to unify storage for AI training, AI inference, and analytics, supporting over 100 million files while achieving up to 5.5× faster small-file writes and 3.2× faster small-file reads.

Tuhu (9690.HK) is an integrated online and offline automotive service platform. By the end of 2025, we had grown to 162.3 million registered users and operated 8,008 service centers across China. As our business continued to expand, our infrastructure had to support increasingly diverse data management and compute workloads. Over the years, multiple storage systems—including NFS, Alluxio, MinIO, and SeaweedFS—had been deployed independently to meet different application requirements. While each solution addressed a specific use case, the overall storage architecture became fragmented, making data movement expensive and operations increasingly difficult.

When evaluating a unified storage architecture, we built a unified AI storage platform based on JuiceFS + TiKV + Ceph object storage cluster (RADOS), considering our existing private cloud infrastructure and Ceph deployment.

Today, this platform serves AI training, AI inference, and big data processing workloads through a single storage infrastructure. It currently manages over 100 million files. In low-concurrency benchmark tests, it achieved up to a 5.5x improvement in small-file sequential write performance and up to a 3.2x improvement in small-file read performance.

In this article, we’ll explain how we designed and deployed our unified AI storage platform on top of its existing Ceph infrastructure. We’ll also share several production optimization practices, including Ceph RADOS data paths, erasure-coded pools, write amplification reduction for small files, and stability improvements for containerized deployments.

From multiple storage systems to a unified storage foundation

In the early stages of our company's growth, storage infrastructure was built independently for different applications to maximize development speed and satisfy immediate application requirements. As the platform evolved, this approach gradually resulted in multiple storage systems coexisting in production.

The rapid adoption of cloud-native computing, large-scale AI training, and AI inference exposed the limitations of this architecture. Problems related to architectural consistency, data mobility, operational complexity, and storage performance became increasingly apparent.

We identified four major challenges.

High operational complexity: Production environments relied on numerous storage solutions, including NFS, Alluxio, MinIO, Ceph, SeaweedFS, and various cloud block storage services. Each system had its own deployment model, access protocol, operational workflow, and troubleshooting methodology. As storage clusters continued to grow, day-to-day maintenance, capacity planning, and version upgrades became increasingly difficult.
Data silos: Different storage systems exposed different interfaces and lacked a unified data access layer. AI training pipelines typically require POSIX-compliant file semantics, while surrounding tools for data processing, model management, backup, and archiving often depend on object storage APIs. Because these storage systems could not naturally share the same data, datasets frequently had to be copied or synchronized between platforms. This increased operational overhead, storage consumption, and workflow complexity.
Rising storage costs: As business continued to grow, both the number of files and total storage capacity increased rapidly. Meanwhile, infrastructure costs—including hardware procurement, datacenter resources, and storage operations—continued to rise. Improving storage utilization while maintaining performance and reliability became a key objective for the infrastructure team.
Higher performance requirements from AI workloads: AI training, AI inference, disaggregated storage and compute, and big data analytics placed higher demands on the underlying storage system's concurrency, throughput, access latency, and stability. In particular, large-scale small-file access, multi-node concurrent reads, model file loading, and online inference services required the storage system not only to provide stable capacity support but also to deliver predictable performance under high-concurrency conditions.

As a result, the existing architecture based on multiple independent storage systems could no longer support long-term application growth. We decided to consolidate fragmented storage resources into a unified storage foundation that would simplify operations while enabling seamless data sharing across different compute environments.

The new storage platform was designed around two major capability areas:

Block storage for infrastructure and virtualization workloads: This capability provided reliable, high-performance storage for virtual machine images, cloud disks, and traditional infrastructure services.
Unified data access for cloud-native workloads, including Kubernetes applications, traditional microservices, big data analytics, AI training, AI inference, and emerging middleware such as vector databases. Among these, AI training and inference posed more concentrated challenges in terms of file semantics, object interfaces, data throughput, small-file performance, and multi-node concurrent access.

Therefore, building a unified cloud storage foundation involved both block storage capability for virtualized environments and data access capability for AI and cloud-native workloads. The following section will focus on the latter—how to build a unified data access foundation based on JuiceFS to support AI training, AI inference, and big data processing.

Building a unified AI storage platform with JuiceFS

When designing a unified storage platform, we evaluated several architectural approaches based on our existing private cloud infrastructure, storage investments, and workload characteristics.

Our primary goals were to support large-scale data access, enable disaggregated storage and compute, provide multiple access protocols, and deliver high reliability without introducing unnecessary operational complexity.

Rather than building a new distributed storage system from scratch, we chose a modular architecture based on JuiceFS, TiKV, and Ceph RADOS. This approach allowed us to leverage our existing Ceph infrastructure while introducing a unified file system capable of serving AI training, inference, and big data processing workloads through a single storage platform.

A key design principle of this architecture is the separation of metadata and user data.

JuiceFS provides the unified file system layer, while the metadata engine and Ceph RADOS independently handle metadata management and data storage. At the storage layer, Ceph distributes data and manages failure domains using the CRUSH algorithm, providing a decentralized and highly reliable storage backend.

The architecture also adopts a lightweight client model.

JuiceFS clients are responsible for file system semantics, metadata operations, cache management, and request scheduling. Once data is written to Ceph RADOS, data durability—whether implemented through replication or erasure coding—is handled entirely by the storage cluster.

This division of responsibilities keeps compute nodes lightweight, minimizes interference with AI training workloads, and fully utilizes the reliability and scalability already provided by the existing Ceph deployment.

For our private cloud environment, this modular approach significantly reduced engineering and operational overhead compared with developing and maintaining an integrated distributed storage system. At the same time, it provides unified access to AI training, AI inference, and big data workloads through a common storage platform.

Unified AI storage architecture

Our unified AI storage platform consists of four logical layers.

Workload layer

At the top are the application workloads, including virtual machines, Kubernetes-based containerized applications, traditional microservices, AI training, AI inference, and Apache Spark jobs.

Cache layer

The cache layer consists of two components:

DataCache Pool: It accelerates repeated access to training datasets, model files, and other hot data, reducing load on the storage backend.
KVCache Pool: It’s designed for large language model (LLM) inference. It caches context data across multiple storage tiers, reducing latency and improving the responsiveness of online inference services.

image: Tuhu unified cloud storage architecture
caption: Tuhu unified cloud storage architecture

Storage protocol layer

JuiceFS provides a unified data access entry point here. AI training jobs access data via POSIX FUSE or CSI Driver; big data Spark jobs access data through the JuiceFS Hadoop Java SDK; and object storage toolchains access the same data via JuiceFS S3 Gateway. This approach unifies file storage, object storage, and big data access interfaces around a single underlying data set, reducing data movement and redundant storage across systems.

It should be noted that block storage scenarios, such as VM cloud disks, are primarily handled by Ceph RBD, which is part of the overall cloud storage system's block storage capability. The focus of this article is the file, object, and big data access foundation built on JuiceFS. While both share the underlying Ceph storage resources, they differ in access semantics and service targets.

Storage engine layer

JuiceFS uses a metadata-data decoupled architecture. For file systems with hundreds of millions of files, metadata capabilities directly impact path resolution, directory traversal, file attribute queries, small-file access, and concurrent job startup performance. TiKV provides distributed transactions, strong consistency, high availability, and horizontal scalability, offering stable metadata support for large-scale file systems. We built a five-node TiKV cluster for the metadata layer.

For the data layer, we use Ceph RADOS as the underlying data storage engine for JuiceFS. Ceph RADOS reuses existing private cloud storage resources, reducing duplicate infrastructure costs. In addition, by creating an erasure-coded pool for physical data storage, we improve physical disk capacity utilization while maintaining reliability, mitigating cost pressures from data growth.

In this architecture, data communication follows two paths:

The metadata path between clients and TiKV, handling path lookups, directory structures, file attributes, and state information
The data read/write path between clients and Ceph RADOS, handling actual data block access

Through librados, JuiceFS interacts directly with the underlying Ceph RADOS. Compared to traditional gateway-based storage architectures, this reduces multi-layer forwarding overhead, allowing clients to access the storage cluster more directly after metadata interactions and shortening the data access path.

image: JuiceFS architecture at Tuhu
caption: JuiceFS architecture at Tuhu

Production deployment and optimization practices

Critical scenario deployment and storage consolidation

In LLM application scenarios, our company has dedicated self-built compute clusters for model inference services such as MiniMax. Meanwhile, custom models developed by internal application lines have gradually been migrated to JuiceFS after passing production validation. As a result, core LLM inference and training applications have begun to unify onto the new file storage foundation, with the original NFS clusters and Alluxio storage clusters entering phased decommissioning.

In core middleware and backup scenarios, including large-scale ClickHouse deployments, multiple core middleware systems and full production backup tasks have also been migrated to JuiceFS. As the unified file foundation has proven its capacity, stability, and access efficiency, MinIO and SeaweedFS have also entered decommissioning, reducing the operational complexity of maintaining multiple parallel storage systems.

Cache acceleration for AI training and inference

For AI training scenarios, training data typically has a "write once, read many" pattern. We use JuiceFS Community Edition's local data caching capability to cache hot training data on compute nodes. This reduces backend storage pressure and improves multi-node concurrent read efficiency.

For LLM inference scenarios, models serving online requests or performing iterative token generation are sensitive to context data loading latency and throughput. With limited single-node GPU memory and memory capacity, we use a multi-level caching mechanism to cache context data across multiple levels, improving online inference response efficiency.

Benchmarking and small I/O optimization

After deploying the unified storage platform, we focused on evaluating its performance under small-file workloads.

Using juicefs bench and fio, we benchmarked small-file operations and random I/O to compare the new JuiceFS + Ceph RADOS architecture with the previous JuiceFS + MinIO deployment.

To minimize the impact of caching, all local caches were disabled during testing, allowing I/O requests to be served directly by the storage backend. It's worth noting that these benchmarks were intended to validate the architecture rather than measure peak cluster performance. The tests were conducted with only four client threads under low concurrency.

Even under low-concurrency test conditions, the new architecture demonstrated substantial performance improvements.

The two tables below show test results for JuiceFS + MinIO:

Item	Throughput	Latency
Large file write	3,377.65 MiB/s	1.21 s/file
Large file read	4,248.77 MiB/s	0.96 s/file
Small file write	147.2 files/s	27.17 ms/file
Small file read	1,176.8 files/s	3.40 ms/file

Item	IOPS	Throughput (MiB/s)
Random read	57,504	1,483
Random write	1,194	1,521

The two tables below show test results for JuiceFS + Ceph RADOS:

Item	Throughput	Latency
Large file write	4,984.12 MiB/s	0.82 s/file
Large file read	4,817.87 MiB/s	0.85 s/file
Small file write	817.0 files/s	4.90 ms/file
Small file read	3,776.4 files/s	1.06 ms/file

Item	IOPS	Throughput (MiB/s)
Random read	77,984	3,394
Random write	3,836	4,120

Test result highlights:

Small-file write throughput improved by up to 5.5x, with significantly lower latency for individual file writes.
Small-file read throughput improved by up to 3.2x, delivering higher read throughput and lower per-file read latency.
Random I/O performance also improved. Both throughput and IOPS increased, with random-write IOPS improving by more than 3x.

From an architectural perspective, the JuiceFS + MinIO solution requires S3 interface and object storage gateway overhead, while the JuiceFS + Ceph RADOS solution uses librados for more direct interaction with the underlying storage cluster. This reduces gateway forwarding overhead. As a result, Ceph RADOS shows more significant performance advantages in scenarios sensitive to access paths, such as small-file writes and random writes.

However, in random small-file read tests, since local caching was disabled, data still needed to be retrieved from physical disks in real time. Therefore, read performance did not show the same order-of-magnitude improvement as writes, but still achieved a stable ~1.4x improvement overall.

Small-file space amplification and write amplification optimization

In erasure coding (EC) mode, traditional EC algorithms require alignment with stripe units. In a 6+3 configuration, even writing a 10-byte tiny file requires filling 6 data chunks to 4 KB each plus 3 coding chunks, resulting in 36 KB of physical storage—causing severe write amplification and thousands of times space waste.

image: Before:after write amplification optimization
caption: Before/after write amplification optimization

To address this, we introduced Ceph's Fast EC feature to optimize small-object writes.

On one hand, Fast EC alleviates small-file space amplification. When enabled, small objects are no longer forced to fill entire stripes. For a 10-byte file, the logical size remains 10 bytes, while physical space occupies the first stripe; with a 4 KB minimum allocation unit and parity data, physical footprint is reduced to 16 KB.

On the other hand, Fast EC also improves read/write efficiency in small I/O scenarios. Reads no longer require fetching full stripes; writes can use parity delta writes (PDW) to reduce unnecessary network interactions and I/O overhead, improving overall small-file read/write performance.

JuiceFS Mount Pod stability optimization

In containerized environments, AI training applications occasionally encountered bad file descriptor errors, which severely impacted training efficiency. Investigation revealed that these errors were related to Mount Pod out-of-memory (OOM) conditions.

Periodic OOM due to metadata backups: When the cluster reached tens of millions or hundreds of millions of files, JuiceFS' default hourly metadata backups generated large backup files. Since these backup tasks were randomly assigned to clients in the cluster, and the underlying RADOS architecture imposes size limits on individual object writes (for example, 128 MB), backup tasks could fail repeatedly in large-scale file system scenarios, causing Mount Pod memory usage to grow continuously and eventually trigger OOM.

To address this, we disabled automatic backup on Mount Pods, preventing online clients from performing automatic backups. Instead, dedicated scheduled tasks handle metadata backups centrally, avoiding additional backup pressure on application clients.

image: 周期性 OOM 客户端内存占用
caption: Periodic OOM memory usage

Occasional OOM due to page cache growth: During frequent reads of large amounts of local cache, the Linux kernel page cache continued to grow. In such cases, even if the container's actual memory usage was only about 1.1GB, Kubernetes OOM decisions based on working_set_bytes—which includes page cache—could trigger OOM kills when the metric rose to high levels (for example, 33.5 GB), causing occasional client disconnections.

To address this, we increased limits.memory based on historical monitoring data. Then, we physically isolated cache resources, ensuring that different PVCs did not share the same local cache directory. For nodes with limited memory (such as CPU nodes), we capped local cache capacity (data cache size) within a reasonable range (for example, 10 GB to 50 GB) and enabled cache reclamation once the limit was reached. This prevents uncontrolled growth of the kernel page cache inside the container.

image: 偶发性 OOM 客户端内存占用
caption: Occasional OOM memory usage

Through these optimizations, Mount Pod stability improved significantly in large-scale file system and high-frequency cache access scenarios. This provides a solid foundation for onboarding more AI training and inference workloads to the unified file foundation.

Future plans

We plan to focus on three aspects for the continued evolution of the AI cloud storage infrastructure:

Unified management from private cloud to hybrid cloud: As AI training jobs are scheduled across both private and public clouds, the storage foundation needs stronger cross-environment data management capabilities. Building on the existing private Ceph RADOS architecture, we’ll continue exploring JuiceFS' integration with public cloud object storage.

By pushing cross-cloud data synchronization, data distribution, and access path management down to the storage layer as much as possible, we can reduce the application's awareness of differences between cloud environments, providing more consistent mount paths and access experiences. This will improve training job migration efficiency across different compute environments and reduce data management complexity in multi-cloud settings.

Promoting lakehouse integration to reduce cross-system data movement: Previously, data movement between offline HDFS big data clusters and algorithm storage clusters relied on synchronization tools, increasing pipeline complexity, data redundancy, sync latency, and management costs.

We plan to further adopt and refine the JuiceFS Hadoop Java SDK so that Spark jobs in the algorithm layer can directly read and write the underlying unified storage pool. This will allow big data computing, algorithm training, and data archiving to operate around a single storage foundation, reducing cross-system data replication and redundant storage, further breaking down data silos, and advancing storage-compute separation and lakehouse integration.

Building tiered storage to accelerate LLM inference: As LLM inference services scale, a single storage medium can no longer simultaneously meet cost, capacity, and latency requirements. We plan to build capabilities in data tiering and KVCache tiering.
For data tiering: JuiceFS' current data tiering feature is primarily designed for public cloud object storage. For private Ceph RADOS environments, we plan to collaborate with the community to explore mounting multiple Ceph storage pools of different performance tiers (for example, NVMe Pool and HDD Pool) under the same JuiceFS file system, allowing data with different access frequencies to reside on appropriate storage tiers.
For KVCache tiering: As LLM inference evolves toward prefill/decode (PD) separation in distributed architectures, relying solely on single-node GPU memory to hold context data faces capacity and cost bottlenecks. Building on the unified storage foundation, we’ll gradually explore a four-tier KVCache architecture composed of HBM (GPU memory), DRAM, local SSD, and shared storage, providing more flexible context data management and access acceleration for large-scale AI inference clusters.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

Amazon S3 Files: How It Works, Performance Boundaries, and a Comparison with JuiceFS

Thu, 23 Jul 2026 06:35:00 +0000

AWS announced Amazon S3 Files to let you mount an S3 bucket as a high‑performance shared file system on compute nodes without moving any data.

This isn't the industry's first attempt to make S3 accessible as a file system. From the early days of s3fs, to AWS' later release of Mountpoint for Amazon S3, and now S3 Files, the path to "using S3 like a file system" has been many years in the making. The key difference is that earlier solutions mostly operated at the access layer, while S3 Files finally wraps shared access, full file system semantics, and a managed high‑performance layer into a native offering.

This makes S3 Files a new option worth analyzing on its own. For workloads that need file‑style access to existing S3 data, it provides a native, lightweight solution. But in more complex scenarios such as AI model training or big data analytics, its actual performance still needs to be evaluated based on its underlying implementation and runtime behavior.

In this article, we’ll analyze S3 Files’ implementation, performance boundaries, and how it differs from JuiceFS.

S3 Files: an S3 native file system with EFS as the high‑performance layer

S3 Files uses Amazon Elastic File System (EFS) as a managed high‑performance storage layer to handle low‑latency data and metadata. On top of that, it provides full file system semantics for S3, including consistency, file locking, and POSIX permissions.
Think of it this way: AWS adds an EFS-based file system access layer on top of object storage, so that data previously accessible only through object APIs can be used directly by compute nodes as directories, files, and mount points. Changes between the file system and S3 are synchronized automatically by the service in the background.
With this architecture, S3 Files does not migrate the entire dataset. Instead, it only places a portion of the current working set into the high‑performance layer on demand, while the “source of truth” for the data remains in S3.

How S3 Files works: mount, import, and sync

For S3 Files, mounting is only the beginning. What really matters is the data path after mounting: how the scope is determined, what gets imported on first access, which requests go to the high‑performance layer, and how writes are synced back to S3. These mechanisms directly determine the performance boundaries and cost structure discussed later.
The figure below shows S3 Files mount architecture:

image: S3 Files mount architecture
caption: S3 Files mount architecture

Taking an EC2 instance mounting an existing S3 bucket as an example, the real questions are not about the mount command itself, but about what happens after mounting: how data is imported, accessed, and synced. Here are the key technical details.

Scope: import the entire bucket or a specific prefix

In S3, files are organized using prefixes. You can think of them as similar to “subdirectories” or “sub-buckets” within a bucket, but they are not actual directories. S3 Files can mount an entire S3 bucket (i.e., s3://my-bucket/) or restrict the scope to a specific prefix (i.e., only s3://my-bucket/data/ml/). This is especially important for huge S3 buckets containing millions or even hundreds of millions of objects, because a broader scope adds more metadata sync overhead and potentially more operational complexity.
When using S3 Files on a compute node, AWS provides a custom mount client amazon-efs-utils. When mounting, you use the file system ID assigned by AWS for the S3 Files instance, not the bucket name.
You can create a local mount directory and mount it using the dedicated s3files file system type as shown below:

sudo yum -y install amazon-efs-utils
sudo mkdir /mnt/s3files
sudo mount -t s3files file-system-id:/ /mnt/s3files

When mounting, you can specify a subdirectory (or actually a prefix) in the mount path (for example, file-system-id:/data/ml/). However, this only affects the local mount point, as the underlying metadata sync still applies to the entire bucket, and other mount points could inadvertently access other subdirectories. Very often in practice, a better approach is to limit the scope to a specific prefix when creating the S3 Files instance, rather than using the entire bucket and trying to restrict access afterward.

First access: import triggers and size thresholds

S3 Files does not immediately move the entire dataset into the high‑performance layer after mounting. Data import is triggered by access events. The default mode is trigger=ON_DIRECTORY_FIRST_ACCESS: when you first access a directory, the system imports the metadata of files under that directory and asynchronously moves the data of small files that meet the criteria into the EFS high‑performance layer.

If configured as trigger=ON_FILE_ACCESS, the first directory traversal only imports metadata; data enters the high‑performance layer only when a file is actually read for the first time. This saves space and import cost, but first‑read latency is higher.

The most critical control parameter here is sizeLessThan, which takes a value in bytes. By default, only files smaller than 128 KiB (i.e., sizeLessThan=131072) are moved into the high‑performance layer during import. Larger files only have their metadata imported; their content is still retrieved mainly from S3. In other words, S3 Files optimizes primarily for small files and low‑latency access, not for warming up all data into the high‑performance layer. For AI training datasets that consist mainly of 10 MB‑level images or video files, this is crucial: even after a directory traversal, those large files may not actually enter the high‑performance layer under default settings.

Sync intervals and conflict resolution

S3 Files automatically maintains bidirectional sync between the file system and S3 in the background. When data in S3 changes, the file system view updates accordingly. Writes from compute nodes first land in the EFS high‑performance layer and then are batch‑synced back to S3 asynchronously. By default, the system aggregates modifications for a period before writing back.

The conflict resolution principle is clear: S3 is always the source of truth. If a file system modification hasn’t yet been synced back and the corresponding object has been updated by another application in S3, the system uses the latest version from S3 and moves the conflicting file to a .s3files-lost+found-* directory.

Performance boundaries and cost structure of S3 Files

The previous section explained how S3 Files works. This section discusses the resulting performance boundaries and cost structure. High‑performance layer occupancy, large‑file read path, write flow, and the amplification effects of partial updates and directory operations are the four most important aspects to evaluate.

Occupancy, eviction, and cost of the EFS high‑performance layer

S3 Files does not use capacity‑based LRU eviction. Instead, it uses access‑time‑based lifecycle management. By default, data that has been synced to S3 and not read for 30 days is evicted from the EFS high‑performance layer. This period is controlled by daysAfterLastAccess (configurable from 1 to 365 days).

This means the cost depends on how much data needs to reside in EFS and for how long. If your working set is large and stays active for a long time, costs will rise accordingly.

Large‑file direct reads and random reads: essentially client passthrough

S3 Files does not route all large‑file reads through the EFS high‑performance layer. With the default sizeLessThan of 128 KiB, only files below that threshold have their data moved into the high‑performance layer during import. For files already synced to S3, reads of 128 KiB or larger are streamed directly from S3.

In other words, S3 Files focuses on optimizing small‑file access and low latency, not on keeping large‑file reads in the high‑performance layer.

This direct‑read path requires that the compute resources themselves have permission to read the source bucket. AWS documentation explicitly requires roles to have s3:GetObject and s3:GetObjectVersion permissions; otherwise, the client cannot read directly from S3.

Cost of sequential writes: large‑scale writes introduce additional flow costs

The write path of S3 Files is not directly to S3. All writes first go into the EFS high‑performance layer and then are asynchronously synced back to S3.

This means that if your workload continuously generates large amounts of result data – for example, sequentially writing hundreds of terabytes of training outputs or analysis results – that data will incur two extra costs when flowing through S3 Files:

Data flow cost: Writes go into the high‑performance layer, then are later synced to S3. Compared to writing directly to S3, this path has an additional intermediate flow overhead.
Short‑term residency cost: After data is synced, it’s not immediately evicted from the high‑performance layer. It stays until the eviction condition is met. So temporary data from large writes may occupy EFS capacity for some time.

Using the current AWS US East (N. Virginia) regional pricing as an example: writing to EFS is about $0.06/GB, and reading it back for sync to S3 is about $0.03/GB. For the data flow alone, each 1 TB written adds roughly $90 in extra cost. If that data continues to reside in EFS after sync, further high‑performance layer storage costs will apply.

This is why S3 Files is better suited for reading existing data than for sustaining large‑scale, continuous result writing.

Partial updates and directory operations: amplification due to the object model

S3 Files does not chunk data at the file system level. It tries to maintain a direct mapping between files and S3 objects. The drawback: when you perform a small random write or append to a large file – from the application’s perspective a tiny update – the backend sync back to S3 can easily amplify into a full object write and versioning overhead.

For example, a user appends a 100 KB image key to a 100 GB LMDB file through S3 Files. The application sees a small write. But this modification is not immediately written back to S3. It’s aggregated for up to 60 seconds and then synced to the bucket. Unlike block storage, which would only update a single block, this type of update is more likely to be amplified into an object write, sync latency, and version storage costs. The larger the file and the more frequent the modifications, the more serious this issue becomes.

Directory renaming is similarly constrained by S3’s flat namespace. S3 does not have native directory metadata. Therefore, when you perform rename or mv, S3 Files cannot just update a single metadata entry. Instead, it must write new objects for every file under the directory and delete the old ones. For a directory with tens of millions of objects, this significantly prolongs sync time and increases S3 request costs. Until the sync completes, the file system view and the S3 view may be temporarily inconsistent.

Overall, S3 Files’ strengths are native access, zero data migration, and good compatibility with existing S3 assets. The drawback, however, is that once the workload shifts to large-file reads, continuous writes, frequent partial updates, or operations on large directories, both performance and cost are amplified more quickly. For this reason, S3 Files is better suited for lightweight shared access scenarios, while in heavy workloads such as training, data production, and large-scale analytics, its drawbacks tend to emerge earlier.

JuiceFS vs. S3 Files: two different architectural philosophies

As we saw earlier, many of S3 Files’ boundaries are not accidental; they are typical of solutions that try to map files directly to S3 objects. Whether it’s the early s3fs, Mountpoint for Amazon S3 (which focuses on high‑throughput reads), or today’s S3 Files, all of them try to maintain a direct mapping between files and S3 objects in exchange for transparent access to existing S3 data.

The advantage of this approach is transparency and low‑effort adoption. The cost has inherent limitations due to the S3 object model. That’s why directory operations tend to devolve into per‑object requests, and partial updates on large files easily amplify into write amplification, sync delays, and extra costs.

This is precisely why the difference between JuiceFS and these solutions is not a matter of a single feature or metric, but a fundamental divergence in architectural philosophy. JuiceFS is not an access layer that “mounts S3 as a file system”; it’s a cloud‑native distributed file system built on top of object storage. It adopts a decoupled metadata and data architecture: file data is stored in the underlying object storage, while metadata is independently managed by a high‑performance key‑value store. This makes JuiceFS much better suited for heavier production workloads such as training, analytics, and data production.

image: 1JuiceFS 架构图.drawio
caption: JuiceFS Community Edition

To help you compare these two architectures, here is a comprehensive table comparing JuiceFS and S3 Files:

Comparison basis	JuiceFS	Amazon S3 Files
Architecture	Metadata and data decoupled; files are chunked and written to object storage	EFS‑based proxy; data is not chunked; 1:1 mapping based on file size (< 128 KB by default)
Core cost	Open source software; costs from object storage, metadata engine, and cache resources	In addition to S3 storage, pay for EFS storage ($ 0.30/GB) plus data sync fees
Read/Write amplification (random writes)	Very low in many cases. Because of chunking, a partial random write typically updates only the affected block, not the whole file	Can be very high in data production scenarios. Random modification of a large file leads to rewriting and retransmitting the entire object
Tiering strategy	Based on capacity and access heat; hot data is automatically warmed up into local disk/memory cache on compute nodes	Based on file size and access time. Small files (< 128 KB) are cached in EFS; large files bypass EFS and read/write directly to S3
Small‑file performance	Relies on a fully in‑memory metadata engine (like Redis or TiKV); well suited for many small files and metadata operations	Depends on EFS performance and the NFS protocol
Large‑file throughput	Can use local NVMe / memory cache to boost throughput	Depends on EFS gateway or S3 direct performance; large‑scale parallel throughput is tied to capacity quotas
Cache consistency	Strong consistency (close‑to‑open) enforced by the independent metadata service	NFS close‑to‑open. But when concurrent modifications conflict between S3 and the file system, local EFS data is discarded into lost+found and S3 is forced as the source of truth
POSIX compatibility	Nearly 100% compatible. Supports hard links, atomic rename, and full lock semantics	Subset of NFSv4.1/4.2. Does not support hard links or atomic rename
Permission management	Supports standard POSIX permissions, ACLs, extended ACLs, and more	Supports standard POSIX permissions, ACLs, extended ACLs, and more
Encryption and security	In‑transit encryption and at‑rest encryption	In‑transit TLS encryption and at‑rest SSE‑KMS encryption
AI‑specific optimizations	Deeply optimized for AI‑common formats (such as LMDB and Safetensors) with mmap reads and local data warm-up	No AI‑specific optimizations; relies on basic streaming reads

Summary

There is no silver bullet. The right choice depends on your specific scenario.

The launch of S3 Files fills a gap in the AWS official ecosystem: seamless, zero‑migration native conversion of S3 into a file system. Its design philosophy is clear: 100% format transparency with the S3 object ecosystem, with targeted optimizations for small‑file (<128 KB) read/write performance in AI scenarios.

When to choose S3 Files

If your core need is to allow legacy applications, shell scripts, or traditional software to access existing S3 data as files without changing your architecture, or if you need a general shared file space that is primarily used for read‑only, small‑file, sequential workloads, then S3 Files is the more natural choice. Its native managed service, plug‑and‑play, and zero data migration can significantly lower the barrier to entry (though you may need to pay high EFS storage and sync costs for that convenience).

When to choose JuiceFS

If your workloads involve AI model training, data production, high‑performance computing (HPC), or big data analytics – facing tens of millions of small files, random reads/writes of terabyte‑scale large files, or demanding higher mmap performance, cache hit rates, and overall throughput – then JuiceFS is the better fit. Compared to S3 Files, JuiceFS’ data chunking, independent metadata engine, and more flexible caching system make it better suited for heavy‑workload, long‑running production and AI training/inference file system scenarios.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

JuiceFS 1.4: Lower Costs, Faster Metadata, and Better Control for Massive Data Management

Tue, 14 Jul 2026 08:42:00 +0000

JuiceFS Community Edition 1.4 is released. It’s the fifth major release since the open source edition was introduced in 2021 and is now the new Long-Term Support (LTS) version. We’ll continue to maintain both v1.4 and v1.3, while v1.2 has reached end of maintenance.

JuiceFS has now surpassed 14.2k GitHub stars. According to anonymous usage statistics reported by users, the total amount of data managed by JuiceFS Community Edition has exceeded 1.4 EB, representing more than 700× growth since 2022.

As JuiceFS is increasingly adopted for large-scale data management, high-concurrency workloads, and multi-user shared environments, long-standing challenges such as storage cost optimization, metadata performance, and resource governance have become more prominent. These are the primary focus areas of the 1.4 release.

In this post, we'll walk through the key improvements in JuiceFS 1.4, including tiered storage, faster metadata operations, enhanced resource management, more reliable data synchronization, metadata change tracking, and broader platform compatibility.

Lower storage costs with file- and directory-level tiered storage

As file systems continue to grow, different datasets naturally diverge in access frequency, performance requirements, and retention periods. Using a single storage type uniformly makes it difficult to simultaneously meet the performance needs of frequently accessed data and the cost-control requirements of infrequently accessed data. Object storage typically offers different storage classes based on access patterns, including hot data, warm (infrequent access) data, and cold (archival) data storage.

JuiceFS has supported setting object storage types via --storage-class since v1.1, but the configuration granularity was mainly at the file system default or mount point level. JuiceFS 1.4 integrates storage class into the file system semantics, supporting storage tier settings on a per-file or per-directory basis. Directory-level configurations can be inherited by subsequently created files and subdirectories. This facilitates tiered management by project, dataset, or application directory.

The storage tiers can be configured flexibly according to the object storage vendor being used. When writing new data, JuiceFS writes it to the corresponding object storage type based on the configuration of the file or its parent directory. For existing data, you can also adjust the metadata configuration and leverage the data migration capabilities on the object storage side to move it to a new storage tier. This capability is suitable for scenarios such as AI training datasets, log archiving, backup data, historical experiment data, and offline analysis results. For archive storage, it’s still necessary to evaluate retrieval latency and fees. For more implementation details, usage methods, and future evolution, see A Deep Dive into JuiceFS 1.4 Tiered Storage.

Faster metadata operations: batch delete, batch clone, and hotspot read optimization

In workloads involving massive numbers of small files, large directories, or high-concurrency access, metadata operations often become the primary performance bottleneck.

JuiceFS 1.4 addresses write transaction overhead and hotspot read overhead in metadata operations with optimizations including batch delete, batch clone, and Redis client-side caching.

Batch delete and clone: reducing transaction overhead

Previously, deleting a large number of files required the system to process them one by one, sequentially updating directory entries, inodes, space statistics, trash, and quota metadata. JuiceFS 1.4 consolidates the deletion of multiple non-directory files within the same directory into a batch transaction, reducing the repetitive overhead of per-file operations. This is applicable to scenarios like large directory cleanup, temporary data reclamation, training sample cleanup, and log directory deletion.

Batch cloning targets directory replication and snapshot scenarios. juicefs clone does not copy the underlying data blocks but creates new file records at the metadata layer and reuses the source file's data block references. JuiceFS 1.4 further reduces the metadata transactions generated by per-file cloning by processing clones of multiple files within the same directory in batch. This is ideal for AI dataset version management, experiment environment preparation, and large-scale directory snapshots.

image: deletion
caption: Throughput improvements for batch delete

image: clone
caption: Throughput improvements for batch delete

Redis client-side caching: reducing hotspot metadata read overhead

In high-concurrency reads, path resolution, directory entry lookups, and file attribute queries generate a large number of repeated requests. When Redis is used as the metadata engine, these requests require round trips between the client and Redis. This may impact access latency and increase Redis load.

JuiceFS 1.4 caches hot inode attributes and directory entries locally on the client side. When cached, it can reduce repeated queries to Redis. When related metadata changes, the local state is updated through a cache invalidation mechanism. It's important to note that this capability caches metadata, not file content.

It’s particularly beneficial for read-heavy workloads with stable hot paths, such as AI training data loading, large-scale container startup, and multi-task concurrent reading. For more implementation details, see Faster Metadata Operations with Batch Unlink, Batch Clone, and Redis Client-Side Caching.

Improved resource management: user quotas and trash usage statistics

In distributed storage environments, storage resources are often shared among multiple users, teams, and projects.

Without effective governance, accidental writes or abnormal workloads from a single user can quickly consume large amounts of storage space or inodes. This affects both system stability and operating costs. Quota management is a critical means of establishing predictable resource boundaries in shared environments.

JuiceFS Community Edition 1.4 introduces user and group quotas, allowing administrators to monitor, configure, and enforce resource limits based on identities.

Resource governance is now extended beyond file system and directory quotas to include user- and group-level quotas, making it especially suitable for shared clusters and AI training platforms.

To reduce metadata overhead in multi-client environments, JuiceFS uses asynchronous accounting so that usage statistics converge gradually over time. For details, see Quota Design in Distributed Architectures: Implementation and Use Cases in JuiceFS.

Supported quota types include:

Quota type	Scope	Design goal	Typical use case
Total file system quota	Entire file system	Prevents overall resource runaway	Cost budget control, capacity limit
Subdirectory quota	Directory subtree	Blocks abnormal write behavior	Prevents misoperations, small‑file storms
User quota	Per user	Isolates impact between different applications	Multi‑tenant data management
User group quota	Project or department	Cost allocation and team limits	Shared environment for AI projects

JuiceFS 1.4 also improves trash space visibility.

Deleted files may remain in the trash for a retention period, making it difficult to understand why storage space has not yet been reclaimed. The enhanced summary tool now reports trash usage, helping administrators identify storage consumption and make informed cleanup, retention, or expansion decisions.

Expanded capabilities: sync, backup, and change tracking

More reliable large-scale sync

Large-scale migration, cross-cloud sync, backup, and archival workloads often face interruptions, security requirements, and bandwidth contention.

JuiceFS 1.4 significantly enhances juicefs sync with three major capabilities:

Resumable sync: It reduces recovery costs after task interruptions. During synchronization, JuiceFS records the task progress. If the task exits abnormally or is manually interrupted, it can resume from the saved state, reducing repeated scanning and processing. This capability is suitable for migration and backup scenarios with a large number of objects, long task durations, or unstable cross-cloud links.
Data encryption and decryption: In cross-cloud backup and archiving scenarios, client-side encryption is a common compliance requirement. JuiceFS 1.4 supports completing encrypted writes, decryption recovery, or re-encryption within the sync pipeline, reducing reliance on external encryption tools. This capability is suitable for off-site backup, sensitive data migration, key rotation, and compliance auditing. However, it requires careful management of key storage and recovery processes.
Global traffic control: It provides bandwidth constraints for concurrent multiple sync tasks. Compared to per-process rate limiting, version 1.4 can centrally manage the overall bandwidth usage of multiple sync tasks, reducing the impact of sync tasks on online application and other network activities. This is suitable for cross-cloud transfers, multi-task concurrent backups, data center migrations, and shared outbound link scenarios. For implementation details, see JuiceFS Sync for PB-Scale Data Transfers: Resumable Sync, Encryption, and Bandwidth Control.

image: checkpoint
caption: Checkpoint save, restore, and cleanup workflow

Changelog: metadata changes become traceable

JuiceFS Community Edition 1.4 introduces a metadata changelog capability, which records metadata change events across the file system.

Previously, troubleshooting relied primarily on client-side access logs, which only reflected operations performed through individual mount points. In multi-client deployments, reconstructing a complete sequence of events was often difficult.

A changelog records metadata operations—including file creation, deletion, attribute updates, and renames—directly at the metadata layer, providing a unified source for troubleshooting, auditing, and incremental processing.
Administrators can now quickly identify accidental deletions, unexpected renames, permission changes, and metadata modifications without collecting logs from every client.

When issues like accidental deletion, abnormal renaming, or unexpected permission or attribute changes occur, administrators can review the relevant change records based on the changelog. This reduces dependence on single-client logs and shortens the troubleshooting path. It also provides a more unified source of metadata changes for operational auditing.

In backup, migration, and recovery scenarios, the changelog can serve as a reference for incremental processing. For large-scale file systems, numerous changes may occur between two full backups or migration tasks. By recording the metadata changes during this period, the changelog can provide input for subsequent incremental backups, migrations, or recovery processes, reducing reliance on full scans.

Better support across different environments

JuiceFS Community Edition 1.4 further improves compatibility across diverse deployment environments.

On Windows clients, the release improves cross-platform consistency and stability, including user mapping, permission mapping, and file access behavior, reducing compatibility issues when Linux and Windows clients access the same file system.

For the Java SDK and the Hadoop ecosystem, JuiceFS 1.4 adds Kerberos authentication, completing support for Hadoop secure mode.

JuiceFS 1.3 already introduced Apache Ranger integration for authorization and access control. Together, Kerberos authenticates who the user is, while Ranger determines what the user can access, providing a more complete security model for enterprise big data platforms.

On the storage backend side, JuiceFS 1.4 also adds support for SMB/CIFS-based storage. This makes it easier to integrate with existing NAS or file-sharing infrastructure.

Continued growth in scale and AI adoption

According to anonymous usage statistics, JuiceFS Community Edition now powers nearly 70,000 file systems managing over 1.4 EB of data, with deployment scale continuing to grow.

image: count and total data volume
caption: JuiceFS Community Edition file system count and total data volume

Over the past year, AI applications have continued to expand from model training to inference services, agents, and multi-cloud scheduling, placing higher demands on data storage. These changes are also reflected in the use cases shared by community users, covering areas such as large language models, autonomous driving, quantitative investment, and computing platforms. We thank these users for sharing their real-world practices, providing valuable references for more teams building AI data infrastructure.
New user stories:

AI training and large language models

AIGC

Autonomous driving and robotics

Inference and AI agents

Quant investment

JuiceFS+MinIO: Ariste AI Achieved 3x Faster I/O and Cut Storage Costs by 40%+

Big data

NAVER, Korea's No.1 Search Engine, Chose JuiceFS over Alluxio for AI Storage

Development of JuiceFS 1.4 spanned nearly a year. During the release cycle, the community reported 366 issues, merged 515 pull requests, and welcomed contributions from 59 contributors. We sincerely thank everyone who reported issues, contributed code, improved documentation, and helped evolve JuiceFS for increasingly demanding production environments. Your participation drives the rapid growth of JuiceFS.

Download and try JuiceFS 1.4 here.

JuiceFS Sync for PB-Scale Data Transfers: Resumable Sync, Encryption, and Bandwidth Control

Wed, 08 Jul 2026 09:42:00 +0000

In scenarios such as data migration, cross-cloud synchronization, and object storage backup, juicefs sync is commonly used to transfer large volumes of data. When datasets grow to the TB- or PB-scale, with millions or even billions of objects, a single synchronization task may run for hours or even days.

As these long-running jobs progress, several common challenges tend to emerge:

After network interruptions, process crashes, or node restarts, tasks often struggle to resume from a consistent state and may need to rescan or reprocess data.
Backup workflows may expose plaintext data and face compliance or security requirements.
When multiple sync jobs run concurrently, bandwidth contention becomes significant, while the overall transfer process lacks effective global control.

To address these challenges, JuiceFS 1.4 introduces three major enhancements to sync: resumable sync, data encryption/decryption, and global traffic control.

In this article, we’ll explain the use cases, implementation details, and configuration methods for each feature.

Resumable sync

In earlier versions, if a synchronization task failed or was interrupted, rerunning juicefs sync required rescanning both the source and destination before determining which objects had already been synchronized and which still needed to be copied.

For workloads involving hundreds of millions of objects or large files, the scan itself could incur substantial time and object-storage request costs.

To address this issue, JuiceFS 1.4 introduces a resumable sync mechanism for sync. When enabled, synchronization progress is periodically saved to the destination. If the task is interrupted, rerunning the same command automatically locates and loads the matching checkpoint and resumes from the last unfinished position, avoiding a full restart.

How it works

When resumable sync is enabled, sync stores a JSON state file on the destination side:

.juicefs-sync-checkpoint.<hash>.json

The <hash> value is derived from the source, destination, and key synchronization parameters. This ensures that a task only loads checkpoints created for itself, preventing accidental reuse across different jobs.
The workflow is shown below:

image: Checkpoint 的流程
caption: Resumable sync workflow

Checkpoint save, restore, and cleanup workflow in juicefs sync:

When sync starts, it first looks for a checkpoint matching the current task.
If a matching checkpoint is found, execution resumes from the saved state. Otherwise, synchronization starts normally with a fresh scan. sync traverses multiple prefixes concurrently, maintaining independent state for each prefix, including:
- Whether traversal is complete
- The last scanned position
- Pending objects to synchronize
- Failed objects
When restoring from a checkpoint:
- Pending and failed objects recorded in the checkpoint are re-added to the task queue.
- Prefixes that were not fully traversed resume scanning from their saved positions.
- Fully traversed prefixes only continue processing unfinished objects recorded in the checkpoint.
During execution, progress is saved asynchronously at a configurable interval, which defaults to every 10 seconds.
After successful completion, the checkpoint file is automatically removed. If the task fails or is interrupted, the checkpoint is retained for resumption on the next execution of the same command.

image: 集群模式
caption: Checkpoint architecture in cluster mode

In cluster mode, only a single checkpoint exists and is maintained centrally by Manager.
Workers do not directly read or write checkpoint files on the destination. Instead, they:

Pull tasks from Manager
Execute synchronization
Report results back to Manager

Manager aggregates completed objects, failed objects, statistics, and multipart-upload state into the global checkpoint.

Usage

# Enable resumable sync.
juicefs sync --enable-checkpoint SRC DST

# Customize checkpoint save interval (default: 10s).
juicefs sync --enable-checkpoint --checkpoint-interval 30s SRC DST

# Ignore existing checkpoints and restart from scratch.
juicefs sync --enable-checkpoint --checkpoint-force-reset SRC DST

Data encryption and decryption

For cross-cloud backup and archival workflows, client-side encryption is often required to satisfy compliance requirements such as data sovereignty, encryption at rest, and secure migration of sensitive data.

Previously, juicefs sync did not provide built-in encryption capabilities. Users who wanted to write encrypted data to the destination typically had to use external tools for additional processing.

In JuiceFS 1.4, streaming encryption and decryption are integrated directly into the synchronization pipeline, enabling three common workflows:

Encrypt-on-write: Encrypt plaintext data before writing it to the destination, suitable for encrypted backup and archiving.
Decrypt-on-read: Read encrypted data from the source and write decrypted data to the destination, suitable for data recovery or plaintext migration.
Re-encryption: Decrypts source data with an old key and re-encrypts it with a new key before writing to the destination, suitable for key rotation or cryptographic algorithm migration.

Chunk-based streaming encryption

To support object storage Range GET operations while avoiding excessive memory usage for large files all at once, sync uses a fixed-size 1 MiB chunk-based streaming encryption scheme.

A file is first divided into plaintext chunks:

[chunk 1: 1 MiB][chunk 2: 1 MiB] ... [chunk N: ≤1 MiB]

Each plaintext chunk is encrypted independently.

Each encrypted chunk consists of a 4-byte header and the ciphertext data, where the 4-byte header stores the actual ciphertext length (ct_len):

Each encrypted block: [4B ct_len][ciphertext + padding]

Encrypted file: [encrypted chunk 1][encrypted chunk 2] ... [encrypted chunk N]

The encrypted block size is determined by the plaintext chunk size plus encryption overhead: plainChunkSize + overhead. The plainChunkSize is fixed at 1 MiB, and the overhead depends on the encryption algorithm and key type used.

image: 分块流式加密
caption: Chunk-based streaming encryption architecture in juicefs sync

This design allows random reads to retrieve only the required encrypted chunk rather than downloading the entire file. Because encrypted objects contain additional headers, padding, and encryption metadata, the destination object is typically larger than the original plaintext file.

Supported algorithms

The table below shows the supported algorithms:

Option	Symmetric cipher	Key encapsulation	Typical use case
aes256gcm-rsa (default)	AES-256-GCM	RSA	General-purpose workloads
chacha20-rsa	ChaCha20-Poly1305	RSA	Environments without efficient AES hardware acceleration
sm4gcm	SM4-GCM	SM2	Scenarios requiring Chinese commercial cryptography standards

Usage

The following examples use RSA keys.

Generate a key pair:

# Generate an RSA private key (the public key is derived automatically).
openssl genrsa -out private.pem 2048

# Generate a password-protected private key.
openssl genrsa -aes256 -out private.pem 2048

Scenario 1: Encrypt and write to destination

juicefs sync /local/data s3://mybucket/backup 
    --encrypt-rsa-key /path/to/private.pem

Scenario 2: Decrypt and read from source for data recovery or plaintext migration

juicefs sync s3://mybucket/backup /local/data 
    --decrypt-rsa-key /path/to/private.pem

Scenario 3: Re-encrypt for key rotation or algorithm migration

# Decrypt data encrypted with the old key and re-encrypt with the new key to new storage.
juicefs sync s3://old-bucket/encrypted s3://new-bucket/re-encrypted 
    --decrypt-rsa-key /path/to/old-private.pem 
    --encrypt-rsa-key /path/to/new-private.pem

If the private key is password-protected, the password can be provided via environment variables:

# For encryption scenarios, use JFS_ENCRYPT_RSA_PASSPHRASE.
export JFS_ENCRYPT_RSA_PASSPHRASE="your-passphrase"
juicefs sync /local/data s3://mybucket/backup --encrypt-rsa-key private.pem

# For decryption scenarios, use JFS_DECRYPT_RSA_PASSPHRASE.
export JFS_DECRYPT_RSA_PASSPHRASE="your-passphrase"
juicefs sync s3://mybucket/backup /local/data --decrypt-rsa-key private.pem

Notes:

Encrypted data is stored using a JuiceFS-specific format and can only be decrypted through juicefs sync with the corresponding key.
Back up encryption keys carefully. Once a private key is lost, encrypted data cannot be recovered.

Global traffic control

In earlier versions, juicefs sync already supported per-process rate limiting via --bwlimit. However, when multiple sync processes run concurrently—such as multiple Workers in a distributed sync, or multiple independent sync tasks sharing the same egress link—per-process limiting cannot constrain total bandwidth usage. The egress link may still be saturated, affecting other application traffic.

JuiceFS 1.4 introduces the --traffic-control-url parameter. Multiple sync processes can connect to the same external traffic control service, which allocates bandwidth quotas uniformly, enabling cross-process, cross-task global rate limiting.

How it works

Global traffic control uses a token bucket model. Before transmitting data, each sync process requests byte credits from the same traffic-control service.

image: 全局流量
caption: Global traffic-control workflow in juicefs sync

Each process periodically requests a certain number of bytes (credit) before data transfer.

The traffic-control service determines:

How many bytes to grant
How long the granted quota remains valid

When credits are exhausted, the process requests additional credits.

If a quota is about to expire before being fully consumed, the unused portion is returned to the service in advance.

The service exposes a simple HTTP API for granting and reclaiming quotas. This must be implemented by the user or integrated with an existing service:

POST /traffic-control
Content-Type: application/json

Request:
{"bytes": 1048576}
  bytes > 0: Request byte credits.
  bytes < 0: Return unused credits.


Response:
{"granted": 524288, "expired": 1000}
  granted: Number of bytes granted this time.
  expired: Credit validity period (milliseconds).

During synchronization, sync requests quotas from the traffic control service before transmitting data. If no credits are available, transmission blocks until new credits are obtained. In this way, multiple sync tasks can share a single global bandwidth limit, preventing the total traffic from becoming uncontrolled even when individual tasks have their own limits.

Usage

# Deploy a traffic-control service first.
# (Example: listen on port 8080 and cap total bandwidth at 100 Mbps)
# (Service implementation is user-defined; JuiceFS only calls the API)

# Multiple sync processes share the same control service.
juicefs sync SRC1 DST1 --traffic-control-url http://127.0.0.1:8080/traffic-control &
juicefs sync SRC2 DST2 --traffic-control-url http://127.0.0.1:8080/traffic-control &

--traffic-control-url can be combined with --bwlimit.

The two mechanisms are independent:

--bwlimit limits the bandwidth of a single sync process.
--traffic-control-url limits aggregate bandwidth across multiple processes.

# Per-process limit: 50 Mbps. All processes combined respect the service-side cap.
juicefs sync SRC DST 
    --bwlimit 50 
    --traffic-control-url http://controller:8080/traffic-control

Summary

JuiceFS 1.4 enhancements to sync include:

Resumable sync reduces recovery costs after task interruptions.
Encryption and decryption improve the security of backups and archival data.
Global traffic control enables multiple synchronization tasks to share bandwidth in a coordinated manner.

For scenarios such as data migration, cross-cloud sync, object storage backup, and encrypted archiving, users can combine these capabilities flexibly based on task scale, network environment, and security requirements.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

42x Faster Writes & 85% Throughput Gain: JuiceFS for Multi-Cloud AI Agents

Wed, 01 Jul 2026 09:59:51 +0000

Xingchen Zhengtu Technology Co., Ltd. is a startup focused on AI search and multi‑modal artificial intelligence generated content (AIGC) applications for e‑commerce. Our products include:

Gensmo, which focuses on fashion and provides virtual try‑on, style recommendations, and product search.
ZooClaw, which offers AI agent services for a wider range of daily and work scenarios.

In this article, we’ll share our thinking and experience in unified storage selection, architecture design, and performance tuning. We’ve been running JuiceFS in production for more than a year. It manages over 100 million files. Our application runs across Oracle Cloud, DigitalOcean, and Google Cloud Platform (GCP). JuiceFS has become the unified storage layer supporting model training, inference, data processing, and online agents.

Unified storage requirements and design approach

Four scenarios, four I/O profiles

So far, we have four main storage scenarios that support Gensmo and ZooClaw.

image: Application-types-Xingchen-Zhengtu
caption: Application types at Xingchen Zhengtu Technology

Scenario 1: Model training
We build our own models, including Gensmo's Try‑on model and video generation models. They show try‑on results, 360‑degree model actions, or special effects to B2B and B2C customers. Model training involves large‑file sequential writes and checkpoint saving. Storage requirements: high capacity and high‑performance sequential I/O.

Scenario 2: Model inference
Inference services primarily need high‑concurrency sequential reads. Data is loaded into local cache to increase hit rates.

Scenario 3: Data processing
We crawl product, clothing, and review data from overseas independent e‑commerce sites. This data is used for model training and application operations analysis. This scenario involves many small files (a few hundred kilobytes per image). High IOPS and concurrency are challenges.
To optimize data processing, we use Ray Data to process datasets in parallel and consolidate large volumes of small files into reusable Parquet datasets ranging from tens to hundreds of gigabytes. These datasets serve as a shared data foundation that can be reused across downstream workloads such as embedding generation, retrieval, recommendation, training, and inference. This greatly reduces pressure on the file system and serves both training and inference needs.

Scenario 4: Online agents
Online agents are different from the offline scenarios above. There are still many small files, but these files are generated by online services. Each agent only reads and writes its own data. No cross‑agent distributed processing is required. The storage system must support high concurrency and fast response. It does not need cross‑agent data coordination.

In summary, these four scenarios have two types of storage requirements:

Offline training, inference, and data processing need high throughput, high concurrency, and caching.
Online agents care more about low latency, data isolation, and stability.

After clarifying these application needs, a natural question arises: do we need a multi‑cloud architecture? From the very beginning of platform construction, our answer was yes.

Cloud neutrality is not a concept; it’s bargaining power

The goal of cloud neutrality is not technical purity. It’s to meet the core needs of the infrastructure team: keeping compute resources and workloads portable, and maintaining bargaining power with different cloud providers.

For overseas business, if compute and storage are locked into a single cloud provider for a long time, the ability to flexibly adjust compute resources becomes limited as the application grows or prices change. This is especially true in AI, where GPU resource prices and availability fluctuate greatly. A resource that is cheap today may become expensive or scarce later. The compute scale needed after application growth may also exceed what the original cloud provider can offer.

Therefore, we want to decouple the storage layer from any specific cloud vendor. Data should stay cloud‑neutral. This way, training, inference, or online agent workloads can move to a cloud that better meets cost and performance requirements. There is no need to copy or reconfigure data repeatedly.

POSIX: The foundation of a unified storage experience

Another critical question in platform construction was: how do we give our engineering team a consistent operational experience across multiple clouds and multiple object storage backends?

For a single application scenario, using object storage directly is enough. But when training, inference, data processing, and online agents share the same data system, the development and operations cost of different object storage interfaces grows quickly. Therefore, we wanted a unified abstraction on top of the underlying storage. POSIX file system semantics are the best fit for this.

With JuiceFS, we map underlying object storage (whether it’s GCS, S3, or R2) into a POSIX file system and mount it as a local path. From local development to production, the engineering team always sees the same file system interface and access path. They don't need to care about which cloud or which object storage holds the data.

In short, an ideal cloud storage experience lets engineers not feel the multi‑cloud environment at all. They always see a local path. This was a major reason we later chose JuiceFS.

Selection: From GCS Fuse and S3 Fuse to JuiceFS

Because offline and online scenarios have very different needs, our storage selection took two different paths.

Offline: Choosing JuiceFS from the start

For offline scenarios, we faced a multi‑cloud environment and high-throughput requirements. Therefore, before building the system, our team evaluated mainstream solutions and compared them against our critical needs:

Self‑built parallel file systems: Highest performance, but high cost, hardware‑locked, limited cross‑cloud capability.
Cloud‑managed parallel file systems: Convenient, but locks you into a single cloud, still expensive.
Plain FUSE: Low cost, but insufficient POSIX semantics and performance.
Cache orchestration layers: Needs underlying storage and adds operational complexity.

Solution	Cloud-neutral	POSIX semantics	High throughput	Distributed cache	Cost/Operations
Self-built parallel file system (for example, Lustre)	❌ Hardware-dependent	✅	✅✅	Partial	High cost, operationally intensive
Cloud‑managed parallel file system (for example, Filestore)	❌ Locked to a single cloud	✅	✅	✅	High cost, lower operational overhead
Object storage + FUSE (S3FS / GCS Fuse)	⚠️ Cloud-dependent	❌	❌	❌	Low cost, lightweight operations
Cache orchestration layer (Alluxio/Fluid)	✅	✅	✅	✅	Requires an underlying storage layer, operationally complex
JuiceFS	✅ Backend of your choice	✅ Full	✅	✅ Built-in	Object storage cost with CSI integration

Compared with the alternatives, JuiceFS was the only solution that simultaneously met our core requirements: cloud neutrality, full POSIX compatibility, built-in distributed caching, and object storage as the backend. Every other option fell short in at least one of these areas. As a result, for our offline workloads, choosing JuiceFS was a straightforward decision from the very beginning.

Agent: Migrating from GCS Fuse to JuiceFS

Our early infrastructure was primarily deployed on Google Cloud, where Google Cloud Storage (GCS) was mounted into GKE Pods using GCS Fuse. In practice, however, we found that this architecture could not meet the reliability, performance, and cloud-neutrality requirements of our online agent workloads.

The biggest problem was data loss under SIGKILL. GCS Fuse uses an asynchronous writeback mechanism. After an application's write call returns success, data may still sit in a local buffer and not be truly written to GCS. If a Pod is terminated by an OOM kill or a SIGKILL signal, data that appears to have been written successfully may be permanently lost. In an agent workload, this directly causes session data loss.

The second issue was inadequate small-file performance and incomplete POSIX semantics. An agent's working directory typically contains numerous small files with frequent append operations. GCS Fuse introduces high latency for operations such as open and stat, while providing incomplete support for POSIX features such as rename, flock, and symlink. As a result, it falls short of the stability requirements for online services.

The third issue was cloud lock-in and limited stability under high concurrency. GCS Fuse is tightly coupled to the GCP ecosystem, making it incompatible with our cloud-neutral infrastructure strategy. In addition, its stability under highly concurrent agent workloads proved insufficient.

These limitations ultimately led us to migrate our online agent workloads to JuiceFS.

JuiceFS addresses the data loss problem through its write path and independent metadata engine. It separates data from metadata: data chunks are first uploaded to object storage, and only after the corresponding metadata is atomically committed to the independent metadata engine is the write considered successful. In other words, a successful write truly means the data has been durably persisted, so a SIGKILL cannot cause the loss of already acknowledged data.

More fundamentally, GCS Fuse exposes object storage through a file system interface, whereas JuiceFS builds a true file system on top of object storage. Its independent metadata engine, together with full POSIX compatibility, cloud neutrality, built-in distributed caching, and a mature ecosystem of tools, makes JuiceFS a much better fit for online agent workloads that demand reliability, consistency, and high-concurrency access.

Today, our online agent platform runs stably in production. JuiceFS has become the unified storage foundation for multiple workloads across the company.

A new architecture: Deploying JuiceFS across multiple clouds

Offline workloads: Compute mobility across clouds with unified metadata + R2

To meet the requirements of cloud neutrality, compute mobility, and high throughput for offline workloads, we designed the following architecture:

We chose Cloudflare R2 as the underlying object storage backend. Because R2 is independent of any cloud provider and does not charge for egress traffic, it’s well suited for high-throughput training workloads spanning multiple clouds. By comparison, while services such as GCS and Amazon S3 offer low storage costs, their egress fees can be substantial, significantly increasing the cost of offline training. For example, storing 1 TB of data in GCS costs roughly $20 per month, while egress charges can range from $20 to $140.

image: Offline-storage-architecture
caption: Offline storage architecture

On top of R2, we deployed JuiceFS Enterprise Edition to provide a unified file system across multiple clouds. Whether compute resources are running on Oracle Cloud or DigitalOcean, training, inference, and data processing jobs all access the same file system through a consistent namespace. This allows engineers to work without worrying about the underlying cloud infrastructure.

Our compute layer consists of H100 GPUs on Oracle Cloud and H200 GPUs on DigitalOcean, running a unified training and inference platform based on Slurm and KubeRay. Each GPU node contributes its local NVMe SSD to JuiceFS' distributed cache, forming a shared cache pool across the cluster. Datasets are fetched from R2 on first access, while subsequent reads are served almost entirely from the distributed cache, effectively masking the latency of cross-cloud data access.

The entire infrastructure is managed as code using Terraform. Networks, storage, training jobs, Ray clusters, and inference services can all be provisioned automatically. As long as a cloud provider supports Kubernetes, compute resources and workloads can be deployed seamlessly, enabling rapid cross-cloud scaling and flexible resource allocation.

Online workloads: Prioritizing low latency with in-cloud metadata

For online agent workloads, using ZooClaw as an example, our primary objective is to provide a unified storage foundation for a large number of agents while supporting centralized management, directory isolation, and usage-based billing. Unlike offline workloads, the online environment prioritizes low latency, efficient small-file writes, and high-concurrency access. Cross-cloud storage paths inevitably introduce additional I/O latency, making them unsuitable for latency-sensitive online services. Therefore, we keep the object storage, metadata service, and application Pods within the same cloud whenever possible.

Our current online deployment runs entirely on GCP. GCS serves as the object storage backend, while the metadata layer is provided by an independent three-node Raft cluster deployed within a private GCP VPC. This keeps object storage, metadata service, and application Pods within the same cloud. It reduces access latency and improves IOPS for small‑file write‑intensive scenarios.

On Kubernetes, we mount a single RWX PersistentVolumeClaim (PVC) through JuiceFS CSI. Each bot Pod accesses its own isolated directory through a dedicated subPath, while access is restricted by environment-specific tokens, providing file system-level isolation. From each agent's perspective, it sees only its own local working directory. From the platform's perspective, however, all data resides in a single unified storage system, simplifying centralized management and billing.

image: Online-storage-architecture
caption: Online storage architecture

If GCP resources or costs become unsuitable in the future, this architecture can still be migrated. We use Terraform and Kubernetes orchestration. We can spin up the same compute and storage structure on another cloud and then synchronize the metadata and data over. Online agent workloads can naturally be switched bot by bot, user by user, or tenant by tenant. There is no need for a one‑time full migration.

Looking back at both deployment models, their objectives are fundamentally different. Offline workloads emphasize cross-cloud data sharing, compute mobility, and high throughput, while online agent workloads prioritize low latency and high concurrency, without sacrificing the flexibility to migrate when needed. Rather than adopting a one-size-fits-all backend, we tailored the JuiceFS deployment to the requirements of each workload. This approach preserves a unified data management model and developer experience while allowing each scenario to use the most appropriate metadata and object storage architecture.

Optimization practice: Distributed cache, writeback, and S3 Gateway

After establishing a unified storage architecture, we continued to optimize performance and fine-tune access strategies for different workloads.

One cache, two optimization strategies

Distributed caching is one of the most important capabilities of JuiceFS, directly affecting IOPS, throughput, and access latency. Although both our offline and online deployments rely on distributed caching, their goals and optimization strategies differ significantly.

For offline workloads, the primary objective is to support high-throughput training and data processing while enabling cross-cloud data sharing and compute mobility. To achieve this, we cache as much data from Cloudflare R2 as possible on local storage. Training, inference, and data processing jobs run on H100 and H200 GPU nodes equipped with NVMe SSDs. Each node contributes roughly 50 TB of local cache, allowing a cluster of more than a dozen nodes to provide several hundred terabytes of distributed cache capacity.

The first access to a dataset requires fetching data from R2, which is slow. Once the initial read completes, however, subsequent training, data processing, and inference workloads are served almost entirely from the distributed cache, delivering I/O performance close to that of local NVMe storage. Since offline workloads write large checkpoint and model weight files—often hundreds of gigabytes or even several terabytes in size—data integrity is critical. Therefore, we typically leave writeback disabled to ensure that writes are durably persisted before being acknowledged.

For online agent workloads, the priorities are low-latency, high-concurrency access to small files while maintaining data isolation for each agent. Here, the cache is primarily used to accelerate small-file writes and access. Each agent Pod mounts the same RWX PVC and accesses its own isolated directory using a dedicated subPath. The cache expiration time is set to 3,600 seconds to maximize cache hits for frequently accessed files.
Because each agent typically accesses only its own directory, the cache strategy does not require strong cross‑agent consistency. Data is only eventually consistent with object storage during necessary offline analysis or operations debugging.

In online scenarios, to further improve small file write and high concurrency performance, we can use writeback together with caching. Writeback trades a controlled amount of data safety risk for higher write throughput. This means that for multiple agents running on the same node, if one agent fails during writing, only that single agent's output, such as a presentation, an image, or a temporary document, is affected. That data can be regenerated. With writeback, online agents get a significant performance boost for high‑concurrency small file writes, while the overall system remains stable and data isolation is preserved.

One copy of data, multiple interfaces

In our architecture, JuiceFS S3 Gateway serves as the data distribution layer, exposing data stored in JuiceFS through a standard S3-compatible interface.

In agent workloads, whether the data consists of configuration files or generated artifacts such as presentations, images, or videos, everything is ultimately stored in the same JuiceFS file system. However, these files often need to be shared with external users through URLs, making POSIX mounts unsuitable for external access.

To address this, we expose the same data directly through JuiceFS S3 Gateway. Internal services continue to access files through the POSIX interface, while external applications retrieve exactly the same data through standard S3 or HTTP APIs, eliminating the need for duplicate copies.

To further improve security and performance, we place Cloudflare Workers and a CDN in front of the S3 Gateway. Incoming requests first pass through a worker for path validation and access control before being forwarded to the gateway. CDN edge caching and ETag validation further reduce origin fetches.

This design provides two major benefits:

Multiple layers of isolation ensure data security, including JuiceFS directory isolation, S3 Gateway access controls, and code-level validation implemented in Cloudflare workers.
CDN caching reduces cross-region access latency and significantly improves download performance for large files such as videos and images. For users around the world, this means that even when the data resides in a GCP region in the eastern United States, content can still be served efficiently from the nearest CDN edge location.

From an architectural perspective, internal training, inference, and agent services access data through the POSIX file system, while external distribution is handled through the S3 Gateway. A single copy of the data can support multiple access methods without any additional replication.

Performance results

Offline workloads: ~4× higher sequential write throughput, 7–8 GB/s cached read performance

For our offline workloads, we conducted benchmark tests on sequential read and write performance. The results before and after optimization are summarized below.

For sequential writes, a single process achieved approximately 700 MB/s when writing model outputs or checkpoints. By leveraging parallel writes across multiple processes and multiple nodes, throughput exceeded 1 GB/s, providing sufficient performance for large-scale training workloads.
For sequential reads, after aggregating numerous small files into larger files during data processing and loading them into the distributed cache, cached sequential read throughput reached 6.7–7.8 GB/s, approaching the performance of local NVMe storage. Inference workloads can also load checkpoints directly from the local cache, eliminating the need to copy data across nodes.

Benchmark (JuiceFS on R2, offline)	Baseline	Optimized (distributed cache + tuning)
Sequential writes (large blocks)	~231 MB/s	~714 MB/s
Sequential writes (20–50 GB datasets)	~256–265 MB/s	840 MB/s–1.1 GB/s
Sequential reads (cache hit)	—	6.7–7.8 GB/s
Sequential reads (cold read from R2)	—	~427 MB/s

Beyond performance improvements, distributed caching also streamlined our engineering workflows. Training, inference, and data processing all share the same file path, eliminating the need to copy checkpoints between nodes or services. Newly generated model weights can be loaded directly by inference services, reducing data movement while shortening the path from model training to deployment.

Online workloads: ~42× faster small-file writes and ~85% higher large-file throughput

Our initial deployment placed the metadata service on OCI while using Cloudflare R2 as the object storage backend. Since the online services ran on GCP, every metadata request had to traverse the public Internet, resulting in a metadata round-trip time (RTT) of approximately 12.7 ms. Small-file throughput was limited to around 24 files/s, and occasional 30-second PUT timeouts from R2 could even affect bot stability.

We introduced two major optimizations.

We enabled writeback and tuned the cache TTL, improving large-file write throughput by approximately 85%.
We migrated both the metadata service and object storage into GCP. The metadata layer now runs as a three-node Raft cluster within a private VPC, while the object storage backend was migrated to GCS and paired with local NVMe caching.

After these optimizations, metadata RTT dropped to approximately 5.8 ms, while small-file throughput increased to around 1,000 files/s, representing an overall performance improvement of roughly 42×.

Conclusion

After more than a year of production use, JuiceFS has become the core storage layer of Xingchen Zhengtu Technology's infrastructure. Today, it reliably supports more than 100 million files across three cloud providers and multiple application scenarios. More importantly, it has unified the storage architecture for training, inference, data processing, and online agent workloads.

For a startup serving a global market, infrastructure must be both flexible and easy to operate so that engineering efforts can remain focused on product innovation. Our unified storage platform provides a consistent interface for applications and developers, while allowing the underlying infrastructure to be optimized for different workloads. Offline workloads prioritize compute mobility and cost optimization, whereas online workloads focus on low latency and high concurrency while retaining the flexibility to migrate when necessary.

This design keeps a consistent upper‑layer experience, while making compute cost negotiable and resources portable. It sets a solid foundation for future expansion to more clouds and regions.
If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

Monitoring JuiceFS with Better Stack

Thu, 25 Jun 2026 06:57:00 +0000

After deployment, JuiceFS feels like a local drive, but underneath it's a sophisticated distributed system. This perfectly reflects one of its core design principles: distributed systems are complex, but from a user's perspective, they should be simple to use.

Even so, that simplicity on the surface doesn't negate the need for deep visibility. For any critical storage system, gaining real-time visibility into its operations is crucial to prevent subtle performance degradations from escalating into significant incidents.

Fortunately, JuiceFS exposes a suite of monitoring metrics, including throughput, IOPS, latency, data size, and many more, in the widely adopted Prometheus format, making it ready for modern monitoring stacks. Traditionally, you would probably pair Prometheus with Grafana to collect these metrics and visualize them. This is indeed a powerful combination. However, deploying, managing, and maintaining these systems yourself adds operational overhead again. Ironically, you may want to monitor them too, and trust me, you would rather not create yet another monitoring stack just to monitor your Prometheus and Grafana combo.

That's where Better Stack comes in. It is a fully managed SaaS observability platform that combines user-friendly dashboards, tracing, logging, error tracking, incident management, automatic alerting, and even AI-powered SRE, all for a predictable, cost-effective price. With Better Stack, you get the power of the best-in-class tools out of the box without the operational overhead.

image: Better Stack
caption: Better Stack

In this post, we'll guide you through setting up a comprehensive monitoring system for JuiceFS using Better Stack, from metric ingestion to intelligent alerting, so you can ensure your file system remains healthy and performant.

Preparing the JuiceFS file system

Before diving into setting up Better Stack for monitoring, you'll need an existing JuiceFS file system that is actively publishing metrics. JuiceFS Community Edition and JuiceFS Enterprise Edition (our cloud service is based on JuiceFS Enterprise Edition) both expose real-time status metrics in Prometheus format, but they do it in slightly different ways.

For the JuiceFS Community Edition, after mounting the file system, JuiceFS automatically exposes metrics via http://localhost:9567/metrics by default on the mounting host where the JuiceFS client is running. You can customize this port using the --metrics option if needed.

On the other hand, for JuiceFS Enterprise Edition & Cloud Service, metrics are exposed through the console via dedicated API endpoints. You'll need to replace VOLUME_NAME with your file system name and API_TOKEN with your API token. In this case, both Prometheus and JSON formats are available for metrics:

Prometheus: https://juicefs.com/api/vol/VOLUME_NAME/metrics?token=API_TOKEN
JSON: https://juicefs.com/api/volume/VOLUME_NAME/status?token=YOUR_TOKEN

A quick but important note: metrics are only generated when the file system is mounted. So before proceeding, ensure your JuiceFS file system is properly mounted and accessible. In this guide, we will use the JuiceFS Cloud Service, as it's the simplest to get started. If you haven't set up JuiceFS yet, please refer to the documentation for detailed instructions. Once you have created the first file system, URLs for the metrics mentioned above would be available under its Monitor tab.

image: JuiceFS file system metrics APIs
caption: JuiceFS file system metrics APIs in the Monitor tab

Setting up a metrics source in Better Stack

With your JuiceFS file system up and running (don't forget to mount the file system to a host machine) and publishing metrics, the next step is to configure Better Stack to start ingesting that data.

First, if you haven't already, register for a Better Stack account. The process is seamless. Using a work email is recommended, and the platform provides clear guidance to help you set up your account and organization.

image: Logging into Better Stack
caption: Logging into Better Stack

Once you're logged in, follow these steps:

In the left-hand navigation panel, head to Telemetry.
Under the Sources section, click Connect source.
Give your telemetry data source a descriptive name, such as "jfs-better-stack" or "juicefs-production", to easily identify it later.

image: Configuring telemetry data source Configuring telemetry data source
caption: Configuring telemetry data source

Now, you'll configure how Better Stack should collect your metrics. In the collector settings:

Under Metrics, choose the Prometheus scrape option and click Connect source.
In the URLs to scrape section, input the JuiceFS metrics endpoint as described above.

image: Configuring URLs to scrape
caption: Configuring URLs to scrape

Note that if you are not using the JuiceFS Cloud Service and your JuiceFS endpoint is behind a firewall, you'll need to allow traffic from Better Stack's scrape servers. The list of IP addresses to add to the allowlist is available in their documentation and from here.

After saving the configuration, Better Stack will begin scraping the endpoint. Your JuiceFS metrics should be received within a few seconds.

Creating a dashboard with AI SRE

With your JuiceFS metrics flowing into Better Stack, it's time to visualize them. You could build a dashboard manually, but Better Stack provides a smarter and more efficient way to do it by using AI SRE.

What is AI SRE?

AI SRE (Site Reliability Engineering) is Better Stack's chat-based site reliability assistant. It's an autonomous AI agent that can read your telemetry data, analyze incidents, build dashboards, and even write code to fix errors. Instead of waiting for humans to manually set up charts and queries, AI SRE can generate comprehensive dashboards for you based on a prompt.

It's notable that AI SRE is a paid feature. If you're on the free plan, you can still create dashboards manually using the drag-and-drop chart builder.

Creating a JuiceFS monitoring dashboard with a single prompt

Once your metrics source is ready, follow these steps:

From the left panel, head to Telemetry and then Metrics.
Click Create dashboard and select the Create with AI option.
In the prompt field, give AI SRE a clear description of what you need. For example: "Create me a dashboard to track ALL JuiceFS metrics, such as latency, data size, etc."
Also make sure to select the metrics Source you created earlier (for example, "jfs-better-stack") so that AI SRE has the proper context and data to work with.

image: Creating a dashboard with AI SRE
caption: Creating a dashboard with AI SRE

Give the platform a few minutes for the dashboard to be created. AI SRE will analyze your JuiceFS metrics and automatically generate a complete set of charts and panels for the important performance indicators such as throughput, IOPS, latency, and storage utilization. For my first time trying this, it just worked like a charm as shown below.

image: AI SRE
caption: Charts and panels generated by AI SRE

AI SRE is a powerful feature that does so much more than create dashboards. It can analyze incidents, perform root cause analysis, suggest fixes, and even open pull requests. We've only scratched the surface in this post. This is your first step toward a smarter, AI-assisted observability workflow. After building your dashboard, you can further customize it by adding panels, editing queries, or setting alerts directly from the graphs.

Conclusion

In this post, we have walked through how to build a complete observability system for JuiceFS with Better Stack. We started by setting up the JuiceFS file system and getting its Prometheus-formatted metrics, then created a metrics source in Better Stack to ingest the data. We examined rapid creation of a full dashboard with AI SRE.

We hope this guide helps you gain better visibility into your JuiceFS deployment. If you have any questions or run into issues, we'd love to hear from you. Join the JuiceFS community on GitHub or Discord. And don't forget to check out Better Stack's documentation and their amazing YouTube channel for practical insights about distributed file storage, observability, AI, and more.

JuiceFS 1.4: Faster Metadata Operations with Batch Unlink, Batch Clone, and Redis Client-Side Caching

Thu, 18 Jun 2026 06:10:00 +0000

In large-scale file access scenarios such as AI training and dataset management, metadata often becomes the first performance bottleneck as file counts and concurrency grow. Whether you're deleting millions of small files, cloning large datasets, or traversing directories under heavy concurrency, metadata performance directly impacts application efficiency.

JuiceFS Community Edition 1.4 introduces three major metadata optimizations:

Batch unlink for large-scale file deletion
Batch clone for metadata cloning
Redis client-side caching for hot metadata reads

These improvements reduce transaction commits, network round trips, and redundant metadata lookups. In tests on a flat directory containing 100,000 files, batch unlink improved performance by up to 93×, while batch clone achieved up to 24× speedup.

In this article, we’ll explain the motivation, design, and performance benefits behind these optimizations.

Deletion: From one‑by‑one to batched transactions

Under JuiceFS' metadata-data separation architecture, deleting a file involves much more than removing a directory entry. The system must also:

Update inode reference counts
Reclaim inode and space resources
Process trash entries
Update quota statistics

These operations must typically be completed within the same transaction.

When a directory contains hundreds of thousands or even millions of files, the traditional file-by-file deletion approach used by rm -rf quickly becomes a bottleneck. Each unlink request goes through the FUSE protocol, switches between kernel and user space, and triggers a separate metadata transaction.

As the number of files grows, the overhead from system calls, context switches, network round trips, and transaction commits accumulates rapidly.

To mitigate this issue, JuiceFS previously introduced the juicefs rmr command. Unlike rm -rf, rmr bypasses the FUSE layer and sends deletion requests directly to the client. It also supports multi-threaded deletion (50 threads by default), significantly improving throughput.

However, each file deletion still requires its own metadata transaction. Deleting 100,000 files still means executing 100,000 transactions.

Batch unlink takes optimization one step further by merging many independent deletion operations within the same directory into a single batch transaction, further removing network overhead.

Core design

The key is to turn many small transactions into fewer large ones. JuiceFS adds a batch unlink interface at the metadata engine layer. It allows the client to delete multiple non‑directory files under the same directory in one call.

When recursively clearing a directory, JuiceFS reduces deletion overhead in two ways:

Different subdirectories are handled concurrently with multi‑threaded deletion.
Inside each directory, normal files and symlinks are grouped into batches and sent to BatchUnlink.

This merges many unlink operations into fewer batch transactions at the metadata level.
It's important to note that BatchUnlink does not directly delete directories. Directory removal still follows the standard recursive workflow: empty the subdirectory first, and then delete the subdirectory itself. Therefore, BatchUnlink only applies to regular files and symbolic links within the same directory.

This restriction preserves correct recursive deletion semantics while avoiding consistency risks to the directory tree structure.

image: JuiceFS delete operations
caption: Optimization path of JuiceFS delete operations

Implementation across metadata engines

JuiceFS uses different batching strategies depending on the metadata backend to minimize transaction commits and network round trips.

SQL backends (MySQL, PostgreSQL, etc.): Previously, each file deletion required its own sequence of INSERT, DELETE, and UPDATE statements. With BatchUnlink, the system:

Fetches all edge records for the target entries in a single batch query.
Retrieves the relevant inode attributes in a single locked batch query.
Executes edge deletions, inode state updates (decrementing nlink or marking for cleanup), and delfile entry insertions — all within one transaction.

Instead of executing one transaction per file, the entire batch can now be completed in a single transaction.

Redis backend: The optimization uses Redis pipelines and transactions. Where individual deletions previously required separate command round trips, BatchUnlink collects all HDEL (dentry removal), ZADD (enqueue for cleanup), SET (inode attribute update), and INCRBY (counter update) commands for multiple files into a single pipeline, executed atomically within one MULTI/EXEC transaction. To avoid blocking Redis' single-threaded event loop for too long, batch size is capped at 250 entries.

TiKV backend: BatchUnlink consolidates multiple deletions into a single transaction, using TiKV's batch write capability to reduce network round trips and transaction overhead. For distributed key-value backends, this kind of batching allows the backend's concurrent write capacity to be more fully utilized.

The figure below shows benchmark results on a flat directory of 100,000 files using juicefs rmr --threads 16. BatchUnlink delivers meaningful improvements across all metadata backends, with TiKV and Redis showing the largest gains.

image: deletion
caption: Throughput improvement for batch deletion across metadata backends

Clone: From one‑by‑one copy to batched references

juicefs clone creates fast copies of files or directories for training dataset version management, experiment snapshots, and large-scale directory duplication. Its efficiency comes from the fact that cloning doesn't immediately copy the underlying data blocks. Instead, it creates new file records at the metadata layer and reuses the source file's existing block references. New data blocks are only allocated when the clone is actually written to. This avoids the time and storage overhead of a full copy.

For large directory clones, the same problem as deletion arises: processing files one by one generates a large number of short transactions and network round trips. The core idea behind batch clone is to merge the clone operations for multiple files in the same directory into a single batch transaction. When recursively cloning a directory, the system reads directory entries in batches as a stream. For each batch, all non-directory entries are collected and cloned together in one operation.

One key implementation detail is inode pre-allocation: before entering the transaction, the system uses nextInode to pre-allocate target inodes for all entries to be cloned. This avoids lock contention from repeatedly requesting inodes inside the transaction. Once inside the transaction, the system batch-queries all source file attributes (with row locks), builds all the insertion data for target nodes, edges, chunks, symlinks, and xattrs, and then inserts everything in a single batch.

Batch clone uses each backend's native batch write capabilities in a similar way to batch unlink. The per-backend implementation details won't be repeated here.

The performance gains vary across backends depending on:

Transaction models
Network communication overhead
Batch insertion efficiency for metadata records such as nodes, edges, and chunk references

Results on a flat directory of 100,000 files are shown below. MySQL sees the largest improvement at approximately 24x; Redis at approximately 5x; TiKV at approximately 2x.

image: clone
caption: Throughput improvement for batch clone across metadata backends

Redis client-side caching: Keeping hot metadata local

In high-concurrency metadata workloads such as AI training dataset access and large-scale container startup, network round trips between JuiceFS clients and Redis often become a major performance bottleneck.

Consider the following operation:

open("/mnt/jfs/dataset/images/cat.jpg")

Before the file can be opened, the Linux Virtual File System (VFS) must resolve every component in the path:

Look up dataset.
Look up images.
Look up cat.jpg.

image: VFS
caption: VFS path resolution

If the images directory contains hundreds of thousands of files and training jobs perform random access across the dataset, each lookup requires a GET request to Redis.
Under heavy concurrency, this results in large numbers of network round trips and increased Redis CPU utilization. Even though a single Redis query takes only a few dozen microseconds, network latency pushes each lookup to hundreds of microseconds or even milliseconds. When thousands of training processes are accessing files simultaneously, this overhead becomes significant.

How it works: Redis 6.0 client-side caching

Redis 6.0 introduced client-side caching, which allows clients to cache hot keys locally and receive invalidation notifications whenever those keys are modified.

Based on this capability, JuiceFS caches two categories of metadata in client memory:

Inode attribute cache. Keyed by inode number, this stores the complete attribute data for a file, such as type, size, permissions, and timestamps. The caching is implemented transparently through hook mechanisms in the Redis driver layer. On query, it first checks the local cache; on hit, it returns immediately without any network request. On modification, it automatically invalidates the corresponding cache. Application logic requires no awareness of the cache.
Directory entry cache. Keyed by "parent inode + path separator + filename," this caches the results of directory lookups. Unlike the inode attribute cache, the lookup logic for entry cache is embedded directly in the directory lookup path rather than being intercepted transparently at the driver layer. When entries for a directory are invalidated, all related cache entries under that directory are cleared using prefix matching. This allows path resolution and repeated access to hot entries in the same directory to be served from local memory.

Introducing client-side caching creates a consistency challenge in multi-mount scenarios. When multiple clients share the same JuiceFS file system, an operation on one client — creating, deleting, renaming, or updating attributes of a file or directory — can invalidate cached inode attributes or directory entries on other clients. Without an effective invalidation mechanism, subsequent reads could hit stale metadata, causing the directory entries or file attributes seen by one client to diverge from the actual state in the backend.

To address this, JuiceFS introduces a Tracking and Broadcast Invalidation (BCAST) model on top of Redis' client-side caching mechanism. After connecting to Redis, each client declares the metadata key prefixes it wants to track. When those keys are modified, Redis sends invalidation notifications to the relevant clients. On receiving a notification, the client clears the corresponding inode attribute cache or entry cache entries, so that subsequent accesses fetch fresh data from the metadata engine.

In addition, at client initialization, JuiceFS warms up metadata for the root directory of the mount point. Since these files are typically the most frequently accessed, benchmarks show this warm-up significantly improves overall access performance.

Through this mechanism, hot metadata can be reused locally. When the metadata changes, the related caches are evicted in time, reducing the risk of stale metadata.

When to use it

Redis client‑side caching works best in read‑heavy, write‑light scenarios with repeated access to hot metadata. AI training dataset loading is a good example: the dataset is usually read‑only during training, and tasks repeatedly access the same directories and files, so inode attribute cache and entry cache hit often, reducing redundant lookups and remote metadata queries.

The benefit is even more obvious when there is higher network latency between the client and the Redis metadata engine, such as in cross-availability-zone deployments.

Redis 6.0 or later is required to use this feature. The default cache expiration time is 1 minute, which provides a safety net in case of network interruptions or connection anomalies where invalidation notifications may not arrive, preventing stale entries from persisting indefinitely. For workloads with stricter consistency requirements, the expiration time can be shortened or client-side caching can be disabled entirely to reduce the risk of reading stale metadata.

Summary

These three optimizations each target a different path through the metadata layer:

Batch unlink merges multiple independent unlink operations within the same directory into a single batch transaction.
Batch clone merges multiple independent clone operations within the same directory into a single batch transaction.
Redis client-side caching keeps hot metadata in client memory, bringing read latency from network-level down to memory-level, with broadcast invalidation to maintain consistency across multiple clients.

BatchUnlink and BatchClone are internal interfaces. Users do not call them directly. Just use the right commands: juicefs rmr for deleting large directories, juicefs clone for copying directories. The optimization will be applied automatically.

One thing worth noting: both batch operations work by merging regular files within the same directory into a single batch transaction. Subdirectories are handled recursively by concurrent goroutines. The larger the directory, the greater the benefit.

Batch operations mainly merge ordinary files under the same directory into one batch transaction. Subdirectories are handled recursively by concurrent goroutines. The larger the directory, the bigger the benefit.

All optimizations above are available in JuiceFS Community Edition 1.4. Upgrade the client to get the performance gains.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

How Gongjiyun Keeps Model Distribution Fast Enough for Cross-Cloud Elastic Inference

Wed, 10 Jun 2026 09:48:58 +0000

Founded in 2023 at Tsinghua University, Gongjiyun provides compute platforms and Model as a Service (MaaS) for artificial intelligence generated content (AIGC) enterprises and research institutions. We aim to alleviate the mismatch between elastic compute demand and supply. By aggregating idle IDC resources and edge resources, the platform offers containerized services, delivering rapidly schedulable compute for volatile workloads such as AI inference, video rendering, data processing, and data synthesis.

In cross-cloud elastic inference scenarios, compute tasks can be scheduled to different regions, cloud environments, and clusters, but model files and application data are large and cannot be migrated as quickly as compute resources. Especially in online inference, the model repository is read‑heavy and frequently accessed – storage access performance directly affects service startup, elastic scaling, and request latency.

To address this, we built an object storage acceleration solution on top of JuiceFS, integrating users’ existing object storage into elastic inference clusters. Through a unified namespace, metadata import, FUSE mount, distributed cache, and data warm-up, it improves access efficiency for model repositories across clouds and clusters. In a case study with a leading text‑to‑image model community, the solution supports a tens‑of‑TB model repository, dynamic loading of checkpoints and low-rank adaptations (LoRAs), and elastic scaling of hundreds of GPUs at peak, while keeping additional latency within the customer’s acceptance range.

In this post, we'll walk through why storage — not compute — is the real bottleneck in cross-cloud elastic inference, how we evaluated and chose JuiceFS, and the step-by-step optimizations that brought latency from +10s down to under 2s in production.

Elastic demand is widespread, but supply is hard to match

As AI applications grow rapidly, compute demand continues to increase, but resource usage patterns differ across scenarios. Compared to training, which has stable resource needs, AI inference, data processing, and data synthesis are often more volatile: office applications may see higher traffic during the day, entertainment apps during evenings or weekends, and project‑based data processing may consume large amounts of compute in short bursts then idle. For small teams or exploratory applications, elastic compute also helps them better evaluate the relationship between per‑request cost and application value.

On the supply side, compute infrastructure is capital‑intensive. Resource providers are not incapable of offering elastic services, but they prefer long‑term dedicated leases to recover costs and reduce risk. As a result, low price, stability, and elasticity are difficult to achieve together:

Dedicated leases are low‑cost and stable but lack elasticity.
Spot resources are cheap and elastic but uncertain.
On‑demand resources are elastic and stable but expensive.

In China, this contradiction is further reflected by a market dominated by dedicated leases, with elastic supply accounting for a small share.

We aim to resolve this mismatch between elastic demand and supply. By aggregating idle IDC and edge resources, the platform offers containerized services, providing rapidly schedulable compute for AI inference, video rendering, data processing, and data synthesis. At lower resource costs, we help users quickly spin up tasks during peaks, schedule them across clusters, and handle elastic demand, while enabling resource providers to improve utilization and monetize idle capacity beyond dedicated leases.

Compute can be scheduled: How does storage keep up?

As elastic compute platforms evolve, compute resource scheduling is easy. Container images can be synchronized across clusters via registries and distribution networks, tasks can be launched in different resource pools by schedulers, and traffic can be distributed via unified ingress and traffic management.

But model and data files are typically large, making cross‑cloud, cross‑cluster migration costly and slow, unable to match the sub‑second startup and release of compute. Therefore, in cross‑cloud elastic inference architectures, the real limitation on system elasticity is often not compute scheduling, but the efficiency of data and model distribution.

Different application scenarios have different storage requirements:

Model training, development, and debugging: These involve complex read‑write needs, including code repositories, model files, experiment results, and intermediate state. They also require high environment stability; users cannot tolerate state loss from frequent host switching. Thus, the platform typically provides long‑term stable compute resources and runtime environments, and storage needs can be met by existing stable storage systems.
Data processing: This can be split further. If a single processing job has high application value and can cover cross‑cloud network transfer costs, you can build a pipeline that continuously pulls data from S3 or other object storage, processes it in the compute cluster, and writes back streaming. The system does not need large local storage. If the data scale is larger or per‑job value is low, local storage acts as a one‑time cache. Data flows through and does not need to be persisted.

What is truly more challenging is the online inference scenario. Online inference services cannot tolerate downtime. However, the resources used by an elastic computing platform may come from idle resource pools. These resources could be preempted. Once resources in a certain data center or cluster become unavailable, the platform must be able to migrate tasks to other providers or other clusters in time. This means not only computing tasks must be migrated. Model files and related storage access capabilities must also be migrated at the same time

Online inference has higher requirements for service continuity and cross-cluster migration capabilities, but its storage access pattern is also more clear. Compared to training, development, and debugging scenarios, inference workloads are typically read heavy. The core needs focus on efficient model loading, reading model weights, and accessing the model repository. For large models and online applications, model loading speed directly affects service startup time, elastic scaling efficiency, and request response stability. Therefore, inference scenarios are not suitable for simply adopting traditional read-write hybrid storage architectures. Instead, they are better suited for specialized optimizations around model distribution, read only access, and cache acceleration.

In addition, an elastic computing platform usually does not host a user's complete application system. The user's primary cloud account, application database, model management system, and even some fixed computing resources often already exist in other clouds or on premises. For the platform to integrate with the user's application, it must be compatible with the user's existing model repository and model management processes. It cannot require the user to fully migrate the entire system.

Therefore, to support cross-cloud elastic inference, we need more than just compute scheduling capabilities. We need a cross-cloud high-performance storage and model distribution solution tailored for model inference scenarios. This solution must support hosting a large model repository and high-performance reading, it must adapt to the user's existing model management system. And it must provide stable data access capabilities when resources are migrated across clouds and clusters.

Why JuiceFS: Unified cross-cloud access, strongly consistent metadata, and high-performance cache

Facing cross-cloud elastic inference scenarios, the storage system needs to meet several conditions at the same time:

It must provide a unified access point across different clouds and clusters. It must support shared read-write access and unified metadata management.
It must be compatible with the user's existing object storage and model repository to avoid data migration.
It needs low operational complexity and good read performance.

When evaluating storage options, we considered Ceph:

Ceph is mature. It’s suitable for building unified storage within a single data center or a stable resource domain.
However, in cross cloud elastic inference scenarios, Ceph requires high network stability and operational skills. The overall integration cost is higher. So we did not choose it.

We also evaluated Alluxio. However, in a multi-cloud environment, multiple clusters need to access the same underlying object storage data concurrently. The workload is not purely read only; there are also occasional writes. This scenario requires strong data consistency. Therefore, Alluxio was not chosen for production.

We finally chose JuiceFS mainly because:

It uses object storage as the database.
It provides a unified namespace and consistent file system view through an independent metadata service. This allows multiple clusters to access the same model data as a file system.
This architecture is suitable for cross-cloud and cross-cluster model distribution and shared reading.
It’s also compatible with the user's existing object storage and model repository, reducing data migration and application integration costs.

The decision to further adopt JuiceFS Enterprise Edition was mainly due to its distributed caching capabilities and managed metadata service. In this scenario, the value of JuiceFS is not just providing a file system interface. It combines object storage, unified namespace, metadata management, and cache acceleration into a storage access layer that is better suited for cross-cloud elastic inference.

image: JuiceFS 架构图（第四版）-第 2 页-winfsp (2)
caption: JuiceFS Enterprise Edition architecture

Practical: Object storage acceleration based on JuiceFS

Based on JuiceFS, the platform encapsulates an object storage acceleration product. This product connects the user's existing object storage to the elastic inference cluster. It provides the storage as a high-performance file system for the application. The overall process is as follows.

Create a file system. The user provides object storage access credentials, for example, AK/SK for S3-compatible storage. The credential permissions can be configured as read only or read-write based on application needs. The platform creates a corresponding JuiceFS file system based on that object storage.
Import metadata. The platform uses the JuiceFS import feature to scan the metadata of files in object storage. Then, it imports that metadata into the JuiceFS metadata service. In this way, the model files originally stored by the user in object storage can be accessed as file system directories in JuiceFS.
Create a cache group. Within each cluster that may host workloads, the platform sets up a JuiceFS cache group. This forms a distributed cache group. Before running a task, the platform can warm-up model files. It caches hot data in the target cluster in advance. This reduces the time needed to pull data from remote object storage when the inference service starts.
Mount to application Pods. When the user's application runs, the platform uses the FUSE client to mount the JuiceFS file system into the application Pod. For the application, model files appear as local file system paths. Therefore, the original model reading logic usually does not need modification.
Enable node local cache. Besides the cluster level cache group, the node where the FUSE client runs can also provide local cache. This improves repeated read and model loading performance. It further reduces direct access to remote object storage.

This object storage acceleration product essentially productizes the JuiceFS metadata import, distributed cache, data warm-up, and FUSE mounting process. It allows the user's existing object storage to serve cross-cloud inference tasks in a way that feels closer to a local file system.

In addition, the JuiceFS cache group is independent from the file system access point. This characteristic, on one hand, adds management complexity on the platform side, because the platform needs to manage the relationships among the file system, cache groups, mount points, and task scheduling. On the other hand, it provides a foundation for cache isolation, independent scheduling, and fine-grained management based on clusters, users, or application scenarios in the future.

Production case study: A leading text-to-image model community

Scenario, challenges, and acceptance criteria

One of the most representative cases in this object storage acceleration solution involves a leading Chinese text-to-image model community hosting tens of terabytes of model data, including large checkpoint base models and a larger number of smaller LoRA models. In practice, inference jobs typically load a checkpoint first, then load one or more LoRA models to perform combined inference.

The company already operated compute infrastructure at scale — several thousand GPUs — but its workload, serving creative design and production use cases, exhibited significant variability. Overall average utilization was below 50%, yet during morning and afternoon peak hours on weekdays, load could reach 140% of normal capacity, degrading the user experience. The customer therefore needed a highly elastic compute supply.

We provided a high-elasticity resource model: compute support at the scale of hundreds of GPUs was available only during weekday peak hours — 10:00–12:00 AM and 2:00–6:00 PM — with resources scaling to zero at all other times.

This meant the platform needed to provision hundreds of GPUs within a window of minutes, while consuming zero resources outside peak hours. For the customer, this model delivers large-scale compute during peak periods while avoiding payment for idle capacity. For the platform, it enables more efficient utilization and monetization of idle compute resources.

The technical challenges were significant:

A model repository of this scale cannot simply be replicated to every elastic cluster.
Inference services do not load all models once at startup. Model reads and switches happen continuously as user requests arrive, resulting in high access frequency. Therefore, the object storage acceleration solution needed to support not just large-scale model repository access, but stable read performance under continuous dynamic loading.

The customer's performance requirements were also strict. During acceptance testing, a portion of production traffic was routed to the elastic cluster. The requirement was that both the median and mean inference latency of the elastic cluster must stay within 2 seconds of the customer's own cluster. Given that individual inference jobs take on the order of tens of seconds, this requirement left virtually no room for additional latency introduced by the storage layer. In the first few rounds of testing, both median and mean inference latency on the elastic cluster exceeded the customer's own cluster by approximately 10 seconds — failing the acceptance criteria.

Performance optimization: Reducing additional latency on the elastic cluster

Optimization began with the median. A high median indicates that a significant proportion of requests are experiencing performance degradation, not just a small number of outliers inflating the tail. JuiceFS monitoring revealed that the cluster's cache hit rate was not reaching the expected level. In the current architecture, a cache miss requires a round trip over the public internet to the customer's object storage on Alibaba Cloud. This significantly increases model loading time and then affects inference request latency.

To solve this, the platform used the isolation capability of the JuiceFS cache group. It assigned dedicated cache nodes to this customer, reserved enough cache space, and warmed up the core model data. After warming up, the access path for core models achieved nearly 100% cache hit rate. This effectively avoided the performance loss from cross public network backfilling.

The second factor affecting the median was metadata access latency. Because the platform uses a unified cross-cluster architecture, the metadata service is accessed over the public internet, for example, via JuiceFS Cloud Service or a deployment on a remote host, and this latency affects overall model read performance.

The platform took two measures to address this issue:

Enabling JuiceFS' open cache to keep metadata in local memory as much as possible. Since this workload is predominantly read-only, caching is an effective way to reduce metadata access overhead.
Tuning the cluster's network rate-limiting policy. While the platform cannot directly control network equipment in edge data centers, it can apply node-level rate limiting to prevent any single node from saturating the available bandwidth, improving overall network stability. After these optimizations, cluster-wide performance improved meaningfully and the median metric gradually reached the customer's requirement.

Once the median met the target, the mean still showed a gap. This indicated that long-tail requests remained, with a small number of requests taking significantly longer than normal and pulling up the overall average. Further analysis traced this to node-level local cache — specifically, the FUSE cache quota. With limited cache capacity, the elastic cluster experienced more frequent cache evictions than the customer's own cluster, causing some requests to reload model data from scratch and increasing mean inference latency. The platform addressed this by increasing the FUSE local cache quota in the production environment, reducing eviction frequency, improving tail latency, and ultimately bringing the mean metric within acceptance. The system passed validation and has been running stably since.

Multi-tenant cache management

After the single-tenant case was validated, the solution entered multi-tenant operation. As different tenants began time-sharing the same elastic nodes, a new issue emerged: cache contention between tenants.

In the elastic resource model, FUSE clients do not actively clear node cache on exit. This is a reasonable design in single-tenant scenarios, where cached data from previous jobs can be reused by subsequent jobs to improve hit rates. However, in multi-tenant scenarios, one tenant's data can occupy node cache for extended periods. This leaves insufficient cache capacity for the next tenant, who is then forced to fall back to object storage, causing a noticeable performance drop.

To address this, we deployed an independent daemon process on each node that performs a global cache garbage collection (GC) pass before the application FUSE client starts. The eviction strategy references the JuiceFS FUSE client implementation, using a 2-random policy to balance collection efficiency and performance overhead. Coordination across nodes is handled via Kubernetes distributed locks: only the client that acquires the lock executes GC, preventing multiple clients from running cache collection simultaneously and creating excessive network and I/O pressure.

This mechanism effectively mitigates the problem of historical jobs occupying cache resources in multi-tenant scenarios, allowing different tenants sharing elastic resources to maintain consistent cache performance.

Conclusion

For elastic compute to reliably serve production traffic, compute scheduling alone is not enough. Model data and hot data must remain stably accessible across clouds and clusters.

Built on JuiceFS, we’ve combined object storage, unified namespace, metadata management, distributed caching, and FUSE mounting into an object storage acceleration solution purpose-built for elastic inference. This is not simply about mounting object storage as a file system. It’s about building a data access layer around the access patterns of model inference: one that supports warm-up, caching, isolation, and management.

This represents Gongjiyun's current progress in elastic compute and cross-cloud storage acceleration. As AI inference scenarios continue to evolve, model distribution, cache management, and multi-cluster data access will continue to surface new engineering challenges. We look forward to exchanging ideas with developers, AI application teams, and infrastructure practitioners, and to exploring more stable and efficient data access solutions for elastic compute environments.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

Reducing Data Storage Costs: A Deep Dive into JuiceFS 1.4 Tiered Storage

Wed, 03 Jun 2026 08:45:00 +0000

JuiceFS Community Edition 1.4 introduces enhanced tiered storage capabilities, allowing users to set object storage classes at the file or directory level. This makes it possible to manage different storage tiers for data under a unified file system interface. In this article, we’ll discuss this feature’s application background, evolution, usage model, implementation, and future plans.

Application background

In real‑world scenarios, different files have different access patterns and performance requirements. Some data is read or written frequently and demands low latency and high throughput. Other data is rarely accessed after being written, and the main concern is long‑term storage cost. Tiered storage addresses this by placing data in the appropriate storage layer based on access patterns, balancing performance and cost.
Typically, data can be classified into three categories:

Hot data: Frequently accessed, requires low latency and high throughput.
Warm (infrequent access) data: Accessed occasionally, but still requires fast retrieval when needed.
Cold (archival) data: Primarily for long‑term retention, very low access frequency, can tolerate some restoration delay in exchange for lower cost.

Object storage already offers tiering capabilities. For example, Amazon S3 provides S3 Standard for frequently accessed data, S3 Standard‑IA for infrequent but still millisecond‑accessible data, and Glacier / Deep Archive for long‑term archiving. These storage classes differ in access latency, minimum storage duration, and pricing.
The table below compares main S3 storage classes:

Storage class	Use case	First byte latency	Minimum storage duration fee
S3 Standard	General-purpose storage for frequently accessed data	Milliseconds	N/A
S3 Standard-IA	Infrequently accessed data requiring millisecond access	Milliseconds	30 days
S3 Glacier Deep Archive	Archiving very rarely accessed data with very low cost	Hours	180 days

For JuiceFS, which is built on top of object storage, the key is to translate these capabilities into file‑system‑level tiering: users set storage tiers for files, directories, or datasets, and JuiceFS maps them to the underlying object storage while handling writes, migrations, and restore operations.

Evolution of JuiceFS tiering capabilities

The evolution of JuiceFS tiering has moved from being “passively unaware of object storage classes” to “actively managing storage tiers at file and directory granularity.”

Before v1.1, JuiceFS did not provide a way to configure storage classes. While users could manually change the storage class of objects at the object storage side, these changes were not recognized or managed by JuiceFS at the file system level. For standard and infrequent‑access classes that support direct access, normal read/write operations usually continued to work. However, if objects were moved to archival storage, access would fail because those objects cannot be read directly.

Starting with v1.1, JuiceFS supports setting the object storage class via --storage-class. For example, you can specify the default storage class for the file system at format time or override the storage class used for data written to a specific mount point during mount. This gave JuiceFS a basic ability to leverage object storage tiering. However, the configuration granularity remained coarse – primarily at the file system default or mount‑point level – and did not allow fine‑grained management per directory, per file, or per dataset.

Version 1.4 further advances tiering capabilities to the file and directory level. You can assign a storage tier to individual files or directories based on data temperature. When a directory is assigned a tier, newly created files and subdirectories under it automatically inherit that configuration. Compared to the previous default or mount‑point level settings, v1.4 is better suited for tiered management by project, directory, dataset, or data temperature.

How to configure tiered storage

The key to tiered storage in JuiceFS 1.4 is translating object storage classes into file‑system‑manageable tiers. The usage model consists of two steps:

Map tier IDs to object storage classes.
Assign files or directories to those tier IDs.

This allows users to organise tiering policies by file, directory, or dataset without specifying the underlying storage class on every write.

The figure below shows mapping tier IDs to storage classes:

image: Mapping tier IDs to storage classes
caption: Mapping tier IDs to storage classes

For example, map tier IDs 1–3 to different storage classes:

juicefs config redis://localhost --tier-id 1 --tier-sc STANDARD_IA -y  
juicefs config redis://localhost --tier-id 2 --tier-sc INTELLIGENT_TIERING -y  
juicefs config redis://localhost --tier-id 3 --tier-sc GLACIER_IR -y

After mapping, set the storage tier for a file or directory:

juicefs tier set redis://localhost --id 1 /path/to/file  
juicefs tier set redis://localhost --id 2 /path/to/dir

Directory‑level settings have inheritance semantics. Once a directory is assigned a tier ID, newly created files and subdirectories will inherit that tier. To apply the tier to existing data under the directory, use -r to recursively set the tier:

juicefs tier set redis://localhost --id 2 /path/to/dir -r

image: existing data
caption: Directory inheritance and recursive setting for existing data

For archival storage classes such as Glacier, a restore request must be issued before reading:

juicefs tier restore redis://localhost /path/to/dir -r

Implementation

From an implementation perspective, the key to tiered storage in v1.4 is storing tier information in metadata and using the tier ID to decide the object storage behavior during writes, migrations, and reads.

image: Write, migration, and archival read flow
caption: Write, migration, and archival read flow

Metadata design

JuiceFS uses tier-id on files and directories to indicate the storage tier. A value of 0 means the default storage tier; values 1 to 3 correspond to user‑configured object storage classes.

Thus, the storage tier is no longer just an external state at the object storage side, but becomes part of the file system metadata that JuiceFS can understand and manage. When writing new data, migrating existing data, or checking file status, JuiceFS can determine the intended storage class based on this metadata.

Migrating existing data

For existing data, changing the storage tier involves not only updating the metadata tier-id but also changing the actual storage class of the underlying objects. When a directory is set recursively, JuiceFS processes all files and subdirectories under it and uses the object storage’s copy capability to migrate existing objects to the new storage class.

If only the mapping from a tier ID to a storage class is changed, the actual storage class of existing objects is not automatically updated. In that case, you must use tier set --force to explicitly trigger the change. This will rewrite the objects with the new storage class.

Write path

When a new file is written, JuiceFS determines the target storage class based on the file’s own tier-id or, if not set, the inherited tier-id from its parent directory. For directories that already have a storage tier assigned, new data can be written directly to the corresponding storage tier. This avoids the overhead of first writing to the default tier and then migrating later.

Read path

For storage classes that support immediate access (for example, Standard and Standard‑IA), reads are transparent to the application, and JuiceFS simply reads the data from object storage as usual.

For archival classes such as Glacier and Deep Archive, objects cannot be read directly. You must first issue a restore request using juicefs tier restore. This sends a request to the object storage service. Whether and when the objects become readable depends on the cloud provider’s restore mechanism. After restoration completes, applications can retry the read.

Therefore, archival storage is suitable for data that is accessed very infrequently and can tolerate restoration delay. It’s not appropriate for workloads that require online access at any time. When using archival tiers, you must consider storage cost, restoration time, and restoration costs.

Future plans

Reducing operational costs of archival storage

Archival storage classes have low long‑term storage costs, but they often come with complex cost models for writes, restores, early deletion costs, and lifecycle transitions. Writing data directly to archival storage may incur extra costs in scenarios with frequent changes or bulk migrations.

In the future, JuiceFS could combine object storage lifecycle management. Data could first be written to standard storage with specific object tags. Users could then use cloud‑vendor lifecycle rules to automatically and cost‑effectively transition data to infrequent‑access or archival tiers based on those tags. This would preserve JuiceFS’ file‑system‑level tiering capabilities while leveraging native batch transition mechanisms to reduce overhead.

Extending tiering to multi‑bucket, multi‑cloud

Currently, tiered storage works on different storage classes within the same object storage backend. In the future, JuiceFS could extend “tier” to different buckets, different object storage services, or even different cloud providers. Tiering would no longer be limited to a single backend.

For example, hot data could be stored in a local high‑performance MinIO cluster backed by SSDs, while cold or archival data resides in low‑cost cloud archival buckets. Policies could then gradually move data from the hot tier to the cold tier. With such an architecture, JuiceFS could offer cross‑bucket, cross‑cloud, and cross‑media tiered data management under a unified file system namespace.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.

JuiceFS at Xiaomi: Unified Storage for AI, Big Data, and Cloud‑Native Workloads

Wed, 27 May 2026 08:03:25 +0000

Xiaomi is one of the world's leading smartphone companies. Since 2021, its storage team has been building a file storage platform based on JuiceFS, initially providing file storage capabilities for cloud‑native and some application scenarios. After Xiaomi announced its comprehensive AI strategy in 2024, issues with the previous heterogeneous storage system became more evident in areas such as technology selection, data flow, and development/operations. Leveraging multi‑protocol access, elastic scalability, multi‑cloud adaptability, and high performance, the team decided to build a unified file storage foundation centered on JuiceFS to support big data, cloud‑native, and AI workloads.

To achieve this goal, the platform further developed core capabilities, including a capacity layer, a performance layer, and a cache layer. These reduce the complexity of multi‑system access and data movement while balancing large‑scale storage with high‑performance access. Over the past two years, with the rapid growth of generative AI and autonomous driving, the platform has supported typical scenarios such as large‑model training, autonomous driving training, inference acceleration, and big‑data cloud migration. Today, the platform can handle hundreds of billions of files and EB‑scale storage, covering the entire AI storage chain from raw data and training data to model file distribution.

Storage architecture challenges under the AI strategy

Before 2023, Xiaomi, like most companies, had built multiple storage systems for different application scenarios. In the big data area, the data platform was mainly based on HDFS; AI workloads, before the rise of large language models, relied primarily on high‑performance file storage services on the cloud, such as Parallel File System (PFS) and Network Attached Storage (NAS).

During this period, we also began to introduce JuiceFS and built an internal self‑developed File Storage Service (FDS), using components like JuiceFS CSI Driver to provide file storage for cloud‑native and some application scenarios. As application needs evolved, these storage systems grew independently. This led to a complex heterogeneous storage landscape.

In 2024, after Xiaomi announced its comprehensive AI strategy, the shortcomings of the previous storage system became more pronounced in areas such as technology selection, access, data flow, and development/operations.

These challenges included:

High selection and access costs: With many storage systems and inconsistent capabilities, application teams had to understand and adapt to each one, raising the barrier to entry.
Low data flow efficiency: The lack of a unified access method across systems led to frequent cross‑system data copying. This hurt development efficiency.
Scattered development and operations efforts: Multiple systems were maintained and evolved independently, making it difficult to focus resources on the mission-critical infrastructure required for the AI strategy.

To address these issues, we conducted in‑depth internal discussions and architectural adjustments in 2024, and began redesigning a unified storage architecture for AI, big data, and cloud‑native scenarios.

Building a unified file foundation with JuiceFS

Selection rationale: Multi‑protocol support, elasticity, multi‑cloud, high Performance

JuiceFS is a distributed file system that natively supports multi‑protocol access, elastic scaling, and high‑performance reads/writes. This makes it a perfect fit for both native AI and big data storage needs.

image: JuiceFS 架构图（第四版）-第 2 页-winfsp (2)
caption: JuiceFS Community Edition architecture

In the cloud-native field, we’ve been using JuiceFS since 2021, continuously conducting internal development and iterative optimization. At the same time, we maintain close collaboration with the JuiceFS open-source community to jointly drive technology evolution and real-world adoption.

In AI scenarios, model training and inference rely heavily on POSIX semantics, which aligns naturally with JuiceFS capabilities. Meanwhile, in the big data area, we were already promoting HDFS replacement during cloud migration, a practice with many mature industry examples, so adapting the HDFS protocol was also feasible.

Considering multi-protocol support, elastic scalability, multi-cloud adaptability, and high-performance read/write, we ultimately chose JuiceFS as the core component of our unified file storage foundation. This solved the problems of complex data flow, high access costs, and scattered operations caused by using different file systems across multiple platforms and application units.

image: xiaomi Evolution
caption: Evolution of the unified file storage foundation based on JuiceFS

Storage layer capability construction

Our core goal is to build a unified file storage layer on top of JuiceFS, providing large capacity, high performance, and standardized access interfaces to uniformly support the three core application scenarios: big data, cloud-native, and AI.

On the client side, we fully leverage JuiceFS’ multi-protocol capabilities, offering access methods including POSIX, Hadoop SDK, Python SDK, and S3 Gateway. They’re all already in use internally.

image: JuiceFS unified file storage
caption: Architecture of the JuiceFS unified file storage foundation

On the data plane, the architecture consists of three layers:

Capacity layer: Built on public cloud object storage, designed for EB‑scale storage, supporting multi-cloud deployments across different strategic data centers and multiple cloud providers.
Performance layer: Large‑scale tuning based on Ceph and all‑flash nodes, designed for AI training and other scenarios with high throughput and low latency requirements.
Cache layer: Given the “write once, read many, seldom modify” characteristic of AI training datasets, we developed a high‑performance distributed cache system based on NVMe and RDMA to reduce repeated read costs and improve training data access efficiency.

On the control plane, we made custom enhancements to the Community Edition:

For metadata, we built a distributed metadata service based on the Raft protocol to integrate with internal infrastructure systems and support multi-system access, improving reliability and scalability.
For backend management, we built a unified management service responsible for data lifecycle management, tiered storage, garbage collection, and warm-up of hot data from the capacity layer to the performance or cache layers.

Through these efforts, JuiceFS has gradually become the unified file storage foundation at Xiaomi, supporting both large‑scale capacity storage and high‑performance access for AI training. The architecture is now running in production and provides the high throughput required for large model training.

Our practices

During the construction of the unified file storage foundation, JuiceFS has gradually covered Xiaomi’s mission-critical application scenarios, including big data, cloud-native, and AI:

In terms of scale, the solution can support EB‑level storage and hundreds of billions of files.
In terms of capability, the coordinated design of the capacity, performance, and cache layers balances large‑scale storage with high performance.

Below we describe two typical scenarios: big data cloud migration and the AI storage pipeline.

Big data cloud migration and unified lakehouse storage

In its early days, our big data system was mainly built on the Hadoop ecosystem, where HDFS used a previous‑generation coupled architecture. Over time, this architecture showed problems such as performance fluctuations, complex operations, and high total cost. In contrast, cloud storage offers significant advantages in elastic scaling, resource utilization, and cost control. Therefore, starting in 2021, we systematically began migrating big data to the cloud.

From cold data to the lakehouse layer

Our big data cloud migration went through three stages:

Cold data migration: We first migrated cold data from HDFS to cloud storage, a process lasting over two years.
Lakehouse layer migration: We self‑developed a unified lakehouse file system, promoting the evolution from coupled to decoupled storage and compute.
Unified storage foundation based on JuiceFS: After selecting JuiceFS, we migrated the entire lakehouse layer to JuiceFS.

Lakehouse construction can leverage Iceberg’s native support for object storage access (like OSS or S3). However, our application spans multiple regions globally using several cloud vendors. Adapting to each vendor individually would incur high access and maintenance costs.

Thus, we chose JuiceFS to uniformly access different cloud storage. Upper‑layer services simply switch the backend storage address via the SDK to adapt to access in different cloud environments, greatly reducing multi‑cloud complexity.

For data migration, our self‑developed data‑factory platform supports transparently switching a table’s underlying storage to the new architecture and gradually migrates existing data to the cloud in the background, with little or no impact on application. Moreover, JuiceFS supports multi-cloud and on‑premises deployment. If future cost or strategic considerations require switching to self‑built storage, data can be smoothly migrated back via JuiceFS. This preserves architectural flexibility.

Hot table cache acceleration for compute efficiency

After data was in the cloud, we further analyzed access patterns of the lakehouse layer. For daily reporting and analysis tasks, computation is usually concentrated on day‑level or week‑level hot data, not requiring frequent full scans. Therefore, the performance focus for the lakehouse layer was not simply improving full‑scan throughput but rather increasing hot data access efficiency and task execution stability.

Based on this, we built a hot table warm-up capability in cooperation with the lakehouse layer. The system identifies hot tables and their hot partitions based on daily access statistics, and preloads related data into the cache layer before task execution via a warm-up interface. For periodic reporting tasks that must be completed by 8 AM, hot data is warmed up before computation. This reduces remote reads and repeated access.

Through offline and online testing, after hot table caching, compute efficiency improved by about 10-20%, with reductions in both computation time and resource consumption. The cache size has reached PB level, with average throughput around 200 GB/s. The cache layer also reduces cross‑cloud bandwidth pressure and cloud storage API call costs: by improving the hot data hit rate, repeated cross-cloud reads can be reduced, thereby lowering bandwidth consumption and access costs.

image: Hot table warm-up architecture
caption: Hot table warm-up architecture

Benefits for big data

Benefits for our big data application include:

Performance: After switching to JuiceFS, sequential read/write performance improved significantly, more than doubling in some scenarios. Overall task duration decreased by about 10–30%.
Cost: By Xiaomi's internal cost metrics, the unified storage architecture has greatly lowered storage costs – about 70% in China and 90% in overseas regions. The overseas legacy solution, which used HDFS with three replicas on cloud instances and EBS, had a high replication factor and thus higher costs.
Stability and operations: Under the previous mixed architecture, many compute tasks easily consumed node resources, raising node load and affecting storage performance. With the decoupled storage‑compute architecture, compute tasks run on dedicated nodes, task durations are more stable, and scaling and management are more flexible.

AI one‑stop storage

AI storage consists of three stages:

Raw data stage: Storing large volumes of raw data, which undergoes processing (for example, ETL) to produce training datasets, then is fed into high‑performance training environments.
Training stage: Training tasks require high throughput and low latency to reduce I/O wait time and increase GPU utilization. After training, model files are generated for subsequent inference.
Inference stage: Model files must be quickly distributed to specific nodes for rapid startup of inference tasks.

image: AI storage flow
caption: AI storage flow

Previously, data flowed among multiple systems, causing inconvenience for both application teams and internal operations. By adopting JuiceFS uniformly, we can meet diverse needs based on different storage tiers.

Requirements and solutions by stage

AI one-stop storage needs to cover three stages: raw data, training data, and model files. The requirements for capacity, performance, cost, and distribution efficiency differ at each stage. The table below compares the application needs for each stage with previous and current solutions.

Use case	Application requirements	Previous solution	Current solution (JuiceFS)
Raw data	Large capacity, low cost; support high‑concurrency data processing; scale to PB+	Direct use of object storage; HDFS; other low‑cost storage	Capacity‑oriented JuiceFS: multi‑cloud object storage underlying, shielding vendor differences; EB capacity, hundreds of billions of files; millions of concurrent tasks
Training data	High throughput, low latency; reduce I/O wait time; improve GPU utilization	PFS, NAS (good performance but high cost)	Performance‑oriented/cache‑oriented JuiceFS: TB/s throughput, low latency; async checkpoint to reduce I/O wait; cache acceleration
Model files	Fast distribution; efficient loading; quick inference startup	P2P distribution; workflow distribution; PFS	Cache‑accelerated JuiceFS: cache improves model loading; up to 16 GB/s sequential load per node; several times faster than local disk or FDS

High‑performance cache acceleration: improving efficiency and cutting costs

In AI training, training datasets typically have the characteristics of "write once, read many times, and modify very little." This is a typical read-heavy, write-light access pattern, making it suitable for improving data access efficiency through caching.

Take our internal autonomous driving training as an example. Once a dataset version matures, its data volume may continue to grow within the version cycle, but existing data is rarely modified. While the previous high‑performance file storage met training performance requirements, it had some performance redundancy and cost waste for such repetitive reads. Therefore, we began promoting a high‑performance cache acceleration solution based on JuiceFS.

The cache solution offers several advantages:

Short I/O path: Clients operate on files directly, greatly shortening the I/O path for fast responses.
Performance optimization: Through RDMA and zero‑copy optimization, performance has significantly improved – throughput more than 20% higher than previous high‑performance storage, with ongoing optimization.
Cost reduction: The previous PFS‑based storage used replication (though some used EC, replication was more common for stability). With the cache solution, single‑copy storage reduces costs by more than 60%.
Resource consolidation: For CPU training, GPU nodes typically have NVMe drives (about 10 TB each), which were previously used in scattered ways with low utilization. Now, we consolidate these NVMe resources into a unified cache pool to accelerate nearby GPU training and data processing tasks.

Future plans

Looking ahead, we’ll focus on three directions:

Continuously improve the stability, performance, and scalability of the unified file storage foundation. As AI application grows rapidly, training, inference, and data processing tasks demand higher throughput, lower latency, and greater reliability. We’ll continue optimizing the underlying architecture and critical paths to enhance service capabilities under large‑scale concurrent access.
Strengthen lifecycle management for massive data. Current data volumes continue to grow, but management across storage tiers, access frequencies, and retention periods can be further optimized. We’ll refine tiered storage, archiving, warm-up, and cleanup strategies based on data temperature, access patterns, and cost models, reducing unit storage cost and improving resource utilization.
Enhance data management and analysis capabilities. On top of the unified file storage foundation, we’ll build data management capabilities for application users, helping them better understand data distribution, access behavior, and resource usage, supporting data management, cost optimization, and application decisions.

We look forward to continuous exchanges with industry peers to explore more technical practices. If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.