JuiceFS Blog

How to Access JuiceFS via S3 and WebDAV Protocols

Thu, 15 Oct 2026 01:11:00 +0000

Different applications access data in different ways: some rely on POSIX file interfaces, some are built on the S3 API, and some users want to browse and edit remote files through a file manager. JuiceFS' multi-protocol access capability is designed precisely to let these tools access the same file system.

In this article, we'll explain the shared initialization path of S3 Gateway and WebDAV, describe their applicable scenarios and common mappings, and dive into their implementation principles and boundaries.

A shared initialization path

JuiceFS consists of a metadata engine and object storage:

The metadata engine manages directories, attributes, locks, and sessions.
Object storage stores the actual file data blocks.

image: 1JuiceFS 架构图.drawio
caption: JuiceFS Community Edition architecture

From an implementation perspective, S3 Gateway and WebDAV first go through the same service initialization path. The service process creates a JuiceFS client based on the metadata address, listen address, and common parameters such as subdir, read-only, cache, limits, access-log, and metrics. Then, it initializes the metadata client, object storage, data block cache, and VFS configuration in sequence, ultimately obtaining an fs.FileSystem.

image: Shared service initialization path of S3 Gateway and WebDAV
caption: Shared service initialization path of S3 Gateway and WebDAV

In other words, the common foundation of the two is not a FUSE mount point, but a native JuiceFS client with sessions, caching, metrics, and background tasks. Before S3 requests and WebDAV requests enter their respective protocol adaptation layers, they already share the same JuiceFS file system and the same file system namespace.

Applicable scenarios and common mappings

The shared initialization path explains how the two connect to the same file system. In actual use, what matters more is choosing the access method: which scenarios are suitable for S3 Gateway, which are suitable for WebDAV, and how common requests map to JuiceFS operations.

Linux applications, training jobs, or workloads requiring full file system semantics should prefer FUSE/CSI.
Applications already built on the S3 SDK, AWS CLI, MinIO Client, or object storage toolchains should prefer S3 Gateway.
Desktop file manager access, DAV/rclone synchronization, Office/apps collaboration, and simple sharing should prefer WebDAV.

Mapping of common S3 Gateway APIs to JuiceFS file system operations:

S3 API	Purpose	Equivalent file system operation
GetObject	Download an object	open/read
PutObject	Upload or overwrite an object	create/write/rename
HeadObject	Query object metadata	stat/getxattr
DeleteObject(s)	Delete one or more objects	unlink/remove
ListObjects V1/V2	List objects by prefix	readdir+filter
CopyObject	Copy an object server-side	copy/read+write
MakeBucket	Create a bucket	mkdir
DeleteBucket	Delete an empty bucket	rmdir
NewMultipartUpload	Create a multipart upload session	create
PutObjectPart	Write a part	write
CompleteMultipart	Merge and commit the object	copy + atomic rename
Tags/Metadata	Read/write tags and metadata	get/setxattr

Mapping of common WebDAV methods to JuiceFS file system operations:

HTTP method	Purpose	Equivalent file system operation
GET	Download a file	read
PUT	Upload or overwrite a file	write/create
DELETE	Delete a file or directory	remove
MKCOL	Create a directory	mkdir
PROPFIND	Query properties or list a directory	stat/readdir
PROPPATCH	Modify custom properties	setxattr
COPY	Copy a file or directory	cp
MOVE	Move or rename	mv/rename
LOCK	Lock resources	File lock
UNLOCK	Unlock resources	Release a file lock
OPTIONS	Query server capabilities	Capability negotiation
HEAD	Query HTTP metadata only	Lightweight stat

S3 Gateway principles: from object semantics to file system operations

S3 Gateway converts S3 protocol requests into JuiceFS file system operations. This section covers four aspects:

The request path
Internal state
Key semantic conversions
Permission coordination

Request path: from the MinIO S3 server to `jfsObjects`

The MinIO S3 server handles the S3 protocol layer, including request routing, signature verification, authentication and authorization, and multipart upload orchestration. After requests pass through the protocol layer, they enter the ObjectLayer implemented by JuiceFS, namely jfsObjects. jfsObjects converts object-level reads, writes, listings, deletions, and attribute access into JuiceFS file system calls.

image: Call relationship among
caption: Call relationship among MinIO S3 server, jfsObjects, and JuiceFS file system

State storage: how the internal namespace carries the Gateway state

During request conversion, the Gateway does not only access user-visible object paths; it also uses an internal namespace to store state needed by the service itself. For example, .sys/tmp is used for temporary writes, .sys/uploads is used for multipart upload state, and .minio.sys is used to store MinIO/IAM-related metadata.

These internal paths are not application data in the ordinary user's understanding, but they are the foundation the Gateway needs to maintain upload processes, part states, and permission configurations.

Semantic conversion: `PutObject`, `Multipart`, `Bucket`, and `Metadata`

The S3 object model and the file system model are not completely the same, so the Gateway needs to perform semantic conversions at several key points.

PutObject must ensure that an object is visible only after the upload is complete. To prevent clients from reading incomplete objects, JuiceFS S3 Gateway writes to a temporary file and saves object attributes such as metadata and tag, commits to the target path via rename after completion. Leveraging the atomicity of file system rename, the Gateway can switch an object write from “uncommitted” to “committed.”

image: PutObject write flow
caption: PutObject write flow: from temporary files to atomic rename commit

Multipart upload needs to merge multiple parts into the final object. JuiceFS S3 Gateway can use file system capabilities such as CopyFileRange to combine already written parts and commit the final file via rename. This mainly adjusts file system metadata relationships rather than moving all data again.

image: Part merging and final commit flow
caption: Part merging and final commit flow of multipart upload

Bucket mapping handles the relationship between the object namespace and file paths. In single-bucket mode, the file system name corresponds to the bucket; in multi-bucket mode, top-level directories map to buckets. Object attributes such as metadata, tag, and ETag are stored through file extended attributes (xattr).

Permission coordination: IAM and multiple instances

S3 Gateway also needs to handle identity and permission information, including root users, regular users, policies, service accounts, and more. These configurations are stored in the JuiceFS file system through internal paths, allowing them to be reused across multiple Gateway instances.

Therefore, multiple Gateway instances can share the same set of IAM configurations; however, each instance caches IAM information, so permission changes are not synchronized instantaneously. In actual deployments, we need to consider the refresh cycle.

WebDAV principles: from file protocol to JuiceFS operations

WebDAV starts from a different point than S3 Gateway. It targets file access scenarios, with common clients including macOS Finder, DAV/rclone, and some Office/apps clients. Since the WebDAV protocol includes concepts such as files, directories, and attributes, it does not need to first handle the model conversion between buckets, objects, and keys on the one hand and paths and files on the other, unlike S3 Gateway.

Request path: from HTTP entry to `webdavFS`

After a WebDAV request enters JuiceFS:

1.It passes through net/http and indexHandler. This layer mainly handles HTTP entry-related capabilities, such as authentication, transport, and conversion of directory access requests.

2.The request enters x/net/webdav.Handler, which handles the WebDAV protocol layer and converts protocol methods into file access requests that webdavFS can process.

image: WebDAV request path
caption: WebDAV request path: from HTTP entry to JuiceFS file system operations

3.After entering webdavFS, the file access requests passed in by the Handler are converted into JuiceFS file system calls such as open, create, read/write, rename, delete, and attribute setting. Since the WebDAV protocol is close to file system semantics, the focus here is not to rebuild an object model, but to handle how request methods, file open flags, and attribute states map onto JuiceFS operations.

State boundaries: dead properties and LOCK

WebDAV adaptation involves not only data reads and writes but also the protocol's own state. Dead properties can be saved as JSON in xattr, thus landing in JuiceFS' file attribute system; however, LOCK state is different—it's currently kept in the service process memory. Single-instance deployments usually have no problem; in multi-instance deployments, different instances cannot automatically synchronize lock state.

These limitations usually do not affect Finder browsing, simple sharing, or DAV/rclone synchronization. However, if an application strongly relies on WebDAV file lock semantics, deployment mode and client behavior need to be verified separately.

S3 Gateway and WebDAV share the same JuiceFS client initialization path and access the same file system namespace. The difference is that S3 Gateway needs to convert object semantics into file system operations, while WebDAV is closer to file access semantics.

Therefore, the choice of access method should ultimately depend on the application itself: whether it organizes data as objects or operates on data as files and directories.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Nydus + JuiceFS: Reducing Container Startup Time for AI Inference from 116s to 1.4s

Fri, 09 Oct 2026 06:56:00 +0000

In large-scale AI inference services, when a cluster scales out, newly added inference instances must go through a series of cold-start steps before they can serve requests: container image preparation, file system mounting, runtime and inference framework initialization, and model weight loading. As image sizes and model weights continue to grow, the time spent on data preparation becomes increasingly prominent. During large-scale scaling events, many instances start simultaneously, further increasing the concurrent pressure on the registry, network, and backend storage.

To address these issues, the industry has developed several optimization strategies, including reducing image size, using P2P distribution to alleviate concurrent pull pressure, and employing on-demand loading to reduce data transfer on the critical startup path.
In this article, we’ll explore a shared‑storage‑based approach: combining Nydus’ on‑demand image loading capabilities with JuiceFS’ distributed storage and caching mechanisms. By sharing storage and reusing cached data, this approach reduces redundant data reads during large-scale scale-outs and shortens container startup time. For AI inference scenarios, container images and model weights can be managed together under a unified storage, caching, and prefetching system, accelerating both types of data while reducing the complexity of building and maintaining two separate data pipelines.

Why OCI image distribution is slow

Traditional Open Container Initiative (OCI) images are typically organized as compressed tar layers. In the image‑pull mode, a node must fully download and extract the required layers before the container can start. In practice, however, only about 6%–20% of files in an image are actually accessed during container startup; the rest of the download and extraction overhead is wasted on non‑critical‑path data.

In large clusters, this “prepare everything first, then start” approach causes three problems:

Long cold‑start times: For GB‑sized images (like AI inference images), full download and extraction significantly delay new instance startup, often taking minutes.
Registry bottleneck: During HPA‑triggered bursts or large‑scale rollouts, hundreds of nodes simultaneously request GB‑sized layers from the registry, easily overwhelming the registry and congesting the network.
Redundant network and storage usage: Without an efficient P2P or distributed sharing mechanism between nodes, the same layers are repeatedly transferred across the cluster, resulting in significant bandwidth consumption and duplicated storage.

Dragonfly+Nydus: what it solves and what it leaves behind

To mitigate registry bandwidth pressure during concurrent image pulls in large clusters, a common solution is to introduce P2P distribution systems like Dragonfly. The core idea is to build a P2P distribution network via a scheduler and node‑side agents, allowing nodes to share image data and reduce egress traffic from the registry.

However, P2P distribution primarily optimizes the data transfer path; it does not change the fundamental OCI mechanism of downloading and extracting layers before starting the container. Nydus, on the other hand, changes the loading approach: it transforms “prepare everything then start” into “fetch lightweight metadata first, and then read actual data on demand,” reducing data transfer on the critical startup path.

Combined, Nydus handles on‑demand loading, while Dragonfly distributes the on‑demand data, reducing both the actual image read volume and the concurrent access pressure on the registry.

However, this combination introduces additional infrastructure and operational complexity. Besides Nydus snapshotter, one must deploy and maintain Dragonfly’s scheduler, distributor, and node‑side components, and handle P2P network connectivity, cache space management, node anomalies, and cache eviction. In large‑scale GPU inference clusters, this means maintaining a separate data distribution and caching system independent of the storage layer.

This leads to an alternative thought: if the cluster already has a mature distributed file system like JuiceFS, could image data distribution and caching be offloaded to the shared storage system, reusing existing storage infrastructure while reducing the cost of building and maintaining extra P2P components?

Architecture redesign: Nydus blobs hosted by JuiceFS

The core idea of the Nydus+JuiceFS solution is to transform image data distribution—originally relying on P2P network scheduling—into shared storage and cache reuse based on a distributed file system. In this architecture, JuiceFS serves as the underlying unified data engine hosting Nydus image blob data, and worker nodes read data through the standard POSIX interface.

image: Nydus +JuiceFS
caption: Nydus +JuiceFS architecture and data flow

After an image is built and converted to Nydus format during the CI/CD phase, it’s split into two parts:

Bootstrap (metadata): Very small (typically a few hundred KB, for example, ~450KB), stored in the Image Registry, used for fast indexing.
Blobs (actual data blocks): Contains all file content (typically several GB, for example, ~1GB), persisted directly to the JuiceFS shared file system after conversion (for example, /var/lib/nydus/blobs).

When a worker node starts a container, containerd triggers nydus-snapshotter, and nydusd mounts the container’s read‑only layer via FUSE. Then, when the application issues actual read requests, nydusd looks up the index in the Bootstrap and reads the corresponding blob data on‑demand through the JuiceFS mount point.

The key change in this path is: the registry only distributes the manifest and the extremely lightweight bootstrap; the massive blob data transfer is entirely handled by JuiceFS’ distributed caching system. This fundamentally eliminates the registry network bottleneck during high‑concurrency pulls.

For AI inference scenarios, this architecture can further unify the storage paths for images and model weights. In addition to Nydus image blobs, model weights can also be stored in JuiceFS, allowing both types of data to reuse the same storage, caching, and prefetching mechanisms. This accelerates data loading while reducing the complexity of building and maintaining two data pipelines.

Comparison basis	OCI full image pull	Dragonfly+Nydus	JuiceFS+Nydus
Core concept	Full layer pull + layer extraction	P2P network offloading + on‑demand image loading	Storage‑layer decoupling + distributed caching + on‑demand loading
Cold start method & time	Full download and extraction required; typically 30s to 10min+	Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds	Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds
Image data transfer path	Registry → worker (full local download & extraction)	Registry / source → Dragonfly P2P network → worker on‑demand read	JuiceFS → worker on‑demand read via POSIX, accelerated by local & distributed caches
Core components	Registry + containerd	Nydus, Manager, Scheduler, Seed Peer, and dfdaemon	JuiceFS + nydus‑snapshotter
Image & model weight synergy	Image pulled from the registry to local node; model weights loaded via a separate path (NAS or object storage); two systems managed independently	Images loaded on‑demand via Nydus, distributed by Dragonfly; model weights still rely on independent external storage	Image blobs and model weights stored together in JuiceFS, sharing the same cache and warm-up mechanisms
Operational complexity	Easy to overwhelm registry bandwidth under high concurrency	Requires extra maintenance of P2P scheduler, node network, and caching system	Can reuse existing JuiceFS storage & cache infrastructure, reducing independent image distribution components
Local disk pressure	High: Stores full image and extracted data	High and unpredictable: Requires cache space for P2P; rapid scaling may cause disk full	Controllable: Unified shared storage pool, uses local NVMe for P2P distributed cache
Network requirements	Standard TCP	Requires widely open P2P ports across nodes; may be invalid in cross‑VPC/NAT environments	Standard LAN – only requires normal high‑bandwidth inter‑node LAN; no intrusive network requirements
Suitable scenarios	General workloads with no cold‑start requirements and small clusters (<20 nodes)	Ultra‑large clusters across data centers / clouds / edge computing environments without unified high‑bandwidth storage; strong ops capabilities of the team	Ultra-large clusters across data centers / clouds / edge computing environments without unified high-bandwidth storage, where JuiceFS is already deployed or can be deployed, aiming for unified image+model management and simplified ops

Runtime mechanism: how Nydus and JuiceFS coordinate

Startup phase: fetching bootstrap (metadata)

When the application starts, nydusd only needs to fetch the lightweight bootstrap file tree from the registry. The bootstrap describes the image file tree and block indexes, but contains no actual file content.

After the container root filesystem (rootfs) is mounted, the subsequent startup process proceeds. In our test scenarios, containers can be ready and pass health checks within hundreds of milliseconds, without transferring the full large image data before startup.

Read phase: on‑demand blob access

When the application process actually issues read operations (such as open() or read()), nydusd captures the I/O requests via the Linux FUSE mechanism and locates the corresponding blob data block and offset using the bootstrap index.

This process is completely transparent to upper‑layer applications. For the underlying data path, the read granularity changes from full image layers to the data blocks corresponding to specific files and offsets. nydusd directly accesses the JuiceFS mount point (for example, /var/lib/nydus/blobs) to read the target block, completely eliminating the runtime fetch‑from‑registry path.

Cache phase: data reuse via JuiceFS cache groups

JuiceFS’ distributed cache groups can reuse already‑cached blob data across the cluster. On a cache hit, data is returned to the kernel at near‑local‑NVMe speed. On a cache miss, only the specified offset block is pulled on‑demand and automatically filled into the cache. In addition to the automatic fill‑on‑miss, blobs can be pre-loaded into the cache to ensure full cache hits during on‑demand reads.

For AI inference scenarios, the same mechanism can also be applied to loading model weight files.

Production test data

In production, we compared the solutions above against a large AI inference image.

Container startup time comparison

The table below shows the test results for container startup time:

Scenario	OCI overlayfs	Nydus+JuiceFS	Comment
Cold pull + run	116 seconds	16 seconds	OCI uses public cloud registry; Nydus+JuiceFS uses JuiceFS + public cloud object storage
Cold pull + run (with distributed cache warm-up)	N/A	~1.4 seconds	Required blobs pre‑loaded in JuiceFS distributed cache
Warm pull + run	~0 seconds	0.22 seconds	OCI: same tag already on node; Nydus+JuiceFS: blobcache local cache

From the test results, we can see:

7x speedup without cache warm-up: When neither local nor distributed cache is warmed up, Nydus+JuiceFS reduces container startup time from 116s to 16s.
~99.9% reduction in data read volume during startup: In the cold pull + run scenario, read volume drops from 11 GB to 11 MB. Traditional OCI requires full download and extraction before startup, while Nydus+JuiceFS only fetches bootstrap metadata and reads blobs actually accessed during startup.
1.4s startup with distributed cache warm-up: When blobs are cached in JuiceFS distributed cache, startup takes 1.4s; with local blobcache warmed up (warm pull + run), startup takes only 0.22s.

On‑demand file read performance

The table below shows test results for: on-demand file read performance:

Scenario	Traditional OCI (local disk direct read)	Nydus+JuiceFS (distributed cache hit)
First‑block read latency (TTFB)	~0.9 milliseconds	~3.2 milliseconds
Small file / scattered configuration loading (IOPS)	Baseline	Close to baseline
Large file sequential read throughput	Local disk limit	GB/s level (JuiceFS NVMe distributed cache)

From the test results, we can see:

Time to first byte (TTFB): On distributed cache hit, first‑block read latency is ~3.2 milliseconds, about 2.3 milliseconds higher than local disk direct read (0.9 milliseconds). The extra latency comes from the FUSE data path and cache access, but remains at millisecond level, generally imperceptible to the application side.
Small file IOPS: Performance is very close to local disk baseline. In scenarios with dense Python dependency and configuration file reads, JuiceFS caching introduces no noticeable IOPS bottleneck.
Large file sequential throughput: On cache hit, Nydus+JuiceFS achieves GB/s sequential read throughput. Compared to traditional OCI, which is limited by single‑node local disk performance, JuiceFS can leverage NVMe distributed cache to provide high‑bandwidth data access.

Summary: combining on‑demand loading with shared cache

Nydus changes image loading from full pull to on‑demand reading, while JuiceFS places data blocks into a shared storage and distributed cache system. Together, they reduce container cold‑start time from 116s to 16s, and to ~1.4s with distributed cache warm-up; the data read volume during startup drops from 11 GB to 11 MB.

The value of this solution extends beyond just image startup acceleration. For AI inference scenarios, image blobs and model weights can be uniformly hosted by JuiceFS, sharing the same caching and prefetching mechanisms. During large‑scale scaling, distributed caching also reduces duplicate reads of the same data across many instances. This lessens concurrent pressure on the registry and backend storage.

In environments lacking shared storage, Dragonfly+Nydus is an excellent P2P alternative. However, in clusters that already have JuiceFS infrastructure, Nydus+JuiceFS delivers extreme image distribution and model loading acceleration with minimal operational overhead.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

JuiceFS Enterprise 5.4: From 100 Billion Files to 1 Million Clients

Tue, 29 Sep 2026 15:07:00 +0000

Following the support for 100-billion-file environments introduced in JuiceFS Enterprise Edition 5.3, v5.4 further improves JuiceFS across several dimensions for large-scale deployments. At this scale, even small per-operation overheads can add up to significant resource consumption. Once metadata is distributed across multiple zones, the system also needs to coordinate across nodes while maintaining performance, consistency, and stability.

To address these challenges, JuiceFS Enterprise Edition 5.4 introduces the following new capabilities and improvements:

Support for up to 1 million simultaneously mounted clients
Fast cloning of directories containing massive files
More than 90% RDMA bandwidth utilization
Up to 100% higher random-read performance under high concurrency
On-demand data synchronization for mirror file systems
More flexible caching strategies for a wider range of workloads
Multiple operational improvements for large-scale data

Supporting 1 million clients

As AI agents move toward large-scale deployment, the JuiceFS team faced a new challenge: supporting connections and concurrent access from up to 1 million clients on top of the metadata workload of a 100-billion-file file system. The team spent months exploring and optimizing the architecture, and introduced this capability in JuiceFS 5.4.

JuiceFS uses a distributed architecture that partitions metadata across multiple metadata nodes, allowing metadata for hundreds of billions of files to be managed across multiple zones. To allow these nodes to handle a large number of client connections at the same time, v5.4 co-locates metadata nodes with network proxies, enabling clients to connect to any metadata node.

The node that receives a request handles requests for its local partition directly. Requests involving other partitions are forwarded internally to the corresponding nodes. Multiple client sessions share the same physical connections between nodes, reducing the number of connections and the associated maintenance overhead.

Connection reuse is only part of the challenge. Querying large numbers of client sessions, reporting session status, and cleaning up expired sessions can also introduce significant overhead.

In v5.4, indexes are used to locate client sessions instead of scanning unrelated sessions. Paginated queries and sampled reporting reduce the overhead of session status management. Expired sessions are cleaned up in batches to prevent cleanup work from becoming concentrated in a single pass.

With these optimizations, JuiceFS 5.4 successfully handled 1 million simultaneous client connections in stress tests. Previously, when clients connected separately to multiple metadata nodes, the total number of connections could reach tens of millions. After optimization, 10 internal proxy nodes handled 1 million client connections, while connection reuse reduced the number of physical backend connections from the proxies to the metadata leader to about 400. This significantly reduced connection-management pressure on the leader.

image: 1-million-client test architecture
caption: 1-million-client test architecture

Fast cloning for large directories

When creating branches of training data or preparing test environments, users often need an independently modifiable copy of a directory. As directory sizes grow, cloning becomes more complex: in a multi-zone architecture, the metadata for a single directory tree may span multiple metadata zones, so cloning must coordinate subtree operations across nodes while preserving the complete directory hierarchy.

v5.4 introduces cross-zone directory cloning. It recursively processes subtrees while preserving the original cross-zone layout of the directory tree. Cloning only copies metadata. The underlying object data remains shared, so there is no need to copy it. Subsequent writes to the clone do not affect the source files.

In a deployment with 30 metadata zones, cloning 100 million files took 1 minute and 40 seconds, averaging 1 million files per second. More zones can provide greater parallelism, while actual performance also depends on metadata load and how the directory tree is distributed across zones.

Note that cross-zone directory cloning is not atomic. If new data continues to be written to the source directory during cloning, different subtrees in the cloned directory may reflect the source directory at different points in time. If strict consistency is required, pause writes to the source directory before starting the clone.

Performance improvements

More than 90% RDMA bandwidth utilization

Starting with JuiceFS 5.3, clients can use RDMA to communicate with distributed cache nodes, reducing CPU overhead during data transfers and improving transfer efficiency. v5.4 further improves bandwidth utilization and transfer stability.

In a test environment with a 400 Gbps RDMA NIC on the client and an 800 Gbps RDMA NIC on the distributed cache node, a single client achieved 45 GB/s of read throughput. A single distributed cache node delivered 90 GB/s of data throughput, with NIC utilization exceeding 90%.

Higher random-read IOPS under high concurrency

In multi-process training, HPC, and rendering scenarios, high-concurrency random reads can stress not only storage and network resources, but also client lock contention and request-processing overhead.

Over the past six months, the JuiceFS team has continuously optimized high-concurrency random-read performance by simplifying frequently executed code paths and reducing lock contention.

As a result, random-read IOPS has more than doubled overall compared with the previous implementation. With distributed caching, a single client reached up to 189,000 IOPS. With local caching, a single client reached up to 156,000 IOPS.

image: Random-read IOPS comparison
caption: Random-read IOPS comparison

Test environment: A dual-socket AMD server with 128 cores and 256 threads, 1 TB of DDR4 memory, a 100 Gbps NIC, and a 3.84 TB NVMe SSD used as the cache disk. The system ran Linux 5.14.

Optimizing data access across regions

On-demand synchronization to reduce data replication

In multi-region, multi-provider, and multi-cluster compute environments, a JuiceFS mirror file system can automatically synchronize files and directories from the source. Metadata is continuously synchronized to keep the directory structure and object versions up to date. Previously, object data was synchronized in full in the background by default.

However, a workload in the mirror region may only need a subset of the models, datasets, or historical directories stored at the source. For example, if the source contains 10 PB of data but the mirror accesses only 1% of it, full synchronization would transfer and store a large amount of data that the workload never uses.

v5.4 supports on-demand synchronization of object data for mirror file systems. When enabled, metadata continues to synchronize, while object data is no longer synchronized automatically in the background. When a client reads data, JuiceFS first checks the object storage in the mirror region. If the data has not been synchronized yet, JuiceFS fetches it from the source and stores it in the mirror on demand. This reduces unnecessary data transfer and storage consumption.

If an application requires local read performance on the first access, you can warm up the required data in advance. If a complete copy of the data is required, you can continue to use background synchronization.

Accelerating metadata access with read-only nodes

For workloads that only need to accelerate metadata access, v5.4 introduces a new mode that does not require creating a mirror file system.

Users can deploy read-only metadata service nodes in other regions to accelerate metadata reads for local clients. Clients continue to mount the source file system, while the acceleration nodes can scale dynamically to match access requirements.

More flexible cache management

Improving hot-data cache hit rates

Data from one-time scans is often unlikely to be accessed again. Caching this data can evict hot data that is accessed repeatedly. By default, when a request misses the distributed cache, the cache node fetches the data from object storage and stores it in the cache. v5.4 introduces a new cache-aside mode. When enabled, a cache miss causes the application node to read directly from object storage instead of filling the cache with the requested data.

Preserving cache hit rates when adding multiple cache nodes

When multiple cache nodes are added at once, read requests may be routed to new nodes that do not yet have the required data, triggering additional reads from object storage.

v5.4 introduces the --commission parameter. After a new node is enabled, it retains the node mapping from before scaling. If the data is not available in its local cache, the new node fetches it from the original node instead. This reduces reads from the source during cache scaling.

Reducing the number of files on large cache disks

On large-capacity cache disks, storing each cache block as a separate file can cause the number of files on the cache disk to grow continuously.

The new merge-cache mode combines multiple cache blocks into larger files and stores their indexes separately. This significantly reduces the number of files that the cache disk needs to manage.

This feature is currently available as a public beta.

Warming up specific byte ranges

If a workload reads only part of a large file, warming up the entire file consumes additional cache space and transfers unnecessary data.

v5.4 allows you to specify one or more byte ranges to warm up. JuiceFS only warms up cache blocks that intersect those ranges, aligning cache preparation with the data the workload actually needs to read.

This capability enables more precise warm-up for data formats such as Lance and Parquet.

Simplifying operations for massive files

Restoring files by deletion time

When restoring files from the trash, users may know approximately when an accidental deletion occurred but have difficulty identifying the target files by name or keyword alone.

The restore command now supports the start-time and end-time parameters. In addition to the existing keyword filters, these parameters allow users to restrict the restore operation by file deletion time.

Reducing GC and `fsck` memory usage

When checking large file systems, GC and fsck need to process large numbers of records, and sorting can consume significant amounts of memory.

v5.4 adds external sorting modes to both commands, using disk storage during sorting to reduce memory requirements. The external sorting mode for GC is available only with dry-run, allowing users to inspect and evaluate the operation without actually deleting data.

Restricting access with token allowlists

v5.4 supports access control through token allowlists. An allowlist can specify multiple first-level subdirectories under the root directory.

Clients using the token can only access and modify content within those subdirectories, allowing different teams or workloads to be restricted to their designated data.

Summary: scale changes everything

JuiceFS 5.3 brought the platform into a new phase of scale. At the same time, the rapid development of AI continues to push the boundaries of data volume and system load.

As file counts, client counts, and request volumes all reach new levels, performance, stability, consistency, and operational complexity are no longer independent concerns. They interact with one another and become increasingly difficult to manage as the system scales.

v5.4 addresses these system-level engineering challenges across large-scale deployments, covering client connectivity, cross-zone directories, data access, mirror synchronization, and cache management. JuiceFS Cloud Service users can now try JuiceFS Enterprise Edition 5.4 online. Users running on-premises deployments can contact the JuiceFS team for upgrade support.

JuiceFS is used across a range of demanding workloads, including LLMs and multimodal models, autonomous driving, embodied AI, and quantitative investment. We’ll continue working with organizations at the forefront of these fields to improve JuiceFS and build infrastructure that can keep pace with their applications.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Infinite Workspaces for AI Agents: A Practical Architecture with JuiceFS, SQLite, and Litestream

Wed, 23 Sep 2026 05:23:00 +0000

Most conversations about AI sandboxes start and end with boot speed and compute isolation: How fast does the container boot? Can the agent run untrusted code safely? But these discussions miss a fundamental truth: an autonomous agent turns a fresh machine into a rambling digital workspace. It clones code repos, installs heavy node_modules or Python virtual environments, downloads browser binaries, compiles artifacts, generates multi-gigabyte datasets, and writes massive trace logs.

"The files are not incidental. They are the working memory of the task."
– Aniket Maurya, Founder @ Celesto AI

The naive solution is to overprovision the root disk. However, determining the size of a boot disk before a task exposes its scope is an ineffective effort. A "quick bug fix" might require a full Next.js build chain, while a "data benchmark" might produce multi-gigabyte Parquet tables. Coupling system state to workspace state creates a bottleneck.

Instead, an emerging architectural pattern is radical in its simplicity: decouple the system disk from the durable workspace. Provide a POSIX-compliant mount that acts like an infinite, persistent drive, backed entirely by affordable object storage. But how do you build such a system without introducing complex services that defeat the purpose of ephemeral sandboxes? Recently, we co-hosted an Office Hours session with Celesto AI, a platform building secure sandboxes for autonomous AI agents, and the webinar was a delightful exploration of exactly this challenge.

Storage	Purpose
Root disk	Operating system, runtime, package manager internals, and system-level state
Workspace	Repositories, generated files, datasets, build artifacts, logs, and project state

Here is the creative engineering stack making this possible: JuiceFS as the file system layer, SQLite as the embedded metadata engine, and Litestream as the durability glue.

The creative trio: JuiceFS + SQLite + Litestream

Object stores (S3, GCS) are excellent for storing files and performing key-based lookups, but they are terrible at listing directories, handling stat calls, or managing millions of small files in batches (i.e., renaming a directory). You need a metadata engine to act as the "brain" of the file system, mapping paths to block locations.

Usually, JuiceFS deployments use centralized or clustered metadata services, such as Redis, TiKV, or FoundationDB. For isolated AI sandboxes, a creative alternative leverages the three familiar JuiceFS components working in harmony:

The JuiceFS client provides the POSIX bridge. It is responsible for the complex task of communicating with both the metadata engine and the object store. It is the "worker" that reads and writes the actual data, as well as performs administrative tasks such as status checks and data compaction.
SQLite acts as the metadata engine. Instead of connecting to a remote metadata server, JuiceFS also supports using a local SQLite database file as its metadata store. This means the entire file system metadata (every inode, directory entry, and block pointer) lives inside a single, standard .db file on the local disk.
Litestream provides the durability backing for that SQLite file. Litestream continuously takes SQLite snapshots and streams the SQLite write-ahead log (WAL) frames to the same object store.

Here is the magic: When the sandbox is running, JuiceFS uses the local SQLite file to resolve ls, cd, and read operations. The actual file blocks are already safe in the object store. (Although, the sync-interval configuration needs to be taken into consideration.) When the user stops the sandbox, the local SQLite file might be discarded, but Litestream has already replicated the metadata engine to the object store.

image: JuiceFS + SQLite + Litestream
caption: JuiceFS + SQLite + Litestream for AI agent workspaces

When the sandbox resumes on a fresh VM (or a different host entirely), the system simply downloads the latest SQLite snapshot from the object store and replays the WAL. Within seconds, the local SQLite can be fully reconstructed. JuiceFS mounts it, points back to the same object store blocks, and the agent is looking at the exact same workspace directory it left behind.

The JuiceFS file system doesn't care which computer it runs on. But it does remember its folder structure and file pointers with the portable SQLite snapshots and WAL frames. With this setup, we turned the working memory of the AI agent into files and blocks in the object store, making it durable and accessible instead of locking it to a specific boot disk or block device.

An unconventional but effective combo

JuiceFS supports interchangeable metadata engines. Choosing SQLite + Litestream subverts the typical playbook by prioritizing simplicity and per-agent isolation over global scale. Here is why this particular stack fits the AI agent use case so well:

Easy to set up: While it is not completely hands-off (you still manage the JuiceFS client, the local SQLite instance, and Litestream's replication configuration like Celesto AI does), this stack is drastically simpler to bootstrap than a distributed database cluster.
Per-agent isolation: Each sandbox gets its own dedicated SQLite metadata file. This eliminates noisy-neighbor problems. Metadata contention simply does not happen across agents, because each one has its own independent "brain" living right next to its mount.
Performance that fits the scope: A single agent workload typically produces tens of thousands to a few million files. SQLite handles this range comfortably with low latency and minimal resource footprint. While this approach obviously does not scale to the hundreds of billions of files that JuiceFS Enterprise Edition can manage globally, it falls squarely into the comfortable zone for individual agent workspaces.

Of course, this approach comes with important trade-offs and considerations:

No cross-agent sharing: The file system is inherently tied to a single SQLite instance. Agents cannot mount and share the same workspace concurrently. If your use case requires collaborative access or shared datasets across sandboxes, you would need a different strategy.
Resume latency depends on object store speed: When a sandbox resumes, the system must download the latest SQLite snapshot from the object store and replay the WAL frames. This adds noticeable startup time compared to a hot-mounted local disk. Tuning Litestream's sync-interval involves a direct trade-off between durability guarantees and resume performance.
Single-writer limitation: SQLite is fundamentally a single-writer database. Heavy parallel writes from multiple processes within the same sandbox can become a bottleneck. For autonomous agents following a largely sequential workflow, this limitation is rarely a practical issue. But it is worth keeping in mind for highly concurrent, multi-threaded agent designs.

The stack is not a universal solution. It is a deliberate, opinionated choice that trades global scalability and shared access for extreme simplicity and tight per-agent isolation. And for the AI sandbox use case, that trade-off often makes perfect sense.

The performance reality

This architecture is beautiful, but it isn't magic. You are trading local SSD latency for object storage durability and infinite scale. The benchmarks reveal a clear pattern:

Small files need attention: Writing 5,000 tiny 4 KiB files to an object-backed file system involves heavy metadata churn. Even with SQLite's fast local access, every close() and fsync() triggers WAL writes and Litestream replication. In these scenarios, the local root disk (ext4/XFS) will outperform the durable mount by a factor of 2x to 5x. See Celesto's blog post for more details.
Large sequential workloads shine: For datasets, model checkpoints, build artifacts, and logs (files > 10 MB), the object-backed mount achieves upwards of 200 MiB/s, which is nearly 90% of the performance of the local disk. The background block uploads are streamed efficiently. See Celesto's blog post for more details.

The golden rule for agents is to use the object-backed mount drive for durable, long-term state reports and large artifacts. Use the local ephemeral disk for high-churn, temporary scratch work that doesn't need to survive a restart. This hybrid approach gives you infinite capacity without sacrificing performance where it truly hurts.

Final thoughts

Cloudflare CFO Thomas Seifert recently predicted that within five years, non-human traffic could be as much as 1,000 times human traffic, making humans "a rounding error on the internet." He caveated that he's "called it wrong at every point along the way," but the direction is hard to dispute: AI and machine traffic already overtook human traffic in May 2026.

image: Cloudflare exec
caption: “Humans will be a rounding error on the internet” says Cloudflare exec

If that is true for network packets, it is likely true for storage as well.

Every autonomous agent session produces a workspace. Multiply that by millions of agents running continuously, resuming, retrying, and branching. Human developers generate storage in bursts tied to working hours. Agents generate it constantly, at machine speed.

This raises an uncomfortable question: if non-human consumers are about to dominate storage demand, are we designing for the right consumer? The combo of JuiceFS + SQLite + Litestream is one answer, but not the only one. Other approaches worth watching include object storage as the primary interface, shared metadata engine with concurrent access at agent scale (i.e., the JuiceFS Enterprise Edition), copy-on-write workspace snapshots, and many others.

Portable, shareable, scalable file storage for AI agents is still in its opening moves.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Keeping GPUs Fed: How Meta's AI Storage Architecture Aligns with JuiceFS

Thu, 17 Sep 2026 03:32:00 +0000

Introduction

Recently, I came across an eye-opening blog post and presentation from Meta's engineering team detailing their AI storage blueprint at scale. Meta operates hundreds of exabyte-scale storage clusters serving everything from Facebook and Instagram to Meta AI.

What caught my attention wasn't just the scale—it was the architecture. After years of wrestling with AI's brutal latency and throughput demands, Meta has converged on a clear pattern: data-metadata separation, rich client design, and comprehensive caching.

Here's the fascinating part: this blueprint isn't unique to Meta. JuiceFS shares these same first principles of design. The hyperscaler's new construction closely resembles the cloud-native architecture that JuiceFS has advocated for years.

In this post, I'll analyze both architectures, compare their core components, dive into their cross-region capabilities, and explain why this convergence in architecture makes the most sense for a distributed file system.

Why storage matters for AI workloads

Before diving into the architectures, it's worth understanding why storage has become such a critical component for AI infrastructure.

Meta puts it succinctly: "If AI is the brain, storage is the memory: capability and speed are highly dependent on the size of memory and speed of retrieval." Yet while AI compute performance has roughly tripled every two years, storage and interconnect performance growth have been far more modest. This disparity has made storage bottlenecks one of the primary contributors to GPU stalls for AI workloads, which is directly impacting both expenditures and time to market.

image: Dataloading across two GPUs
caption: Dataloading across two GPUs (Reference: Meta)

The math is brutal. During model training, hundreds of thousands of GPUs iterate over vast datasets across multiple epochs. GPUs synchronize their state periodically. If a single GPU stalls due to slow storage, it slows down the entire training job. In Meta's presentation, if GPUs were idle for about 20% of the time, "that 20% stall means about tens of millions of dollars loss every hour" for a data center the size of Manhattan.

Beyond the direct dollar cost, there is an even more consequential dimension: opportunity cost. The AI industry is moving at an unprecedented pace. Major model releases, which took roughly 104 weeks between 2020 and 2022, have now been compressed to approximately 4-week cycles by 2026. Every week of delay in shipping a frontier model translates to lost market share, diminished competitive advantage, and missed revenue opportunities.

In short, storage is what makes the difference between a model shipping in weeks versus months and between a billion-dollar data center running at 80% versus 100% efficiency.

Meta's AI storage blueprint: the evolved architecture

After iterating on their storage infrastructure, Meta has arrived at a definitive architectural blueprint to solve the GPU stall problem at its root while continuing to support existing external and internal products. The training stack has been gradually migrating to the BLOB-storage interface. The motivation is more about performance: AI workloads demand predictable, bounded pMax latency. Traditional layered metadata lookups could add hundreds of milliseconds per request, stalling GPUs during training. By moving to a BLOB-centric interface, Meta leverages flash drives to deliver the low, predictable latencies required to keep thousands of GPUs productive.

The three-component architecture

The diagram below illustrates the request flow for Meta's BLOB storage getObject API and shows the system's three major components.

image: BLOB storage getObject request flow
caption: BLOB storage getObject request flow (Reference: Meta)

The Tectonic data storage: At the foundation sits Tectonic, a regional, multi-tenant storage fabric providing high durability and availability through erasure coding. It supports tiering across HDDs and flash drives, intelligently managing hot, cold, and warm data placement.
The BLOB server and metadata layer: Operating on top of Tectonic, the BLOB layer handles API requests, stores metadata, and exposes a global, infinitely scalable storage fabric with policies allowing users to trade off between durability and availability, abstracting away the complexity of the underlying block storage.
The rich client SDK: Meta eliminated the data-plane proxy and built a rich client SDK with a Tectonic BlockClient embedded within it, enabling direct data reading from Tectonic storage servers to clients. The client also implements hedged reads and dynamic concurrency control to reduce tail latencies.

To maximize cache efficiency, Meta also leverages spare memory on GPU hosts as a distributed data cache by integrating peers from the Owl subsystem directly into the BLOB-storage client SDK, which yields an average cache hit rate of 80%.

Cross-region data access with tiered caching

Meta's cross-region approach treats storage as a global data lake, but with tiered caches accelerating access across geographical boundaries.

image: Cross-region data access with multi-tier caching
caption: Cross-region data access with multi-tier caching (Reference: Meta)

Cache Tier	Location	Role
L1 Cache	GPU host memory (RAM)	Hottest, most frequently accessed data
L2 Cache	GPU host flash (SSD)	Local flash tier for warm data
L3 Cache	Regional flash-based BLOB storage	Regional cache before falling back to global BLOB storage

The results speak for themselves: average ingestion time dropped by 93%, and worst-case ingestion time dropped by 97%. This is what it looks like when storage stops being the bottleneck and starts enabling AI research velocity.

JuiceFS: cloud-native, AI-native from day one

For readers already familiar with JuiceFS, the parallels to Meta's design are immediately apparent. The architecture Meta converged upon after years of iteration shares the same core principles that have guided JuiceFS from day one.

JuiceFS was architected from the ground up on the data-metadata separation principle, which also consists of three core components:

Object storage: JuiceFS uses object storage as the reliable, scalable backend for actual data storage. It supports all major public cloud offerings (AWS S3, Alibaba OSS, Google Cloud Storage, etc.) as well as self-hosted solutions like Ceph and MinIO. This layer ensures that AI datasets, which can be as large as petabytes or exabytes, are safe, accessible, and can grow almost without limit.
Metadata engine: The metadata engine stores all file system metadata, including file names, sizes, permissions, directory structures, and the mapping between files and their underlying data blocks. JuiceFS supports multiple engine options, including Redis, MySQL, PostgreSQL, TiKV, and many others. For extremely large-scale deployments, JuiceFS Enterprise Edition offers a proprietary Raft-based distributed metadata engine that provides high availability and strong consistency across multiple nodes.
JuiceFS client: The client handles all file I/O logic, including data slicing, merging, caching, and POSIX semantics enforcement. It communicates with both the metadata engine and object storage directly, coordinating reads and writes intelligently.

Cross-region strategy: mirrors and replication

The JuiceFS Enterprise Edition also provides a native approach to support cross-region and cross-cloud data access:

Mirror file systems: Deploy a full metadata cluster and optionally an object storage replica in a mirror region. Metadata syncs automatically from the source region with approximately 1-second latency under normal network conditions.
Async data replication: JuiceFS supports cross-region and cross-cloud asynchronous replication with a one-to-many topology. Data written in the primary region is replicated to multiple mirror regions in the background, enabling global datasets without manual copying.
Distributed cache: JuiceFS clients on mounting hosts can form a distributed cache group using a consistent hashing ring, sharing cached data blocks with one another. This is particularly effective for model training, where datasets are repeatedly accessed across GPU nodes.
Smart read strategy: Clients preferentially read from the local region. If the data isn't fully synced yet, they seamlessly fall back to the source region.

image: JuiceFS mirror file system architecture
caption: JuiceFS Enterprise Edition mirror file system

This design resonates with Meta's global data lake + tiered cache approach but offers a more flexible, cloud-native model that works across public clouds, private data centers, and hybrid environments.

Architectural similarities

On the surface, Meta mainly exposes a BLOB-style interface, while JuiceFS primarily provides a POSIX file system. Yet beneath these protocol differences, the architectural parallels are noteworthy. Both systems converged on the same three-component design with comprehensive caching and cross-region capabilities:

Dimension	Meta's BLOB-Storage	JuiceFS
Data storage layer	Tectonic: regional block storage fabric with erasure coding, tiering across HDDs and flash	Object storage: S3, OSS, GCS, Ceph, MinIO, and others
Metadata layer	Unified metadata scheme backed by ZippyDB with O(1) chunk lookups	Metadata engine: Redis, MySQL, PostgreSQL, TiKV, JuiceFS Enterprise Edition
Rich client	"Fat client SDK" with embedded Tectonic BlockClient, streaming directly from storage servers	JuiceFS client: handles all I/O logic, slicing, merging, caching; supports FUSE, POSIX, Hadoop SDK, CSI, S3 Gateway
Distributed cache	Tiered caching layers	Memory + local cache + distributed cache group
Cross-region	Tiered caching with global BLOB storage data lakes	Primary region + mirror file systems

A concrete example: explicit cache hydration

One interesting example is how both systems handle proactive cache hydration, which is the practice of loading data into the cache before it's actually needed.

Meta exposes a prefetch() API through the BLOB-storage SDK, while JuiceFS provides a warmup subcommand. Both serve the same purpose: allowing the client side to signal to the storage system exactly which data will be needed in the near future so it can be preloaded into cache before the GPU finishes its current work.

This is especially critical in distributed training, where thousands of GPUs may simultaneously read the same dataset. Without cache hydration, the initial requests would have to read from the underlying storage (Tectonic for BLOB and object store for JuiceFS) with higher latency. With prefetch() or juicefs warmup, data is staged in the distributed cache before the GPUs need it. In essence, both systems recognize the limitations of network-based storage and passive caching, and they both provide explicit prefetching as an important tool to compensate for what the storage layer alone cannot guarantee.

Conclusion

Meta's AI storage blueprint is a testament to the architectural patterns that emerge when a hyperscaler evolves its data infrastructure to hundreds of exabytes. After years of iteration, they arrived at a design that's similar to what JuiceFS has championed from day one: data-metadata separation, a rich client that handles I/O logic directly, tiered caching, and cross-region capabilities.

The same architectural pattern is also available to teams of any size. JuiceFS brought it into the big data era and now brings it into the AI age, with the flexibility of multi-cloud cross-region strategies, distributed caching, and a rich ecosystem of access protocols.

Hopefully, storage is no longer the bottleneck in your AI training stack. At least, it doesn't have to be.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Everything Is a File for Agents: Hello Uses JuiceFS to Build Stateful AI Workspaces

Thu, 10 Sep 2026 06:29:00 +0000

Hello is a Chinese mobility and technology company, offering car-pooling, car-hailing, battery swapping, shared bike, e-bike, and e-scooter rental services. We have 800 million+ registered users.

As our AI business continues to grow, we need to support multiple workloads simultaneously, including model training, data processing, and AI agents. To address issues such as data fragmentation, inconsistent access methods, and complex cross-cloud data flows caused by multiple storage systems, we introduced JuiceFS (an open-source, high-performance distributed file system) in production to build a unified file data foundation that connects Notebook, training, evaluation, and agent environments.

Among these workloads, agents have become an important application type and are already used in several real-world application scenarios. We’ve also deployed an internal personal agent for employees. Unlike training workloads, which primarily focus on high-performance I/O and access to massive numbers of files, agents place greater emphasis on data movement during task execution, state persistence, and the accumulation of results.

This article focuses on how we use JuiceFS to persist user profiles, conversations, memories, skills, execution traces, and application artifacts, and how a shared workspace decouples agent state from compute Pods.

Hello's AI infrastructure and storage landscape

Our business scenarios are diverse, including customer service, review, scheduling, analytics, and broadcasting. To support AI applications across these businesses, we’ve progressively built an AI platform and a large language model (LLM) platform covering model development, training, inference, and serving.
The AI platform supports scenarios including Notebook, training, offline inference, online inference, and image building. Notebook environments need to mount user directories and datasets. Training and inference jobs need to access input data, models, and intermediate results. The image-building process also generates reusable caches.

The LLM platform aggregates different model services and provides model invocation and benchmarking capabilities. The data and reports generated during model benchmarking also need to be persistently stored.

For different types of data and workloads, we primarily use three types of storage:

CPFS provides high-performance parallel storage.
JuiceFS serves as the shared file storage foundation for the platform. It supports data requiring POSIX semantics, multi-node sharing, and a unified namespace, including Notebook environments, task datasets, model benchmarking reports, and image build caches.
Object storage is mainly used for object data such as multimedia attachments.

A small number of directories on development machines are still used as a transitional solution and have not yet been migrated to the unified storage system.

Overall, this storage architecture already covers the data persistence and cross-environment sharing requirements of our AI platform.

image: Hello's AI infrastructure
caption: Storage architecture in Hello's AI infrastructure

New storage requirements introduced by agent workloads

As agents have gradually moved into production, our AI capabilities have expanded beyond model development, training, and inference to agents capable of invoking tools and completing application workflows.
Related applications include image review for two-wheeled vehicle operations and trip recording review for ride-sharing services. These applications have delivered practical results by expanding review coverage, reducing manual effort, and lowering service costs.
Building on the deployment of vertical agents, we further introduced Jarvis Claw, a personal agent for employees. Employees can interact with Jarvis Claw through DingTalk or the web. By combining capabilities such as long-term memory, skills, Model Context Protocol (MCP), and Cron, it can assist with tasks such as answering questions, generating reports, reviewing content, and performing analysis.

Unlike vertical agents designed for a single application workflow, a personal agent needs to preserve state across multiple rounds of interaction and continuous tasks.

For example:

In an IT support scenario, Jarvis Claw needs to preserve user requests, attachments, conversations, knowledge retrieval results, and tool invocation results.
In an operations reporting scenario, it needs to continuously access historical source tables and scripts while generating new data tables, charts, and reports.
In an operations attribution scenario, it also needs to preserve data extraction scripts, intermediate data, and final analysis results for later inspection and re-execution.

A long-running agent needs to preserve five types of data:

User profiles and behavioral configurations
Conversation history and runtime state
Long-term memory that can be reused across conversations
Skills, MCP servers, channel configurations, and scheduled task configurations
Application artifacts such as scripts, CSV files, Excel spreadsheets, charts, and reports

These states naturally take the form of files:

Profiles are stored as Markdown files.
Conversation history is stored in SQLite.
Skills consist of directories and scripts.
Task configurations use JSON.
Application results exist as ordinary files such as data tables, charts, and code.

The complete state of an agent ultimately consists of a continuously evolving collection of files and directories that are created, modified, and related to one another.

A long-running agent therefore requires a unified shared storage layer that supports POSIX semantics, SQLite write-ahead logging (WAL), atomic rename operations, and access to large numbers of small files. It must also allow multiple Pods to share the same user state. Even if a compute Pod is destroyed, recreated, or rescheduled, the user's profile, conversations, memories, and task artifacts must remain available.

How JuiceFS stores the complete state of an agent

To meet these requirements, we extended our existing JuiceFS-based shared file storage foundation to agent workloads and use it to store a complete workspace for each user.

The platform assigns each employee an independent Jarvis Claw directory based on the employee account. The directory structure is similar to:

/userdata/aiplatform-jarvis-claw/user/<staff_id>/data/workspace

The user's profile, conversations, memories, skills, task configurations, and application artifacts are all stored in this directory. The Jarvis Claw Pod is responsible only for computation.

When a Pod is destroyed, recreated, or rescheduled, it only needs to remount the existing workspace to continue using the previous state. This makes the agent stateful while keeping the compute Pods stateless.

What agent state is stored in the workspace?

The workspace contains profile files such as:

AGENTS.md
SOUL.md
PROFILE.md
HEARTBEAT.md
MEMORY.md

These files describe the agent's identity, behavior, and user preferences and are combined into the system prompt.

Conversation and runtime state are primarily stored in files and directories such as:

sessions/
transcripts/
dialog/
tool_results/
history.db
chats.json
inbox_events.json

history.db uses SQLite and also generates WAL and SHM files.
User messages are written to transcripts/ at the beginning of a task. Even if the task is interrupted later, the conversation content generated before the interruption is preserved.
Long-term memory is stored in directories such as:

memory/
digest/
mem_agent/
mem_session/
mem_metadata/
resource/

Jarvis Claw manages memory using ReMe and a Memory-as-File approach.
Some memory is written directly by the agent during execution. Another part is generated by periodically reviewing historical conversations, extracting user preferences and information that remains useful over time, and writing the results to directories such as digest/. Internally, this process is referred to as Dream.
The workspace also stores task and capability configurations, including:

cron_jobs.json
cron_runs/
mcp.json
channels.json
dingtalk_webhooks.json
skills/
skill.json

Skill packages are stored as ZIP files in object storage. When a Pod starts, they are downloaded and extracted into the user's workspace.
Scripts, data tables, charts, and reports generated by the agent are stored in directories such as output/ and scripts/, as well as in files including Excel spreadsheets, CSV files, and SVG images.

The actual workspace also contains auxiliary content such as browser data, backups, and migration files.

Together, these files and directories form the complete state of an agent and are continuously created, modified, and accumulated as tasks are executed.

How a unified file system supports data movement and operations

Storing agent workspaces in JuiceFS not only provides state persistence and storage-compute separation, but also allows agents to share the same file system with environments such as Notebook, training, and evaluation.

When permissions allow it, Notebook can directly access an agent's workspace. Data prepared in Notebook can be directly provided to an agent. Likewise, data tables and scripts generated by an agent can be reused for analysis or evaluation without repeatedly uploading or copying data between different systems.

Operational tasks such as inspection, migration, backup, and per-user storage usage analysis can also be performed using standard tools such as ls, du, and rsync. Engineers can directly inspect conversations, memories, configurations, and task artifacts to troubleshoot issues during execution.

However, as the workspace evolves from an ordinary data directory into a working environment that an agent can actively read from, write to, and execute scripts within, the platform must provide not only file persistence and sharing, but also appropriate access controls and execution boundaries.

Security boundaries for shared workspaces

An agent may not only read and modify files in its workspace, but also execute commands or code through Shell, MCP, and REPL. Without effective constraints, its access could extend beyond the user's workspace to other users' data, service source code, and sensitive files inside the container.

Therefore, we established security boundaries across four layers:

User data isolation
Tool invocation governance
File access control
Subprocess management

How different users' data is isolated

We manage file system permissions through JuiceFS and OpenACL, assigning each user an independent workspace and controlling the directories that different users are allowed to mount and access.

With this mechanism, each agent can access only the workspace for which it has been authorized and cannot read other users' data.

File system permissions provide the basic data boundary between users, but they cannot cover all access behavior generated when an agent executes tools and code inside a container. Therefore, in addition to user-level permissions, the platform also needs to further restrict tool invocations and their execution environments.

How the system determines whether a tool invocation is allowed

When the model decides to invoke a tool, the operation is not executed immediately. Instead, the governance layer first evaluates the operation through assert_policy().
The result falls into one of four categories:

ALLOW: Allow the operation.
DENY: Reject unknown tools and dangerous operations such as sudo.
ASK: Request manual approval.
SANDBOX_FALLBACK: Execute the operation in a sandbox environment.

When manual approval is required, the system sends an approval request to the frontend through a side channel. After the user approves the request, a generalized version of the rule can be written to policy.yaml so that similar future invocations can reuse it.
For Shell operations, execute_shell_command() fixes the current working directory to the user's workspace rather than the service source directory. Then, it selects the appropriate sandbox configuration based on the governance result. The configuration prioritizes dynamically generated fine-grained rules from the governance layer, followed by globally configured default rules registered when the service starts. Bare-process execution is allowed only in local development environments. This layer determines whether a tool invocation is allowed to execute. For approved operations, the platform must further restrict the files that can actually be accessed based on the execution method.

How file access permissions are restricted for the main process and subprocesses

Agents access files mainly in two ways, which require different control mechanisms.

File tools such as read_file, write_file, edit, grep, glob, and view_image run directly inside the main service process and do not create subprocesses.

As a result, they cannot be intercepted by bubblewrap. The system uses application-level path checks, such as is_protected_path(), to constrain relative paths to the user's workspace while blocking access to protected locations such as /app, which contains service source code and installation files. Protected entries are also filtered during directory listing.

Shell commands, stdio MCP servers, and REPL environments used to execute model-generated code, on the other hand, create subprocesses.

For these operations, application-level path checks alone cannot establish a complete security boundary. Therefore, the system uses bubblewrap and Linux mount namespaces to construct an isolated file system view for subprocesses.

Within this restricted view:

The user's workspace is mounted with read and write access.
Other approved paths are exposed according to policy rules.
Service source directories such as /app are hidden or mounted as read-only.
Sensitive paths such as ~/.ssh and ~/.aws are blocked.

As a result, the agent can operate normally within the user's workspace without gaining access to all files in the container.

If the production environment is configured to use bubblewrap but the sandbox cannot actually be created, Shell and MCP use a fail-closed policy: execution is rejected rather than falling back to an unisolated bare process. The REPL follows the same rule when executing model-generated Python code. If the sandbox is unavailable, execution is denied.

How subprocess lifecycles and sandbox boundaries are managed

Each tool invocation that creates a subprocess uses an independent sandbox and process group with --die-with-parent enabled. When a user executes /stop, a task times out, or a new task within the same Scope replaces the current one, the system progressively cancels the current task and uses killpg to clean up the entire process group. This prevents Shell processes, MCP servers, or their child processes from continuing to run in the background.
Although stdio MCP servers are launched by the SDK itself, their execution commands are also prefixed with bubblewrap so that they use the same file system view as Shell commands. The REPL must likewise enter the sandbox when executing model-generated Python code, preventing execution methods other than Shell from becoming a path around the security boundary.
There are also several implementation details to consider:

After /app is overlaid with tmpfs, it must be remounted as read-only to prevent writes to the temporary /app from silently succeeding.
Before entering the sandbox, VIRTUAL_ENV and PATH entries pointing to protected locations must also be cleaned up to avoid problems with the Python runtime environment.
When the service starts, it performs an actual /bin/true execution inside the sandbox as a self-check rather than merely verifying that the bubblewrap binary exists.

The current solution primarily addresses file system and service source code isolation. Network isolation through --unshare-net has not yet been implemented. In addition, CPU and memory quotas have not yet been configured beyond execution timeouts.

Summary

Our production practice can be summarized in four points:

Everything about an agent's state can be represented as files. Profiles are Markdown files. History is stored in SQLite. Skills are directories. Memory and configuration are stored as files. Application artifacts include Excel spreadsheets, CSV files, charts, and scripts.
A shared file system provides state persistence and storage-compute separation. Agent Pods are responsible only for computation, while agent state and workspaces are stored in JuiceFS. Pods can be rebuilt at any time without causing user data to disappear with the compute instance.
The same file system connects multiple AI environments. Notebook, training, evaluation, and Jarvis Claw can access the same set of files. Data no longer needs to be repeatedly copied between different systems, and operations can rely on standard file-based tools and workflows.
Opening the workspace requires layered security boundaries. JuiceFS and OpenACL manage user-level file system permissions. Application-level path checks restrict the file access scope of tools running in the main process. Bubblewrap controls subprocesses created by Shell commands, MCP servers, and code execution.

Next, we plan to further explore agent self-evolution. Agent telemetry, memories, and evaluation data will continue to accumulate in the file system. Models will continue to improve, and this accumulated data will become part of the foundation for the agent's future evolution.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Scaling PB-Scale Scientific Data Workloads: How the American Museum of Natural History Powers Research Computing with JuiceFS

Thu, 03 Sep 2026 05:29:00 +0000

In cutting-edge scientific fields like astrophysics and genomics, high-throughput reading of massive datasets and zero-tolerance data integrity are the absolute lifebloods of computational analysis.

Recently, the JuiceFS team sat down for an in-depth conversation with the Scientific Computing team at the American Museum of Natural History (AMNH). They explained how they use JuiceFS Community Edition to handle large amounts of research data within an Open Storage Network (OSN) S3-compatible storage system which was funded by a grant from the National Science Foundation (NSF), creating a highly reliable and consistent storage setup for academic use.

The workload: 95% read-heavy traffic

The scientific computing environment at AMNH supports extraordinarily intense interdisciplinary research, operating at a massive scale. Currently, AMNH's OSN nodes have the capacity to store around 3 PB of research data, with approximately 30% of that total volume actively accessed via the NFS + JuiceFS layer. The overall storage architecture is precisely tailored to handle these specific academic patterns:

A 95% Read-Heavy Profile: Workloads include the system's massive reference datasets such as genomic sequences and astronomical survey data in read-only format. The calculations are very throughput- and latency-sensitive. Once a dataset is written, it will be read many times and rarely changed.
The OSN (Open Storage Network) Model: AMNH partners with the Open Storage Network. Standard rack servers and an external JBOD are hosted on-site at the museum but managed remotely by OSN administrators. This setup provides AMNH with cost-effective S3-compatible endpoints while allowing them to maintain physical ownership of the hardware and data.
Seamless Multi-Protocol Access: AMNH adopted JuiceFS Community Edition as the distributed file system layer, allowing mounting S3 buckets seamlessly and enabling traditional NFS access to this underlying S3-compatible storage for high-performance computing (HPC) clusters.

image: NFS + JuiceFS + OSN
caption: AMNH's NFS + JuiceFS + OSN storage architecture

Data integrity first, without silent corruption

For scientific research, data consistency is crucial. To prove and continuously observe the system's resilience, the AMNH team conducted rounds of tests during the PoC phase and watched JuiceFS logs and metrics closely in production.

"We have recently observed timeout warnings when JuiceFS writes to the OSN object store, but data integrity issues were never reported. We do want to understand why, though."

– Scientific Computing Team, AMNH

That was one of the discussions we had during the meeting. It turns out that JuiceFS configures all object store SDKs with RetryMaxAttempts=1 and implements custom logic for retries. This way, JuiceFS is able to unify the behavior across all object stores it supports. Thus, in this case, the warning is a precaution, and the data integrity is ensured.

When comparing JuiceFS to other file systems that claim higher raw performance metrics (often achieved by using "unsafe" write-handling or skipping strict metadata commits), AMNH opted for data consistency.

"We use almost all the default configurations for the JuiceFS client with Redis as the metadata engine, and it's been running in production steadily for 4 years now."

– Scientific Computing Team, AMNH

Current & future tuning options

The conversation also continued into how AMNH tunes and considers the JuiceFS client configurations to match their scientific computing patterns needed for the future.

The write-back configuration

Currently, the only non-default configuration AMNH utilizes is --writeback , which smooths out the client-side writing throughput. There are always tradeoffs. Write-back sacrifices a bit of data consistency and requires high disk reliability on the mounting machine. However, given AMNH's current workload pattern (mostly reads), both teams agreed that it is acceptable either way, write-back enabled or not.

High-performance caching

The teams also evaluated the differences between the Linux kernel's native page cache and JuiceFS' cache for client-side metadata. While the kernel page cache is the fastest, the JuiceFS client-side metadata cache offers more flexibility. In a read-heavy setup, this configuration can save many roundtrips to the metadata engine. Currently, the metadata engine is co-located with the JuiceFS client-mounting host, but in the future, this configuration can be a nice add-on for performance.

On the file side, since the hardware uses HDDs managed by the OSN object store, the JuiceFS client's local data read cache significantly lowers latency after the first read. Increasing the --cachesize can further improve performance when needed.

The potential of using SQLite as the metadata engine

In a highly creative approach, the teams discussed utilizing SQLite on local NVMe storage as their JuiceFS metadata engine. The JuiceFS team highlighted that for standalone sandboxes or read-heavy workloads, SQLite is an exceptional, low-maintenance choice, provided the metadata file is backed up for resilience (which is true for Redis as well). Since for AMNH's current scenario, the primary I/O bottleneck usually lies in the backend object storage rather than the database, SQLite's simplicity makes it incredibly attractive for specific workloads.

Final thoughts

Managing multi-petabyte scale, strong data consistency, and high-throughput research workloads is a difficult balancing act for any infrastructure team. AMNH has built a resilient, low-maintenance storage environment that lets scientists work without creating massive engineering overhead, using an open architecture and JuiceFS Community Edition on top of an OSN backend.

Reflecting on their four-year journey of running this system stably in production, the AMNH team summarized their experience seamlessly:

"The implementation on our side is straightforward, and we are happy about it overall."

– Scientific Computing Team, AMNH

As data demands in scientific computing continue to grow, this blueprint of combining cost-effective object storage with a robust, reliable file system layer provides a highly reproducible success story for academic and research institutions worldwide.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and our Discord server.

How Tencent Cloud Built a Unified Storage Platform with JuiceFS and FoundationDB

Thu, 27 Aug 2026 08:39:00 +0000

As big data and AI workloads continue to grow, more enterprises are transitioning from tightly coupled storage-compute architectures to storage-compute separation architectures. However, in real-world deployments, enterprises often need to manage multiple storage systems simultaneously, including Cloud Object Storage (COS), S3, and HDFS. These backends differ in access protocols, authentication mechanisms, file system semantics, and metadata performance. This creates challenges in unified access, existing data reuse, and cache acceleration.

To address these challenges, the Tencent Cloud (a cloud computing service operated by Tencent, a multinational technology company headquartered in China) team built an enterprise-grade unified storage solution based on JuiceFS+FoundationDB. The solution provides full POSIX semantics for newly created file systems while also integrating existing object storage and HDFS data into a unified namespace, enabling centralized access and cache acceleration.

In this architecture, FoundationDB serves as the metadata engine, supporting metadata management at the tens-of-billions scale with strong transactional consistency. This is also the first time the JuiceFS community has shared a production practice using FoundationDB for metadata management. Local caching and distributed caching are introduced to reduce remote access latency. Going forward, our team plans to further evolve the platform in areas including lakehouse integration, AI training acceleration, intelligent data tiering, and cross-cluster federation.

Challenges: storage gets more complex after storage-compute separation

During early customer discussions and proof-of-concept (POC) projects, we found that when enterprises migrate from storage-compute integrated architectures to storage-compute separation architectures, their biggest concerns are not the architecture model itself, but two questions:

Can existing data and applications be migrated smoothly?
Can performance remain close to the original integrated architecture after migration?

Enterprises often operate multiple storage systems simultaneously, including COS, S3, and HDFS. These backends differ in access protocols, SDKs, authentication methods, and file system semantics. If Spark, Hive, Flink, or AI training frameworks need to integrate with each storage system separately, development and maintenance costs can quickly increase.

Performance is another major challenge in adopting storage-compute separation. When data access paths become longer, remote storage and network communication introduce additional overhead. Meanwhile, running multiple storage systems independently makes it difficult to share and reuse caching capabilities.

Therefore, a unified storage solution must not only solve multi-backend integration challenges, but also provide reusable cache acceleration capabilities.

In addition, although object storage is well suited for massive-scale data storage, it’s generally less efficient than traditional file systems for metadata-intensive operations such as:

List
Stat
Directory traversal
Small-file access

Different storage backends also provide inconsistent support for file system semantics such as directories, permissions, quotas, and snapshots. These differences further complicate unified access for big data and AI workloads.

Based on these requirements, we chose JuiceFS as the foundation for building an enterprise-grade unified storage platform.

Architecture: unified access with two operating modes

JuiceFS adopts a metadata-data separation architecture. The metadata engine can be selected based on workload scale and consistency requirements, including Redis, TiKV, and other options.

For this solution, we selected FoundationDB as the metadata engine, mainly because of its:

Strong transaction model
Ordered key-value data model
Automatic data distribution
Horizontal scalability

These capabilities enable metadata management at the tens-of-billions scale.

Based on JuiceFS' core architecture, we built a unified storage platform consisting of:

A unified client layer
A service-oriented metadata layer
Multi-level caching
Multi-backend storage adapters

JuiceFS provides fundamental capabilities including:

File system semantics
Multi-protocol access
Object storage-based data paths
Local caching

On top of these capabilities, we further extended the system with the following capabilities to meet the multi-tenant and multi-storage requirements of cloud service scenarios:

Unified authentication
Multi-volume management
Unified namespace
Existing data integration
Distributed caching

On the client side, big data and AI applications can access the system through multiple interfaces:

HDFS interface
POSIX mount
S3 interface
Python SDK

The client supports:

Hadoop SDK
Python SDK
FUSE mount
Local cache
Service discovery

This allows frameworks such as Spark, Hive, Flink, Presto, and various AI training frameworks to access storage without adapting individually to different backend systems.

image: JuiceFS+FoundationDB
caption: Unified storage architecture based on JuiceFS+FoundationDB

For metadata access, we converted JuiceFS' metadata access path into a service-oriented architecture.

Clients no longer connect directly to FoundationDB. Instead, they communicate with stateless metadata services through RPC. The metadata service centrally provides:

Metadata APIs
Authentication and authorization
Multi-volume management
Quota management
Directory protection
File protection
Task management

ZooKeeper is used for:

Service registration
Service discovery
Cross-node state synchronization

The metadata service can scale horizontally as needed, while FoundationDB acts as the backend metadata engine responsible for metadata persistence and transaction processing.

image: Data access architecture
caption: Data access architecture for managed mode and external mode

To cover both new file system creation and existing data ingestion scenarios, the underlying storage layer provides two modes:

Managed mode is designed for newly created file systems. File metadata is written to FoundationDB through the metadata service, and file data is split according to JuiceFS' chunk/slice/block model and written to object storage. This provides full POSIX file system semantics.
External mode is designed for existing data reuse. Its core is the Unified File System (UFS) abstraction layer, which abstracts backends such as COS, S3, and HDFS into a unified file system interface, without migrating or taking over the existing metadata and data of those backends. Metadata requests are passed through the metadata service via UFS to the underlying storage, while data reads and writes are performed using the native protocols of the respective backends.

image: UFS multi‑storage adaptation architecture
caption: UFS multi‑storage adaptation architecture in external mode

The unified namespace organizes both modes within a single directory structure. To upper‑layer applications, it presents a unified path and access entry. As for the underlying implementation, different directories can correspond either to managed file systems or to mounted existing HDFS or object storage data. This allows new and existing data to coexist under the same access system without requiring large‑scale data migration beforehand.

On the data access path, the system provides both local caching and distributed caching. Read requests first check the local cache; if missed, they access the distributed cache; if still missed, they fall back to the underlying storage. This reduces remote access latency and backend storage pressure.

FoundationDB: design trade‑offs for 10‑billion‑level metadata

In the unified storage access layer, the metadata engine determines the system's scalability limit and consistency capabilities. When file counts reach the billion or even 10‑billion level, the metadata system itself becomes the critical bottleneck.

Why we chose FoundationDB

During the metadata engine selection phase, we focused on several key questions:

Whether a single cluster can support 10‑billion‑level metadata
Whether the transaction model is strong enough
Whether automatic sharding and horizontal scaling are supported
Whether operational complexity is manageable

Traditional relational databases tend to hit bottlenecks in single‑volume scale and horizontal scalability, so we focused our evaluation on distributed key-value solutions, including TiKV and FoundationDB:

TiKV has been widely adopted in large‑scale metadata scenarios and offers strong horizontal scalability.
FoundationDB better meets our requirements in terms of strict transaction consistency, ordered key‑value model, automatic data distribution, multi‑replica strong consistency, and operational complexity.

Ultimately, we chose FoundationDB as the metadata engine because it’s better suited for the strongly consistent, high‑concurrency, range‑scannable access patterns typical of file system metadata. This practice also allowed our team to accumulate experience in modeling, tuning, and operating FoundationDB in large‑scale metadata scenarios, providing a reference for future use cases such as Hive table metadata management.

Key design: multi‑volume isolation and range scans

File system metadata naturally has two types of access patterns:

A single cluster must host multiple file systems, requiring clear boundaries between volumes.
Operations such as directory traversal, attribute queries, and chunk index queries often rely on prefix scans.

Therefore, key design must both satisfy multi‑volume isolation and closely align with file system access paths, avoiding hot spots or large‑transaction issues at scale.

Leveraging FoundationDB's ordered key-values, we used JuiceFS metadata design method:

Using fsname as the key prefix to distinguish different file systems
Organizing attributes, directory entries, chunk indexes, and other metadata related to the same file within a close key space to improve access locality
Encoding numeric fields in big‑endian order so that lexicographic order aligns with numeric order for sequential scans.

image: FoundationDB metadata key
caption: FoundationDB metadata key encoding design

On this basis, multiple file systems can share the same FoundationDB cluster while maintaining independent key spaces. For operations such as list that may scan large amounts of data, pagination is used to control single‑transaction size. In transaction conflict scenarios, optimistic locking and automatic retries are used to reduce lock contention.

How to avoid FoundationDB's hard limits

FoundationDB has explicit hard limits on key size, value size, and transaction size:

A single key is limited to 10 KB.
A single value to 100 KB.
A single transaction to 10 MB total size.

These limits cannot be adjusted via configuration parameters. When using FoundationDB as a file system metadata engine, we need to pay special attention to operations that tend to enlarge metadata size, such as frequent random writes, repeated truncates, fallocate punch hole, and copyFileRange, which may cause individual values to grow continuously. Large‑scale file copies, large file truncation, batch metadata import, or deleting an entire file system may trigger oversized transactions.

The highest risk comes from chunk slice accumulation. In JuiceFS' data model, the slice list corresponding to a chunk is stored in a single value, and each slice occupies about 24 bytes. When a chunk accumulates about 4,266 slices, it may hit FoundationDB's 100 KB value limit. Operations such as frequent random writes to the same chunk, repeated truncates, fallocate punch hole, and copyFileRange will continuously increase the slice count. Without a governance mechanism, a single value can keep growing, eventually causing transaction commit failures.

This issue also has a subtle implementation aspect: some paths use atomic append operations like AppendIfFits. This is a blind‑write operation; the transaction cannot know the total value size after the append in advance. If the append exceeds 100 KB, failure may not occur until the transaction commit phase. This means the problem is not exposed early during writes but instead accumulates silently until a commit fails. Therefore, relying solely on retries after failure is insufficient; safety boundaries must be set in the data model and write path.

To address chunk slice accumulation, we implemented and strengthened the compaction mechanism. The core idea is to detect chunks with too many slices during read/write operations and merge multiple small slices into fewer large slices to control individual value growth. For lightly fragmented chunks, asynchronous compaction can be triggered to avoid blocking writes. When the slice count reaches a safety threshold, synchronous compaction is triggered to prevent it from growing to the 100 KB limit. In practice, maxSlices is set to 2,500, corresponding to about 60 KB, leaving about 40 KB of headroom below the 100 KB limit.

During compaction implementation, concurrency and reclamation issues also need to be handled. The merge process uses CAS semantics to ensure the chunk has not been modified concurrently, avoiding data loss. If compaction fails, normal writes are not affected, and it can be retried later. Obsolete slices generated after compaction are passed to the Garbage Collection (GC) process, where reference counting determines whether they are still in use. A delayed‑deletion mechanism supports accidental deletion recovery and background reclamation.

In addition to chunk slice accumulation, we also examined other risk items such as file system configurations, Kerberos tokens, POSIX locks, extended attributes, and UFS path‑type keys. Overall, these risks are manageable: most values are small, and key design is constrained by file name or path length. The real area requiring governance is still the single‑value expansion caused by chunk slice accumulation.

Cache acceleration system: two‑level caching to reduce remote access overhead

After storage‑compute separation, data access paths shift from local disks to remote object storage or HDFS. For read‑intensive workloads such as big data analytics and AI training, if every read goes directly back to the underlying storage, network latency and backend access pressure are amplified. Therefore, the unified storage layer must not only address "how to access multiple storage types" but also "how to make remote data reads faster."

Our approach is to introduce two‑level caching: client local cache as L1, and distributed cache as L2. Read requests first check the local cache; if hit, return directly. If missed, they access the distributed cache. If still missed, they fall back to the underlying storage and fill the data into the cache system. This provides both the lowest latency from local SSD or memory and the ability to reuse hot data across multiple clients through an independent SSD cache cluster.

Local caching mainly uses JuiceFS Community Edition's existing capabilities, extended to support external mode. It runs inside the client process, caches data at block granularity by default, and supports least recently used (LRU) eviction, sequential read prefetching, and OS cache control. The distributed cache is our self‑developed independent cache cluster. Clients use consistent hashing to route the same data block to the same cache node, making it suitable for multi‑node shared reads, scenarios with limited client local storage, or workloads requiring cross‑task hot data reuse.

The most critical area in the cache system is consistency. In managed mode, every write generates a new slice ID. This causes the cache key to change, so old caches are naturally not hit. Meanwhile, objects in object storage are never modified in place after being written, so no additional invalidation mechanism is needed.

External mode is more complex, because files in the underlying HDFS or object storage may be directly overwritten by external systems. If the cache key contains only the path, stale data may be read. To address this, we introduced a fingerprint mechanism in external mode, incorporating file path, block offset, and file fingerprint into the cache key. The fingerprint is composed of file modification time and size, and is frozen at open time. When the file is modified externally, the fingerprint changes, so old caches are naturally not hit, satisfying close‑to‑open consistency.

This brings a trade‑off: managed mode provides finer‑grained cache isolation based on slice IDs, while external mode relies on file fingerprints, which are more coarse‑grained and may invalidate larger cache ranges. However, for scenarios where existing data may be modified by external systems, this is a necessary trade‑off for consistency, and it’s also an area for future optimization.

In addition to basic read caching, the system supports cache warm-up, cache‑stampede protection, TTL‑based eviction, and transparent degradation. For example:

The warmup command can pre‑load hot data.
Distributed caching supports batch-warming by directory prefix.
When the cache layer becomes unavailable, the system can automatically degrade to direct underlying storage access to prevent cache failures from affecting application reads.

Multi‑tenancy and elastic scaling

In cloud provider scenarios, the unified storage layer must serve multiple business units, tenants, and file systems simultaneously. Therefore, we designed the metadata service as a stateless architecture. Clients perform service discovery and load balancing through ZooKeeper, and metadata service nodes can scale horizontally on demand to increase overall access capacity and availability.

Multi‑volume isolation relies on FoundationDB's key prefix design. Each file system uses an independent fsname prefix, and different volumes have independent key spaces, allowing a single FoundationDB cluster to host multiple file systems. New file systems can be added online without restarting the metadata service.

image: Stateless metadata
caption: Stateless metadata service and FoundationDB multi‑volume isolation architecture

In multi‑node deployments, runtime information such as quotas, permissions, directory protection, and task states must also be synchronized across nodes. We use ZooKeeper for cross‑node change notifications, combined with debounce merging, precise key‑level refresh, and periodic polling for fallback, ensuring configuration changes are promptly synchronized to all metadata service nodes.

Through stateless metadata services, fsname‑based isolation, and ZooKeeper state synchronization, the system can achieve both multi‑tenant isolation and horizontal scalability at both the service and metadata layers.

10‑billion‑level metadata validation

We conducted 10‑billion‑level metadata tests based on JuiceFS 1.3.0 and FoundationDB 7.3. The test environment comprised 6 x86 servers running Kylin V10 SP3, with FoundationDB deployed using triple replication and the SSD storage engine.

The tests were performed after 10.8 billion metadata entries had already been written. Results showed that at the 10‑billion scale, FoundationDB single‑operation latency remained largely consistent with baseline data. Some operations, such as readdir_1k and lookup, even performed better. Overall, after reaching 10.8 billion inodes in a single volume, metadata operation latency remained in the sub‑millisecond to 2 ms range, indicating no significant performance degradation at scale.

The table below shows benchmark single‑operation latency (μs/op, lower is better):

Operation	Redis-always	FoundationDB (official)	Current environment (10B+)	vs. Redis
mkdir	558	1,842	1,763	3.2x
create	570	1,761	1,720	3.0x
rename	728	1,911	1,849	2.5x
unlink	658	1,940	1,887	2.9x
lookup	173	1,029	817	4.7x
getattr	87	504	414	4.8x
write	553	1,747	1,710	3.1x
readdir_10	280	1,744	1,357	4.8x
readdir_1k	1,490	15,276	1,185	0.8x

In extreme write tests, we used 15 concurrent processes to perform juicefs clone for metadata cloning, validating FoundationDB's write ceiling. Results showed that under the 6‑node + SSD configuration, stable‑phase write throughput was approximately 11,000–15,000 inodes/s. Bottleneck analysis indicated that the main pressure was on storage write queues and disk I/O, rather than CPU or network. Future improvements can be made by adding storage processes, upgrading disk performance, or scaling Commit Proxy and TLog components.

Inode creation:

Phase	Inode creation rate	Description
Initial peak	23,000+ inodes/s	Peak performance when the write queue is not yet full
Stable phase (after 5–6 minutes)	15,000 inodes/s	Write queue gradually fills up
After queue full (after 10+ minutes)	11,000–13,000 inodes/s	Disk I/O becomes the bottleneck; FDB throttling protection is triggered

For high availability, we validated scenarios with 1‑node and 2‑node failures. With 1‑node failure, the service remained available, and FoundationDB automatically triggered replica repair and data migration. With 2‑node failure, the service remained available, but the cluster's fault tolerance dropped to zero—it could no longer tolerate additional failures. After recovery, the system triggered data rebalancing, with a complete recovery time of approximately 4 hours in testing. This process introduces additional I/O pressure, so production environments must pay close attention to disk watermarks, single‑disk failures, and resource consumption during recovery.

For capacity planning, test data showed that at 5 billion inodes, FoundationDB disk usage was approximately 6.1 TB; at 10.8 billion inodes, it was approximately 10.1 TB. After accounting for triple replication, compression, and reserved space, approximately 120 GB per 1 billion inodes can be used as a baseline capacity estimate.

Through this validation, we’ve essentially confirmed that FoundationDB can support 10‑billion‑level metadata per volume while maintaining stable metadata access performance at that scale. At the same time, the triple‑replica architecture provides good failure recovery capabilities, though production environments still require careful capacity planning, disk monitoring, and I/O management during recovery.

Summary and future plans

At present, this unified storage access layer built on JuiceFS+FoundationDB has completed core capability construction in big data storage‑compute separation scenarios, including unified client access, leveraging of external data, multi‑level cache acceleration, 10‑billion‑level metadata management, and multi‑tenant scalability. Moving forward, we’ll continue to evolve around lakehouse integration, AI training, and storage governance.

In the lakehouse direction, we plan to strengthen integration with table formats such as Iceberg and Hudi through the unified storage layer, enabling upper‑layer data lake and data warehouse scenarios to reuse unified storage access, permission management, and cache acceleration capabilities.

For AI training, we’ll further optimize cache warm-up and near‑local caching to reduce remote read amplification and GPU wait times in training jobs. We’ll also explore model checkpoint management using snapshot capabilities, making intermediate state saving, recovery, and reuse more efficient during training.

In storage governance, we’ll explore intelligent tiering capabilities, automatically migrating data between SSD, HDD, and object storage based on access frequency, achieving a better balance between performance and cost. For larger‑scale cloud deployments, we’ll continue to advance cross‑cluster federation capabilities, enabling metadata synchronization, routing, and unified access across multiple clusters.

Going forward, we’ll continue to iterate around performance, cost, elasticity, and data governance, enabling more workloads to access and manage underlying data in a unified manner.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Giving AI Agents a Real File System

Thu, 20 Aug 2026 08:42:00 +0000

The new root user

"Everything is a file."

For decades, this single sentence defined the core Unix philosophy. Today, the primary operator interacting with those files is no longer a human at a terminal but an autonomous LLM agent spinning up environments, editing code, and executing scripts.

Because LLMs were pre-trained on codebases, wiki pages, documentation, directory trees, and terminal logs, POSIX file system operations serve as the zero-friction API for agentic reasoning. Giving an agent access to a familiar workspace lets it leverage standard developer tools immediately.

The following is an example output of an AI agent performing code reviews:

# Verified, the following command returns only the method definition.
# Your change doesn't have any actual invocation of this method.
grep -r "store.addDelayedStaging" . --include="*.go"

However, while a classic POSIX interface is necessary, modern AI agents demand far more from their storage than standard local file systems were built to deliver. To support a massive number of ephemeral, fast-moving agent sessions, the underlying storage layer must evolve as well.

Architectural paradox

This shift introduces a fundamental storage paradox for AI infrastructure engineers:

At the surface layer: The agent requires a workspace that strictly behaves like a classic, local POSIX file system so that standard developer tools (git, python, grep, compilers, etc.) function seamlessly without custom glue code.
Under the hood: Traditional instance-bound storage breaks down when managing hundreds or thousands of ephemeral, short-lived, or parallel agent sessions. The underlying backend must provide remote snapshotting, consistent multi-agent concurrent access, elasticity, and scalability while still being compatible with that familiar file interface.

To build scalable infrastructure for autonomous AI agents, we must preserve the classic Unix interface while completely re-engineering the storage engine beneath it.

Why AI agents love the file system

If APIs are rigid software contracts, POSIX file systems are the open playground. Giving an LLM agent direct access to a file system workspace unlocks operational advantages that structured tool calls simply cannot match.

Offloading the context window

Even the most advanced models with million-token context windows experience severe latency, high costs, and performance or attention degradation when a massive repository or gigabytes of build logs are included in an LLM prompt. The file system serves as secondary memory, allowing the agent to store raw state externally and selectively read, seek, or stream only the relevant fragments it needs at runtime.

The universal abstraction

A file system requires zero protocol overhead when an agent has shell execution access, unlike external services or database integrations that need custom JSON schemas or Model Context Protocol (MCP) tool wrappers. LLMs already possess a profound understanding of standard file operations and directory navigation out of the box. Exposing a POSIX file system gives the agent a universal workspace across any language, framework, or file format, without needing to define or maintain a single custom tool schema.

Unlocking the Unix ecosystem

By granting an agent a file system, it immediately inherits decades of battle-tested CLI tools: grep, awk, git, sed, compilers, and static analyzers. Rather than inventing "AI-native code search," an agent can simply execute grep or analyze git diff using tools already optimized for speed and scale.

Database-driven metadata meets elastic storage

When building agentic platforms, the naive approach is to rely on local block storage attached directly to a compute instance (e.g., standard ephemeral instance disks or container volumes). While simple initially, instance-bound disks rapidly fail under the demands of autonomous agents:

Ephemeral state loss: When an agent container scales down, crashes, or migrates to another host, all workspace state and context stored on instance-tied disks are lost unless manually synced.
Concurrent data access: Multi-agent workflows often require sub-agents (e.g., a planner, coder, and reviewer) to operate on the same repository simultaneously. Instance-attached disks cannot safely allow multi-node concurrent reads and writes without file lock corruption or race conditions.
Non-native data backup: Compute-instance disks lack built-in data backup or snapshotting, which makes rolling back the environment after an agent error slow and expensive.

To overcome these constraints, modern agent file architectures decouple the metadata engine (directory trees, permissions, file locks, and state) from the data storage layer (file payloads).

By backing the metadata layer with specialized database engines and offloading raw file blocks to elastic object or remote storage, file system operations essentially become database queries. The agent gets the exact POSIX semantics its tools expect, while the infrastructure gains consistency guarantees, automatic backups, and infinite horizontal scale.

The ecosystem landscape

Innovative projects across the storage industry demonstrate how database backends are powering this new era of AI agent file storage. Here, we are exploring a few notable approaches.

Lightweight embedded approach: AgentFS with SQLite

AgentFS, developed by Turso, embodies the lightweight embedded approach within the database-as-file-system paradigm. Designed specifically for agent sandboxing, it stores every piece of runtime state (file operations, key-value entries, and tool call histories) in a single SQLite database. The underlying engine is Turso, a full rewrite of SQLite in Rust that adds native asynchronous I/O and concurrent access, with optional cloud sync for portability across environments.

image: AgentFS architecture
caption: AgentFS architecture

AgentFS exposes a POSIX-like virtual file system via an SDK or a FUSE mount. This unified storage foundation delivers copy-on-write isolation for safe agent execution, instant SQL-queryable audit trails for debugging and compliance, and single-file snapshots that make agent state trivial to share, version, and reproduce.

Client-server transactional approach: TigerFS with PostgreSQL

TigerFS, developed by Timescale, mounts a PostgreSQL database as a transactional file system over FUSE (Linux) or NFS (macOS). Every file maps to a row, directories map to tables, and file contents become columns. Every write is a full ACID transaction, enabling multiple agents and humans to read and write the same files concurrently with true ACID guarantees.

image: TigerFS- how it works
caption: TigerFS: how it works

Multi-agent task coordination becomes as simple as moving a file (`mv task.md ./doing/`), with PostgreSQL transactions natively preventing race conditions without custom API orchestration. TigerFS offers two modes: file‑first for workspaces with atomic writes and reversible savepoints, and data‑first for exploring existing databases with Unix tools like `ls`, `cat`, and `grep`, with filters pushed down as SQL. Every change is versioned and fully reversible, and the file system itself serves as the API.

Decoupled distributed approach: JuiceFS with object storage

JuiceFS takes a fundamentally different architectural path: it decouples metadata and data storage entirely. File data is split into chunks and stored in S3-compatible object storage, while metadata lives in a separate, pluggable engine, supporting Redis, MySQL, PostgreSQL, TiKV, SQLite, JuiceFS Enterprise Edition (EE) metadata engine, and more. This separation enables JuiceFS to deliver strict POSIX compliance alongside shared, multi-node concurrent access, allowing hundreds of agent instances to safely work across a shared workspace. JuiceFS can be connected using FUSE, Kubernetes CSI, or SDKs (like Python, Hadoop, and S3 gateway), which makes it a flexible storage option for agents as well as other AI processes, including data preparation, model training, and deployment in different regions. Every committed change is immediately visible across all mounts with strong consistency, and the system scales elastically with object storage economics.

image: 1JuiceFS 架构图.drawio
caption: JuiceFS architecture

Together, projects like AgentFS, TigerFS, and JuiceFS represent distinct architectural explorations in the quest to provide agents with a persistent, shareable file system. Each tackles the problem from a different angle, yet all share a common recognition: agents naturally work with files. There is no clear consensus on which approach will become the dominant standard, and perhaps no single winner exists. What is certain is that the file system is re-emerging as a core primitive for AI agent orchestration, precisely because the POSIX interface aligns so naturally with how agents read, write, and organize data, and ultimately because agents complete tasks by acting on data and files.

Agents' full toolbox

While a database-backed file system provides a zero-friction home for code and context, a file system alone isn't enough for complex enterprise workflows. Real-world agents also need to orchestrate across multiple systems of record, for example:

The file system: As discussed above, the file system serves as the primary workspace for source code, local logs, build artifacts, and raw data manipulation.
Databases & APIs: The structured state layer for transaction logs, vector search embeddings, and relational data that models need to query or update.
The Model Context Protocol (MCP): An open standard protocol that standardizes how LLM applications connect to external data sources and tools through MCP servers. It does not replace existing APIs. Instead, it provides a common interface and transport layer on top of them so clients can discover and use capabilities without custom integrations for every application or model.

By combining elastic file workspaces with standardized protocols like MCP, developers can build agents that are autonomous, practical, and powerful.

Final thoughts

We aren't just building sandboxes anymore. Instead, we are designing a new type of operating system. In this model, the LLM acts as the CPU, the context window is the fast L1 cache, and database-backed file systems serve as persistent NVMe drives.

The industry is still actively debating what an agent's ultimate memory architecture should look like: whether it should rely on vector databases, structured state stores, or file-based paradigms. While the ideal abstraction may continue to evolve, the file system is already proving to be a core foundation in agent infrastructure. Let's honor the classic Unix philosophy and give agents seamless access to decades of battle-tested software tools through a familiar POSIX interface and the performance and scale that modern distributed backends deliver.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

GPFS vs. Alluxio vs. JuiceFS: Architecture and Use Cases Compared

Thu, 13 Aug 2026 10:33:00 +0000

In AI workloads, scenarios such as training, inference, model distribution, agents, and data lakes each have distinct requirements for throughput, latency, concurrent access, POSIX compatibility, consistency, cost, and operational complexity.

In this article, we’ll first examine the typical I/O patterns of AI workloads, identifying the practical challenges these workloads pose to storage systems. Then, we’ll compare General Parallel File System (GPFS), Alluxio, and JuiceFS, analyzing their architectural differences, file system semantics, caching mechanisms, cost models, and applicability across different AI scenarios.

I/O patterns and storage challenges in AI workloads

Based on our interactions with enterprises across various AI domains, we can categorize common requirements as follows.

Autonomous driving: large‑scale data generation and training

Autonomous driving is one of the most data‑intensive AI storage scenarios. Large fleets of data‑collection vehicles continuously generate images, videos, sensor data, and trajectory logs, which undergo cleaning, labeling, and format conversion before entering model training pipelines. Common data formats include .mcap, .pack, TFRecord, and LMDB. These environments require highly stable and scalable storage due to the long data pipelines, diverse formats, and heavy training workloads.

LLM training: full‑pipeline data access

Foundation model scenarios cover multiple stages, including data cleaning, model training, checkpoint reads/writes, and inference services. Model weights, training data, and intermediate results are accessed repeatedly across different stages. The storage system must sustain long‑running jobs while maintaining stable data access during task failures, node outages, and training recovery.

Multimodal models: coexistence of small files, aggregated data, and model files

AIGC scenarios include text‑to‑image, image‑to‑image, text‑to‑video, image‑to‑video, and 3D generation. Training inputs may consist of many individual images or video clips, or they may be aggregated into formats like LMDB or Parquet to improve training efficiency. The training process also generates checkpoints and outputs model files such as safetensors. This makes the data landscape more complex than in single‑task training scenarios.

Compute platforms: multi‑cloud collaboration and model distribution

Compute platforms focus on distributing models and data across multiple environments. Users may pull models from external repositories or upload their own, and then run training or inference workloads across different clusters and cloud environments. The key challenge is reducing redundant copying while enabling consistent access to the same data across different compute environments.

Quantitative finance: balancing performance and cost

Quantitative finance typically involves smaller data volumes than autonomous driving or AIGC. However, as techniques such as Transformer‑based models, neural network training, time‑series modeling, and market graph structure analysis are increasingly adopted, storage cost is becoming a more explicit selection factor.

In recent discussions with quantitative finance customers, we’ve observed growing attention to storage costs. High‑performance parallel file systems like GPFS are inherently performance‑oriented, especially in large‑capacity, all‑flash configurations. In some on‑premises deployments, the storage investment for a large‑capacity all‑flash GPFS cluster can approach the cost of an entire 5090 GPU cluster.

AI agents: data sharing in short‑lived sandboxes

AI agents represent a rapidly emerging scenario. They often involve many short‑lived sandboxes—each executing a subtask with a lifetime of just seconds or even less.

While these tasks run briefly, the context, model files, tool files, and intermediate results must be shared among subtasks. If each sandbox independently mounts a file system, and the mount process itself takes several seconds, task scheduling efficiency suffers. A more practical approach is to pre‑mount the file system on the host and then expose it to sandboxes via bind mounts, PVCs, or similar mechanisms.

From a storage perspective, AI agents are less concerned with sheer capacity and more focused on data continuity, shared access, and mount efficiency within short‑lived tasks. As agent complexity grows, demands on file system semantics and data sharing capabilities will continue to rise.

The table below shows typical I/O patterns and storage requirements for AI workloads:

Scenario	Typical I/O pattern	Key storage challenges	Selection criteria
Autonomous driving	Large‑file throughput, mmap random reads, small‑file reads	Large data scale, long training pipelines, high random‑read pressure	Throughput, caching, metadata capabilities, capacity cost
LLM training	Large‑file reads/writes, mixed reads, checkpoint reads/writes	Full‑pipeline access, high stability requirements for long‑running tasks	Stable throughput, concurrent access, fault recovery
Multimodal models	Small‑file reads, aggregated large‑file reads, model file access	Coexistence of small and large files, significant multi‑task concurrency	Caching, metadata management, multi‑task concurrency
Compute platforms	Model distribution, cross‑cluster access, multi‑cloud collaboration	Data must be consistently accessible across environments	Unified namespace, multi‑cloud distribution, cache governance
Quantitative finance	Large‑file sequential reads, small‑file reads, training/backtesting access	Rising cost sensitivity, need to balance performance and cost	Capacity cost, scalability, long‑term operations
AI agents	Small I/O, multi‑client sharing, short‑lived access	Mount efficiency, data continuity, task isolation	File system semantics, shared access, mount methods

GPFS vs. JuiceFS

From PFS to GPFS: capabilities and limitations of parallel file systems

To understand GPFS, it helps to first understand parallel file systems (PFSs). Intuitively, a parallel file system separates metadata and data, enabling multiple clients to access underlying storage resources in parallel. In this architecture, metadata and data travel different paths. Clients do not need to funnel all I/O through a single node; they can concurrently access underlying disks or storage nodes. This allows hundreds of compute nodes to simultaneously read and write block devices or storage resources, breaking the network bottleneck of a single path and enabling horizontal scaling.

image: Parallel file system architecture
caption: Parallel file system architecture

GPFS, Lustre, and BeeGFS are all typical examples of parallel file systems.

GPFS, later renamed IBM Storage Scale, is a mature parallel file system that has been widely used in high-performance computing environments for many years. It provides strong advantages in throughput, concurrent access, and consistency, making it suitable for workloads that require high performance and reliability.

However, as AI infrastructure continues to scale and organizations increasingly prioritize cost optimization, the high cost and operational complexity of GPFS can become important considerations during storage selection.

Typical GPFS workloads include quantitative finance, genomic analysis, physics simulations, and weather forecasting.

Architecture advantages and trade-offs: metanode, token locks, and strong consistency

The architectural advantages of GPFS mainly come from two mechanisms:

Metanode, which affects metadata coordination
Distributed token locking, which determines how consistency is maintained during concurrent access

Metanode-based metadata coordination

Under the Metanode model, a GPFS cluster usually consists of:

I/O servers
data disks
metadata disks

Metadata is stored on dedicated metadata disks. Unlike systems with centralized metadata services, GPFS allows clients to participate in parts of metadata coordination.

In other words, GPFS assigns different clients as coordination nodes for different files or inodes instead of routing all metadata requests through a fixed centralized service.

Therefore, GPFS clients need to maintain communication with each other, typically through port 1191.

When network issues or connection failures occur, the cluster manager must determine node status and remove failed nodes from the cluster to prevent split-brain conditions and data inconsistency.

This design helps distribute metadata coordination pressure. Under stable network and storage conditions, it can support high levels of concurrent access.

However, it also introduces higher environmental requirements. If network quality is unstable or storage latency increases, coordination efficiency may degrade. In severe cases, users may experience:

Request stalls
Long waiters
Recovery operations requiring node restarts

These problems do not necessarily indicate a weakness of GPFS itself. Instead, they reflect that high-performance, strongly consistent architectures require reliable networking, disk, and cluster state management.

Distributed token locks

Distributed token locking is another key mechanism behind GPFS consistency.

GPFS manages read and write operations through tokens:

Read operations require read tokens
Write operations require write tokens

When multiple clients access the same file, GPFS controls concurrent access by granting, revoking, and transferring tokens.

For example, if Client A holds a write token for a file and Client B wants to read or modify the same file, GPFS must revoke the token from A. A must flush dirty data to disk and release the token before B can proceed. This ensures strong consistency, but it also relies on stable, fast network and disk responses.

If disk writes become slow or network communication is interrupted during token revocation, long waiters can appear. In practice, requests waiting on Revoke Token or Reopen Token operations are not uncommon. This illustrates GPFS' inherent trade‑off: token‑based strong consistency and concurrency control come at the cost of increased system complexity.

Early InfiniBand networks or high‑quality fiber networks could reliably support this mechanism due to their low latency and high reliability. However, in some newer deployments—particularly those using RoCE—suboptimal network conditions, hardware quality, or operational expertise can amplify stability challenges in token coordination, especially when hundreds of clients coexist in a single cluster.

Overall, this mechanism enables high‑performance parallel access but demands high network, disk stability, and cluster operational capabilities. Client anomalies also require handling token recovery and cluster state restoration. Therefore, it’s essential to assess whether your team has the necessary deployment, monitoring, and troubleshooting skills.

Based on practical experience, GPFS deployments also require attention to several engineering considerations:

mmap scenarios: GPFS employs special mechanisms for mmap, such as pagepool, to reduce memory copying and improve performance. However, this also creates complex interactions with OS memory management. Large‑scale reliance on mmap patterns is generally not recommended without thorough validation.
Hot files and large directories: Hot files, hot directories, massive small files, or many clients concurrently accessing the same directory can become performance bottlenecks due to metadata coordination and lock contention. Mitigation typically involves directory splitting, data sharding, and access pattern optimization.
Operational management and monitoring: GPFS' management interface is not particularly user‑friendly for newcomers, and some monitoring information is not intuitive. Many teams supplement with external monitoring systems like Grafana to better observe cluster state, performance metrics, and anomalies.
Cluster Export Services (CES), Active File Management (AFM), and other components: While these features enable more complex scenarios, they also introduce additional configuration, operational, and troubleshooting overhead. Teams without long‑term GPFS experience should carefully evaluate this complexity upfront.
Capacity planning: Many GPFS or CPFS deployments emphasize expansion over contraction. Capacity planning early on is crucial to avoid either excessive initial capacity (and long‑term cost) or insufficient capacity (hampering business growth).

Performance comparison

Intuitively, many people assume that JuiceFS—built on object storage and independent metadata services—cannot be directly compared to GPFS. However, in certain AI workloads, both systems do present comparable alternatives.

It’s important to note that GPFS performance depends heavily on the synergy of underlying disks, storage servers, networking, clients, and the parallel file system itself. JuiceFS, on the other hand, is influenced by object storage performance, metadata services, client caching, distributed cache groups, and mount modes. Therefore, comparisons should not be reduced to single performance numbers—they must account for I/O patterns, deployment architecture, data scale, and access paths.

The following tests were performed with JuiceFS Enterprise Edition. The Community Edition shares the same core architecture, so Community Edition users can also refer to these testing methodologies and results.

Sequential reads: GPFS better per node, JuiceFS scales through cache groups

In single‑node scenarios, the performance models differ markedly. In our tests, with two 400 Gbps NICs, a single GPFS node achieved about 100 GB/s for sequential reads.

Under TCP mode (200 Gbps NIC), a single JuiceFS node peaked at about 20 GB/s. With RDMA (two 400 Gbps NICs), it reached about 55 GB/s. For higher aggregate throughput, JuiceFS can scale horizontally by adding cache nodes. For example, in one autonomous driving customer deployment with about 150 cache nodes (each with 160 Gbps NICs), the aggregated application throughput reached about 2.3 TB/s.

Sequential writes: GPFS excels at synchronous writes; JuiceFS uses writeback for scale

For sequential writes, GPFS holds an advantage under synchronous write semantics. Its write capability stems from the underlying parallel storage system—data written is immediately accessible by other clients under strong consistency semantics. This is well suited for scenarios requiring write reliability, real‑time visibility, and consistency.

JuiceFS sequential write performance depends on whether writeback is enabled. In synchronous mode, writes go to the backend object store, and performance is constrained by backend storage, protocol overhead, and network latency. With writeback enabled, data is first written to client‑local cache and later uploaded asynchronously—aggregate throughput improves, but real‑time visibility and consistency semantics change.

Thus, GPFS is better for scenarios requiring synchronous persistence and real‑time visibility. JuiceFS with writeback is suitable for workloads that can tolerate asynchronous upload semantics.

Random reads: GPFS better at high concurrency, JuiceFS competitive in certain scenarios

In 4K single‑process random read tests, we compared JuiceFS, GPFS, and local disk /tmp (EXT4).

The figure below shows the test results:

At iodepth 1, 2, and 4, JuiceFS outperformed GPFS.
At iodepth 8, GPFS surpassed JuiceFS and stabilized around 80K IOPS.
JuiceFS peaked around 68K IOPS at iodepth 4 and 8 and then gradually declined at deeper I/O depths.

image: JuiceFS 4K single-process random read performance
caption: JuiceFS 4K single-process random read performance

Multi‑process random reads present more complex behaviors. When multiple processes concurrently read the same file, GPFS' consistency and lock mechanisms can introduce extra overhead. To validate this, we tested both reading the same file and different files.

The following figure shows the test results:

GPFS reads of different files scaled rapidly with numjobs, reaching ~433K IOPS at numjobs=12.
GPFS reads of the same file dropped after numjobs=2, suggesting consistency coordination overhead.
JuiceFS reached ~258K IOPS at numjobs=12 and stabilized around 250K thereafter.

image: JuiceFS 4K multi‑process random read performance
caption: JuiceFS 4K multi‑process random read performance

All tests were run with local caching disabled, and data was served from distributed cache. The results show that GPFS has higher random read performance under high concurrency, while JuiceFS remains highly capable—sufficient for most AI training requirements.

Random writes: GPFS superior at high concurrency

Random writes better expose architectural differences between the two systems. JuiceFS' random write performance depends on whether writeback is enabled. Without it, performance is mainly constrained by the object storage backend. With writeback enabled, performance reflects the client's local cache path capabilities.

Under writeback enabled and cache‑dir set to local NVMe storage, 4K multi‑process random write results showed:

From numjobs 1 to 3, JuiceFS increased from ~28K to ~56K IOPS, significantly higher than GPFS.
At numjobs=12, both were close (~51K vs. ~50K).
Beyond numjobs=16, GPFS continued to rise to ~66K (at 16) and ~81K (at 20), while JuiceFS was in the 50–60K range.

image: JuiceFS 4K multi‑process random write performance
caption: JuiceFS 4K multi‑process random write performance

These results indicate GPFS holds an advantage in random write performance, while JuiceFS closes the gap significantly when writeback is enabled. However, random writes are not common in AI workloads, so this should not be a primary evaluation criterion.

Selection summary

The main differences between GPFS and JuiceFS are summarized in the table below:

Aspect	GPFS (symmetric, decentralized)	JuiceFS (separated metadata & data)
Metadata architecture	Distributed, embedded in local memory	Independent, high-performance external database (cloud-native, lightweight)
Data storage layer	Expensive, tightly coupled shared SAN / parallel disk	Cost-effective, highly reliable, highly elastic object storage
Locking & concurrency	Distributed token-based locks (strong consistency)	Optimistic concurrency (Community Edition) / Single-threaded core (Enterprise Edition)
Typical use cases	Select HPC and scientific computing scenarios	AI research, training, inference acceleration, large-scale data management

GPFS is better suited for environments with ample budgets and stringent requirements for low latency, strong consistency, high‑concurrency access, and random write capability—typical of traditional HPC, scientific computing, and some quantitative finance applications. Its performance also depends on stable networking, storage hardware, and professional operational expertise.

For AI research, training, inference acceleration, and large‑scale data management challenges involving scalability, performance, cost, and multi‑cloud management, JuiceFS is the more suitable solution—and has been validated in production across all the aforementioned domains.

Alluxio vs. JuiceFS

Alluxio is another solution that enterprises often compare with JuiceFS when evaluating AI storage architectures.

Both solutions can provide filesystem access and cache acceleration on top of object storage. However, they differ significantly in product positioning, data organization models, and metadata architectures.

From the perspective of product evolution, the two projects have taken different approaches:

Alluxio’s recent AI-focused development has mainly concentrated on its Enterprise AI product line. The latest release of the Alluxio open-source repository is v2.9.4, released in June 2024.
JuiceFS continues to evolve both its open-source and enterprise editions. Many new capabilities are first introduced, validated, and refined in the open-source edition before gradually becoming available in the enterprise edition to serve more production users.

Key architectural differences

Data organization and consistency boundaries

Alluxio uses a 1:1 transparent caching model, which preserves the original organization of files in the underlying storage system. Existing data does not need to be imported or reorganized in advance, and the caching layer can be added or removed as needed.

image: Comparison of data organization
caption: Comparison of data organization models between Alluxio and JuiceFS

JuiceFS, on the other hand, splits files into data chunks and stores them in object storage. It uses a metadata service to maintain file system semantics and data chunk mappings. From the application perspective, users see a complete file system, while the objects stored in the backend object storage are JuiceFS-managed data chunks rather than the original files.

This difference also affects the source of truth and consistency boundaries:

For Alluxio, the actual data typically remains in object storage or other UFS systems. The cache layer mainly accelerates access. If applications bypass Alluxio and directly modify data in the underlying storage, additional cache invalidation, refresh, or synchronization mechanisms may be required.
JuiceFS combines metadata services and object storage into a complete file system. File states and data mappings are managed centrally by JuiceFS, and application reads and writes go through JuiceFS. Therefore, its consistency boundary exists within the file system itself and does not rely on additional synchronization between the cache layer and backend storage.

Cache and namespace organization

Alluxio emphasizes a unified namespace and shared cache pool. It can integrate multiple underlying storage systems, including OSS, S3, HDFS, Ceph, MinIO, and NAS, into a single namespace and accelerate access through a unified distributed cache.

For organizations that already have multiple storage systems and do not want to migrate or reorganize existing data, this approach provides greater flexibility. However, when multiple workloads share the same cache pool, resource management and isolation typically require policies such as directory-based controls, priorities, and time to live (TTL) settings.

JuiceFS typically uses an independent filesystem as the management unit. Different object storage backends, such as OSS, COS, and TOS, usually correspond to different file systems with their own cache configurations.

Compared with Alluxio’s emphasis on a unified view across multiple data sources and shared caching, JuiceFS focuses more on clear boundaries between file system management, permissions, and data governance.

In addition, JuiceFS Enterprise Edition can connect multiple buckets into a single file system and use the same cache resources for acceleration. This enables multi-source data access scenarios while maintaining clear file system-level management boundaries.

Metadata architecture

Alluxio Enterprise distributes caching and part of the state management responsibilities across workers. Workers not only provide data caching but also participate in managing cache status and data location information. This brings data access closer to compute nodes or GPU nodes. Coordination and management components still exist, but cache-related states are not entirely centralized in an independent metadata service. As a result, when workers frequently restart or scale in and out, additional attention is required for recovering cache states and coordinating data locations.

image: Comparison of distributed cache architectures
caption: Comparison of distributed cache architectures

JuiceFS manages core file system metadata through an independent metadata service.

The open-source edition supports metadata engines such as Redis and TiKV, while the enterprise edition provides a highly available metadata service to support consistency and transactional operations. Distributed cache nodes are responsible only for caching data blocks. Therefore, cache node failures mainly affect cache hit rates and access performance, but they do not change file system metadata states.

How architectural differences affect real-world usage

POSIX compatibility

Alluxio exposes data from object storage or other UFS systems through file system interfaces, but its file system semantics are not always complete.

Capabilities such as timestamps, file locking, hard links, symbolic links, extended attributes, and ACLs may require additional configuration or have usage limitations. For read acceleration and lightweight access scenarios, these limitations may not be significant.

However, if an application intends to use Alluxio as a full-featured file system, these factors should be carefully evaluated.

JuiceFS aims to provide complete file system capabilities on top of object storage. Therefore, JuiceFS continuously improves POSIX compatibility, covering common file system semantics such as directories, permissions, timestamps, and file locks, as well as more advanced system calls such as ioctl-based immutable and read-only settings. In production environments, JuiceFS POSIX compatibility already covers the vast majority of workloads and approaches full POSIX compatibility.

Write performance and write amplification

Alluxio preserves the original file format, which provides transparent access and reduces the cost of onboarding existing datasets. However, random writes, overwrites, and partial updates require careful consideration of write amplification. The reason is that object storage generally does not support in-place modification of object contents. If the backend maintains a 1:1 file layout, a small modification may trigger rewriting a larger amount of data, potentially requiring the entire object or file to be rewritten.

JuiceFS uses a chunk-based data layout. For random writes or append operations, JuiceFS only needs to process affected data chunks and update corresponding metadata, without being constrained by the original object storage file layout. Therefore, JuiceFS can better control write amplification for partial updates and random write workloads. However, if users need to restore JuiceFS data into the original file format in object storage, additional export or migration operations are required.

Deployment and engineering capabilities

Alluxio Enterprise supports full-component Kubernetes deployment. It also supports temporary write caching without persisting data to object storage, such as:

temporary decompression workloads
intermediate computation results
short-lived data that can be discarded after use

These capabilities are closer to cache-layer acceleration and compute-side optimization scenarios.

JuiceFS Enterprise Edition currently typically deploys metadata services on virtual machines or physical servers, mainly due to requirements for metadata stability and system reliability.

JuiceFS focuses more on long-term filesystem capabilities, including:

large-scale metadata management
Trash and data recovery
transactional operations
cache governance
seamless upgrades
open-source ecosystem integration

In production, JuiceFS has already supported up to 500 billion files—an important demonstration of its capability as a complete file system in large‑scale data management scenarios.

Selection recommendations

The difference between Alluxio and JuiceFS is not simply about the number of features. They are designed to solve different problems:

If data already exists in object storage, HDFS, Ceph, MinIO, or NAS, and organizations do not want to migrate or reorganize existing data but only need a pluggable cache layer to accelerate reads, Alluxio can be a suitable choice.
JuiceFS is likely a better fit, if organizations want to build a complete file system on top of object storage while supporting:
- mixed read/write workloads
- full POSIX semantics
- strong consistency
- elastic metadata scaling
- tens of thousands of concurrent clients
- multi-cloud data management

Summary

GPFS, Alluxio, and JuiceFS address different core storage challenges:

GPFS focuses on low latency and high-concurrency read/write performance for high-performance computing workloads.
Alluxio primarily provides cache acceleration for existing datasets.
JuiceFS provides a complete, scalable file system built on top of object storage.

Therefore, storage selection should not be based only on individual performance metrics. Organizations should first determine whether they need high-performance shared storage, a data acceleration layer, or a complete file system designed for multi-cloud and large-scale data management.

The final decision should consider consistency requirements, data scale, cost, and operational complexity.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and the community on Discord.

JuiceFS Blog

How to Access JuiceFS via S3 and WebDAV Protocols

A shared initialization path

Applicable scenarios and common mappings

S3 Gateway principles: from object semantics to file system operations

Request path: from the MinIO S3 server to jfsObjects

State storage: how the internal namespace carries the Gateway state

Semantic conversion: PutObject, Multipart, Bucket, and Metadata

Permission coordination: IAM and multiple instances

WebDAV principles: from file protocol to JuiceFS operations

Request path: from HTTP entry to webdavFS

State boundaries: dead properties and LOCK

Nydus + JuiceFS: Reducing Container Startup Time for AI Inference from 116s to 1.4s

Why OCI image distribution is slow

Dragonfly+Nydus: what it solves and what it leaves behind

Architecture redesign: Nydus blobs hosted by JuiceFS

Runtime mechanism: how Nydus and JuiceFS coordinate

Startup phase: fetching bootstrap (metadata)

Read phase: on‑demand blob access

Cache phase: data reuse via JuiceFS cache groups

Production test data

Container startup time comparison

On‑demand file read performance

Summary: combining on‑demand loading with shared cache

JuiceFS Enterprise 5.4: From 100 Billion Files to 1 Million Clients

Supporting 1 million clients

Fast cloning for large directories

Performance improvements

More than 90% RDMA bandwidth utilization

Higher random-read IOPS under high concurrency

Optimizing data access across regions

On-demand synchronization to reduce data replication

Accelerating metadata access with read-only nodes

More flexible cache management

Improving hot-data cache hit rates

Preserving cache hit rates when adding multiple cache nodes

Reducing the number of files on large cache disks

Warming up specific byte ranges

Simplifying operations for massive files

Restoring files by deletion time

Reducing GC and fsck memory usage

Restricting access with token allowlists

Summary: scale changes everything

Infinite Workspaces for AI Agents: A Practical Architecture with JuiceFS, SQLite, and Litestream

The creative trio: JuiceFS + SQLite + Litestream

An unconventional but effective combo

The performance reality

Final thoughts

Keeping GPUs Fed: How Meta's AI Storage Architecture Aligns with JuiceFS

Introduction

Why storage matters for AI workloads

Meta's AI storage blueprint: the evolved architecture

The three-component architecture

Cross-region data access with tiered caching

JuiceFS: cloud-native, AI-native from day one

Cross-region strategy: mirrors and replication

Architectural similarities

A concrete example: explicit cache hydration

Conclusion

Everything Is a File for Agents: Hello Uses JuiceFS to Build Stateful AI Workspaces

Hello's AI infrastructure and storage landscape

New storage requirements introduced by agent workloads

How JuiceFS stores the complete state of an agent

What agent state is stored in the workspace?

How a unified file system supports data movement and operations

Security boundaries for shared workspaces

How different users' data is isolated

How the system determines whether a tool invocation is allowed

How file access permissions are restricted for the main process and subprocesses

How subprocess lifecycles and sandbox boundaries are managed

Summary

Scaling PB-Scale Scientific Data Workloads: How the American Museum of Natural History Powers Research Computing with JuiceFS

The workload: 95% read-heavy traffic

Data integrity first, without silent corruption

Current & future tuning options

The write-back configuration

High-performance caching

The potential of using SQLite as the metadata engine

Final thoughts

How Tencent Cloud Built a Unified Storage Platform with JuiceFS and FoundationDB

Request path: from the MinIO S3 server to `jfsObjects`

Semantic conversion: `PutObject`, `Multipart`, `Bucket`, and `Metadata`

Request path: from HTTP entry to `webdavFS`

Reducing GC and `fsck` memory usage