JuiceFS Blog

How Horizon Robotics Uses JuiceFS for EB-Scale Autonomous Driving Data

Thu, 05 Nov 2026 08:56:00 +0000

Horizon Robotics is a Chinese technology company that develops artificial intelligence (AI) chips used in self-driving cars and advanced driver assistance systems. We have partnered with 40+ original equipment manufacturers (OEMs) and brands globally and are the trusted choice for smart driving mass production among China's top 10 OEMs.

Architecture is shaped by scale. At the exabyte level, with hundreds of billions of files, hundreds of thousands of clients, and 200 PB of data added and deleted every week, the assumptions built into open source file systems begin to break down.

We use both JuiceFS Community Edition and Enterprise Edition to manage EB-scale data for our AIDI intelligent driving R&D platform. This article focuses on our experience with JuiceFS Community Edition, sharing how our team addressed four engineering challenges, the results we achieved, and the trade-offs we made.

New storage challenges at EB scale

AIDI supports our closed-loop intelligent driving R&D workflow. Data collected from road tests and production vehicles is filtered and labeled before entering model training. Trained models are evaluated and released, and data from newly identified issues feeds back into the pipeline. Storage underpins this entire workflow. How quickly data becomes available at each stage directly affects whether computing resources sit idle.

Over the past few years, the scale of this workflow has grown dramatically. Data capacity has increased from petabytes (PB) to exabytes (EB), while training capacity has grown from thousands to tens of thousands of GPUs. As models iterate faster and data flows back into the pipeline more frequently, storage workloads have increasingly exceeded the assumptions behind the original architecture.

Today, the platform faces the following scale and workload characteristics:

Capacity: EB scale (1018 bytes) and still growing, well beyond the scale covered by conventional JuiceFS Community Edition testing.
File count: Hundreds of billions (1011) of files. Individual directories, such as the trash, can contain tens of billions of entries.
Weekly data changes: Approximately 200 PB of data added and another 200 PB deleted, leaving tens of billions of objects to be deleted from object storage.
Client count: Hundreds of thousands of clients connecting through Kubernetes (K8s) training clusters, the Container Storage Interface (CSI), an S3-compatible gateway, and direct access through software development kits (SDKs).
Workload characteristics: Large sequential I/O for datasets coexists with small random I/O for training checkpoints. Access frequencies vary greatly between hot and cold data, and multiple tenants share a single metadata cluster.

These conditions led us to four engineering challenges:

How can we prevent metadata hotspots and maintain stability with hundreds of billions of files?
How can we govern data safely and audibly when approximately 200 PB is deleted every week?
How can we balance training throughput and storage costs at EB scale?
How can we connect, rate-limit, and monitor hundreds of thousands of clients?

We began using JuiceFS Community Edition on AIDI in 2021 and have continuously customized it to address these challenges. As data volumes and use cases expanded, we gradually introduced JuiceFS Enterprise Edition as well. Both editions are now used on the platform. The sections below focus on our work with Community Edition and explain the changes we made and the trade-offs involved.

Challenge 1: Optimizing metadata for hundreds of billions of files

Reducing metadata hotspots and large-directory pressure

Once the file count reached hundreds of billions, metadata access through TiKV exposed three bottlenecks:

During peak training periods, the CPU on nodes hosting the leader replica was saturated, while follower replicas still had spare capacity.
Transactions requested global timestamps from the Timestamp Oracle (TSO) service in Placement Driver (PD), adding a network round trip.
Hot metadata, including mount configuration and capacity information, was queried repeatedly.

Further parameter tuning alone could no longer relieve these bottlenecks. We needed to change the access paths.

To address them, we modified the Region routing cache in the TiKV Go client so follower replicas could handle read requests. We also eliminated one TSO round trip from frequent point lookups and introduced a read cache for hot metadata.

We validated the changes against both old and new versions through continuous integration tests. Then, we delivered them as build-time patches, avoiding the need to maintain a long-lived fork of the upstream dependency. When scaling the cache cluster in or out, we also minimize large-scale cache-key remapping to reduce the risk of cache stampedes.

image: Metadata access paths
caption: Metadata access paths

Large directories containing billions of entries also put pressure on client memory. We adopted streaming, batched directory scans to avoid building the entire result set in memory during a single readdir call. This prevents out-of-memory failures and supports kernel-level readdir caching.

Rate limiting to keep the cluster stable

Hundreds of thousands of clients share one Meta cluster, and a single client can overwhelm it. To prevent one tenant's traffic from affecting the entire cluster, we independently limit resource consumption in the metadata and data planes.

On the metadata side, we use token buckets based on PD Resource Groups. Request Units (RUs) measure and limit each tenant's resource consumption. In the data plane, we separately limit traffic to the cache cluster and traffic that falls back to object storage.

Clients that exceed their budgets are throttled before their load can overwhelm shared resources. We manage rate-limit budgets declaratively: operators maintain the desired state in YAML files, and the manager compares that state with the current configuration and automatically generates SQL statements to create, alter, or delete database tables as needed.

image: Multi-tenant rate limiting and resource budgets
caption: Multi-tenant rate limiting and resource budgets

Metadata disaster recovery: cold backups as a last line of defense

Metadata is critical to the entire system. TiKV stores the relationships between hundreds of billions of files and their metadata. If that metadata cannot be recovered, the file contents may still exist in object storage, but the system may no longer know which files they belong to.

We use three layers of protection against different failure scenarios:

TiKV Raft replication protects against hardware failures.
Periodic full snapshots of TiKV are backed up to object storage in a separate region. These backups provide a recovery path for accidentally deleted metadata in a keyspace, logical metadata corruption, and region-wide failures.
Object storage retains the file contents, allowing datasets to be used again after metadata recovery. When recovery is needed, we rebuild the TiKV cluster and restore the latest snapshot.

Cold backups come with a trade-off: metadata can only be restored to the most recent backup. Changes made afterward may be lost, and the potential recovery point depends on the backup interval. For our current training workloads, this is an acceptable trade-off.

Challenge 2: Data visibility and governance, reducing costs by deleting 200 PB every week

When approximately 200 PB of data is deleted each week, logical deletion is only the beginning. Object storage continues to accumulate billions of objects awaiting reclamation, and a single-node scan cannot keep up. Failed asynchronous deletions can also leave orphaned objects whose keys no longer reveal which files they belonged to.

We built a governance workflow that connects deletion decisions, recovery from accidental deletion, space reclamation, and leak auditing.

Deletion decisions: dashboards and application-owner review

Application owners must decide what to delete and when. A trash cannot replace that decision-making process.

We provide dashboards showing dataset access patterns, changes in access frequency, and capacity growth trends. Application owners use this information to plan deletions and capacity expansion. Once they confirm a decision, they can initiate deletion or recovery themselves. Deleted data first enters the trash.

The dashboards rely on a metadata statistics pipeline that makes data queryable the next day (T+1). We first export the full set of file metadata from the Meta cluster to the data lakehouse to establish a baseline. After that, we synchronize incremental changes daily through the changelog.

Application owners can query capacity, access patterns, and lifecycle data directly without asking the storage team to run queries for them. With approximately 200 PB of data added and deleted every week, any interruption or data drift in this pipeline can undermine capacity planning and reclamation decisions. Keeping the pipeline reliable is therefore essential.

image: Pipeline for exporting fine-grained statistics to the data lakehouse
caption: Pipeline for exporting fine-grained statistics to the data lakehouse

Recovering deleted data: a seven-day trash

Data deleted through the platform remains in the trash for seven days. This window provides time to recover accidentally deleted data and absorbs the daily deletion volume, which can reach petabytes. It also prevents reclamation workloads from placing immediate pressure on the metadata engine.
The trash uses different mechanisms for different paths:

Mechanism	Design	Purpose
Paths that bypass the trash (mount options)	Skip the trash for paths explicitly designated as non-recoverable, such as temporary training directories.	Prevent large volumes of data from overwhelming the trash.
Scheduled batch cleanup	Delete data older than seven days in batches organized by time bucket. The process is observable and supports recovery.	Manage trash capacity while preserving recovery capabilities.
Streaming deletion lookup	Scan deleted files as a stream to locate the requested data.	Enable recovery within seconds and support deletion audits.

During the retention period, application owners can restore data themselves through the recovery platform or command-line tools, without filing an operations ticket. Recovery rolls back metadata and rebuilds object references without copying the underlying data. The platform supports petabyte-scale daily recovery, and a single PB-scale recovery can finish within hours. Both deletion and recovery enforce permissions through access control lists (ACLs) and record audit logs.

Reclaiming space: distributed garbage collection

The standard Community Edition garbage collection (GC) relies on single-node scans. This cannot keep pace with approximately 200 PB of weekly deletions. We modified the process to record files and data slices awaiting deletion in TiKV, and then divide reclamation work into multiple shards for parallel processing by multiple workers.

Workers acquire tasks through leases. After completing a task, a worker deletes its task record and claims the next batch. Tasks remain in TiKV until completion. If a worker fails, another worker can take over unfinished work. This approach distributes reclamation load without requiring a separate message queue.

Leases determine which worker handles a task. Reference counts in the deletion transaction determine whether an object can actually be deleted. Physical deletion proceeds only when the reference count meets the deletion condition—in the current implementation, a negative value. This prevents concurrent processing or retries from causing duplicate deletions.

On the object storage side, reclamation uses batch deletion, exponential backoff and retries, and audit records.

We apply the same distributed processing model to large-file compaction. Files that grow through repeated appends can accumulate many slices, increasing read overhead. A separate gc_cluster distributes compaction tasks from a manager to multiple workers.

These workers periodically merge slices online to control read amplification.

image: Sharded gc_slices processing pipeline
caption: Sharded gc_slices processing pipeline

Auditing orphaned objects with `leak-check`

In addition to reclaiming objects based on metadata records, we use leak-check to identify objects that were missed by the deletion process.

The standard fsck check in JuiceFS looks for cases where metadata references an object that is missing. leak-check performs the reverse check: it finds objects that still exist in storage but are no longer referenced by metadata.

On the object storage side, we list objects concurrently across 32 subprefixes under chunks/, reducing the bottleneck caused by scanning through a single connection. We independently obtain the object inventory and the metadata reference inventory. We then use two external-sort (extsort) operations and merge the sorted results as streams to generate a candidate list. This avoids building a full in-memory map containing hundreds of billions of entries.

image: Dual-extsort merge in leak-check
caption: Dual-extsort merge in leak-check

The comparison results are only candidates for cleanup; they cannot be deleted immediately. Objects being written may temporarily lack metadata references. By default, the audit generates a list without deleting anything and filters out recently created objects. Deletion happens only after the candidates are reviewed and confirmed. This process helps find orphaned objects while reducing the risk of deleting writes that are still in progress.

Challenge 3: Balancing training performance and EB-scale storage costs

Training requires throughput on the order of hundreds of GB/s. It also combines large sequential I/O for datasets with small random I/O for checkpoints. At EB scale, we cannot keep all data in the hot tier. When compute runs across multiple clouds, data must also be accessible from locations close to the workloads.

Three-tier caching: from clients to a centralized cluster

Our three-tier cache expands the sharing scope at each level:

L1 cache: Reuses data blocks within the current client.
L2 cache: Shares data blocks between clients through peer-to-peer (P2P) transfers.
L3 cache: A centralized cluster that serves hot training data for both reads and writes.

Reads check L1, L2, and L3 in sequence. Only when all three tiers miss does the client access object storage.
When multiple training jobs read the same dataset, L1 caching alone cannot share data across clients, so clients may fetch the same data from object storage repeatedly. L2 allows clients to act as cache peers. A block already cached by one client can be reused by another through the range-read (getrange) interface. When dozens of jobs read the same data concurrently, the total amount fetched from object storage can approach the size of a single copy. This saves bandwidth by making better use of existing local disks and memory.
Requests that miss both L1 and L2 proceed to the centralized L3 cache cluster, which handles PB-scale data. L3 serves hot working sets shared across clients and absorbs most of the read throughput. The same cluster provides both read caching and write aggregation: read caching serves large sequential dataset I/O, while write aggregation handles small random checkpoint I/O. The cluster can scale horizontally to increase hot-data capacity.
A cache failure could force hundreds of thousands of clients to fetch data directly from object storage at once, increasing the load on the storage backend. We use health checks to remove unhealthy nodes and reintroduce them after recovery. If the read cache becomes entirely unavailable, clients fall back to direct reads from object storage, trading higher latency for continued access.
We also record cache hit counts, bytes served, latency, and slow requests at each tier. These metrics help us detect cache degradation and changes in fallback traffic.

Metadata caching: avoiding repeated lookups

When a training job reads a file, it must do more than retrieve data blocks. It also repeatedly looks up files, retrieves file attributes, and opens files through operations such as lookup, getattr, and open. Each operation requires a metadata query.
To reduce repeated requests to TiKV, the client uses an open cache to retain query results.
The challenge is that different file types change at different rates. Dataset images and packaged files usually remain unchanged for long periods and can be cached longer. Indexes and logs, by contrast, may change continuously and need to reflect updates quickly.

Using the same time-to-live (TTL) for every file creates a trade-off: a long TTL risks serving stale metadata, while a short TTL forces frequently accessed but unchanged files to query the backend repeatedly.
Therefore, we define three TTL tiers by file type. Static files use longer TTLs, dynamic files use TTLs measured in seconds, and other files use the default TTL. The configuration supports hot reloading through YAML.
For static files, a cache hit renews the lease (expire_lease), resetting the expiration time. Frequently used files can therefore remain cached, while cold files expire naturally. The modification-time filter (filter_mtime) prevents recently modified files—which may still be receiving writes—from entering the cache. This avoids caching metadata before it has stabilized.

image: File-type-based metadata caching policies
caption: File-type-based metadata caching policies

Controlling read amplification: coordinating prefetching and caching

Prefetching can accelerate sequential reads, but excessive prefetching can also increase load on the storage backend. We control prefetching independently at the cache layer and the kernel layer.

At the cache layer, the client uses block sequence numbers in object keys to fetch subsequent blocks in advance. A deduplication table and bounded queue limit concurrency. When the queue is full, the client drops prefetch requests rather than blocking normal reads.
At the kernel layer, dynamic read-ahead adjusts the window based on whether access is sequential or random. A global watermark limits the overall load, and the configuration supports hot reloading.

image: Two-stage prefetch pipeline
caption: Two-stage prefetch pipeline

Keeping data close to compute and tiering by access frequency

The changes described above improve access to hot data. Controlling storage costs at EB scale also requires deciding where data should reside.

When training workloads run across multiple clouds, we allow buckets in different regions to share a single file-system namespace. Metadata records each object's owning bucket and storage tier. Read requests use this information to route to the appropriate bucket, so migrating data does not require application access paths to change.

Data moves directly between buckets on the server side, without consuming client bandwidth. During migration, the read path falls back across the hot tier, cold tier, and full pool as needed. Declarative configuration controls the owning bucket and migration policy for datasets or directories.

When training compute moves to another cloud, we can migrate data to the corresponding region in advance, reducing dependence on cross-cloud dedicated bandwidth. Infrequently accessed data can be moved to lower-cost buckets or cold-storage pools such as tape libraries and retrieved when needed.

This approach keeps hot data close to compute while controlling storage costs at EB scale. Capacity scaling no longer depends on a single cloud provider. Critical datasets can also retain backup buckets in other clouds, providing an alternative read path during cloud outages.

image: Architecture for data locality and hot:cold tiering
caption: Architecture for data locality and hot/cold tiering

Challenge 4: Connecting and managing hundreds of thousands of clients

When hundreds of thousands of clients connect simultaneously, multi-tenant access control, connection methods, and client observability all need to be designed for scale. We address these concerns through access control, training-job integration, and client monitoring.

Multi-tenancy: access control

Our production kernel versions do not support enabling POSIX-compliant ACLs on the Filesystem in Userspace (FUSE) mount path. Therefore, we moved tenant authorization into a user-space ACL Manager.

The ACL Manager uses buckets as the unit of authorization and manages permissions through the READER and MAINTAINER role groups. It supplements lightweight directory access protocol (LDAP) group information using globally unique identifier (GUID) files and records audit logs.

image: ACL Manager
caption: ACL Manager

With this design, LDAP only needs to maintain group membership. The ACL Manager implements ACL semantics in user space, bypassing kernel-version limitations and rejecting unauthorized access to shared file systems during authorization.
The trade-off is additional user-space authorization overhead, which we mitigate with a three-level inode cache.

Access methods: S3 Gateway and a mountless SDK

In addition to POSIX mounts, we provide two access paths: an S3 Gateway built on the MinIO plugin layer and an in-process SDK for direct data access.

Some applications already use the S3 protocol and need to keep their existing interfaces while accessing the same data. The S3 Gateway translates object-protocol requests into file-system operations. It shares the same file-system namespace with POSIX mounts and the SDK, reusing the same permissions, caches, and underlying data.

The plugin layer implements directory-level tenant isolation and user-token validation, and fixes slow paginated queries in ListObjectsV2. We also removed support for non-S3 backends to create a lightweight image. For cross-cloud dedicated-network scenarios, we provide a lightweight SDK for read-side access via the S3 protocol.

The other access path targets highly concurrent training workloads. As the number of jobs and dataloader processes grows, the number of simultaneously active client data connections can reach the millions. FUSE access requires requests to pass through the kernel, while short-lived jobs repeatedly mount and unmount the file system. The resulting connection and mount overhead accumulates.

To address this, we developed an in-process Python SDK that reads data directly without requiring a mount. Connections are created and exit with the process. The trade-off is that the SDK must manage process connections, sessions after fork, and heartbeats itself.

The SDK starts with the container and integrates with existing data-loading pipelines through fsspec and ffrecord. When dataloader processes are forked, each child process recreates its session and heartbeat thread and reports metrics independently. During operation, the SDK periodically reports metrics and receives heartbeat intervals and rate-limit policies from the server.

Training workloads use read-only access, and write operations are blocked at multiple layers. Path validation and cross-bucket isolation further prevent unauthorized access.

image: Mountless direct-read architecture for the Python SDK
caption: Mountless direct-read architecture for the Python SDK

Metric observability and client upgrades

Each of hundreds of thousands of clients can generate hundreds of metrics. If Prometheus scraped every client individually, the number of time series would grow rapidly.

Instead, clients report metrics to MetricManager in shards, and Prometheus scrapes the aggregated results. MetricManager controls time-series cardinality by merging or dropping some fine-grained labels. The trade-off is that some observability dimensions are no longer available.

The aggregation service must also support hot configuration updates, recovery from failures, dropping data under overload, and expiration-based cleanup. Otherwise, the service itself could become a bottleneck.

Rolling out upgrades across hundreds of thousands of clients must also minimize disruption to training workloads. The Gateway and file-system client (FS) images are produced through the same build pipeline. Combined with stateless clients and rolling restarts in K8s, this supports staged upgrades.

Upgrades to the direct-read SDK and S3 Gateway are transparent to training jobs. For POSIX mounts, the disruption window can be kept under control.

We are evaluating several general-purpose improvements for potential upstream contributions, including batched cache APIs, streaming scans through ReaddirFrom, and dynamic read-ahead. Features more tightly coupled to our application requirements, such as ACLs and the end-to-end data governance workflow, will continue to evolve internally.

Conclusion

Our experience managing EB-scale capacity, hundreds of billions of files, and hundreds of thousands of clients comes down to four principles.

Distribute the workload to eliminate centralized bottlenecks. We distribute metadata reads across follower replicas, eliminate the extra TSO round trip for frequent point lookups, and optimize hot keys to prevent individual nodes from becoming overloaded. We also customize the metadata access path to improve overall performance. When designing the architecture, we must account for the impact of failures in components that carry global workloads.
Govern the data lifecycle with an end-to-end workflow. Dashboards support deletion decisions by application owners; a seven-day trash and PB-scale recovery protect against accidental deletion; GC reclaims space; and leak-check audits orphaned objects. Recovery throughput must keep pace with deletion volume so that reclamation anomalies remain within an acceptable space-reclamation delay, reducing both accidental data loss and the risk of permanent storage leaks.
Protect training performance through tiered caching and bounded prefetching. L1 client caching, L2 P2P caching between clients, and the centralized L3 cluster absorb load at progressively broader sharing scopes. Prefetching is constrained by global limits and queue watermarks. When queues fill, requests are dropped rather than allowed to slow normal reads through additional read amplification. Data locality and hot/cold tiering further reduce cross-cloud access costs and EB-scale storage expenses.
Limit the impact of individual tenants through access control and resource budgets. ACLs define the scope of access, while token buckets constrain each tenant's resource consumption in both the metadata and data planes. Declarative RU-based resource budgets make these limits manageable and help prevent a single client from destabilizing the shared cluster.

As data volumes, compute capacity, and application complexity continue to grow, we’ll keep improving the metadata engine, data plane, performance engineering, governance mechanisms, and observability. Our goal is to prepare the architecture for the next order-of-magnitude increase before it arrives. Identifying potential bottlenecks early and addressing them proactively will remain a central focus of our work.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

How Zhuoyu Built a Petabyte-Scale Data Platform for Autonomous Driving with JuiceFS

Thu, 29 Oct 2026 05:36:00 +0000

Founded in 2016, Zhuoyu Technology (ZYT) focuses on the research, development, and large-scale application of mobile intelligence technologies. Leveraging our proprietary native multimodal foundation models and generative AI technologies, we build a unified mobile intelligence technology base that can efficiently adapt to multiple scenarios. It not only provides automakers with open, customizable, production-oriented intelligent driving assistance solutions (covering L2 and above), but also delivers underlying technology for autonomous mobile robots in diverse scenarios, empowering a full-spectrum intelligent mobility ecosystem from the ground to near-space.

As R&D scale expands, training and inference compute are gradually moving toward multi-data-center and hybrid-cloud environments, while data volume and access demands continue to grow. Compute resources can be scheduled on demand, but task startup can still be affected by mount status, origin data fetches, and model loading. How to make data ready in time and reduce compute waiting has become a key goal of storage architecture evolution.

We built a unified data foundation on JuiceFS. Through unified access, multiple storage pools, and shared caching, it supports data ingestion, capacity scaling, and hot data reuse. Combined with proactive warm-up and read optimization, it accelerates data delivery. Today, the online cluster carries 100-PB-scale data, with online read capability reaching the 100 GB/s range. This article covers the architecture evolution and production practices.

The autonomous driving data loop: growing volumes and repeated data consumption

Autonomous driving R&D revolves around a continuous data loop. Data collected by road-testing and production vehicles is filtered and labeled before entering model training. The resulting models then go through evaluation and deployment, while newly identified issues and newly collected data feed into the next iteration. At each stage, how quickly data becomes available directly affects whether compute resources have to wait and, ultimately, how efficiently R&D teams can iterate.

From a storage perspective, autonomous driving R&D is a data-intensive pipeline. Multimodal data continuously flows back into the platform and is repeatedly consumed by different R&D workloads. Data collected by production and road-testing vehicles, together with simulation-generated data, first enters shared multimodal storage. Downstream workloads—including data filtering, labeling, model training, simulation evaluation, and individual development environments—then consume the data.

image: Multimodal data flow R&D
caption: Multimodal data flow and consumption in autonomous driving R&D

As data volumes and iteration frequency increase, storage must support both continuous capacity growth and efficient data access to keep downstream workloads supplied with data.

Coupling between access, operations, and scaling in the original architecture

Before adopting JuiceFS, we used both S3 APIs and file-based access. Multiple access paths coexisted, storage configuration was scattered across applications, and compute-side teams were responsible for maintaining mount clients.

image: Multiple storage access paths and distributed configuration
caption: Multiple storage access paths and distributed configuration before the migration

We identified three core challenges:

Mounting coupled with workload scheduling: Task startup depended on mount clients being ready. Maintaining those clients on the compute side increased operational overhead and could leave compute resources waiting for storage.
Storage configuration coupled with applications: Storage details such as buckets, endpoints, and credentials were embedded in application code. Changes to underlying storage resources required application-side changes.
Capacity coupled with bandwidth scaling: Data volumes continued to grow while workloads repeatedly accessed hotspot data. Local caches could not be reused across nodes, making it difficult to scale storage capacity and read bandwidth independently.

Therefore, the new architecture needed to simplify application integration and mount management while allowing storage capacity and read bandwidth to scale independently.

Architecture upgrade: building a unified data platform on JuiceFS

Technology selection: S3 as the primary interface, JuiceFS as the storage foundation

To maintain compatibility with existing workloads while supporting future growth, we chose JuiceFS as the storage foundation for four main reasons:

Multiple access protocols: JuiceFS supports interfaces such as S3 and POSIX, allowing different workloads to use the access method that best fits their needs.
Capacity scaling: JuiceFS uses object storage as the underlying data store, providing room for continued data growth.
Cache acceleration: The Community Edition already provides local caching, giving us a foundation for further read-path optimizations.
Open source ecosystem: An active community makes it easier to continue improving the platform based on production requirements.

image: 1JuiceFS 架构图.drawio
caption: JuiceFS Community Edition architecture

Unified access layer: decoupling applications from storage

In the new architecture, applications access storage through an SDK provided by our internal data platform. The SDK provides a unified logical namespace and handles the mapping between logical paths and underlying storage locations and credentials. Applications integrated through the SDK primarily use S3 access, reducing the need to manage mount clients on compute nodes. Existing workloads can continue to use the POSIX interface.

image: Unified access layer and logical namespace based on the SDK
caption: Unified access layer and logical namespace based on the SDK

Using S3 as the primary access path also makes it easier to implement traffic isolation, monitoring, and auditing at the access layer.

The unified access layer separates application changes from storage changes, reducing the impact of underlying resource changes on application integration. We can continue scaling capacity, adding caches, and integrating external data sources while keeping the interfaces exposed to applications stable.

Building the core capabilities

We extended the JuiceFS Community Edition with multiple storage pools, distributed caching, and Under File System (UFS). The resulting architecture is shown below.

image: multiple storage pools, distributed caching, and UFS
caption: Architecture integrating multiple storage pools, distributed caching, and UFS

Multiple storage pools: scaling capacity and isolating failures

We added routing to the data layer of a JuiceFS volume. Based on policies such as file size and configured weights, the router distributes writes across different storage pools. The backends can be multiple S3 buckets or file storage systems.

image: multiple storage pools in a single volume
caption: Write routing and capacity scaling across multiple storage pools in a single volume

This gives us a one-to-many relationship between a JuiceFS volume and its underlying buckets. When an object storage resource reaches its scaling limit, we can add another storage pool to handle subsequent writes without migrating the existing data as a whole. Multiple independent storage pools also help reduce the impact of failures within any single resource domain.

Distributed caching: reusing hot data across nodes

Initially, we used local caching, which prevented nodes from sharing cached data. When a workload was scheduled to a new node and the required data was not available locally, the node had to fetch the data from the source again and keep another copy in its local cache. This increased read latency and reduced cache utilization.

To address this, we developed distributed caching of the Community Edition. Multiple clients can reuse hot data in a shared cache, while object storage remains the authoritative source of data. The read path works as follows:

The client uses consistent hashing to select a cache node.
If the data is available on the cache node's local disk, the node returns it directly.
If the data is not cached and fetching from the source is allowed, the cache node reads it from S3, returns the data to the client, and keeps a copy locally.

Local client-side caching remains available, giving us a multi-level caching architecture together with distributed caching.

image: Distributed cache read path
caption: Distributed cache read path

UFS: integrating external data and accelerating access

We also developed UFS of the Community Edition to bring external data sources into the JuiceFS namespace. UFS provides unified access to external data and accelerates reads through caching, while the authoritative copy remains in the original system.

For example, suppose an existing dataset is stored in S3 and we want to accelerate access using the JuiceFS cache. We can map the dataset to a path in JuiceFS without migrating the original data. Then, applications can access the data through the mapped path.

image: unified namespace with UFS
caption: Mapping external data sources into a unified namespace with UFS

The current implementation uses read-only mappings. It supports the same backend storage types as the Community Edition. Metadata can be loaded on demand or warmed up in advance. Even without warm-up, JuiceFS can retrieve directory and file metadata from the external storage as needed when a directory is accessed.

Production at hundreds-of-petabytes scale: architecture and optimizations

Our production clusters now manage hundreds of petabytes of data, deliver hundreds of GB/s of bandwidth, and provide petabyte-scale cache capacity. To improve data preparation and workload read performance, we optimized cache orchestration, model delivery, network transport, and data formats.

Cache orchestration: building caches as data arrives and proactively warming hot data

Based on statistics collected through our SDK, we found that roughly 80% of the data is accessed within eight days after it arrives. Therefore, we build distributed caches asynchronously as data is ingested and plan cache capacity and TTLs based on access patterns, so that workloads can read data from the cache. Cached data is reclaimed when it expires or when cache capacity becomes insufficient.

image: Cache lifecycle driven by hotspot windows and TTLs
caption: Cache lifecycle driven by hotspot windows and TTLs

For workloads that need explicit control over data preparation, we introduced Fluid for proactive orchestration. A Dataset describes the dataset and its source, while applications access the data through the corresponding Kubernetes PVC.

As part of the workload workflow, DataLoad works with the JuiceFS Runtime to warm the data. The application starts consuming the data only after it’s ready. The Fluid Controller coordinates resources and state throughout the process, managing the lifecycle from cache creation and application consumption to cache reclamation. For workloads that are not integrated with Fluid, we also provide APIs that allow applications to control cache warming and eviction themselves.

image: Fluid-based cache warm-up and lifecycle orchestration
caption: Fluid-based cache warm-up and lifecycle orchestration

Inference workloads: large-scale model delivery

For inference workloads, data preparation primarily means delivering models. An inference Pod cannot start serving requests until its model has been loaded. Previously, Pods loaded models directly from the model repository at startup. As the number of nodes increased, the repository had to handle a large number of repeated reads. Therefore, we placed a cache layer between the model repository and inference nodes.

With this cache layer, models are first warmed up into the distributed cache and then distributed over the cache network to the local caches on inference nodes. Requests for the same data block are routed to the same cache node through consistent hashing, allowing the cache layer to aggregate I/O and reduce repeated reads from the source.

When multiple inference Pods load the same model, the shared cache can aggregate their read requests and keep the amount of data fetched from the model repository close to the size of a single copy of the model. In our internal inference workloads, this approach improved deployment speed by more than 10× compared with the previous architecture.

image: Inference model delivery through shared caching
caption: Inference model delivery through shared caching

Training workloads: TCP zero-copy optimization

Once data is in the cache, workloads still need to continuously read it over the network. To reduce the overhead of this path, we optimized data transfer between clients and cache nodes, starting with TCP because it applies to a broad range of environments.

In the conventional read path, ReadAt copies data from kernel space into a user-space buffer, and then conn.Write copies it back into the kernel for transmission. When the requested data is available in cacheFile on the server and the requested range is valid, we can use io.CopyN → sendfile(2) to send the file data directly to the TCP socket. This eliminates this user-space copy.

image: Zero-copy TCP send path for cache hits
caption: Zero-copy TCP send path for cache hits

To evaluate the performance of the optimized cache path, we ran a 1 MiB sequential-read benchmark. The combined bandwidth across multiple servers was 1,200 Gbit/s. As client load increased, measured throughput reached approximately 135–138 GiB/s. At around 135 GiB/s, P99 latency for 1 MiB requests was approximately 2 ms. When throughput increased further to around 138 GiB/s, P99 latency rose to approximately 20–30 ms. This shows that as throughput approaches the bandwidth ceiling, a trade-off remains between throughput and tail latency.

image: Sequential read benchmark
caption: Sequential read benchmark

For the throughput-oriented MLPerf Storage UNet3D workload, we first used a high-density single-client topology. We concentrated the concurrent read load from eight accelerator ranks on a single node with a single 200 Gbit/s data ingress, allowing us to measure the data delivery ceiling of a single client. The tests covered three accelerator configurations: H20, H100, and B200. The H100 and B200 tests reached the limits of the 200G network and the client read path, shifting the bottleneck from backend storage to the client data ingress. In the next phase, we’ll extend the UNet3D training workload to multiple clients and combine it with RDMA to further evaluate sustained data delivery and accelerator utilization for large-scale, multi-node, multi-accelerator training.

image: Theoretical bandwidth requirements and measured results for MLPerf UNet3D
caption: Theoretical bandwidth requirements and measured results for MLPerf UNet3D

Lance columnar storage: reducing I/O at the data format layer

In addition to improving storage-path throughput, we also reduced the amount of data and number of I/O operations required by workloads by changing the data format.

For example, consider labeling data. A single asset may correspond to multiple JSON files, which were previously packaged into tar archives. Downstream processing still had to parse the JSON frame by frame, resulting in repeated serialization and format conversion and generating a large number of small I/O operations.

We switched the labeling data to Lance and flattened commonly used JSON fields into columns. Data is passed in memory through Arrow batches and written to Lance tables in batches. Downstream workloads select only the required columns and read them in batches, reducing unnecessary reads, per-frame JSON processing, and repeated serialization.

image: Per-frame JSON processing vs. batched data flow with Lance
caption: Per-frame JSON processing vs. batched data flow with Lance

The benefits vary across pipelines. I/O time decreased by approximately 20%–90%, IOPS requirements decreased by approximately 90%, and bandwidth consumption decreased by more than 60%. These improvements significantly reduced the demand that application workloads placed on storage resources.

Combining Lance with JuiceFS for targeted data warm-up

Column-based reads also allow us to make data warm-up more targeted. We use layout metadata such as the Lance Manifest to locate the physical byte ranges of the selected columns. Then, we map those ranges to the corresponding JuiceFS data blocks and warm only the required blocks, reducing unnecessary reads.

image: JuiceFS data blocks for targeted warm-up
caption: Mapping Lance columns to JuiceFS data blocks for targeted warm-up

These capabilities have been merged into JuiceFS Community Edition: byte-range data warm-up (PR #7398) and a tool for parsing Lance datasets (PR #7399). Together, they can generate a list of files and byte ranges based on a specified dataset version and selected columns, which can then be used for targeted data warm-up.

Conclusion

Looking back at this evolution, we built a unified data platform on JuiceFS and gradually improved it along three paths:

A stable access layer: The unified access layer abstracts differences between underlying storage systems. This minimizes the impact of storage scaling and deployment changes on application integration.
Shared hot data: Distributed caching allows data to be reused across nodes. This reduces repeated reads from the source when workloads move between nodes, clusters scale out, or multiple copies of the same data are accessed.
Proactive data preparation: Fluid orchestration and APIs allow data to be prepared before workloads consume it. This connects workload execution to the state of data preparation.

Together, these three approaches support data delivery at the hundreds-of-petabytes scale and performance requirements of hundreds of GB/s and above, while reducing the operational overhead and coordination required between storage and application teams. We continue to optimize cache orchestration, network transport, and data formats to make data available faster and reduce compute wait time, providing a more reliable storage foundation for autonomous driving R&D.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Scaling Data Management for Agent Sandboxes with JuiceFS

Thu, 22 Oct 2026 05:57:00 +0000

Recently, while working with cloud providers and agent teams to deploy sandboxes, we found that beyond getting data into the sandbox, we also need to consider how the data required by tasks and the data generated during execution can be continuously used across tasks and stages. Sandboxes can be quickly created and destroyed with tasks, but data often needs to be retained, shared, and transferred.

As sandbox concurrency scales further, the challenges go beyond data persistence and shared access. Operational issues such as client scale, cache reuse, small files, and fault isolation become more prominent.

This article combines our practice in agentic reinforcement learning (RL) scenarios and JuiceFS deployment experience in cloud provider sandboxes to introduce how sandbox data is organized, accessed across clouds, and mounted. We’ll also discuss the problems and optimization directions encountered under large-scale concurrency.

How to manage long-lived data in short-lived sandboxes

Take agentic RL as an example. A task typically goes through task distribution, sandbox execution, trajectory collection, reward evaluation, and training optimization. Data generated during task execution continues to be used by subsequent stages.

First, the compute lifecycle and data lifecycle separate. When an agent task starts, it needs to read datasets, code repositories, dependency environments, and model weights. During execution, it continuously produces trajectories, logs, intermediate files, and evaluation results. The sandbox can be destroyed after the task ends, but some of this data still needs to be retained for replay, analysis, evaluation, and subsequent training.

Second, data sharing and task isolation must coexist. Each sandbox typically needs an independent working directory and access boundary, but different agents, evaluators, or training pipelines may access the same dataset, code repository, or reuse results from previous tasks.

Third, data needs to flow across different compute environments. When tasks are distributed across self-built environments and sandboxes from different cloud providers, maintaining independent data copies in each environment not only increases synchronization and management costs but also limits the flexibility of task scheduling.

How JuiceFS manages and accesses sandbox data

To address these data management issues, we introduce how JuiceFS organizes and accesses data in actual sandbox environments.

Organizing shared data and task data with directories

Different customers adopt different data organization methods based on their application characteristics. Based on the usage patterns of existing customers, a typical approach is:

Datasets directory: stores core data such as evaluation datasets, training datasets, and benchmark task inputs.
Repo directory: stores code repositories, toolkits, and snapshots of related dependency environments.
Task directories: based on different tasks, runtime IDs, and training versions, independent directories are created in the file system to store data generated during each task run.

In this way, public data such as datasets and repos can be shared and accessed by multiple tasks, while each task has its own independent data space. At the same time, enterprises can decide which data needs to be retained long-term and which can be cleaned up after task completion, enabling more flexible data lifecycle management.

How cross-cloud sandboxes access the same data

Some customers use sandbox services from different providers across multiple cloud platforms simultaneously for reasons such as security, load balancing, and resource scheduling.

image: Multi-cloud compute access
caption: Multi-cloud compute access

Our customers' core data (such as training sets and inference data) is usually stored in their own data platform or file system, while sandbox environments from different cloud providers run on their respective cloud infrastructures. The challenge is how sandboxes on different clouds can access the same data without maintaining separate data copies.

In such scenarios, JuiceFS Enterprise Edition's cross-cloud data access capability can be used to keep unified data on the customer's own platform while allowing sandboxes in different cloud environments to access it on demand. With fine-grained permission control, data isolation can be achieved between sandboxes, and resources such as datasets, code repositories, and evaluation data on different clouds can be managed within a unified namespace.

For example, after data is written from cloud A, a sandbox in cloud B can quickly read it and continue the subsequent task flow. This improves the flexibility of task scheduling and reduces the complexity of data management in multi-cloud environments.

JuiceFS data mounting mode for strongly isolated sandboxes

To enable sandboxes to actually access this data, we also need to solve the problem of how to mount the file system into isolated environments.

Currently, sandbox solutions provided by cloud providers are different somewhat from self-built sandboxes in enterprises. From the existing cloud sandbox architectures, most are based on lightweight virtualization technologies such as Firecracker, with each cloud provider optimizing to different degrees.

The core characteristic of such sandboxes is that they create a strongly isolated environment inside a virtual machine. This keeps it isolated from the host and other sandbox instances. Therefore, under this strong isolation architecture, data access cannot simply rely on host shared directories.

To address this, the current approach is to run an independent JuiceFS client alongside each sandbox. The client is launched with the sandbox through a sidecar-like mechanism provided by the cloud provider and exposes a local mount path to the application sandbox.
In this mode, each sandbox connects to the backend JuiceFS service with an independent identity, accesses the metadata service deployed in the customer environment through the VPC network, reads data from the corresponding object storage, and mounts shared or exclusive directories as needed by the task. For data that needs to be shared across multiple tasks and components, a shared directory enables unified access, while data unique to each task is isolated and protected through independent subdirectories, ensuring data security.

For permission management, the token access control capability provided by JuiceFS Enterprise Edition can further enable fine-grained permission management, ensuring secure and controllable data access between different sandboxes.

JuiceFS in cloud provider sandboxes

We’ve implemented JuiceFS in sandbox environments on Tencent Cloud and Alibaba Cloud.

Tencent Cloud: data mounting methods in agent runtime

Tencent Cloud uses the agent runtime mechanism, with three levels of isolation: file system level, instance level, and storage level.

The first level is mounting block storage inside the sandbox. The block storage is exclusively used by a single sandbox, and the sandbox creates an independent file system on it. In this case, other environments cannot access this file system, achieving strong data isolation.
The second level is mounting storage services provided by Tencent Cloud itself. Through mount, the corresponding storage can be mounted into a specific sandbox environment, which is essentially mounting inside the sandbox.
The third level is the JuiceFS file system or subdirectory-level isolation. JuiceFS can mount the entire file system to a sandbox or only a specified subdirectory (subpath). Multiple sandboxes can share the same JuiceFS file system while dividing their data access scopes through different subdirectories, striking a balance between data sharing and task isolation.

image: Tencent Cloud agent sandbox file isolation solution
caption: Tencent Cloud agent sandbox file isolation solution

Alibaba Cloud: FC and ACS

Alibaba Cloud has two product lines:

The Function Compute (FC) Sandbox: Its implementation is similar to Tencent Cloud, also using a sidecar-like process mode to start a client in each sandbox to complete JuiceFS mounting.
The Container Compute Service (ACS): It also provides sandbox services but with a different implementation. ACS integrates JuiceFS mounting capability into its driver component. In actual use, customers decide whether and how to use this capability. The related code is also planned to be open-sourced in its components.

image: Alibaba Cloud FC agent sandbox
caption: Alibaba Cloud FC agent sandbox

Currently, whether on Tencent Cloud or Alibaba Cloud, JuiceFS mounting capability supports both Community Edition and Enterprise Edition.

Sandbox lifecycle management and mount control

In the current sidecar-like client mode, JuiceFS mounting is completed during sandbox startup, and the entire mount lifecycle is managed by the sandbox platform. For example, in Alibaba Cloud, the FC management platform is responsible; in Tencent Cloud, the agent runtime platform takes on this role.

Each sandbox corresponds to an independent JuiceFS client, which is created with the sandbox and completes directory mounting. The token, mount parameters, cache directory, and other configurations required during mounting are uniformly injected and managed by the runtime platform.

Under this mechanism, there are some differences between the Enterprise Edition and the Community Edition. JuiceFS Enterprise Edition provides more comprehensive management through the console and token capabilities, and cloud provider platforms have largely integrated these capabilities into their own mount management processes.

For the application side, the sandbox only needs to see the mounted data directory and does not need to directly manage the JuiceFS client. The creation, configuration, mounting, and exit of the client are all handled by the sandbox platform.

Challenges and optimization in large-scale agent sandbox scenarios

Client scaling bottleneck

Currently, one sandbox corresponds to one independent JuiceFS client, and each client is considered an independent client instance in the Enterprise Edition. As the number of sandboxes grows, the number of clients and the corresponding processes, connections, and metadata requests also increase.

However, most cloud provider sandbox implementations are based on Firecracker virtualization, and native Firecracker does not support passthrough mounting like VirtIO-FS. The number of clients a single file system can support is around 100,000. We’ve already received requests from some customers for millions or even more sandboxes. This makes client architecture optimization an important direction for future evolution.

One direction being explored is introducing a Mount Pod-like mode, where a JuiceFS client runs on the host or a designated node, and multiple sandboxes share the mounting capability. This can reduce the number of client instances and resource consumption, and further increase the supported sandbox scale.

Cache reuse and data loading efficiency

Currently, one sandbox corresponds to one independent JuiceFS client, and sandboxes may be distributed across different nodes. This makes it difficult to fully reuse client cache across tasks. Without a shared cache layer, a large amount of repeated data reads and loading pressure is directly transmitted to object storage; for cross-cloud sandbox scenarios, it may further increase bandwidth pressure on cross-cloud links.

To address this, we plan to combine task scheduling, host disk resources, and JuiceFS caching capabilities to improve cache reuse efficiency for hot data. For example, schedule related tasks to nearby nodes as much as possible, and use host disks to build a distributed cache, bringing frequently accessed data such as model datasets as close to compute as possible and reducing repeated loading from object storage.

When using cloud provider sandboxes, it’s necessary to combine the cache resources the cloud provider can offer; for self-built sandboxes, planning can be based on the customer's own infrastructure to balance performance, cache capacity, and resource cost.

Small files and metadata performance

During agent execution, data such as logs and trajectories often manifests as continuous small I/O writes, large numbers of small files, and frequent flushes. This places higher demands on JuiceFS metadata and write performance.

For such scenarios, we’ll work with application teams to explore optimization strategies, such as evaluating whether large numbers of small I/Os can be merged into larger-grained write requests, or using batch writes to reduce frequent small file operations. Adjusting the data write pattern on the application side can also help reduce pressure on the underlying metadata service.

Fault isolation and platform governance

In large-scale concurrency scenarios, it’s necessary to prevent a single abnormal task from affecting shared caches, metadata services, or object storage. For example, when abnormal access generates a large number of requests, its impact scope must be controlled as much as possible to avoid affecting other tasks.

If shared caches, shared clients, or client pools are further introduced, the original isolation boundaries will also change. After multiple sandboxes share underlying resources, fault isolation, resource limits, access permissions, and lifecycle management of shared clients need to be reconsidered.

On the other hand, as more platforms treat sandboxes as a public service for different teams or customers, data management will further extend to multi-tenant scenarios. Clear data and permission boundaries must be maintained between tenants, and different types of data require corresponding lifecycle policies.

Observability also needs to be improved accordingly, not only to observe client and cache status but also to understand overall storage usage, the running status of shared resources, and the scope of abnormal impact. This will provide a basis for subsequent resource governance.

From current practice, sandboxes are not difficult to create and destroy quickly; the real challenge is that the data behind them cannot disappear with them. Trajectories, logs, evaluation results, and training samples still need to be used by subsequent tasks, which makes storage gradually evolve from "providing a directory for the sandbox" to participating in the entire agent data pipeline over the long term.

As the scale grows from hundreds of thousands to millions, designs that were once simple and straightforward, such as sandbox clients and independent caches, also need to be reconsidered. We are still validating different approaches for client reuse, caching, and isolation, and we welcome exchanges with teams working on agent sandboxes, agentic RL, or related infrastructure about the problems they encounter.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

How to Access JuiceFS via S3 and WebDAV Protocols

Thu, 15 Oct 2026 01:11:00 +0000

Different applications access data in different ways: some rely on POSIX file interfaces, some are built on the S3 API, and some users want to browse and edit remote files through a file manager. JuiceFS' multi-protocol access capability is designed precisely to let these tools access the same file system.

In this article, we'll explain the shared initialization path of S3 Gateway and WebDAV, describe their applicable scenarios and common mappings, and dive into their implementation principles and boundaries.

A shared initialization path

JuiceFS consists of a metadata engine and object storage:

The metadata engine manages directories, attributes, locks, and sessions.
Object storage stores the actual file data blocks.

image: 1JuiceFS 架构图.drawio
caption: JuiceFS Community Edition architecture

From an implementation perspective, S3 Gateway and WebDAV first go through the same service initialization path. The service process creates a JuiceFS client based on the metadata address, listen address, and common parameters such as subdir, read-only, cache, limits, access-log, and metrics. Then, it initializes the metadata client, object storage, data block cache, and VFS configuration in sequence, ultimately obtaining an fs.FileSystem.

image: Shared service initialization path of S3 Gateway and WebDAV
caption: Shared service initialization path of S3 Gateway and WebDAV

In other words, the common foundation of the two is not a FUSE mount point, but a native JuiceFS client with sessions, caching, metrics, and background tasks. Before S3 requests and WebDAV requests enter their respective protocol adaptation layers, they already share the same JuiceFS file system and the same file system namespace.

Applicable scenarios and common mappings

The shared initialization path explains how the two connect to the same file system. In actual use, what matters more is choosing the access method: which scenarios are suitable for S3 Gateway, which are suitable for WebDAV, and how common requests map to JuiceFS operations.

Linux applications, training jobs, or workloads requiring full file system semantics should prefer FUSE/CSI.
Applications already built on the S3 SDK, AWS CLI, MinIO Client, or object storage toolchains should prefer S3 Gateway.
Desktop file manager access, DAV/rclone synchronization, Office/apps collaboration, and simple sharing should prefer WebDAV.

Mapping of common S3 Gateway APIs to JuiceFS file system operations:

S3 API	Purpose	Equivalent file system operation
GetObject	Download an object	open/read
PutObject	Upload or overwrite an object	create/write/rename
HeadObject	Query object metadata	stat/getxattr
DeleteObject(s)	Delete one or more objects	unlink/remove
ListObjects V1/V2	List objects by prefix	readdir+filter
CopyObject	Copy an object server-side	copy/read+write
MakeBucket	Create a bucket	mkdir
DeleteBucket	Delete an empty bucket	rmdir
NewMultipartUpload	Create a multipart upload session	create
PutObjectPart	Write a part	write
CompleteMultipart	Merge and commit the object	copy + atomic rename
Tags/Metadata	Read/write tags and metadata	get/setxattr

Mapping of common WebDAV methods to JuiceFS file system operations:

HTTP method	Purpose	Equivalent file system operation
GET	Download a file	read
PUT	Upload or overwrite a file	write/create
DELETE	Delete a file or directory	remove
MKCOL	Create a directory	mkdir
PROPFIND	Query properties or list a directory	stat/readdir
PROPPATCH	Modify custom properties	setxattr
COPY	Copy a file or directory	cp
MOVE	Move or rename	mv/rename
LOCK	Lock resources	File lock
UNLOCK	Unlock resources	Release a file lock
OPTIONS	Query server capabilities	Capability negotiation
HEAD	Query HTTP metadata only	Lightweight stat

S3 Gateway principles: from object semantics to file system operations

S3 Gateway converts S3 protocol requests into JuiceFS file system operations. This section covers four aspects:

The request path
Internal state
Key semantic conversions
Permission coordination

Request path: from the MinIO S3 server to `jfsObjects`

The MinIO S3 server handles the S3 protocol layer, including request routing, signature verification, authentication and authorization, and multipart upload orchestration. After requests pass through the protocol layer, they enter the ObjectLayer implemented by JuiceFS, namely jfsObjects. jfsObjects converts object-level reads, writes, listings, deletions, and attribute access into JuiceFS file system calls.

image: Call relationship among
caption: Call relationship among MinIO S3 server, jfsObjects, and JuiceFS file system

State storage: how the internal namespace carries the Gateway state

During request conversion, the Gateway does not only access user-visible object paths; it also uses an internal namespace to store state needed by the service itself. For example, .sys/tmp is used for temporary writes, .sys/uploads is used for multipart upload state, and .minio.sys is used to store MinIO/IAM-related metadata.

These internal paths are not application data in the ordinary user's understanding, but they are the foundation the Gateway needs to maintain upload processes, part states, and permission configurations.

Semantic conversion: `PutObject`, `Multipart`, `Bucket`, and `Metadata`

The S3 object model and the file system model are not completely the same, so the Gateway needs to perform semantic conversions at several key points.

PutObject must ensure that an object is visible only after the upload is complete. To prevent clients from reading incomplete objects, JuiceFS S3 Gateway writes to a temporary file and saves object attributes such as metadata and tag, commits to the target path via rename after completion. Leveraging the atomicity of file system rename, the Gateway can switch an object write from “uncommitted” to “committed.”

image: PutObject write flow
caption: PutObject write flow: from temporary files to atomic rename commit

Multipart upload needs to merge multiple parts into the final object. JuiceFS S3 Gateway can use file system capabilities such as CopyFileRange to combine already written parts and commit the final file via rename. This mainly adjusts file system metadata relationships rather than moving all data again.

image: Part merging and final commit flow
caption: Part merging and final commit flow of multipart upload

Bucket mapping handles the relationship between the object namespace and file paths. In single-bucket mode, the file system name corresponds to the bucket; in multi-bucket mode, top-level directories map to buckets. Object attributes such as metadata, tag, and ETag are stored through file extended attributes (xattr).

Permission coordination: IAM and multiple instances

S3 Gateway also needs to handle identity and permission information, including root users, regular users, policies, service accounts, and more. These configurations are stored in the JuiceFS file system through internal paths, allowing them to be reused across multiple Gateway instances.

Therefore, multiple Gateway instances can share the same set of IAM configurations; however, each instance caches IAM information, so permission changes are not synchronized instantaneously. In actual deployments, we need to consider the refresh cycle.

WebDAV principles: from file protocol to JuiceFS operations

WebDAV starts from a different point than S3 Gateway. It targets file access scenarios, with common clients including macOS Finder, DAV/rclone, and some Office/apps clients. Since the WebDAV protocol includes concepts such as files, directories, and attributes, it does not need to first handle the model conversion between buckets, objects, and keys on the one hand and paths and files on the other, unlike S3 Gateway.

Request path: from HTTP entry to `webdavFS`

After a WebDAV request enters JuiceFS:

1.It passes through net/http and indexHandler. This layer mainly handles HTTP entry-related capabilities, such as authentication, transport, and conversion of directory access requests.

2.The request enters x/net/webdav.Handler, which handles the WebDAV protocol layer and converts protocol methods into file access requests that webdavFS can process.

image: WebDAV request path
caption: WebDAV request path: from HTTP entry to JuiceFS file system operations

3.After entering webdavFS, the file access requests passed in by the Handler are converted into JuiceFS file system calls such as open, create, read/write, rename, delete, and attribute setting. Since the WebDAV protocol is close to file system semantics, the focus here is not to rebuild an object model, but to handle how request methods, file open flags, and attribute states map onto JuiceFS operations.

State boundaries: dead properties and LOCK

WebDAV adaptation involves not only data reads and writes but also the protocol's own state. Dead properties can be saved as JSON in xattr, thus landing in JuiceFS' file attribute system; however, LOCK state is different—it's currently kept in the service process memory. Single-instance deployments usually have no problem; in multi-instance deployments, different instances cannot automatically synchronize lock state.

These limitations usually do not affect Finder browsing, simple sharing, or DAV/rclone synchronization. However, if an application strongly relies on WebDAV file lock semantics, deployment mode and client behavior need to be verified separately.

S3 Gateway and WebDAV share the same JuiceFS client initialization path and access the same file system namespace. The difference is that S3 Gateway needs to convert object semantics into file system operations, while WebDAV is closer to file access semantics.

Therefore, the choice of access method should ultimately depend on the application itself: whether it organizes data as objects or operates on data as files and directories.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Nydus + JuiceFS: Reducing Container Startup Time for AI Inference from 116s to 1.4s

Fri, 09 Oct 2026 06:56:00 +0000

In large-scale AI inference services, when a cluster scales out, newly added inference instances must go through a series of cold-start steps before they can serve requests: container image preparation, file system mounting, runtime and inference framework initialization, and model weight loading. As image sizes and model weights continue to grow, the time spent on data preparation becomes increasingly prominent. During large-scale scaling events, many instances start simultaneously, further increasing the concurrent pressure on the registry, network, and backend storage.

To address these issues, the industry has developed several optimization strategies, including reducing image size, using P2P distribution to alleviate concurrent pull pressure, and employing on-demand loading to reduce data transfer on the critical startup path.
In this article, we’ll explore a shared‑storage‑based approach: combining Nydus’ on‑demand image loading capabilities with JuiceFS’ distributed storage and caching mechanisms. By sharing storage and reusing cached data, this approach reduces redundant data reads during large-scale scale-outs and shortens container startup time. For AI inference scenarios, container images and model weights can be managed together under a unified storage, caching, and prefetching system, accelerating both types of data while reducing the complexity of building and maintaining two separate data pipelines.

Why OCI image distribution is slow

Traditional Open Container Initiative (OCI) images are typically organized as compressed tar layers. In the image‑pull mode, a node must fully download and extract the required layers before the container can start. In practice, however, only about 6%–20% of files in an image are actually accessed during container startup; the rest of the download and extraction overhead is wasted on non‑critical‑path data.

In large clusters, this “prepare everything first, then start” approach causes three problems:

Long cold‑start times: For GB‑sized images (like AI inference images), full download and extraction significantly delay new instance startup, often taking minutes.
Registry bottleneck: During HPA‑triggered bursts or large‑scale rollouts, hundreds of nodes simultaneously request GB‑sized layers from the registry, easily overwhelming the registry and congesting the network.
Redundant network and storage usage: Without an efficient P2P or distributed sharing mechanism between nodes, the same layers are repeatedly transferred across the cluster, resulting in significant bandwidth consumption and duplicated storage.

Dragonfly+Nydus: what it solves and what it leaves behind

To mitigate registry bandwidth pressure during concurrent image pulls in large clusters, a common solution is to introduce P2P distribution systems like Dragonfly. The core idea is to build a P2P distribution network via a scheduler and node‑side agents, allowing nodes to share image data and reduce egress traffic from the registry.

However, P2P distribution primarily optimizes the data transfer path; it does not change the fundamental OCI mechanism of downloading and extracting layers before starting the container. Nydus, on the other hand, changes the loading approach: it transforms “prepare everything then start” into “fetch lightweight metadata first, and then read actual data on demand,” reducing data transfer on the critical startup path.

Combined, Nydus handles on‑demand loading, while Dragonfly distributes the on‑demand data, reducing both the actual image read volume and the concurrent access pressure on the registry.

However, this combination introduces additional infrastructure and operational complexity. Besides Nydus snapshotter, one must deploy and maintain Dragonfly’s scheduler, distributor, and node‑side components, and handle P2P network connectivity, cache space management, node anomalies, and cache eviction. In large‑scale GPU inference clusters, this means maintaining a separate data distribution and caching system independent of the storage layer.

This leads to an alternative thought: if the cluster already has a mature distributed file system like JuiceFS, could image data distribution and caching be offloaded to the shared storage system, reusing existing storage infrastructure while reducing the cost of building and maintaining extra P2P components?

Architecture redesign: Nydus blobs hosted by JuiceFS

The core idea of the Nydus+JuiceFS solution is to transform image data distribution—originally relying on P2P network scheduling—into shared storage and cache reuse based on a distributed file system. In this architecture, JuiceFS serves as the underlying unified data engine hosting Nydus image blob data, and worker nodes read data through the standard POSIX interface.

image: Nydus +JuiceFS
caption: Nydus +JuiceFS architecture and data flow

After an image is built and converted to Nydus format during the CI/CD phase, it’s split into two parts:

Bootstrap (metadata): Very small (typically a few hundred KB, for example, ~450KB), stored in the Image Registry, used for fast indexing.
Blobs (actual data blocks): Contains all file content (typically several GB, for example, ~1GB), persisted directly to the JuiceFS shared file system after conversion (for example, /var/lib/nydus/blobs).

When a worker node starts a container, containerd triggers nydus-snapshotter, and nydusd mounts the container’s read‑only layer via FUSE. Then, when the application issues actual read requests, nydusd looks up the index in the Bootstrap and reads the corresponding blob data on‑demand through the JuiceFS mount point.

The key change in this path is: the registry only distributes the manifest and the extremely lightweight bootstrap; the massive blob data transfer is entirely handled by JuiceFS’ distributed caching system. This fundamentally eliminates the registry network bottleneck during high‑concurrency pulls.

For AI inference scenarios, this architecture can further unify the storage paths for images and model weights. In addition to Nydus image blobs, model weights can also be stored in JuiceFS, allowing both types of data to reuse the same storage, caching, and prefetching mechanisms. This accelerates data loading while reducing the complexity of building and maintaining two data pipelines.

Comparison basis	OCI full image pull	Dragonfly+Nydus	JuiceFS+Nydus
Core concept	Full layer pull + layer extraction	P2P network offloading + on‑demand image loading	Storage‑layer decoupling + distributed caching + on‑demand loading
Cold start method & time	Full download and extraction required; typically 30s to 10min+	Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds	Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds
Image data transfer path	Registry → worker (full local download & extraction)	Registry / source → Dragonfly P2P network → worker on‑demand read	JuiceFS → worker on‑demand read via POSIX, accelerated by local & distributed caches
Core components	Registry + containerd	Nydus, Manager, Scheduler, Seed Peer, and dfdaemon	JuiceFS + nydus‑snapshotter
Image & model weight synergy	Image pulled from the registry to local node; model weights loaded via a separate path (NAS or object storage); two systems managed independently	Images loaded on‑demand via Nydus, distributed by Dragonfly; model weights still rely on independent external storage	Image blobs and model weights stored together in JuiceFS, sharing the same cache and warm-up mechanisms
Operational complexity	Easy to overwhelm registry bandwidth under high concurrency	Requires extra maintenance of P2P scheduler, node network, and caching system	Can reuse existing JuiceFS storage & cache infrastructure, reducing independent image distribution components
Local disk pressure	High: Stores full image and extracted data	High and unpredictable: Requires cache space for P2P; rapid scaling may cause disk full	Controllable: Unified shared storage pool, uses local NVMe for P2P distributed cache
Network requirements	Standard TCP	Requires widely open P2P ports across nodes; may be invalid in cross‑VPC/NAT environments	Standard LAN – only requires normal high‑bandwidth inter‑node LAN; no intrusive network requirements
Suitable scenarios	General workloads with no cold‑start requirements and small clusters (<20 nodes)	Ultra‑large clusters across data centers / clouds / edge computing environments without unified high‑bandwidth storage; strong ops capabilities of the team	Ultra-large clusters across data centers / clouds / edge computing environments without unified high-bandwidth storage, where JuiceFS is already deployed or can be deployed, aiming for unified image+model management and simplified ops

Runtime mechanism: how Nydus and JuiceFS coordinate

Startup phase: fetching bootstrap (metadata)

When the application starts, nydusd only needs to fetch the lightweight bootstrap file tree from the registry. The bootstrap describes the image file tree and block indexes, but contains no actual file content.

After the container root filesystem (rootfs) is mounted, the subsequent startup process proceeds. In our test scenarios, containers can be ready and pass health checks within hundreds of milliseconds, without transferring the full large image data before startup.

Read phase: on‑demand blob access

When the application process actually issues read operations (such as open() or read()), nydusd captures the I/O requests via the Linux FUSE mechanism and locates the corresponding blob data block and offset using the bootstrap index.

This process is completely transparent to upper‑layer applications. For the underlying data path, the read granularity changes from full image layers to the data blocks corresponding to specific files and offsets. nydusd directly accesses the JuiceFS mount point (for example, /var/lib/nydus/blobs) to read the target block, completely eliminating the runtime fetch‑from‑registry path.

Cache phase: data reuse via JuiceFS cache groups

JuiceFS’ distributed cache groups can reuse already‑cached blob data across the cluster. On a cache hit, data is returned to the kernel at near‑local‑NVMe speed. On a cache miss, only the specified offset block is pulled on‑demand and automatically filled into the cache. In addition to the automatic fill‑on‑miss, blobs can be pre-loaded into the cache to ensure full cache hits during on‑demand reads.

For AI inference scenarios, the same mechanism can also be applied to loading model weight files.

Production test data

In production, we compared the solutions above against a large AI inference image.

Container startup time comparison

The table below shows the test results for container startup time:

Scenario	OCI overlayfs	Nydus+JuiceFS	Comment
Cold pull + run	116 seconds	16 seconds	OCI uses public cloud registry; Nydus+JuiceFS uses JuiceFS + public cloud object storage
Cold pull + run (with distributed cache warm-up)	N/A	~1.4 seconds	Required blobs pre‑loaded in JuiceFS distributed cache
Warm pull + run	~0 seconds	0.22 seconds	OCI: same tag already on node; Nydus+JuiceFS: blobcache local cache

From the test results, we can see:

7x speedup without cache warm-up: When neither local nor distributed cache is warmed up, Nydus+JuiceFS reduces container startup time from 116s to 16s.
~99.9% reduction in data read volume during startup: In the cold pull + run scenario, read volume drops from 11 GB to 11 MB. Traditional OCI requires full download and extraction before startup, while Nydus+JuiceFS only fetches bootstrap metadata and reads blobs actually accessed during startup.
1.4s startup with distributed cache warm-up: When blobs are cached in JuiceFS distributed cache, startup takes 1.4s; with local blobcache warmed up (warm pull + run), startup takes only 0.22s.

On‑demand file read performance

The table below shows test results for: on-demand file read performance:

Scenario	Traditional OCI (local disk direct read)	Nydus+JuiceFS (distributed cache hit)
First‑block read latency (TTFB)	~0.9 milliseconds	~3.2 milliseconds
Small file / scattered configuration loading (IOPS)	Baseline	Close to baseline
Large file sequential read throughput	Local disk limit	GB/s level (JuiceFS NVMe distributed cache)

From the test results, we can see:

Time to first byte (TTFB): On distributed cache hit, first‑block read latency is ~3.2 milliseconds, about 2.3 milliseconds higher than local disk direct read (0.9 milliseconds). The extra latency comes from the FUSE data path and cache access, but remains at millisecond level, generally imperceptible to the application side.
Small file IOPS: Performance is very close to local disk baseline. In scenarios with dense Python dependency and configuration file reads, JuiceFS caching introduces no noticeable IOPS bottleneck.
Large file sequential throughput: On cache hit, Nydus+JuiceFS achieves GB/s sequential read throughput. Compared to traditional OCI, which is limited by single‑node local disk performance, JuiceFS can leverage NVMe distributed cache to provide high‑bandwidth data access.

Summary: combining on‑demand loading with shared cache

Nydus changes image loading from full pull to on‑demand reading, while JuiceFS places data blocks into a shared storage and distributed cache system. Together, they reduce container cold‑start time from 116s to 16s, and to ~1.4s with distributed cache warm-up; the data read volume during startup drops from 11 GB to 11 MB.

The value of this solution extends beyond just image startup acceleration. For AI inference scenarios, image blobs and model weights can be uniformly hosted by JuiceFS, sharing the same caching and prefetching mechanisms. During large‑scale scaling, distributed caching also reduces duplicate reads of the same data across many instances. This lessens concurrent pressure on the registry and backend storage.

In environments lacking shared storage, Dragonfly+Nydus is an excellent P2P alternative. However, in clusters that already have JuiceFS infrastructure, Nydus+JuiceFS delivers extreme image distribution and model loading acceleration with minimal operational overhead.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

JuiceFS Enterprise 5.4: From 100 Billion Files to 1 Million Clients

Tue, 29 Sep 2026 15:07:00 +0000

Following the support for 100-billion-file environments introduced in JuiceFS Enterprise Edition 5.3, v5.4 further improves JuiceFS across several dimensions for large-scale deployments. At this scale, even small per-operation overheads can add up to significant resource consumption. Once metadata is distributed across multiple zones, the system also needs to coordinate across nodes while maintaining performance, consistency, and stability.

To address these challenges, JuiceFS Enterprise Edition 5.4 introduces the following new capabilities and improvements:

Support for up to 1 million simultaneously mounted clients
Fast cloning of directories containing massive files
More than 90% RDMA bandwidth utilization
Up to 100% higher random-read performance under high concurrency
On-demand data synchronization for mirror file systems
More flexible caching strategies for a wider range of workloads
Multiple operational improvements for large-scale data

Supporting 1 million clients

As AI agents move toward large-scale deployment, the JuiceFS team faced a new challenge: supporting connections and concurrent access from up to 1 million clients on top of the metadata workload of a 100-billion-file file system. The team spent months exploring and optimizing the architecture, and introduced this capability in JuiceFS 5.4.

JuiceFS uses a distributed architecture that partitions metadata across multiple metadata nodes, allowing metadata for hundreds of billions of files to be managed across multiple zones. To allow these nodes to handle a large number of client connections at the same time, v5.4 co-locates metadata nodes with network proxies, enabling clients to connect to any metadata node.

The node that receives a request handles requests for its local partition directly. Requests involving other partitions are forwarded internally to the corresponding nodes. Multiple client sessions share the same physical connections between nodes, reducing the number of connections and the associated maintenance overhead.

Connection reuse is only part of the challenge. Querying large numbers of client sessions, reporting session status, and cleaning up expired sessions can also introduce significant overhead.

In v5.4, indexes are used to locate client sessions instead of scanning unrelated sessions. Paginated queries and sampled reporting reduce the overhead of session status management. Expired sessions are cleaned up in batches to prevent cleanup work from becoming concentrated in a single pass.

With these optimizations, JuiceFS 5.4 successfully handled 1 million simultaneous client connections in stress tests. Previously, when clients connected separately to multiple metadata nodes, the total number of connections could reach tens of millions. After optimization, 10 internal proxy nodes handled 1 million client connections, while connection reuse reduced the number of physical backend connections from the proxies to the metadata leader to about 400. This significantly reduced connection-management pressure on the leader.

image: 1-million-client test architecture
caption: 1-million-client test architecture

Fast cloning for large directories

When creating branches of training data or preparing test environments, users often need an independently modifiable copy of a directory. As directory sizes grow, cloning becomes more complex: in a multi-zone architecture, the metadata for a single directory tree may span multiple metadata zones, so cloning must coordinate subtree operations across nodes while preserving the complete directory hierarchy.

v5.4 introduces cross-zone directory cloning. It recursively processes subtrees while preserving the original cross-zone layout of the directory tree. Cloning only copies metadata. The underlying object data remains shared, so there is no need to copy it. Subsequent writes to the clone do not affect the source files.

In a deployment with 30 metadata zones, cloning 100 million files took 1 minute and 40 seconds, averaging 1 million files per second. More zones can provide greater parallelism, while actual performance also depends on metadata load and how the directory tree is distributed across zones.

Note that cross-zone directory cloning is not atomic. If new data continues to be written to the source directory during cloning, different subtrees in the cloned directory may reflect the source directory at different points in time. If strict consistency is required, pause writes to the source directory before starting the clone.

Performance improvements

More than 90% RDMA bandwidth utilization

Starting with JuiceFS 5.3, clients can use RDMA to communicate with distributed cache nodes, reducing CPU overhead during data transfers and improving transfer efficiency. v5.4 further improves bandwidth utilization and transfer stability.

In a test environment with a 400 Gbps RDMA NIC on the client and an 800 Gbps RDMA NIC on the distributed cache node, a single client achieved 45 GB/s of read throughput. A single distributed cache node delivered 90 GB/s of data throughput, with NIC utilization exceeding 90%.

Higher random-read IOPS under high concurrency

In multi-process training, HPC, and rendering scenarios, high-concurrency random reads can stress not only storage and network resources, but also client lock contention and request-processing overhead.

Over the past six months, the JuiceFS team has continuously optimized high-concurrency random-read performance by simplifying frequently executed code paths and reducing lock contention.

As a result, random-read IOPS has more than doubled overall compared with the previous implementation. With distributed caching, a single client reached up to 189,000 IOPS. With local caching, a single client reached up to 156,000 IOPS.

image: Random-read IOPS comparison
caption: Random-read IOPS comparison

Test environment: A dual-socket AMD server with 128 cores and 256 threads, 1 TB of DDR4 memory, a 100 Gbps NIC, and a 3.84 TB NVMe SSD used as the cache disk. The system ran Linux 5.14.

Optimizing data access across regions

On-demand synchronization to reduce data replication

In multi-region, multi-provider, and multi-cluster compute environments, a JuiceFS mirror file system can automatically synchronize files and directories from the source. Metadata is continuously synchronized to keep the directory structure and object versions up to date. Previously, object data was synchronized in full in the background by default.

However, a workload in the mirror region may only need a subset of the models, datasets, or historical directories stored at the source. For example, if the source contains 10 PB of data but the mirror accesses only 1% of it, full synchronization would transfer and store a large amount of data that the workload never uses.

v5.4 supports on-demand synchronization of object data for mirror file systems. When enabled, metadata continues to synchronize, while object data is no longer synchronized automatically in the background. When a client reads data, JuiceFS first checks the object storage in the mirror region. If the data has not been synchronized yet, JuiceFS fetches it from the source and stores it in the mirror on demand. This reduces unnecessary data transfer and storage consumption.

If an application requires local read performance on the first access, you can warm up the required data in advance. If a complete copy of the data is required, you can continue to use background synchronization.

Accelerating metadata access with read-only nodes

For workloads that only need to accelerate metadata access, v5.4 introduces a new mode that does not require creating a mirror file system.

Users can deploy read-only metadata service nodes in other regions to accelerate metadata reads for local clients. Clients continue to mount the source file system, while the acceleration nodes can scale dynamically to match access requirements.

More flexible cache management

Improving hot-data cache hit rates

Data from one-time scans is often unlikely to be accessed again. Caching this data can evict hot data that is accessed repeatedly. By default, when a request misses the distributed cache, the cache node fetches the data from object storage and stores it in the cache. v5.4 introduces a new cache-aside mode. When enabled, a cache miss causes the application node to read directly from object storage instead of filling the cache with the requested data.

Preserving cache hit rates when adding multiple cache nodes

When multiple cache nodes are added at once, read requests may be routed to new nodes that do not yet have the required data, triggering additional reads from object storage.

v5.4 introduces the --commission parameter. After a new node is enabled, it retains the node mapping from before scaling. If the data is not available in its local cache, the new node fetches it from the original node instead. This reduces reads from the source during cache scaling.

Reducing the number of files on large cache disks

On large-capacity cache disks, storing each cache block as a separate file can cause the number of files on the cache disk to grow continuously.

The new merge-cache mode combines multiple cache blocks into larger files and stores their indexes separately. This significantly reduces the number of files that the cache disk needs to manage.

This feature is currently available as a public beta.

Warming up specific byte ranges

If a workload reads only part of a large file, warming up the entire file consumes additional cache space and transfers unnecessary data.

v5.4 allows you to specify one or more byte ranges to warm up. JuiceFS only warms up cache blocks that intersect those ranges, aligning cache preparation with the data the workload actually needs to read.

This capability enables more precise warm-up for data formats such as Lance and Parquet.

Simplifying operations for massive files

Restoring files by deletion time

When restoring files from the trash, users may know approximately when an accidental deletion occurred but have difficulty identifying the target files by name or keyword alone.

The restore command now supports the start-time and end-time parameters. In addition to the existing keyword filters, these parameters allow users to restrict the restore operation by file deletion time.

Reducing GC and `fsck` memory usage

When checking large file systems, GC and fsck need to process large numbers of records, and sorting can consume significant amounts of memory.

v5.4 adds external sorting modes to both commands, using disk storage during sorting to reduce memory requirements. The external sorting mode for GC is available only with dry-run, allowing users to inspect and evaluate the operation without actually deleting data.

Restricting access with token allowlists

v5.4 supports access control through token allowlists. An allowlist can specify multiple first-level subdirectories under the root directory.

Clients using the token can only access and modify content within those subdirectories, allowing different teams or workloads to be restricted to their designated data.

Summary: scale changes everything

JuiceFS 5.3 brought the platform into a new phase of scale. At the same time, the rapid development of AI continues to push the boundaries of data volume and system load.

As file counts, client counts, and request volumes all reach new levels, performance, stability, consistency, and operational complexity are no longer independent concerns. They interact with one another and become increasingly difficult to manage as the system scales.

v5.4 addresses these system-level engineering challenges across large-scale deployments, covering client connectivity, cross-zone directories, data access, mirror synchronization, and cache management. JuiceFS Cloud Service users can now try JuiceFS Enterprise Edition 5.4 online. Users running on-premises deployments can contact the JuiceFS team for upgrade support.

JuiceFS is used across a range of demanding workloads, including LLMs and multimodal models, autonomous driving, embodied AI, and quantitative investment. We’ll continue working with organizations at the forefront of these fields to improve JuiceFS and build infrastructure that can keep pace with their applications.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Infinite Workspaces for AI Agents: A Practical Architecture with JuiceFS, SQLite, and Litestream

Wed, 23 Sep 2026 05:23:00 +0000

Most conversations about AI sandboxes start and end with boot speed and compute isolation: How fast does the container boot? Can the agent run untrusted code safely? But these discussions miss a fundamental truth: an autonomous agent turns a fresh machine into a rambling digital workspace. It clones code repos, installs heavy node_modules or Python virtual environments, downloads browser binaries, compiles artifacts, generates multi-gigabyte datasets, and writes massive trace logs.

"The files are not incidental. They are the working memory of the task."
– Aniket Maurya, Founder @ Celesto AI

The naive solution is to overprovision the root disk. However, determining the size of a boot disk before a task exposes its scope is an ineffective effort. A "quick bug fix" might require a full Next.js build chain, while a "data benchmark" might produce multi-gigabyte Parquet tables. Coupling system state to workspace state creates a bottleneck.

Instead, an emerging architectural pattern is radical in its simplicity: decouple the system disk from the durable workspace. Provide a POSIX-compliant mount that acts like an infinite, persistent drive, backed entirely by affordable object storage. But how do you build such a system without introducing complex services that defeat the purpose of ephemeral sandboxes? Recently, we co-hosted an Office Hours session with Celesto AI, a platform building secure sandboxes for autonomous AI agents, and the webinar was a delightful exploration of exactly this challenge.

Storage	Purpose
Root disk	Operating system, runtime, package manager internals, and system-level state
Workspace	Repositories, generated files, datasets, build artifacts, logs, and project state

Here is the creative engineering stack making this possible: JuiceFS as the file system layer, SQLite as the embedded metadata engine, and Litestream as the durability glue.

The creative trio: JuiceFS + SQLite + Litestream

Object stores (S3, GCS) are excellent for storing files and performing key-based lookups, but they are terrible at listing directories, handling stat calls, or managing millions of small files in batches (i.e., renaming a directory). You need a metadata engine to act as the "brain" of the file system, mapping paths to block locations.

Usually, JuiceFS deployments use centralized or clustered metadata services, such as Redis, TiKV, or FoundationDB. For isolated AI sandboxes, a creative alternative leverages the three familiar JuiceFS components working in harmony:

The JuiceFS client provides the POSIX bridge. It is responsible for the complex task of communicating with both the metadata engine and the object store. It is the "worker" that reads and writes the actual data, as well as performs administrative tasks such as status checks and data compaction.
SQLite acts as the metadata engine. Instead of connecting to a remote metadata server, JuiceFS also supports using a local SQLite database file as its metadata store. This means the entire file system metadata (every inode, directory entry, and block pointer) lives inside a single, standard .db file on the local disk.
Litestream provides the durability backing for that SQLite file. Litestream continuously takes SQLite snapshots and streams the SQLite write-ahead log (WAL) frames to the same object store.

Here is the magic: When the sandbox is running, JuiceFS uses the local SQLite file to resolve ls, cd, and read operations. The actual file blocks are already safe in the object store. (Although, the sync-interval configuration needs to be taken into consideration.) When the user stops the sandbox, the local SQLite file might be discarded, but Litestream has already replicated the metadata engine to the object store.

image: JuiceFS + SQLite + Litestream
caption: JuiceFS + SQLite + Litestream for AI agent workspaces

When the sandbox resumes on a fresh VM (or a different host entirely), the system simply downloads the latest SQLite snapshot from the object store and replays the WAL. Within seconds, the local SQLite can be fully reconstructed. JuiceFS mounts it, points back to the same object store blocks, and the agent is looking at the exact same workspace directory it left behind.

The JuiceFS file system doesn't care which computer it runs on. But it does remember its folder structure and file pointers with the portable SQLite snapshots and WAL frames. With this setup, we turned the working memory of the AI agent into files and blocks in the object store, making it durable and accessible instead of locking it to a specific boot disk or block device.

An unconventional but effective combo

JuiceFS supports interchangeable metadata engines. Choosing SQLite + Litestream subverts the typical playbook by prioritizing simplicity and per-agent isolation over global scale. Here is why this particular stack fits the AI agent use case so well:

Easy to set up: While it is not completely hands-off (you still manage the JuiceFS client, the local SQLite instance, and Litestream's replication configuration like Celesto AI does), this stack is drastically simpler to bootstrap than a distributed database cluster.
Per-agent isolation: Each sandbox gets its own dedicated SQLite metadata file. This eliminates noisy-neighbor problems. Metadata contention simply does not happen across agents, because each one has its own independent "brain" living right next to its mount.
Performance that fits the scope: A single agent workload typically produces tens of thousands to a few million files. SQLite handles this range comfortably with low latency and minimal resource footprint. While this approach obviously does not scale to the hundreds of billions of files that JuiceFS Enterprise Edition can manage globally, it falls squarely into the comfortable zone for individual agent workspaces.

Of course, this approach comes with important trade-offs and considerations:

No cross-agent sharing: The file system is inherently tied to a single SQLite instance. Agents cannot mount and share the same workspace concurrently. If your use case requires collaborative access or shared datasets across sandboxes, you would need a different strategy.
Resume latency depends on object store speed: When a sandbox resumes, the system must download the latest SQLite snapshot from the object store and replay the WAL frames. This adds noticeable startup time compared to a hot-mounted local disk. Tuning Litestream's sync-interval involves a direct trade-off between durability guarantees and resume performance.
Single-writer limitation: SQLite is fundamentally a single-writer database. Heavy parallel writes from multiple processes within the same sandbox can become a bottleneck. For autonomous agents following a largely sequential workflow, this limitation is rarely a practical issue. But it is worth keeping in mind for highly concurrent, multi-threaded agent designs.

The stack is not a universal solution. It is a deliberate, opinionated choice that trades global scalability and shared access for extreme simplicity and tight per-agent isolation. And for the AI sandbox use case, that trade-off often makes perfect sense.

The performance reality

This architecture is beautiful, but it isn't magic. You are trading local SSD latency for object storage durability and infinite scale. The benchmarks reveal a clear pattern:

Small files need attention: Writing 5,000 tiny 4 KiB files to an object-backed file system involves heavy metadata churn. Even with SQLite's fast local access, every close() and fsync() triggers WAL writes and Litestream replication. In these scenarios, the local root disk (ext4/XFS) will outperform the durable mount by a factor of 2x to 5x. See Celesto's blog post for more details.
Large sequential workloads shine: For datasets, model checkpoints, build artifacts, and logs (files > 10 MB), the object-backed mount achieves upwards of 200 MiB/s, which is nearly 90% of the performance of the local disk. The background block uploads are streamed efficiently. See Celesto's blog post for more details.

The golden rule for agents is to use the object-backed mount drive for durable, long-term state reports and large artifacts. Use the local ephemeral disk for high-churn, temporary scratch work that doesn't need to survive a restart. This hybrid approach gives you infinite capacity without sacrificing performance where it truly hurts.

Final thoughts

Cloudflare CFO Thomas Seifert recently predicted that within five years, non-human traffic could be as much as 1,000 times human traffic, making humans "a rounding error on the internet." He caveated that he's "called it wrong at every point along the way," but the direction is hard to dispute: AI and machine traffic already overtook human traffic in May 2026.

image: Cloudflare exec
caption: “Humans will be a rounding error on the internet” says Cloudflare exec

If that is true for network packets, it is likely true for storage as well.

Every autonomous agent session produces a workspace. Multiply that by millions of agents running continuously, resuming, retrying, and branching. Human developers generate storage in bursts tied to working hours. Agents generate it constantly, at machine speed.

This raises an uncomfortable question: if non-human consumers are about to dominate storage demand, are we designing for the right consumer? The combo of JuiceFS + SQLite + Litestream is one answer, but not the only one. Other approaches worth watching include object storage as the primary interface, shared metadata engine with concurrent access at agent scale (i.e., the JuiceFS Enterprise Edition), copy-on-write workspace snapshots, and many others.

Portable, shareable, scalable file storage for AI agents is still in its opening moves.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Keeping GPUs Fed: How Meta's AI Storage Architecture Aligns with JuiceFS

Thu, 17 Sep 2026 03:32:00 +0000

Introduction

Recently, I came across an eye-opening blog post and presentation from Meta's engineering team detailing their AI storage blueprint at scale. Meta operates hundreds of exabyte-scale storage clusters serving everything from Facebook and Instagram to Meta AI.

What caught my attention wasn't just the scale—it was the architecture. After years of wrestling with AI's brutal latency and throughput demands, Meta has converged on a clear pattern: data-metadata separation, rich client design, and comprehensive caching.

Here's the fascinating part: this blueprint isn't unique to Meta. JuiceFS shares these same first principles of design. The hyperscaler's new construction closely resembles the cloud-native architecture that JuiceFS has advocated for years.

In this post, I'll analyze both architectures, compare their core components, dive into their cross-region capabilities, and explain why this convergence in architecture makes the most sense for a distributed file system.

Why storage matters for AI workloads

Before diving into the architectures, it's worth understanding why storage has become such a critical component for AI infrastructure.

Meta puts it succinctly: "If AI is the brain, storage is the memory: capability and speed are highly dependent on the size of memory and speed of retrieval." Yet while AI compute performance has roughly tripled every two years, storage and interconnect performance growth have been far more modest. This disparity has made storage bottlenecks one of the primary contributors to GPU stalls for AI workloads, which is directly impacting both expenditures and time to market.

image: Dataloading across two GPUs
caption: Dataloading across two GPUs (Reference: Meta)

The math is brutal. During model training, hundreds of thousands of GPUs iterate over vast datasets across multiple epochs. GPUs synchronize their state periodically. If a single GPU stalls due to slow storage, it slows down the entire training job. In Meta's presentation, if GPUs were idle for about 20% of the time, "that 20% stall means about tens of millions of dollars loss every hour" for a data center the size of Manhattan.

Beyond the direct dollar cost, there is an even more consequential dimension: opportunity cost. The AI industry is moving at an unprecedented pace. Major model releases, which took roughly 104 weeks between 2020 and 2022, have now been compressed to approximately 4-week cycles by 2026. Every week of delay in shipping a frontier model translates to lost market share, diminished competitive advantage, and missed revenue opportunities.

In short, storage is what makes the difference between a model shipping in weeks versus months and between a billion-dollar data center running at 80% versus 100% efficiency.

Meta's AI storage blueprint: the evolved architecture

After iterating on their storage infrastructure, Meta has arrived at a definitive architectural blueprint to solve the GPU stall problem at its root while continuing to support existing external and internal products. The training stack has been gradually migrating to the BLOB-storage interface. The motivation is more about performance: AI workloads demand predictable, bounded pMax latency. Traditional layered metadata lookups could add hundreds of milliseconds per request, stalling GPUs during training. By moving to a BLOB-centric interface, Meta leverages flash drives to deliver the low, predictable latencies required to keep thousands of GPUs productive.

The three-component architecture

The diagram below illustrates the request flow for Meta's BLOB storage getObject API and shows the system's three major components.

image: BLOB storage getObject request flow
caption: BLOB storage getObject request flow (Reference: Meta)

The Tectonic data storage: At the foundation sits Tectonic, a regional, multi-tenant storage fabric providing high durability and availability through erasure coding. It supports tiering across HDDs and flash drives, intelligently managing hot, cold, and warm data placement.
The BLOB server and metadata layer: Operating on top of Tectonic, the BLOB layer handles API requests, stores metadata, and exposes a global, infinitely scalable storage fabric with policies allowing users to trade off between durability and availability, abstracting away the complexity of the underlying block storage.
The rich client SDK: Meta eliminated the data-plane proxy and built a rich client SDK with a Tectonic BlockClient embedded within it, enabling direct data reading from Tectonic storage servers to clients. The client also implements hedged reads and dynamic concurrency control to reduce tail latencies.

To maximize cache efficiency, Meta also leverages spare memory on GPU hosts as a distributed data cache by integrating peers from the Owl subsystem directly into the BLOB-storage client SDK, which yields an average cache hit rate of 80%.

Cross-region data access with tiered caching

Meta's cross-region approach treats storage as a global data lake, but with tiered caches accelerating access across geographical boundaries.

image: Cross-region data access with multi-tier caching
caption: Cross-region data access with multi-tier caching (Reference: Meta)

Cache Tier	Location	Role
L1 Cache	GPU host memory (RAM)	Hottest, most frequently accessed data
L2 Cache	GPU host flash (SSD)	Local flash tier for warm data
L3 Cache	Regional flash-based BLOB storage	Regional cache before falling back to global BLOB storage

The results speak for themselves: average ingestion time dropped by 93%, and worst-case ingestion time dropped by 97%. This is what it looks like when storage stops being the bottleneck and starts enabling AI research velocity.

JuiceFS: cloud-native, AI-native from day one

For readers already familiar with JuiceFS, the parallels to Meta's design are immediately apparent. The architecture Meta converged upon after years of iteration shares the same core principles that have guided JuiceFS from day one.

JuiceFS was architected from the ground up on the data-metadata separation principle, which also consists of three core components:

Object storage: JuiceFS uses object storage as the reliable, scalable backend for actual data storage. It supports all major public cloud offerings (AWS S3, Alibaba OSS, Google Cloud Storage, etc.) as well as self-hosted solutions like Ceph and MinIO. This layer ensures that AI datasets, which can be as large as petabytes or exabytes, are safe, accessible, and can grow almost without limit.
Metadata engine: The metadata engine stores all file system metadata, including file names, sizes, permissions, directory structures, and the mapping between files and their underlying data blocks. JuiceFS supports multiple engine options, including Redis, MySQL, PostgreSQL, TiKV, and many others. For extremely large-scale deployments, JuiceFS Enterprise Edition offers a proprietary Raft-based distributed metadata engine that provides high availability and strong consistency across multiple nodes.
JuiceFS client: The client handles all file I/O logic, including data slicing, merging, caching, and POSIX semantics enforcement. It communicates with both the metadata engine and object storage directly, coordinating reads and writes intelligently.

Cross-region strategy: mirrors and replication

The JuiceFS Enterprise Edition also provides a native approach to support cross-region and cross-cloud data access:

Mirror file systems: Deploy a full metadata cluster and optionally an object storage replica in a mirror region. Metadata syncs automatically from the source region with approximately 1-second latency under normal network conditions.
Async data replication: JuiceFS supports cross-region and cross-cloud asynchronous replication with a one-to-many topology. Data written in the primary region is replicated to multiple mirror regions in the background, enabling global datasets without manual copying.
Distributed cache: JuiceFS clients on mounting hosts can form a distributed cache group using a consistent hashing ring, sharing cached data blocks with one another. This is particularly effective for model training, where datasets are repeatedly accessed across GPU nodes.
Smart read strategy: Clients preferentially read from the local region. If the data isn't fully synced yet, they seamlessly fall back to the source region.

image: JuiceFS mirror file system architecture
caption: JuiceFS Enterprise Edition mirror file system

This design resonates with Meta's global data lake + tiered cache approach but offers a more flexible, cloud-native model that works across public clouds, private data centers, and hybrid environments.

Architectural similarities

On the surface, Meta mainly exposes a BLOB-style interface, while JuiceFS primarily provides a POSIX file system. Yet beneath these protocol differences, the architectural parallels are noteworthy. Both systems converged on the same three-component design with comprehensive caching and cross-region capabilities:

Dimension	Meta's BLOB-Storage	JuiceFS
Data storage layer	Tectonic: regional block storage fabric with erasure coding, tiering across HDDs and flash	Object storage: S3, OSS, GCS, Ceph, MinIO, and others
Metadata layer	Unified metadata scheme backed by ZippyDB with O(1) chunk lookups	Metadata engine: Redis, MySQL, PostgreSQL, TiKV, JuiceFS Enterprise Edition
Rich client	"Fat client SDK" with embedded Tectonic BlockClient, streaming directly from storage servers	JuiceFS client: handles all I/O logic, slicing, merging, caching; supports FUSE, POSIX, Hadoop SDK, CSI, S3 Gateway
Distributed cache	Tiered caching layers	Memory + local cache + distributed cache group
Cross-region	Tiered caching with global BLOB storage data lakes	Primary region + mirror file systems

A concrete example: explicit cache hydration

One interesting example is how both systems handle proactive cache hydration, which is the practice of loading data into the cache before it's actually needed.

Meta exposes a prefetch() API through the BLOB-storage SDK, while JuiceFS provides a warmup subcommand. Both serve the same purpose: allowing the client side to signal to the storage system exactly which data will be needed in the near future so it can be preloaded into cache before the GPU finishes its current work.

This is especially critical in distributed training, where thousands of GPUs may simultaneously read the same dataset. Without cache hydration, the initial requests would have to read from the underlying storage (Tectonic for BLOB and object store for JuiceFS) with higher latency. With prefetch() or juicefs warmup, data is staged in the distributed cache before the GPUs need it. In essence, both systems recognize the limitations of network-based storage and passive caching, and they both provide explicit prefetching as an important tool to compensate for what the storage layer alone cannot guarantee.

Conclusion

Meta's AI storage blueprint is a testament to the architectural patterns that emerge when a hyperscaler evolves its data infrastructure to hundreds of exabytes. After years of iteration, they arrived at a design that's similar to what JuiceFS has championed from day one: data-metadata separation, a rich client that handles I/O logic directly, tiered caching, and cross-region capabilities.

The same architectural pattern is also available to teams of any size. JuiceFS brought it into the big data era and now brings it into the AI age, with the flexibility of multi-cloud cross-region strategies, distributed caching, and a rich ecosystem of access protocols.

Hopefully, storage is no longer the bottleneck in your AI training stack. At least, it doesn't have to be.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Everything Is a File for Agents: Hello Uses JuiceFS to Build Stateful AI Workspaces

Thu, 10 Sep 2026 06:29:00 +0000

Hello is a Chinese mobility and technology company, offering car-pooling, car-hailing, battery swapping, shared bike, e-bike, and e-scooter rental services. We have 800 million+ registered users.

As our AI business continues to grow, we need to support multiple workloads simultaneously, including model training, data processing, and AI agents. To address issues such as data fragmentation, inconsistent access methods, and complex cross-cloud data flows caused by multiple storage systems, we introduced JuiceFS (an open-source, high-performance distributed file system) in production to build a unified file data foundation that connects Notebook, training, evaluation, and agent environments.

Among these workloads, agents have become an important application type and are already used in several real-world application scenarios. We’ve also deployed an internal personal agent for employees. Unlike training workloads, which primarily focus on high-performance I/O and access to massive numbers of files, agents place greater emphasis on data movement during task execution, state persistence, and the accumulation of results.

This article focuses on how we use JuiceFS to persist user profiles, conversations, memories, skills, execution traces, and application artifacts, and how a shared workspace decouples agent state from compute Pods.

Hello's AI infrastructure and storage landscape

Our business scenarios are diverse, including customer service, review, scheduling, analytics, and broadcasting. To support AI applications across these businesses, we’ve progressively built an AI platform and a large language model (LLM) platform covering model development, training, inference, and serving.
The AI platform supports scenarios including Notebook, training, offline inference, online inference, and image building. Notebook environments need to mount user directories and datasets. Training and inference jobs need to access input data, models, and intermediate results. The image-building process also generates reusable caches.

The LLM platform aggregates different model services and provides model invocation and benchmarking capabilities. The data and reports generated during model benchmarking also need to be persistently stored.

For different types of data and workloads, we primarily use three types of storage:

CPFS provides high-performance parallel storage.
JuiceFS serves as the shared file storage foundation for the platform. It supports data requiring POSIX semantics, multi-node sharing, and a unified namespace, including Notebook environments, task datasets, model benchmarking reports, and image build caches.
Object storage is mainly used for object data such as multimedia attachments.

A small number of directories on development machines are still used as a transitional solution and have not yet been migrated to the unified storage system.

Overall, this storage architecture already covers the data persistence and cross-environment sharing requirements of our AI platform.

image: Hello's AI infrastructure
caption: Storage architecture in Hello's AI infrastructure

New storage requirements introduced by agent workloads

As agents have gradually moved into production, our AI capabilities have expanded beyond model development, training, and inference to agents capable of invoking tools and completing application workflows.
Related applications include image review for two-wheeled vehicle operations and trip recording review for ride-sharing services. These applications have delivered practical results by expanding review coverage, reducing manual effort, and lowering service costs.
Building on the deployment of vertical agents, we further introduced Jarvis Claw, a personal agent for employees. Employees can interact with Jarvis Claw through DingTalk or the web. By combining capabilities such as long-term memory, skills, Model Context Protocol (MCP), and Cron, it can assist with tasks such as answering questions, generating reports, reviewing content, and performing analysis.

Unlike vertical agents designed for a single application workflow, a personal agent needs to preserve state across multiple rounds of interaction and continuous tasks.

For example:

In an IT support scenario, Jarvis Claw needs to preserve user requests, attachments, conversations, knowledge retrieval results, and tool invocation results.
In an operations reporting scenario, it needs to continuously access historical source tables and scripts while generating new data tables, charts, and reports.
In an operations attribution scenario, it also needs to preserve data extraction scripts, intermediate data, and final analysis results for later inspection and re-execution.

A long-running agent needs to preserve five types of data:

User profiles and behavioral configurations
Conversation history and runtime state
Long-term memory that can be reused across conversations
Skills, MCP servers, channel configurations, and scheduled task configurations
Application artifacts such as scripts, CSV files, Excel spreadsheets, charts, and reports

These states naturally take the form of files:

Profiles are stored as Markdown files.
Conversation history is stored in SQLite.
Skills consist of directories and scripts.
Task configurations use JSON.
Application results exist as ordinary files such as data tables, charts, and code.

The complete state of an agent ultimately consists of a continuously evolving collection of files and directories that are created, modified, and related to one another.

A long-running agent therefore requires a unified shared storage layer that supports POSIX semantics, SQLite write-ahead logging (WAL), atomic rename operations, and access to large numbers of small files. It must also allow multiple Pods to share the same user state. Even if a compute Pod is destroyed, recreated, or rescheduled, the user's profile, conversations, memories, and task artifacts must remain available.

How JuiceFS stores the complete state of an agent

To meet these requirements, we extended our existing JuiceFS-based shared file storage foundation to agent workloads and use it to store a complete workspace for each user.

The platform assigns each employee an independent Jarvis Claw directory based on the employee account. The directory structure is similar to:

/userdata/aiplatform-jarvis-claw/user/<staff_id>/data/workspace

The user's profile, conversations, memories, skills, task configurations, and application artifacts are all stored in this directory. The Jarvis Claw Pod is responsible only for computation.

When a Pod is destroyed, recreated, or rescheduled, it only needs to remount the existing workspace to continue using the previous state. This makes the agent stateful while keeping the compute Pods stateless.

What agent state is stored in the workspace?

The workspace contains profile files such as:

AGENTS.md
SOUL.md
PROFILE.md
HEARTBEAT.md
MEMORY.md

These files describe the agent's identity, behavior, and user preferences and are combined into the system prompt.

Conversation and runtime state are primarily stored in files and directories such as:

sessions/
transcripts/
dialog/
tool_results/
history.db
chats.json
inbox_events.json

history.db uses SQLite and also generates WAL and SHM files.
User messages are written to transcripts/ at the beginning of a task. Even if the task is interrupted later, the conversation content generated before the interruption is preserved.
Long-term memory is stored in directories such as:

memory/
digest/
mem_agent/
mem_session/
mem_metadata/
resource/

Jarvis Claw manages memory using ReMe and a Memory-as-File approach.
Some memory is written directly by the agent during execution. Another part is generated by periodically reviewing historical conversations, extracting user preferences and information that remains useful over time, and writing the results to directories such as digest/. Internally, this process is referred to as Dream.
The workspace also stores task and capability configurations, including:

cron_jobs.json
cron_runs/
mcp.json
channels.json
dingtalk_webhooks.json
skills/
skill.json

Skill packages are stored as ZIP files in object storage. When a Pod starts, they are downloaded and extracted into the user's workspace.
Scripts, data tables, charts, and reports generated by the agent are stored in directories such as output/ and scripts/, as well as in files including Excel spreadsheets, CSV files, and SVG images.

The actual workspace also contains auxiliary content such as browser data, backups, and migration files.

Together, these files and directories form the complete state of an agent and are continuously created, modified, and accumulated as tasks are executed.

How a unified file system supports data movement and operations

Storing agent workspaces in JuiceFS not only provides state persistence and storage-compute separation, but also allows agents to share the same file system with environments such as Notebook, training, and evaluation.

When permissions allow it, Notebook can directly access an agent's workspace. Data prepared in Notebook can be directly provided to an agent. Likewise, data tables and scripts generated by an agent can be reused for analysis or evaluation without repeatedly uploading or copying data between different systems.

Operational tasks such as inspection, migration, backup, and per-user storage usage analysis can also be performed using standard tools such as ls, du, and rsync. Engineers can directly inspect conversations, memories, configurations, and task artifacts to troubleshoot issues during execution.

However, as the workspace evolves from an ordinary data directory into a working environment that an agent can actively read from, write to, and execute scripts within, the platform must provide not only file persistence and sharing, but also appropriate access controls and execution boundaries.

Security boundaries for shared workspaces

An agent may not only read and modify files in its workspace, but also execute commands or code through Shell, MCP, and REPL. Without effective constraints, its access could extend beyond the user's workspace to other users' data, service source code, and sensitive files inside the container.

Therefore, we established security boundaries across four layers:

User data isolation
Tool invocation governance
File access control
Subprocess management

How different users' data is isolated

We manage file system permissions through JuiceFS and OpenACL, assigning each user an independent workspace and controlling the directories that different users are allowed to mount and access.

With this mechanism, each agent can access only the workspace for which it has been authorized and cannot read other users' data.

File system permissions provide the basic data boundary between users, but they cannot cover all access behavior generated when an agent executes tools and code inside a container. Therefore, in addition to user-level permissions, the platform also needs to further restrict tool invocations and their execution environments.

How the system determines whether a tool invocation is allowed

When the model decides to invoke a tool, the operation is not executed immediately. Instead, the governance layer first evaluates the operation through assert_policy().
The result falls into one of four categories:

ALLOW: Allow the operation.
DENY: Reject unknown tools and dangerous operations such as sudo.
ASK: Request manual approval.
SANDBOX_FALLBACK: Execute the operation in a sandbox environment.

When manual approval is required, the system sends an approval request to the frontend through a side channel. After the user approves the request, a generalized version of the rule can be written to policy.yaml so that similar future invocations can reuse it.
For Shell operations, execute_shell_command() fixes the current working directory to the user's workspace rather than the service source directory. Then, it selects the appropriate sandbox configuration based on the governance result. The configuration prioritizes dynamically generated fine-grained rules from the governance layer, followed by globally configured default rules registered when the service starts. Bare-process execution is allowed only in local development environments. This layer determines whether a tool invocation is allowed to execute. For approved operations, the platform must further restrict the files that can actually be accessed based on the execution method.

How file access permissions are restricted for the main process and subprocesses

Agents access files mainly in two ways, which require different control mechanisms.

File tools such as read_file, write_file, edit, grep, glob, and view_image run directly inside the main service process and do not create subprocesses.

As a result, they cannot be intercepted by bubblewrap. The system uses application-level path checks, such as is_protected_path(), to constrain relative paths to the user's workspace while blocking access to protected locations such as /app, which contains service source code and installation files. Protected entries are also filtered during directory listing.

Shell commands, stdio MCP servers, and REPL environments used to execute model-generated code, on the other hand, create subprocesses.

For these operations, application-level path checks alone cannot establish a complete security boundary. Therefore, the system uses bubblewrap and Linux mount namespaces to construct an isolated file system view for subprocesses.

Within this restricted view:

The user's workspace is mounted with read and write access.
Other approved paths are exposed according to policy rules.
Service source directories such as /app are hidden or mounted as read-only.
Sensitive paths such as ~/.ssh and ~/.aws are blocked.

As a result, the agent can operate normally within the user's workspace without gaining access to all files in the container.

If the production environment is configured to use bubblewrap but the sandbox cannot actually be created, Shell and MCP use a fail-closed policy: execution is rejected rather than falling back to an unisolated bare process. The REPL follows the same rule when executing model-generated Python code. If the sandbox is unavailable, execution is denied.

How subprocess lifecycles and sandbox boundaries are managed

Each tool invocation that creates a subprocess uses an independent sandbox and process group with --die-with-parent enabled. When a user executes /stop, a task times out, or a new task within the same Scope replaces the current one, the system progressively cancels the current task and uses killpg to clean up the entire process group. This prevents Shell processes, MCP servers, or their child processes from continuing to run in the background.
Although stdio MCP servers are launched by the SDK itself, their execution commands are also prefixed with bubblewrap so that they use the same file system view as Shell commands. The REPL must likewise enter the sandbox when executing model-generated Python code, preventing execution methods other than Shell from becoming a path around the security boundary.
There are also several implementation details to consider:

After /app is overlaid with tmpfs, it must be remounted as read-only to prevent writes to the temporary /app from silently succeeding.
Before entering the sandbox, VIRTUAL_ENV and PATH entries pointing to protected locations must also be cleaned up to avoid problems with the Python runtime environment.
When the service starts, it performs an actual /bin/true execution inside the sandbox as a self-check rather than merely verifying that the bubblewrap binary exists.

The current solution primarily addresses file system and service source code isolation. Network isolation through --unshare-net has not yet been implemented. In addition, CPU and memory quotas have not yet been configured beyond execution timeouts.

Summary

Our production practice can be summarized in four points:

Everything about an agent's state can be represented as files. Profiles are Markdown files. History is stored in SQLite. Skills are directories. Memory and configuration are stored as files. Application artifacts include Excel spreadsheets, CSV files, charts, and scripts.
A shared file system provides state persistence and storage-compute separation. Agent Pods are responsible only for computation, while agent state and workspaces are stored in JuiceFS. Pods can be rebuilt at any time without causing user data to disappear with the compute instance.
The same file system connects multiple AI environments. Notebook, training, evaluation, and Jarvis Claw can access the same set of files. Data no longer needs to be repeatedly copied between different systems, and operations can rely on standard file-based tools and workflows.
Opening the workspace requires layered security boundaries. JuiceFS and OpenACL manage user-level file system permissions. Application-level path checks restrict the file access scope of tools running in the main process. Bubblewrap controls subprocesses created by Shell commands, MCP servers, and code execution.

Next, we plan to further explore agent self-evolution. Agent telemetry, memories, and evaluation data will continue to accumulate in the file system. Models will continue to improve, and this accumulated data will become part of the foundation for the agent's future evolution.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.

Scaling PB-Scale Scientific Data Workloads: How the American Museum of Natural History Powers Research Computing with JuiceFS

Thu, 03 Sep 2026 05:29:00 +0000

In cutting-edge scientific fields like astrophysics and genomics, high-throughput reading of massive datasets and zero-tolerance data integrity are the absolute lifebloods of computational analysis.

Recently, the JuiceFS team sat down for an in-depth conversation with the Scientific Computing team at the American Museum of Natural History (AMNH). They explained how they use JuiceFS Community Edition to handle large amounts of research data within an Open Storage Network (OSN) S3-compatible storage system which was funded by a grant from the National Science Foundation (NSF), creating a highly reliable and consistent storage setup for academic use.

The workload: 95% read-heavy traffic

The scientific computing environment at AMNH supports extraordinarily intense interdisciplinary research, operating at a massive scale. Currently, AMNH's OSN nodes have the capacity to store around 3 PB of research data, with approximately 30% of that total volume actively accessed via the NFS + JuiceFS layer. The overall storage architecture is precisely tailored to handle these specific academic patterns:

A 95% Read-Heavy Profile: Workloads include the system's massive reference datasets such as genomic sequences and astronomical survey data in read-only format. The calculations are very throughput- and latency-sensitive. Once a dataset is written, it will be read many times and rarely changed.
The OSN (Open Storage Network) Model: AMNH partners with the Open Storage Network. Standard rack servers and an external JBOD are hosted on-site at the museum but managed remotely by OSN administrators. This setup provides AMNH with cost-effective S3-compatible endpoints while allowing them to maintain physical ownership of the hardware and data.
Seamless Multi-Protocol Access: AMNH adopted JuiceFS Community Edition as the distributed file system layer, allowing mounting S3 buckets seamlessly and enabling traditional NFS access to this underlying S3-compatible storage for high-performance computing (HPC) clusters.

image: NFS + JuiceFS + OSN
caption: AMNH's NFS + JuiceFS + OSN storage architecture

Data integrity first, without silent corruption

For scientific research, data consistency is crucial. To prove and continuously observe the system's resilience, the AMNH team conducted rounds of tests during the PoC phase and watched JuiceFS logs and metrics closely in production.

"We have recently observed timeout warnings when JuiceFS writes to the OSN object store, but data integrity issues were never reported. We do want to understand why, though."

– Scientific Computing Team, AMNH

That was one of the discussions we had during the meeting. It turns out that JuiceFS configures all object store SDKs with RetryMaxAttempts=1 and implements custom logic for retries. This way, JuiceFS is able to unify the behavior across all object stores it supports. Thus, in this case, the warning is a precaution, and the data integrity is ensured.

When comparing JuiceFS to other file systems that claim higher raw performance metrics (often achieved by using "unsafe" write-handling or skipping strict metadata commits), AMNH opted for data consistency.

"We use almost all the default configurations for the JuiceFS client with Redis as the metadata engine, and it's been running in production steadily for 4 years now."

– Scientific Computing Team, AMNH

Current & future tuning options

The conversation also continued into how AMNH tunes and considers the JuiceFS client configurations to match their scientific computing patterns needed for the future.

The write-back configuration

Currently, the only non-default configuration AMNH utilizes is --writeback , which smooths out the client-side writing throughput. There are always tradeoffs. Write-back sacrifices a bit of data consistency and requires high disk reliability on the mounting machine. However, given AMNH's current workload pattern (mostly reads), both teams agreed that it is acceptable either way, write-back enabled or not.

High-performance caching

The teams also evaluated the differences between the Linux kernel's native page cache and JuiceFS' cache for client-side metadata. While the kernel page cache is the fastest, the JuiceFS client-side metadata cache offers more flexibility. In a read-heavy setup, this configuration can save many roundtrips to the metadata engine. Currently, the metadata engine is co-located with the JuiceFS client-mounting host, but in the future, this configuration can be a nice add-on for performance.

On the file side, since the hardware uses HDDs managed by the OSN object store, the JuiceFS client's local data read cache significantly lowers latency after the first read. Increasing the --cachesize can further improve performance when needed.

The potential of using SQLite as the metadata engine

In a highly creative approach, the teams discussed utilizing SQLite on local NVMe storage as their JuiceFS metadata engine. The JuiceFS team highlighted that for standalone sandboxes or read-heavy workloads, SQLite is an exceptional, low-maintenance choice, provided the metadata file is backed up for resilience (which is true for Redis as well). Since for AMNH's current scenario, the primary I/O bottleneck usually lies in the backend object storage rather than the database, SQLite's simplicity makes it incredibly attractive for specific workloads.

Final thoughts

Managing multi-petabyte scale, strong data consistency, and high-throughput research workloads is a difficult balancing act for any infrastructure team. AMNH has built a resilient, low-maintenance storage environment that lets scientists work without creating massive engineering overhead, using an open architecture and JuiceFS Community Edition on top of an OSN backend.

Reflecting on their four-year journey of running this system stably in production, the AMNH team summarized their experience seamlessly:

"The implementation on our side is straightforward, and we are happy about it overall."

– Scientific Computing Team, AMNH

As data demands in scientific computing continue to grow, this blueprint of combining cost-effective object storage with a robust, reliable file system layer provides a highly reproducible success story for academic and research institutions worldwide.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and our Discord server.

JuiceFS Blog

How Horizon Robotics Uses JuiceFS for EB-Scale Autonomous Driving Data

New storage challenges at EB scale

Challenge 1: Optimizing metadata for hundreds of billions of files

Reducing metadata hotspots and large-directory pressure

Rate limiting to keep the cluster stable

Metadata disaster recovery: cold backups as a last line of defense

Challenge 2: Data visibility and governance, reducing costs by deleting 200 PB every week

Deletion decisions: dashboards and application-owner review

Recovering deleted data: a seven-day trash

Reclaiming space: distributed garbage collection

Auditing orphaned objects with leak-check

Challenge 3: Balancing training performance and EB-scale storage costs

Three-tier caching: from clients to a centralized cluster

Metadata caching: avoiding repeated lookups

Controlling read amplification: coordinating prefetching and caching

Keeping data close to compute and tiering by access frequency

Challenge 4: Connecting and managing hundreds of thousands of clients

Multi-tenancy: access control

Access methods: S3 Gateway and a mountless SDK

Metric observability and client upgrades

Conclusion

How Zhuoyu Built a Petabyte-Scale Data Platform for Autonomous Driving with JuiceFS

The autonomous driving data loop: growing volumes and repeated data consumption

Coupling between access, operations, and scaling in the original architecture

Architecture upgrade: building a unified data platform on JuiceFS

Technology selection: S3 as the primary interface, JuiceFS as the storage foundation

Unified access layer: decoupling applications from storage

Building the core capabilities

Multiple storage pools: scaling capacity and isolating failures

Distributed caching: reusing hot data across nodes

UFS: integrating external data and accelerating access

Production at hundreds-of-petabytes scale: architecture and optimizations

Cache orchestration: building caches as data arrives and proactively warming hot data

Inference workloads: large-scale model delivery

Training workloads: TCP zero-copy optimization

Lance columnar storage: reducing I/O at the data format layer

Combining Lance with JuiceFS for targeted data warm-up

Conclusion

Scaling Data Management for Agent Sandboxes with JuiceFS

How to manage long-lived data in short-lived sandboxes

How JuiceFS manages and accesses sandbox data

Organizing shared data and task data with directories

How cross-cloud sandboxes access the same data

JuiceFS data mounting mode for strongly isolated sandboxes

JuiceFS in cloud provider sandboxes

Tencent Cloud: data mounting methods in agent runtime

Alibaba Cloud: FC and ACS

Sandbox lifecycle management and mount control

Challenges and optimization in large-scale agent sandbox scenarios

Client scaling bottleneck

Cache reuse and data loading efficiency

Small files and metadata performance

Fault isolation and platform governance

How to Access JuiceFS via S3 and WebDAV Protocols

A shared initialization path

Applicable scenarios and common mappings

S3 Gateway principles: from object semantics to file system operations

Request path: from the MinIO S3 server to jfsObjects

State storage: how the internal namespace carries the Gateway state

Semantic conversion: PutObject, Multipart, Bucket, and Metadata

Permission coordination: IAM and multiple instances

WebDAV principles: from file protocol to JuiceFS operations

Request path: from HTTP entry to webdavFS

State boundaries: dead properties and LOCK

Nydus + JuiceFS: Reducing Container Startup Time for AI Inference from 116s to 1.4s

Why OCI image distribution is slow

Dragonfly+Nydus: what it solves and what it leaves behind

Architecture redesign: Nydus blobs hosted by JuiceFS

Runtime mechanism: how Nydus and JuiceFS coordinate

Startup phase: fetching bootstrap (metadata)

Read phase: on‑demand blob access

Cache phase: data reuse via JuiceFS cache groups

Production test data

Container startup time comparison

On‑demand file read performance

Summary: combining on‑demand loading with shared cache

JuiceFS Enterprise 5.4: From 100 Billion Files to 1 Million Clients

Supporting 1 million clients

Fast cloning for large directories

Auditing orphaned objects with `leak-check`

Request path: from the MinIO S3 server to `jfsObjects`

Semantic conversion: `PutObject`, `Multipart`, `Bucket`, and `Metadata`

Request path: from HTTP entry to `webdavFS`

Reducing GC and `fsck` memory usage