Evolving JuiceFS Enterprise Edition for Exabyte-Scale AI Storage

As AI workloads scale from petabytes to exabytes, storage systems face challenges that go far beyond capacity. File counts can reach hundreds of billions, client connections can grow from thousands to hundreds of thousands or even millions, and training throughput continues to increase. At the same time, data reclamation, cross-cloud access, multi-tenant isolation, and daily operations all become more demanding.

These challenges put pressure on four key areas of the storage architecture: the metadata engine, data lifecycle management, the data path, and the client and control planes.

The previous article explored how Horizon Robotics, an autonomous driving technology company, uses JuiceFS Community Edition at exabyte scale. This article focuses on JuiceFS Enterprise Edition and how these four areas have evolved to meet the demands of exabyte-scale data, hundreds of billions of files, and millions of clients.

Metadata engine: from single-node performance to horizontal scaling across hundreds of billions of files

Metadata challenges at the scale of hundreds of billions of files

When a file system manages hundreds of billions of files, metadata access faces three major challenges:

• Hotspots: During peak training periods, a small number of heavily accessed directories and frequent metadata operations can turn a single metadata node into a bottleneck.
• Increased latency: Path resolution, attribute lookups, and file opens all require metadata access. Any additional round trip can directly slow down training jobs.
• Scale-related overhead: Large directories, such as those in the trash, can contain tens of billions of entries. Global information, including mount configurations and capacity data, is also queried repeatedly.

To address these challenges, JuiceFS Enterprise Edition has evolved its metadata engine with a custom-built metadata service. The service distributes hotspots, reduces access latency, and supports horizontal scaling.

Metadata engine design: in-memory, lock-free, and multi-zone

The Enterprise Edition metadata engine combines an in-memory architecture, single-threaded lock-free processing, and Raft.

All core data structures are processed by a single thread. This design ensures atomicity by default and avoids latency spikes caused by lock contention between threads. Keeping metadata in memory eliminates the need to access disks for metadata requests, keeping per-request processing latency in the hundreds of microseconds. Raft handles replication across nodes and automatic failover. A single metadata process can manage approximately 300 million files with 30 GiB of memory.

However, an in-memory design also means that memory consumption directly affects hardware costs. At the scale of hundreds of billions of files, even a small difference in per-file memory usage can translate into substantial resource overhead.

JuiceFS Enterprise Edition uses several techniques to reduce memory consumption, including memory pools, manual memory management, serialization and compression of idle directories, and compact representations of small files. These optimizations have reduced average memory usage per file from nearly 600 bytes to approximately 100 bytes—about 27% of that of an HDFS NameNode and 3.7% of that of a CephFS MDS.

Higher memory efficiency allows the same hardware to manage more files. In other words, it significantly reduces the metadata infrastructure costs of managing hundreds of billions of files.

Building on the capabilities of a single process, Enterprise Edition uses multiple metadata zones to scale horizontally. Each zone manages part of the file tree, while clients combine the zones into a single namespace.

The design keeps subtrees complete whenever possible, so most operations stay within a single zone and require fewer distributed transactions. Cross-zone migration uses links and references, remaining transparent to POSIX paths. Extremely large directories can also be split into multiple virtual subdirectories, allowing finer-grained migration and preventing a single large directory from remaining a long-term burden on one zone.

The high-order bits of each inode encode its zone information. Clients automatically discover the zone topology and connect to all zones. Adding a zone therefore requires no changes to application access patterns.

For reliability, each zone consists of three instances: a service instance, a hot standby instance, and a cold standby instance. The service and hot standby instances use different memory compression strategies. Snapshots and change logs are continuously persisted to disk and can also be backed up to a remote location (see the Trash, background tasks, and audit backups section).

Version evolution and scale validation

Starting with v5.2, Enterprise Edition focused on improving multi-zone stability and reducing metadata memory consumption, bringing file management into the hundreds-of-billions range. In production, a single cluster now manages more than 50 billion files while keeping metadata access latency at the millisecond level.

In v5.3, the maximum number of zones increased to 1,024, allowing a single file system to support more than 500 billion files. The release also introduced automatic hotspot migration and supporting operations tools. Frequently accessed directories can be split into smaller units and migrated concurrently to distribute hotspots more quickly. Periodic self-checks detect and repair intermediate states that may remain after cross-zone migrations.

In v5.4, the focus expanded from file count to client scale. Metadata nodes and network proxies can now be deployed together, and clients can connect to any node. Requests for other zones are forwarded internally by the cluster. Combined with connection reuse, this architecture supports up to a million client connections. In the last section, we’ll cover this design in more detail.

Data lifecycle management: built-in deletion, recovery, and auditing

Data lifecycle challenges when deleting petabytes of data

Deleting petabytes of data requires more than reclaiming storage space promptly. The system must also support recovery from accidental deletion, deletion traceability, and audits for orphaned objects.

Object storage can accumulate billions of objects awaiting reclamation. An accidental deletion can disrupt training jobs, while leftover objects must be continuously detected and accounted for.

Trash, background tasks, and audit backups

The trash is enabled by default in Enterprise Edition. Deleted files first move to the trash, where the retention period is configurable (typically seven days). During this period, users can restore files through the console or a restore command. Restoration only updates metadata; it does not copy the underlying data.

Starting with v5.3, the trash in multi-zone environments records each file's original path to ensure that files are restored to the correct location. In v5.4, users can filter deleted files by time range and keywords. This makes it easier to find accidentally deleted files among large volumes of deletions.

Storage reclamation is handled by a unified background tasks framework. Tasks such as chunk compaction, trash cleanup, and cross-cloud replication are scheduled by the metadata service and executed concurrently across clients throughout the cluster.

This approach offers three main benefits:

• Scalable deletion: Deletion capacity scales with the number of clients. To speed up reclamation, simply add more mount points.
• Controllable execution: Deletion traffic can be limited by access token. Background tasks can be disabled for specific clients, while the metadata service centrally configures the execution windows for chunk compaction.
• Observable progress: Both chunk compaction and deletion tasks expose monitoring metrics.

Auditing and disaster recovery are also built into the system. Change logs record every deletion operation and the session that initiated it. Combined with audit logs, they help answer the questions: who deleted what, and when?

The console provides centralized configuration for metadata snapshots and change logs. By default, backups are sent to remote object storage every hour. Local backups are retained for two days, and a retention period of at least 30 days is recommended.

On the object storage side, garbage collection and consistency-checking tools audit orphaned objects. In v5.4, both tools gained an external sorting mode that uses disk space instead of memory for sorting. This keeps memory consumption manageable when checking hundreds of millions of files. External sorting is intended for inspection and assessment only; it does not delete any objects.

Data insights and dataset branching

Deletion decisions require reliable data. The console's monitoring dashboard provides metrics such as the number of files in the trash, pending object chunks for deletion, and fragmented chunks. File searches filter results on the metadata side, supporting hot/cold data analysis and data aging assessments. Directory, user, and group quotas help enforce storage budgets across workloads.

For dataset preparation, v5.4 supports cross-zone directory cloning. Cloning a directory containing 100 million files takes approximately 1 minute and 40 seconds, or about 1 million files per second.

Data path: balancing performance and cost with caching and mirroring

Accessing EB-scale data: throughput, storage costs, and cross-cloud access

Training workloads require throughput at the terabyte-per-second level. Dataset loading involves large volumes of random reads, while training checkpoint writes involve sequential writes of large files. These workloads have different IOPS and write-throughput requirements.

At the same time, storing all EB-scale data in a hot storage tier is not possible. Compute resources deployed across multiple clouds also need to access data from nearby locations whenever possible.

Therefore, read performance, storage costs, and data locality must be addressed together.

Multi-level caching and distributed cache groups

Enterprise Edition turns shared caching across clients into a distributed caching solution. Multiple clients can form a cache group and share data blocks through consistent hashing. When a node fails, the system automatically removes it from the group. Cache misses fall back to object storage, so read operations can continue without interruption.

Application clients can also be configured to use cache without serving cached data to other clients. This allows a dedicated cache cluster, made up of stable nodes, to serve frequently accessed training data.

The read path is hierarchical. Data retrieved from a cache group is also retained in the local cache of the requesting client, so subsequent accesses do not need to traverse the network again.

Enterprise Edition also provides a comprehensive set of tools for cache group operations and efficiency:

• Data warm-up to load data into the cache before workloads need it.
• Cache group backup to improve cache hit rates while group membership changes.
• Scale-out protection to let new nodes retrieve data from existing nodes on a cache miss, reducing requests to object storage.
• No-backfill-on-miss to protect hot data from being displaced by less frequently accessed data.
• Cache block compaction to reduce the number of files on large cache disks.
• Range-based warm-up to warm up specific byte ranges within files.

The transport layer has also evolved through the introduction of the custom IORPC protocol in v5.2 and RDMA support starting with v5.3.

The following benchmarks illustrate the performance improvements:

• v5.2: With 100 nodes equipped with 100 Gbps network interfaces, aggregate read bandwidth reached 1.2 TB/s.
• v5.3: With 160 Gbps network interfaces and 1 MB random reads, CPU utilization dropped to approximately one-third of that of the TCP-based approach.
• v5.4: In an environment with 400 Gbps client nodes and 800 Gbps cache nodes, a single client achieved 45 GB/s of read bandwidth, while a single cache node achieved 90 GB/s. Network interface utilization exceeded 90%, and random-read IOPS under high concurrency more than doubled.

Cross-region mirroring and on-demand synchronization

To bring data closer to compute resources, Enterprise Edition provides mirror file systems. A metadata replica is deployed in the mirror region and kept in sync with the source. Clients prioritize reads from local object storage. Writes follow a read/write separation model: write operations are committed to the source cluster to maintain consistency across regions.

Starting with v5.3, read requests can fall back to the source cluster when the mirror is unavailable. In v5.4, mirroring gained on-demand data synchronization. Only metadata is continuously synchronized. Object data is fetched on demand when first accessed and then cached locally. This avoids synchronizing an entire dataset when only a small portion is needed.

If the goal is to accelerate metadata reads alone, read-only metadata nodes can be deployed directly.

Elastic capacity and onboarding existing data

Object storage provides the underlying capacity. Storage can scale up or down on demand, and a single file system can manage multiple buckets across regions and providers while providing caching for all of them.

Existing data in object storage can be onboarded transparently through the import feature. The process creates metadata entries without copying the underlying data. Starting with v5.2, data can be imported on demand when first accessed. Starting with v5.3, all imported objects can benefit from cache acceleration. Hot data is served by caches and mirrors, while cold data remains in object storage. This means that exabyte-scale capacity no longer needs to be provisioned based on peak demand.

Clients and control plane: isolation, observability, and operations at a million connections

Control plane challenges at the million-client scale

Hundreds of thousands of clients connect through Kubernetes and CSI, the S3 Gateway, SDKs, and other interfaces. The number of active connections can reach one million.

At this scale, any client could overwhelm a shared metadata cluster. Each client also generates hundreds of metrics, making direct collection impractical due to time-series cardinality. Multi-tenant environments require clear access controls and resource budgets. Upgrades must also minimize disruption to training workloads.

Connection reuse: supporting a million clients

In v5.4, Enterprise Edition expanded client connectivity to the million-client scale by introducing a network proxy layer. The proxy can be deployed together with metadata nodes or separately.

Clients can connect to any metadata node. Requests for the local zone are handled directly, while cross-zone requests are forwarded internally. Multiple client sessions share physical connections between nodes.

In stress tests, 10 internal proxy nodes handled one million client connections while requiring only about 400 physical backend connections to metadata service nodes.

Session management was also optimized. The system uses index-based lookups instead of full scans, while paginated queries and sampled reporting keep the amount of session state under control. Expired sessions are cleaned up in batches.

Multi-tenant isolation, observability, and operations

In multi-tenant environments, Enterprise Edition uses access tokens to define client access boundaries. These controls include allowed IP ranges, read/write permissions, accessible subdirectories, and whether a client can participate in background tasks.

In v5.4, Enterprise Edition extended the existing single-subdirectory restriction with a whitelist mechanism. This allows multiple top-level subdirectories to be configured for a client.

Resource controls cover different types of traffic, including uploads, downloads, deletions, and background tasks. The system also supports session-level CPU quotas to prevent a single tenant from continuously consuming resources needed by the shared cluster. In v5.2, global rate-limit messages were consolidated to reduce the overhead of synchronizing rate-limit state.

The client access layer supports multiple interfaces and client types, including POSIX through FUSE and CSI, the S3 Gateway, the Hadoop SDK, the Python SDK, and Windows clients. This flexibility accommodates different training and data-processing workflows.

For observability, the console integrates Prometheus and Grafana. It provides metrics for file systems, client sessions, and cluster health, and can also integrate with existing monitoring and alerting systems.

For daily operations, Enterprise Edition has supported centralized client upgrades and diagnostic information collection through the console since v5.2. The metadata service also supports rolling upgrades without service interruption.

Conclusion

From v5.2 to v5.4, the evolution of JuiceFS Enterprise Edition for large-scale environments has not been about adding isolated features. Instead, the goal has been to make the metadata engine, data lifecycle, data path, and client and control planes all scale together.

• Metadata: An in-memory engine, multi-zone architecture, and automatic balancing extend single-node capabilities to hundreds of billions of files and beyond.
• Data lifecycle: The trash, background tasks, auditing, and backups are integrated into a unified framework.
• Data path: Distributed caching, RDMA, mirroring, and data import bring hot data closer to compute resources while preserving the capacity elasticity of object storage.
• Clients and control plane: Connection reuse, access controls, rate limiting, and centralized observability support hundreds of thousands to millions of connections.

These capabilities address common challenges that emerge as infrastructure scales. The goal is to let the file system handle more of the foundational work itself, reducing the need for platform teams to build and maintain additional systems around it.

JuiceFS Enterprise Edition is already running in production environments at the exabyte scale, and its architecture continues to evolve. We hope these design choices offer useful insights to teams planning their AI data infrastructure.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.