JuiceFS Enterprise Edition 5.2: Supporting Hundreds of Billions of Files and Windows Clients

JuiceFS Enterprise Edition 5.2 has been released, marking a milestone in managing file systems at the scale of hundreds of billions of files. This upgrade focuses on improving stability for hyperscale clusters, optimizing distributed cache network performance, and enhancing usability and security, aiming to support complex high-performance application scenarios such as high-concurrency access.

Designed for high-performance scenarios, JuiceFS Enterprise Edition has been widely adopted in the machine learning field since 2019. It has become one of the core infrastructures in the AI industry. Our customers include large language model (LLM) companies like MiniMax, generative AI service and application providers like Lepton AI, autonomous driving companies such as Momenta and Horizon Robotics, and numerous leading tech enterprises across industries using AI technologies.

In JuiceFS Enterprise Edition 5.2, we support hundreds of billions of files, achieve TiB-level aggregate bandwidth, introduce Windows client support, and deliver other key optimizations. Dive into the article to learn more.

Hyperscale stability enhancement: Supporting hundreds of billions of files

In AI applications, particularly in fields like autonomous driving, data volumes are growing at an unprecedented rate. For example, a single autonomous vehicle can generate up to 20 TB of data daily. With the widespread adoption of autonomous driving technology, this data volume is expanding exponentially. Distributed file systems must not only meet current application capacity requirements but also possess the capability to support continuous future expansion. However, scaling from tens of billions to hundreds of billions of files presents severe challenges for distributed file systems, including:

• Managing significantly increased metadata complexity
• Maintaining strong consistency while scaling performance in high-concurrency environments
• Ensuring system stability during seamless scaling

The multi-zone mechanism, which implements metadata partitioning and parallel processing, serves as JuiceFS' core strategy for large-scale clusters. In version 5.2, we achieved another breakthrough in file system scale by optimizing the stability of the multi-zone mechanism and memory compression strategies. In addition, this version addresses issues in extreme scenarios for the multi-zone mechanism, further enhancing system reliability.

This optimized architecture has been rigorously validated in production environments across multiple enterprises, demonstrating:

• Single-cluster management of over 50 billion files
• Consistent average metadata request latency of 1 ms
• Daily read and write bandwidth of 50 GiB/s and 30 GiB/s respectively
• Peak read speeds exceeding 200 GiB/s

To further validate version 5.2's capabilities in high-concurrency metadata processing and system stability, we deployed an even larger test cluster internally. Using the mdtest tool, we:

• Wrote 130 billion files within 12 hours
• Sustained mixed workloads at this scale for 24 hours

Hardware configuration:

• 36 GCP c3-standard-176 servers
• Each equipped with:
  • 176 vCPUs
  • 704 GB RAM
  • 100 Gbps network interfaces

Performance results:

• Total files written: >130 billion
• Metadata cluster stability: Maintained 8 million IOPS on average

Distributed cache network optimization: Achieving TiB-level aggregate bandwidth

As data volumes and model sizes experience explosive growth, scenarios where multiple clients frequently access the same data have become increasingly common. Distributed caching aggregates local caches from multiple nodes to form a large-capacity cache pool, improving cache hit rates, enhancing read bandwidth and IOPS, while reducing read latency to meet high-performance requirements.

However, data exchange between nodes heavily relies on network performance. Insufficient bandwidth can limit transfer speeds and increase latency; high network latency may degrade cache responsiveness and reduce system efficiency. In addition, the CPU resources consumed by network data processing can also become a bottleneck, constraining overall performance.

In version 5.2, we implemented several optimizations for network transmission between distributed cache nodes, including:

• ​​Zero-copy technology​​ to reduce system calls and data copying
• ​​Multiplexing, small packet aggregation​​, and ​​SO_RCVLOWAT​​ (receive low watermark) socket configuration to address CPU utilization degradation caused by network fragmentation
• ​​Optimized cyclic redundancy check (CRC) verification​​ to reduce CPU overhead

These optimizations significantly improve distributed cache performance, reduce system latency, and meet the demands of high concurrency and large-scale access.

The figures below show the key performance improvements in Enterprise Edition 5.2:

The figure above demonstrates that after a series of optimizations, version 5.2 has significantly reduced CPU overhead:

• ​​Client CPU overhead​​ was reduced by more than half.
• ​​Cache node CPU overhead​​, benefiting from zero-copy technology, was reduced to one-third of pre-optimization levels.

Compared to version 5.1, JuiceFS 5.2 demonstrates significantly improved network bandwidth utilization. In a 15-node GCP cluster with 200 Gbps nodes, aggregate bandwidth increased by about 20%.

Furthermore, we deployed a distributed cache cluster consisting of 100 GCP nodes (100 Gbps/node). When conducting sequential read tests using the fio tool, the cluster achieved:

• Aggregate read bandwidth of 1.2 TB/s
• Near-saturation of TCP/IP network bandwidth

These results demonstrate the system's enhanced horizontal scaling capabilities, proving its ability to:

• Fully leverage hardware performance at scale
• Support high-concurrency access scenarios
• Handle massive data volumes efficiently

First release of Windows clients (beta)

Many Windows users hope to use JuiceFS for file sharing. Due to stability and compatibility issues in earlier versions, the Windows client remained in internal testing until now. With this new release, we officially launch the Windows client (beta), delivering significantly improved usability and stability to meet most daily usage scenarios.

Key enhancements include:

• Fixed multiple compatibility issues​​ in API calls and user identity management
• Expanded toolchain support​​, with subcommands like debug, stat, and info now fully functional on Windows
• Added background service parameters​​, enabling direct mounting as a system service without third-party tools

We invite you to download and test this beta release. Please report any issues encountered, and our team will promptly address feedback to refine the Windows experience.

Other important feature optimizations

Features

• Enhanced Python SDK compatibility with improved performance.
• Optimized shared block device support: JuiceFS enables clients to use block devices to accelerate data I/O. Previous versions required mounting the block device on every client node, requiring multi-machine mounting support. The new version removes this constraint, allowing a single client to mount the block device locally while other clients access it remotely via the network.
• On-demand import of object storage files: JuiceFS supports importing existing objects from object storage into the file system for cross-protocol access and caching. However, batch imports could overwhelm metadata services and take a long time. The new version introduces on-demand import—objects are only imported upon first access after configuring the bucket association.
• Support for cache disks of different sizes: Data caching is a core JuiceFS feature. Earlier versions enforced uniform cache disk sizes. The new version introduces a weight-based mechanism to using disks of different sizes, improving flexibility and resource utilization.

Stability

• Reduced read amplification: JuiceFS uses readahead and prefetch to enhance file read performance. However, these strategies could cause significant read amplification in specific scenarios, unnecessarily consuming object storage bandwidth. The new version adjusts these mechanisms, significantly reducing read amplification when handling specific file sizes and 4 MiB random reads.
• Enhanced global quality of service (QoS)​​: File system-wide QoS relies on metadata service coordination. As the number of clients increases, the additional QoS-related messages multiply, sometimes even affecting normal application requests. The new version merges QoS-related requests, greatly reducing the pressure on metadata services.

Usability

• One-click client upgrade via the console: Version upgrade is an inevitable part of software maintenance. Previously, upgrading JuiceFS clients required administrators to log into each client node and manually execute upgrade commands. The new version introduces a one-click upgrade button in the console, allowing quick and effortless upgrades.
• One-click collection of client debug information: In the new version, users don’t need to log into individual nodes to gather debug information. Instead, users can directly execute debug commands from the console and download the results.
• Enhanced file recovery with the restore command: When users need to recover files filtered by multiple keywords or restore from a complete file list, the new restore command offers a more user-friendly experience and better performance.

Security

Encrypted object storage credentials: Client configuration files locally store access credentials such as object storage access keys and secret keys. If leaked, these credentials could have catastrophic consequences for user data. In the new version, such credentials are now advanced encryption standard (AES) encrypted by default, and operators are prohibited from explicitly decrypting them.

Summary

JuiceFS Enterprise Edition 5.2 achieves a breakthrough in scaling from tens of billions to hundreds of billions of files, laying a robust foundation for the continuous growth of AI workloads. This release significantly reduces CPU overhead and delivers TiB-level network bandwidth through optimized distributed caching performance. In addition, it introduces the first Windows client (beta) and enhances multiple key features, further improving system performance, usability, and stability.

Cloud service users can now experience JuiceFS Enterprise Edition 5.2 directly online, while on-premises deployment users may obtain upgrade support through official channels.

The optimizations in v5.2 are driven by real-world customer needs and industry trends. We remain committed to tracking advancements in AI, focusing on product innovation to help enterprises tackle evolving data challenges with a simple and adaptable architecture.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Slack.