GPFS vs. Alluxio vs. JuiceFS: Architecture and Use Cases Compared

In AI workloads, scenarios such as training, inference, model distribution, agents, and data lakes each have distinct requirements for throughput, latency, concurrent access, POSIX compatibility, consistency, cost, and operational complexity.

In this article, we’ll first examine the typical I/O patterns of AI workloads, identifying the practical challenges these workloads pose to storage systems. Then, we’ll compare General Parallel File System (GPFS), Alluxio, and JuiceFS, analyzing their architectural differences, file system semantics, caching mechanisms, cost models, and applicability across different AI scenarios.

I/O patterns and storage challenges in AI workloads

Based on our interactions with enterprises across various AI domains, we can categorize common requirements as follows.

Autonomous driving: large‑scale data generation and training

Autonomous driving is one of the most data‑intensive AI storage scenarios. Large fleets of data‑collection vehicles continuously generate images, videos, sensor data, and trajectory logs, which undergo cleaning, labeling, and format conversion before entering model training pipelines. Common data formats include .mcap, .pack, TFRecord, and LMDB. These environments require highly stable and scalable storage due to the long data pipelines, diverse formats, and heavy training workloads.

LLM training: full‑pipeline data access

Foundation model scenarios cover multiple stages, including data cleaning, model training, checkpoint reads/writes, and inference services. Model weights, training data, and intermediate results are accessed repeatedly across different stages. The storage system must sustain long‑running jobs while maintaining stable data access during task failures, node outages, and training recovery.

Multimodal models: coexistence of small files, aggregated data, and model files

AIGC scenarios include text‑to‑image, image‑to‑image, text‑to‑video, image‑to‑video, and 3D generation. Training inputs may consist of many individual images or video clips, or they may be aggregated into formats like LMDB or Parquet to improve training efficiency. The training process also generates checkpoints and outputs model files such as safetensors. This makes the data landscape more complex than in single‑task training scenarios.

Compute platforms: multi‑cloud collaboration and model distribution

Compute platforms focus on distributing models and data across multiple environments. Users may pull models from external repositories or upload their own, and then run training or inference workloads across different clusters and cloud environments. The key challenge is reducing redundant copying while enabling consistent access to the same data across different compute environments.

Quantitative finance: balancing performance and cost

Quantitative finance typically involves smaller data volumes than autonomous driving or AIGC. However, as techniques such as Transformer‑based models, neural network training, time‑series modeling, and market graph structure analysis are increasingly adopted, storage cost is becoming a more explicit selection factor.

In recent discussions with quantitative finance customers, we’ve observed growing attention to storage costs. High‑performance parallel file systems like GPFS are inherently performance‑oriented, especially in large‑capacity, all‑flash configurations. In some on‑premises deployments, the storage investment for a large‑capacity all‑flash GPFS cluster can approach the cost of an entire 5090 GPU cluster.

AI agents: data sharing in short‑lived sandboxes

AI agents represent a rapidly emerging scenario. They often involve many short‑lived sandboxes—each executing a subtask with a lifetime of just seconds or even less.

While these tasks run briefly, the context, model files, tool files, and intermediate results must be shared among subtasks. If each sandbox independently mounts a file system, and the mount process itself takes several seconds, task scheduling efficiency suffers. A more practical approach is to pre‑mount the file system on the host and then expose it to sandboxes via bind mounts, PVCs, or similar mechanisms.

From a storage perspective, AI agents are less concerned with sheer capacity and more focused on data continuity, shared access, and mount efficiency within short‑lived tasks. As agent complexity grows, demands on file system semantics and data sharing capabilities will continue to rise.

The table below shows typical I/O patterns and storage requirements for AI workloads:

GPFS vs. JuiceFS

From PFS to GPFS: capabilities and limitations of parallel file systems

To understand GPFS, it helps to first understand parallel file systems (PFSs). Intuitively, a parallel file system separates metadata and data, enabling multiple clients to access underlying storage resources in parallel. In this architecture, metadata and data travel different paths. Clients do not need to funnel all I/O through a single node; they can concurrently access underlying disks or storage nodes. This allows hundreds of compute nodes to simultaneously read and write block devices or storage resources, breaking the network bottleneck of a single path and enabling horizontal scaling.

GPFS, Lustre, and BeeGFS are all typical examples of parallel file systems.

GPFS, later renamed IBM Storage Scale, is a mature parallel file system that has been widely used in high-performance computing environments for many years. It provides strong advantages in throughput, concurrent access, and consistency, making it suitable for workloads that require high performance and reliability.

However, as AI infrastructure continues to scale and organizations increasingly prioritize cost optimization, the high cost and operational complexity of GPFS can become important considerations during storage selection.

Typical GPFS workloads include quantitative finance, genomic analysis, physics simulations, and weather forecasting.

Architecture advantages and trade-offs: metanode, token locks, and strong consistency

The architectural advantages of GPFS mainly come from two mechanisms:

• Metanode, which affects metadata coordination
• Distributed token locking, which determines how consistency is maintained during concurrent access

Metanode-based metadata coordination

Under the Metanode model, a GPFS cluster usually consists of:

• I/O servers
• data disks
• metadata disks

Metadata is stored on dedicated metadata disks. Unlike systems with centralized metadata services, GPFS allows clients to participate in parts of metadata coordination.

In other words, GPFS assigns different clients as coordination nodes for different files or inodes instead of routing all metadata requests through a fixed centralized service.

Therefore, GPFS clients need to maintain communication with each other, typically through port 1191.

When network issues or connection failures occur, the cluster manager must determine node status and remove failed nodes from the cluster to prevent split-brain conditions and data inconsistency.

This design helps distribute metadata coordination pressure. Under stable network and storage conditions, it can support high levels of concurrent access.

However, it also introduces higher environmental requirements. If network quality is unstable or storage latency increases, coordination efficiency may degrade. In severe cases, users may experience:

• Request stalls
• Long waiters
• Recovery operations requiring node restarts

These problems do not necessarily indicate a weakness of GPFS itself. Instead, they reflect that high-performance, strongly consistent architectures require reliable networking, disk, and cluster state management.

Distributed token locks

Distributed token locking is another key mechanism behind GPFS consistency.

GPFS manages read and write operations through tokens:

• Read operations require read tokens
• Write operations require write tokens

When multiple clients access the same file, GPFS controls concurrent access by granting, revoking, and transferring tokens.

For example, if Client A holds a write token for a file and Client B wants to read or modify the same file, GPFS must revoke the token from A. A must flush dirty data to disk and release the token before B can proceed. This ensures strong consistency, but it also relies on stable, fast network and disk responses.

If disk writes become slow or network communication is interrupted during token revocation, long waiters can appear. In practice, requests waiting on Revoke Token or Reopen Token operations are not uncommon. This illustrates GPFS' inherent trade‑off: token‑based strong consistency and concurrency control come at the cost of increased system complexity.

Early InfiniBand networks or high‑quality fiber networks could reliably support this mechanism due to their low latency and high reliability. However, in some newer deployments—particularly those using RoCE—suboptimal network conditions, hardware quality, or operational expertise can amplify stability challenges in token coordination, especially when hundreds of clients coexist in a single cluster.

Overall, this mechanism enables high‑performance parallel access but demands high network, disk stability, and cluster operational capabilities. Client anomalies also require handling token recovery and cluster state restoration. Therefore, it’s essential to assess whether your team has the necessary deployment, monitoring, and troubleshooting skills.

Based on practical experience, GPFS deployments also require attention to several engineering considerations:

• mmap scenarios: GPFS employs special mechanisms for mmap, such as pagepool, to reduce memory copying and improve performance. However, this also creates complex interactions with OS memory management. Large‑scale reliance on mmap patterns is generally not recommended without thorough validation.
• Hot files and large directories: Hot files, hot directories, massive small files, or many clients concurrently accessing the same directory can become performance bottlenecks due to metadata coordination and lock contention. Mitigation typically involves directory splitting, data sharding, and access pattern optimization.
• Operational management and monitoring: GPFS' management interface is not particularly user‑friendly for newcomers, and some monitoring information is not intuitive. Many teams supplement with external monitoring systems like Grafana to better observe cluster state, performance metrics, and anomalies.
• Cluster Export Services (CES), Active File Management (AFM), and other components: While these features enable more complex scenarios, they also introduce additional configuration, operational, and troubleshooting overhead. Teams without long‑term GPFS experience should carefully evaluate this complexity upfront.
• Capacity planning: Many GPFS or CPFS deployments emphasize expansion over contraction. Capacity planning early on is crucial to avoid either excessive initial capacity (and long‑term cost) or insufficient capacity (hampering business growth).

Performance comparison

Intuitively, many people assume that JuiceFS—built on object storage and independent metadata services—cannot be directly compared to GPFS. However, in certain AI workloads, both systems do present comparable alternatives.

It’s important to note that GPFS performance depends heavily on the synergy of underlying disks, storage servers, networking, clients, and the parallel file system itself. JuiceFS, on the other hand, is influenced by object storage performance, metadata services, client caching, distributed cache groups, and mount modes. Therefore, comparisons should not be reduced to single performance numbers—they must account for I/O patterns, deployment architecture, data scale, and access paths.

The following tests were performed with JuiceFS Enterprise Edition. The Community Edition shares the same core architecture, so Community Edition users can also refer to these testing methodologies and results.

Sequential reads: GPFS better per node, JuiceFS scales through cache groups

In single‑node scenarios, the performance models differ markedly. In our tests, with two 400 Gbps NICs, a single GPFS node achieved about 100 GB/s for sequential reads.

Under TCP mode (200 Gbps NIC), a single JuiceFS node peaked at about 20 GB/s. With RDMA (two 400 Gbps NICs), it reached about 55 GB/s. For higher aggregate throughput, JuiceFS can scale horizontally by adding cache nodes. For example, in one autonomous driving customer deployment with about 150 cache nodes (each with 160 Gbps NICs), the aggregated application throughput reached about 2.3 TB/s.

Sequential writes: GPFS excels at synchronous writes; JuiceFS uses writeback for scale

For sequential writes, GPFS holds an advantage under synchronous write semantics. Its write capability stems from the underlying parallel storage system—data written is immediately accessible by other clients under strong consistency semantics. This is well suited for scenarios requiring write reliability, real‑time visibility, and consistency.

JuiceFS sequential write performance depends on whether writeback is enabled. In synchronous mode, writes go to the backend object store, and performance is constrained by backend storage, protocol overhead, and network latency. With writeback enabled, data is first written to client‑local cache and later uploaded asynchronously—aggregate throughput improves, but real‑time visibility and consistency semantics change.

Thus, GPFS is better for scenarios requiring synchronous persistence and real‑time visibility. JuiceFS with writeback is suitable for workloads that can tolerate asynchronous upload semantics.

Random reads: GPFS better at high concurrency, JuiceFS competitive in certain scenarios

In 4K single‑process random read tests, we compared JuiceFS, GPFS, and local disk /tmp (EXT4).

The figure below shows the test results:

• At iodepth 1, 2, and 4, JuiceFS outperformed GPFS.
• At iodepth 8, GPFS surpassed JuiceFS and stabilized around 80K IOPS.
• JuiceFS peaked around 68K IOPS at iodepth 4 and 8 and then gradually declined at deeper I/O depths.

Multi‑process random reads present more complex behaviors. When multiple processes concurrently read the same file, GPFS' consistency and lock mechanisms can introduce extra overhead. To validate this, we tested both reading the same file and different files.

The following figure shows the test results:

• GPFS reads of different files scaled rapidly with numjobs, reaching ~433K IOPS at numjobs=12.
• GPFS reads of the same file dropped after numjobs=2, suggesting consistency coordination overhead.
• JuiceFS reached ~258K IOPS at numjobs=12 and stabilized around 250K thereafter.

All tests were run with local caching disabled, and data was served from distributed cache. The results show that GPFS has higher random read performance under high concurrency, while JuiceFS remains highly capable—sufficient for most AI training requirements.

Random writes: GPFS superior at high concurrency

Random writes better expose architectural differences between the two systems. JuiceFS' random write performance depends on whether writeback is enabled. Without it, performance is mainly constrained by the object storage backend. With writeback enabled, performance reflects the client's local cache path capabilities.

Under writeback enabled and cache‑dir set to local NVMe storage, 4K multi‑process random write results showed:

• From numjobs 1 to 3, JuiceFS increased from ~28K to ~56K IOPS, significantly higher than GPFS.
• At numjobs=12, both were close (~51K vs. ~50K).
• Beyond numjobs=16, GPFS continued to rise to ~66K (at 16) and ~81K (at 20), while JuiceFS was in the 50–60K range.

These results indicate GPFS holds an advantage in random write performance, while JuiceFS closes the gap significantly when writeback is enabled. However, random writes are not common in AI workloads, so this should not be a primary evaluation criterion.

Selection summary

The main differences between GPFS and JuiceFS are summarized in the table below:

GPFS is better suited for environments with ample budgets and stringent requirements for low latency, strong consistency, high‑concurrency access, and random write capability—typical of traditional HPC, scientific computing, and some quantitative finance applications. Its performance also depends on stable networking, storage hardware, and professional operational expertise.

For AI research, training, inference acceleration, and large‑scale data management challenges involving scalability, performance, cost, and multi‑cloud management, JuiceFS is the more suitable solution—and has been validated in production across all the aforementioned domains.

Alluxio vs. JuiceFS

Alluxio is another solution that enterprises often compare with JuiceFS when evaluating AI storage architectures.

Both solutions can provide filesystem access and cache acceleration on top of object storage. However, they differ significantly in product positioning, data organization models, and metadata architectures.

From the perspective of product evolution, the two projects have taken different approaches:

• Alluxio’s recent AI-focused development has mainly concentrated on its Enterprise AI product line. The latest release of the Alluxio open-source repository is v2.9.4, released in June 2024.
• JuiceFS continues to evolve both its open-source and enterprise editions. Many new capabilities are first introduced, validated, and refined in the open-source edition before gradually becoming available in the enterprise edition to serve more production users.

Key architectural differences

Data organization and consistency boundaries

Alluxio uses a 1:1 transparent caching model, which preserves the original organization of files in the underlying storage system. Existing data does not need to be imported or reorganized in advance, and the caching layer can be added or removed as needed.

JuiceFS, on the other hand, splits files into data chunks and stores them in object storage. It uses a metadata service to maintain file system semantics and data chunk mappings. From the application perspective, users see a complete file system, while the objects stored in the backend object storage are JuiceFS-managed data chunks rather than the original files.

This difference also affects the source of truth and consistency boundaries:

• For Alluxio, the actual data typically remains in object storage or other UFS systems. The cache layer mainly accelerates access. If applications bypass Alluxio and directly modify data in the underlying storage, additional cache invalidation, refresh, or synchronization mechanisms may be required.
• JuiceFS combines metadata services and object storage into a complete file system. File states and data mappings are managed centrally by JuiceFS, and application reads and writes go through JuiceFS. Therefore, its consistency boundary exists within the file system itself and does not rely on additional synchronization between the cache layer and backend storage.

Cache and namespace organization

Alluxio emphasizes a unified namespace and shared cache pool. It can integrate multiple underlying storage systems, including OSS, S3, HDFS, Ceph, MinIO, and NAS, into a single namespace and accelerate access through a unified distributed cache.

For organizations that already have multiple storage systems and do not want to migrate or reorganize existing data, this approach provides greater flexibility. However, when multiple workloads share the same cache pool, resource management and isolation typically require policies such as directory-based controls, priorities, and time to live (TTL) settings.

JuiceFS typically uses an independent filesystem as the management unit. Different object storage backends, such as OSS, COS, and TOS, usually correspond to different file systems with their own cache configurations.

Compared with Alluxio’s emphasis on a unified view across multiple data sources and shared caching, JuiceFS focuses more on clear boundaries between file system management, permissions, and data governance.

In addition, JuiceFS Enterprise Edition can connect multiple buckets into a single file system and use the same cache resources for acceleration. This enables multi-source data access scenarios while maintaining clear file system-level management boundaries.

Metadata architecture

Alluxio Enterprise distributes caching and part of the state management responsibilities across workers. Workers not only provide data caching but also participate in managing cache status and data location information. This brings data access closer to compute nodes or GPU nodes. Coordination and management components still exist, but cache-related states are not entirely centralized in an independent metadata service. As a result, when workers frequently restart or scale in and out, additional attention is required for recovering cache states and coordinating data locations.

JuiceFS manages core file system metadata through an independent metadata service.

The open-source edition supports metadata engines such as Redis and TiKV, while the enterprise edition provides a highly available metadata service to support consistency and transactional operations. Distributed cache nodes are responsible only for caching data blocks. Therefore, cache node failures mainly affect cache hit rates and access performance, but they do not change file system metadata states.

How architectural differences affect real-world usage

POSIX compatibility

Alluxio exposes data from object storage or other UFS systems through file system interfaces, but its file system semantics are not always complete.

Capabilities such as timestamps, file locking, hard links, symbolic links, extended attributes, and ACLs may require additional configuration or have usage limitations. For read acceleration and lightweight access scenarios, these limitations may not be significant.

However, if an application intends to use Alluxio as a full-featured file system, these factors should be carefully evaluated.

JuiceFS aims to provide complete file system capabilities on top of object storage. Therefore, JuiceFS continuously improves POSIX compatibility, covering common file system semantics such as directories, permissions, timestamps, and file locks, as well as more advanced system calls such as ioctl-based immutable and read-only settings. In production environments, JuiceFS POSIX compatibility already covers the vast majority of workloads and approaches full POSIX compatibility.

Write performance and write amplification

Alluxio preserves the original file format, which provides transparent access and reduces the cost of onboarding existing datasets. However, random writes, overwrites, and partial updates require careful consideration of write amplification. The reason is that object storage generally does not support in-place modification of object contents. If the backend maintains a 1:1 file layout, a small modification may trigger rewriting a larger amount of data, potentially requiring the entire object or file to be rewritten.

JuiceFS uses a chunk-based data layout. For random writes or append operations, JuiceFS only needs to process affected data chunks and update corresponding metadata, without being constrained by the original object storage file layout. Therefore, JuiceFS can better control write amplification for partial updates and random write workloads. However, if users need to restore JuiceFS data into the original file format in object storage, additional export or migration operations are required.

Deployment and engineering capabilities

Alluxio Enterprise supports full-component Kubernetes deployment. It also supports temporary write caching without persisting data to object storage, such as:

• temporary decompression workloads
• intermediate computation results
• short-lived data that can be discarded after use

These capabilities are closer to cache-layer acceleration and compute-side optimization scenarios.

JuiceFS Enterprise Edition currently typically deploys metadata services on virtual machines or physical servers, mainly due to requirements for metadata stability and system reliability.

JuiceFS focuses more on long-term filesystem capabilities, including:

• large-scale metadata management
• Trash and data recovery
• transactional operations
• cache governance
• seamless upgrades
• open-source ecosystem integration

In production, JuiceFS has already supported up to 500 billion files—an important demonstration of its capability as a complete file system in large‑scale data management scenarios.

Selection recommendations

The difference between Alluxio and JuiceFS is not simply about the number of features. They are designed to solve different problems:

• If data already exists in object storage, HDFS, Ceph, MinIO, or NAS, and organizations do not want to migrate or reorganize existing data but only need a pluggable cache layer to accelerate reads, Alluxio can be a suitable choice.
• JuiceFS is likely a better fit, if organizations want to build a complete file system on top of object storage while supporting:
  • mixed read/write workloads
  • full POSIX semantics
  • strong consistency
  • elastic metadata scaling
  • tens of thousands of concurrent clients
  • multi-cloud data management

Summary

GPFS, Alluxio, and JuiceFS address different core storage challenges:

• GPFS focuses on low latency and high-concurrency read/write performance for high-performance computing workloads.
• Alluxio primarily provides cache acceleration for existing datasets.
• JuiceFS provides a complete, scalable file system built on top of object storage.

Therefore, storage selection should not be based only on individual performance metrics. Organizations should first determine whether they need high-performance shared storage, a data acceleration layer, or a complete file system designed for multi-cloud and large-scale data management.

The final decision should consider consistency requirements, data scale, cost, and operational complexity.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and the community on Discord.