5.5× Faster Small-File Writes: Tuhu Built a Unified AI Storage Platform with JuiceFS + Ceph RADOS

TL;DR:
Tuhu leverages JuiceFS, TiKV, and Ceph RADOS to unify storage for AI training, AI inference, and analytics, supporting over 100 million files while achieving up to 5.5× faster small-file writes and 3.2× faster small-file reads.

Tuhu (9690.HK) is an integrated online and offline automotive service platform. By the end of 2025, we had grown to 162.3 million registered users and operated 8,008 service centers across China. As our business continued to expand, our infrastructure had to support increasingly diverse data management and compute workloads. Over the years, multiple storage systems—including NFS, Alluxio, MinIO, and SeaweedFS—had been deployed independently to meet different application requirements. While each solution addressed a specific use case, the overall storage architecture became fragmented, making data movement expensive and operations increasingly difficult.

When evaluating a unified storage architecture, we built a unified AI storage platform based on JuiceFS + TiKV + Ceph object storage cluster (RADOS), considering our existing private cloud infrastructure and Ceph deployment.

Today, this platform serves AI training, AI inference, and big data processing workloads through a single storage infrastructure. It currently manages over 100 million files. In low-concurrency benchmark tests, it achieved up to a 5.5x improvement in small-file sequential write performance and up to a 3.2x improvement in small-file read performance.

In this article, we’ll explain how we designed and deployed our unified AI storage platform on top of its existing Ceph infrastructure. We’ll also share several production optimization practices, including Ceph RADOS data paths, erasure-coded pools, write amplification reduction for small files, and stability improvements for containerized deployments.

From multiple storage systems to a unified storage foundation

In the early stages of our company's growth, storage infrastructure was built independently for different applications to maximize development speed and satisfy immediate application requirements. As the platform evolved, this approach gradually resulted in multiple storage systems coexisting in production.

The rapid adoption of cloud-native computing, large-scale AI training, and AI inference exposed the limitations of this architecture. Problems related to architectural consistency, data mobility, operational complexity, and storage performance became increasingly apparent.

We identified four major challenges.

• High operational complexity: Production environments relied on numerous storage solutions, including NFS, Alluxio, MinIO, Ceph, SeaweedFS, and various cloud block storage services. Each system had its own deployment model, access protocol, operational workflow, and troubleshooting methodology. As storage clusters continued to grow, day-to-day maintenance, capacity planning, and version upgrades became increasingly difficult.

• Data silos: Different storage systems exposed different interfaces and lacked a unified data access layer. AI training pipelines typically require POSIX-compliant file semantics, while surrounding tools for data processing, model management, backup, and archiving often depend on object storage APIs. Because these storage systems could not naturally share the same data, datasets frequently had to be copied or synchronized between platforms. This increased operational overhead, storage consumption, and workflow complexity.

• Rising storage costs: As business continued to grow, both the number of files and total storage capacity increased rapidly. Meanwhile, infrastructure costs—including hardware procurement, datacenter resources, and storage operations—continued to rise. Improving storage utilization while maintaining performance and reliability became a key objective for the infrastructure team.

• Higher performance requirements from AI workloads: AI training, AI inference, disaggregated storage and compute, and big data analytics placed higher demands on the underlying storage system's concurrency, throughput, access latency, and stability. In particular, large-scale small-file access, multi-node concurrent reads, model file loading, and online inference services required the storage system not only to provide stable capacity support but also to deliver predictable performance under high-concurrency conditions.

As a result, the existing architecture based on multiple independent storage systems could no longer support long-term application growth. We decided to consolidate fragmented storage resources into a unified storage foundation that would simplify operations while enabling seamless data sharing across different compute environments.

The new storage platform was designed around two major capability areas:

• Block storage for infrastructure and virtualization workloads: This capability provided reliable, high-performance storage for virtual machine images, cloud disks, and traditional infrastructure services.

• Unified data access for cloud-native workloads, including Kubernetes applications, traditional microservices, big data analytics, AI training, AI inference, and emerging middleware such as vector databases. Among these, AI training and inference posed more concentrated challenges in terms of file semantics, object interfaces, data throughput, small-file performance, and multi-node concurrent access.

Therefore, building a unified cloud storage foundation involved both block storage capability for virtualized environments and data access capability for AI and cloud-native workloads. The following section will focus on the latter—how to build a unified data access foundation based on JuiceFS to support AI training, AI inference, and big data processing.

Building a unified AI storage platform with JuiceFS

When designing a unified storage platform, we evaluated several architectural approaches based on our existing private cloud infrastructure, storage investments, and workload characteristics.

Our primary goals were to support large-scale data access, enable disaggregated storage and compute, provide multiple access protocols, and deliver high reliability without introducing unnecessary operational complexity.

Rather than building a new distributed storage system from scratch, we chose a modular architecture based on JuiceFS, TiKV, and Ceph RADOS. This approach allowed us to leverage our existing Ceph infrastructure while introducing a unified file system capable of serving AI training, inference, and big data processing workloads through a single storage platform.

A key design principle of this architecture is the separation of metadata and user data.

JuiceFS provides the unified file system layer, while the metadata engine and Ceph RADOS independently handle metadata management and data storage. At the storage layer, Ceph distributes data and manages failure domains using the CRUSH algorithm, providing a decentralized and highly reliable storage backend.

The architecture also adopts a lightweight client model.

JuiceFS clients are responsible for file system semantics, metadata operations, cache management, and request scheduling. Once data is written to Ceph RADOS, data durability—whether implemented through replication or erasure coding—is handled entirely by the storage cluster.

This division of responsibilities keeps compute nodes lightweight, minimizes interference with AI training workloads, and fully utilizes the reliability and scalability already provided by the existing Ceph deployment.

For our private cloud environment, this modular approach significantly reduced engineering and operational overhead compared with developing and maintaining an integrated distributed storage system. At the same time, it provides unified access to AI training, AI inference, and big data workloads through a common storage platform.

Unified AI storage architecture

Our unified AI storage platform consists of four logical layers.

Workload layer

At the top are the application workloads, including virtual machines, Kubernetes-based containerized applications, traditional microservices, AI training, AI inference, and Apache Spark jobs.

Cache layer

The cache layer consists of two components:

• DataCache Pool: It accelerates repeated access to training datasets, model files, and other hot data, reducing load on the storage backend.

• KVCache Pool: It’s designed for large language model (LLM) inference. It caches context data across multiple storage tiers, reducing latency and improving the responsiveness of online inference services.

Storage protocol layer

JuiceFS provides a unified data access entry point here. AI training jobs access data via POSIX FUSE or CSI Driver; big data Spark jobs access data through the JuiceFS Hadoop Java SDK; and object storage toolchains access the same data via JuiceFS S3 Gateway. This approach unifies file storage, object storage, and big data access interfaces around a single underlying data set, reducing data movement and redundant storage across systems.

It should be noted that block storage scenarios, such as VM cloud disks, are primarily handled by Ceph RBD, which is part of the overall cloud storage system's block storage capability. The focus of this article is the file, object, and big data access foundation built on JuiceFS. While both share the underlying Ceph storage resources, they differ in access semantics and service targets.

Storage engine layer

JuiceFS uses a metadata-data decoupled architecture. For file systems with hundreds of millions of files, metadata capabilities directly impact path resolution, directory traversal, file attribute queries, small-file access, and concurrent job startup performance. TiKV provides distributed transactions, strong consistency, high availability, and horizontal scalability, offering stable metadata support for large-scale file systems. We built a five-node TiKV cluster for the metadata layer.

For the data layer, we use Ceph RADOS as the underlying data storage engine for JuiceFS. Ceph RADOS reuses existing private cloud storage resources, reducing duplicate infrastructure costs. In addition, by creating an erasure-coded pool for physical data storage, we improve physical disk capacity utilization while maintaining reliability, mitigating cost pressures from data growth.

In this architecture, data communication follows two paths:

• The metadata path between clients and TiKV, handling path lookups, directory structures, file attributes, and state information

• The data read/write path between clients and Ceph RADOS, handling actual data block access

Through librados, JuiceFS interacts directly with the underlying Ceph RADOS. Compared to traditional gateway-based storage architectures, this reduces multi-layer forwarding overhead, allowing clients to access the storage cluster more directly after metadata interactions and shortening the data access path.

Production deployment and optimization practices

Critical scenario deployment and storage consolidation

In LLM application scenarios, our company has dedicated self-built compute clusters for model inference services such as MiniMax. Meanwhile, custom models developed by internal application lines have gradually been migrated to JuiceFS after passing production validation. As a result, core LLM inference and training applications have begun to unify onto the new file storage foundation, with the original NFS clusters and Alluxio storage clusters entering phased decommissioning.

In core middleware and backup scenarios, including large-scale ClickHouse deployments, multiple core middleware systems and full production backup tasks have also been migrated to JuiceFS. As the unified file foundation has proven its capacity, stability, and access efficiency, MinIO and SeaweedFS have also entered decommissioning, reducing the operational complexity of maintaining multiple parallel storage systems.

Cache acceleration for AI training and inference

For AI training scenarios, training data typically has a "write once, read many" pattern. We use JuiceFS Community Edition's local data caching capability to cache hot training data on compute nodes. This reduces backend storage pressure and improves multi-node concurrent read efficiency.

For LLM inference scenarios, models serving online requests or performing iterative token generation are sensitive to context data loading latency and throughput. With limited single-node GPU memory and memory capacity, we use a multi-level caching mechanism to cache context data across multiple levels, improving online inference response efficiency.

Benchmarking and small I/O optimization

After deploying the unified storage platform, we focused on evaluating its performance under small-file workloads.

Using juicefs bench and fio, we benchmarked small-file operations and random I/O to compare the new JuiceFS + Ceph RADOS architecture with the previous JuiceFS + MinIO deployment.

To minimize the impact of caching, all local caches were disabled during testing, allowing I/O requests to be served directly by the storage backend. It's worth noting that these benchmarks were intended to validate the architecture rather than measure peak cluster performance. The tests were conducted with only four client threads under low concurrency.

Even under low-concurrency test conditions, the new architecture demonstrated substantial performance improvements.

The two tables below show test results for JuiceFS + MinIO:

Item	Throughput	Latency
Large file write	3,377.65 MiB/s	1.21 s/file
Large file read	4,248.77 MiB/s	0.96 s/file
Small file write	147.2 files/s	27.17 ms/file
Small file read	1,176.8 files/s	3.40 ms/file

Item	IOPS	Throughput (MiB/s)
Random read	57,504	1,483
Random write	1,194	1,521

The two tables below show test results for JuiceFS + Ceph RADOS:

Item	Throughput	Latency
Large file write	4,984.12 MiB/s	0.82 s/file
Large file read	4,817.87 MiB/s	0.85 s/file
Small file write	817.0 files/s	4.90 ms/file
Small file read	3,776.4 files/s	1.06 ms/file

Item	IOPS	Throughput (MiB/s)
Random read	77,984	3,394
Random write	3,836	4,120

Test result highlights:

• Small-file write throughput improved by up to 5.5x, with significantly lower latency for individual file writes.
• Small-file read throughput improved by up to 3.2x, delivering higher read throughput and lower per-file read latency.
• Random I/O performance also improved. Both throughput and IOPS increased, with random-write IOPS improving by more than 3x.

From an architectural perspective, the JuiceFS + MinIO solution requires S3 interface and object storage gateway overhead, while the JuiceFS + Ceph RADOS solution uses librados for more direct interaction with the underlying storage cluster. This reduces gateway forwarding overhead. As a result, Ceph RADOS shows more significant performance advantages in scenarios sensitive to access paths, such as small-file writes and random writes.

However, in random small-file read tests, since local caching was disabled, data still needed to be retrieved from physical disks in real time. Therefore, read performance did not show the same order-of-magnitude improvement as writes, but still achieved a stable ~1.4x improvement overall.

Small-file space amplification and write amplification optimization

In erasure coding (EC) mode, traditional EC algorithms require alignment with stripe units. In a 6+3 configuration, even writing a 10-byte tiny file requires filling 6 data chunks to 4 KB each plus 3 coding chunks, resulting in 36 KB of physical storage—causing severe write amplification and thousands of times space waste.

To address this, we introduced Ceph's Fast EC feature to optimize small-object writes.

On one hand, Fast EC alleviates small-file space amplification. When enabled, small objects are no longer forced to fill entire stripes. For a 10-byte file, the logical size remains 10 bytes, while physical space occupies the first stripe; with a 4 KB minimum allocation unit and parity data, physical footprint is reduced to 16 KB.

On the other hand, Fast EC also improves read/write efficiency in small I/O scenarios. Reads no longer require fetching full stripes; writes can use parity delta writes (PDW) to reduce unnecessary network interactions and I/O overhead, improving overall small-file read/write performance.

JuiceFS Mount Pod stability optimization

In containerized environments, AI training applications occasionally encountered bad file descriptor errors, which severely impacted training efficiency. Investigation revealed that these errors were related to Mount Pod out-of-memory (OOM) conditions.

Periodic OOM due to metadata backups: When the cluster reached tens of millions or hundreds of millions of files, JuiceFS' default hourly metadata backups generated large backup files. Since these backup tasks were randomly assigned to clients in the cluster, and the underlying RADOS architecture imposes size limits on individual object writes (for example, 128 MB), backup tasks could fail repeatedly in large-scale file system scenarios, causing Mount Pod memory usage to grow continuously and eventually trigger OOM.

To address this, we disabled automatic backup on Mount Pods, preventing online clients from performing automatic backups. Instead, dedicated scheduled tasks handle metadata backups centrally, avoiding additional backup pressure on application clients.

Occasional OOM due to page cache growth: During frequent reads of large amounts of local cache, the Linux kernel page cache continued to grow. In such cases, even if the container's actual memory usage was only about 1.1GB, Kubernetes OOM decisions based on working_set_bytes—which includes page cache—could trigger OOM kills when the metric rose to high levels (for example, 33.5 GB), causing occasional client disconnections.

To address this, we increased limits.memory based on historical monitoring data. Then, we physically isolated cache resources, ensuring that different PVCs did not share the same local cache directory. For nodes with limited memory (such as CPU nodes), we capped local cache capacity (data cache size) within a reasonable range (for example, 10 GB to 50 GB) and enabled cache reclamation once the limit was reached. This prevents uncontrolled growth of the kernel page cache inside the container.

Through these optimizations, Mount Pod stability improved significantly in large-scale file system and high-frequency cache access scenarios. This provides a solid foundation for onboarding more AI training and inference workloads to the unified file foundation.

Future plans

We plan to focus on three aspects for the continued evolution of the AI cloud storage infrastructure:

• Unified management from private cloud to hybrid cloud: As AI training jobs are scheduled across both private and public clouds, the storage foundation needs stronger cross-environment data management capabilities. Building on the existing private Ceph RADOS architecture, we’ll continue exploring JuiceFS' integration with public cloud object storage.

By pushing cross-cloud data synchronization, data distribution, and access path management down to the storage layer as much as possible, we can reduce the application's awareness of differences between cloud environments, providing more consistent mount paths and access experiences. This will improve training job migration efficiency across different compute environments and reduce data management complexity in multi-cloud settings.

• Promoting lakehouse integration to reduce cross-system data movement: Previously, data movement between offline HDFS big data clusters and algorithm storage clusters relied on synchronization tools, increasing pipeline complexity, data redundancy, sync latency, and management costs.

We plan to further adopt and refine the JuiceFS Hadoop Java SDK so that Spark jobs in the algorithm layer can directly read and write the underlying unified storage pool. This will allow big data computing, algorithm training, and data archiving to operate around a single storage foundation, reducing cross-system data replication and redundant storage, further breaking down data silos, and advancing storage-compute separation and lakehouse integration.

• Building tiered storage to accelerate LLM inference: As LLM inference services scale, a single storage medium can no longer simultaneously meet cost, capacity, and latency requirements. We plan to build capabilities in data tiering and KVCache tiering.

• For data tiering: JuiceFS' current data tiering feature is primarily designed for public cloud object storage. For private Ceph RADOS environments, we plan to collaborate with the community to explore mounting multiple Ceph storage pools of different performance tiers (for example, NVMe Pool and HDD Pool) under the same JuiceFS file system, allowing data with different access frequencies to reside on appropriate storage tiers.

• For KVCache tiering: As LLM inference evolves toward prefill/decode (PD) separation in distributed architectures, relying solely on single-node GPU memory to hold context data faces capacity and cost bottlenecks. Building on the unified storage foundation, we’ll gradually explore a four-tier KVCache architecture composed of HBM (GPU memory), DRAM, local SSD, and shared storage, providing more flexible context data management and access acceleration for large-scale AI inference clusters.

If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and community on Discord.