How Tencent Cloud Built a Unified Storage Platform with JuiceFS and FoundationDB

As big data and AI workloads continue to grow, more enterprises are transitioning from tightly coupled storage-compute architectures to storage-compute separation architectures. However, in real-world deployments, enterprises often need to manage multiple storage systems simultaneously, including Cloud Object Storage (COS), S3, and HDFS. These backends differ in access protocols, authentication mechanisms, file system semantics, and metadata performance. This creates challenges in unified access, existing data reuse, and cache acceleration.

To address these challenges, the Tencent Cloud (a cloud computing service operated by Tencent, a multinational technology company headquartered in China) team built an enterprise-grade unified storage solution based on JuiceFS+FoundationDB. The solution provides full POSIX semantics for newly created file systems while also integrating existing object storage and HDFS data into a unified namespace, enabling centralized access and cache acceleration.

In this architecture, FoundationDB serves as the metadata engine, supporting metadata management at the tens-of-billions scale with strong transactional consistency. This is also the first time the JuiceFS community has shared a production practice using FoundationDB for metadata management. Local caching and distributed caching are introduced to reduce remote access latency. Going forward, our team plans to further evolve the platform in areas including lakehouse integration, AI training acceleration, intelligent data tiering, and cross-cluster federation.

Challenges: storage gets more complex after storage-compute separation

During early customer discussions and proof-of-concept (POC) projects, we found that when enterprises migrate from storage-compute integrated architectures to storage-compute separation architectures, their biggest concerns are not the architecture model itself, but two questions:

• Can existing data and applications be migrated smoothly?
• Can performance remain close to the original integrated architecture after migration?

Enterprises often operate multiple storage systems simultaneously, including COS, S3, and HDFS. These backends differ in access protocols, SDKs, authentication methods, and file system semantics. If Spark, Hive, Flink, or AI training frameworks need to integrate with each storage system separately, development and maintenance costs can quickly increase.

Performance is another major challenge in adopting storage-compute separation. When data access paths become longer, remote storage and network communication introduce additional overhead. Meanwhile, running multiple storage systems independently makes it difficult to share and reuse caching capabilities.

Therefore, a unified storage solution must not only solve multi-backend integration challenges, but also provide reusable cache acceleration capabilities.

In addition, although object storage is well suited for massive-scale data storage, it’s generally less efficient than traditional file systems for metadata-intensive operations such as:

• List
• Stat
• Directory traversal
• Small-file access

Different storage backends also provide inconsistent support for file system semantics such as directories, permissions, quotas, and snapshots. These differences further complicate unified access for big data and AI workloads.

Based on these requirements, we chose JuiceFS as the foundation for building an enterprise-grade unified storage platform.

Architecture: unified access with two operating modes

JuiceFS adopts a metadata-data separation architecture. The metadata engine can be selected based on workload scale and consistency requirements, including Redis, TiKV, and other options.

For this solution, we selected FoundationDB as the metadata engine, mainly because of its:

• Strong transaction model
• Ordered key-value data model
• Automatic data distribution
• Horizontal scalability

These capabilities enable metadata management at the tens-of-billions scale.

Based on JuiceFS' core architecture, we built a unified storage platform consisting of:

• A unified client layer
• A service-oriented metadata layer
• Multi-level caching
• Multi-backend storage adapters

JuiceFS provides fundamental capabilities including:

• File system semantics
• Multi-protocol access
• Object storage-based data paths
• Local caching

On top of these capabilities, we further extended the system with the following capabilities to meet the multi-tenant and multi-storage requirements of cloud service scenarios:

• Unified authentication
• Multi-volume management
• Unified namespace
• Existing data integration
• Distributed caching

On the client side, big data and AI applications can access the system through multiple interfaces:

• HDFS interface
• POSIX mount
• S3 interface
• Python SDK

The client supports:

• Hadoop SDK
• Python SDK
• FUSE mount
• Local cache
• Service discovery

This allows frameworks such as Spark, Hive, Flink, Presto, and various AI training frameworks to access storage without adapting individually to different backend systems.

For metadata access, we converted JuiceFS' metadata access path into a service-oriented architecture.

Clients no longer connect directly to FoundationDB. Instead, they communicate with stateless metadata services through RPC. The metadata service centrally provides:

• Metadata APIs
• Authentication and authorization
• Multi-volume management
• Quota management
• Directory protection
• File protection
• Task management

ZooKeeper is used for:

• Service registration
• Service discovery
• Cross-node state synchronization

The metadata service can scale horizontally as needed, while FoundationDB acts as the backend metadata engine responsible for metadata persistence and transaction processing.

To cover both new file system creation and existing data ingestion scenarios, the underlying storage layer provides two modes:

• Managed mode is designed for newly created file systems. File metadata is written to FoundationDB through the metadata service, and file data is split according to JuiceFS' chunk/slice/block model and written to object storage. This provides full POSIX file system semantics.
• External mode is designed for existing data reuse. Its core is the Unified File System (UFS) abstraction layer, which abstracts backends such as COS, S3, and HDFS into a unified file system interface, without migrating or taking over the existing metadata and data of those backends. Metadata requests are passed through the metadata service via UFS to the underlying storage, while data reads and writes are performed using the native protocols of the respective backends.

The unified namespace organizes both modes within a single directory structure. To upper‑layer applications, it presents a unified path and access entry. As for the underlying implementation, different directories can correspond either to managed file systems or to mounted existing HDFS or object storage data. This allows new and existing data to coexist under the same access system without requiring large‑scale data migration beforehand.

On the data access path, the system provides both local caching and distributed caching. Read requests first check the local cache; if missed, they access the distributed cache; if still missed, they fall back to the underlying storage. This reduces remote access latency and backend storage pressure.

FoundationDB: design trade‑offs for 10‑billion‑level metadata

In the unified storage access layer, the metadata engine determines the system's scalability limit and consistency capabilities. When file counts reach the billion or even 10‑billion level, the metadata system itself becomes the critical bottleneck.

Why we chose FoundationDB

During the metadata engine selection phase, we focused on several key questions:

• Whether a single cluster can support 10‑billion‑level metadata
• Whether the transaction model is strong enough
• Whether automatic sharding and horizontal scaling are supported
• Whether operational complexity is manageable

Traditional relational databases tend to hit bottlenecks in single‑volume scale and horizontal scalability, so we focused our evaluation on distributed key-value solutions, including TiKV and FoundationDB:

• TiKV has been widely adopted in large‑scale metadata scenarios and offers strong horizontal scalability.

• FoundationDB better meets our requirements in terms of strict transaction consistency, ordered key‑value model, automatic data distribution, multi‑replica strong consistency, and operational complexity.

Ultimately, we chose FoundationDB as the metadata engine because it’s better suited for the strongly consistent, high‑concurrency, range‑scannable access patterns typical of file system metadata. This practice also allowed our team to accumulate experience in modeling, tuning, and operating FoundationDB in large‑scale metadata scenarios, providing a reference for future use cases such as Hive table metadata management.

Key design: multi‑volume isolation and range scans

File system metadata naturally has two types of access patterns:

• A single cluster must host multiple file systems, requiring clear boundaries between volumes.
• Operations such as directory traversal, attribute queries, and chunk index queries often rely on prefix scans.

Therefore, key design must both satisfy multi‑volume isolation and closely align with file system access paths, avoiding hot spots or large‑transaction issues at scale.

Leveraging FoundationDB's ordered key-values, we used JuiceFS metadata design method:

• Using fsname as the key prefix to distinguish different file systems
• Organizing attributes, directory entries, chunk indexes, and other metadata related to the same file within a close key space to improve access locality
• Encoding numeric fields in big‑endian order so that lexicographic order aligns with numeric order for sequential scans.

On this basis, multiple file systems can share the same FoundationDB cluster while maintaining independent key spaces. For operations such as list that may scan large amounts of data, pagination is used to control single‑transaction size. In transaction conflict scenarios, optimistic locking and automatic retries are used to reduce lock contention.

How to avoid FoundationDB's hard limits

FoundationDB has explicit hard limits on key size, value size, and transaction size:

• A single key is limited to 10 KB.
• A single value to 100 KB.
• A single transaction to 10 MB total size.

These limits cannot be adjusted via configuration parameters. When using FoundationDB as a file system metadata engine, we need to pay special attention to operations that tend to enlarge metadata size, such as frequent random writes, repeated truncates, fallocate punch hole, and copyFileRange, which may cause individual values to grow continuously. Large‑scale file copies, large file truncation, batch metadata import, or deleting an entire file system may trigger oversized transactions.

The highest risk comes from chunk slice accumulation. In JuiceFS' data model, the slice list corresponding to a chunk is stored in a single value, and each slice occupies about 24 bytes. When a chunk accumulates about 4,266 slices, it may hit FoundationDB's 100 KB value limit. Operations such as frequent random writes to the same chunk, repeated truncates, fallocate punch hole, and copyFileRange will continuously increase the slice count. Without a governance mechanism, a single value can keep growing, eventually causing transaction commit failures.

This issue also has a subtle implementation aspect: some paths use atomic append operations like AppendIfFits. This is a blind‑write operation; the transaction cannot know the total value size after the append in advance. If the append exceeds 100 KB, failure may not occur until the transaction commit phase. This means the problem is not exposed early during writes but instead accumulates silently until a commit fails. Therefore, relying solely on retries after failure is insufficient; safety boundaries must be set in the data model and write path.

To address chunk slice accumulation, we implemented and strengthened the compaction mechanism. The core idea is to detect chunks with too many slices during read/write operations and merge multiple small slices into fewer large slices to control individual value growth. For lightly fragmented chunks, asynchronous compaction can be triggered to avoid blocking writes. When the slice count reaches a safety threshold, synchronous compaction is triggered to prevent it from growing to the 100 KB limit. In practice, maxSlices is set to 2,500, corresponding to about 60 KB, leaving about 40 KB of headroom below the 100 KB limit.

During compaction implementation, concurrency and reclamation issues also need to be handled. The merge process uses CAS semantics to ensure the chunk has not been modified concurrently, avoiding data loss. If compaction fails, normal writes are not affected, and it can be retried later. Obsolete slices generated after compaction are passed to the Garbage Collection (GC) process, where reference counting determines whether they are still in use. A delayed‑deletion mechanism supports accidental deletion recovery and background reclamation.

In addition to chunk slice accumulation, we also examined other risk items such as file system configurations, Kerberos tokens, POSIX locks, extended attributes, and UFS path‑type keys. Overall, these risks are manageable: most values are small, and key design is constrained by file name or path length. The real area requiring governance is still the single‑value expansion caused by chunk slice accumulation.

Cache acceleration system: two‑level caching to reduce remote access overhead

After storage‑compute separation, data access paths shift from local disks to remote object storage or HDFS. For read‑intensive workloads such as big data analytics and AI training, if every read goes directly back to the underlying storage, network latency and backend access pressure are amplified. Therefore, the unified storage layer must not only address "how to access multiple storage types" but also "how to make remote data reads faster."

Our approach is to introduce two‑level caching: client local cache as L1, and distributed cache as L2. Read requests first check the local cache; if hit, return directly. If missed, they access the distributed cache. If still missed, they fall back to the underlying storage and fill the data into the cache system. This provides both the lowest latency from local SSD or memory and the ability to reuse hot data across multiple clients through an independent SSD cache cluster.

Local caching mainly uses JuiceFS Community Edition's existing capabilities, extended to support external mode. It runs inside the client process, caches data at block granularity by default, and supports least recently used (LRU) eviction, sequential read prefetching, and OS cache control. The distributed cache is our self‑developed independent cache cluster. Clients use consistent hashing to route the same data block to the same cache node, making it suitable for multi‑node shared reads, scenarios with limited client local storage, or workloads requiring cross‑task hot data reuse.

The most critical area in the cache system is consistency. In managed mode, every write generates a new slice ID. This causes the cache key to change, so old caches are naturally not hit. Meanwhile, objects in object storage are never modified in place after being written, so no additional invalidation mechanism is needed.

External mode is more complex, because files in the underlying HDFS or object storage may be directly overwritten by external systems. If the cache key contains only the path, stale data may be read. To address this, we introduced a fingerprint mechanism in external mode, incorporating file path, block offset, and file fingerprint into the cache key. The fingerprint is composed of file modification time and size, and is frozen at open time. When the file is modified externally, the fingerprint changes, so old caches are naturally not hit, satisfying close‑to‑open consistency.

This brings a trade‑off: managed mode provides finer‑grained cache isolation based on slice IDs, while external mode relies on file fingerprints, which are more coarse‑grained and may invalidate larger cache ranges. However, for scenarios where existing data may be modified by external systems, this is a necessary trade‑off for consistency, and it’s also an area for future optimization.

In addition to basic read caching, the system supports cache warm-up, cache‑stampede protection, TTL‑based eviction, and transparent degradation. For example:

• The warmup command can pre‑load hot data.
• Distributed caching supports batch-warming by directory prefix.
• When the cache layer becomes unavailable, the system can automatically degrade to direct underlying storage access to prevent cache failures from affecting application reads.

Multi‑tenancy and elastic scaling

In cloud provider scenarios, the unified storage layer must serve multiple business units, tenants, and file systems simultaneously. Therefore, we designed the metadata service as a stateless architecture. Clients perform service discovery and load balancing through ZooKeeper, and metadata service nodes can scale horizontally on demand to increase overall access capacity and availability.

Multi‑volume isolation relies on FoundationDB's key prefix design. Each file system uses an independent fsname prefix, and different volumes have independent key spaces, allowing a single FoundationDB cluster to host multiple file systems. New file systems can be added online without restarting the metadata service.

In multi‑node deployments, runtime information such as quotas, permissions, directory protection, and task states must also be synchronized across nodes. We use ZooKeeper for cross‑node change notifications, combined with debounce merging, precise key‑level refresh, and periodic polling for fallback, ensuring configuration changes are promptly synchronized to all metadata service nodes.

Through stateless metadata services, fsname‑based isolation, and ZooKeeper state synchronization, the system can achieve both multi‑tenant isolation and horizontal scalability at both the service and metadata layers.

10‑billion‑level metadata validation

We conducted 10‑billion‑level metadata tests based on JuiceFS 1.3.0 and FoundationDB 7.3. The test environment comprised 6 x86 servers running Kylin V10 SP3, with FoundationDB deployed using triple replication and the SSD storage engine.

The tests were performed after 10.8 billion metadata entries had already been written. Results showed that at the 10‑billion scale, FoundationDB single‑operation latency remained largely consistent with baseline data. Some operations, such as readdir_1k and lookup, even performed better. Overall, after reaching 10.8 billion inodes in a single volume, metadata operation latency remained in the sub‑millisecond to 2 ms range, indicating no significant performance degradation at scale.

The table below shows benchmark single‑operation latency (μs/op, lower is better):

In extreme write tests, we used 15 concurrent processes to perform juicefs clone for metadata cloning, validating FoundationDB's write ceiling. Results showed that under the 6‑node + SSD configuration, stable‑phase write throughput was approximately 11,000–15,000 inodes/s. Bottleneck analysis indicated that the main pressure was on storage write queues and disk I/O, rather than CPU or network. Future improvements can be made by adding storage processes, upgrading disk performance, or scaling Commit Proxy and TLog components.

Inode creation:

For high availability, we validated scenarios with 1‑node and 2‑node failures. With 1‑node failure, the service remained available, and FoundationDB automatically triggered replica repair and data migration. With 2‑node failure, the service remained available, but the cluster's fault tolerance dropped to zero—it could no longer tolerate additional failures. After recovery, the system triggered data rebalancing, with a complete recovery time of approximately 4 hours in testing. This process introduces additional I/O pressure, so production environments must pay close attention to disk watermarks, single‑disk failures, and resource consumption during recovery.

For capacity planning, test data showed that at 5 billion inodes, FoundationDB disk usage was approximately 6.1 TB; at 10.8 billion inodes, it was approximately 10.1 TB. After accounting for triple replication, compression, and reserved space, approximately 120 GB per 1 billion inodes can be used as a baseline capacity estimate.

Through this validation, we’ve essentially confirmed that FoundationDB can support 10‑billion‑level metadata per volume while maintaining stable metadata access performance at that scale. At the same time, the triple‑replica architecture provides good failure recovery capabilities, though production environments still require careful capacity planning, disk monitoring, and I/O management during recovery.

Summary and future plans

At present, this unified storage access layer built on JuiceFS+FoundationDB has completed core capability construction in big data storage‑compute separation scenarios, including unified client access, leveraging of external data, multi‑level cache acceleration, 10‑billion‑level metadata management, and multi‑tenant scalability. Moving forward, we’ll continue to evolve around lakehouse integration, AI training, and storage governance.

In the lakehouse direction, we plan to strengthen integration with table formats such as Iceberg and Hudi through the unified storage layer, enabling upper‑layer data lake and data warehouse scenarios to reuse unified storage access, permission management, and cache acceleration capabilities.

For AI training, we’ll further optimize cache warm-up and near‑local caching to reduce remote read amplification and GPU wait times in training jobs. We’ll also explore model checkpoint management using snapshot capabilities, making intermediate state saving, recovery, and reuse more efficient during training.

In storage governance, we’ll explore intelligent tiering capabilities, automatically migrating data between SSD, HDD, and object storage based on access frequency, achieving a better balance between performance and cost. For larger‑scale cloud deployments, we’ll continue to advance cross‑cluster federation capabilities, enabling metadata synchronization, routing, and unified access across multiple clusters.

Going forward, we’ll continue to iterate around performance, cost, elasticity, and data governance, enabling more workloads to access and manage underlying data in a unified manner.

If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.