In cutting-edge scientific fields like astrophysics and genomics, high-throughput reading of massive datasets and zero-tolerance data integrity are the absolute lifebloods of computational analysis.
Recently, the JuiceFS team sat down for an in-depth conversation with the Scientific Computing team at the American Museum of Natural History (AMNH). They explained how they use JuiceFS Community Edition to handle large amounts of research data within an Open Storage Network (OSN) S3-compatible storage system which was funded by a grant from the National Science Foundation (NSF), creating a highly reliable and consistent storage setup for academic use.
The workload: 95% read-heavy traffic
The scientific computing environment at AMNH supports extraordinarily intense interdisciplinary research, operating at a massive scale. Currently, AMNH's OSN nodes have the capacity to store around 3 PB of research data, with approximately 30% of that total volume actively accessed via the NFS + JuiceFS layer. The overall storage architecture is precisely tailored to handle these specific academic patterns:
-
A 95% Read-Heavy Profile: Workloads include the system's massive reference datasets such as genomic sequences and astronomical survey data in read-only format. The calculations are very throughput- and latency-sensitive. Once a dataset is written, it will be read many times and rarely changed.
-
The OSN (Open Storage Network) Model: AMNH partners with the Open Storage Network. Standard rack servers and an external JBOD are hosted on-site at the museum but managed remotely by OSN administrators. This setup provides AMNH with cost-effective S3-compatible endpoints while allowing them to maintain physical ownership of the hardware and data.
-
Seamless Multi-Protocol Access: AMNH adopted JuiceFS Community Edition as the distributed file system layer, allowing mounting S3 buckets seamlessly and enabling traditional NFS access to this underlying S3-compatible storage for high-performance computing (HPC) clusters.
Data integrity first, without silent corruption
For scientific research, data consistency is crucial. To prove and continuously observe the system's resilience, the AMNH team conducted rounds of tests during the PoC phase and watched JuiceFS logs and metrics closely in production.
"We have recently observed timeout warnings when JuiceFS writes to the OSN object store, but data integrity issues were never reported. We do want to understand why, though."
– Systems and Scientific Computing Team, American Museum of Natural History
That was one of the discussions we had during the meeting. It turns out that JuiceFS configures all object store SDKs with RetryMaxAttempts = 1 and implements custom logic for retries. This way, JuiceFS is able to unify the behavior across all object stores it supports. Thus, in this case, the warning is a precaution, and the data integrity is ensured.
When comparing JuiceFS to other file systems that claim higher raw performance metrics (often achieved by using "unsafe" write-handling or skipping strict metadata commits), AMNH opted for data consistency.
"We use almost all the default configurations for the JuiceFS client with Redis as the metadata engine, and it's been running in production steadily for 4 years now."
– Systems and Scientific Computing Team, American Museum of Natural History
Current & future tuning options
The conversation also continued into how AMNH tunes and considers the JuiceFS client configurations to match their scientific computing patterns needed for the future.
The write-back configuration
Currently, the only non-default configuration AMNH utilizes is --writeback , which smooths out the client-side writing throughput. There are always tradeoffs. Write-back sacrifices a bit of data consistency and requires high disk reliability on the mounting machine. However, given AMNH's current workload pattern (mostly reads), both teams agreed that it is acceptable either way, write-back enabled or not.
High-performance caching
The teams also evaluated the differences between the Linux kernel's native page cache and JuiceFS' cache for client-side metadata. While the kernel page cache is the fastest, the JuiceFS client-side metadata cache offers more flexibility. In a read-heavy setup, this configuration can save many roundtrips to the metadata engine. Currently, the metadata engine is co-located with the JuiceFS client-mounting host, but in the future, this configuration can be a nice add-on for performance.
On the file side, since the hardware uses HDDs managed by the OSN object store, the JuiceFS client's local data read cache significantly lowers latency after the first read. Increasing the --cachesize can further improve performance when the workloads need it.
The potential of using SQLite as the metadata engine
In a highly creative approach, the teams discussed utilizing SQLite on local NVMe storage as their JuiceFS metadata engine. The JuiceFS team highlighted that for standalone sandboxes or read-heavy workloads, SQLite is an exceptional, low-maintenance choice, provided the metadata file is backed up for resilience (which is true for Redis as well). Since for AMNH's current scenario, the primary I/O bottleneck usually lies in the backend object storage rather than the database, SQLite's simplicity makes it incredibly attractive for specific workloads.
Final thoughts
Managing multi-petabyte scale, strong data consistency, and high-throughput research workloads is a difficult balancing act for any infrastructure team. AMNH has built a resilient, low-maintenance storage environment that lets scientists work without creating massive engineering overhead, using an open architecture and JuiceFS Community Edition on top of an OSN backend.
Reflecting on their four-year journey of running this system stably in production, the AMNH team summarized their experience seamlessly:
"The implementation on our side is straightforward, and we are happy about it overall."
– Systems and Scientific Computing Team, American Museum of Natural History
As data demands in scientific computing continue to grow, this blueprint of combining cost-effective object storage with a robust, reliable file system layer provides a highly reproducible success story for academic and research institutions worldwide.
If you have any questions for this article, feel free to join JuiceFS discussions on GitHub and our Discord server.
