Mirror file system

If you wish to use the same JuiceFS file system across two or more regions, while maintaining a similar high-performance experience, you can consider using JuiceFS' "Mirror file system" feature: create one or more complete "read-only mirrors" for a file system with the same content, but with a different volume name and access credentials. Each mirror file system is billed separately.

Starting from 5.0, JuiceFS experimentally supports writing to mirror file systems. Read this section to learn more.

How it works

To build a mirror file system, firstly, the mirror region needs to establish metadata synchronization, a separate metadata cluster will be deployed in the mirror region, and configured to become a mirror of the source region. The basic mirror unit is the whole metadata service, meaning you can't choose to mirror just a single file system. This is usually not a problem since the metadata mirroring overhead is relatively small. Once the mirror metadata region has been configured and deployed, it'll automatically sync the metadata from the source region, under normal cross region network conditions, synchronization latency is usually around a second, there's also a monitoring task running in Console which periodically check data differences between source and mirror regions.

As for object storage, users can choose from different setup depending on their performance requirement and budget:

• Solution #1: Share object storage between regions. To simplify the setup, and lower the object storage costs, consider sharing the object storage service across both regions. To increase mirror region performance, use a distributed cache cluster, as shown below.

  

• Solution #2: Use dedicated object storage bucket in the mirror region. If your mirror region needs all the data from the source file system with the best performance, use a dedicated object storage service in the mirror region, and enable replication to keep them synchronized.

  

With setup 2, a dedicated object storage is used in the mirror region, so credentials for both object storage services should be provided via the --access-key2 and --secret-key2 options. The mirror metadata service will watch for all data modifications in the Raft changelog, and dispatch data synchronization tasks to the clients (in the form of background tasks), client will then pull data from the source object storage, and upload to the target object storage. And if synchronization speed isn't ideal, simply mount more clients to increase concurrency. This mechanism is introduced thoroughly in the replication docs.

Mirror region clients will preferentially read data from the object storage in the current region. If the data synchronization has not yet been completed, it will instead try to read from the object storage in the source region, as shown in the following figure:

One-to-many replication

There's no limit to the number of mirror regions, and every mirror file system can use object storage service hosted in its local region, and enable data replication, and use this method to achieve one-to-many replication.

This topic is also interconnected to data replication. The feature itself only achieves one-to-one replication, so for one-to-many, mirror must be utilized, this is covered in more detail in data replication.

Get started

If you have confirmed your requirements, contact Juicedata engineers to deploy the mirror metadata service in the specified cloud provider (or data center) region. If you use on-premises deployment, you can also obtain the deployment documentation from us and implement it yourself.

Once the mirror region is created and data synchronization is established, go to the file system settings page. Then, you can click Create mirror volume in the Mirror Volumes section.

Note that:

• The mirror file system has its own name and access credentials (token). Be careful not to mix these up when mounting.
• If data replication is enabled, the first time the mirror file system is mounted, it automatically starts synchronizing historical data. However, to make the mirror file system production-ready faster, it is recommended to manually perform a full synchronization in advance using other methods (such as using the juicefs sync command). During the operation of the mirror region client, data from the object storage is also fully synchronized regularly (once a week by default).
• By default, the mirror file system is read-only, and data cannot be written or modified. Any write operations on the host mount point will fail. However, special attention should be paid to CSI Driver. It should not use directories that do not exist in the file system (for example, using dynamic provisioning or attempting to mount non-existing subdirectories), as this will cause CSI Controller to fail to initialize and thus be unable to mount.
• If you need to write via your mirror file system, you can select "Read & Write" permission on the access control tab from the JuiceFS Web Console, note that mirror write support is still a beta feature, read below section to learn more.

Write to mirror file systems (experimental)

Starting from version 5.0, JuiceFS experimentally supports writing to mirror file systems. The principle is not complicated: the client in the mirror region implements read-write separation; metadata read requests are processed through the metadata service in the mirror region, while write requests are written across regions to the source cluster's metadata service and object storage. Therefore, if you need to write files in the mirror region, you no longer need to mount the source file system via public network and arrange all write operations at these specific mount points. Instead, you can directly write through the client of the mirror file system.

warning

The write support for mirror file systems is still in the public testing stage. If you need to use it, test it thoroughly with Juicedata engineers and adopt it cautiously.

Enabling write support for mirror file systems does not require special operations. You only need to create a token with read-write permissions on the Access Token page as usual. Therefore, if you want to manage permissions for the mirror file system and restrict write operations to specific clients, simply manage the read-write permissions of each token.