Smooth Upgrade: Implementation and Usage

In the past, upgrading JuiceFS versions or modifying mount parameters required users to remount the file system. This causes temporary interruptions in service. Once the underlying file system service is interrupted, it affects the normal operation of other components in the system. To avoid such issues, we’ve introduced the smooth upgrade feature in the latest release of JuiceFS 1.2. This allows users to restart or upgrade without service downtime. In this post, we’ll share how we implemented the smooth upgrade feature and how to use it.

Implementation

FUSE request handling process

To upgrade without interrupting read and write requests, it's essential to understand the entire process of user read/write requests.

As shown above, when a user uses juicefs mount xxxxx /jfs and opens /jfs/a.txt, the main behaviors of JuiceFS and the operating system are as follows:

1. After the user runs juicefs mount, the go-fuse module of JuiceFS opens /dev/fuse to obtain mount fd.
2. The user initiates an open system call, which enters the virtual file system (VFS) layer. The VFS layer then forwards the request to the FUSE layer.
3. The FUSE layer writes the open request into mount fd in the corresponding format according to the protocol.
4. The go-fuse module reads mount fd and calls the corresponding implementation in JuiceFS after parsing the request.
5. go-fuse writes the processing result of this request into mount fd according to the format.
6. The FUSE module in kernel space reads mount fd to obtain the processing result of this request.

Through the example process above, you can see that the /dev/fuse device file serves as the communication bridge between the JuiceFS user-space file system and the kernel. The two communicate via the file handle mount fd obtained when the go-fuse module opens the /dev/fuse device file during executing the juicefs mount command. The kernel writes requests and JuiceFS uses go-fuse to process and respond to requests. Their relationship can be viewed as a client-server protocol, where the kernel acts as the client requesting service, and the user-space JuiceFS plays the role of the server that handles these requests.

How to pass file handles (mount fd) between processes

By analyzing the FUSE request handling process, it's evident that if we can ensure kernel requests don't get lost or encounter errors, we can smoothly complete the upgrade of JuiceFS. Even if requests are delayed in processing, it’s acceptable.

The key here lies in maintaining the consistency of the file handle (mount fd) obtained during the initial mount. The new process must inherit and use the mount fd of the old process, because from the kernel's perspective, nothing has changed. Therefore, the process is transparent to the application.

Note that every time the FUSE device is opened, a new session is allocated. A new session cannot access resources opened in previous sessions. Therefore, to achieve a smooth upgrade, we must maintain the old mount fd.

Before delving into the specifics of how to achieve mount fd consistency, it's essential to understand that starting from version 1.2, JuiceFS has been split into two processes: a daemon process and a service process. When the service process exits abnormally, the daemon restarts the service process. During this process, read-only requests remain unaffected, but write requests fail. During smooth upgrades, between new and old daemon processes, and between the daemon and service processes, we use Unix domain sockets to pass file handles.

The figure above shows how mount fd and status information are transmitted during a smooth upgrade. The specific steps, labeled from 1 to 7, are marked accordingly in the figure.

1. The old service process obtains mount fd by opening the /dev/fuse device during the initial mount.
2. The old service process passes mount fd to the old daemon process.
3. The new daemon process obtains mount fd from the old daemon process.
4. The new daemon process sends a SIGHUP signal to the old service process. At this point, the old service process stops receiving new FUSE requests and flushes the data already written to the object storage. If certain requests are not completed within 3 seconds, they will be interrupted.
5. Upon receiving the SIGHUP signal, the old service process generates an intermediate status file in the /tmp directory.
6. The new service process loads the intermediate status file.
7. The new service process obtains mount fd from the new daemon process.

During the upgrade process, JuiceFS persists all newly written data in memory to the object storage and saves the open file handles to disk. After the new process starts, it restores these open files, ensuring that subsequent FUSE operations proceed without issues. If any FUSE request fails to complete during the upgrade, it will be forcibly interrupted. This may impact the application. It's recommended to perform upgrade operations during periods of low workload. Following the steps above, the new JuiceFS service process obtains the mount fd previously used by the old JuiceFS. The new service process can then handle requests read from the mount fd. The workflow is as follows:

Usage

To use the smooth upgrade feature, you only need to use the new binary or parameters and remount.

Client upgrade

For example, if there is a running mount process juicefs mount redis://127.0.0.1:6379/0 /mnt/jfs -d, and you want to deploy a new JuiceFS client without unmounting the mount point, you can follow these steps:

1.Back up the current binary:

cp juicefs juicefs.bak

2.Download the new binary to replace the current juicefs binary.

3.Use the juicefs mount command to complete the smooth upgrade:

juicefs mount redis://127.0.0.1:6379/0 /mnt/jfs -d

Adjust mount parameters dynamically

In the process above, if you want to adjust the log level to debug without unmounting the mount point, you can execute the following command:

# Adjust the log level.
juicefs mount redis://127.0.0.1:6379/0 /mnt/jfs --debug -d

Note

1. Currently, smooth upgrade in v1.2-beta1 is only supported on Linux platforms. When v1.2 is officially released, macOS platforms will be supported simultaneously.
2. Smooth upgrade requires the JuiceFS client versions of both the new and old processes to be at least v1.2.
3. The FUSE parameters in the new mount parameters should remain consistent with the old mount parameters. Otherwise, the smooth upgrade will continue to overwrite the mount on the current mount point.
4. During the upgrade process, FUSE requests that are not completed within 3 seconds may be interrupted. Some applications may be affected if they cannot automatically retry interrupted file system operations.

Feel free to download and try out JuiceFS v1.2.0-beta1. If you have any feedback or suggestions, you can join JuiceFS discussions on GitHub and our community on Slack.