JuiceFS Cloud Service Quick Start Guide

tip

This is a quick start guide for JuiceFS Cloud Service. If you are using the JuiceFS Community Edition, please visit Community Edition Documentation.

Concept at a glance

JuiceFS is a distributed file system driven by a combination of object storage and database. "Object storage" is used for data storage, and "database" is used for metadata storage. The resulting file system supports shared mount use on any networked server.

JuiceFS Cloud Service provides out-of-the-box self-developed database, so you just need to prepare object storage.

Working environment

• Storage: With JuiceFS, all data will be stored in your own object storage, you can create the object storage Bucket manually in the cloud platform in advance, or you can let JuiceFS client automatically create the Bucket for you. either way, you need to prepare the secret key to access the API of object storage.
• Python: Make sure you have Python installed, Python 3.5+ is recommended, Python 2.6+ is supported as a minimum.
• FUSE: JuiceFS achieves POSIX compatibility via FUSE, make sure you have FUSE installed:
  • Most Linux distributions already have FUSE built-in as a kernel module, use fusermount -V to verify.
  • Install macFUSE if you are using macOS. With Apple silicon you might also need to refer to Startup Disk security policy control for a Mac with Apple silicon.

Windows caveats

JuiceFS client for Windows is not yet available for open use, contact us if you are in need.

On the other hand, you may choose to use JuiceFS client in Windows Subsystem for Linux (WSL), installation steps are just the same as Linux.

Creating a file system

To use JuiceFS cloud service, you need to create a file system in the official website console, new users should visit JuiceFS official website registration first. Then login to your console and click the "Create File System" button.

In the Region field, select your service provider, and the actual object storage region. JuiceFS client will automatically match you with the fastest metadata storage cluster.

note

If the desired service provider / region is not in the list, send a request as shown in the console dialogue, our engineers will deliver support promptly.

You can create any number of file systems, each associated with an object storage bucket.

By default, the JuiceFS Cloud Service client automatically creates and binds a bucket. You can also click "Advanced Options" in "Create File System" to bind a created bucket. In addition, you can also adjust the compression algorithm of the file system in "Advanced Options". The compression feature is enabled by default. If the data format you store (such as Parquet, ORC) has been compressed, please turn off the compression feature of JuiceFS.

note

If you are using Tencent Cloud's COS object storage, you must click "Advanced Options" to modify the automatically generated Bucket Name and add the APPID suffix to it. The complete format is <bucket>-<APPID>. Click here to see how to get the APPID of Tencent Cloud.

Upon creation, you will be redirected to the settings page, follow the instructions to mount the newly created file system. Note the Token in the settings page, this is unique for each file system, and is used to authenticate when you mount this file system.

Mounting the file system

JuiceFS can be mounted on any computers connected to the Internet, and then be used like a local disk. But with production environments, machines should be in the same region as the file system's object storage, in order to achieve the best cost and performance.

Client installation

note

JuiceFS Cloud Service Client is different from the JuiceFS Community Edition Client. If you need to use both, change the name of the program to avoid name conflict. For example, rename the cloud service client to juicefs-cloud.

For Linux and macOS, use the command below to install JuiceFS client (might need root permission). The installation path in the example command is set to /usr/local/bin by convention, you can also modify it according to the actual situation.

curl

curl -L https://juicefs.com/static/juicefs -o /usr/local/bin/juicefs && chmod +x /usr/local/bin/juicefs

wget

wget https://juicefs.com/static/juicefs -o /usr/local/bin/juicefs && chmod +x /usr/local/bin/juicefs

If you encounter a Permission denied error during the execution of the above command, please switch to the root user to execute.

In the terminal, run juicefs --help and expect a help message printed, indicating that the client was installed successfully. If you see command not found, check your environment variables and ensure /usr/local/bin is in PATH.

Mounting

To mount the file system that you just created, run:

sudo juicefs mount $VOL_NAME $MOUNTPOINT

Please replace $VOL_NAME with the file system name set when "Creating a file system", and replace $MOUNTPOINT with the path of the JuiceFS mount point.

The client will interactively ask for credentials. You'll be needing the token of the file system, and the Access Key ID / Access Key Secret required to access the object storage (must have full access to the underlying bucket).

$ sudo juicefs mount myjfs /jfs
Token for myjfs: xxxxx
Access key ID for oss://juicefs-myjfs: xxxxx
Access key secret for oss://juicefs-myjfs: xxxxx
OK, myjfs is ready at /jfs.

info

Credentials are required when mounting for the first time, upon a successful mount, relevant information will be stored in the $HOME/.juicefs/$VOL_NAME, so you won't need to provide them again on the next mount.

Now that JuiceFS has been mounted on the system, the client will run as a daemon in the background, logs will be written to syslog (Facility: LOG_USER) or /var/log/juicefs.log, refer to "Fault diagnosis and analysis" for specifics.

Auto-mount on boot

After confirming that the mount is successful and can be used normally, you can refer to this section to set the automatic mount on boot.

Linux

Add an entry in file system table (/etc/fstab) so that JuiceFS is automatically mounted on boot. JuiceFS 4.8+ supports --update-fstab to make this easier:

# Needs root permission to modify /etc/fstab
$ sudo juicefs mount --update-fstab $VOL_NAME $MOUNTPONT
$ grep $VOL_NAME /etc/fstab
<VOL_NAME> <MOUNTPONT> juicefs _netdev 0 0

If you'd like to control this process by hand, note that:

• A soft link needs to be created from /sbin/mount.juicefs to the juicefs executable, the /sbin/mount.juicefs command is called when the operating system parses the fstab.

• All parameters provided in your juicefs mount command must also be included in the fstab options to take effect. Remember to remove the prefixing hyphen(s), and add their values with =, for example:

  $ sudo juicefs mount --update-fstab $VOL_NAME $MOUNTPONT -b --max-uploads=1 --prefetch 2 --writeback miniovol /jfs -o max_read=3
  # -o stands for FUSE options, and is handled differently
  $ grep $VOL_NAME /etc/fstab
  <VOL_NAME>  <MOUNTPONT> juicefs _netdev,background,max-uploads=1,max_read=3,prefetch=2,writeback 0 0

note

CentOS 6 does not automatically mount the network file system at boot time by default, you need to run the following command to enable automatic mounting.

sudo chkconfig --add netfs

macOS

1. Create ~/Library/LaunchAgents/com.juicefs.VOL_NAME.plist as follows, replace capitalized words to their true values.

   <?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
   <plist version="1.0">
   <dict>
           <key>Label</key>
           <string>com.juicefs.VOL_NAME</string>
           <key>ProgramArguments</key>
           <array>
                   <string>/usr/local/bin/juicefs</string>
                   <string>mount</string>
                   <string>VOL_NAME</string>
                   <string>MOUNTPOINT</string>
           </array>
           <key>RunAtLoad</key>
           <true/>
   </dict>
   </plist>

2. Run the following command and see if mount succeeds.

   launchctl load ~/Library/LaunchAgents/com.juicefs.VOL_NAME.plist
   launchctl start ~/Library/LaunchAgents/com.juicefs.VOL_NAME
   ls $MOUNTPOINT

3. If the mount is not successful, add this piece to the above plist file:

   <key>StandardErrorPath</key>
   <string>/tmp/mycommand.err</string>
   <key>StandardOutPath</key>
   <string>/tmp/mycommand.out</string>

   Reload to check juicefs logs:

   launchctl unload ~/Library/LaunchAgents/com.juicefs.VOL_NAME.plist
   launchctl load ~/Library/LaunchAgents/com.juicefs.VOL_NAME.plist
   cat /tmp/mycommand.out
   cat /tmp/mycommand.err

Import existing files

Juicesync is the recommended tool if you need to move data in and out of JuiceFS, but thanks to POSIX compatibility, you can use any other tool you like.

But if the associating object storage already have files uploaded before you use JuiceFS. Use juicefs import to import them into JuiceFS.

juicefs import BUCKET-NAME/PREFIX TARGET-DIR-IN-JUICEFS

For example, the following command imports all files in my-bucket starting with my-files into the /jfs/my-files folder.

juicefs import my-bucket/my-files /jfs/my-files

info

juicefs import only writes the metadata of the files to the metadata engine of the JuiceFS cloud service, the file remains in the object storage as is.

After the import is complete, you can change the properties of these files in JuiceFS as you like, such as changing the file name or permissions, etc. These changes will not be synchronized to the object storage. Similarly, deleting imported files in the file system will only delete the metadata, not the actual files in the object storage.

It is recommended to fully import data into JuiceFS for best performance and full product functionality experience.

Unmounting the file system

caution

Force unmounting a file system in use may result in data loss. Make sure you have backed up all your important data.

Use the umount command to unmount the file system.

umount /jfs

If the command returns the target is busy error, it means that the file system is in use, use lsof MOUNTPOINT to find and kill the process of the relevant application, and try again.

For Linux, you can use the -l option to perform a delayed unmount (unmounting the file system immediately, but waiting until the device is no longer busy to clean up the resources).

umount -l /jfs

For macOS, use diskutil to force an unmount:

diskutil unmount force /jfs

Also with macOS, you can find JuiceFS in Finder - Locations - (Your computer), and then press the eject key to umount.

You can kill the juicefs client process manually, if you don't, it'll exit on its own.

Delete file system

If you need to delete a file system created in the cloud service, the following steps should be strictly followed for data security:

1. Backup all data in the file system
2. Unmount the file system on all host
3. Delete the file system in the console
4. Delete file system related configuration file: rm $HOME/.juicefs/$VOL_NAME.conf