Skip to main content

Python SDK

In some permission-restricted environments (such as unprivileged containers, and most of the serverless environments), mounting file systems is restricted due to the inability to use the FUSE module. To address these limitations and better support AI scenarios, JuiceFS Enterprise Edition introduced the Python SDK in version 5.1, allowing applications to directly access JuiceFS within the process.

warning

Python SDK is currently in beta phase, use with caution and thoroughly test it before shipping to production. Contact a Juicedata engineer if you encounter any problem.

Installation​

Ensure that Python 3.8 or above is installed, then install the JuiceFS Python SDK via pip:

pip install https://static.juicefs.com/misc/juicefs-5.2.15.20250930-py3-none-any.whl

If you want to install the Ceph version of the JuiceFS Python SDK, use the following command:

pip install https://static.juicefs.com/misc/juicefs-5.2.9.202605220801-py3-none-any.whl

Initialize the JuiceFS client​

In Python SDK, you access the file system via the juicefs.Client object. If you have already mounted the file system in the client environment before, meaning that the *.conf configuration file of the file system is present in ~/.juicefs, you only need to specify the file system name when initializing the Client object:

import juicefs

# Initialize the client object using the file system named myjfs
jfs = juicefs.Client("myjfs")

If not, you need to provide the file system credentials, including the volume name, token, and access keys for object storage:

import os
import juicefs

# Get configuration information from environment variables, or fill them in directly.
volume = os.getenv("VOLUME_NAME")
jfs_token = os.getenv("TOKEN")
ak = os.getenv("ACCESS_KEY")
sk = os.getenv("SECRET_KEY")

# Initialize the JuiceFS client
jfs = juicefs.Client(volume,           # File system name
                     token=jfs_token,  # Token obtained from the JuiceFS console
                     access_key=ak,    # Access Key for object storage
                     secret_key=sk)    # Secret Key for object storage

Basic file operations​

To help users quickly get started, the JuiceFS Python SDK is designed with reference to some functions in Python's built-in functions and the os package. Here are some basic file operation examples.

List files in a directory​

Use the listdir() method to list files in a specified directory:

jfs.listdir('/')

Create a directory​

You can use the makedirs() method to create a directory:

jfs.makedirs("/files")

Check if a file or directory exists​

Use the exists() method to check if a file or directory exists:

if jfs.exists("/files/hello.txt"):
    print("File exists")
else:
    print("File does not exist")

Write to a file​

Use the open() method to open a file and use the write() method to write content:

with jfs.open("/files/hello.txt", "w") as f:
    f.write("hello")

Append content​

Use the open() method to open a file in append mode and write content:

with jfs.open("/files/hello.txt", "a+") as f:
    f.write(" world")

Read a file​

Use the open() method to open a file and use the read() method to read content:

with jfs.open("/files/hello.txt") as f:
    data = f.read()
    print(data)

Delete a file​

Use the remove() method to delete a file:

jfs.remove("/files/hello.txt")

Advanced operations​

Change file permissions​

Use the chmod() method to change file permissions. The permission parameter is an octal number:

jfs.chmod("/files/hello.txt", 0o777)

Use the symlink() method to create a symbolic link:

jfs.symlink("/files/hello.txt", "/files/link")

Use the readlink() method to read the target file of a symbolic link:

link_target = jfs.readlink("/files/link")
print(link_target)

Use the unlink() method to delete a symbolic link:

jfs.unlink("/files/link")

Set and get extended attributes​

Use the setxattr() and getxattr() methods to set and get extended attributes of a file:

jfs.setxattr("/files/hello.txt", "user.key", b"value\0")
xx = jfs.getxattr("/files/hello.txt", "user.key")
print(xx)

Integrate with Ray​

JuiceFS Python SDK can be integrated with Ray through the fsspec interface. The example is as follows:

import fsspec
import ray

# This import statement will automatically call fsspec.register_implementation()
from juicefs.spec import JuiceFS

# Initialize JuiceFS file system client
jfs = fsspec.filesystem("juicefs", name="<volume-name>",
                        conf_dir="/root/.juicefs", cache_group="CACHEGROUP",
                        cache_size="20480", cache_dir="/dev/shm/cache",
                        no_sharing=True)

# Read data
ds = ray.data.read_csv("example.csv", filesystem=jfs)
ds.count()
ds.schema()

API reference​

Client class​

The Client class is the client class for JuiceFS Cloud Service, used to interact with JuiceFS.

Initialization method​

Client initialization accomplishes two things: authentication (fetch client config file) and establish client session with Metadata Service, this is roughly the same as the commandline workflow, where one would first auth and then mount to create a mount point. Thus, the options used during initialization is the same as juicefs auth and juicefs mount, refer to their respective documentations for full description.

Some important caveats:

  • For some options, the default value are different from the FUSE client, to better accommodate SDK scenario:
    • The default value of cache_dir is memory.
    • The default value of cache_size is 100M.
    • The default value of put_timeout is 60s and the default value of get_timeout is 5s.
  • Do not change console_url unless you are using an on-prem setup, point it to the actual Web Console address deployed in your environment.
class Client(name, *, token="", conf_dir="", console_url="https://juicefs.com",
             bucket=None, access_key=None, secret_key=None, session_token=None, shards=0, storage_class=None,
             bucket2=None, access_key2=None, secret_key2=None, session_token2=None, shards2=0, storage_class2=None,
             rsa_key_path="", rsa_passphrase=None, internal=False, external=False,
             max_uploads=20, max_downloads=200, prefetch=1, put_timeout="60s", get_timeout="5s",
             upload_limit='', download_limit='', writeback=False, writeback_threshold_size="0",
             metacache=True, max_cached_inodes=500000, opencache=False,
             attr_cache="1s", entry_cache="0s", dir_entry_cache="1s",
             buffer_size="300M", cache_size="100M", cache_items=0, free_space_ratio=0.1, cache_dir="memory",
             cache_evict="2-random", cache_expire="0s", cache_scan_interval="3600s", verify_cache_checksum="extend",
             cache_group="", group_ip="", group_weight=100, group_weight_unit="0M", group_port=0, no_sharing=False,
             second_group="", cache_partial_only=False, cache_large_write=False, cache_try_dio=True,
             fill_group_cache=False, cache_priority=0,
             mount_point="/jfs", access_log="", debug=False, flip=False, no_bgjob=False, log="", read_only=False)

open()​

Client.open(path, mode='r', buffering=-1, encoding=None, errors=None)

Parameter description:

  • path (str): File path.
  • mode (str): File open mode. Supports combinations of r, w, a, x with b/t and +, and exactly one of r, w, a, and x must be specified.
  • buffering (int): Buffer size. Default is -1, which means using the default buffering.
  • encoding (str): File encoding. Not supported in binary mode.
  • errors (str): Error handling strategy. Not supported in binary mode.

Return value:

A File object

makedirs()​

Client.makedirs(path, mode=0o777, exist_ok=False)

Parameter description:

  • path (str): Directory path
  • mode (int): Directory permissions
  • exist_ok (bool): Ignore the error if the directory exists

Return value:

No return value

exists()​

Client.exists(path)

Parameter description:

path (str): File or directory path

Return value:

A boolean indicating whether the file or directory exists

remove()​

Client.remove(path)

Parameter description:

path (str): File path

Return value:

No return value

chmod()​

Client.chmod(path, mode)

Parameter description:

  • path (str): File path
  • mode (int): File permissions

Return value:

No return value

Client.symlink(src, dst)

Parameter description:

  • src (str): Source file path
  • dst (str): Target symbolic link path

Return value:

No return value

Client.readlink(path)

Parameter description:

path (str): Symbolic link path

Return value:

The target path of the symbolic link

Client.unlink(path)

Parameter description:

path (str): Symbolic link path

Return value:

No return value

setxattr()​

Client.setxattr(path, name, value, flags=0)

Parameter description:

  • path (str): File path
  • name (str): The extended attribute name
  • value (bytes): The extended attribute value
  • flags (int): Extended attribute flags

Return value:

No return value

getxattr()​

Client.getxattr(path, name)

Parameter description:

  • path (str): File path
  • name (str): The extended attribute name

Return value:

The extended attribute value

listdir()​

Client.listdir(path, detail=False)

Parameter description:

  • path (str): Directory path.
  • detail (bool): Whether to return a list with detailed file information. Default is False, which returns only the list of file names.

Return value:

A list of file names in the directory; if detail=True, a list of (file name, stat result) tuples

stat()​

Client.stat(path)

Parameter description:

path (str): File or directory path

Return value:

An os.stat_result object containing the file metadata

lstat()​

Client.lstat(path)

Parameter description:

path (str): File or directory path

Return value:

Like stat(), but does not follow symbolic links

mkdir()​

Client.mkdir(path, mode=0o777)

Parameter description:

  • path (str): Directory path
  • mode (int): Directory permissions

Return value:

No return value

rmdir()​

Client.rmdir(path)

Parameter description:

path (str): Directory path. The directory must be empty.

Return value:

No return value

rename()​

Client.rename(old, new)

Parameter description:

  • old (str): Original file or directory path
  • new (str): New file or directory path

Return value:

No return value

truncate()​

Client.truncate(path, size)

Parameter description:

  • path (str): File path
  • size (int): Truncated file size

Return value:

No return value

chown()​

Client.chown(path, uid, gid)

Parameter description:

  • path (str): File path
  • uid (int): New file owner UID
  • gid (int): New file group GID

Return value:

No return value

Client.link(src, dst)

Parameter description:

  • src (str): Source file path
  • dst (str): Target hard link path

Return value:

No return value

rmr()​

Client.rmr(path)

Parameter description:

path (str): File or directory path to remove recursively

Return value:

No return value

utime()​

Client.utime(path, times=None)

Parameter description:

  • path (str): File path
  • times (tuple): (atime, mtime) timestamp tuple. Default is None, which means the current time is used.

Return value:

No return value

listxattr()​

Client.listxattr(path)

Parameter description:

path (str): File path

Return value:

A list containing all the extended attribute names of the file

removexattr()​

Client.removexattr(path, name)

Parameter description:

  • path (str): File path
  • name (str): The extended attribute name to remove

Return value:

No return value

summary()​

Client.summary(path, depth=0, entries=1)

Get the summary of a directory, including the number of files, directories, and total size within the directory and its subdirectories.

Parameter description:

  • path (str): Directory path.
  • depth (int): Directory depth for the summary. Default is 0, which means summarizing everything.
  • entries (int): Maximum number of subdirectory entries to return. Default is 1.

Return value:

A dictionary with the directory summary, such as Files (number of files), Dirs (number of directories), Size (data size), and Entries (subdirectory details) fields.

clone()​

Client.clone(src, dst, preserve=False, follow_link=True)

Clone a file or directory. Like the juicefs clone command, cloning only copies the metadata without actually copying the object storage data, so it is very fast.

Parameter description:

  • src (str): Source file or directory path.
  • dst (str): Destination path.
  • preserve (bool): Whether to preserve the metadata of the source file (mode, UID, GID, atime, mtime). Default is False.
  • follow_link (bool): When True, symbolic links are cloned as regular files; when False, they are cloned as links. Default is True.

Return value:

No return value

set_quota()​

Client.set_quota(path, capacity=0, inodes=0, create=False, strict=False)

Set a quota for a directory. Refer to Capacity and quota for quota management.

Parameter description:

  • path (str): Directory path.
  • capacity (int): Capacity quota in bytes. Default is 0.
  • inodes (int): File count quota. Default is 0.
  • create (bool): Whether to automatically create the directory if it does not exist. Default is False.
  • strict (bool): Reserved parameter.

Return value:

No return value

get_quota()​

Client.get_quota(path)

Get the quota information of a directory.

Parameter description:

path (str): Directory path

Return value:

A dictionary with the quota information of the directory, such as MaxSpace (capacity quota) and MaxInodes (file count quota) fields

del_quota()​

Client.del_quota(path)

Delete the quota of a directory.

Parameter description:

path (str): Directory path

Return value:

No return value

list_quota()​

Client.list_quota()

List quotas configured for all directories.

Return value:

A dictionary containing all directories with quota set and their quota information

merge()​

Client.merge(dest, sources, overwrite=False, append=False)

Merge files from multiple sources into one destination file.

Parameter description:

  • dest (str): Destination file path.
  • sources (str or list): Source file path or a list of source file paths.
  • overwrite (bool): Whether to overwrite the destination if it already exists. Default is False.
  • append (bool): Whether to append to the destination if it already exists and is not a regular file. Default is False.

Return value:

The operation result

info()​

Client.info(path, inode=0, recursive=False)

Get detailed information of a file or directory.

Parameter description:

  • path (str): File or directory path.
  • inode (int): The inode number of the file. Default is 0, in which case the lookup is performed by path.
  • recursive (bool): Whether to recursively get detailed information of a directory. Default is False.

Return value:

A dictionary with the detailed information of the file or directory

warmup()​

Client.warmup(paths, threads=10, priority=0, retry=3, max_failure=0, evict=False, follow_link=False, check=False, background=False, samples=0, expand=0)

Warm up the cache. Like the juicefs warmup command, it downloads files to the cache in advance to speed up subsequent access.

Parameter description:

  • paths (str): File or directory path to warm up.
  • threads (int): Download concurrency. Default is 10.
  • priority (int): Cache block priority. Valid values are 0, 1, 2, and 3. The higher the number, the higher the priority. Default is 0.
  • retry (int): Maximum retry times for downloading a single data block. Default is 3.
  • max_failure (int): Maximum number of failed data blocks allowed. Default is 0.
  • evict (bool): Actively evict the cache content of the given path. Default is False.
  • follow_link (bool): Whether to follow symbolic links. Default is False.
  • check (bool): Check whether the given path has been cached. Default is False.
  • background (bool): Whether to run in the background. Default is False.
  • samples (int): Number of data blocks to sample for warmup. Default is 0, which means warming up all data.
  • expand (int): Additional range to warm up. Default is 0.

Return value:

A dictionary with the warmup result

trash()​

Client.trash(cursor=0, limit=100, query="", mode=1, startTs=0, endTs=0)

List the files in the trash.

Parameter description:

  • cursor (int): Pagination cursor. Default is 0.
  • limit (int): Maximum number of files to return. Default is 100.
  • query (str): Query keyword. Default is an empty string.
  • mode (int): Query mode. Default is 1.
  • startTs (int): Start timestamp. Default is 0.
  • endTs (int): End timestamp. Default is 0.

Return value:

A tuple of (file list, next cursor), where each item in the file list is a (file name, stat result) tuple

status()​

Client.status()

Get the status of the file system.

Return value:

A dictionary with the status information of the file system

File Class​

The File class is used for file operations in JuiceFS, for reading and writing files.

Initialization method​

Usually the Client.open() method is used to initialize the File object and will not be created directly.

fileno()​

File.fileno()

Return value:

The file descriptor

isatty()​

File.isatty()

Return value:

A boolean indicating whether the file is a TTY

read()​

File.read(size=-1)

Parameter description:

size (int): Number of bytes to read. Default is -1, which means reading the entire file

Return value:

The read byte data

write()​

File.write(data)

Parameter description:

data (bytes): Data to write

Return value:

The number of bytes written

readline()​

File.readline()

Read a line, up to a newline or the end of the file.

Return value:

The read line of data

close()​

File.close()

Return value:

No return value

flush()​

File.flush()

Return value:

No return value

fsync()​

File.fsync()

Force write file data to the backend storage.

Return value:

No return value

readlines()​

File.readlines(hint=-1)

Parameter description:

hint (int): Number of lines to read. Default is -1, which means reading all lines

Return value:

A list containing the lines of the file

writelines()​

File.writelines(lines)

Parameter description:

lines (list): List of lines to write

Return value:

No return value

seek()​

File.seek(offset, whence=0)

Parameter description:

  • offset (int): Offset
  • whence (int): Reference point for the offset. 0 means from the beginning of the file. 1 means from the current position. 2 means from the end of the file

Return value:

The new file pointer position

tell()​

File.tell()

Return value:

The current file pointer position

truncate()​

File.truncate(size=None)

Parameter description:

size (int): Truncated file size. Default is the current file pointer position

Return value:

The truncated file size

readable()​

File.readable()

Return value:

A boolean indicating whether the file is readable

writable()​

File.writable()

Return value:

A boolean indicating whether the file is writable

seekable()​

File.seekable()

Return value:

A boolean indicating whether the file is seekable

Last updated on
Follow us on X
Join us on Discord