Skip to main content

Python SDK

In some permission-restricted environments (such as unprivileged containers and most serverless environments), mounting file systems is restricted due to the inability to use the FUSE module. To address these limitations and better support AI scenarios, JuiceFS Cloud Service and Enterprise Edition introduced the Python SDK in version 5.1, allowing applications to directly access JuiceFS within the process.

Note

The JuiceFS Python SDK is currently in its beta phase. If you experience any issues while using it, we'd greatly appreciate your feedback.

Installation​

Ensure that Python 3.8 or above is installed, then install the JuiceFS Python SDK via pip:

pip install https://static.juicefs.com/misc/juicefs-5.1.5.202411290726-py3-none-any.whl

Initialize the JuiceFS client​

Before using it, you need to initialize a Client object.

If you have previously mounted the file system using the cloud service client, meaning that the *.conf configuration file of the file system already exists in the current user's ~/.juicefs directory, you only need to specify the file system name when initializing the Client object:

import juicefs

# Initialize the client object using the file system named myjfs
jfs = juicefs.Client("myjfs")

If not, you need to provide JuiceFS configuration information, including the file system name, token, and access keys for object storage:

import os
import juicefs

# Get configuration information from environment variables, or fill them in directly.
volume = os.getenv("VOLUME_NAME")
jfs_token = os.getenv("TOKEN")
ak = os.getenv("ACCESS_KEY")
sk = os.getenv("SECRET_KEY")

# Initialize the JuiceFS client
jfs = juicefs.Client(volume,           # File system name
                     token=jfs_token,  # Token obtained from the JuiceFS console
                     access_key=ak,    # Access Key for object storage
                     secret_key=sk)    # Secret Key for object storage

Basic file operations​

To help users quickly get started, the JuiceFS Python SDK is designed with reference to some functions in Python's built-in functions and the os package. Here are some basic file operation examples.

List files in a directory​

Use the listdir() method to list files in a specified directory:

jfs.listdir('/')

Create a directory​

You can use the makedirs() method to create a directory:

jfs.makedirs("/files")

Check if a file or directory exists​

Use the exists() method to check if a file or directory exists:

if jfs.exists("/files/hello.txt"):
    print("File exists")
else:
    print("File does not exist")

Write to a file​

Use the open() method to open a file and use the write() method to write content:

with jfs.open("/files/hello.txt", "w") as f:
    f.write("hello")

Append content​

Use the open() method to open a file in append mode and write content:

with jfs.open("/files/hello.txt", "a+") as f:
    f.write(" world")

Read a file​

Use the open() method to open a file and use the read() method to read content:

with jfs.open("/files/hello.txt") as f:
    data = f.read()
    print(data)

Delete a file​

Use the remove() method to delete a file:

jfs.remove("/files/hello.txt")

Advanced operations​

Change file permissions​

Use the chmod() method to change file permissions. The permission parameter is an octal number:

jfs.chmod("/files/hello.txt", 0o777)

Use the symlink() method to create a symbolic link:

jfs.symlink("/files/hello.txt", "/files/link")

Use the readlink() method to read the target file of a symbolic link:

link_target = jfs.readlink("/files/link")
print(link_target)

Use the unlink() method to delete a symbolic link:

jfs.unlink("/files/link")

Set and get extended attributes​

Use the setxattr() and getxattr() methods to set and get extended attributes of a file:

jfs.setxattr("/files/hello.txt", "user.key", b"value\0")
xx = jfs.getxattr("/files/hello.txt", "user.key")
print(xx)

API reference​

Client class​

The Client class is the client class for JuiceFS Cloud Service, used to interact with JuiceFS.

Initialization method​

tip

For parameter descriptions of the initialization method, please refer to the option descriptions of the juicefs auth and juicefs mount commands. .

It is important to note that the default values ​​of some options are different from the FUSE client. For example, the default value of cache_dir is memory and the default value of cache_size is 100M.

class Client(name, *, token="", conf_dir="", console_url="https://juicefs.com",
             bucket=None, access_key=None, secret_key=None, session_token=None, shards=0, storage_class=None,
             bucket2=None, access_key2=None, secret_key2=None, session_token2=None, shards2=0, storage_class2=None,
             rsa_key_path="", rsa_passphrase=None, internal=False, external=False,
             max_uploads=20, max_downloads=200, prefetch=1, put_timeout="60s", get_timeout="5s",
             upload_limit='', download_limit='', writeback=False,
             metacache=True, max_cached_inodes=500000, opencache=False,
             attr_cache="1s", entry_cache="0s", dir_entry_cache="1s",
             buffer_size="300M", cache_size="100M", free_space_ratio=0.1, cache_dir="memory",
             cache_evict="2-random", cache_expire="0s", verify_cache_checksum="full",
             cache_group="", group_ip="", group_weight=100, group_weight_unit="0M", group_port=0, no_sharing=False,
             cache_partial_only=False, cache_large_write=False, fill_group_cache=False, cache_priority=0,
             mount_point="/jfs", access_log="", debug=False, flip=False,
             **kwargs)

open()​

Client.open(path, mode='r', buffering=-1, encoding=None, errors=None, newline=None)

Parameter description:

  • path (str): File path
  • mode (str): File open mode
  • buffering (int): Buffer size
  • encoding (str): File encoding
  • errors (str): Error handling strategy
  • newline (str): Newline handling strategy

Return value:

A File object

makedirs()​

Client.makedirs(path, mode=0o777, exist_ok=False)

Parameter description:

  • path (str): Directory path
  • mode (int): Directory permissions
  • exist_ok (bool): Ignore the error if the directory exists

Return value:

No return value

exists()​

Client.exists(path)

Parameter description:

path (str): File or directory path

Return value:

A boolean indicating whether the file or directory exists

remove()​

Client.remove(path)

Parameter description:

path (str): File path

Return value:

No return value

chmod()​

Client.chmod(path, mode)

Parameter description:

  • path (str): File path
  • mode (int): File permissions

Return value:

No return value

Client.symlink(src, dst)

Parameter description:

  • src (str): Source file path
  • dst (str): Target symbolic link path

Return value:

No return value

Client.readlink(path)

Parameter description:

path (str): Symbolic link path

Return value:

The target path of the symbolic link

Client.unlink(path)

Parameter description:

path (str): Symbolic link path

Return value:

No return value

setxattr()​

Client.setxattr(path, name, value, flags=0)

Parameter description:

  • path (str): File path
  • name (str): The extended attribute name
  • value (bytes): The extended attribute value
  • flags (int): Extended attribute flags

Return value:

No return value

getxattr()​

Client.getxattr(path, name)

Parameter description:

  • path (str): File path
  • name (str): The extended attribute name

Return value:

The extended attribute value

File Class​

The File class is used for file operations in JuiceFS, for reading and writing files.

Initialization method​

Usually the Client.open() method is used to initialize the File object and will not be created directly.

fileno()​

File.fileno()

Return value:

The file descriptor

isatty()​

File.isatty()

Return value:

A boolean indicating whether the file is a TTY

read()​

File.read(size=-1)

Parameter description:

size (int): Number of bytes to read. Default is -1, which means reading the entire file

Return value:

The read byte data

write()​

File.write(data)

Parameter description:

data (bytes): Data to write

Return value:

The number of bytes written

close()​

File.close()

Return value:

No return value

flush()​

File.flush()

Return value:

No return value

readlines()​

File.readlines(hint=-1)

Parameter description:

hint (int): Number of lines to read. Default is -1, which means reading all lines

Return value:

A list containing the lines of the file

writelines()​

File.writelines(lines)

Parameter description:

lines (list): List of lines to write

Return value:

No return value

seek()​

File.seek(offset, whence=0)

Parameter description:

  • offset (int): Offset
  • whence (int): Reference point for the offset. 0 means from the beginning of the file. 1 means from the current position. 2 means from the end of the file

Return value:

The new file pointer position

tell()​

File.tell()

Return value:

The current file pointer position

truncate()​

File.truncate(size=None)

Parameter description:

size (int): Truncated file size. Default is the current file pointer position

Return value:

No return value

readable()​

File.readable()

Return value:

A boolean indicating whether the file is readable

writable()​

File.writable()

Return value:

A boolean indicating whether the file is writable

seekable()​

File.seekable()

Return value:

A boolean indicating whether the file is seekable

Last updated on
Follow us on X
Let's chat on Slack