JuiceFS S3 Gateway
JuiceFS splits and upload files to the underlying object storage, and applications often use the exposed POSIX API. However, if you need to use an S3-compatible API to access JuiceFS files, JuiceFS S3 Gateway provides a solution. Here is its architecture:
S3 Gateway is usually used to:
- Expose the S3 API for JuiceFS file system, enabling applications to access JuiceFS via the S3 SDK.
- Use tools like s3cmd, AWS CLI and MinIO Client to access and modify files stored in JuiceFS.
- When transferring data across regions, use S3 Gateway as an unified data export endpoint. This eliminates metadata latency and improves performance. See Sync across regions using S3 Gateway.
Deploy S3 Gateway
Similar to the mount
command, gateway
reads the local JuiceFS Client configuration file (~/.juicefs/$VOL_NAME.conf
by default). If this configuration does not exist, you need to authenticate via the Web Console using the auth
command to generate it:
juicefs auth $VOL_NAME --token=xxx --access-key=xxx --secret-key=xxx
S3 Gateway is implemented using the open source MinIO code, so you have to provide some MinIO related variables:
export MINIO_ACCESS_KEY="admin"
export MINIO_SECRET_KEY="password"
Without these variables, you will encounter errors like MINIO_ROOT_USER should be specified as an environment variable with at least 3 characters
.
With the credential variables in place, run the gateway
command to launch the gateway:
juicefs gateway myjfs 127.0.0.1:8888
Expected output:
2023/03/21 20:15:49.945403 juicefs[97188] <INFO>: connected to 47.103.20.252:9308 [client.go:874]
2023/03/21 20:15:49.965411 juicefs[97188] <INFO>: Cache: /Users/herald/.juicefs/cache/jfs8 capacity: 102400 MB [disk_cache.go:747]
Endpoint: http://127.0.0.1:8888
Browser Access:
http://127.0.0.1:8888
Object API (Amazon S3 compatible):
Go: https://docs.min.io/docs/golang-client-quickstart-guide
Java: https://docs.min.io/docs/java-client-quickstart-guide
Python: https://docs.min.io/docs/python-client-quickstart-guide
JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
.NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
Assuming your server's public IP is 111.2.3.4
and you want to make the gateway accessible over the internet on port 9000
, adjust the startup command like this:
juicefs gateway myjfs 111.2.3.4:9000
Access JuiceFS S3 Gateway
You can access JuiceFS S3 Gateway using various S3 API-supported clients, desktop applications, and web applications. Ensure the correct address and port are used.
The following examples assume that JuiceFS S3 Gateway is running on the local host and being accessed by third-party clients. Adjust the gateway's address according to your specific scenario.
Use the AWS CLI
Download and install the AWS Command Line Interface (AWS CLI) from the AWS website.
Configure it:
$ aws configure
AWS Access Key ID [None]: admin
AWS Secret Access Key [None]: 12345678
Default region name [None]:
Default output format [None]:
The program guides you interactively to add new configurations. Use the same values for Access Key ID
as MINIO_ROOT_USER
and Secret Access Key
as MINIO_ROOT_PASSWORD
. Leave the region name and output format blank.
Now you can use the aws s3
command to access JuiceFS storage, for example:
# List buckets
$ aws --endpoint-url http://localhost:9000 s3 ls
# List objects in bucket
$ aws --endpoint-url http://localhost:9000 s3 ls s3://<bucket>
Use the MinIO Client
To avoid compatibility issues, we recommend using the RELEASE.2021-04-22T17-40-00Z
version of the MinIO Client (mc
). You can find historical versions with different architectures of mc
at this address. For example, for the amd64 architecture, you can download the RELEASE.2021-04-22T17-40-00Z
version of mc
from this link.
After installing mc
, add a new alias:
mc alias set juicefs http://localhost:9000 admin 12345678
Then, you can freely copy, move, add, and delete files and folders between the local disk, JuiceFS storage, and other cloud storage services using the mc
client.
$ mc ls juicefs/jfs
[2021-10-20 11:59:00 CST] 130KiB avatar-2191932_1920.png
[2021-10-20 11:59:00 CST] 4.9KiB box-1297327.svg
[2021-10-20 11:59:00 CST] 21KiB cloud-4273197.svg
[2021-10-20 11:59:05 CST] 17KiB hero.svg
[2021-10-20 11:59:06 CST] 1.7MiB hugo-rocha-qFpnvZ_j9HU-unsplash.jpg
[2021-10-20 11:59:06 CST] 16KiB man-1352025.svg
[2021-10-20 11:59:06 CST] 1.3MiB man-1459246.ai
[2021-10-20 11:59:08 CST] 19KiB sign-up-accent-left.07ab168.svg
[2021-10-20 11:59:10 CST] 11MiB work-4997565.svg
Common features
Multi-bucket support
By default, JuiceFS S3 Gateway only allows one bucket. The bucket name is the file system name. If you need multiple buckets, you can add --multi-buckets
at startup to enable multi-bucket support. This parameter exports each subdirectory under the top-level directory of the JuiceFS file system as a separate bucket. Creating a bucket means creating a subdirectory with the same name at the top level of the file system.
juicefs gateway myjfs localhost:9000 --multi-buckets
Retain ETags
By default, JuiceFS S3 Gateway does not save or return object ETag information. You can enable this with --keep-etag
.
Enable object tags
Object tags are not supported by default, but you can use --object-tag
to enable them.
Enable virtual host-style requests
By default, JuiceFS S3 Gateway supports path-style requests in the format of http://mydomain.com/bucket/object
. The MINIO_DOMAIN
environment variable is used to enable virtual host-style requests. If the request's Host
header information matches (.+).mydomain.com
, the matched pattern $1
is used as the bucket, and the path is used as the object.
For example:
export MINIO_DOMAIN=mydomain.com
Adjust the IAM refresh interval
The default refresh interval for Identity and Access Management (IAM) caching is 5 minutes. You can adjust this using --refresh-iam-interval
. The value of this parameter is a time string with a unit, such as "300ms", "-1.5h", or "2h45m." Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", and "h".
For example, to set a refresh interval of 1 minute:
juicefs gateway xxxx xxxx --refresh-iam-interval 1m
Multiple gateway instances
JuiceFS' distributed architecture allows you to run multiple JuiceFS S3 Gateway instances across different nodes simultaneously. This enhances both the availability and performance of the S3 Gateways. Each S3 Gateway instance independently handles requests but accesses the same JuiceFS file system. Keep the following points in mind:
- Ensure that all instances are started with the same user at initialization; use the same UID and GID for all instances.
- IAM refresh times may vary between nodes, but avoid setting refresh intervals too short to prevent excessive pressure on JuiceFS.
- The address and port listened by each instance can be freely configured. If multiple instances are started on the same machine, ensure that there is no conflict in port numbers.
Advanced features
The core feature of JuiceFS S3 Gateway is to provide the S3 API. Now, the support for the S3 protocol is comprehensive. Version 1.2 supports IAM and bucket event notifications.
These advanced features require the RELEASE.2021-04-22T17-40-00Z
version of the mc
client. For the usage of these advanced features, see the MinIO documentation or the mc
command-line help information.
If you are unsure about the available features or how to use a specific feature, you can append -h
to a subcommand to view the help information.
Identity and access control
Regular users
Before version 5.1, juicefs gateway
only created a superuser when starting, and this superuser belonged only to that process. Even if multiple gateway processes shared the same file system, their users were isolated between processes. You could set different superusers for each gateway process, and they were independent and unaffected by each other.
Starting from version 5.1, juicefs gateway
still requires setting a superuser at startup, and this superuser remains isolated per process. However, it allows adding new users using mc admin user add
. Newly added users are shared across the same file system. You can manage new users using mc admin user
. This supports adding, disabling, enabling, and deleting users, as well as viewing all users and displaying user information and policies.
$ mc admin user -h
NAME:
mc admin user - manage users
USAGE:
mc admin user COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]
COMMANDS:
add Add a new user
disable Disable a user
enable Enable a user
remove Remove a user
list List all users
info Display information of a user
policy Export user policies in JSON format
svcacct Manage service accounts
An example of adding a user:
# Add a new user.
$ mc admin user add myjfs user1 admin123
# List current users.
$ mc admin user list myjfs
enabled user1
# List current users in JSON format.
$ mc admin user list myjfs --json
{
"status": "success",
"accessKey": "user1",
"userStatus": "enabled"
}
Service accounts
Service accounts are used to create a copy of an existing user with the same permissions, allowing different applications to use separate access keys. The privileges for service accounts inherit from their parent users. They can be managed using the command:
$ mc admin user svcacct -h
NAME:
mc admin user svcacct - manage service accounts
USAGE:
mc admin user svcacct COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]
COMMANDS:
add Add a new service account
ls List services accounts
rm Remove a service account
info Get a service account information
set Edit an existing service account
enable Enable a service account
disable Disable a services account
Service accounts inherit privileges from their parent users and cannot have policies attached directly.
For example, let's say there is an existing user named user1
. You can create a service account called svcacct1
for it as follows:
mc admin user svcacct add myjfs user1 --access-key svcacct1 --secret-key 123456abc
If the parent user, user1
, has read-only permissions, svcacct1
will also inherit these permissions. To grant different permissions to svcacct1
, you must adjust the privileges of the parent user.
AssumeRole security token service
The S3 Gateway Security Token Service (STS) is a service that allows clients to request temporary credentials to access MinIO resources. The working principle of temporary credentials is almost the same as default administrator credentials but with some differences:
- Temporary credentials are short-lived. They can be configured to last from minutes to hours. After expiration, the gateway no longer recognizes them and does not allow any form of API request access.
- Temporary credentials do not need to be stored with the application. They are dynamically generated and provided to the application when requested. When temporary credentials expire, applications can request new credentials.
The AssumeRole
operation returns a set of temporary security credentials. You can use them to access gateway resources. AssumeRole
requires authorization credentials for an existing gateway user and returns temporary security credentials, including an access key, secret key, and security token. Applications can use these temporary security credentials to sign requests for gateway API operations. The policies applied to these temporary credentials inherit from gateway user credentials.
By default, AssumeRole
creates temporary security credentials with a validity period of one hour. However, you can specify the duration of the credentials using the optional parameter DurationSeconds
, which can range from 900 (15 minutes) to 604,800 (7 days).
API request parameters
-
Version
Indicates the STS API version information. The only supported value is '2011-06-15', borrowed from the AWS STS API documentation for compatibility.
Parameter Value Type String Require Yes -
AUTHPARAMS
Indicates the STS API authorization information. If you are familiar with AWS Signature V4 authorization headers, this STS API supports the signature V4 authorization as described here.
-
DurationSeconds
Duration in seconds. This value can range from 900 seconds (15 minutes) to 7 days. If the value is higher than this setting, the operation fails. By default, this value is set to 3,600 seconds.
Parameter Value Type Integer Valid range From 900 to 604,800 Required No -
Policy
A JSON-format IAM policy that you want to use as an inline session policy. This parameter is optional. Passing a policy to this operation returns new temporary credentials. The permissions of the generated session are the intersection of preset policy names and the policy set here. You cannot use this policy to grant more permissions than allowed by the assumed preset policy names.
Parameter Value Type String Valid range From 1 to 2,048 Required No
Response elements
The XML response of this API is similar to AWS STS AssumeRole
.
Errors
The XML error response of this API is similar to AWS STS AssumeRole
.
A POST
request example
http://minio:9000/?Action=AssumeRole&DurationSeconds=3600&Version=2011-06-15&Policy={"Version":"2012-10-17","Statement":[{"Sid":"Stmt1","Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}]}&AUTHPARAMS
A response example
<?xml version="1.0" encoding="UTF-8"?>
<AssumeRoleResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
<AssumeRoleResult>
<AssumedRoleUser>
<Arn/>
<AssumeRoleId/>
</AssumedRoleUser>
<Credentials>
<AccessKeyId>Y4RJU1RNFGK48LGO9I2S</AccessKeyId>
<SecretAccessKey>sYLRKS1Z7hSjluf6gEbb9066hnx315wHTiACPAjg</SecretAccessKey>
<Expiration>2019-08-08T20:26:12Z</Expiration>
<SessionToken>eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJZNFJKVTFSTkZHSzQ4TEdPOUkyUyIsImF1ZCI6IlBvRWdYUDZ1Vk80NUlzRU5SbmdEWGo1QXU1WWEiLCJhenAiOiJQb0VnWFA2dVZPNDVJc0VOUm5nRFhqNUF1NVlhIiwiZXhwIjoxNTQxODExMDcxLCJpYXQiOjE1NDE4MDc0NzEsImlzcyI6Imh0dHBzOi8vbG9jYWxob3N0Ojk0NDMvb2F1dGgyL3Rva2VuIiwianRpIjoiYTBiMjc2MjktZWUxYS00M2JmLTg3MzktZjMzNzRhNGNkYmMwIn0.ewHqKVFTaP-j_kgZrcOEKroNUjk10GEp8bqQjxBbYVovV0nHO985VnRESFbcT6XMDDKHZiWqN2vi_ETX_u3Q-w</SessionToken>
</Credentials>
</AssumeRoleResult>
<ResponseMetadata>
<RequestId>c6104cbe-af31-11e0-8154-cbc7ccf896c7</RequestId>
</ResponseMetadata>
</AssumeRoleResponse>
Use the AWS CLI with the AssumeRole API
-
Start the gateway and create a user named
foobar
. -
Configure the AWS CLI:
[foobar]
region = us-east-1
aws_access_key_id = foobar
aws_secret_access_key = foo12345 -
Use the AWS CLI to request the
AssumeRole
API.noteIn the command below,
--role-arn
and--role-session-name
have no significance for the gateway. You can set them to any value that meets the command line requirements.$ aws --profile foobar --endpoint-url http://localhost:9000 sts assume-role --policy '{"Version":"2012-10-17","Statement":[{"Sid":"Stmt1","Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}]}' --role-arn arn:xxx:xxx:xxx:xxxx --role-session-name anything
{
"AssumedRoleUser": {
"Arn": ""
},
"Credentials": {
"SecretAccessKey": "xbnWUoNKgFxi+uv3RI9UgqP3tULQMdI+Hj+4psd4",
"SessionToken": "eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJLOURUSU1VVlpYRVhKTDNBVFVPWSIsImV4cCI6MzYwMDAwMDAwMDAwMCwicG9saWN5IjoidGVzdCJ9.PetK5wWUcnCJkMYv6TEs7HqlA4x_vViykQ8b2T_6hapFGJTO34sfTwqBnHF6lAiWxRoZXco11B0R7y58WAsrQw",
"Expiration": "2019-02-20T19:56:59-08:00",
"AccessKeyId": "K9DTIMUVZXEXJL3ATUOY"
}
}
Access the AssumeRole API in Go applications
See the MinIO official example program.
Superusers set by environment variables cannot use the AssumeRole APIs. Only users added by mc admin user add
can use these APIs.
Permission management
By default, newly created users have no permissions and need to be granted permissions using mc admin policy
before they can be used. This command supports adding, deleting, updating, and listing policies, as well as adding, deleting, and updating permissions for users.
$ mc admin policy -h
NAME:
mc admin policy - manage policies defined in the MinIO server
USAGE:
mc admin policy COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]
COMMANDS:
add Add a new policy
remove Remove a policy
list List all policies
info Show information on a policy
set Set an IAM policy on a user or group
unset Unset an IAM policy for a user or group
update Attach a new IAM policy to a user or group
The gateway includes the following common policies:
readonly
: Read-only users.readwrite
: Read-write users.writeonly
: Write-only users.consoleAdmin
: Read-write-admin users, where "admin" means the ability to use management APIs such as creating users.
For example, to set a user as a read-only user:
# Set user1 as a read-only user.
$ mc admin policy set myjfs readonly user=user1
# Check user policy.
$ mc admin user list myjfs
enabled user1 readonly
For custom policies, use mc admin policy add
:
$ mc admin policy add -h
NAME:
mc admin policy add - add new policy
USAGE:
mc admin policy add TARGET POLICYNAME POLICYFILE
POLICYNAME:
Name of the canned policy on MinIO server.
POLICYFILE:
Name of the policy file associated with the policy name.
EXAMPLES:
1. Add a new canned policy 'writeonly'.
$ mc admin policy add myjfs writeonly /tmp/writeonly.json
The policy file to be added here must be in JSON format with IAM-compatible syntax, and no more than 2,048 characters. This syntax allows for more fine-grained access control. If you are unfamiliar with this, you can first use the following command to see the simple policies and then modify them accordingly.
$ mc admin policy info myjfs readonly
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::*"
]
}
]
}
User group management
JuiceFS S3 Gateway supports creating user groups, similar to Linux user groups, and uses mc admin group
for management. You can set one or more users to a group and grant permissions uniformly to the group. This usage is similar to user management.
$ mc admin group -h
NAME:
mc admin group - manage groups
USAGE:
mc admin group COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]
COMMANDS:
add Add users to a new or existing group
remove Remove a group or its members
info Display group information
list List all groups
enable Enable a group
disable Disable a group
Anonymous access management
In addition to user-specific permissions, anonymous access management is also possible. This allows specific objects or buckets to be accessible to anyone. You can use the mc policy
command to manage this functionality.
Name:
mc policy - manage anonymous access to buckets and objects
USAGE:
mc policy [FLAGS] set PERMISSION TARGET
mc policy [FLAGS] set-json FILE TARGET
mc policy [FLAGS] get TARGET
mc policy [FLAGS] get-json TARGET
mc policy [FLAGS] list TARGET
PERMISSION:
Allowed policies are: [none, download, upload, public].
FILE:
A valid S3 policy JSON filepath.
EXAMPLES:
1. Set bucket to "download" on Amazon S3 cloud storage.
$ mc policy set download s3/burningman2011
2. Set bucket to "public" on Amazon S3 cloud storage.
$ mc policy set public s3/shared
3. Set bucket to "upload" on Amazon S3 cloud storage.
$ mc policy set upload s3/incoming
4. Set policy to "public" for bucket with prefix on Amazon S3 cloud storage.
$ mc policy set public s3/public-commons/images
5. Set a custom prefix based bucket policy on Amazon S3 cloud storage using a JSON file.
$ mc policy set-json /path/to/policy.json s3/public-commons/images
6. Get bucket permissions.
$ mc policy get s3/shared
7. Get bucket permissions in JSON format.
$ mc policy get-json s3/shared
8. List policies set to a specified bucket.
$ mc policy list s3/shared
9. List public object URLs recursively.
$ mc policy --recursive links s3/shared/
The gateway has built-in support for four types of anonymous permissions by default:
none
: Disallows anonymous access (typically used to clear existing permissions).download
: Allows anyone to read.upload
: Allows anyone to write.public
: Allows anyone to read and write.
The following example shows how to set an object to allow anonymous downloads:
# Set testbucket1/afile for anonymous access.
mc policy set download useradmin/testbucket1/afile
# View specific permissions.
mc policy get-json useradmin/testbucket1/afile
$ mc policy --recursive links useradmin/testbucket1/
http://127.0.0.1:9001/testbucket1/afile
# Directly download the object.
wget http://127.0.0.1:9001/testbucket1/afile
# Clear download permission for a file.
mc policy set none useradmin/testbucket1/afile
Configuration effective time
All management API updates for JuiceFS S3 Gateway take effect immediately and are persisted to the JuiceFS file system. Clients that accept these API requests also immediately reflect these changes.
However, in a multi-server gateway setup, the situation is slightly different. This is because when the gateway handles request authentication, it uses in-memory cached information as the validation baseline. Otherwise, reading configuration file content for every request would pose unacceptable performance issues. However, caching also introduces potential inconsistencies between cached data and the configuration file.
Currently, JuiceFS S3 Gateway's cache refresh strategy involves forcibly updating the in-memory cache every 5 minutes (certain operations also trigger cache update operations). This ensures that configuration changes take effect within a maximum of 5 minutes in a multi-server setup. You can adjust this time by using the --refresh-iam-interval
parameter. If immediate effect on a specific gateway is required, you can manually restart it.
Bucket event notifications
You can use bucket event notifications to monitor events happening on objects within a storage bucket and trigger certain actions in response.
Currently supported object event types include:
s3:ObjectCreated:Put
s3:ObjectCreated:CompleteMultipartUpload
s3:ObjectAccessed:Head
s3:ObjectCreated:Post
s3:ObjectRemoved:Delete
s3:ObjectCreated:Copy
s3:ObjectAccessed:Get
Supported global events include:
s3:BucketCreated
s3:BucketRemoved
You can use the mc
client tool with the event subcommand to set up and monitor event notifications. Notifications sent by MinIO for publishing events are in JSON format. See the JSON structure.
To reduce dependencies, JuiceFS S3 Gateway has cut support for certain event destination types. Currently, storage bucket events can be published to the following destinations:
- Redis
- MySQL
- PostgreSQL
- Webhooks
$ mc admin config get myjfs | grep notify
notify_webhook publish bucket notifications to webhook endpoints
notify_mysql publish bucket notifications to MySQL databases
notify_postgres publish bucket notifications to Postgres databases
notify_redis publish bucket notifications to Redis datastores
Here, assuming the JuiceFS file system name is images
, enable the S3 Gateway service and define its alias as myjfs
in mc. For the S3 Gateway, the JuiceFS file system name images
serves as a bucket name.
Use Redis to publish events
Redis event destination supports two formats: namespace
and access
.
In the namespace
format, the gateway synchronizes objects in the bucket to entries in a Redis hash. Each entry corresponds to an object in the storage bucket, with the key set to "bucket name/object name" and the value as JSON-formatted event data specific to that gateway object. Any updates or deletions of objects also update or delete corresponding entries in the hash.
In the access
format, the gateway uses RPUSH to add events to a list. Each element in this list is a JSON-formatted list with two elements:
- A timestamp string
- A JSON object containing event data related to operations on the bucket
In this format, elements in the list are not updated or deleted.
To use notification destinations in namespace
and access
formats:
-
Configure Redis with the gateway.
Use the
mc admin config set
command to configure Redis as the event notification destination:# Command-line parameters
# mc admin config set myjfs notify_redis[:name] address="xxx" format="namespace|access" key="xxxx" password="xxxx" queue_dir="" queue_limit="0"
# An example
$ mc admin config set myjfs notify_redis:1 address="127.0.0.1:6379/1" format="namespace" key="bucketevents" password="yoursecret" queue_dir="" queue_limit="0"You can use
mc admin config get myjfs notify_redis
to view the configuration options. Different types of destinations have different configuration options. For Redis type, it has the following configuration options:$ mc admin config get myjfs notify_redis
notify_redis enable=off format=namespace address= key= password= queue_dir= queue_limit=0Here are the meanings of each configuration option:
notify_redis[:name] Supports setting multiple Redis instances with different names.
address* (address) Address of the Redis server. For example: localhost:6379.
key* (string) Redis key to store/update events. The key is created automatically.
format* (namespace*|access) Whether it is namespace or access. Default is 'namespace'.
password (string) Password for the Redis server.
queue_dir (path) Directory to store unsent messages, for example, '/home/events'.
queue_limit (number) Maximum limit of unsent messages. Default is '100000'.
comment (sentence) Optional comment description.The gateway supports persistent event storage. Persistent storage backs up events when the Redis broker is offline and replays events when the broker comes back online. You can set the directory for event storage using the
queue_dir
field and the maximum limit for storage usingqueue_limit
. For example, you can setqueue_dir
to/home/events
, and you can setqueue_limit
to 1,000. By default,queue_limit
is 100,000. Before updating the configuration, you can use themc admin config get
command to get the current configuration.$ mc admin config get myjfs notify_redis
notify_redis:1 address="127.0.0.1:6379/1" format="namespace" key="bucketevents" password="yoursecret" queue_dir="" queue_limit="0"
# Effective after restart
$ mc admin config set myjfs notify_redis:1 queue_limit="1000"
Successfully applied new settings.
Please restart your server 'mc admin service restart myjfs'.
# Note that you cannot use `mc admin service restart myjfs` to restart. JuiceFS S3 Gateway does not currently support this functionality. You need to manually restart JuiceFS S3 Gateway when prompted after configuring with `mc`.After using the
mc admin config set
command to update the configuration, restart JuiceFS S3 Gateway to apply the changes. JuiceFS S3 Gateway will output a line similar toSQS ARNs: arn:minio:sqs::1:redis
.Based on your needs, you can add multiple Redis destinations by providing the identifier for each Redis instance (like the "1" in the example "notify_redis:1") along with the configuration parameters for each instance.
-
Enable bucket notifications.
Now you can enable event notifications on a bucket named "images." When a JPEG file is created or overwritten, a new key is created or an existing key is updated in the previously configured Redis hash. If an existing object is deleted, the corresponding key is also removed from the hash. Therefore, the rows in the Redis hash map to
.jpg
objects in the "images" bucket.To configure bucket notifications, you need to use the Amazon Resource Name (ARN) information outputted by the gateway in the previous steps. See more information about ARNs.
You can use the
mc
tool to add these configuration details. Assuming the gateway service alias is myjfs, you can execute the following script:mc event add myjfs/images arn:minio:sqs::1:redis --suffix .jpg
mc event list myjfs/images
arn:minio:sqs::1:redis s3:ObjectCreated:*,s3:ObjectRemoved:*,s3:ObjectAccessed:* Filter: suffix=".jpg" -
Verify Redis.
Start the
redis-cli
Redis client program to check the content in Redis. Running themonitor
Redis command will output every command executed on Redis.redis-cli -a yoursecret
127.0.0.1:6379> monitor
OKUpload a file named
myphoto.jpg
to theimages
bucket.mc cp myphoto.jpg myjfs/images
In the previous terminal, you can see the operations performed by the gateway on Redis:
127.0.0.1:6379> monitor
OK
1712562516.867831 [1 192.168.65.1:59280] "hset" "bucketevents" "images/myphoto.jpg" "{\"Records\":[{\"eventVersion\":\"2.0\",\"eventSource\":\"minio:s3\",\"awsRegion\":\"\",\"eventTime\":\"2024-04-08T07:48:36.865Z\",\"eventName\":\"s3:ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"admin\"},\"requestParameters\":{\"principalId\":\"admin\",\"region\":\"\",\"sourceIPAddress\":\"127.0.0.1\"},\"responseElements\":{\"content-length\":\"0\",\"x-amz-request-id\":\"17C43E891887BA48\",\"x-minio-origin-endpoint\":\"http://127.0.0.1:9001\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"Config\",\"bucket\":{\"name\":\"images\",\"ownerIdentity\":{\"principalId\":\"admin\"},\"arn\":\"arn:aws:s3:::images\"},\"object\":{\"key\":\"myphoto.jpg\",\"size\":4,\"eTag\":\"40b134ab8a3dee5dd9760a7805fd495c\",\"userMetadata\":{\"content-type\":\"image/jpeg\"},\"sequencer\":\"17C43E89196AE2A0\"}},\"source\":{\"host\":\"127.0.0.1\",\"port\":\"\",\"userAgent\":\"MinIO (darwin; arm64) minio-go/v7.0.11 mc/RELEASE.2021-04-22T17-40-00Z\"}}]}"Here, you can see that the gateway executed the
HSET
command on theminio_events
key.In the
access
format,minio_events
is a list, and the gateway callsRPUSH
to add it to the list. In themonitor
command, you can see:127.0.0.1:6379> monitor
OK
1712562751.922469 [1 192.168.65.1:61102] "rpush" "aceesseventskey" "[{\"Event\":[{\"eventVersion\":\"2.0\",\"eventSource\":\"minio:s3\",\"awsRegion\":\"\",\"eventTime\":\"2024-04-08T07:52:31.921Z\",\"eventName\":\"s3:ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"admin\"},\"requestParameters\":{\"principalId\":\"admin\",\"region\":\"\",\"sourceIPAddress\":\"127.0.0.1\"},\"responseElements\":{\"content-length\":\"0\",\"x-amz-request-id\":\"17C43EBFD35A53B8\",\"x-minio-origin-endpoint\":\"http://127.0.0.1:9001\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"Config\",\"bucket\":{\"name\":\"images\",\"ownerIdentity\":{\"principalId\":\"admin\"},\"arn\":\"arn:aws:s3:::images\"},\"object\":{\"key\":\"myphoto.jpg\",\"size\":4,\"eTag\":\"40b134ab8a3dee5dd9760a7805fd495c\",\"userMetadata\":{\"content-type\":\"image/jpeg\"},\"sequencer\":\"17C43EBFD3DACA70\"}},\"source\":{\"host\":\"127.0.0.1\",\"port\":\"\",\"userAgent\":\"MinIO (darwin; arm64) minio-go/v7.0.11 mc/RELEASE.2021-04-22T17-40-00Z\"}}],\"EventTime\":\"2024-04-08T07:52:31.921Z\"}]"
Use MySQL to publish events
The MySQL notification destination supports two formats: namespace
and access
.
If you use the namespace
format, the gateway synchronizes objects in the bucket to rows in the database table. Each row has two columns:
key_name
. It is the bucket name plus the object name.value
. It is the JSON-formatted event data about that gateway object.
If objects are updated or deleted, the corresponding rows in the table are also updated or deleted.
If you use the access
format, the gateway adds events to the table. Rows have two columns:
event_time
. It is the time the event occurred on the gateway server.event_data
. It is the JSON-formatted event data about that gateway object.
In this format, rows are not deleted or modified.
The following steps show how to use the notification destination in namespace
format. The access
format is similar and not further described here.
-
Ensure the MySQL version meets the minimum requirements.
JuiceFS S3 Gateway requires MySQL version 5.7.8 or above, because it uses the JSON data type introduced in MySQL 5.7.8.
-
Configure MySQL to the gateway.
Use the
mc admin config set
command to configure MySQL as the event notification destination.mc admin config set myjfs notify_mysql:myinstance table="minio_images" dsn_string="root:123456@tcp(172.17.0.1:3306)/miniodb"
You can use
mc admin config get myjfs notify_mysql
to view the configuration options. Different destination types have different configuration options. For MySQL type, the following configuration options are available:$ mc admin config get myjfs notify_mysql
format=namespace dsn_string= table= queue_dir= queue_limit=0 max_open_connections=2Here are the meanings of each configuration item:
KEY:
notify_mysql[:name] Publish bucket notifications to the MySQL database. When multiple MySQL server endpoints are required, you can add a user-specified "name" to each configuration, for example, "notify_mysql:myinstance."
ARGS:
dsn_string* (string) MySQL data source name connection string, for example, "<user>:<password>@tcp(<host>:<port>)/<database>".
table* (string) Name of the database table to store/update events. The table is automatically created.
format* (namespace*|access) 'namespace' or 'access.' The default is 'namespace.'
queue_dir (path) The directory for storing unsent messages, for example, '/home/events'.
queue_limit (number) The maximum limit of unsent messages. The default is '100000'.
comment (sentence) Optional comment description.dsn_string
is required and must be in the format<user>:<password>@tcp(<host>:<port>)/<database>
.MinIO supports persistent event storage. Persistent storage backs up events when the MySQL connection is offline and replays events when the broker comes back online. You can set the storage directory for events using the
queue_dir
field, and the maximum storage limit usingqueue_limit
. For example, you can setqueue_dir
to/home/events
, andqueue_limit
to 1,000. By default,queue_limit
is set to 100,000.Before updating the configuration, you can use the
mc admin config get
command to get the current configuration.$ mc admin config get myjfs/ notify_mysql
notify_mysql:myinstance enable=off format=namespace host= port= username= password= database= dsn_string= table= queue_dir= queue_limit=0Update the MySQL notification configuration using the
mc admin config set
command with thedsn_string
parameter:mc admin config set myjfs notify_mysql:myinstance table="minio_images" dsn_string="root:xxxx@tcp(127.0.0.1:3306)/miniodb"
You can add multiple MySQL server endpoints as needed, by providing the identifier of the MySQL instance (for example, "myinstance") and the configuration parameter information for each instance.
After updating the configuration with the
mc admin config set
command, restart the gateway to apply the configuration changes. The gateway server will output a line during startup similar toSQS ARNs: arn:minio:sqs::myinstance:mysql
. -
Enable bucket notifications.
Now you can enable event notifications on a bucket named "images." When a file is uploaded to the bucket, a new record is inserted into MySQL, or an existing record is updated. If an existing object is deleted, the corresponding record is also deleted from the MySQL table. Therefore, each row in the MySQL table corresponds to an object in the bucket.
To configure bucket notifications, you need to use the ARN information outputted by MinIO in previous steps. See more information about ARNs.
Assuming the gateway service alias is myjfs, you can execute the following script:
# Add notification configuration to the 'images' bucket using the MySQL ARN. The --suffix parameter is used to filter events.
mc event add myjfs/images arn:minio:sqs::myinstance:mysql --suffix .jpg
# Print the notification configuration on the 'images' bucket.
mc event list myjfs/images
arn:minio:sqs::myinstance:mysql s3:ObjectCreated:*,s3:ObjectRemoved:*,s3:ObjectAccessed:* Filter: suffix=”.jpg” -
Verify MySQL.
Open a new terminal and upload a JPEG image to the
images
bucket:mc cp myphoto.jpg myjfs/images
Open a MySQL terminal and list all records in the
minio_images
table. You will find a newly inserted record.
Use PostgreSQL to publish events
The method of publishing events using PostgreSQL is similar to publishing MinIO events using MySQL, with PostgreSQL version 9.5 or above required. The gateway uses PostgreSQL 9.5's INSERT ON CONFLICT
(aka UPSERT
) feature and 9.4's jsonb
data type.
Use a webhook to publish events
Webhooks use a push model to get data instead of continually pulling.
-
Configure a webhook to the gateway.
The gateway supports persistent event storage. Persistent storage backs up events when the webhook is offline and replays events when the broker comes back online. You can set the directory for event storage using the
queue_dir
field, and the maximum storage limit usingqueue_limit
. For example, you can setqueue_dir
to/home/events
andqueue_limit
to 1,000. By default,queue_limit
is 100,000.KEY:
notify_webhook[:name] Publish bucket notifications to webhook endpoints.
ARGS:
endpoint* (url) Webhook server endpoint, for example, http://localhost:8080/minio/events.
auth_token (string) Opaque token or JWT authorization token.
queue_dir (path) The directory for storing unsent messages, for example, '/home/events'.
queue_limit (number) The maximum limit of unsent messages. The default is '100000'.
client_cert (string) The client certificate for mTLS authentication of the webhook.
client_key (string) The client certificate key for mTLS authentication of the webhook.
comment (sentence) Optional comment description.Use the
mc admin config set
command to update the configuration. The endpoint here is the service that listens for webhook notifications. Save the configuration file and restart the MinIO service to apply the changes. Note that when restarting MinIO, this endpoint must be up and accessible.mc admin config set myjfs notify_webhook:1 queue_limit="0" endpoint="http://localhost:3000" queue_dir=""
-
Enable bucket notifications.
Now you can enable event notifications. When a file is uploaded to the bucket, an event is triggered. Here, the ARN value is
arn:minio:sqs::1:webhook
. See more information about ARNs.mc mb myjfs/images-thumbnail
mc event add myjfs/images arn:minio:sqs::1:webhook --event put --suffix .jpgIf the command report cannot create a bucket, please check if the S3 Gateway has enabled Multi-bucket support.
-
Use Thumbnailer to verify.
Thumbnailer is a project that generates thumbnails using MinIO's
listenBucketNotification
API. JuiceFS uses Thumbnailer to listen to gateway notifications. If a file is uploaded to the gateway service, Thumbnailer listens to that notification, generates a thumbnail, and uploads it to the gateway service.To install Thumbnailer:
git clone https://github.com/minio/thumbnailer/
npm installOpen the Thumbnailer's
config/webhook.json
configuration file, add the configuration for the MinIO server, and start Thumbnailer using:NODE_ENV=webhook node thumbnail-webhook.js
Thumbnailer runs on
http://localhost:3000/
.Next, configure the MinIO server to send messages to this URL (mentioned in step 1) and set up bucket notifications using
mc
(mentioned in step 2). Then upload an image to the gateway server:mc cp ~/images.jpg myjfs/images
.../images.jpg: 8.31 KB / 8.31 KB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 100.00% 59.42 KB/s 0sAfter a moment, use
mc ls
to check the content of the bucket. You will see a thumbnail.mc ls myjfs/images-thumbnail
[2017-02-08 11:39:40 IST] 992B images-thumbnail.jpg