JuiceFS Java Client Parameter Reference
Add the following parameters to the core-site.xml of Hadoop.
Basic Configuration
| Item | Default Value | Description |
|---|---|---|
| fs.jfs.impl | com.juicefs.JuiceFileSystem | Specify the implementation of the storage type jfs://. |
| fs.AbstractFileSystem.jfs.impl | com.juicefs.JuiceFS | Specify the implementation of AbstractFileSystem for MapReduce. |
| juicefs.token | Credential for JuiceFS volume, checkout from the setting page of JuiceFS web console. | |
| juicefs.accesskey | Access Key for object store. It doesn’t need if the node has priviledge to access the object store. | |
| juicefs.secretkey | Secret Key for object store. It doesn’t need if the node has priviledge to access the object store. |
Cache Configuration
| Item | Default Value | Description |
|---|---|---|
| juicefs.cache-dir | /var/jfsCache | Cache directory, you could specify more than one folder, seperate by colon, or use wildcards *. Typically, the component doesn’t have permission to create these diectories, you need to create manully and give 0777 permission for components could share data in these folders. |
| juicefs.cache-size | 10240 | Cache capacity (unit MB), if multiple diectories are configured, this is total capacity for all cache folders. |
| juicefs.cache-replica | 1 | Number of cache replica. |
| juicefs.memory-size | 300 | Maximum memory used for read-ahead, unit MB. |
| juicefs.auto-create-cache-dir | true | Whether to create cache directories automatically. When set to false, it will ignore the a cache directory that doesn’t exist. |
| juicefs.free-space | 0.2 | Minimum free space ratio. When free space is under the ratio, it will clear the cache to free disk space, defaut 20%. |
| juicefs.metacache | true | Enable metadata cache. |
| juicefs.discover-nodes-url | Specify the node list of cluster, refresh every 10 minutes. If you use YARN manage resource, nodes could be discovered by RESTful API (seperate by comma): http://resourmanager1:8088/ws/v1/cluster/nodes/. You could also give a path of the configuration file, e.g file:///etc/nodes or jfs://your-jfs-name/etc/nodes. The configuration file must be plain text, one line for a hostname. |
Other Configuration
| Item | Default Value | Description |
|---|---|---|
| juicefs.access-log | The filepath for access log (e.g /tmp/juicefs.access.log), read and write permission is required for every Hadoop component. Log file will rotate automatically, and retain last 7 files. | |
| juicefs.debug | false | Enable DEBUG level log. |
| juicefs.max-uploads | 50 | Maximum number of concurrency for writing object store. |
| juicefs.superuser | hdfs | Specify the superuser |
| juicefs.bucket | Specify the bucket name of object store. | |
| juicefs.upload-limit | 0 | Speed limit for writing object store by a single process, units Byte/second. |
| juicefs.object-timeout | 5 | Timeout for visiting object store |
| juicefs.external | false | Using external domain to access object store |
| juicefs.rsaPrivKeyPath | The file path of RSA Private Key for data encryption. | |
| juicefs.rsaPassphase | The passphrase of RSA Key for data encryption. | |
| juicefs.file.checksum | false | Enable checksum for copying data via Hadoop DistCp |
When using multiple JuiceFS filesystems, all of above items can be specified for a single filesystem, the filesystem name JFS_NAME should to be placed in the middle of the configuration item, such as:
<property>
<name>juicefs.{JFS_NAME}.token</name>
<value>xxx</value>
</property>