Skip to main content

Other Components

Deploy JuiceFS Java Client in Hadoop environment first.

Apache Spark

Modify conf/spark-env.sh, add following lines at the front:

export HADOOP_HOME=YOUR_HADOOP_HOME
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_DIST_CLASSPATH=`hadoop classpath`

Modify bin/config.sh, add following lines at the front:

export HADOOP_HOME=YOUR_HADOOP_HOME
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_CLASSPATH=`hadoop classpath`

Presto

Copy juicefs-hadoop.jar to plugin/hive-hadoop2 folder.

DataX

  • Copy juicefs-hadoop.jar to datax/plugin/writer/hdfswriter/libs folder.

  • Modify the configuration file of DataX:

    "defaultFS": "jfs://your-jfs-name",
    "hadoopConfig": {
    "fs.jfs.impl": "com.juicefs.JuiceFileSystem",
    "fs.AbstractFileSystem.jfs.impl": "com.juicefs.JuiceFS",
    "juicefs.access-log": "/tmp/juicefs.access.log",
    "juicefs.token": "xxxxxxxxxxxxx",
    "juicefs.accesskey": "xxxxxxxxxxxxx",
    "juicefs.secretkey": "xxxxxxxxxxxxx"
    }