Skip to main content

自定义组件

首先配置好 Hadoop 基础环境

Apache Spark

修改 conf/spark-env.sh,在前面增加以下配置:

export HADOOP_HOME=YOUR_HADOOP_HOME
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_DIST_CLASSPATH=`hadoop classpath`

修改 bin/config.sh,在前面增加以下配置:

export HADOOP_HOME=YOUR_HADOOP_HOME
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_CLASSPATH=`hadoop classpath`

Presto

juicefs-hadoop.jar 放到 plugin/hive-hadoop2 目录

DataX

  • juicefs-hadoop.jar 放到 datax/plugin/writer/hdfswriter/libs 目录

  • 修改 DataX 配置文件

    "defaultFS": "jfs://your-jfs-name",
    "hadoopConfig": {
    "fs.jfs.impl": "com.juicefs.JuiceFileSystem",
    "fs.AbstractFileSystem.jfs.impl": "com.juicefs.JuiceFS",
    "juicefs.access-log": "/tmp/juicefs.access.log",
    "juicefs.token": "xxxxxxxxxxxxx",
    "juicefs.accesskey": "xxxxxxxxxxxxx",
    "juicefs.secretkey": "xxxxxxxxxxxxx"
    }