Skip to main content

自定义组件

首先配置好 Hadoop 基础环境

Apache Spark

修改 conf/spark-env.sh，在前面增加以下配置：

export HADOOP_HOME=YOUR_HADOOP_HOME
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_DIST_CLASSPATH=`hadoop classpath`

修改 bin/config.sh，在前面增加以下配置：

export HADOOP_HOME=YOUR_HADOOP_HOME
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_CLASSPATH=`hadoop classpath`

Presto

将 juicefs-hadoop.jar 放到 plugin/hive-hadoop2 目录

DataX

  • 将 juicefs-hadoop.jar 放到 datax/plugin/writer/hdfswriter/libs 目录

  • 修改 DataX 配置文件

    "defaultFS": "jfs://your-jfs-name",
    "hadoopConfig": {
        "fs.jfs.impl": "com.juicefs.JuiceFileSystem",
        "fs.AbstractFileSystem.jfs.impl": "com.juicefs.JuiceFS",
        "juicefs.access-log": "/tmp/juicefs.access.log",
        "juicefs.token": "xxxxxxxxxxxxx",
        "juicefs.accesskey": "xxxxxxxxxxxxx",
        "juicefs.secretkey": "xxxxxxxxxxxxx"
    }