自定义组件
首先配置好 Hadoop 基础环境
Apache Spark
修改 conf/spark-env.sh
,在前面增加以下配置:
export HADOOP_HOME=YOUR_HADOOP_HOME
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_DIST_CLASSPATH=`hadoop classpath`
Apache Flink
修改 bin/config.sh
,在前面增加以下配置:
export HADOOP_HOME=YOUR_HADOOP_HOME
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_CLASSPATH=`hadoop classpath`
Presto
将 juicefs-hadoop.jar
放到 plugin/hive-hadoop2
目录
DataX
将
juicefs-hadoop.jar
放到datax/plugin/writer/hdfswriter/libs
目录修改 DataX 配置文件
"defaultFS": "jfs://your-jfs-name",
"hadoopConfig": {
"fs.jfs.impl": "com.juicefs.JuiceFileSystem",
"fs.AbstractFileSystem.jfs.impl": "com.juicefs.JuiceFS",
"juicefs.access-log": "/tmp/juicefs.access.log",
"juicefs.token": "xxxxxxxxxxxxx",
"juicefs.accesskey": "xxxxxxxxxxxxx",
"juicefs.secretkey": "xxxxxxxxxxxxx"
}