spark streaming的程序如果运行方式是yarn client mode，那么如何指定driver和executor的log4j配置文件？
Driver
添加参数--driver-java-options
spark-submit --driver-java-options "-Dlog4j.configuration=file:/data1/conf/log4j-driver.properties"
Executor
由于executor是运行在yarn的集群中的，所以先要将配置文件通过--files上传
spark-submit --files /data1/conf/log4j.properties --conf spark.executor.extraJavaOptions="-Dlog4j.configuration=log4j.properties"
在log4j.properties中要注意配置spark.yarn.app.container.log.dir例如
log4j.rootLogger=INFO, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.append=true
log4j.appender.file.file=${spark.yarn.app.container.log.dir}/stdout
log4j.appender.file.MaxFileSize=256MB
log4j.appender.file.MaxBackupIndex=20
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %p [%t] %c{1}:%L - %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
这样就可以在spark的Web UI中直接查看日志
其他
如果是通过java -cp命令运行自己的jar包，可以通过下面的方式添加log4j的配置
java -cp -Dlog4j.configuration=file:${APP_HOME}/conf/log4j.properties
