本地环境IDEA, 但把Jar包上传到Spark集群的时候, Job会失败. 错误信息
2017-03-29 18:12:48,190 ERROR [Driver] yarn.ApplicationMaster (Logging.scala:logError(95)) - User class threw exception: java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:133)
at com.dtise.common.elasticsearch.ESClientSingleton.getClientWithHost(ESClientSingleton.java:39)
at com.dtise.common.elasticsearch.ESClientSingleton.getTestClient(ESClientSingleton.java:29)
at com.dtise.common.elasticsearch.ESClientProvider.getTestClient(ESClientProvider.java:22)
at com.dtise.data.batch.bigtable.ElasticHelper$.createIndexWithMapping(ElasticHelper.scala:70)
at com.dtise.data.batch.bigtable.ElasticHelper$.importToEs(ElasticHelper.scala:19)
at com.dtise.data.batch.bigtable.BigTablePipeline$.start(BigTablePipeline.scala:22)
at com.dtise.data.batch.bigtable.BigTableGo$.main(BigTableGo.scala:57)
at com.dtise.data.batch.bigtable.BigTableGo.main(BigTableGo.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
- 发现是Guava版本的问题, 运行时Guava的版本低.
- 我们要的是18.0的版本, 于是将pom依赖中所有低于18.0的都exclude掉, 但还是不行.
关于查看依赖, 用的以下两种方法:- 可以用
mvn dependency:tree -Dverbose | grep guava查看, 这个方便一下子搜索出来. 记得加上-Dverbose - 用IDEA的话, 通过
Maven Projecttab里面的Show Dependencies,可以看到图形化的包依赖.可以直接exclude不合的包.
- 可以用
- 看cdh上oozie job的log, 有这么一句
guava-14.0.1.jar -> resource { scheme: "hdfs" host: "nameservice1" port: -1 file: "/user/oozie/share/lib/lib_20170216215144/spark/guava-14.0.1.jar" } size: 2189117 timestamp: 1487253111175 type: FILE visibility: PUBLIC- 在/user/oozie/share/lib/lib_20170216215144/spark里面找到一个guava-14.0.1.jar, 将其改名, 并上传一个guava-18.0.jar
- 但还是不行, oozie里面记住了之前的版本, 还是报错找不到guava-14.0.1.jar. 为什么oozie会记住用某个版本的jar还有待调查.
- 将guava-18.0.jar改名成guava-14.0.1.jar. 问题解决.
