背景
最近在写hiveUDF的时候，遇到了一些反序列的问题，具体的报错如下：
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 11 in stage 6.0 failed 4 times, most recent failure: Lost task 11.3 in stage 6.0 (TID 105) (dw-csprd-bigdata-athena-dn-096.shizhuang-inc.com executor 3): com.esotericsoftware.kryo.KryoException: Unable to find class: scala.collection.immutable.HashMap$$$Lambda$9/282828951
Serialization trace:
mergef$1 (scala.collection.immutable.HashMap$$anon$1)
defaultMerger (scala.collection.immutable.HashMap$)
_2 (scala.Tuple2)
head (scala.collection.immutable.$colon$colon)
factories (com.fasterxml.jackson.module.scala.deser.UnsortedMapDeserializerModule$$anon$1)
_additionalDeserializers (com.fasterxml.jackson.databind.cfg.DeserializerFactoryConfig)
_factoryConfig (com.fasterxml.jackson.databind.deser.BeanDeserializerFactory)
_factory (com.fasterxml.jackson.databind.deser.DefaultDeserializationContext$Impl)
_deserializationContext (com.fasterxml.jackson.databind.ObjectMapper)
objectMapper (xxx.xxxUDF)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:160)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:181)
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
分析
我们的代码类似如下：
class xxxUDF extends GenericUDF {
@transient
var argumentOIs: Array[ObjectInspector] = _
val objectMapper = new ObjectMapper()
objectMapper.registerModule(DefaultScalaModule)
objectMapper.configure(Feature.ALLOW_UNQUOTED_FIELD_NAMES, true)
objectMapper.configure(Feature.ALLOW_SINGLE_QUOTES, true)
val result: Text = new Text()
其中spark的配置是使用kryo序列化，spark.serializer=org.apache.spark.serializer.KryoSerializer
可以看到是objectMapper类在driver端在把UDF传给executor的时候，需要做UDF的序列化，而序列化的时候，就会把objectMapper字段进行序列化，
这样在executor端进行task.run的时候会把 objectMapper反序列化出来，这个时候如果对应的类的成员方法如果没有进行kryo的注册，就会直接报序列化的错误，
而spark目前的默认注册的kryo类在KryoSerializer.scala中，如下:
...
kryo.register(None.getClass)
kryo.register(Nil.getClass)
kryo.register(Utils.classForName("scala.collection.immutable.$colon$colon"))
kryo.register(Utils.classForName("scala.collection.immutable.Map$EmptyMap$"))
kryo.register(classOf[ArrayBuffer[Any]])
...
解决
在objectMapper加上@transient注解，使该对象不被序列化，这样在反序列化的时候，就不会反序列化该对象
