hadoop - Getting java.lang.NoSuchFieldError: INT_8 error while running spark job through oozie -

-

i getting java.lang.nosuchfielderror: int_8 error when trying execute spark job using oozie on cloudera 5.5.1 version. on appreciated.

please find error stackstrace below.

16/01/28 11:21:17 warn tasksetmanager: lost task 0.2 in stage 20.0 (tid 40, zlab-physrv1): java.lang.nosuchfielderror: int_8     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter.convertfield(catalystschemaconverter.scala:327)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter.convertfield(catalystschemaconverter.scala:312)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter$$anonfun$convertfield$1.apply(catalystschemaconverter.scala:517)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter$$anonfun$convertfield$1.apply(catalystschemaconverter.scala:516)     @ scala.collection.indexedseqoptimized$class.foldl(indexedseqoptimized.scala:51)     @ scala.collection.indexedseqoptimized$class.foldleft(indexedseqoptimized.scala:60)     @ scala.collection.mutable.arrayops$ofref.foldleft(arrayops.scala:108)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter.convertfield(catalystschemaconverter.scala:516)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter.convertfield(catalystschemaconverter.scala:312)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter.convertfield(catalystschemaconverter.scala:521)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter.convertfield(catalystschemaconverter.scala:312)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter$$anonfun$convert$1.apply(catalystschemaconverter.scala:305)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter$$anonfun$convert$1.apply(catalystschemaconverter.scala:305)     @ scala.collection.traversablelike$$anonfun$map$1.apply(traversablelike.scala:244)     @ scala.collection.traversablelike$$anonfun$map$1.apply(traversablelike.scala:244)     @ scala.collection.iterator$class.foreach(iterator.scala:727)     @ scala.collection.abstractiterator.foreach(iterator.scala:1157)     @ scala.collection.iterablelike$class.foreach(iterablelike.scala:72)     @ org.apache.spark.sql.types.structtype.foreach(structtype.scala:92)     @ scala.collection.traversablelike$class.map(traversablelike.scala:244)     @ org.apache.spark.sql.types.structtype.map(structtype.scala:92)     @ org.apache.spark.sql.execution.datasources.parquet.catalystschemaconverter.convert(catalystschemaconverter.scala:305)     @ org.apache.spark.sql.execution.datasources.parquet.parquettypesconverter$.convertfromattributes(parquettypesconverter.scala:58)     @ org.apache.spark.sql.execution.datasources.parquet.rowwritesupport.init(parquettablesupport.scala:55)     @ parquet.hadoop.parquetoutputformat.getrecordwriter(parquetoutputformat.java:277)     @ parquet.hadoop.parquetoutputformat.getrecordwriter(parquetoutputformat.java:251)     @ org.apache.spark.sql.execution.datasources.parquet.parquetoutputwriter.<init>(parquetrelation.scala:94)     @ org.apache.spark.sql.execution.datasources.parquet.parquetrelation$$anon$3.newinstance(parquetrelation.scala:272)     @ org.apache.spark.sql.execution.datasources.defaultwritercontainer.writerows(writercontainer.scala:233)     @ org.apache.spark.sql.execution.datasources.insertintohadoopfsrelation$$anonfun$run$1$$anonfun$apply$mcv$sp$3.apply(insertintohadoopfsrelation.scala:150)     @ org.apache.spark.sql.execution.datasources.insertintohadoopfsrelation$$anonfun$run$1$$anonfun$apply$mcv$sp$3.apply(insertintohadoopfsrelation.scala:150)     @ org.apache.spark.scheduler.resulttask.runtask(resulttask.scala:66)     @ org.apache.spark.scheduler.task.run(task.scala:88)     @ org.apache.spark.executor.executor$taskrunner.run(executor.scala:214)     @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145)     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615)     @ java.lang.thread.run(thread.java:745)

as per idea used error when ever there difference on jars have used generate code , jars have used currently.

note: when trying submit same 1 using spark-submit command it's running fine.

regards nisith

finally able debug , fix issue. issue installation 1 of data nodes having older version of parquet jars(5.2 cdh distribution). after replacing jars current version jars working fine.


Comments

Post a Comment

Popular posts from this blog

Hatching array of circles in AutoCAD using c# -

-
i beginner in autocad plugins .i have created array of circle using following code. now, have hatch circles in array .how that??how hatch circles @ once? moreover, how can return hatch method , use hatch objects.sorry language.thanks in advance. [assembly: commandclass(typeof(beamsection.class1))] namespace beamsection { public class class1 { //class1 obj = new class1(); [commandmethod("beamatsupport")] public void addlightweightpolyline() { // current document , database document acdoc = application.documentmanager.mdiactivedocument; database accurdb = acdoc.database; promptpointresult ppr = acdoc.editor.getpoint("\nselect starting point "); var ucs = acdoc.editor.currentusercoordinatesystem; point3d startingpt = ppr.value.transformby(ucs); list<double> radius = new list<double>(); radius.add(0.5);
Read more

ios - UITEXTFIELD InputView Uipicker not working in swift -

-
hello trying pop uipickerview view programmatically when user clicks on textfield. have tried doesn't doing anything. isn't working. nothing happening when click textfield class userprofiletableviewcontroller: uitableviewcontroller,uitextfielddelegate,uipickerviewdelegate,uipickerviewdatasource { var itempicker: uipickerview! = uipickerview() @iboutlet weak var gendertxtfield: uitextfield! var gender = ["male","female"] override func viewdidload() { super.viewdidload() gendertxtfield.delegate = self itempicker!.delegate = self itempicker!.datasource = self itempicker!.backgroundcolor = uicolor.blackcolor() self.gendertxtfield.inputview = itempicker } func numberofcomponentsinpickerview(pickerview: uipickerview) -> int{ return 1 } // returns # of rows in each component.. func pickerview(pickerview: uipickerview, numberofrowsincomponent component:
Read more

Python Pig Latin Translator -

-
so have been working on @ codecademy.com while trying learn python. don't know if blind , can't see why won't work or what. here's code won't let me use. function print variable if had typed in , print "empty" if didn't type anything. print "welcome english pig latin translator!" original = raw_input("what's name?") if len(original) > 0 print original else print "empty" if len(original) > 0 else the above lines contains syntax errors. try: if len(original) > 0: else: note added colons, ( : ).
Read more