spark reading data from mysql in parallel -

-

im trying read data mysql , write parquet file in s3 specific partitions follows:

df=sqlcontext.read.format('jdbc')\    .options(driver='com.mysql.jdbc.driver',url="""jdbc:mysql://<host>:3306/<>db?user=<usr>&password=<pass>""",          dbtable='tbl',          numpartitions=4 )\    .load()   df2=df.withcolumn('updated_date',to_date(df.updated_at)) df2.write.parquet(path='s3n://parquet_location',mode='append',partitionby=['updated_date'])

my problem open 1 connection mysql (instead of 4) , doesn't write parquert until fetches data mysql, because table in mysql huge (100m rows) process failed on outofmemory.

is there way configure spark open more 1 connection mysql , write partial data parquet?

you should set these properties:

partitioncolumn,  lowerbound,  upperbound,  numpartitions

as documented here: http://spark.apache.org/docs/latest/sql-programming-guide.html#jdbc-to-other-databases


Comments

Post a Comment

Popular posts from this blog

Hatching array of circles in AutoCAD using c# -

-
i beginner in autocad plugins .i have created array of circle using following code. now, have hatch circles in array .how that??how hatch circles @ once? moreover, how can return hatch method , use hatch objects.sorry language.thanks in advance. [assembly: commandclass(typeof(beamsection.class1))] namespace beamsection { public class class1 { //class1 obj = new class1(); [commandmethod("beamatsupport")] public void addlightweightpolyline() { // current document , database document acdoc = application.documentmanager.mdiactivedocument; database accurdb = acdoc.database; promptpointresult ppr = acdoc.editor.getpoint("\nselect starting point "); var ucs = acdoc.editor.currentusercoordinatesystem; point3d startingpt = ppr.value.transformby(ucs); list<double> radius = new list<double>(); radius.add(0.5); ...
Read more

ios - UITEXTFIELD InputView Uipicker not working in swift -

-
hello trying pop uipickerview view programmatically when user clicks on textfield. have tried doesn't doing anything. isn't working. nothing happening when click textfield class userprofiletableviewcontroller: uitableviewcontroller,uitextfielddelegate,uipickerviewdelegate,uipickerviewdatasource { var itempicker: uipickerview! = uipickerview() @iboutlet weak var gendertxtfield: uitextfield! var gender = ["male","female"] override func viewdidload() { super.viewdidload() gendertxtfield.delegate = self itempicker!.delegate = self itempicker!.datasource = self itempicker!.backgroundcolor = uicolor.blackcolor() self.gendertxtfield.inputview = itempicker } func numberofcomponentsinpickerview(pickerview: uipickerview) -> int{ return 1 } // returns # of rows in each component.. func pickerview(pickerview: uipickerview, numberofrowsincomponent component:...
Read more