python - pyspark on cluster, make sure all nodes are used -

-

deployment info: "pyspark --master yarn-client --num-executors 16 --driver-memory 16g --executor-memory 2g "

i turning 100,000 line text file (in hdfs dfs format) rdd object corpus = sc.textfile("my_file_name"). when execute corpus.count() 100000. realize these steps performed on master node.

now, question when perform action new_corpus=corpus.map(some_function), job automatically distributed pyspark among available slaves (16 in case)? or have specify something?

notes:

  • i don't think gets distributed (or @ least not on 16 nodes) because when new_corpus.count(), prints out [stage some_number:> (0+2)/2], not [stage some_number:> (0+16)/16]
  • i don't think doing corpus = sc.textfile("my_file_name",16) solution me because function want apply works @ line level , therefore should applied 100,000 times (the goal of parallelization speed process, having each slave taking 100000/16 lines). should not applied 16 times on 16 subsets of original text file.

your observations not correct. stages not "executors". in spark have jobs, tasks , stages. job kicked off master driver , task assigned different worker nodes stage collection of task has same shuffling dependencies. in case shuffling happens once.

to check if executors 16, have resource manager. @ port 4040 since using yarn.

also if use rdd.map(), should parallelize according defined partitions , not executors set in sc.textfile("my_file_name", numpartitions).

here overview again: https://spark.apache.org/docs/1.6.0/cluster-overview.html


Comments

Post a Comment

Popular posts from this blog

Hatching array of circles in AutoCAD using c# -

-
i beginner in autocad plugins .i have created array of circle using following code. now, have hatch circles in array .how that??how hatch circles @ once? moreover, how can return hatch method , use hatch objects.sorry language.thanks in advance. [assembly: commandclass(typeof(beamsection.class1))] namespace beamsection { public class class1 { //class1 obj = new class1(); [commandmethod("beamatsupport")] public void addlightweightpolyline() { // current document , database document acdoc = application.documentmanager.mdiactivedocument; database accurdb = acdoc.database; promptpointresult ppr = acdoc.editor.getpoint("\nselect starting point "); var ucs = acdoc.editor.currentusercoordinatesystem; point3d startingpt = ppr.value.transformby(ucs); list<double> radius = new list<double>(); radius.add(0.5); ...
Read more

ios - UITEXTFIELD InputView Uipicker not working in swift -

-
hello trying pop uipickerview view programmatically when user clicks on textfield. have tried doesn't doing anything. isn't working. nothing happening when click textfield class userprofiletableviewcontroller: uitableviewcontroller,uitextfielddelegate,uipickerviewdelegate,uipickerviewdatasource { var itempicker: uipickerview! = uipickerview() @iboutlet weak var gendertxtfield: uitextfield! var gender = ["male","female"] override func viewdidload() { super.viewdidload() gendertxtfield.delegate = self itempicker!.delegate = self itempicker!.datasource = self itempicker!.backgroundcolor = uicolor.blackcolor() self.gendertxtfield.inputview = itempicker } func numberofcomponentsinpickerview(pickerview: uipickerview) -> int{ return 1 } // returns # of rows in each component.. func pickerview(pickerview: uipickerview, numberofrowsincomponent component:...
Read more