java - launch and submit job spark -


i try "run" spark jobs width java application, searching, found following 2 methods: clientsarguments , sparklauncher.

could explain me difference between two? difference between launch , submit job/application spark? thank you.

sparklauncher wrapper library spark-submit, coverts sparklauncher code spark-submit script , trigger jobs.

the mechanism same spark-submit script, if @ source code of sparklauncher, uses processbuilder construct shell.

if want use sparklauncher, need specify $java_home, $sprak_home , other essential parameters. there limitation sparklauncher, machine sparklauncher runs must have $java_home , $spark_home(spark library) used sparklauncher locating script , related dependencies. soft of impossible cloud environment cloudfoundry etc.

you assume sparklauncher equals spark-submit script, choose client or master, local or yarn mode.

clientsarguments class yarn script, works yarn-mode.


-

Comments

Post a Comment

Popular posts from this blog

loops - Spock: How to use test data with @Stepwise -

i used @stepwise automation. 8 test methods running sequentially complete process. 1 of methods take parameter , want pass different parameters method. but problem is: method takes first set of parameter, parameters processed and instead of proceeding next method, method takes next set of parameter. the source looks like: @stepwise class aosearchpagetest extends gebreportingspec { def "first_method"() { } def "second_method"() { when: @ particularpage then: process(area) process(coutntry) process(airport) process(dest_area) process(dest_coutntry) process(dest_airport) where: area << ["asia","europe"] country << ["japan","uk"] port << ["narita","london"] dest_area << ["asia","europe"] dest_country << ["k...
Read more

amazon web services - S3 Pre-signed POST validate file type? -

i know it's possible put limit on file size content-length-range header. possible validate file type? https://docs.aws.amazon.com/amazons3/latest/dev/httppostforms.html#policyconditions i see there content-type header, if set say, audio/mp3 allow mp3 files , return error if file not mp3? i found previous question answers mention validating file size: s3 direct upload restricting file size , type you can specify content-type @ post request. you can specify @ signature, post must done content-type: { "expiration": "2007-12-01t12:00:00.000z", "conditions": [ {"acl": "public-read" }, {"bucket": "johnsmith" }, {"content-type: "audio/mp3"} ] } creating html form (using aws signature version 4) edit: @ previous question found, checking content-type.
Read more