java - Colons in Apache Spark application path -


i'm submitting apache spark application yarn programmatically:

package application.restapplication;  import org.apache.hadoop.conf.configuration; import org.apache.spark.sparkconf; import org.apache.spark.deploy.yarn.client; import org.apache.spark.deploy.yarn.clientarguments;  public class app {     public static void main(string[] args1) {         string[] args = new string[] {                 "--class", "org.apache.spark.examples.javawordcount",                 "--jar", "/opt/spark/examples/jars/spark-examples_2.11-2.0.0.jar",                 "--arg", "hdfs://hadoop-master:9000/input/file.txt"         };         configuration config = new configuration();         system.setproperty("spark_yarn_mode", "true");         sparkconf sparkconf = new sparkconf();         clientarguments cargs = new clientarguments(args);         client client = new client(cargs, config, sparkconf);         client.run();     } }

i have problem line: "--arg", "hdfs://hadoop-master:9000/input/file.txt" - more colons:

16/08/29 09:54:16 error yarn.applicationmaster: uncaught exception:  java.lang.numberformatexception: input string: "9000/input/plik2.txt"     @ java.lang.numberformatexception.forinputstring(numberformatexception.java:65)     @ java.lang.integer.parseint(integer.java:580)     @ java.lang.integer.parseint(integer.java:615)     @ scala.collection.immutable.stringlike$class.toint(stringlike.scala:272)     @ scala.collection.immutable.stringops.toint(stringops.scala:29)     @ org.apache.spark.util.utils$.parsehostport(utils.scala:935)     @ org.apache.spark.deploy.yarn.applicationmaster.waitforsparkdriver(applicationmaster.scala:547)     @ org.apache.spark.deploy.yarn.applicationmaster.runexecutorlauncher(applicationmaster.scala:405)     @ org.apache.spark.deploy.yarn.applicationmaster.run(applicationmaster.scala:247)     @ org.apache.spark.deploy.yarn.applicationmaster$$anonfun$main$1.apply$mcv$sp(applicationmaster.scala:749)     @ org.apache.spark.deploy.sparkhadooputil$$anon$1.run(sparkhadooputil.scala:71)     @ org.apache.spark.deploy.sparkhadooputil$$anon$1.run(sparkhadooputil.scala:70)     @ java.security.accesscontroller.doprivileged(native method)     @ javax.security.auth.subject.doas(subject.java:422)     @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1657)     @ org.apache.spark.deploy.sparkhadooputil.runassparkuser(sparkhadooputil.scala:70)     @ org.apache.spark.deploy.yarn.applicationmaster$.main(applicationmaster.scala:747)     @ org.apache.spark.deploy.yarn.executorlauncher$.main(applicationmaster.scala:774)     @ org.apache.spark.deploy.yarn.executorlauncher.main(applicationmaster.scala)

how write (as argument) path file colons? try various combinations slashes, backslashes, %3a, etc...

according utils#parsehostport invoked during call, spark seems consider port text behind last : :

def parsehostport(hostport: string): (string, int) = {     // check cache first.     val cached = hostportparseresults.get(hostport)     if (cached != null) {       return cached     }      val indx: int = hostport.lastindexof(':')     // potentially broken - when dealing ipv6 addresses example, sigh ...     // hadoop not support ipv6 right now.     // now, assume if port exists, valid - not check if int > 0     if (-1 == indx) {       val retval = (hostport, 0)       hostportparseresults.put(hostport, retval)       return retval     }      val retval = (hostport.substring(0, indx).trim(), hostport.substring(indx + 1).trim().toint)     hostportparseresults.putifabsent(hostport, retval)     hostportparseresults.get(hostport) }

as consequence, whole string 9000/input/file.txt supposed single port number. suggests not supposed refer input file hdfs file system. guess more skilled in apache spark give better advice.


-

Comments

Post a Comment

Popular posts from this blog

amazon web services - S3 Pre-signed POST validate file type? -

i know it's possible put limit on file size content-length-range header. possible validate file type? https://docs.aws.amazon.com/amazons3/latest/dev/httppostforms.html#policyconditions i see there content-type header, if set say, audio/mp3 allow mp3 files , return error if file not mp3? i found previous question answers mention validating file size: s3 direct upload restricting file size , type you can specify content-type @ post request. you can specify @ signature, post must done content-type: { "expiration": "2007-12-01t12:00:00.000z", "conditions": [ {"acl": "public-read" }, {"bucket": "johnsmith" }, {"content-type: "audio/mp3"} ] } creating html form (using aws signature version 4) edit: @ previous question found, checking content-type.
Read more

c# - Check Keyboard Input Winforms -

i wondering if instead of doing this protected override void onkeydown(keyeventargs e) { if (e.keycode == keys.a) console.writeline("the key down."); } i set bool method , this: if(keydown(keys.a)) // whatever here i've been sitting here ages trying figure out how it. can't wrap head around it. in case wondering, plan call bool inside different method, check input. since want perform action after pressing key, using keydown event enough. but in cases suppose want check if specific key down in middle of process, can use getkeystate method way: [dllimport("user32.dll", charset = charset.auto, exactspelling = true)] public static extern short getkeystate(int keycode); public const int key_pressed = 0x8000; public static bool iskeydown(keys key) { return convert.toboolean(getkeystate((int)key) & key_pressed); } you should know, each time check key state using example iskeydown(keys.a) method returns true if ke...
Read more