pyspark - How to know how many files spark created after saving data frame -


i have data frame , im saving csv file databricks.spark.csv using save function on dataframe. how can know how many files spark created (spark dividing files automatically)

+1 anshul's comment, can use getnumpartitions number of partitions of rdd, , number of file number.

btw,why need know saved file number?


-

Comments

Post a Comment

Popular posts from this blog

loops - Spock: How to use test data with @Stepwise -

i used @stepwise automation. 8 test methods running sequentially complete process. 1 of methods take parameter , want pass different parameters method. but problem is: method takes first set of parameter, parameters processed and instead of proceeding next method, method takes next set of parameter. the source looks like: @stepwise class aosearchpagetest extends gebreportingspec { def "first_method"() { } def "second_method"() { when: @ particularpage then: process(area) process(coutntry) process(airport) process(dest_area) process(dest_coutntry) process(dest_airport) where: area << ["asia","europe"] country << ["japan","uk"] port << ["narita","london"] dest_area << ["asia","europe"] dest_country << ["k...
Read more

amazon web services - S3 Pre-signed POST validate file type? -

i know it's possible put limit on file size content-length-range header. possible validate file type? https://docs.aws.amazon.com/amazons3/latest/dev/httppostforms.html#policyconditions i see there content-type header, if set say, audio/mp3 allow mp3 files , return error if file not mp3? i found previous question answers mention validating file size: s3 direct upload restricting file size , type you can specify content-type @ post request. you can specify @ signature, post must done content-type: { "expiration": "2007-12-01t12:00:00.000z", "conditions": [ {"acl": "public-read" }, {"bucket": "johnsmith" }, {"content-type: "audio/mp3"} ] } creating html form (using aws signature version 4) edit: @ previous question found, checking content-type.
Read more