pyspark - How to know how many files spark created after saving data frame -


i have data frame , im saving csv file databricks.spark.csv using save function on dataframe. how can know how many files spark created (spark dividing files automatically)

+1 anshul's comment, can use getnumpartitions number of partitions of rdd, , number of file number.

btw,why need know saved file number?


-

Comments

Post a Comment

Popular posts from this blog

How to use SUM() in MySQL for calculated values -

in mysql, dont know how use function sum calculated values. could me please? in code below, latest part. thing want add row @ end, summarizing "calculado" column thank in advance select p.id_order ref, concat (di.firstname,' ',di.lastname) nombre, ca.weight peso, if (ca.weight<1,(1), if (ca.weight>=1 , (ca.weight<3),(2), if (ca.weight>=3 , (ca.weight<5),(3), if (ca.weight>=5 , (ca.weight<8),(4), if (ca.weight>=8 , (ca.weight<10),(5), if (ca.weight>=10 , (ca.weight<15),(6), if (ca.weight>=15 , (ca.weight<20),(7), if (ca.weight>=20,(ca.weight*(0.5)),'error' )))))))) calculado, p.invoice_date fecha ps_orders p left join ps_address di on (p.id_address_delivery = di.id_address) left join ps_order_carrier ca on (p.id_order = ca.id_order) date(p.invoice_date) >= concat(date_format(last_day(now...
Read more

ios - IBOutlet for image button not correctly referencing button after recreating outlet -

i have custom button image. have iboutlet reference in viewcontroller.swift file. the button working hooked wrong outlet. removed of outlets , actions button , similar buttons , remade them. @ point stopped working. i first tried change image button code: star1backbuttonreference.setimage(uiimage(named: "starfull_48dp.png"), forstate: this didn't anything. after thinking problem code changing image decided check few things. so tried hide button this: star1backbuttonreference.hidden = true this did not either think there problem iboutlet reference star1backbuttonreference , found instance change/hide image , reference. the reference this: @iboutlet var star1backbuttonreference: uibutton! i checked referencing outlet on storyboard , correct (one side viewcontroller other side star1backbuttonreference). there must mistake have done in connecting outlet or deleting old one. if deleting referencing outlet must remove it's reference conn...
Read more

java - AEM: 403 Forbidden occurs when call a Post servlet -

my problem similar this: cq5: 403 forbidden occurs when call post servlet in aem 6.1 according accepted answer of above topic, must remove post apache sling referrer filter. wonder if action harmful system? , have better way fix issue? p/s: sorry english. if testing code on author mode, 403 forbidden error request. requires csrf token. or can declare dependency granite.csrf.standalone before can use framework. on publish should work fine.
Read more