pyspark - How to know how many files spark created after saving data frame -


i have data frame , im saving csv file databricks.spark.csv using save function on dataframe. how can know how many files spark created (spark dividing files automatically)

+1 anshul's comment, can use getnumpartitions number of partitions of rdd, , number of file number.

btw,why need know saved file number?


-

Comments

Post a Comment

Popular posts from this blog

javascript - How do you prevent nested queries/catches in sequelize? -

i think i'm preventing nested queries as possible, i'm not sure. understand calls here can executed in single select query, did simplify example. // example in typescript // find user user.find({where:{username:'user'}}) // if found user .then(function(user) { return user.find({where:{username:'other_user'}}) // if found other_user .then(function(other_user) { // stuff return whatever_i_need } // if went wrong, go straight parent catch .catch(function(err) { // stuff throw new error() } } // if previous .then() returned success .then(function(data) { return user.find({where:{username:'yet_another_user'}}) // if found yet_another_user .then(function(yet_another_user) { // stuff return whatever_i_need_again ...
Read more

ruby on rails - ActiveRecord order not working -

in controller, after reprioritizing "child", lists children in new order: @child.parent.children.sort_by{|g| g.priority } this, surprisingly, doesn't work: @child.parent.children.order(priority: :asc) why doesn't .order work? instead of sorting correctly, it's sorting previous order before operation run, assume it's using cached results (the query run before in operation). how 1 bust cache, if indeed problem? i've tried @child.reload after reprioritizing no avail. if have ordering on children association might try use reorder apply new ordering: @child.parent.children.reorder(priority: :asc)
Read more

java - ConcurrentModificationException upon committing transaction with Hibernate -

in our application have upgraded hibernate 3.5.6-final 4.2.21.final , getting concurrentmodificationexception when database transaction committed: java.util.concurrentmodificationexception: null @ java.util.arraylist$itr.checkforcomodification(arraylist.java:901) @ java.util.arraylist$itr.next(arraylist.java:851) @ org.hibernate.engine.spi.actionqueue.executeactions(actionqueue.java:386) @ org.hibernate.engine.spi.actionqueue.executeactions(actionqueue.java:304) @ org.hibernate.event.internal.abstractflushingeventlistener.performexecutions(abstractflushingeventlistener.java:349) @ org.hibernate.event.internal.defaultflusheventlistener.onflush(defaultflusheventlistener.java:56) @ org.hibernate.internal.sessionimpl.flush(sessionimpl.java:1195) @ org.hibernate.internal.sessionimpl.managedflush(sessionimpl.java:404) @ org.hibernate.engine.transaction.internal.jdbc.jdbctransaction.beforetransactioncommit(jdbctransaction.java:101) @ org.hibernat...
Read more