scala - How does RDD.aggregate action work (i.e. how to understand the parameters)? -


can give detailed explanation of how below agrregate action in spark produces result of (9,4)

val rdd = sc.parallelize(list(1,2,3,3))  rdd.aggregate((0, 0)) ((x, y) => (x._1 + y, x._2 + 1), (x, y) => (x._1 + y._1, x._2 + y._2))  res : (9,4)

this spark 2.1.0 here (which should not matter much, but...)

go the official documentation of aggregate (aka scaladoc) , read:

aggregate elements of each partition, , results partitions, using given combine functions , neutral "zero value". function can return different result type, u, type of rdd, t. thus, need 1 operation merging t u , 1 operation merging 2 u's, in scala.traversableonce. both of these functions allowed modify , return first argument instead of creating new u avoid memory allocation.

the signature follows (removed implicit parameter not particularly interesting):

aggregate[u](zerovalue: u)(seqop: (u, t) ⇒ u, combop: (u, u) ⇒ u): u

the scaladoc says:

zerovalue initial value accumulated result of each partition seqop operator, , initial value combine results different partitions combop operator - typically neutral element (e.g. nil list concatenation or 0 summation)

in case, zerovalue (0, 0).

seqop operator used accumulate results within partition

in case, seqop (x, y) => (x._1 + y, x._2 + 1) function accepting 2 pairs, unfortunately named x , y (which i'd call p1 , p2 @ least or using pattern matching , partial function, i.e. case ((x1, y1), (x2, y2)) => ...).

given you've got n partitions (you can check out using rdd.getnumpartition), seqop going called n times.

the scaladoc says:

combop associative operator used combine results different partitions

which means combop combine results of seqop , apply function:

(x, y) => (x._1 + y._1, x._2 + y._2)

it's again badly written see , i'd call a noise. i'd write function follows:

{ case ((x1, y1), (x2, y2)) => (x1 + x2, y1 + y2) }

follow types , give proper names, , in scala becomes easier ;-)


Comments

Post a Comment

Popular posts from this blog

'hasOwnProperty' in javascript -

i use obj.hasownproperty judge whether object has property, when replaced obj[prop] !== undefined , not normal implementation, ask, why behind method can not use it? object.hasownproperty(prop); object[prop] !== undefined; obj[prop] !== undefined wrong 2 reasons: you can explicitly set property undefined , obj[prop] = undefined; . obj.hasownproperty(prop) return true in case. obj[prop] follow prototype chain, return property that's inherited. obj.hasownproperty(prop) returns true if property exists directly in object, returns false inherited properties.
Read more

How to understand 2 main() functions after using uftrace to profile the C++ program? -

i trying uftrace profile following simple c++ program: #include <iostream> class { public: a() {std::cout << "a created" << std::endl;} ~a() {std::cout << "a destroyed" << std::endl;} }; int main() { a; return 0; } the profile result this: # uftrace a.out created destroyed # duration tid function 2.026 [ 4828] | __cxa_atexit(); [ 4828] | main() { [ 4828] | __static_initialization_and_destruction_0() { 89.397 [ 4828] | std::ios_base::init::init(); 0.768 [ 4828] | __cxa_atexit(); 93.029 [ 4828] | } /* __static_initialization_and_destruction_0 */ 94.425 [ 4828] | } /* main */ [ 4828] | main() { [ 4828] | a::a() { 11.104 [ 4828] | std::operator <<(); 10.825 [ 4828] | std::basic_ostream::operator <<(); 24.514 [ 4828] | } /* a::a */ [ 4828] | a::~a() { 0.978 [ 4828] | ...
Read more

android - Unable to generate FCM token from dynamically instantiated Firebase -

our business logic requires instantiate firebase during runtime. fetch firebase credentials (app id, api key etc.) default firebase location after knowing user's public key , create firebase instance using credentials. this means there 2 firebase instances used within app: the default "index" firebase gives credentials second the actual firebase intend use point forward the second firebase initialised this: firebaseapp app = firebaseapp.initializeapp(<context>, <options>, <app_label>); our problem traditional method of retrieving fcm token using firebaseinstanceidservice , ontokenrefresh() fails since ontokenrefresh() method called first firebase instance. directly calling string token = firebaseinstanceid.getinstance(app).gettoken(); returns null never ready when called. have tried polling test if token generated @ point. no luck. how can generate fcm token reliably firebase instance instantiated during runtime? i...
Read more