hive - Spark partitions - using DISTRIBUTE BY option -


we have spark environment should process 50mm rows. these rows contains key column. unique number of keys close 2000. process of 2000 keys in parallel. using spark sql following

hivecontext.sql("select * bigtbl distribute key")

subsequently have mappartitions works nicely on partitions in parallel. trouble is, creates 200 partitions default. using command following able increase partitions

hivecontext.sql("set spark.sql.shuffle.partitions=500");

however during real production run not know number of unique keys. want auto managed. there way please.

thanks

bala

i suggest use "repartition" function , register repartitioned new temp table , further cache faster processing.

val distinctvalues = hivecontext.sql("select key bigtbl").distinct().count() // find count distinct values   hivecontext.sql("select * bigtbl distribute key")        .repartition(distinctvalues.toint) // repartition number of distinct values        .registertemptable("newbigtbl") // register repartitioned table temp table  hivecontext.cachetable("newbigtbl") // cache repartitioned table improving query performance

for further queries use "newbigtbl"


Comments

Post a Comment

Popular posts from this blog

'hasOwnProperty' in javascript -

i use obj.hasownproperty judge whether object has property, when replaced obj[prop] !== undefined , not normal implementation, ask, why behind method can not use it? object.hasownproperty(prop); object[prop] !== undefined; obj[prop] !== undefined wrong 2 reasons: you can explicitly set property undefined , obj[prop] = undefined; . obj.hasownproperty(prop) return true in case. obj[prop] follow prototype chain, return property that's inherited. obj.hasownproperty(prop) returns true if property exists directly in object, returns false inherited properties.
Read more

How to understand 2 main() functions after using uftrace to profile the C++ program? -

i trying uftrace profile following simple c++ program: #include <iostream> class { public: a() {std::cout << "a created" << std::endl;} ~a() {std::cout << "a destroyed" << std::endl;} }; int main() { a; return 0; } the profile result this: # uftrace a.out created destroyed # duration tid function 2.026 [ 4828] | __cxa_atexit(); [ 4828] | main() { [ 4828] | __static_initialization_and_destruction_0() { 89.397 [ 4828] | std::ios_base::init::init(); 0.768 [ 4828] | __cxa_atexit(); 93.029 [ 4828] | } /* __static_initialization_and_destruction_0 */ 94.425 [ 4828] | } /* main */ [ 4828] | main() { [ 4828] | a::a() { 11.104 [ 4828] | std::operator <<(); 10.825 [ 4828] | std::basic_ostream::operator <<(); 24.514 [ 4828] | } /* a::a */ [ 4828] | a::~a() { 0.978 [ 4828] | ...
Read more

android - Unable to generate FCM token from dynamically instantiated Firebase -

our business logic requires instantiate firebase during runtime. fetch firebase credentials (app id, api key etc.) default firebase location after knowing user's public key , create firebase instance using credentials. this means there 2 firebase instances used within app: the default "index" firebase gives credentials second the actual firebase intend use point forward the second firebase initialised this: firebaseapp app = firebaseapp.initializeapp(<context>, <options>, <app_label>); our problem traditional method of retrieving fcm token using firebaseinstanceidservice , ontokenrefresh() fails since ontokenrefresh() method called first firebase instance. directly calling string token = firebaseinstanceid.getinstance(app).gettoken(); returns null never ready when called. have tried polling test if token generated @ point. no luck. how can generate fcm token reliably firebase instance instantiated during runtime? i...
Read more