multithreading - Multiprocessing with elastic search in python -


got stuck problem while working multiprocessing in python.

i have python script calls 7 functions in parallel. these 7 functions again call 5 functions each same. these 5 child functions use elastic search me results parent function. , parent function results given top parent function. when call top parent script, 35 threads created in parallel(desired).

now script seems running in parallel , have 1 es cluster (my pc) , 1 node in it. searched bit , understood node getting locked first thread using es first , next thread starts. need more nodes in cluster.

node.name: "data one" node.master: true node.data: false  node.name: "data two" node.master: false node.data: true  node.name: "data three" node.master: false node.data: true 

after create nodes way, doubt how let threads know node use , create indexes on node only. way? :

es_client=elasticsearch(['node one'])   //'node two' 'node three' , on rest of threads? 

i'm trying figure out why code executing threads 1 one. appreciated. wrong in understanding somewhere here?. guys reading long post.

edit: code format

"grandparentscript.py"

func_topmostscript():     q1=queue.queue()     p1=process(target=func_fieldfunction(field1,q1)).start()     a=q1.get()     q2=queue.queue()     p2=process(target=func_fieldfunction(field2,q2)).start()     b=q2.get()     q3=queue.queue()     p3=process(target=func_fieldfunction(field3,q3)).start()     c=q3.get()     q4=queue.queue()     p4=process(target=func_fieldfunction(field4,q4)).start()     d=q4.get()     q5=queue.queue()     p5=process(target=func_fieldfunction(field5,q5)).start()     e=q5.get()     q6=queue.queue()     p6=process(target=func_fieldfunction(field6,q6)).start()     f=q6.get()     q7=queue.queue()     p7=process(target=func_fieldfunction(field7,q7)).start()     g=q7.get()     //do p,q,r,s,t,u,v 

"parentscript.py"

func_parentscript(field,queue):     c1=process(target=func_child1())     c2=process(target=func_child2())     c3=process(target=func_child3())     c4=process(target=func_child4())     c5=process(target=func_child5())     # c1,c2,c3,c4,c5 c     queue.put(c) 

each of child function called uses es client , sends output parent script.


Comments

Popular posts from this blog

Command prompt result in label. Python 2.7 -

javascript - How do I use URL parameters to change link href on page? -

amazon web services - AWS Route53 Trying To Get Site To Resolve To www -