python 3.x - Setting Celery Worker to Load only Relevant Scripts -


summary: how can make celery worker read relevant scripts respective queues?

detailed: let's have 2 processes, light_process , heavy_process, on celery project. project organized as:

proj/     /__init__.py     /celery.py     /light_tasks.py     /heavy_tasks.py 

with celery.py being:

from celery import celery  app = celery('proj',              broker=[broker_url],              backend=[backend_url],              include=['proj.light_tasks','proj.heavy_tasks'])  app.conf.task_routes = {     'proj.light_tasks.light_process':{         'queue':'light_queue'     },     'proj.heavy_tasks.heavy_process':{         'queue':'heavy_queue'     } } 

light_tasks.py being:

from proj.celery import app import mylib  @app.task def light_process(arg):     return mylib.light_process(arg) 

and heavy_tasks.py :

from proj.celery import app import mylib  lotsa_ram = mylib.load('file_that_i_absolutely_need_in_memory')  @app.task def heavy_process(arg):     return lotsa_ram.heavy_process(arg) 

now, know if wanted run worker runs light process start as:

$ celery worker -a proj -q light_queue 

what need know is: if that, worker load heavy file, because it's in same project, or load file if it's relevant queue? , if load heavy_tasks.py anyway default, how can make worker run/load relevant script?

i understand keep different projects connected sabe broker, don't know if right way approach problem. besides, i'd keep on single repo avoid confusions while maintaining setup.


Comments