python 3.x - Setting Celery Worker to Load only Relevant Scripts -
summary: how can make celery worker read relevant scripts respective queues?
detailed: let's have 2 processes, light_process , heavy_process, on celery project. project organized as:
proj/ /__init__.py /celery.py /light_tasks.py /heavy_tasks.py with celery.py being:
from celery import celery app = celery('proj', broker=[broker_url], backend=[backend_url], include=['proj.light_tasks','proj.heavy_tasks']) app.conf.task_routes = { 'proj.light_tasks.light_process':{ 'queue':'light_queue' }, 'proj.heavy_tasks.heavy_process':{ 'queue':'heavy_queue' } } light_tasks.py being:
from proj.celery import app import mylib @app.task def light_process(arg): return mylib.light_process(arg) and heavy_tasks.py :
from proj.celery import app import mylib lotsa_ram = mylib.load('file_that_i_absolutely_need_in_memory') @app.task def heavy_process(arg): return lotsa_ram.heavy_process(arg) now, know if wanted run worker runs light process start as:
$ celery worker -a proj -q light_queue what need know is: if that, worker load heavy file, because it's in same project, or load file if it's relevant queue? , if load heavy_tasks.py anyway default, how can make worker run/load relevant script?
i understand keep different projects connected sabe broker, don't know if right way approach problem. besides, i'd keep on single repo avoid confusions while maintaining setup.
Comments
Post a Comment