Celery chunks large data set
I'm trying to use celery's chunks functionality to divide my iterable dataset into pieces, which is then sent to a celery task for further processing. I have a query_set that I got from making the following sqlalchemy call query_set = MyModel.query.join(OtherModel).all()) Currently, query_set is a list of tuples. The lenth of query_results is at 40,000 and growing. I have another function (celery task) that crunches the data in query_set, whose definition is #celery_app.task def crunch_qs(query_set): . . . . . . Since query_set is a list of tuples, I figured I could pass it directly to crunch_qs like this crunched_qs = crunch_qs.chunks(query_set, 5000)() results = crunched_qs.get() That did not work. It gave me an unexpected result. It was unpacking the items in each query_set's tuple and sending them to crunch_qs. So crunch_qs would receive **query_set on first iteration, which raised the following error TypeError: crunch_qs() takes exactly 1 argument (10 given) len(query_set) = 10 I also tried.. crunched_qs = crunch_qs.chunks((row,) for row in query_set, 5000)() results = crunched_qs.get() That worked a little better. The TypeError went away. However, my crunch_qs function is now getting each row (tuple) as a parameter instead of a list of tuples whose length is 5000. Any help/ideas on how to pass a list of tuples to celery chunks would be highly appreciated. Thanks in advance
How to load DICOM files in Python?
iPerf3 as a ssh.exec command using paramiko in Python
MemoryError when trying to append
Access the result of a previous calculation in custom function passed to apply()
multithreading: Second thread is waiting until first thread is finished
What determines the size of int in numpy?
sending html in django http response
SQLite not accepting unique command
How to run a Python package as a scheduled daemon job?
merge two tables when one table has key, but the other has only a range of the key
python mechanize: how can i make mechanize's submit() be able to open local HTML file?
Extracting Info From Specific Sub-elements Using ElementTree XML
Django delete old and new file before/after save new
How to make a response on the web using MQTT and Flask?
Four-of-a-Kind Dice Game
Python subprocess - how to redirect only input