Celery chunks large data set
I'm trying to use celery's chunks functionality to divide my iterable dataset into pieces, which is then sent to a celery task for further processing. I have a query_set that I got from making the following sqlalchemy call query_set = MyModel.query.join(OtherModel).all()) Currently, query_set is a list of tuples. The lenth of query_results is at 40,000 and growing. I have another function (celery task) that crunches the data in query_set, whose definition is #celery_app.task def crunch_qs(query_set): . . . . . . Since query_set is a list of tuples, I figured I could pass it directly to crunch_qs like this crunched_qs = crunch_qs.chunks(query_set, 5000)() results = crunched_qs.get() That did not work. It gave me an unexpected result. It was unpacking the items in each query_set's tuple and sending them to crunch_qs. So crunch_qs would receive **query_set on first iteration, which raised the following error TypeError: crunch_qs() takes exactly 1 argument (10 given) len(query_set) = 10 I also tried.. crunched_qs = crunch_qs.chunks((row,) for row in query_set, 5000)() results = crunched_qs.get() That worked a little better. The TypeError went away. However, my crunch_qs function is now getting each row (tuple) as a parameter instead of a list of tuples whose length is 5000. Any help/ideas on how to pass a list of tuples to celery chunks would be highly appreciated. Thanks in advance
Populate XML values from HTML Web Form Using Python
Gunicorn + Flask-Restful : High CPU usage while starting
Pymongo Regex match with list
Python 2.7 The 'packaging' package is required; normally this is bundled with this package
EOF Error Pickle
Graphviz: write result to file
Debug behavior differ from normal execution in python
nodejs unable to pass more than one arguments to the script with spawn
Setting indents while writing to an xml. Python.
Python Numpy's argsort duplicate issue [duplicate]
How to write Python script like shell script for UNIX?
Why does pythons slice indexing give counter intuitive results? [duplicate]
How to solve “Insufficient Permission” for userUsageReport with Google API?
Python: TypeError: 'list' object is not callable on global variable
sudo/suid non-root nesting fails
Difference between a list & a stack in python?