python


Celery chunks large data set


I'm trying to use celery's chunks functionality to divide my iterable dataset into pieces, which is then sent to a celery task for further processing.
I have a query_set that I got from making the following sqlalchemy call
query_set = MyModel.query.join(OtherModel).all())
Currently, query_set is a list of tuples. The lenth of query_results is at 40,000 and growing.
I have another function (celery task) that crunches the data in query_set, whose definition is
#celery_app.task
def crunch_qs(query_set):
. . .
. . .
Since query_set is a list of tuples, I figured I could pass it directly to crunch_qs like this
crunched_qs = crunch_qs.chunks(query_set, 5000)()
results = crunched_qs.get()
That did not work. It gave me an unexpected result. It was unpacking the items in each query_set's tuple and sending them to crunch_qs.
So crunch_qs would receive **query_set[0] on first iteration, which raised the following error
TypeError: crunch_qs() takes exactly 1 argument (10 given)
len(query_set[0]) = 10
I also tried..
crunched_qs = crunch_qs.chunks((row,) for row in query_set, 5000)()
results = crunched_qs.get()
That worked a little better. The TypeError went away. However, my crunch_qs function is now getting each row (tuple) as a parameter instead of a list of tuples whose length is 5000.
Any help/ideas on how to pass a list of tuples to celery chunks would be highly appreciated.
Thanks in advance

Related Links

How to load DICOM files in Python?
iPerf3 as a ssh.exec command using paramiko in Python
MemoryError when trying to append
Access the result of a previous calculation in custom function passed to apply()
multithreading: Second thread is waiting until first thread is finished
What determines the size of int in numpy?
sending html in django http response
SQLite not accepting unique command
How to run a Python package as a scheduled daemon job?
merge two tables when one table has key, but the other has only a range of the key
python mechanize: how can i make mechanize's submit() be able to open local HTML file?
Extracting Info From Specific Sub-elements Using ElementTree XML
Django delete old and new file before/after save new
How to make a response on the web using MQTT and Flask?
Four-of-a-Kind Dice Game
Python subprocess - how to redirect only input

Categories

HOME
batch-processing
shopify
dynamics-crm
vb6
rotation
nuxeo
google-contacts
reportportal
h2
sbt-assembly
rocketmq
ll
apiconnect
actionscript-2
gimp
contact-form-7
facebook-javascript-sdk
viber
uiview
event-log
eclipse-luna
lenskit
ssms-2016
bar-chart
arraylist
ms-access-2007
zoho
statusbar
logarithm
pdfsharp
auditing
maximo
mangodb
forum
clockwork
receipt
nsurlconnection
mapguide
threshold
honeysql
android-download-manager
mongoexport
deepstream.io
fax
twiki
flex4.5
jupyter-console
stereo-3d
paho
powermta
z3py
rdw
web-deployment-project
tcserver
hpcc
httplib2
emgu
createobject
cjson
debugdiag
galleriffic
unsatisfiedlinkerror
soda
dbscan
qt-linguist
asp.net-mvc-2
galaxy
sonarqube5.3
hateoas
zuora
xjc
divide-by-zero
bridge
rspec3
web-component-tester
hiera
lexicographic
obfuscar
java-melody
cpu-speed
cloudpebble
and-operator
errorprovider
node-inspector
java-collections-api
opensocial
broadcasting
flash-cc
winrt-httpclient
operations
real-time-updates
image-zoom
installshield-2009
va-list
backbone-relational
marmalade-edk
crocodoc
couchdb-lucene
scrollto
caliper
subtract
getstring
die
supersized
curljs
dcpu-16
symbol-server
surefire
file-encodings
communicationexception
visual-c++-2008-express
callgrind
lazy-c++
interface-design
self-reference

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App