python


Celery chunks large data set


I'm trying to use celery's chunks functionality to divide my iterable dataset into pieces, which is then sent to a celery task for further processing.
I have a query_set that I got from making the following sqlalchemy call
query_set = MyModel.query.join(OtherModel).all())
Currently, query_set is a list of tuples. The lenth of query_results is at 40,000 and growing.
I have another function (celery task) that crunches the data in query_set, whose definition is
#celery_app.task
def crunch_qs(query_set):
. . .
. . .
Since query_set is a list of tuples, I figured I could pass it directly to crunch_qs like this
crunched_qs = crunch_qs.chunks(query_set, 5000)()
results = crunched_qs.get()
That did not work. It gave me an unexpected result. It was unpacking the items in each query_set's tuple and sending them to crunch_qs.
So crunch_qs would receive **query_set[0] on first iteration, which raised the following error
TypeError: crunch_qs() takes exactly 1 argument (10 given)
len(query_set[0]) = 10
I also tried..
crunched_qs = crunch_qs.chunks((row,) for row in query_set, 5000)()
results = crunched_qs.get()
That worked a little better. The TypeError went away. However, my crunch_qs function is now getting each row (tuple) as a parameter instead of a list of tuples whose length is 5000.
Any help/ideas on how to pass a list of tuples to celery chunks would be highly appreciated.
Thanks in advance

Related Links

How to refer to the local module in Python?
Is close() necessary when using iterator on a Python file object [duplicate]
Django Admin “Edit Selection” Action?
How to change firefox proxy from webdriver?
Is it possible to hook up a more robust HTML parser to Python mechanize?
Enable Unicode “globally” in Python
Dynamically import a callable given the full module path?
python chaining
py2app and xml.etree.ElementTree
What is the difference between isinstance('aaa', basestring) and isinstance('aaa', str)?
Is this essential functional programming feature missing from python?
Hooking into a wave-out on different platforms
What causes subprocess.call to output blank file when attempting db export with mysqldump?
How do I remove something form a list, plus string matching?
Reference to Part of List - Python
Django Delete all but last five of queryset

Categories

HOME
arrays
crystal-reports
admin-on-rest
crate
oauth
webrtc
otrs
zend-framework2
sbt-assembly
flyway
devstack
websphere-liberty
openwrt
ipfs
maxima
postmessage
ida
uicollectionview
anychart
ip-camera
profile
cep
codelite
fetch-api
vcard
system.data.sqlite
pitest
ifstream
pycparser
phpspreadsheet
datalog
jdbi
resuming-training
amd
matlab-cvst
php-ews
lumen-5.3
fqdn
appstore-approval
modelattribute
collapsingtoolbarlayout
nofollow
windows-nt
coordinate-transformation
achievements
date-format
picking
direct3d12
kendo-menu
manjaro
ytplayerview
digits
crash-dumps
coypu
gulp-livereload
jazz
subversion-edge
code-first-migrations
kallithea
java-melody
errorprovider
suffix-array
factors
hamsterdb
exiv2
nservicebus4
ruboto
objcmongodb
tfs-power-tools
continuous-testing
redirectstandardoutput
shim
applicationcontext
flash-cs5.5
nssavepanel
nsindexpath
throttling
posting
squeel
hadoop-plugins
trampolines
plinq
virtualquery

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App