python


Celery chunks large data set


I'm trying to use celery's chunks functionality to divide my iterable dataset into pieces, which is then sent to a celery task for further processing.
I have a query_set that I got from making the following sqlalchemy call
query_set = MyModel.query.join(OtherModel).all())
Currently, query_set is a list of tuples. The lenth of query_results is at 40,000 and growing.
I have another function (celery task) that crunches the data in query_set, whose definition is
#celery_app.task
def crunch_qs(query_set):
. . .
. . .
Since query_set is a list of tuples, I figured I could pass it directly to crunch_qs like this
crunched_qs = crunch_qs.chunks(query_set, 5000)()
results = crunched_qs.get()
That did not work. It gave me an unexpected result. It was unpacking the items in each query_set's tuple and sending them to crunch_qs.
So crunch_qs would receive **query_set[0] on first iteration, which raised the following error
TypeError: crunch_qs() takes exactly 1 argument (10 given)
len(query_set[0]) = 10
I also tried..
crunched_qs = crunch_qs.chunks((row,) for row in query_set, 5000)()
results = crunched_qs.get()
That worked a little better. The TypeError went away. However, my crunch_qs function is now getting each row (tuple) as a parameter instead of a list of tuples whose length is 5000.
Any help/ideas on how to pass a list of tuples to celery chunks would be highly appreciated.
Thanks in advance

Related Links

How can I get the default colors in GTK?
Ndb entry .put() not executed?
Tensorflow: NaNs propagating throughout network, even though using sparse_softmax etc
How to do a Python argparse mutually required argument group
scaling websocket game application server
Is there a way to find a character's Unicode code point in Python 2.7?
figtext datetime function matplotlib
environment variable in ubuntu
Cross-platform Python Executables
Extract subarray from collection of 2D coordinates?
tkinter populate treeview using threading pool
How to make a function determining the winner of Tic-Tac-Toe more concise
Django update model entry using form fails
ctypes using HRESULT(python)
How to export property values with django-import-export
Plotting Coordinate Lines Using Matplotlib

Categories

HOME
caching
sas
signalr
tinyos
dynamics-crm
json-ld
playframework
jscript
rocketmq
apple-numbers
webdav
adsense
triggers
avl-tree
actionscript-2
netflix-feign
diagram
oclint
xamarin-studio
nixos
glpk
movies
tcpclient
logarithm
widevine
owl-api
cgal
sharp
jackson-dataformat-csv
android-vpn-service
svnkit
oracle-xml-db
geo
von-neumann
text-files
image-compression
ntfs-mft
alchemy.js
cross-entropy
tripwire
paho
resuming-training
rich-text-editor
asmx
rapidweaver
defold
pluck
paas
barcode-printing
icefaces
visible
angularjs-ng-pattern
pax-web
shipitjs
achievements
faraday
picking
logcat
proxygen
mfc-feature-pack
flexjson
multinomial
java.util.calendar
jta
apigee-baas
webhdfs
connection-reset
prezto
graphical-logo
viper-architecture
ibm-data-studio
pgm
robocode
hg-git
ejabberd-saas
candidate-key
dache
lov
roxygen
dynamic-binding
linkedhashset
sunspot-rails
cosm
deploying
shim
die
text-services-framework
visual-web-gui
executable-format
phpcrawl
exitstatus
scala-2.8
pci-bus
visual-studio-2010-beta-2
lazy-c++
system-codedom-compiler
evb
web-analytics-tools
cots

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App