python


How to correctly instance a IPython cluster made of local and remote machines


I'm facing big problems in instancing a cluster for IPython. I think the doc is pretty complicated.
My system is made of the following things:
Local computer, Ubuntu 12.04 x86_64, IPython 2.1, 8 cores processor
Remote computer, Ubuntu 12.04 x86_64, IPython 2.1, 16 cores processor, access through passwordless SSH from local computer.
I have access with no password to the remote server via ssh linello#myserveripaddress.
I think this situation is pretty standard for many people that carry scientific computations.
I want to instanciate an IPython notebook with 8 local and 16 remote engines (=24 total engines) to exploit for some embarassingly parallel computations.
If I start my ipython cluster with a ipcluster start and then start an IPython instance, where I instanciate the Client class, I just get only 8 local clients
In[1]: from IPython import parallel
In[2]: clients = parallel.Client()
In[3]: print clients.ids
Out[3]: [0,1,2,3,4,5,6,7]
This means that only the local engines started. So I modified my ipcluster_config.py script to the following:
c = get_config()
c.IPClusterEngines.engine_launcher_class = 'SSH'
c.LocalControllerLauncher.controller_args = ["--ip='myserverip'"]
c.SSHEngineSetLauncher.engines = {
'localhost': 4,
'tyler': 4,
'par': 4,
}
c.SSHEngineSetLauncher.engine_cmd = ['/ipengine']
where myserverip is actually my server ip 192.168.xxx.yyy
I really don't know how to correctly configure my ipython profile to do have 24 engines in my ipython kernel.
I found this really confusing in the beginning as well. I still do to some extent, but I think I have a case working that looks similar to yours.
The only thing that looks to be missing is telling the engines where the controller is. There's an option in the ipcluster config that looks like:
c.SSHEngineLauncher.engine_args = ['--location=xxx.xxx.xxx.xxx']
where xxx.xxx.xxx.xxx is the ip address of the machine running the controller. I think that's the likely issue with your config.
I usually test this sort of thing by running ipcluster from the prompt. It's usually fairly explicit about the problem. I hope that helps.
Also, I usually don't need to specify:
c.SSHEngineSetLauncher.engine_cmd = ['/ipengine']

Related Links

Recommendations for a simple 2D graphics python library that can output to screen and pdf?
Reading numpy arrays outside of Python
Return value from thread
Is this control structure a code smell?
Django: How do I validate unique_together from within the model
Munging non-printable characters to dots using string.translate()
Error when using astWCS trying to create WCS object
if there any better way to read bb function souce code.i was very faint
PyQt4 signals and slots
Sql Alchemy What is wrong?
How do I plot a graph in Python?
Reordering matrix elements to reflect column and row clustering in naiive python
making urllib request in Python from the client side
Grid within a frame?
how to create http headers from scratch
Mapping a range of values to another

Categories

HOME
paypal
json
visual-studio-2015
crate
tinyos
itext
ubuntu-16.04
label
aem
actionscript
hana
angular2-directives
ruby-on-rails-3
webdav
snap.svg
tumblr
file-io
vsftpd
eval
rfid
tee
cpanel
x-frame-options
uiview
pyyaml
anychart
prestodb
ng2-dragula
koa
uiautomator
url.action
avplayeritem
servicemix
badge
sumo
java-ee-7
pipelinedb
lync-2013
version-numbering
multichoiceitems
xor
clockwork
stocks
division
passenger
bing-maps-api
broadleaf-commerce
office365connectors
installshield-2012
intellilock
cgo
onbackpressed
jrules
amd
rich-text-editor
web-deployment-project
text-classification
photon-controller
cdk
addin-express
barcode-printing
pspice
gameanalytics
picking
spservices
java.util.calendar
mogrify
manjaro
divide-by-zero
swift2.1
robocode
musl
java-melody
activity-streams
pgagent
kendo-mvvm
and-operator
ogr2ogr
suffix-array
mov
sthttprequest
author
yahoo-boss-api
initialization-vector
pyjade
opendata
magicalrecord-2.2
nscolor
manage.py
symphony-cms
code-conversion
web-safe-fonts
cadisplaylink
xmlslurper
unions
z-machine
text-services-framework
point-sprites
selectonemenu
text-size
executable-format
scala-2.8
web-statistics
internals
handheld

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App