python


How to correctly instance a IPython cluster made of local and remote machines


I'm facing big problems in instancing a cluster for IPython. I think the doc is pretty complicated.
My system is made of the following things:
Local computer, Ubuntu 12.04 x86_64, IPython 2.1, 8 cores processor
Remote computer, Ubuntu 12.04 x86_64, IPython 2.1, 16 cores processor, access through passwordless SSH from local computer.
I have access with no password to the remote server via ssh linello#myserveripaddress.
I think this situation is pretty standard for many people that carry scientific computations.
I want to instanciate an IPython notebook with 8 local and 16 remote engines (=24 total engines) to exploit for some embarassingly parallel computations.
If I start my ipython cluster with a ipcluster start and then start an IPython instance, where I instanciate the Client class, I just get only 8 local clients
In[1]: from IPython import parallel
In[2]: clients = parallel.Client()
In[3]: print clients.ids
Out[3]: [0,1,2,3,4,5,6,7]
This means that only the local engines started. So I modified my ipcluster_config.py script to the following:
c = get_config()
c.IPClusterEngines.engine_launcher_class = 'SSH'
c.LocalControllerLauncher.controller_args = ["--ip='myserverip'"]
c.SSHEngineSetLauncher.engines = {
'localhost': 4,
'tyler': 4,
'par': 4,
}
c.SSHEngineSetLauncher.engine_cmd = ['/ipengine']
where myserverip is actually my server ip 192.168.xxx.yyy
I really don't know how to correctly configure my ipython profile to do have 24 engines in my ipython kernel.
I found this really confusing in the beginning as well. I still do to some extent, but I think I have a case working that looks similar to yours.
The only thing that looks to be missing is telling the engines where the controller is. There's an option in the ipcluster config that looks like:
c.SSHEngineLauncher.engine_args = ['--location=xxx.xxx.xxx.xxx']
where xxx.xxx.xxx.xxx is the ip address of the machine running the controller. I think that's the likely issue with your config.
I usually test this sort of thing by running ipcluster from the prompt. It's usually fairly explicit about the problem. I hope that helps.
Also, I usually don't need to specify:
c.SSHEngineSetLauncher.engine_cmd = ['/ipengine']

Related Links

gRPC unary-streaming continues after disconnect / blocks on interrupt
Don't Get the concept of Variable Classes [on hold]
Python script isn't working
Delete nth row from a list of list - Python [duplicate]
How can I delete the Read files after --->>(for root, dirs, files in os.walk()) <<— with function os.remove() in python
Filter Pandas DataFrame with Nested Arrays
Convert String to datetime object pyton
Getting Python SQL error - Must declare the scalar variable “#p1offset” when using Tkinter
Gensim saved dictionary has no id2token
What is the purpose of the LibraryLoader.LoadLibrary function in Python ctypes?
Enumerating all mazes of given size n*m
python exit from class after handling exception
Python: Coordinates Boxes around Polyline
Install multiple version of Python packages like node does it
margin giving wrong output for pandas
Python re: why does [,-_] match “=”?

Categories

HOME
swift
rest
vue.js
nuxeo
ubuntu-16.04
jasmine
smartphone
playframework
hana
chaiscript
apache2
orientation
flyway
webdav
phaser-framework
pygame
httpclient
constructor
height
octopus-deploy
ssr
react-leaflet
typeahead
lstm
session-timeout
jsonserializer
styles
plupload
aws-cognito
epicor
dspic
richfaces
header-files
searchbar
python-textprocessing
geopandas
guile
hash-collision
image-quality
reply
dpi
partial-application
weinre
datadog
ivy
piecewise
helix-3d-toolkit
modelandview
node-horseman
superpowered
viewstate
remote-server
rider
bing-maps-api
mms
swagger-editor
image-compression
deepstream.io
web-development-server
ical-dotnet
react-native-router-flux
amazon-clouddrive
estimation
w3-total-cache
ti-basic
workflow-foundation-4.5
modelattribute
instruction-set
tofixed
hpcc
paas
collapsingtoolbarlayout
freetype2
asteriskami
windows-nt
com-interop
remoteapp
django-south
financial
code-readability
uialertview
uos
shift-jis
low-level
word-2013
cpan
obfuscar
line-numbers
jquery-autocomplete
firebug-lite
surrogate-key
phpdocx
bounds
winrt-httpclient
java.util.date
loop-invariant
dllexport
surveyor-gem
google-closure-library
dynamic-data
fotoware
gdt
die
maven-ant-tasks
separation-of-concerns
flash-cs5.5
shared-objects
surf
rubycas
ets
squeel
pureftpd
recordset
google-instant
front-controller
webrat
oggvorbis
windows-controls
lazy-c++
hmacsha1
divx

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App