Hi Prof. Anubhav
(#1):
As you wrote, I confirmed I can connect using the native MongoDB tools, e.g., action 1-3 was succeed in our environment.
However, actually, I doubt the problem is about connection to mongo.
I think it is because SSH without ~/.ssh/id_rsa_XXX doesn’t work, regardless of the mongo.
I’m sorry I was not clear, or I’m still misunderstanding.
My problem is, when we execute “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” (by the NEW version of fireworks), it fails with error
File “/home/taka/venv/std_python3/bin//qlaunch”, line 11, in
sys.exit(qlaunch())
File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fireworks/scripts/qlaunch_run.py”, line 210, in qlaunch
pre_non_default, args.command, non_default))
File “”, line 2, in run
File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 29, in opens
self.open()
File “/home/taka/venv/std_python3/lib/python3.5/site-packages/fabric/connection.py”, line 541, in open
self.client.connect(**kwargs)
File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 424, in connect
passphrase,
File “/home/taka/venv/std_python3/lib/python3.5/site-packages/paramiko/client.py”, line 715, in _auth
raise SSHException('No authentication methods available')
paramiko.ssh_exception.SSHException: No authentication methods available
I think that it is because the our supercomputer (remote fireworker and NOT mongo server) requires ~/.ssh/id_rsa, however, with fabric.Connection( … in 201 line in /scripts/qlaunch_run.py doesn’t specify key_filename.
Actually, following small script works well for me.
import fabric
import os
host = “my.remote.supercomputer.ac.jp”
key_filename="/home/taka/.ssh/id_rsa"
with fabric.Connection(
host=host,
user=“my_user”,
config=fabric.Config({‘run’: {‘shell’: “/usr/bin/zsh -l -c”}}),
connect_kwargs={‘password’: None, ‘key_filename’: key_filename}) as conn:
connect_kwargs={‘password’: None}) as conn: # fails with the same error
for r in ["/path/to/dir"]:
r = os.path.expanduser(r)
with conn.cd(r):
conn.run("ls")
For me, it seems the problem is not related to mongo, because above script works regardless to mongo and port 27018.
Or, can SSH tunnel solve such a problem? If my explanation is still insufficient of misunderstanding, please write.
I’m sorry for prolonging my question and thanks for your patience.
(#2):
The updated script works well for us!
That will be very helpful for us.
We really appreciate the work.
Thanks,
2018年8月21日火曜日 2時15分20秒 UTC+9 Anubhav Jain:
···
Also regarding issue #1, did you try using the native Mongo tools to connect?
e.g.,
- Start the SSH tunnel
- Type “mongo --port 27018 mydb” to open a tunnel
- Use the mongo shell to switch to your fireworks database (e.g. “use fireworks”), authorize (e.g., “db.auth(xxx)”), and execute a query successfully (e.g., “db.fireworks.findOne()”).
If you are able to do that, then you should be able to issue the same queries as in #3 with FireWorks as well. It will just be a matter of configuring the files properly. But, first you should confirm whether you can connect even using the native MongoDB tools.
On Monday, August 20, 2018 at 10:10:58 AM UTC-7, Anubhav Jain wrote:
Hi Akira
Richard Gowers has contributed a fix for the Fabric issue - FWS now uses Fabric 2.3.1. The changes are reflected in the latest version of FWS, v1.7.8.
On Sunday, August 12, 2018 at 3:17:21 AM UTC-7, Akira Takahashi wrote:
Hi Prof. Anubhav
Thanks for fast reply.
However, I’m afraid that, I could not solve my problems.
-
As written in https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69, I tried to execute
ssh -i /path/to/ssh_public_key user@remote_host_address -Nf -L 27018:localhost:27017
and execute
mongod --config /my/mongodb/path/mongodb.config # only dbpath is written in mongodb.config
and on another terminal, I did
qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50
, however, it failed to connect.
( Fireworks require my password with the message:
[remote_host_address] Login password for ‘user’
, however, it failed and repeated infinitely. )
Actually, I added key_filename option to qlaunch_run.py like
for h in args.remote_host:
with settings(host_string=h, user=args.remote_user,
password=args.remote_password,
key_filename="/path/to/ssh_public_key", # added
shell=args.remote_shell):
, then, connection succeeded.
Is my command wrong, or does anything else should be configured?
-
I tried to update the code of FireWorks, however, Fabric2 seems greatly different from Fabric1, then I think that considerable codes should be replaced.
Since I am not familiar with Fabric1 or Fabric2, maybe it takes a lot of time for me and maybe I would break backward compatibility of FireWorks.
Therefore, I’m afraid that, I think that I should NOT update FireWorks code.
Thanks,
Akira Takahashi
2018年8月11日土曜日 11時59分02秒 UTC+9 Anubhav Jain:
Hi Akira
- FireWorks has some options for including SSL information but not SSH. You can create an SSH tunnel instead, there should be many examples online e.g.
https://stackoverflow.com/questions/42718547/how-to-connect-remote-mongodb-with-pymongo
https://gist.github.com/umidjons/ec7a60415c452787f2666ef6761bbf69
- Most of the FireWorks developers are not using the remote launch features, so it is not so well maintained. The best solution is if we can upgrade the requirements to using fabric2.x.x. Are you familiar with Python? If so perhaps you can update the code to work with Fabric 2.x.x and submit a pull request we can merge in. I have created a ticket for this on the FireWorks Github page: https://github.com/materialsproject/fireworks/issues/306 . If you have trouble or think you are not able to do it, please respond there and we can try and figure out the next step.
On Thursday, August 9, 2018 at 6:42:29 PM UTC-7, Akira Takahashi wrote:
Hi,
I am tackling execution to qlaunch to remote host.
I have two question about that.
- To connect our remote computer, we need ssh secret key.
How can I use ssh key by fireworks?
- In my environment using python3, “qlaunch -rh XXX.XXX.XXX.XXX -ru XXX rapidfire -m 50” failed with error message
“Remote options require the Fabric package to be installed!”.
I think that this is because fabric2.X.X is not compatible with fabric1.X.X and
from fabric.api import settings, run, cd
(in scripts/qlaunch.py)
is failed because fabric.api no longer exists.
However, even when I downgrade fabric from 2.3.1 to 1.14.0 by pip install, it still with error
File "........./lib/python3.5/site-packages/fabric/context_ managers.py", line 535
def accept(channel, (src_addr, src_port), (dest_addr, dest_port)):
, I think, because fabric1 is not python3 compatible.
Now I use fireworks by python2, however, using fireworks by python3 is more useful for me.
How should I configure my environment?
I would appreciate your help.