Connecting to MongoDB Atlas (cloud service)

FireWorks
1

Hi,

I have a running mongodb-cluster on Altas which I am trying to set up as a Launchpad for fireworks.

I have created a new (empty) database “my_db”, and my “.fireworks/my_launchpad.yaml” looks like this:

host: cluster0-shard-00-00-xxxxx.mongodb.net:27017,cluster0-shard-00-01-xxxxx.mongodb.net:27017,cluster0-shard-00-02-xxxxx.mongodb.net:27017
logdir: null
name: my_db
password: password
port: 27017
strm_lvl: INFO
username: username
ssl: true

``

When I try to connect with the following code:

from fireworks import LaunchPad
from fireworks.fw_config import LAUNCHPAD_LOC
lpad = LaunchPad.from_file(LAUNCHPAD_LOC)

``

I get the following error:

Traceback (most recent call last):

File “”, line 3, in
lpad = LaunchPad.from_file(LAUNCHPAD_LOC)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 286, in from_file
return cls.from_format(f.read(), f_format=f_format)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 254, in from_format
yaml.safe_load(f_str)))

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 224, in from_dict
ssl_ca_certs, ssl_certfile, ssl_keyfile, ssl_pem_passphrase)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 155, in init
self.db.authenticate(username, password)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/database.py”, line 1274, in authenticate
connect=True)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 614, in _cache_credentials
sock_info.authenticate(credentials)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/pool.py”, line 688, in authenticate
auth.authenticate(credentials, self)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 563, in authenticate
auth_func(credentials, sock_info)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 540, in _authenticate_default
return _authenticate_scram(credentials, sock_info, ‘SCRAM-SHA-1’)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 262, in _authenticate_scram
res = sock_info.command(source, cmd)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/pool.py”, line 579, in command
unacknowledged=unacknowledged)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/network.py”, line 150, in command
parse_write_concern_error=parse_write_concern_error)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)

OperationFailure: Authentication failed.

``

If I change “name” in my_launchpad.yaml to “admin”, which is the user authentication database on Atlas I am able to connect, however this db has certain restrictions so it can not be used as a fireworks db.

When I run the following code (with “name”=“admin”):

lpad.get_fw_ids()

``

I get the following error:

Traceback (most recent call last):

File “”, line 1, in
lpad.get_fw_ids()

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 620, in get_fw_ids
for fw in getattr(self, coll).find(criteria, {“fw_id”: True}, sort=sort).limit(limit):

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 1189, in next
if len(self.__data) or self._refresh():

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 1104, in _refresh
self.__send_message(q)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 982, in __send_message
helpers._check_command_response(first)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)

OperationFailure: cannot do raw queries on admin in atlas

``

Everything works fine however if I in the LaunchPad constructor pass username and password directly to the MongoClient constructor instead of the in the subsequent “if username:…” statement:

user_creds = {‘username’: username, ‘password’: password} if username else {}

self.connection = MongoClient(host, port, ssl=self.ssl,

ssl_ca_certs=self.ssl_ca_certs, ssl_certfile=self.ssl_certfile,

ssl_keyfile=self.ssl_keyfile, ssl_pem_passphrase=self.ssl_pem_passphrase,

socketTimeoutMS=MONGO_SOCKET_TIMEOUT_MS, connect=False, **user_creds)

self.db = self.connection[name]

if username:

self.db.authenticate(username, password)

``

I am using fireworks 1.8.0 and pymongo 3.7.2. The Mongodb instance on Atlas is version 3.6

If anyone has successfully used an Atlas mongodb instance with fireworks, please let be know if I am missing out on something. I have used mlab.com without problems earlier. Any help here is much appreciated.

Regards,

Audun

2

Hi Audun

I would suggest first trying to connect using the mongodb shell (e.g., “mongo” command line tool) before working with Python, FireWorks, etc. This will just make sure you have things working with mongo tools (MongoDB Atlas and mongo shell). If you are having trouble connecting with the mongo shell, I would suggest contacting MongoDB Atlas support.

Once you’ve figured out the credentials and settings that work to connect via the mongo shell, then just use the same credentials for FireWorks. If it works with mongo shell but not with FireWorks, please send another message.

···

On Sat, Oct 20, 2018 at 1:40 PM agrav [email protected] wrote:

Hi,

I have a running mongodb-cluster on Altas which I am trying to set up as a Launchpad for fireworks.

I have created a new (empty) database “my_db”, and my “.fireworks/my_launchpad.yaml” looks like this:

host: cluster0-shard-00-00-xxxxx.mongodb.net:27017,cluster0-shard-00-01-xxxxx.mongodb.net:27017,cluster0-shard-00-02-xxxxx.mongodb.net:27017
logdir: null
name: my_db
password: password
port: 27017
strm_lvl: INFO
username: username
ssl: true

``

When I try to connect with the following code:

from fireworks import LaunchPad
from fireworks.fw_config import LAUNCHPAD_LOC
lpad = LaunchPad.from_file(LAUNCHPAD_LOC)

``

I get the following error:

Traceback (most recent call last):

File “”, line 3, in
lpad = LaunchPad.from_file(LAUNCHPAD_LOC)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 286, in from_file
return cls.from_format(f.read(), f_format=f_format)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 254, in from_format
yaml.safe_load(f_str)))

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 224, in from_dict
ssl_ca_certs, ssl_certfile, ssl_keyfile, ssl_pem_passphrase)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 155, in init
self.db.authenticate(username, password)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/database.py”, line 1274, in authenticate
connect=True)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 614, in _cache_credentials
sock_info.authenticate(credentials)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/pool.py”, line 688, in authenticate
auth.authenticate(credentials, self)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 563, in authenticate
auth_func(credentials, sock_info)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 540, in _authenticate_default
return _authenticate_scram(credentials, sock_info, ‘SCRAM-SHA-1’)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 262, in _authenticate_scram
res = sock_info.command(source, cmd)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/pool.py”, line 579, in command
unacknowledged=unacknowledged)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/network.py”, line 150, in command
parse_write_concern_error=parse_write_concern_error)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)

OperationFailure: Authentication failed.

``

If I change “name” in my_launchpad.yaml to “admin”, which is the user authentication database on Atlas I am able to connect, however this db has certain restrictions so it can not be used as a fireworks db.

When I run the following code (with “name”=“admin”):

lpad.get_fw_ids()

``

I get the following error:

Traceback (most recent call last):

File “”, line 1, in
lpad.get_fw_ids()

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 620, in get_fw_ids
for fw in getattr(self, coll).find(criteria, {“fw_id”: True}, sort=sort).limit(limit):

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 1189, in next
if len(self.__data) or self._refresh():

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 1104, in _refresh
self.__send_message(q)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 982, in __send_message
helpers._check_command_response(first)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)

OperationFailure: cannot do raw queries on admin in atlas

``

Everything works fine however if I in the LaunchPad constructor pass username and password directly to the MongoClient constructor instead of the in the subsequent “if username:…” statement:

user_creds = {‘username’: username, ‘password’: password} if username else {}

self.connection = MongoClient(host, port, ssl=self.ssl,

ssl_ca_certs=self.ssl_ca_certs, ssl_certfile=self.ssl_certfile,

ssl_keyfile=self.ssl_keyfile, ssl_pem_passphrase=self.ssl_pem_passphrase,

socketTimeoutMS=MONGO_SOCKET_TIMEOUT_MS, connect=False, **user_creds)

self.db = self.connection[name]

if username:

self.db.authenticate(username, password)

``

I am using fireworks 1.8.0 and pymongo 3.7.2. The Mongodb instance on Atlas is version 3.6

If anyone has successfully used an Atlas mongodb instance with fireworks, please let be know if I am missing out on something. I have used mlab.com without problems earlier. Any help here is much appreciated.

Regards,

Audun

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

3

Hi Anubhav, thank you for the reply.

I have now successfully connected to atlas with MongoDB shell (version v3.6.3). I can use my intended FireWorks database and read/write documents to/from collections in the db.

Atlas suggests two connection strings for the mongo shell, depending on version, and both works for me:

v3.6+:

mongo “mongodb+srv://cluster0-xxxxx.mongodb.net/test” --username username

v3.4+

mongo “mongodb://cluster0-shard-00-00-xxxxx.mongodb.net:27017,cluster0-shard-00-01-xxxxx.mongodb.net:27017,cluster0-shard-00-02-xxxxx.mongodb.net:27017/test?replicaSet=Cluster0-shard-0” --ssl --authenticationDatabase admin --username usename–password password

I have also successfully connected with pymongo. All three alternatives below works. Note that alt 1 is similar to what happens in the LaunchPad constructor, except that I also pass the optional keyword argument “source=‘admin’”. If I leave this keyword out, I get the authentication error mentioned in the original post. I am still not able to use the database with FireWorks.

import pymongo
import yaml
from fireworks.fw_config import LAUNCHPAD_LOC

alt 1

db_dict = yaml.load(open(LAUNCHPAD_LOC))
client = pymongo.MongoClient(db_dict[‘host’],
db_dict[‘port’],
ssl=db_dict[‘ssl’],
connect=False,
authSource=‘admin’
)
db = client[db_dict[‘name’]]
if db_dict.get(‘username’):
db.authenticate(db_dict[‘username’], db_dict[‘password’], source=‘admin’)
list(db[‘projects’].find())

alt 2

db_dict = yaml.load(open(LAUNCHPAD_LOC))
client = pymongo.MongoClient(db_dict[‘host’],
db_dict[‘port’],
ssl=db_dict[‘ssl’],
username=db_dict[‘username’],
password=db_dict[‘password’],
connect=False,
)
db = client[db_dict[‘name’]]
list(db[‘projects’].find())

alt 3

db_dict = yaml.load(open(LAUNCHPAD_LOC))
conn_str = ‘mongodb://{}:{}@{}/test?ssl=true&authSource=admin’.format(
db_dict[‘username’],
db_dict[‘password’],
db_dict[‘host’])
client = pymongo.MongoClient(conn_str)
db = client[db_dict[‘name’]]
list(db[‘projects’].find())

``

(Please note that above and in my previous post I have replaced the actual credentials and db-uri with dummy values for anonymity)

A solution would perhaps be to allow for an “authsource” key in the my_launchpad.yaml for specifying a authentication db other than the db specified by “name” ?

Audun

søndag 21. oktober 2018 23.48.46 UTC+2 skrev ajain følgende:

···

Hi Audun

I would suggest first trying to connect using the mongodb shell (e.g., “mongo” command line tool) before working with Python, FireWorks, etc. This will just make sure you have things working with mongo tools (MongoDB Atlas and mongo shell). If you are having trouble connecting with the mongo shell, I would suggest contacting MongoDB Atlas support.

Once you’ve figured out the credentials and settings that work to connect via the mongo shell, then just use the same credentials for FireWorks. If it works with mongo shell but not with FireWorks, please send another message.

On Sat, Oct 20, 2018 at 1:40 PM agrav [email protected] wrote:

Hi,

I have a running mongodb-cluster on Altas which I am trying to set up as a Launchpad for fireworks.

I have created a new (empty) database “my_db”, and my “.fireworks/my_launchpad.yaml” looks like this:

host: cluster0-shard-00-00-xxxxx.mongodb.net:27017,cluster0-shard-00-01-xxxxx.mongodb.net:27017,cluster0-shard-00-02-xxxxx.mongodb.net:27017
logdir: null
name: my_db
password: password
port: 27017
strm_lvl: INFO
username: username
ssl: true

``

When I try to connect with the following code:

from fireworks import LaunchPad
from fireworks.fw_config import LAUNCHPAD_LOC
lpad = LaunchPad.from_file(LAUNCHPAD_LOC)

``

I get the following error:

Traceback (most recent call last):

File “”, line 3, in
lpad = LaunchPad.from_file(LAUNCHPAD_LOC)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 286, in from_file
return cls.from_format(f.read(), f_format=f_format)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 254, in from_format
yaml.safe_load(f_str)))

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 224, in from_dict
ssl_ca_certs, ssl_certfile, ssl_keyfile, ssl_pem_passphrase)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 155, in init
self.db.authenticate(username, password)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/database.py”, line 1274, in authenticate
connect=True)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 614, in _cache_credentials
sock_info.authenticate(credentials)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/pool.py”, line 688, in authenticate
auth.authenticate(credentials, self)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 563, in authenticate
auth_func(credentials, sock_info)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 540, in _authenticate_default
return _authenticate_scram(credentials, sock_info, ‘SCRAM-SHA-1’)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/auth.py”, line 262, in _authenticate_scram
res = sock_info.command(source, cmd)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/pool.py”, line 579, in command
unacknowledged=unacknowledged)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/network.py”, line 150, in command
parse_write_concern_error=parse_write_concern_error)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)

OperationFailure: Authentication failed.

``

If I change “name” in my_launchpad.yaml to “admin”, which is the user authentication database on Atlas I am able to connect, however this db has certain restrictions so it can not be used as a fireworks db.

When I run the following code (with “name”=“admin”):

lpad.get_fw_ids()

``

I get the following error:

Traceback (most recent call last):

File “”, line 1, in
lpad.get_fw_ids()

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 620, in get_fw_ids
for fw in getattr(self, coll).find(criteria, {“fw_id”: True}, sort=sort).limit(limit):

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 1189, in next
if len(self.__data) or self._refresh():

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 1104, in _refresh
self.__send_message(q)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/cursor.py”, line 982, in __send_message
helpers._check_command_response(first)

File “/home/audun/miniconda3/envs/fireworks/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)

OperationFailure: cannot do raw queries on admin in atlas

``

Everything works fine however if I in the LaunchPad constructor pass username and password directly to the MongoClient constructor instead of the in the subsequent “if username:…” statement:

user_creds = {‘username’: username, ‘password’: password} if username else {}

self.connection = MongoClient(host, port, ssl=self.ssl,

ssl_ca_certs=self.ssl_ca_certs, ssl_certfile=self.ssl_certfile,

ssl_keyfile=self.ssl_keyfile, ssl_pem_passphrase=self.ssl_pem_passphrase,

socketTimeoutMS=MONGO_SOCKET_TIMEOUT_MS, connect=False, **user_creds)

self.db = self.connection[name]

if username:

self.db.authenticate(username, password)

``

I am using fireworks 1.8.0 and pymongo 3.7.2. The Mongodb instance on Atlas is version 3.6

If anyone has successfully used an Atlas mongodb instance with fireworks, please let be know if I am missing out on something. I have used mlab.com without problems earlier. Any help here is much appreciated.

Regards,

Audun

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

4

Was this issue ever resolved? I am having a similar issue now.

5

Yes, see the latest message from agrav?

···

On Tuesday, December 4, 2018 at 6:39:55 AM UTC-8, Christopher Pashartis wrote:

Was this issue ever resolved? I am having a similar issue now.

6

He found a solution. But you did not implement it, I found out how to fix it in your code.
In a nutshell, you need to add a new kwarg in fireworks/core/launchpad.py, authSource. You must also make sure that self.db.authenticate(username, password, source = self.authSource). You can see my fork on github (cpashartis) for more info. **I’m not sure if these changes are consistent for using self.authSource = None. **

···

On Tuesday, December 4, 2018 at 4:15:45 PM UTC+1, Anubhav Jain wrote:

Yes, see the latest message from agrav?

On Tuesday, December 4, 2018 at 6:39:55 AM UTC-8, Christopher Pashartis wrote:

Was this issue ever resolved? I am having a similar issue now.

7

Hi,

You are right, I forgot to make the modification. It should be there in FWS v1.8.2.

···

On Tuesday, December 4, 2018 at 8:38:48 AM UTC-8, Christopher Pashartis wrote:

He found a solution. But you did not implement it, I found out how to fix it in your code.
In a nutshell, you need to add a new kwarg in fireworks/core/launchpad.py, authSource. You must also make sure that self.db.authenticate(username, password, source = self.authSource). You can see my fork on github (cpashartis) for more info. **I’m not sure if these changes are consistent for using self.authSource = None. **

On Tuesday, December 4, 2018 at 4:15:45 PM UTC+1, Anubhav Jain wrote:

Yes, see the latest message from agrav?

On Tuesday, December 4, 2018 at 6:39:55 AM UTC-8, Christopher Pashartis wrote:

Was this issue ever resolved? I am having a similar issue now.

8

Hi,
I actually ended up using mlab. Will migrate to atlas later. Thanks for the upcoming mod.

Audun

torsdag 6. desember 2018 22.49.43 UTC+1 skrev Anubhav Jain følgende:

···

Hi,

You are right, I forgot to make the modification. It should be there in FWS v1.8.2.

On Tuesday, December 4, 2018 at 8:38:48 AM UTC-8, Christopher Pashartis wrote:

He found a solution. But you did not implement it, I found out how to fix it in your code.
In a nutshell, you need to add a new kwarg in fireworks/core/launchpad.py, authSource. You must also make sure that self.db.authenticate(username, password, source = self.authSource). You can see my fork on github (cpashartis) for more info. **I’m not sure if these changes are consistent for using self.authSource = None. **

On Tuesday, December 4, 2018 at 4:15:45 PM UTC+1, Anubhav Jain wrote:

Yes, see the latest message from agrav?

On Tuesday, December 4, 2018 at 6:39:55 AM UTC-8, Christopher Pashartis wrote:

Was this issue ever resolved? I am having a similar issue now.

9

TL;DR: You have to add authsource: admin to your my_launchpad.yaml file, and make sure you use fireworks v1.8.2 or higher.

Thanks Audun, Christopher and Anubhav for fixing this issue! I was also using mlab and couldn’t figure out why migrating to Atlas just wasn’t working for me.

Kind regards,

Marnik