Authentication failed for calculation on clusters

[Zezhong_Zhang]

Hi,

I am trying to setup atomate on supercomputer clusters that I can submit jobs from my laptop. I have read the Fireworks tutorial “Executing workflows on different types of computing resources”, I can submit jobs to the database and rlaunch or qlaunch -r to pick up the jobs in a cluster, but I am still quite confused. In specific, (a) how to specify which clusters to use in the spec so that the right clusters can pick the right jobs; (b) how to specify nodes upon job submission? I guess it needs to change this in the my_qlaunch.yaml that changes the template submission file; © how to do (a-b) jobs in python? I have now configured the public/private keys that I can login ssh passwordlessly in command line, but not in python. I guess this is due to the the OPENSSH not is supported yet in paramiko (see issue).

But before all of these, I still have problems performing a benchmark calculation first. The example was using the MgO band structure scrip as given in the tutorial.

However, the firework is fizzled and I received the authentication errors as follow.

Traceback (most recent call last):

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/atomate/utils/database.py”, line 42, in init

self.db.authenticate(self.user, self.password)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/database.py”, line 1274, in authenticate

connect=True)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 614, in _cache_credentials

sock_info.authenticate(credentials)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/pool.py”, line 688, in authenticate

auth.authenticate(credentials, self)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 563, in authenticate

auth_func(credentials, sock_info)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 538, in _authenticate_default

return _authenticate_scram(credentials, sock_info, ‘SCRAM-SHA-1’)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 262, in _authenticate_scram

res = sock_info.command(source, cmd)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/pool.py”, line 579, in command

unacknowledged=unacknowledged)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/network.py”, line 150, in command

parse_write_concern_error=parse_write_concern_error)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response

raise OperationFailure(msg % errmsg, code, response)

pymongo.errors.OperationFailure: Authentication failed.

Many thanks for your kind helps.

Best regards,

Zezhong

[Anubhav_Jain]

Hi Zhezhong

a) To determine where you want a job to run, see:

https://materialsproject.github.io/fireworks/controlworker.html

Scroll down to “Controlling the Worker that executes a Firework”

b) To determine which nodes run a job, you need to set the right queue flags in my_qadapter.yaml - assuming your queueing system supports something like this. You might want to have two “FireWorkers” on the same machine - one with a queueadapter for one group of nodes (e.g., high memory node job submission) and another queue adapter for a second group of nodes (e.g., regular job submission).

c) For the authentication failed, it’s unclear where that code is being run, whether you are able to connect successfully at all from some other location, etc… Some things to check:

• are you using the right my_launchpad.yaml

• does your my_launchpad contain the correct credentials?

• do you need to set your authsource to admin database?

• are you using the correct hostname, port, etc.?

There are unfortunately too many potential issues and not enough details for me to give any real opinion on your auth error.

As for “how to do (a-b) jobs in python?” I am not sure what you mean. Note that typically we set up all our workflows, etc. in Python but then manually execute “qlaunch -r rapidfire” on the nodes (i.e., log in manually and execute the command). Sometimes a cron job on the system is used, although you need to be careful to set that up well (e.g. with timeouts on the qlaunch).

···

On Mon, Apr 15, 2019 at 4:45 AM Zezhong Zhang [email protected] wrote:

Hi,

I am trying to setup atomate on supercomputer clusters that I can submit jobs from my laptop. I have read the Fireworks tutorial “Executing workflows on different types of computing resources”, I can submit jobs to the database and rlaunch or qlaunch -r to pick up the jobs in a cluster, but I am still quite confused. In specific, (a) how to specify which clusters to use in the spec so that the right clusters can pick the right jobs; (b) how to specify nodes upon job submission? I guess it needs to change this in the my_qlaunch.yaml that changes the template submission file; © how to do (a-b) jobs in python? I have now configured the public/private keys that I can login ssh passwordlessly in command line, but not in python. I guess this is due to the the OPENSSH not is supported yet in paramiko (see issue).

But before all of these, I still have problems performing a benchmark calculation first. The example was using the MgO band structure scrip as given in the tutorial.

However, the firework is fizzled and I received the authentication errors as follow.

Traceback (most recent call last):

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/atomate/utils/database.py”, line 42, in init

self.db.authenticate(self.user, self.password)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/database.py”, line 1274, in authenticate

connect=True)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 614, in _cache_credentials

sock_info.authenticate(credentials)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/pool.py”, line 688, in authenticate

auth.authenticate(credentials, self)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 563, in authenticate

auth_func(credentials, sock_info)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 538, in _authenticate_default

return _authenticate_scram(credentials, sock_info, ‘SCRAM-SHA-1’)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 262, in _authenticate_scram

res = sock_info.command(source, cmd)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/pool.py”, line 579, in command

unacknowledged=unacknowledged)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/network.py”, line 150, in command

parse_write_concern_error=parse_write_concern_error)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response

raise OperationFailure(msg % errmsg, code, response)

pymongo.errors.OperationFailure: Authentication failed.

Many thanks for your kind helps.

Best regards,

Zezhong

–

You received this message because you are subscribed to the Google Groups “atomate” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

[Zezhong_Zhang]

Hi Anubhav,

Thanks very much for your reply! I am still debugging the authentication error.

1. my_launchpad.yaml is correct as the local laptop can connect to the database using ‘lpad add’ and supercomuter can find the job with ‘lpad get_fws’ and ‘qlaunch’.

2. The credential is correct that I tried the following test code and received ‘True’ from authenticate.

from pymongo import MongoClient

client = MongoClient('mongodb+srv://user:[email protected]')

db=client.admin

db.authenticate(name=user,password=passwd)

I traced the error back to atomate.utils.get_database, as I run the following code give bad authtication error.

config_file='/path/to/db.json'

d = loadfn(config_file)

conn = MongoClient(host=d["host"], port=d["port"])

db = conn[d["database"]]

user = d["admin_user"]

passwd = d["admin_password"]

db.authenticate(user, passwd)

The db.json file is like this

{

“host”: “mongodb+srv://admin:[email protected]”,

“port”: 27017,

“database”: “alloy”,

“collection”: “workflows”,

“admin_user”: “admin”,

“admin_password”: “password”,

“readonly_user”: “user”,

“readonly_password”: “test123”,

“aliases”: {}

}

But if run the following code returns True

config_file='/path/to/db.json'

d = loadfn(config_file)

conn = MongoClient(host=d["host"], port=d["port"]).admin

user = d["admin_user"]

passwd = d["admin_password"]

conn.authenticate(user, passwd)

Many thanks for your help.

Cheers,

Zezhong

···

On Monday, April 15, 2019 at 6:23:19 PM UTC+2, Anubhav Jain wrote:

Hi Zhezhong

a) To determine where you want a job to run, see:

https://materialsproject.github.io/fireworks/controlworker.html

Scroll down to “Controlling the Worker that executes a Firework”

b) To determine which nodes run a job, you need to set the right queue flags in my_qadapter.yaml - assuming your queueing system supports something like this. You might want to have two “FireWorkers” on the same machine - one with a queueadapter for one group of nodes (e.g., high memory node job submission) and another queue adapter for a second group of nodes (e.g., regular job submission).

c) For the authentication failed, it’s unclear where that code is being run, whether you are able to connect successfully at all from some other location, etc… Some things to check:

• are you using the right my_launchpad.yaml

• does your my_launchpad contain the correct credentials?

• do you need to set your authsource to admin database?

• are you using the correct hostname, port, etc.?

There are unfortunately too many potential issues and not enough details for me to give any real opinion on your auth error.

As for “how to do (a-b) jobs in python?” I am not sure what you mean. Note that typically we set up all our workflows, etc. in Python but then manually execute “qlaunch -r rapidfire” on the nodes (i.e., log in manually and execute the command). Sometimes a cron job on the system is used, although you need to be careful to set that up well (e.g. with timeouts on the qlaunch).

On Mon, Apr 15, 2019 at 4:45 AM Zezhong Zhang [email protected] wrote:

Hi,

I am trying to setup atomate on supercomputer clusters that I can submit jobs from my laptop. I have read the Fireworks tutorial “Executing workflows on different types of computing resources”, I can submit jobs to the database and rlaunch or qlaunch -r to pick up the jobs in a cluster, but I am still quite confused. In specific, (a) how to specify which clusters to use in the spec so that the right clusters can pick the right jobs; (b) how to specify nodes upon job submission? I guess it needs to change this in the my_qlaunch.yaml that changes the template submission file; © how to do (a-b) jobs in python? I have now configured the public/private keys that I can login ssh passwordlessly in command line, but not in python. I guess this is due to the the OPENSSH not is supported yet in paramiko (see issue).

But before all of these, I still have problems performing a benchmark calculation first. The example was using the MgO band structure scrip as given in the tutorial.

However, the firework is fizzled and I received the authentication errors as follow.

Traceback (most recent call last):

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/atomate/utils/database.py”, line 42, in init

self.db.authenticate(self.user, self.password)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/database.py”, line 1274, in authenticate

connect=True)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 614, in _cache_credentials

sock_info.authenticate(credentials)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/pool.py”, line 688, in authenticate

auth.authenticate(credentials, self)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 563, in authenticate

auth_func(credentials, sock_info)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 538, in _authenticate_default

return _authenticate_scram(credentials, sock_info, ‘SCRAM-SHA-1’)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 262, in _authenticate_scram

res = sock_info.command(source, cmd)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/pool.py”, line 579, in command

unacknowledged=unacknowledged)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/network.py”, line 150, in command

parse_write_concern_error=parse_write_concern_error)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response

raise OperationFailure(msg % errmsg, code, response)

pymongo.errors.OperationFailure: Authentication failed.

Many thanks for your kind helps.

Best regards,

Zezhong

–

You received this message because you are subscribed to the Google Groups “atomate” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

[Anubhav_Jain]

Hi Zezhong,

Can you see if the following connection code works?

config_file=’/path/to/db.json’

d = loadfn(config_file)

conn = MongoClient(host=d[“host”], port=d[“port”], username=d[“admin_user”], password=d[“admin_password”], authSource=“admin”)

db = conn[d[“database”]]

If this works, we’ll just add an authSource option to the JSON db file configuration for the atomate db and slightly modify the connection code.

Note also that admin_user and admin_password were not intended to correspond to the user/pass of the admin database of your Mongo cluster, but rather a write-enabled user/pass for your atomate database. (If this sentence is confusing, feel free to ignore). Meaning, you don’t need to give credentials to your admin database if you don’t want, you can just provide credentials for your atomate database if you’ve set up a username/pass for your atomate DB.

Best,

Anubhav

···

On Tue, Apr 16, 2019 at 2:11 AM Zezhong Zhang [email protected] wrote:

Hi Anubhav,

Thanks very much for your reply! I am still debugging the authentication error.

1. my_launchpad.yaml is correct as the local laptop can connect to the database using ‘lpad add’ and supercomuter can find the job with ‘lpad get_fws’ and ‘qlaunch’.

2. The credential is correct that I tried the following test code and received ‘True’ from authenticate.

from pymongo import MongoClient

client = MongoClient(‘mongodb+srv://user:p…@alloyproject-4qquz.mongodb.net’)

db=client.admin

db.authenticate(name=user,password=passwd)

I traced the error back to atomate.utils.get_database, as I run the following code give bad authtication error.

config_file=’/path/to/db.json’

d = loadfn(config_file)

conn = MongoClient(host=d[“host”], port=d[“port”])

db = conn[d[“database”]]

user = d[“admin_user”]

passwd = d[“admin_password”]

db.authenticate(user, passwd)

The db.json file is like this

{

“host”: “mongodb+srv://admin:p…@alloyproject-4qquz.mongodb.net”,

“port”: 27017,

“database”: “alloy”,

“collection”: “workflows”,

“admin_user”: “admin”,

“admin_password”: “password”,

“readonly_user”: “user”,

“readonly_password”: “test123”,

“aliases”: {}

}

But if run the following code returns True

config_file=’/path/to/db.json’

d = loadfn(config_file)

conn = MongoClient(host=d[“host”], port=d[“port”]).admin

user = d[“admin_user”]

passwd = d[“admin_password”]

conn.authenticate(user, passwd)

Many thanks for your help.

Cheers,

Zezhong

On Monday, April 15, 2019 at 6:23:19 PM UTC+2, Anubhav Jain wrote:

Hi Zhezhong

a) To determine where you want a job to run, see:

https://materialsproject.github.io/fireworks/controlworker.html

Scroll down to “Controlling the Worker that executes a Firework”

b) To determine which nodes run a job, you need to set the right queue flags in my_qadapter.yaml - assuming your queueing system supports something like this. You might want to have two “FireWorkers” on the same machine - one with a queueadapter for one group of nodes (e.g., high memory node job submission) and another queue adapter for a second group of nodes (e.g., regular job submission).

c) For the authentication failed, it’s unclear where that code is being run, whether you are able to connect successfully at all from some other location, etc… Some things to check:

• are you using the right my_launchpad.yaml

• does your my_launchpad contain the correct credentials?

• do you need to set your authsource to admin database?

• are you using the correct hostname, port, etc.?

There are unfortunately too many potential issues and not enough details for me to give any real opinion on your auth error.

As for “how to do (a-b) jobs in python?” I am not sure what you mean. Note that typically we set up all our workflows, etc. in Python but then manually execute “qlaunch -r rapidfire” on the nodes (i.e., log in manually and execute the command). Sometimes a cron job on the system is used, although you need to be careful to set that up well (e.g. with timeouts on the qlaunch).

On Mon, Apr 15, 2019 at 4:45 AM Zezhong Zhang [email protected] wrote:

Hi,

I am trying to setup atomate on supercomputer clusters that I can submit jobs from my laptop. I have read the Fireworks tutorial “Executing workflows on different types of computing resources”, I can submit jobs to the database and rlaunch or qlaunch -r to pick up the jobs in a cluster, but I am still quite confused. In specific, (a) how to specify which clusters to use in the spec so that the right clusters can pick the right jobs; (b) how to specify nodes upon job submission? I guess it needs to change this in the my_qlaunch.yaml that changes the template submission file; © how to do (a-b) jobs in python? I have now configured the public/private keys that I can login ssh passwordlessly in command line, but not in python. I guess this is due to the the OPENSSH not is supported yet in paramiko (see issue).

But before all of these, I still have problems performing a benchmark calculation first. The example was using the MgO band structure scrip as given in the tutorial.

However, the firework is fizzled and I received the authentication errors as follow.

Traceback (most recent call last):

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/atomate/utils/database.py”, line 42, in init

self.db.authenticate(self.user, self.password)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/database.py”, line 1274, in authenticate

connect=True)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/mongo_client.py”, line 614, in _cache_credentials

sock_info.authenticate(credentials)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/pool.py”, line 688, in authenticate

auth.authenticate(credentials, self)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 563, in authenticate

auth_func(credentials, sock_info)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 538, in _authenticate_default

return _authenticate_scram(credentials, sock_info, ‘SCRAM-SHA-1’)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/auth.py”, line 262, in _authenticate_scram

res = sock_info.command(source, cmd)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/pool.py”, line 579, in command

unacknowledged=unacknowledged)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/network.py”, line 150, in command

parse_write_concern_error=parse_write_concern_error)

File “/home/zzhang/atomate/atomate_env/lib/python3.6/site-packages/pymongo/helpers.py”, line 155, in _check_command_response

raise OperationFailure(msg % errmsg, code, response)

pymongo.errors.OperationFailure: Authentication failed.

Many thanks for your kind helps.

Best regards,

Zezhong

–

You received this message because you are subscribed to the Google Groups “atomate” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–

You received this message because you are subscribed to the Google Groups “atomate” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

[Zezhong_Zhang1]

Dear Jain,

Thanks for your kind help with this. I have tried the connection code and the results are as commented

d = loadfn(config_file)
conn = MongoClient(host=d["host"], port=d["port"], username=d["admin_user"], password=d["admin_password"], authSource="admin")
db = conn[d["database"]]
db.authenticate(user, passwd) # Return authentication failed
conn.admin.authenticate(user, passwd) # Return True
db.authenticate(user,passwd,source='admin’) # Return True

I found that MongoClient can drop the authSource=“admin” option but the db.authenticate cannot drop source=‘admin’, at least for my Mac OS 10.14.4 and a cray cluster on SLURM 18.08.6.

Yes, now I understand that I can create a user with the read and write right for the ’admin_user” in atomate. Thanks very much for pointing it out.

All the best,

Zezhong

···

On 16 Apr 2019, at 18:22, Anubhav Jain <[email protected]> wrote:

config_file='/path/to/db.json'
d = loadfn(config_file)
conn = MongoClient(host=d["host"], port=d["port"], username=d["admin_user"], password=d["admin_password"], authSource="admin")
db = conn[d["database"]]

[Anubhav_Jain]

Hi Zezhong,

Two notes:

1. You shouldn’t have to call “authenticate()” at all in the code I sent. The username and password are already sent in the MongoClient. Can you try it without calling authenticate() at all?

2. For my note about the “user with the read and write right”, I meant to create a user specifically for your atomate database (whatever the “database” parameter is in your db.json file - this should not be the admin database). That user will have read/write access to the atomate database but not read/write access to the admin database.

···

Best,
Anubhav

[Zezhong_Zhang1]

Dear Jain,

I have ran the code without the authenticate() as the one you originally posted and received the following. I think it means the connection is successful. Shall I wait for the update of the atomate to implement this, and pip install upgrade? Or I modify the change locally?

Database(MongoClient(host=[‘alloyproject-shard-00-01-4qquz.mongodb.net:27017’, ‘alloyproject-shard-00-02-4qquz.mongodb.net:27017’, ‘alloyproject-shard-00-00-4qquz.mongodb.net:27017’], document_class=dict, tz_aware=False, connect=True, authsource=‘admin’, replicaset=‘AlloyProject-shard-0’, ssl=True), 'alloy’)

I see, for the mongoldb atlas that I can create user with specific roles, and I can assign read and write right to the an ‘atomate’ role that has access to an ‘atomate’ database only (as shown in figs attached). Hopefully this will be useful for other people using mongoldb atlas.

Best regard,

Zezhong

···

On 17 Apr 2019, at 00:57, Anubhav Jain [email protected] wrote:

Hi Zezhong,

Two notes:

1. You shouldn’t have to call “authenticate()” at all in the code I sent. The username and password are already sent in the MongoClient. Can you try it without calling authenticate() at all?

2. For my note about the “user with the read and write right”, I meant to create a user specifically for your atomate database (whatever the “database” parameter is in your db.json file - this should not be the admin database). That user will have read/write access to the atomate database but not read/write access to the admin database.

On Tue, Apr 16, 2019 at 2:33 PM ‘Zezhong Zhang’ via atomate [email protected] wrote:
Dear Jain,

Thanks for your kind help with this. I have tried the connection code and the results are as commented

d = loadfn(config_file)
conn = MongoClient(host=d[“host”], port=d[“port”], username=d[“admin_user”], password=d[“admin_password”], authSource=“admin”)
db = conn[d[“database”]]
db.authenticate(user, passwd) # Return authentication failed
conn.admin.authenticate(user, passwd) # Return True
db.authenticate(user,passwd,source='admin’) # Return True

I found that MongoClient can drop the authSource=“admin” option but the db.authenticate cannot drop source=‘admin’, at least for my Mac OS 10.14.4 and a cray cluster on SLURM 18.08.6.

Yes, now I understand that I can create a user with the read and write right for the ’admin_user” in atomate. Thanks very much for pointing it out.

All the best,

Zezhong

On 16 Apr 2019, at 18:22, Anubhav Jain [email protected] wrote:

config_file=’/path/to/db.json’
d = loadfn(config_file)
conn = MongoClient(host=d[“host”], port=d[“port”], username=d[“admin_user”], password=d[“admin_password”], authSource=“admin”)
db = conn[d[“database”]]

–
You received this message because you are subscribed to the Google Groups “atomate” group.
To unsubscribe from this group and stop receiving emails from it, send an email to&[email protected].
To post to this group, send email to&[email protected].
For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

–
You received this message because you are subscribed to the Google Groups “atomate” group.
To unsubscribe from this group and stop receiving emails from it, send an email to&[email protected].
To post to this group, send email to&[email protected].
For more options, visit https://groups.google.com/d/optout.

[Anubhav_Jain]

Hi Zezhong,

Ok great!

I’ll try to update the atomate code on Friday, it’s up to you if you want to i) wait and pip install upgrade or ii) make the changes locally and run in the meantime.

The database screenshots you sent are exactly what I was trying to convey - thanks for sharing so others can see it more clearly!

Best,

Anubhav

···

Best,
Anubhav

[Zezhong_Zhang]

Dear Anubhav,

I have updated my local atomate.utils.get_database, and add source=‘admin’ to CalcDb.init in atomate.utils.database (line42) (otherwise, it reports authentication error here).

Now, I can run the MgO band structure tutorial. Atomate seems to access the database and perform vasp calculation with no problem, as the optimisation and static calculations are fine (I have also checked the vasp output files). But theMgO-nscf uniform is fuzzled, the vasp output file and slurm output file are as attached.

Look forward to getting the atomate working on our machine, quite excited!

Best regards,

Zezhong

slurm-3549193.out (9.08 KB)

vasp.out.gz (635 Bytes)

[Steven_Hartman]

Hi,

I’ve also seen this “nameless” VASP error message recently when running nscf-line calculations. If it’s the same one I’ve had, it happens because the KPOINTS file doesn’t list the tetrahedra, which are required for ISMEAR = -5. I’m not sure if this is a bug or if we’re just doing it wrong. I’ve been working around it by switching to ISMEAR 0, SIGMA 0.01 for nscf-line, using the update_wflows command.

···

On Wednesday, April 24, 2019 at 12:24:09 PM UTC-5, Zezhong Zhang wrote:

Dear Anubhav,

I have updated my local atomate.utils.get_database, and add source=‘admin’ to CalcDb.init in atomate.utils.database (line42) (otherwise, it reports authentication error here).

Now, I can run the MgO band structure tutorial. Atomate seems to access the database and perform vasp calculation with no problem, as the optimisation and static calculations are fine (I have also checked the vasp output files). But theMgO-nscf uniform is fuzzled, the vasp output file and slurm output file are as attached.

Look forward to getting the atomate working on our machine, quite excited!

Best regards,

Zezhong

[Zezhong_Zhang1]

Hi Steven,

For me, the nscf-line is fine but the nscf-uniform is fizzled, probably due to the KPOINTS not tetrahedra as you suggested. The KPOINTS file specifies the location of each k-point instead of using the automatic k-point generation. I guess this is to calculate the dielectric matrix at the last k-point with the KPOINT_BSE = -1 0 0 0 specified in the INCAR.

I have checked the pymatgen.io.vasp.sets that ISMEAR = -5 for nscf-uniform calculation, while nscf-line has ISMEAR = 0 and SIGMA = 0.01 by default.

And I am using the atomate default band structure workflow as:

from pymatgen import Structure

from atomate.vasp.workflows.presets.core import wf_bandstructure

truct = Structure.from_file(‘POSCAR’)

wf = wf_bandstructure(struct)

If I update the nscf-uniform with ISMEAR = 0 and SIGMA = 0.01, what is the difference between uniform and nscf-line?

Cheers,

Zezhong

···

On 24 Apr 2019, at 20:43, Steven Hartman [email protected] wrote:

Hi,
I’ve also seen this “nameless” VASP error message recently when running nscf-line calculations. If it’s the same one I’ve had, it happens because the KPOINTS file doesn’t list the tetrahedra, which are required for ISMEAR = -5. I’m not sure if this is a bug or if we’re just doing it wrong. I’ve been working around it by switching to ISMEAR 0, SIGMA 0.01 for nscf-line, using the update_wflows command.

On Wednesday, April 24, 2019 at 12:24:09 PM UTC-5, Zezhong Zhang wrote:
Dear Anubhav,

I have updated my local atomate.utils.get_database, and add source=‘admin’ to CalcDb.init in atomate.utils.database (line42) (otherwise, it reports authentication error here).

Now, I can run the MgO band structure tutorial. Atomate seems to access the database and perform vasp calculation with no problem, as the optimisation and static calculations are fine (I have also checked the vasp output files). But theMgO-nscf uniform is fuzzled, the vasp output file and slurm output file are as attached.

Look forward to getting the atomate working on our machine, quite excited!

Best regards,

Zezhong

–
You received this message because you are subscribed to the Google Groups “atomate” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to&[email protected].
For more options, visit https://groups.google.com/d/optout.

[Steven_Hartman]

Now that you mention it, I suspect it was actually my nscf-uniform which was failing, unfortunately I deleted those workflows so I can’t go back and check. Anyway, the difference between line and uniform modes is the arrangement of the kpoints. “Uniform” uses an evenly spaced kpoint grid, while “line” only places kpoints along the lines in between high-symmetry points such as gamma, X, R, etc, which is useful for plotting band structure diagrams. I don’t know why the uniform KPONTS file doesn’t include tetrahedra, I think opening an issue on the atomate github would be a good idea.

···

On Thursday, April 25, 2019 at 4:09:45 AM UTC-5, Zezhong Zhang wrote:

Hi Steven,

For me, the nscf-line is fine but the nscf-uniform is fizzled, probably due to the KPOINTS not tetrahedra as you suggested. The KPOINTS file specifies the location of each k-point instead of using the automatic k-point generation. I guess this is to calculate the dielectric matrix at the last k-point with the KPOINT_BSE = -1 0 0 0 specified in the INCAR.

I have checked the pymatgen.io.vasp.sets that ISMEAR = -5 for nscf-uniform calculation, while nscf-line has ISMEAR = 0 and SIGMA = 0.01 by default.

And I am using the atomate default band structure workflow as:

from pymatgen import Structure

from atomate.vasp.workflows.presets.core import wf_bandstructure

truct = Structure.from_file(‘POSCAR’)

wf = wf_bandstructure(struct)

If I update the nscf-uniform with ISMEAR = 0 and SIGMA = 0.01, what is the difference between uniform and nscf-line?

Cheers,

Zezhong

On 24 Apr 2019, at 20:43, Steven Hartman [email protected] wrote:

Hi,
I’ve also seen this “nameless” VASP error message recently when running nscf-line calculations. If it’s the same one I’ve had, it happens because the KPOINTS file doesn’t list the tetrahedra, which are required for ISMEAR = -5. I’m not sure if this is a bug or if we’re just doing it wrong. I’ve been working around it by switching to ISMEAR 0, SIGMA 0.01 for nscf-line, using the update_wflows command.

On Wednesday, April 24, 2019 at 12:24:09 PM UTC-5, Zezhong Zhang wrote:
Dear Anubhav,

I have updated my local atomate.utils.get_database, and add source=‘admin’ to CalcDb.init in atomate.utils.database (line42) (otherwise, it reports authentication error here).

Now, I can run the MgO band structure tutorial. Atomate seems to access the database and perform vasp calculation with no problem, as the optimisation and static calculations are fine (I have also checked the vasp output files). But theMgO-nscf uniform is fuzzled, the vasp output file and slurm output file are as attached.

Look forward to getting the atomate working on our machine, quite excited!

Best regards,

Zezhong

–
You received this message because you are subscribed to the Google Groups “atomate” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
For more options, visit https://groups.google.com/d/optout.

[Zezhong_Zhang1]

Thanks Steven for kind explanation for nscf-uniform & line, I have submitted an issue on the GitHub.

[Marnik_Bercx]

Hey everyone,

To follow up on this: I’ve recently started using atomate and was running into issues with the atomate.vasp.firetasks.parse_outputs.VaspToDb firetask. I traced the problem to the authentication to the mongoDB database, so figured I must have made a mistake in setting up my db.json file. Nothing I tried, however, seemed to work.

I then started playing around with pymongo to see if I could somehow understand the problem. Here’s a little notebook on colab with some of the tests I’ve run. I think the problem lies in the initialization of the atomate.utils.database.CalcDb class, i.e. when testing the authentication using db.authenticate. Even though the “authsource” is passed correctly to the MongoClient initialization, it is not passed to the db.authentication() call, and that causes authentication failures down the line, I think.

I’ve made a pull request with a little fix for this, that simply passes the “authsource” kwargs to the source argument for the db.authenticate call. That fixes the problem for me. Hopefully it doesn’t cause any other issues that I’ve missed.

Kind regards,
Marnik

PS: Thanks to Zezhong for pointing out this morning that using source= "admin" in the db.authenticate call fixed the problem for him. Having an office mate that also uses atomate can be really useful!

​

[Anubhav_Jain]

Thanks! I just merged your changes and will push out in the next release. Sorry for the delay in responding to this.

···

On Tuesday, July 23, 2019 at 4:48:14 AM UTC-7, [email protected] wrote:

Hey everyone,

To follow up on this: I’ve recently started using atomate and was running into issues with the atomate.vasp.firetasks.parse_outputs.VaspToDb firetask. I traced the problem to the authentication to the mongoDB database, so figured I must have made a mistake in setting up my db.json file. Nothing I tried, however, seemed to work.

I then started playing around with pymongo to see if I could somehow understand the problem. Here’s a little notebook on colab with some of the tests I’ve run. I think the problem lies in the initialization of the atomate.utils.database.CalcDb class, i.e. when testing the authentication using db.authenticate. Even though the “authsource” is passed correctly to the MongoClient initialization, it is not passed to the db.authentication() call, and that causes authentication failures down the line, I think.

I’ve made a pull request with a little fix for this, that simply passes the “authsource” kwargs to the source argument for the db.authenticate call. That fixes the problem for me. Hopefully it doesn’t cause any other issues that I’ve missed.

Kind regards,
Marnik

PS: Thanks to Zezhong for pointing out this morning that using source= "admin" in the db.authenticate call fixed the problem for him. Having an office mate that also uses atomate can be really useful!

​

[Fengyu_Xie1]

Thanks so much! Your discussion have also saved me from a whole day of debugging!