Some questions about how to share files with multi accounts

Questions

  1. When I use many accounts ( on the same host ) to submit bandstructure_wfs to queue, error raises ( please see Error message ). How to solve it.

  2. How to share files with multi accounts through ssh.

  3. How to run all fireworks of a workflow in a queue, and protect fireworks against being ran in another queue. Or can I force the optimization, the static, the DOS calculation be ran in the same account?( Can I change my_qadapter.yaml to archive it? )

Error message

PermissionError: [Errno 13] Permission denied: ‘/data/home/ypliu/atomate/jobs/block_2018-04-25-13-54-44-462553/launcher_2018-04-25-13-55-03-468076’

Files

There is my config

my_qadapter.yaml:


_fw_name: CommonAdapter

_fw_q_type: LoadSharingFacility

_fw_template_file: /data/home/jbzhao/atomate/config/LSF_template.bsub

rocket_launch: rlaunch -c /data/home/jbzhao/atomate/config singleshot

nodes: 28

ptile: 28

queue: q2680v4m128

walltime: 24:00

account: null

job_name: vasp_test

pre_rocket: module add vasp/5.4.1

post_rocket: null

logdir: null

  1. You need to contact a system administrator to help you make sure all your directories have read/write privileges for all your accounts. You can create a Unix group for this. However, this is not an atomate problem, it has to do with permissions on your filesystem.

  2. I am not sure the question, this sounds like you just need to learn to use the ssh command which is not related to atomate

  3. Currently your “rocket_launch” parameter in your my_qadapter.yaml is running a “singleshot” command. This will run only a single Firework (VASP job) for each queue submission. You can change this command to rapidfire to run many jobs in the same command. See the options on rapidfire for how to achieve this as well as for further options. It is covered in the documentation for the FireWorks code as well as by typing “rlaunch rapidfire -h” to get the help documentation for this command.

I am a little confused by the questions, but you might also want to have different atomate installations for different accounts. i.e., separate databases and separate installations, one for each account you want to run under. But this depends on what you are trying to do.

···

On Friday, April 27, 2018 at 11:47:46 AM UTC-7, [email protected] wrote:

Questions

  1. When I use many accounts ( on the same host ) to submit bandstructure_wfs to queue, error raises ( please see Error message ). How to solve it.
  1. How to share files with multi accounts through ssh.
  1. How to run all fireworks of a workflow in a queue, and protect fireworks against being ran in another queue. Or can I force the optimization, the static, the DOS calculation be ran in the same account?( Can I change my_qadapter.yaml to archive it? )

Error message

PermissionError: [Errno 13] Permission denied: ‘/data/home/ypliu/atomate/jobs/block_2018-04-25-13-54-44-462553/launcher_2018-04-25-13-55-03-468076’

Files

There is my config

my_qadapter.yaml:

_fw_name: CommonAdapter

_fw_q_type: LoadSharingFacility

_fw_template_file: /data/home/jbzhao/atomate/config/LSF_template.bsub

rocket_launch: rlaunch -c /data/home/jbzhao/atomate/config singleshot

nodes: 28

ptile: 28

queue: q2680v4m128

walltime: 24:00

account: null

job_name: vasp_test

pre_rocket: module add vasp/5.4.1

post_rocket: null

logdir: null

Thank you for your replay.

Although when I change ‘ralunch singleshot’ to ‘ralunch rapidfire —nlaunches 4’ , it seems only change the number of FWs I can run in a queue. But it seems cannot force run 4 FWs of the same workflow in queue. If I want to run a workflow (not FW) in a queue, what can I do? ( to run certain FWs by ids can be a solution? )

Best wishes.

在 2018年4月28日星期六 UTC+8上午5:22:28,Anubhav Jain写道:

···
  1. You need to contact a system administrator to help you make sure all your directories have read/write privileges for all your accounts. You can create a Unix group for this. However, this is not an atomate problem, it has to do with permissions on your filesystem.
  2. I am not sure the question, this sounds like you just need to learn to use the ssh command which is not related to atomate
  1. Currently your “rocket_launch” parameter in your my_qadapter.yaml is running a “singleshot” command. This will run only a single Firework (VASP job) for each queue submission. You can change this command to rapidfire to run many jobs in the same command. See the options on rapidfire for how to achieve this as well as for further options. It is covered in the documentation for the FireWorks code as well as by typing “rlaunch rapidfire -h” to get the help documentation for this command.

I am a little confused by the questions, but you might also want to have different atomate installations for different accounts. i.e., separate databases and separate installations, one for each account you want to run under. But this depends on what you are trying to do.

On Friday, April 27, 2018 at 11:47:46 AM UTC-7, [email protected] wrote:

Questions

  1. When I use many accounts ( on the same host ) to submit bandstructure_wfs to queue, error raises ( please see Error message ). How to solve it.
  1. How to share files with multi accounts through ssh.
  1. How to run all fireworks of a workflow in a queue, and protect fireworks against being ran in another queue. Or can I force the optimization, the static, the DOS calculation be ran in the same account?( Can I change my_qadapter.yaml to archive it? )

Error message

PermissionError: [Errno 13] Permission denied: ‘/data/home/ypliu/atomate/jobs/block_2018-04-25-13-54-44-462553/launcher_2018-04-25-13-55-03-468076’

Files

There is my config

my_qadapter.yaml:

_fw_name: CommonAdapter

_fw_q_type: LoadSharingFacility

_fw_template_file: /data/home/jbzhao/atomate/config/LSF_template.bsub

rocket_launch: rlaunch -c /data/home/jbzhao/atomate/config singleshot

nodes: 28

ptile: 28

queue: q2680v4m128

walltime: 24:00

account: null

job_name: vasp_test

pre_rocket: module add vasp/5.4.1

post_rocket: null

logdir: null

It is true that the rlaunch command will just pull whatever Firework is READY to run. There is no way to guarantee that it will pull jobs from the same workflow. This is part of the philosophy of Fireworks being independent jobs.

Can I ask why you need to ensure that a single qlaunch job runs only a single workflow? e.g., why can’t it run more than that or why can’t it run different jobs? I really don’t see the use case for this. For example, if you need all your jobs to run in a particular directory, there are other ways (like “_launch_dir”) to accomplish that.

Some partial workarounds:

  • you can change the rocket_launch command inside my_qadapter.yaml to be a series of “rlaunch singleshot --fwid <FW_ID>”. However I don’t really recommend this as it will be a pain for you to set up and manage.

  • you can set the priority of Fireworks to control the order in which Fireworks are pulled. See: https://materialsproject.github.io/fireworks/priority_tutorial.html . This won’t really give you what you are asking for though

Anubhav

···

On Sunday, April 29, 2018 at 6:30:55 PM UTC-7, [email protected] wrote:

Thank you for your replay.

Although when I change ‘ralunch singleshot’ to ‘ralunch rapidfire —nlaunches 4’ , it seems only change the number of FWs I can run in a queue. But it seems cannot force run 4 FWs of the same workflow in queue. If I want to run a workflow (not FW) in a queue, what can I do? ( to run certain FWs by ids can be a solution? )

Best wishes.

在 2018年4月28日星期六 UTC+8上午5:22:28,Anubhav Jain写道:

  1. You need to contact a system administrator to help you make sure all your directories have read/write privileges for all your accounts. You can create a Unix group for this. However, this is not an atomate problem, it has to do with permissions on your filesystem.
  2. I am not sure the question, this sounds like you just need to learn to use the ssh command which is not related to atomate
  1. Currently your “rocket_launch” parameter in your my_qadapter.yaml is running a “singleshot” command. This will run only a single Firework (VASP job) for each queue submission. You can change this command to rapidfire to run many jobs in the same command. See the options on rapidfire for how to achieve this as well as for further options. It is covered in the documentation for the FireWorks code as well as by typing “rlaunch rapidfire -h” to get the help documentation for this command.

I am a little confused by the questions, but you might also want to have different atomate installations for different accounts. i.e., separate databases and separate installations, one for each account you want to run under. But this depends on what you are trying to do.

On Friday, April 27, 2018 at 11:47:46 AM UTC-7, [email protected] wrote:

Questions

  1. When I use many accounts ( on the same host ) to submit bandstructure_wfs to queue, error raises ( please see Error message ). How to solve it.
  1. How to share files with multi accounts through ssh.
  1. How to run all fireworks of a workflow in a queue, and protect fireworks against being ran in another queue. Or can I force the optimization, the static, the DOS calculation be ran in the same account?( Can I change my_qadapter.yaml to archive it? )

Error message

PermissionError: [Errno 13] Permission denied: ‘/data/home/ypliu/atomate/jobs/block_2018-04-25-13-54-44-462553/launcher_2018-04-25-13-55-03-468076’

Files

There is my config

my_qadapter.yaml:

_fw_name: CommonAdapter

_fw_q_type: LoadSharingFacility

_fw_template_file: /data/home/jbzhao/atomate/config/LSF_template.bsub

rocket_launch: rlaunch -c /data/home/jbzhao/atomate/config singleshot

nodes: 28

ptile: 28

queue: q2680v4m128

walltime: 24:00

account: null

job_name: vasp_test

pre_rocket: module add vasp/5.4.1

post_rocket: null

logdir: null