Having a firetask sourcing a shell file to set up environment for remaining firetasks in workflow

Hi,

for FireWorks to be maximally useful to me, I want to be able to use it to run calculations with completely different codes, with different environments. For that to work, I cannot set the environment in the my_qadapter.yaml file, but have to do it afterwards (otherwise I have to quit and restart the launcher with different qadapters for each job). I would like to be able to control everything from python input files, so it would work like this:

set_env = some_script_that_sets_the_correct_environment(‘my_shell_file’)

…(whatever other tasks will be in the firework)…

firework = Firework([set_env,
other_tasks])
wflow = Workflow([firework], name=‘my job’)
launchpad.add_wf(wflow)

However, while I can write a PyTask to do this (i.e. PyTask(code_to_run_subprocess_and_run_shell_script)), the environment changes apparently have no effect outside that one PyTask. Once the code goes on to the next task in the Firework, the environment has been reset.

Is there any way to achieve this?

Thanks,
Rasmus

Using a ScriptTask, as so
set_env = ScriptTask.from_str(‘source setupenv.sh’)

setting the firework as follows:

#and submit the workflow to the launchpad
firework = Firework([set_env,
do_something,
do_something_else],
name=‘test’)
wflow = Workflow([firework], name='test)
launchpad.add_wf(wflow)

Still gives errors (“no module named X”) for one of the tasks.

···

On Tuesday, July 12, 2016 at 1:48:08 PM UTC-7, Rasmus Karlsson wrote:

Hi,

for FireWorks to be maximally useful to me, I want to be able to use it to run calculations with completely different codes, with different environments. For that to work, I cannot set the environment in the my_qadapter.yaml file, but have to do it afterwards (otherwise I have to quit and restart the launcher with different qadapters for each job). I would like to be able to control everything from python input files, so it would work like this:

set_env = some_script_that_sets_the_correct_environment(‘my_shell_file’)

…(whatever other tasks will be in the firework)…

firework = Firework([set_env,
other_tasks])
wflow = Workflow([firework], name=‘my job’)
launchpad.add_wf(wflow)

However, while I can write a PyTask to do this (i.e. PyTask(code_to_run_subprocess_and_run_shell_script)), the environment changes apparently have no effect outside that one PyTask. Once the code goes on to the next task in the Firework, the environment has been reset.

Is there any way to achieve this?

Thanks,
Rasmus

My understanding is that you’d like to have two fireworks in the same workflow run with different environments, without having to submit with different queue adapter files.

You can override default queueadapter parameters, including “pre_rocket” commands often used to set up the environment, in the construction of the firework, i. e.

firework = Firework([do_something,

			 do_something_else],

			spec = {“_queueadapter”:{“pre_rocket”:”source setupenv.sh”}}, name=“test”)

Will this achieve what you’re looking to do?

Best,

Joey

···

On Jul 12, 2016, at 2:22 PM, Rasmus Karlsson [email protected] wrote:

Using a ScriptTask, as so
set_env = ScriptTask.from_str(‘source setupenv.sh’)

setting the firework as follows:

#and submit the workflow to the launchpad
firework = Firework([set_env,
do_something,
do_something_else],
name=‘test’)
wflow = Workflow([firework], name='test)
launchpad.add_wf(wflow)

Still gives errors (“no module named X”) for one of the tasks.

On Tuesday, July 12, 2016 at 1:48:08 PM UTC-7, Rasmus Karlsson wrote:

Hi,

for FireWorks to be maximally useful to me, I want to be able to use it to run calculations with completely different codes, with different environments. For that to work, I cannot set the environment in the my_qadapter.yaml file, but have to do it afterwards (otherwise I have to quit and restart the launcher with different qadapters for each job). I would like to be able to control everything from python input files, so it would work like this:

set_env = some_script_that_sets_the_correct_environment(‘my_shell_file’)

…(whatever other tasks will be in the firework)…

firework = Firework([set_env,
other_tasks])
wflow = Workflow([firework], name=‘my job’)
launchpad.add_wf(wflow)

However, while I can write a PyTask to do this (i.e. PyTask(code_to_run_subprocess_and_run_shell_script)), the environment changes apparently have no effect outside that one PyTask. Once the code goes on to the next task in the Firework, the environment has been reset.

Is there any way to achieve this?

Thanks,
Rasmus


You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/19d90304-a123-4831-9742-985d7a927d56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Joey,

Wow, that’s exactly what I needed!

I removed the pre_rocket text from my_qadapter.yaml, and inserted all commands needed to setup the environment in
the spec attribute for the Firework object. I also restarted the launcher with

nohup qlaunch -r rapidfire -m 40 --nlaunches infinite &

the -r argument was needed for the launcher to parse the changes in the spec attribute.

Now it works!

Thank you,
Rasmus

···

On Tuesday, July 12, 2016 at 3:01:23 PM UTC-7, Joseph Montoya wrote:

My understanding is that you’d like to have two fireworks in the same workflow run with different environments, without having to submit with different queue adapter files.

You can override default queueadapter parameters, including “pre_rocket” commands often used to set up the environment, in the construction of the firework, i. e.

firework = Firework([do_something,

  		 do_something_else],
  		spec = {“_queueadapter”:{“pre_rocket”:”source setupenv.sh”}}, name=“test”)

Will this achieve what you’re looking to do?

Best,

Joey

On Jul 12, 2016, at 2:22 PM, Rasmus Karlsson [email protected] wrote:

Using a ScriptTask, as so
set_env = ScriptTask.from_str(‘source setupenv.sh’)

setting the firework as follows:

#and submit the workflow to the launchpad
firework = Firework([set_env,
do_something,
do_something_else],
name=‘test’)
wflow = Workflow([firework], name='test)
launchpad.add_wf(wflow)

Still gives errors (“no module named X”) for one of the tasks.

On Tuesday, July 12, 2016 at 1:48:08 PM UTC-7, Rasmus Karlsson wrote:

Hi,

for FireWorks to be maximally useful to me, I want to be able to use it to run calculations with completely different codes, with different environments. For that to work, I cannot set the environment in the my_qadapter.yaml file, but have to do it afterwards (otherwise I have to quit and restart the launcher with different qadapters for each job). I would like to be able to control everything from python input files, so it would work like this:

set_env = some_script_that_sets_the_correct_environment(‘my_shell_file’)

…(whatever other tasks will be in the firework)…

firework = Firework([set_env,
other_tasks])
wflow = Workflow([firework], name=‘my job’)
launchpad.add_wf(wflow)

However, while I can write a PyTask to do this (i.e. PyTask(code_to_run_subprocess_and_run_shell_script)), the environment changes apparently have no effect outside that one PyTask. Once the code goes on to the next task in the Firework, the environment has been reset.

Is there any way to achieve this?

Thanks,
Rasmus


You received this message because you are subscribed to the Google Groups “fireworkflows” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/fireworkflows.
To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/19d90304-a123-4831-9742-985d7a927d56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.