Workflow of workflows

Leip · April 2, 2019, 1:36pm

Hi,

I have a very simple workflow using only PyTask and custom functions. This linear workflow takes one input and then calculate stuff using this input.

Then, I needed to calculate lots of different inputs. Easy, I just used a loop:

for i, work_path in enumerate(work_paths):
fw2a = Firework(PyTask(func=‘workflow_functions.run_command’, […])
fw2b = Firework(PyTask(func=‘workflow_functions.run_command’, […])

wf = Workflow([fw2a, fw2b, [...]],
              {fw2a: fw2b,
               [...]},
              name="Workflow %s" % str(i))

launchpad.add_wf(wf)

rapidfire(launchpad,
FWorker(),
CommonAdapter(“SGE”,
“fireworks_queue”,
rocket_launch="rlaunch -l s singleshot" os.path.join(root_path, “fireworks/my_launchpad.yaml”)),
reserve=False)

This is working fine. But now I need to join all those workflows to use their outputs as input for a “finishing” workflow. How can I do that?

I would have:

wf1 wf2 […] wfn

\ | /

final_workflow

Thanks!

Anubhav_Jain · April 2, 2019, 8:38pm

Hi Leip

Each workflow is independent of the others (by definition), so you can’t join a final workflow to the other workflows.

Typically, if there are job dependencies like what you describe, everything would go in a single workflow. So you would write one workflow that merges together all the other workflows. Note that the Workflow() object has an “append_wf()” function that helps you merge Workflows in case you are interested in this option.

If you really want to keep everything as separate workflows, the best you can do is set the priority of the Fireworks in the “finishing” workflow to be very low, e.g., negative numbers, and the priority of your “main” workflow Fireworks to be high, e.g. positive numbers. This hack will make sure that your “main” workflows will run before your finishing workflow. However, note that this hack won’t do things like (i) not run the final workflow if one of the main one doesn’t finish or (ii) automatically rerun the finishing workflow if you decide to rerun one of the the main workflow. For those sorts of behaviors you’d need to put things within the same workflow.

Best,

Anubhav

···

On Tuesday, April 2, 2019 at 6:36:43 AM UTC-7, Leip wrote:

Hi,