Dear Fireworks support,
With Anubhav’s and my teammate’s help, I am able to run subfireworks in parallel. Thanks!
Now I have a different question:
I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.
It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.
I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the “spec” section, but none worked. I also try to set “_dupefinder” in “spec” to “DupFinderExact” in hoping that different “spec” will somehow make the fireworks unique, but then I got this error:
File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap
File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 114, in run
File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py”, line 70, in rapidfire_process
rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)
File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py”, line 68, in rapidfire
while skip_check or launchpad.run_exists(fworker):
File “”, line 2, in run_exists
File “/opt/common/compbio/lib/python2.7/multiprocessing/managers.py”, line 773, in _callmethod
raise convert_to_error(kind, result)
AttributeError: ‘unicode’ object has no attribute ‘query’
So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.
Hope that my question is not too confusing.
Thanks in advance!