how to make the firetasks via the same firetask class but created by multiple executions of my python script unique

Alex_Dong_Li · June 26, 2015, 12:46pm

Dear Fireworks support,

With Anubhav’s and my teammate’s help, I am able to run subfireworks in parallel. Thanks!

Now I have a different question:

I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.

It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.

I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the “spec” section, but none worked. I also try to set “_dupefinder” in “spec” to “DupFinderExact” in hoping that different “spec” will somehow make the fireworks unique, but then I got this error:

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap

self.run()

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 114, in run

self._target(*self._args, **self._kwargs)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py”, line 70, in rapidfire_process

rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py”, line 68, in rapidfire

while skip_check or launchpad.run_exists(fworker):

File “”, line 2, in run_exists

File “/opt/common/compbio/lib/python2.7/multiprocessing/managers.py”, line 773, in _callmethod

raise convert_to_error(kind, result)

AttributeError: ‘unicode’ object has no attribute ‘query’

So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.

Hope that my question is not too confusing.

Thanks in advance!

Alex.

Anubhav_Jain · June 26, 2015, 3:55pm

Hi Alex,

Sorry, but I am not able to understand your question.

Note that duplicate checking is off by default. So in the default case, you can add identical workflows 100 times and FWS will run it 100 times. It will not try to do anything clever unless you explicitly turn duplicate checking on via the “_dupefinder” spec key

Looking at the error, it looks like it is unrelated to your description. Instead, it looks like “fworker” object being passed into launchpad.run_exists() is a string/unicode object instead of a proper FWorker object. Are you sure you are passing an FWorker object into your launch, e.g. launch_multiprocess()? If not, please do this and refer to the comment strings to see the type of object needed to be passed.

Best,

Anubhav

···

On Fri, Jun 26, 2015 at 5:46 AM, Alex Li [email protected] wrote:

Dear Fireworks support,

With Anubhav’s and my teammate’s help, I am able to run subfireworks in parallel. Thanks!

Now I have a different question:

I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.

It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.

I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the “spec” section, but none worked. I also try to set “_dupefinder” in “spec” to “DupFinderExact” in hoping that different “spec” will somehow make the fireworks unique, but then I got this error:

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap

self.run()

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 114, in run

self._target(*self._args, **self._kwargs)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py”, line 70, in rapidfire_process

rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py”, line 68, in rapidfire

while skip_check or launchpad.run_exists(fworker):

File “”, line 2, in run_exists

File “/opt/common/compbio/lib/python2.7/multiprocessing/managers.py”, line 773, in _callmethod

raise convert_to_error(kind, result)

AttributeError: ‘unicode’ object has no attribute ‘query’

So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.

Hope that my question is not too confusing.

Thanks in advance!

Alex.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/dae8a09f-a5be-4217-a43c-d894bb826f36%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alex_Dong_Li · June 26, 2015, 6:50pm

Anubhav,

Thanks for clarifying the default behavior of dupe checking!

There is no internal collision inside a script (ie: calling the same firetask module in two subworkflows in one python script is fine). However, there is somehow a cross-collision between the same subworkflow in two concurrent executions of the same script (using the different parameters and writing to different output files). I will try to narrow down the problem and come up with a simplest script that can reproduce the problem.

Regards,

Alex.

···

On Friday, June 26, 2015 at 10:55:51 AM UTC-5, Anubhav Jain wrote:

Hi Alex,

Sorry, but I am not able to understand your question.

Note that duplicate checking is off by default. So in the default case, you can add identical workflows 100 times and FWS will run it 100 times. It will not try to do anything clever unless you explicitly turn duplicate checking on via the “_dupefinder” spec key

Looking at the error, it looks like it is unrelated to your description. Instead, it looks like “fworker” object being passed into launchpad.run_exists() is a string/unicode object instead of a proper FWorker object. Are you sure you are passing an FWorker object into your launch, e.g. launch_multiprocess()? If not, please do this and refer to the comment strings to see the type of object needed to be passed.

Best,

Anubhav

On Fri, Jun 26, 2015 at 5:46 AM, Alex Li [email protected] wrote:

Dear Fireworks support,

With Anubhav’s and my teammate’s help, I am able to run subfireworks in parallel. Thanks!

Now I have a different question:

I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.

It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.

I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the “spec” section, but none worked. I also try to set “_dupefinder” in “spec” to “DupFinderExact” in hoping that different “spec” will somehow make the fireworks unique, but then I got this error:

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap

self.run()

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 114, in run

self._target(*self._args, **self._kwargs)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py”, line 70, in rapidfire_process

rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py”, line 68, in rapidfire

while skip_check or launchpad.run_exists(fworker):

File “”, line 2, in run_exists

File “/opt/common/compbio/lib/python2.7/multiprocessing/managers.py”, line 773, in _callmethod

raise convert_to_error(kind, result)

AttributeError: ‘unicode’ object has no attribute ‘query’

So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.

Hope that my question is not too confusing.

Thanks in advance!

Alex.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/dae8a09f-a5be-4217-a43c-d894bb826f36%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Anubhav_Jain · June 26, 2015, 6:54pm

I will try to narrow down the problem and come up with a simplest script that can reproduce the problem

Ok, I think that will be needed in order to debug

Best,

Anubhav

···

On Fri, Jun 26, 2015 at 11:50 AM, Alex Li [email protected] wrote:

Anubhav,

Thanks for clarifying the default behavior of dupe checking!

There is no internal collision inside a script (ie: calling the same firetask module in two subworkflows in one python script is fine). However, there is somehow a cross-collision between the same subworkflow in two concurrent executions of the same script (using the different parameters and writing to different output files). I will try to narrow down the problem and come up with a simplest script that can reproduce the problem.

Regards,

Alex.

On Friday, June 26, 2015 at 10:55:51 AM UTC-5, Anubhav Jain wrote:

Hi Alex,

Sorry, but I am not able to understand your question.

Note that duplicate checking is off by default. So in the default case, you can add identical workflows 100 times and FWS will run it 100 times. It will not try to do anything clever unless you explicitly turn duplicate checking on via the “_dupefinder” spec key

Looking at the error, it looks like it is unrelated to your description. Instead, it looks like “fworker” object being passed into launchpad.run_exists() is a string/unicode object instead of a proper FWorker object. Are you sure you are passing an FWorker object into your launch, e.g. launch_multiprocess()? If not, please do this and refer to the comment strings to see the type of object needed to be passed.

Best,

Anubhav

On Fri, Jun 26, 2015 at 5:46 AM, Alex Li [email protected] wrote:

Dear Fireworks support,

With Anubhav’s and my teammate’s help, I am able to run subfireworks in parallel. Thanks!

Now I have a different question:

I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.

It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.

I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the “spec” section, but none worked. I also try to set “_dupefinder” in “spec” to “DupFinderExact” in hoping that different “spec” will somehow make the fireworks unique, but then I got this error:

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap

self.run()

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 114, in run

self._target(*self._args, **self._kwargs)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py”, line 70, in rapidfire_process

rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py”, line 68, in rapidfire

while skip_check or launchpad.run_exists(fworker):

File “”, line 2, in run_exists

File “/opt/common/compbio/lib/python2.7/multiprocessing/managers.py”, line 773, in _callmethod

raise convert_to_error(kind, result)

AttributeError: ‘unicode’ object has no attribute ‘query’

So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.

Hope that my question is not too confusing.

Thanks in advance!

Alex.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/dae8a09f-a5be-4217-a43c-d894bb826f36%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8362d66b-c3e2-4923-a97c-cb084f3c8cc1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alex_Dong_Li · July 3, 2015, 2:09pm

Anubhav,

I think that we have solved the problem.

Problem recap:

When two python scripts (can be the same script with the different input/output data) run at the same time, one script tries to run the fw_ids created by anther script, and vice versa.

How we solved the problem:

When creating subfirework objects, pass a session unique string to fireworks objects

Pass this session string to launchpad when creating the launchpad object

Modified Method _get_a_fw_to_run() in launchpad class to make it optionally use the session string to narrow down the mongodb results to the session’s fw_ids.

Suggestion for future development:

To optionally allow a python script to run the jobs for the fw_ids it has created ONLY, and not run the jobs for the fw_ids created by another python script.

Thank you very much anyway.

Alex

···

On Friday, June 26, 2015 at 1:55:38 PM UTC-5, Anubhav Jain wrote:

I will try to narrow down the problem and come up with a simplest script that can reproduce the problem

Ok, I think that will be needed in order to debug

Best,

Anubhav

On Fri, Jun 26, 2015 at 11:50 AM, Alex Li [email protected] wrote:

Anubhav,

Thanks for clarifying the default behavior of dupe checking!

There is no internal collision inside a script (ie: calling the same firetask module in two subworkflows in one python script is fine). However, there is somehow a cross-collision between the same subworkflow in two concurrent executions of the same script (using the different parameters and writing to different output files). I will try to narrow down the problem and come up with a simplest script that can reproduce the problem.

Regards,

Alex.

On Friday, June 26, 2015 at 10:55:51 AM UTC-5, Anubhav Jain wrote:

Hi Alex,

Sorry, but I am not able to understand your question.

Note that duplicate checking is off by default. So in the default case, you can add identical workflows 100 times and FWS will run it 100 times. It will not try to do anything clever unless you explicitly turn duplicate checking on via the “_dupefinder” spec key

Looking at the error, it looks like it is unrelated to your description. Instead, it looks like “fworker” object being passed into launchpad.run_exists() is a string/unicode object instead of a proper FWorker object. Are you sure you are passing an FWorker object into your launch, e.g. launch_multiprocess()? If not, please do this and refer to the comment strings to see the type of object needed to be passed.

Best,

Anubhav

On Fri, Jun 26, 2015 at 5:46 AM, Alex Li [email protected] wrote:

Dear Fireworks support,

With Anubhav’s and my teammate’s help, I am able to run subfireworks in parallel. Thanks!

Now I have a different question:

I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.

It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.

I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the “spec” section, but none worked. I also try to set “_dupefinder” in “spec” to “DupFinderExact” in hoping that different “spec” will somehow make the fireworks unique, but then I got this error:

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap

self.run()

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 114, in run

self._target(*self._args, **self._kwargs)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py”, line 70, in rapidfire_process

rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py”, line 68, in rapidfire

while skip_check or launchpad.run_exists(fworker):

File “”, line 2, in run_exists

File “/opt/common/compbio/lib/python2.7/multiprocessing/managers.py”, line 773, in _callmethod

raise convert_to_error(kind, result)

AttributeError: ‘unicode’ object has no attribute ‘query’

So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.

Hope that my question is not too confusing.

Thanks in advance!

Alex.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/dae8a09f-a5be-4217-a43c-d894bb826f36%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8362d66b-c3e2-4923-a97c-cb084f3c8cc1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Anubhav_Jain · July 5, 2015, 12:42am

Hi Alex

One of the points of having a database is so that multiple scripts can enter jobs and multiple scripts can execute jobs. To be clear, this is not a “problem” in the code. If you want to maintain separate lists of jobs, you can either have two separate databases or use the FWorker features to pull only the appropriate jobs. No “hacks” are needed. For details on the second strategy see the second half of this documentation for setting categories of jobs:

https://pythonhosted.org/FireWorks/controlworker.html

I’m happy that you found a solution but I also don’t want to propagate a notion that such a change to the code is necessary. The built in tools let you easily control what jobs get pulled by what scripts.

Best,

Anubhav

···

On Jul 3, 2015, at 7:09 AM, Alex Li [email protected] wrote:

Anubhav,

I think that we have solved the problem.

Problem recap:

When two python scripts (can be the same script with the different input/output data) run at the same time, one script tries to run the fw_ids created by anther script, and vice versa.

How we solved the problem:

When creating subfirework objects, pass a session unique string to fireworks objects

Pass this session string to launchpad when creating the launchpad object

Modified Method _get_a_fw_to_run() in launchpad class to make it optionally use the session string to narrow down the mongodb results to the session’s fw_ids.

Suggestion for future development:

To optionally allow a python script to run the jobs for the fw_ids it has created ONLY, and not run the jobs for the fw_ids created by another python script.

Thank you very much anyway.

Alex

On Friday, June 26, 2015 at 1:55:38 PM UTC-5, Anubhav Jain wrote:

I will try to narrow down the problem and come up with a simplest script that can reproduce the problem

Ok, I think that will be needed in order to debug

Best,

Anubhav

On Fri, Jun 26, 2015 at 11:50 AM, Alex Li [email protected] wrote:

Anubhav,

Thanks for clarifying the default behavior of dupe checking!

There is no internal collision inside a script (ie: calling the same firetask module in two subworkflows in one python script is fine). However, there is somehow a cross-collision between the same subworkflow in two concurrent executions of the same script (using the different parameters and writing to different output files). I will try to narrow down the problem and come up with a simplest script that can reproduce the problem.

Regards,

Alex.

On Friday, June 26, 2015 at 10:55:51 AM UTC-5, Anubhav Jain wrote:

Hi Alex,

Sorry, but I am not able to understand your question.

Note that duplicate checking is off by default. So in the default case, you can add identical workflows 100 times and FWS will run it 100 times. It will not try to do anything clever unless you explicitly turn duplicate checking on via the “_dupefinder” spec key

Looking at the error, it looks like it is unrelated to your description. Instead, it looks like “fworker” object being passed into launchpad.run_exists() is a string/unicode object instead of a proper FWorker object. Are you sure you are passing an FWorker object into your launch, e.g. launch_multiprocess()? If not, please do this and refer to the comment strings to see the type of object needed to be passed.

Best,

Anubhav

On Fri, Jun 26, 2015 at 5:46 AM, Alex Li [email protected] wrote:

Dear Fireworks support,

With Anubhav’s and my teammate’s help, I am able to run subfireworks in parallel. Thanks!

Now I have a different question:

I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.

It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.

I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the “spec” section, but none worked. I also try to set “_dupefinder” in “spec” to “DupFinderExact” in hoping that different “spec” will somehow make the fireworks unique, but then I got this error:

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap

self.run()

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 114, in run

self._target(*self._args, **self._kwargs)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py”, line 70, in rapidfire_process

rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py”, line 68, in rapidfire

while skip_check or launchpad.run_exists(fworker):

File “”, line 2, in run_exists

File “/opt/common/compbio/lib/python2.7/multiprocessing/managers.py”, line 773, in _callmethod

raise convert_to_error(kind, result)

AttributeError: ‘unicode’ object has no attribute ‘query’

So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.

Hope that my question is not too confusing.

Thanks in advance!

Alex.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/dae8a09f-a5be-4217-a43c-d894bb826f36%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8362d66b-c3e2-4923-a97c-cb084f3c8cc1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/39095136-a314-4f47-a6e1-134430d11bde%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alex_Dong_Li · July 5, 2015, 8:32pm

Anubhav,

I tried setting the “_category” to runId and in “spec” and use FWorker(category=runId) and it worked!

I guess that I did not understand how the categroy is used when I was reading the documentation before.

Thanks for the information!

Regards,

Alex.

···

On Sat, Jul 4, 2015 at 7:42 PM, Anubhav Jain [email protected] wrote:

Hi Alex

One of the points of having a database is so that multiple scripts can enter jobs and multiple scripts can execute jobs. To be clear, this is not a “problem” in the code. If you want to maintain separate lists of jobs, you can either have two separate databases or use the FWorker features to pull only the appropriate jobs. No “hacks” are needed. For details on the second strategy see the second half of this documentation for setting categories of jobs:

https://pythonhosted.org/FireWorks/controlworker.html

I’m happy that you found a solution but I also don’t want to propagate a notion that such a change to the code is necessary. The built in tools let you easily control what jobs get pulled by what scripts.

Best,

Anubhav

On Jul 3, 2015, at 7:09 AM, Alex Li [email protected] wrote:

Anubhav,

I think that we have solved the problem.

Problem recap:

When two python scripts (can be the same script with the different input/output data) run at the same time, one script tries to run the fw_ids created by anther script, and vice versa.

How we solved the problem:

When creating subfirework objects, pass a session unique string to fireworks objects

Pass this session string to launchpad when creating the launchpad object

Modified Method _get_a_fw_to_run() in launchpad class to make it optionally use the session string to narrow down the mongodb results to the session’s fw_ids.

Suggestion for future development:

To optionally allow a python script to run the jobs for the fw_ids it has created ONLY, and not run the jobs for the fw_ids created by another python script.

Thank you very much anyway.

Alex

On Friday, June 26, 2015 at 1:55:38 PM UTC-5, Anubhav Jain wrote:

I will try to narrow down the problem and come up with a simplest script that can reproduce the problem

Ok, I think that will be needed in order to debug

Best,

Anubhav

On Fri, Jun 26, 2015 at 11:50 AM, Alex Li [email protected] wrote:

Anubhav,

Thanks for clarifying the default behavior of dupe checking!

There is no internal collision inside a script (ie: calling the same firetask module in two subworkflows in one python script is fine). However, there is somehow a cross-collision between the same subworkflow in two concurrent executions of the same script (using the different parameters and writing to different output files). I will try to narrow down the problem and come up with a simplest script that can reproduce the problem.

Regards,

Alex.

On Friday, June 26, 2015 at 10:55:51 AM UTC-5, Anubhav Jain wrote:

Hi Alex,

Sorry, but I am not able to understand your question.

Note that duplicate checking is off by default. So in the default case, you can add identical workflows 100 times and FWS will run it 100 times. It will not try to do anything clever unless you explicitly turn duplicate checking on via the “_dupefinder” spec key

Looking at the error, it looks like it is unrelated to your description. Instead, it looks like “fworker” object being passed into launchpad.run_exists() is a string/unicode object instead of a proper FWorker object. Are you sure you are passing an FWorker object into your launch, e.g. launch_multiprocess()? If not, please do this and refer to the comment strings to see the type of object needed to be passed.

Best,

Anubhav

On Fri, Jun 26, 2015 at 5:46 AM, Alex Li [email protected] wrote:

Dear Fireworks support,

With Anubhav’s and my teammate’s help, I am able to run subfireworks in parallel. Thanks!

Now I have a different question:

I have a python script that creates a few subworkflows and processes the input data and writes the output files into the unique directories each time it runs.

It runs fine alone (ie no concurrent multiple executions of the same script). However, if I run the same script twice with the different parameters at the same time (or their run times overlap), one would interfere another, eg: the second run would try to use the output files from the first run, which should not as it should use its own unique files.

I played a few tricks, eg: by giving each subworkflow a unique fw_id, name, and by setting the unique values (input/output files) in the “spec” section, but none worked. I also try to set “_dupefinder” in “spec” to “DupFinderExact” in hoping that different “spec” will somehow make the fireworks unique, but then I got this error:

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 258, in _bootstrap

self.run()

File “/opt/common/compbio/lib/python2.7/multiprocessing/process.py”, line 114, in run

self._target(*self._args, **self._kwargs)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/features/multi_launcher.py”, line 70, in rapidfire_process

rapidfire(launchpad, fworker, None, nlaunches, -1, sleep, loglvl)

File “/opt/common/compbio/lib/python2.7/site-packages/fireworks/core/rocket_launcher.py”, line 68, in rapidfire

while skip_check or launchpad.run_exists(fworker):

File “”, line 2, in run_exists

File “/opt/common/compbio/lib/python2.7/multiprocessing/managers.py”, line 773, in _callmethod

raise convert_to_error(kind, result)

AttributeError: ‘unicode’ object has no attribute ‘query’

So could you please help again? Basically I want to make the fireworks think that the firetasks via the same firetask class created by multiple executions of my python script unique.

Hope that my question is not too confusing.

Thanks in advance!

Alex.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/dae8a09f-a5be-4217-a43c-d894bb826f36%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/8362d66b-c3e2-4923-a97c-cb084f3c8cc1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/39095136-a314-4f47-a6e1-134430d11bde%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.