Can't access LaunchPad attributes in Firetask when using _add_launchpad_and_fw_id with rlaunch multi

However, when I have _add_launchpad_and_fw_id set to True with rlaunch multi it doesn’t seem like the usual LaunchPad attributes (e.g., host, port, launches) are accessible. Inside the Firetask, self.launchpad is a multiprocessing.managers.AutoProxy[LaunchPad] object.

Here is an example:

from fireworks.utilities.fw_utilities import explicit_serialize
from fireworks.core.firework import FireTaskBase
from fireworks import Firework, Workflow, LaunchPad

@explicit_serialize
class AddLaunchpadTask(FireTaskBase):
    _fw_name = "AddLaunchpadTask"

    def run_task(self, fw_spec):
        host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
        print(host, port, name)

if __name__=="__main__":
    launchpad = LaunchPad()
    launchpad.reset(password=None, require_password=False)
    fw = Firework([AddLaunchpadTask()], spec={'_add_launchpad_and_fw_id':True})
    launchpad.add_wf(fw)
    launchpad.add_wf(fw)

Running ‘rlaunch rapidfire’ does not throw an error.

Running ‘rlaunch multi 2’ throws the error:

Traceback (most recent call last):
  File "/Users/alexdunn/env1/lib/python2.7/site-packages/fireworks/core/rocket.py", line 224, in run
    m_action = t.run_task(my_spec)
  File "/Users/alexdunn/tests/add_launchpad_test_case.py", line 11, in run_task
    host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
AttributeError: 'AutoProxy[LaunchPad]' object has no attribute 'host'
    m_action = t.run_task(my_spec)
  File "/Users/alexdunn/tests/add_launchpad_test_case.py", line 11, in run_task
    host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
AttributeError: 'AutoProxy[LaunchPad]' object has no attribute 'host'


There is a class DataServer in fw_utilities.py that looks like it manages LaunchPad objects between multiple processes. I am just not sure how to get LaunchPad data from the AutoProxy[LaunchPad] object though. Has anyone else run into this issue before?

When I have _add_launchpad_and_fw_id set to True with rlaunch rapidfire or rlaunch singleshot I can access all the attributes of the LaunchPad object, such as the host with self.launchpad.host. Inside the Firetask, self.launchpad is a Fireworks.core.launchpad.LaunchPad object.

Hi Alex,

Sorry for the late reply on this. I just pushed a fix via commit: f4cecc535a4db93b081a4d7cced574961797d926

If you pull the latest Github version, you should see it. Otherwise (if you are using pip or conda install), you can wait until I release FWS version v1.4.5.

The issue is that the multiprocessing uses an AutoProxy object to manage the shared LaunchPad. I don’t know all the details of this code since it was from Xiaohui Qu. However, AutoProxy objects cannot access attributes of the enclosing class (e.g., can’t access host, port, name) - they can only access methods of that class.

I couldn’t find a quick workaround so I just hacked this so that we get a brand new LaunchPad object by calling the AutoProxy’s to_dict() method and loading the LaunchPad from that.

Let me know if you see any problems with it.

Y

···

On Tuesday, April 18, 2017 at 7:32:45 PM UTC-7, Alex Dunn wrote:

However, when I have _add_launchpad_and_fw_id set to True with rlaunch multi it doesn’t seem like the usual LaunchPad attributes (e.g., host, port, launches) are accessible. Inside the Firetask, self.launchpad is a multiprocessing.managers.AutoProxy[LaunchPad] object.

Here is an example:

from fireworks.utilities.fw_utilities import explicit_serialize
from fireworks.core.firework import FireTaskBase
from fireworks import Firework, Workflow, LaunchPad

@explicit_serialize
class AddLaunchpadTask(FireTaskBase):
    _fw_name = "AddLaunchpadTask"

    def run_task(self, fw_spec):
        host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
        print(host, port, name)


if __name__=="__main__":
    launchpad = LaunchPad()
    launchpad.reset(password=None, require_password=False)
    fw = Firework([AddLaunchpadTask()], spec={'_add_launchpad_and_fw_id':True})
    launchpad.add_wf(fw)
    launchpad.add_wf(fw)

Running ‘rlaunch rapidfire’ does not throw an error.

Running ‘rlaunch multi 2’ throws the error:

Traceback (most recent call last):
  File "/Users/alexdunn/env1/lib/python2.7/site-packages/fireworks/core/rocket.py", line 224, in run
    m_action = t.run_task(my_spec)
  File "/Users/alexdunn/tests/add_launchpad_test_case.py", line 11, in run_task
    host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
AttributeError: 'AutoProxy[LaunchPad]' object has no attribute 'host'
    m_action = t.run_task(my_spec)
  File "/Users/alexdunn/tests/add_launchpad_test_case.py", line 11, in run_task
    host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
AttributeError: 'AutoProxy[LaunchPad]' object has no attribute 'host'



There is a class DataServer in fw_utilities.py that looks like it manages LaunchPad objects between multiple processes. I am just not sure how to get LaunchPad data from the AutoProxy[LaunchPad] object though. Has anyone else run into this issue before?

When I have _add_launchpad_and_fw_id set to True with rlaunch rapidfire or rlaunch singleshot I can access all the attributes of the LaunchPad object, such as the host with self.launchpad.host. Inside the Firetask, self.launchpad is a Fireworks.core.launchpad.LaunchPad object.

Ok, thanks!

···

On Wed, Jul 5, 2017 at 3:32 PM Anubhav Jain [email protected] wrote:

Hi Alex,

Sorry for the late reply on this. I just pushed a fix via commit: f4cecc535a4db93b081a4d7cced574961797d926

If you pull the latest Github version, you should see it. Otherwise (if you are using pip or conda install), you can wait until I release FWS version v1.4.5.

The issue is that the multiprocessing uses an AutoProxy object to manage the shared LaunchPad. I don’t know all the details of this code since it was from Xiaohui Qu. However, AutoProxy objects cannot access attributes of the enclosing class (e.g., can’t access host, port, name) - they can only access methods of that class.

I couldn’t find a quick workaround so I just hacked this so that we get a brand new LaunchPad object by calling the AutoProxy’s to_dict() method and loading the LaunchPad from that.

Let me know if you see any problems with it.

Y

On Tuesday, April 18, 2017 at 7:32:45 PM UTC-7, Alex Dunn wrote:

When I have _add_launchpad_and_fw_id set to True with rlaunch rapidfire or rlaunch singleshot I can access all the attributes of the LaunchPad object, such as the host with self.launchpad.host. Inside the Firetask, self.launchpad is a Fireworks.core.launchpad.LaunchPad object.

However, when I have _add_launchpad_and_fw_id set to True with rlaunch multi it doesn’t seem like the usual LaunchPad attributes (e.g., host, port, launches) are accessible. Inside the Firetask, self.launchpad is a multiprocessing.managers.AutoProxy[LaunchPad] object.

Here is an example:

from fireworks.utilities.fw_utilities import explicit_serialize
from fireworks.core.firework import FireTaskBase
from fireworks import Firework, Workflow, LaunchPad

@explicit_serialize
class AddLaunchpadTask(FireTaskBase):
    _fw_name = "AddLaunchpadTask"

    def run_task(self, fw_spec):
        host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
        print(host, port, name)


if __name__=="__main__":
    launchpad = LaunchPad()
    launchpad.reset(password=None, require_password=False)
    fw = Firework([AddLaunchpadTask()], spec={'_add_launchpad_and_fw_id':True})
    launchpad.add_wf(fw)
    launchpad.add_wf(fw)

Running ‘rlaunch rapidfire’ does not throw an error.

Running ‘rlaunch multi 2’ throws the error:

Traceback (most recent call last):
  File "/Users/alexdunn/env1/lib/python2.7/site-packages/fireworks/core/rocket.py", line 224, in run
    m_action = t.run_task(my_spec)
  File "/Users/alexdunn/tests/add_launchpad_test_case.py", line 11, in run_task
    host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
AttributeError: 'AutoProxy[LaunchPad]' object has no attribute 'host'
    m_action = t.run_task(my_spec)
  File "/Users/alexdunn/tests/add_launchpad_test_case.py", line 11, in run_task
    host, port, name = [getattr(self.launchpad, k) for k in ('host', 'port', 'name')]
AttributeError: 'AutoProxy[LaunchPad]' object has no attribute 'host'



There is a class DataServer in fw_utilities.py that looks like it manages LaunchPad objects between multiple processes. I am just not sure how to get LaunchPad data from the AutoProxy[LaunchPad] object though. Has anyone else run into this issue before?

You received this message because you are subscribed to a topic in the Google Groups “fireworkflows” group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/fireworkflows/JSshaDjqVUQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/e6e562c0-bca4-4162-bd49-34f7b51a2cd1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alex Dunn

B.S. Chemical Engineering 2017

Lawrence Berkeley Lab

Phone: 530-588-3812

Email: [email protected]