Parsing more information from "calcs_reversed" to output in task

Atomate
1

Dear Atomate Team

I’ve encountered some difficulty customizing a the atomate.vasp.fireworks.core.LepsFW to output more information into the task document than the default.

In the original firework, the LepsFW terminates after generating a task document in MongoDB that reports the following fields in the output taskdoc: epsilon_static, epsilon_static_wolfe, epsilon_ionic, piezo_ionic_tensor, piezo_tensor. However, since the DFPT calculation generates other information in calcs_reversed/output section as well, such as force_constants, normalmode_eigenvals etc, I want to customize the firework to report these information in the output taskdoc too.

May I know what is the cleanest way of doing this?

What I have tried is the following:

The Original LepsFW parses the vasp output to MongoDB tasks collection using the atomate.vasp.firetasks.parse_output.VaspToDb, which in turn invokes the atomate.vasp.drones.VaspDrone to do the actual parsing. I made copies of

  • atomate/vasp/firework/core.py

  • atomate/vasp/firetasks/parse_outputs.py

  • atomate/vasp/drones.py
    in my own user directory in a folder named “custom” and inserted a sys.path.append command in both core.py and parse_output.py so that the modified files load each of their subsidiary from

  • custom.atomate.vasp.firetasks.parse_output.VaspToDb

  • custom.vasp.drones.VaspDrone

The modified drones.py’s line 299 is then edited to include terms such as “force_constants” etc. so that they will be parsed to the “output” taskdoc.

I thought this will be sufficient but the following error popped up:

···

Traceback (most recent call last):

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1461, in _refresh_wf

updated_ids = wf.refresh(fw_id)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py”, line 1005, in refresh

updated_ids = updated_ids.union(self.apply_action(m_action, fw.fw_id))

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py”, line 813, in apply_action

apply_mod(mod, self.id_fw[cfid].spec)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1765, in spec

return self.partial_fw.spec

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1838, in partial_fw

self._fw = Firework.from_dict(data)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 155, in _decorator

new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 155, in

new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 118, in _recursive_load

return {k: _recursive_load(v) for k, v in obj.items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 118, in

return {k: _recursive_load(v) for k, v in obj.items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 121, in _recursive_load

return [_recursive_load(v) for v in obj]

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 121, in

return [_recursive_load(v) for v in obj]

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 113, in _recursive_load

return load_object(obj)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 329, in load_object

mod = import(modname, globals(), locals(), [classname], 0)

ModuleNotFoundError: No module named ‘custom’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/rocket.py”, line 357, in run

lp.complete_launch(launch_id, m_action, final_state)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1272, in complete_launch

self._refresh_wf(fw_id)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1476, in _refresh_wf

raise RuntimeError(err_message)

RuntimeError: Error refreshing workflow. The full stack trace is: Traceback (most recent call last):

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1461, in _refresh_wf

updated_ids = wf.refresh(fw_id)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py”, line 1005, in refresh

updated_ids = updated_ids.union(self.apply_action(m_action, fw.fw_id))

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py”, line 813, in apply_action

apply_mod(mod, self.id_fw[cfid].spec)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1765, in spec

return self.partial_fw.spec

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1838, in partial_fw

self._fw = Firework.from_dict(data)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 155, in _decorator

new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 155, in

new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 118, in _recursive_load

return {k: _recursive_load(v) for k, v in obj.items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 118, in

return {k: _recursive_load(v) for k, v in obj.items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 121, in _recursive_load

return [_recursive_load(v) for v in obj]

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 121, in

return [_recursive_load(v) for v in obj]

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 113, in _recursive_load

return load_object(obj)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 329, in load_object

mod = import(modname, globals(), locals(), [classname], 0)

ModuleNotFoundError: No module named ‘custom’

It seems that there are more files and classes involved even after drones.py; however, I am unable to locate them. At this point, the process starts to look very complicated.

May I know if there is a cleaner way of achieving this objective? Or if not, which other files should I modify to overcome this error?

Best wishes

Yaze

p.s.

I’ve tried modifying task_materials.py in atomate/vasp/builder to parse the sections in “calcs_reversed”, but it seems that the “calcs_reversed” is a list with only 1 element. I can only parse the entire “calcs_reversed” to a new taskdoc but am unable to parse only a section of it. Testing with adding the following line

doc[“testing”] = taskdoc[“calcs_reversed”][1]

to task_materials.py leads to IndexError: list index out of range

2

Hi Yaze,

If you make your own custom classes and put them somewhere outside your installed Python packages (e.g., not where atomate is installed), you will have trouble unless you tell Python about the existence of your packages. e.g., if you cannot run “from my_custom_module import CustomDrone” from the Python shell (when running the Python shell in some directory outside where your module is located), then FireWorks will also not be able to find your CustomDrone.

There are some details on this in the “registering FireTasks” section:

https://materialsproject.github.io/fireworks/guide_to_writing_firetasks.html

Step 3: Register your Firetask

When FireWorks bootstraps your Firetask from a database definition, it needs to know where to look for Firetasks.

First, you need to make sure your Firetask is defined in a file location that can be found by Python, i.e. is within Python’s search path and that you can import your Firetask in a Python shell. If Python cannot import your code (e.g., from the shell), neither can FireWorks. This step usually means either installing the code into your site-packages directory (where many Python tools install code) or modifying your PYTHONPATH environment variable to include the location of the Firetask. You can see the locations where Python looks for code by typing import sys followed by print(sys.path). If you are unfamiliar with this topic, some more details about this process can be found here, or try Googling “how does Python find modules?”

Second, you must register your Firetask so that it can be found by the FireWorks software. There are a couple of options for registering your Firetask (you only need to do one of the below):

  1. Use the @explicit_serialize decorator to define your FW name (see the Appendix). No further registration is needed if you use this option.
  2. (or) if you have access to the FireWorks source directory, put your Firetask definition anywhere in fireworks.user_objects or it subdirectories - it will be automatically be found there.
  3. (or) put the Firetask wherever you’d like. However, you need to modify the USER_PACKAGES variable of the FW config to include the package for where to find the Firetask, e.g. “mypackage.my_subpackage”. Note that FireWorks will search within subpackages automatically, so you can just put a root package (but loading will be slightly slower).
    You are now ready to use your Firetask!

Regarding your postscript - unfortunately I did not understand what you intended by this. Note that the calcs_reversed contains details on each individual VASP run. For double relaxation structure optimizations, the length of the list is two, for all other calculatoins, the length of the list is typically one. If you want to know details of the “final” calculation in a series of calculations performed by custodian within the task, use calcs_reversed[0] which will give you information on the final calc.

···

On Monday, December 3, 2018 at 4:39:19 AM UTC-8, Yaze Wu wrote:

Dear Atomate Team

I’ve encountered some difficulty customizing a the atomate.vasp.fireworks.core.LepsFW to output more information into the task document than the default.

In the original firework, the LepsFW terminates after generating a task document in MongoDB that reports the following fields in the output taskdoc: epsilon_static, epsilon_static_wolfe, epsilon_ionic, piezo_ionic_tensor, piezo_tensor. However, since the DFPT calculation generates other information in calcs_reversed/output section as well, such as force_constants, normalmode_eigenvals etc, I want to customize the firework to report these information in the output taskdoc too.

May I know what is the cleanest way of doing this?

What I have tried is the following:

The Original LepsFW parses the vasp output to MongoDB tasks collection using the atomate.vasp.firetasks.parse_output.VaspToDb, which in turn invokes the atomate.vasp.drones.VaspDrone to do the actual parsing. I made copies of

  • atomate/vasp/firework/core.py
  • atomate/vasp/firetasks/parse_outputs.py
  • atomate/vasp/drones.py
    in my own user directory in a folder named “custom” and inserted a sys.path.append command in both core.py and parse_output.py so that the modified files load each of their subsidiary from
  • custom.atomate.vasp.firetasks.parse_output.VaspToDb
  • custom.vasp.drones.VaspDrone

The modified drones.py’s line 299 is then edited to include terms such as “force_constants” etc. so that they will be parsed to the “output” taskdoc.

I thought this will be sufficient but the following error popped up:

Traceback (most recent call last):

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1461, in _refresh_wf

updated_ids = wf.refresh(fw_id)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py”, line 1005, in refresh

updated_ids = updated_ids.union(self.apply_action(m_action, fw.fw_id))

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py”, line 813, in apply_action

apply_mod(mod, self.id_fw[cfid].spec)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1765, in spec

return self.partial_fw.spec

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1838, in partial_fw

self._fw = Firework.from_dict(data)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 155, in _decorator

new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 155, in

new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 118, in _recursive_load

return {k: _recursive_load(v) for k, v in obj.items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 118, in

return {k: _recursive_load(v) for k, v in obj.items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 121, in _recursive_load

return [_recursive_load(v) for v in obj]

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 121, in

return [_recursive_load(v) for v in obj]

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 113, in _recursive_load

return load_object(obj)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 329, in load_object

mod = import(modname, globals(), locals(), [classname], 0)

ModuleNotFoundError: No module named ‘custom’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/rocket.py”, line 357, in run

lp.complete_launch(launch_id, m_action, final_state)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1272, in complete_launch

self._refresh_wf(fw_id)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1476, in _refresh_wf

raise RuntimeError(err_message)

RuntimeError: Error refreshing workflow. The full stack trace is: Traceback (most recent call last):

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1461, in _refresh_wf

updated_ids = wf.refresh(fw_id)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py”, line 1005, in refresh

updated_ids = updated_ids.union(self.apply_action(m_action, fw.fw_id))

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/firework.py”, line 813, in apply_action

apply_mod(mod, self.id_fw[cfid].spec)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1765, in spec

return self.partial_fw.spec

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/core/launchpad.py”, line 1838, in partial_fw

self._fw = Firework.from_dict(data)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 155, in _decorator

new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 155, in

new_args[0] = {k: _recursive_load(v) for k, v in args[0].items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 118, in _recursive_load

return {k: _recursive_load(v) for k, v in obj.items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 118, in

return {k: _recursive_load(v) for k, v in obj.items()}

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 121, in _recursive_load

return [_recursive_load(v) for v in obj]

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 121, in

return [_recursive_load(v) for v in obj]

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 113, in _recursive_load

return load_object(obj)

File “/opt/apps/util/easybuild/software/atomate/0.8.4-intel-2018b-Python-3.6.6/lib/python3.6/site-packages/fireworks/utilities/fw_serializers.py”, line 329, in load_object

mod = import(modname, globals(), locals(), [classname], 0)

ModuleNotFoundError: No module named ‘custom’

It seems that there are more files and classes involved even after drones.py; however, I am unable to locate them. At this point, the process starts to look very complicated.

May I know if there is a cleaner way of achieving this objective? Or if not, which other files should I modify to overcome this error?

Best wishes

Yaze

p.s.

I’ve tried modifying task_materials.py in atomate/vasp/builder to parse the sections in “calcs_reversed”, but it seems that the “calcs_reversed” is a list with only 1 element. I can only parse the entire “calcs_reversed” to a new taskdoc but am unable to parse only a section of it. Testing with adding the following line

doc[“testing”] = taskdoc[“calcs_reversed”][1]

to task_materials.py leads to IndexError: list index out of range