querying fireworks from python using Launchpad object

FireWorks
1

Hello,

I would like to querying fireworks just like in this page of doc : https://materialsproject.github.io/fireworks/query_tutorial.html, but from python.

I can do it from the shell just as indicated in this page, but I need to get these information periodically in a python program

For the moment I use a subprocess to retrieve this information from the CLI (see below), but It would be much faster to do it from a Launchpad object in python …

def get_fws(discr, value):

if discr == "category":

	cmd = "lpad get_fws -q '{\"spec._category\": \"%s\"}' -d ids" % value

elif discr == "state":

	cmd = "lpad get_fws -s %s -d ids" % value

ids_json = subprocess.check_output(cmd, shell=True).decode('utf-8')

ids = json.loads(ids_json)

return ids

I haven’t find it in the doc. But I have found this on this list : launchpad.launches.find()

res = launchpad.launches.find({“spec._category”: “multiseq_long”}) -> may be ok: returns pymongo.cursor.Cursor

nb = launchpad.launches.count({“spec._category”: “multiseq_long”}) -> not OK : gives 0 (should be 26)

So it seems not to be the good way of doing it …

Could you provide us a way to do it ? in a first time, a count() will be sufficient, but an id list would be nice !

Best regards,
David

2

Hi David

The “lpad get_fws” command uses a two step process:

i) Launchpad.get_fw_ids() to get the list of matching FW ids

ii) Launchpad.get_fw_by_id() to get the details of a Firework once you know the ids

Part (ii) is straightforward. For (i) you will just need to pass in your Mongo query dict, e.g. something like {“spec._category”: “category1”}. Note that (i) only takes in a Mongo query dict and doesn’t take any other type of query (e.g., no direct query on state). To see how the other options like “get_fws -s FIZZLED” are translated into a Mongo query dict, look at fireworks.scripts.lpad_run.get_fws(). In fact, the entire source code for how to turn the command line options into Python is in that function.

Best,

Anubhav

···

On Fri, Sep 8, 2017 at 7:59 AM, [email protected] wrote:

Hello,

I would like to querying fireworks just like in this page of doc : https://materialsproject.github.io/fireworks/query_tutorial.html, but from python.

I can do it from the shell just as indicated in this page, but I need to get these information periodically in a python program

For the moment I use a subprocess to retrieve this information from the CLI (see below), but It would be much faster to do it from a Launchpad object in python …

def get_fws(discr, value):

if discr == “category”:

  cmd = "lpad get_fws -q '{\"spec._category\": \"%s\"}' -d ids" % value

elif discr == “state”:

  cmd = "lpad get_fws -s %s -d ids" % value

ids_json = subprocess.check_output(cmd, shell=True).decode(‘utf-8’)

ids = json.loads(ids_json)

return ids

I haven’t find it in the doc. But I have found this on this list : launchpad.launches.find()

res = launchpad.launches.find({“spec._category”: “multiseq_long”}) -> may be ok: returns pymongo.cursor.Cursor

nb = launchpad.launches.count({“spec._category”: “multiseq_long”}) -> not OK : gives 0 (should be 26)

So it seems not to be the good way of doing it …

Could you provide us a way to do it ? in a first time, a count() will be sufficient, but an id list would be nice !

Best regards,
David

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/6b9ba4cb-f32a-4421-aaf1-e81b13ef2ee6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–

Best,
Anubhav

3

Hi Anubhav,

Thank you for the quick response. I managed to use the python interface. It works very fine now.

Thanks a lot !

Best regards,
David

···

Le vendredi 8 septembre 2017 18:03:44 UTC+2, ajain a écrit :

Hi David

The “lpad get_fws” command uses a two step process:

i) Launchpad.get_fw_ids() to get the list of matching FW ids

ii) Launchpad.get_fw_by_id() to get the details of a Firework once you know the ids

Part (ii) is straightforward. For (i) you will just need to pass in your Mongo query dict, e.g. something like {“spec._category”: “category1”}. Note that (i) only takes in a Mongo query dict and doesn’t take any other type of query (e.g., no direct query on state). To see how the other options like “get_fws -s FIZZLED” are translated into a Mongo query dict, look at fireworks.scripts.lpad_run.get_fws(). In fact, the entire source code for how to turn the command line options into Python is in that function.

Best,

Anubhav

On Fri, Sep 8, 2017 at 7:59 AM, [email protected] wrote:

Hello,

I would like to querying fireworks just like in this page of doc : https://materialsproject.github.io/fireworks/query_tutorial.html, but from python.

I can do it from the shell just as indicated in this page, but I need to get these information periodically in a python program

For the moment I use a subprocess to retrieve this information from the CLI (see below), but It would be much faster to do it from a Launchpad object in python …

def get_fws(discr, value):

if discr == "category":

  cmd = "lpad get_fws -q '{\"spec._category\": \"%s\"}' -d ids" % value

elif discr == "state":

  cmd = "lpad get_fws -s %s -d ids" % value

ids_json = subprocess.check_output(cmd, shell=True).decode('utf-8')

ids = json.loads(ids_json)

return ids

I haven’t find it in the doc. But I have found this on this list : launchpad.launches.find()

res = launchpad.launches.find({“spec._category”: “multiseq_long”}) -> may be ok: returns pymongo.cursor.Cursor

nb = launchpad.launches.count({“spec._category”: “multiseq_long”}) -> not OK : gives 0 (should be 26)

So it seems not to be the good way of doing it …

Could you provide us a way to do it ? in a first time, a count() will be sufficient, but an id list would be nice !

Best regards,
David

–

You received this message because you are subscribed to the Google Groups “fireworkflows” group.

To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].

To post to this group, send email to [email protected].

Visit this group at https://groups.google.com/group/fireworkflows.

To view this discussion on the web visit https://groups.google.com/d/msgid/fireworkflows/6b9ba4cb-f32a-4421-aaf1-e81b13ef2ee6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

–
Best,
Anubhav

4

how to write a mongodb queries for a list of FIREWORK IDS using $in

5

You can query mongodb collections directly using lp.db.collection.find(). An example of this is as follows:

from fireworks import LaunchPad

fw_ids = <your list of firework ids>

lp = LaunchPad.autoload()
lp.db.fireworks.find({'fw_id': {'$in': fw_ids}})
6

Thanks

7

Hello,

Is there a command in python to mark the jobs as fizzled using fw_id?

8

No, you can only cancel (“defuse”) jobs via manual command. The fizzled (failed) tag is only set by FireWorks, or you would need to go in and manually hack the database (which is tricky since the Launch state should match the Firework state, and the fw_states key of the Workflow needs to be updated for example).

9

Thank you

10

Firework states : Waiting, Completed, Reserved, Ready. I am able to get these states for my firework jobs and I am able to query thes states using launchpad.get_fw_by_id(fw_id).state

But I am not able to get RUNNING state as output for any firework jobs. Is RUNNING state will be visible for firework jobs