Is it possbile to run one Firework with different queue

Hi,

I am new to Fireworks. All the computing queue in my HPC have no access to outside network. However, there is one queue, which called copyq, is albe to connect to my MongoDB host. I do not want to use “offline” model. The HPC administrator suggested me use copyq to download and upload from MongoDB, and use normal computing queue to do calculation. I am wondering if this is possible with Fireworks. For example, do one Firework with three tasks, write VASP input files with copyq > run VASP with normal queue > Store results and send back to MongoDB with copyq. Is this technically possbile?

Thanks.

Hi Xinyu,

It seems (maybe?) possible but difficult / cumbersome. If I understand correctly, you would have to do something like the following:

  1. In a “normal” operation, you could do everything using the copyq (e.g., pull the job, run VASP, update the database).

  2. But since you need to run VASP on an outside queue, things become difficult. You could try setting up your Firework as follows:

a) The first FireTask pulls the job and writes the VASP inputs (same as always)

b) The second FireTask submits a VASP run in the directory of your VASP inputs to your “normal queue”

c) A third FireTask checks if the VASP run has completed. If not, it FIZZLES (throws an error). If yes, then it just goes to the fourth step

d) A fourth FireTask does the database insertion, etc.

So, what will happen is that the job will run through steps a-b on the copyq. When it gets to step c), it will FIZZLE. At this point, you will have a job waiting in the normal queue and a FIZZLED Firework.

When the job in the normal queue completes, you can rerun the FireWork with task_level_rerun set to True. This will restart the FireWork at task c. It will detect that the job is completed and move on to step (d).

I have not tested the above scheme, so there might be issues when actually trying it. But, at least in principle, I think it could work.

···

On Thursday, May 16, 2019 at 5:37:20 AM UTC-7, Xinyu Lee wrote:

Hi,

I am new to Fireworks. All the computing queue in my HPC have no access to outside network. However, there is one queue, which called copyq, is albe to connect to my MongoDB host. I do not want to use “offline” model. The HPC administrator suggested me use copyq to download and upload from MongoDB, and use normal computing queue to do calculation. I am wondering if this is possible with Fireworks. For example, do one Firework with three tasks, write VASP input files with copyq > run VASP with normal queue > Store results and send back to MongoDB with copyq. Is this technically possbile?

Thanks.