Parallel processing on Windows 10

Hello,

I’m getting an error when I try to run workflows in parallel mode (rlaunch multi) on Windows. I’m running Python 3.6.2 and Fireworks 1.4.8 inside a Conda environment on Windows 10. This is clearly related to the multiprocessing lib in python and differences between Windows and Unix. I have also reproduced the same error by calling **fireworks.features.multi_launcher.launch_multiprocess **from a python script. I have tried the same on Ubuntu 16.04 (identical environment), where everything works fine.

I have seen little evidence that Windows in fact is a target platform for Fireworks, but is the parallel-processing functionality something you would consider adapting to work on Windows as well?

Alternatively I’m looking for advice for workarounds for running workflows in parallel in ‘infinite’ mode.

Here is the traceback when issuing “rlaunch multi 2” on the command-line:

(tailor_py3) C:\Users\audun>rlaunch multi 2
2017-08-07 12:42:44,404 INFO Hostname/IP lookup (this will take a few seconds)
Traceback (most recent call last):
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 193, in _run_module_as_main
main”, mod_spec)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\audun\Miniconda2\envs\tailor_py3\Scripts\rlaunch.exe_main
.py", line 9, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\scripts\rlaunch_run.py”, line 148, in rlaunch
local_redirect=args.local_redirect)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\features\multi_launcher.py”, line 189, in launch_multiprocess
ds = DataServer.setup(launchpad)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\utilities\fw_utilities.py”, line 203, in setup
m.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\managers.py”, line 513, in start
self._process.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\process.py”, line 105, in start
self._popen = self._Popen(self)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in init
reduction.dump(process_obj, to_child)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘DataServer.setup..’
Traceback (most recent call last):
File “”, line 1, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

``

Any help is much appreciated,

Regards,

Audun Gravdal Johansen

Hi Audun,

Unfortunately, I don’t have a Windows box and won’t be able to debug / fix this error. I’ll pass your ticket along to the person that wrote most of the multiprocessing to see if he has any thoughts.

Best,

Anubhav

···

On Monday, August 7, 2017 at 6:21:35 AM UTC-7, agrav wrote:

Hello,

I’m getting an error when I try to run workflows in parallel mode (rlaunch multi) on Windows. I’m running Python 3.6.2 and Fireworks 1.4.8 inside a Conda environment on Windows 10. This is clearly related to the multiprocessing lib in python and differences between Windows and Unix. I have also reproduced the same error by calling **fireworks.features.multi_launcher.launch_multiprocess **from a python script. I have tried the same on Ubuntu 16.04 (identical environment), where everything works fine.

I have seen little evidence that Windows in fact is a target platform for Fireworks, but is the parallel-processing functionality something you would consider adapting to work on Windows as well?

Alternatively I’m looking for advice for workarounds for running workflows in parallel in ‘infinite’ mode.

Here is the traceback when issuing “rlaunch multi 2” on the command-line:

(tailor_py3) C:\Users\audun>rlaunch multi 2
2017-08-07 12:42:44,404 INFO Hostname/IP lookup (this will take a few seconds)
Traceback (most recent call last):
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 193, in _run_module_as_main
main”, mod_spec)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\audun\Miniconda2\envs\tailor_py3\Scripts\rlaunch.exe_main
.py", line 9, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\scripts\rlaunch_run.py”, line 148, in rlaunch
local_redirect=args.local_redirect)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\features\multi_launcher.py”, line 189, in launch_multiprocess
ds = DataServer.setup(launchpad)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\utilities\fw_utilities.py”, line 203, in setup
m.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\managers.py”, line 513, in start
self._process.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\process.py”, line 105, in start
self._popen = self._Popen(self)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in init
reduction.dump(process_obj, to_child)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘DataServer.setup..’
Traceback (most recent call last):
File “”, line 1, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

``

Any help is much appreciated,

Regards,

Audun Gravdal Johansen

Ok, great!

Thanks for the quick reply.

Audun

mandag 7. august 2017 18.53.51 UTC+2 skrev Anubhav Jain følgende:

···

Hi Audun,

Unfortunately, I don’t have a Windows box and won’t be able to debug / fix this error. I’ll pass your ticket along to the person that wrote most of the multiprocessing to see if he has any thoughts.

Best,

Anubhav

On Monday, August 7, 2017 at 6:21:35 AM UTC-7, agrav wrote:

Hello,

I’m getting an error when I try to run workflows in parallel mode (rlaunch multi) on Windows. I’m running Python 3.6.2 and Fireworks 1.4.8 inside a Conda environment on Windows 10. This is clearly related to the multiprocessing lib in python and differences between Windows and Unix. I have also reproduced the same error by calling **fireworks.features.multi_launcher.launch_multiprocess **from a python script. I have tried the same on Ubuntu 16.04 (identical environment), where everything works fine.

I have seen little evidence that Windows in fact is a target platform for Fireworks, but is the parallel-processing functionality something you would consider adapting to work on Windows as well?

Alternatively I’m looking for advice for workarounds for running workflows in parallel in ‘infinite’ mode.

Here is the traceback when issuing “rlaunch multi 2” on the command-line:

(tailor_py3) C:\Users\audun>rlaunch multi 2
2017-08-07 12:42:44,404 INFO Hostname/IP lookup (this will take a few seconds)
Traceback (most recent call last):
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 193, in _run_module_as_main
main”, mod_spec)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\audun\Miniconda2\envs\tailor_py3\Scripts\rlaunch.exe_main
.py", line 9, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\scripts\rlaunch_run.py”, line 148, in rlaunch
local_redirect=args.local_redirect)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\features\multi_launcher.py”, line 189, in launch_multiprocess
ds = DataServer.setup(launchpad)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\utilities\fw_utilities.py”, line 203, in setup
m.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\managers.py”, line 513, in start
self._process.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\process.py”, line 105, in start
self._popen = self._Popen(self)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in init
reduction.dump(process_obj, to_child)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘DataServer.setup..’
Traceback (most recent call last):
File “”, line 1, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

``

Any help is much appreciated,

Regards,

Audun Gravdal Johansen

I can’t reproduce error since I am not using Windows. I did some search on google. I agree that there is some difference between Linux and Windows.

Windows doesn’t have Unix sockets. Instead, …

http://stupidpythonideas.blogspot.com/2014/09/sockets-and-multiprocessing.html

A possible solution is discussed in this thread:

https://github.com/uqfoundation/pathos/issues/67

Second item to note is that Windows works differently than non-windows systems. Much of the time, for the ProcessPool, Windows requires from pathos.helpers import freeze_support, then freeze_support(). It’s unnecessary for non-windows systems. Windows often requires freeze_support to be provided in main.

···

On Monday, August 7, 2017 at 6:21:35 AM UTC-7, agrav wrote:

Hello,

I’m getting an error when I try to run workflows in parallel mode (rlaunch multi) on Windows. I’m running Python 3.6.2 and Fireworks 1.4.8 inside a Conda environment on Windows 10. This is clearly related to the multiprocessing lib in python and differences between Windows and Unix. I have also reproduced the same error by calling **fireworks.features.multi_launcher.launch_multiprocess **from a python script. I have tried the same on Ubuntu 16.04 (identical environment), where everything works fine.

I have seen little evidence that Windows in fact is a target platform for Fireworks, but is the parallel-processing functionality something you would consider adapting to work on Windows as well?

Alternatively I’m looking for advice for workarounds for running workflows in parallel in ‘infinite’ mode.

Here is the traceback when issuing “rlaunch multi 2” on the command-line:

(tailor_py3) C:\Users\audun>rlaunch multi 2
2017-08-07 12:42:44,404 INFO Hostname/IP lookup (this will take a few seconds)
Traceback (most recent call last):
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 193, in _run_module_as_main
main”, mod_spec)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\Users\audun\Miniconda2\envs\tailor_py3\Scripts\rlaunch.exe_main
.py", line 9, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\scripts\rlaunch_run.py”, line 148, in rlaunch
local_redirect=args.local_redirect)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\features\multi_launcher.py”, line 189, in launch_multiprocess
ds = DataServer.setup(launchpad)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\site-packages\fireworks\utilities\fw_utilities.py”, line 203, in setup
m.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\managers.py”, line 513, in start
self._process.start()
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\process.py”, line 105, in start
self._popen = self._Popen(self)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\context.py”, line 322, in _Popen
return Popen(process_obj)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\popen_spawn_win32.py”, line 65, in init
reduction.dump(process_obj, to_child)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\reduction.py”, line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can’t pickle local object ‘DataServer.setup..’
Traceback (most recent call last):
File “”, line 1, in
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 105, in spawn_main
exitcode = _main(fd)
File “c:\users\audun\miniconda2\envs\tailor_py3\lib\multiprocessing\spawn.py”, line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

``

Any help is much appreciated,

Regards,

Audun Gravdal Johansen