Repost from email question

ardunn · June 10, 2019, 8:25pm

This is a copy of a question I got over email, thought I’d put it (and the answer) here so others could find it.

···

Hi Alex,

Thanks for integrating the previous fix so fast, and again for providing Rocketsled.

I wanted to also let you know about a couple changes I had to make to get it to run:

```
 Run on my Windows sandbox, and,
```

 Run RS jobs that were not located in my Python install directory (I wanted to store my RS jobs in a separate work folder)

The changes I made were all in utils.py’s serialize() / deserialize() functions (see attached):

 Change the “/” to Python’s “os.sep” OS-dependent filepath separator so it work son windows

 Modify serialize() to try getting the “wf_creator” function from the RS Job’s module first

Steps to reproduce:

 Copy basic.py somewhere outside the Python installation dir

a. In this case, I used: C:\Users\abe\rmsit\workspaces\projects\ers3_rocketsled_scripts

 Rename “basic.py” to “basic2.py” (or any other name that is not already used by a module in your Python installation dir)

```
 Run basic2.py
```

It errors out with:

Traceback (most recent call last):

File “”, line 1, in

runfile('C:/Users/abe/rmsit/workspaces/projects/ers3_rocketsled_scripts/basic2.py', wdir='C:/Users/abe/rmsit/workspaces/projects/ers3_rocketsled_scripts')

File “C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 786, in runfile

execfile(filename, namespace)

File “C:\ProgramData\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 110, in execfile

exec(compile(f.read(), filename, 'exec'), namespace)

File “C:/Users/abe/rmsit/workspaces/projects/ers3_rocketsled_scripts/basic2.py”, line 96, in

mc.configure(wf_creator=wf_creator, dimensions=x_dim)

File “C:\ProgramData\Anaconda3\lib\site-packages\rocketsled\control.py”, line 219, in configure

wf_creator = serialize(wf_creator)

File “C:\ProgramData\Anaconda3\lib\site-packages\rocketsled\utils.py”, line 122, in serialize

"you sure it's module is in your PYTHONPATH?".format(fun_path))

ImportError: Administrator.rmsit.workspaces.projects.ers3_rocketsled_scripts.basic2.wf_creator couldn’t be serialized to be imported. Are you sure it’s module is in your PYTHONPATH?

I added a print statement to print out the function path it was trying to resolve:

    for _ in range(5):

        try:

            # if we couldn't find the func, try adding the parent folder name to the import path

            full_import_path = all_pkgs[-1] + "." + full_import_path

            all_pkgs = all_pkgs[:-1]

            fun_path = full_import_path + "." + name

            print("fun_path: {}".format(fun_path))

            deserialize(fun_path)

            return fun_path

        except ImportError:

            continue

    else:

        raise ImportError("{} couldn't be serialized to be imported. Are "

                          "you sure it's module is in your PYTHONPATH?".format(fun_path))

This is the output of the print statement:

2019-06-06 11:33:31,422 INFO Optimization collection opt_default hard reset.

fun_path: ers3_rocketsled_scripts.basic2.wf_creator

fun_path: projects.ers3_rocketsled_scripts.basic2.wf_creator

fun_path: workspaces.projects.ers3_rocketsled_scripts.basic2.wf_creator

fun_path: rmsit.workspaces.projects.ers3_rocketsled_scripts.basic2.wf_creator

fun_path: abe.rmsit.workspaces.projects.ers3_rocketsled_scripts.basic2.wf_creator

So it never tried to look up the base case “basic2.wf_creator”. I rearranged the code in the for-loop like so, but will defer to you on whether this is the right thing to do:

    for _ in range(5):

        try:

            fun_path = full_import_path + "." + name

            print("fun_path: {}".format(fun_path))

            deserialize(fun_path)

            return fun_path

        except ImportError:

            # if we couldn't find the func, try adding the parent folder name to the import path

            full_import_path = all_pkgs[-1] + "." + full_import_path

            all_pkgs = all_pkgs[:-1]

            continue

    else:

        raise ImportError("{} couldn't be serialized to be imported. Are "

                          "you sure it's module is in your PYTHONPATH?".format(fun_path))

The output of the print statement now gets “basic2.wf_creator” on the first iteration of the for-loop:

fun_path: basic2.wf_creator

I tested this out with Anaconda3 on a Windows 7 VM. Like I said, I will defer to you on whether these are the right code changes. I just know I had to make these changes to get it to work on my Win7 box, and I figured I’d share so hopefully it makes it easier for you. I would have done a pull request on Github, but I need to check on our corporate policies before I can do that, so I figured I would just email you to get it to you faster. Sorry about that.

If you have any feedback, please feel free to let me know!

Thanks,

Abe

ardunn · June 10, 2019, 8:49pm

Hi Abe,

Thanks for mentioning the separators for Windows compatibility. I will update the code accordingly.

···

The “not being able to find the parent module” is a tricky and persistent issue because it is not well defined how is best to access the module outside the rocketsled directory given only some free-floating function object. But there are ways around it on Linux/Unix without having to further modify serialize()/deserialize().

Here is a short primer on defining wf_creator functions (though this extends to get_z, and predictor as well):

You can passeither in a function object or string representing a function to the wf_creator argument.

**Function object **If you choose to pass a function object, the serializer has to serialize it (basically turn your function object into a string). This is the easiest but probably most error prone way. The serializer iteratively tries to import based on directory names and hopes that eventually it will find a fully defined importable. This works well for modules in packages, but when the function object is not in an importable package’s module but rather in a free module in some folder, you will get an error like the one shown in your stacktrace.

String - in a package: If your wf_creator is in a module which is a part of some package, you can simply do ‘my_package.my_subpackage.my_module.wf_creator’.

String - not in a package: If your wf_creator is free floating in some non-package directory (e.g., C:\Abe\Downloads\basic.py), you can use the syntax “C:\Abe\Downloads\basic.wf_creator”. This basically allows rocketsled to add this folder to your pythonpath env variabe so rocketsled can find the module containing wf_creator. I have just personally tested this on Linux/Unix and it works; on windows, you might encounter some unforseen issues but it is worth a try.

You can find more details at the end of the wf_creator section in the comprehensive guide: https://hackingmaterials.lbl.gov/rocketsled/guide.html

ardunn · June 10, 2019, 8:57pm

P.S. I have an idea in mind for allowing function objects to work even if they are outside of registered python package in your environment. If I implement it in the next rocketsled version, I’ll update this thread.

Abe_Wu · June 11, 2019, 12:16am

Thanks for the response Alex.

I tried the “String - not in a package” method, by specifying wf_creator=“C:\Abe\Downloads\basic.wf_creator”, and it did in fact work on Windows. Thanks for letting me know of this other way of doing it.

However, I think the utils.py fix I submitted might do what you’re looking for, if I’m understanding what you’re saying correctly. For a couple reasons:

It allows you to run a “free floating” RS job without having to hardcode the filepath to the basic.py, and,
If it doesn’t find the wf_creator in the RS job, it still works in the same way as you describe. Namely, if it doesn’t find the wf_creator function in the RS job, it will look in the parent folder recursively. So if the script is in “C:\Abe\Downloads\basic.py”, it will look in these places in order:

basic.wf_creator
downloads.basic.wf_creator
abe.downloads.basic.wf_creator

No worries though if you have a better way to do it. In any case, thanks for all the work you put in to creating/maintaining Rocketsled!

Abe

···

On Monday, June 10, 2019 at 1:57:48 PM UTC-7, Alexander Dunn wrote:

P.S. I have an idea in mind for allowing function objects to work even if they are outside of registered python package in your environment. If I implement it in the next rocketsled version, I’ll update this thread.

Abe_Wu · June 11, 2019, 12:33am

Just to follow up on my previous email, I did confirm that I run into the same error on Linux with it not finding the wf_creator function. It was fixed after I applied the same change to utils.py.

Also, I realized I misspoke when I said “it will look in the parent folder recursively”. It doesn’t recurse, it just does it in the for-loop (same as in the original code).

Thanks,

Abe

···

On Monday, June 10, 2019 at 5:16:48 PM UTC-7, Abe Wu wrote:

Thanks for the response Alex.

However, I think the utils.py fix I submitted might do what you’re looking for, if I’m understanding what you’re saying correctly. For a couple reasons:

It allows you to run a “free floating” RS job without having to hardcode the filepath to the basic.py, and,

If it doesn’t find the wf_creator in the RS job, it still works in the same way as you describe. Namely, if it doesn’t find the wf_creator function in the RS job, it will look in the parent folder recursively. So if the script is in “C:\Abe\Downloads\basic.py”, it will look in these places in order:

basic.wf_creator

downloads.basic.wf_creator

abe.downloads.basic.wf_creator

No worries though if you have a better way to do it. In any case, thanks for all the work you put in to creating/maintaining Rocketsled!

Abe

P.S. I have an idea in mind for allowing function objects to work even if they are outside of registered python package in your environment. If I implement it in the next rocketsled version, I’ll update this thread.

I tried the “String - not in a package” method, by specifying wf_creator=“C:\Abe\Downloads\basic.wf_creator”, and it did in fact work on Windows. Thanks for letting me know of this other way of doing it.
On Monday, June 10, 2019 at 1:57:48 PM UTC-7, Alexander Dunn wrote: