Python bindings for my custom model

Elisa

Hi, I have built the PyBindTutorial and I got the same error:

Jupyter QtConsole 4.3.1
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

Python bindings for the SAMSON API are done in the same structure as the SAMSON SDK itself (see Documentation for SAMSON SDK). Python bindings for the SAMSON API are imported in the following way:

import samson as sam
from samson.Facade import SAMSON        # SAMSON Facade - main interface of SAMSON
from samson.DataModel import Quantity   # Quantities: length, mass, time, etc
from samson.DataModel import Type       # Types: position3, etc

import SE_F2078F9E_F2CB_BA72_EE86_1E01A10B63D4 as pybindtutorial
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-1dded50a96a7> in <module>()
----> 1 import SE_F2078F9E_F2CB_BA72_EE86_1E01A10B63D4 as pybindtutorial

ImportError: generic_type: type "CustomStructuralModel" referenced unknown base type "SBMStructuralModel"

The only change I made to the code was in the CMakeList.txt to point to my pybind11 folder, but I don't think that should be related to this.

DmitriyMarin

No, I do not think that it can be related to a pybind11 version. Did you experience this issue on Windows? I will try to reproduce the problem and will return to you. For now, you may try to use your workaround with py::class_<SBStructuralModel>(m, "SBStructuralModel");

Elisa

Yes, I'm using Windows 10.

By the way, this is not entirely related, but now my module fails to load if the Python Scripting module is not loaded, do you know of a simple way to handle this? So the rest of functionality of my module can be loaded.

Best regards,

Elisa

DmitriyMarin

Dear @Elisa ,

This is not totally related to the Python Scripting Element but to the fact that now your Element is linked to a Python library and users will need to install Python. The Python Scripting Element also takes care of loading the Python library.

There are several possibilities to solve this:
1 . You can ship 2 SAMSON Elements: with Python bindings and without. But it is not that convenient.
2 . Users will need to install Python.
3 . You can ship a Python library together with your SAMSON Element.

I will check the 3rd option and will return to you.

DmitriyMarin

@elisa said in Python bindings for my custom model:

That is not working for me, I get the error I posted before unless before that statement I add py::class_<SBStructuralModel>(m, "SBStructuralModel"); before exposing my structural model when importing in the Python Scripting console.

I have tried to reproduce the problem you are having on Win10 and Win7 with the PyBindTutorial sample, but I could not - for me, it works without the need to expose SBStructuralModel (i.e., py::class_<SBStructuralModel>(m, "SBStructuralModel");) before exposing a custom structural model - there is no error when importing the module in the Python Scripting console.

Could you, please, try to point not to your pybind11 folder but to one provided with the PyBindTutorial sample, like it is already done in the CMakeLists.txt of this sample - just try to build the PyBindTutorial sample as it is.

DmitriyMarin

@dmitriymarin said in Python bindings for my custom model:

Dear @Elisa ,
This is not totally related to the Python Scripting Element but to the fact that now your Element is linked to a Python library and users will need to install Python. The Python Scripting Element also takes care of loading the Python library.
There are several possibilities to solve this:
1 . You can ship 2 SAMSON Elements: with Python bindings and without. But it is not that convenient.
2 . Users will need to install Python.
3 . You can ship a Python library together with your SAMSON Element.
I will check the 3rd option and will return to you.

Concerning the 3rd option. A simple workaround might be to ship your SAMSON Element together with a Python library used for compilation for each platform. You can just copy the Python library in the SAMSON Binaries folder (the folder with SAMSON executable). Then even if a user has no Python installed, your SAMSON Element will be loaded since a dependency to the Python library is resolved.

Elisa

Thank you very much, I will investigate how to ship it with a virtual python environment with pip freeze and venv.

DmitriyMarin

@Elisa , if you came up with a solution, please, share it. In the future, we want to ship the Python Scripting Element with python interpreter and libraries, so that users won't need to install Python themselves.

Elisa

I will! I am dealing with a weird unrelated bug in my programming environment, but will try to get on with this after that :)

DmitriyMarin

Thanks, @Elisa !

DmitriyMarin

I have added a tutorial on how to create Python bindings for functions that return or receive SAMSON physical quantities and types (i.e., SBQuantity, SBVector3, etc). The code for the tutorial can be found in the sample PyBindTutorial SAMSON Element.

Elisa

Thanks for the new tutorial!!

I was looking into shipping my module with a Python virtual environment, to avoid users having to install Anaconda3. So far this is what I got:

If I understood it alright, to develop and compile the module only the variables PYTHON_INCLUDE_DIR and PYTHON_LIBRARY matter, which should be Python 3.6's. I didn't check if they updated when upgrading the Anaconda installation, but it seems easy to keep an Anaconda 5.2 installation to access this in case it wouldn't work.
To execute the Python Scripting (and any other depending on it) it is necessary to specify the variable PYTHON_EXECUTABLE. For this I think you can use a virtual environment. What I did was to use pip freeze
to get a list of modules installed in Anaconda 5.2, then I used the virtualenv module to create a virtual environment out of Anaconda 5.2 and install this modules there. I set the PYTHON_EXECUTABLE variable to this virtual environment ENV_DIR\\Scripts\\python.exe (which you can place on the data folder and then set this using the plugin installation folder, I guess). The only problem is, you have to activate the virtual environment. I tested this by going to the virtual environment folder, activating it from there in the command line, and calling SAMSON-Core.exe then. It worked. I didn't had Anaconda in my path, nor it worked without activating the virtual environment.

I have to update now my Anaconda3 installation for some unrelated work, so I will test if this keeps working in the developers case. Hopefully, I will be able to test if at least users can avoid installing Anaconda3 5.2 soon with a user of our module.

This were the modules I installed in my virtual environment btw:

backcall==0.1.0
colorama==0.3.9
decorator==4.3.0
ipykernel==4.8.2
ipython==6.4.0
ipython-genutils==0.2.0
jedi==0.12.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
numpy==1.14.3
parso==0.2.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
Pygments==2.2.0
python-dateutil==2.7.3
pyzmq==17.0.0
simplegeneric==0.8.1
six==1.11.0
tornado==5.0.2
traitlets==4.3.2
wcwidth==0.1.7

Maybe you could do without numpy or with a more recent version. The virtual environment command was:

python -m virtualenv VENV_DIR --always-copy

I haven't understood yet how these specific versions of the modules are involved when compiling, but in principle the Include folder from my virtual environment, and the include folder from anaconda3 seem the same. Maybe for the Python Scripting module this is different, since it is the module actually creating the ipython console.

DmitriyMarin

Thank you, @Elisa ! This looks great!

Do you want users to use Python in your module or is it just for your module to be loaded in SAMSON if users do not have Python installed?

What is the size of the resulting environment?

I will try the same for the Python Scripting Element, since for the next release we are thinking about shipping it with a Python environment.

Elisa

Thanks!

It would be only for the case that users don't have python or they need very specific Python requirements (exactly those python modules versions, for example).
I think if you ship the Python Scripting Element with a virtual environment, probably other modules don't need to ship it.

The virtual environment is 101MB, but I created it copying everything, maybe it could work if the user has a Python36 installation somewhere, and we just override the modules with the versions we need.

DmitriyMarin

Yes, if Python Scripting is shipped with a Python environment, other SAMSON Elements would not need their own, except if they use some quite specific one.

Can the Python environment you are shipping with your module be extended by installing new modules?

Overriding of existing modules in the user's environment is not suitable since users might need this particular environment, we should not modify a user's environment in any way.

Elisa

The virtual environments doesn't really override existing modules, only once you have activated this environment, but once you exit it, the usual environment is back. It works like Anaconda environments. Once you activate the environment you can install any module you want, remove it or upgrade it, via pip as usual.

DmitriyMarin

Ah, sorry, I see what you meant in the previous comment, I've misunderstood.
I will check later the construction of a Python environment for Python Scripting so that users did not have to install Python. Activation of Python environment on Linux and Mac can be incorporated into the scripts that launch SAMSON, on Windows you can write a .bat file that will be first activating your Python env and then launching SAMSON.

Elisa

Even though I develop this for Windows, I am more familiar with Linux too, so I don't know if it would be possible to incorporate on the launcher the activation of the venv, but it is done with a script created by virtualenv/venv as in Linux.

DmitriyMarin

I think it should be possible, it will be just an additional line in the launcher script. If we will be shipping Python Scripting with the Python environment, I think we might ship the environment with a bare SAMSON installation so that to use SAMSON Elements, that depend on this environment, the user won't need to install the Python Scripting Element.