[ell-i-developers] Re: Embedding python in emulator
- From: Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx>
- To: "ell-i-developers@xxxxxxxxxxxxx" <ell-i-developers@xxxxxxxxxxxxx>
- Date: Wed, 22 Oct 2014 16:04:45 +0300
Hello all,
Calling python from c-code now works. I have been studying and working on
it since more then a week, but it was good learning process. I have now
good understanding for advanced usage of "Embedding python in C" and
"Extending python in C". Both have almost same CPython-API, but usage is
different. In our case, for providing a bridge between the emulator code
and python, I had to write a c-extension module to pass on the reference of
python callback function to the emulator and using the same module to
callback python function using the reference from emulator.
As we are loading the emulator as a shared library and the code to call
python callback using its reference and CPython-API is written in emulator,
the python is already initialized when we loaded the emulator shared
library. The only wrong thing I was doing is to acquire it in wrong context
and then trying to close it in the same python thread. After reading the
python documentation, I realized that I was not locking and unlocking the
python interpreter (Global Interpreter Lock -> GIL) to call CPython APIs.
Now, using the extension module which possess the static reference to the
python callback can be used in the CPython API in emulator.
I had been doing the following which was a good learning process:
1- Using python sub-interpretor (didn't worked because all the modules,
extension modules and local and global variables are initialized again with
new state).
2- Using the c global variables (extern in header files) and using it as
bridge between python and emulator.
3- Calling the python script file emulator to store results in a file and
then read that file in python (ugly solution and very inflexible).
4- Calling the python callback in python script file directly using CPython
API in emulator (didn't worked to load the script file as module).
5- The neat and clean solution is to write a cpython extension module,
compile it, store the .so library of it in the emulator/src/ directory and
the test-scripts/emulator/ library. Then load it in python scripts or
emulator when needed.
Any other solutions or suggestions are welcomed!
On Wed, Oct 15, 2014 at 11:02 PM, Pekka Nikander <pekka.nikander@xxxxxx>
wrote:
> You are not embedding a python interpreter yourself, so you shouldn't call
> Py_Initialize. That's been called already. You shouldn't call Py_Finanize
> neither, as that destroys the Python interpreter there is.
>
> You can check that with Py_IsInitialized.
>
> My guess is that also PyEval_ThreadsInitialized should return true,
> meaning that you have the Python threads initialised.
>
> --Pekka
>
> On 2014–10–15, at 22:53 , Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx>
> wrote:
>
> > I am calling:
> >
> > void Register::pythonCallback()
> > {
> > Py_Initializa();
> > Py_Finalize();
> > }
> >
> > in Register.cpp through = operator in BSRR class
> > On Oct 15, 2014 10:43 PM, "Pekka Nikander" <pekka.nikander@xxxxxx>
> wrote:
> > My gut feeling is that you are somehow acquiring the Python context in a
> wrong way.
> >
> > It would help if you posted the relevant code from your Register.cpp.
> Yes, I did see it today, when I uncommented it, but I didn't really look at
> it from this point of view, and therefore I don't remember what there was.
> >
> > --Pekka
> >
> > On 2014–10–15, at 22:37 , Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx>
> wrote:
> >
> >> Hi Ivan,
> >>
> >> Thanks!
> >>
> >> python 2.7.8 (32-bit manually compiled)
> >> Linux asif-Satellite-C655 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15
> 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> >>
> >> Pekka assisted me solving the issue!
> >>
> >> I am calling the python interpreter from Register.cpp only using (
> Py_Initialize() and Py_Finalize() ), still after running the rf test case
> under gdb, I get following segmentation fault: Does that mean I have to
> call the python interpretor in a seperate thread or so from the
> Register.cpp class? Any guesses ... I am not calling python interpreter
> from the main.c or so.
> >>
> ------------------------------------------------------------------------------------------
> >>
> ==============================================================================
> >> DigitalRead
> >>
> ==============================================================================
> >> Read high from pin :: Test Setup: Set the pin mode to output. Test...
> GPIO:MODER:value=0 -> =40
> >> .GPIO:MODER:value=40 -> =40
> >> GPIO:BSRRL:value=0 -> =0
> >> Here we call python from Register Class
> >>
> >> Program received signal SIGSEGV, Segmentation fault.
> >> 0x0811b581 in wait_for_thread_shutdown () at Python/pythonrun.c:1720
> >> 1720 PyObject *threading =
> PyMapping_GetItemString(tstate->interp->modules,
> >> (gdb) where
> >> #0 0x0811b581 in wait_for_thread_shutdown () at Python/pythonrun.c:1720
> >> #1 Py_Finalize () at Python/pythonrun.c:416
> >> #2 0xf6f8923f in Register::GPIO_Callback_Python (this=0xf7130f78
> <GeneralPurposeInputOutput::GPIOB+84>)
> >> at
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:48
> >> #3 0xf6f8a97d in Register_GPIO_BSRRL::operator= (this=0xf7130f78
> <GeneralPurposeInputOutput::GPIOB+84>, arg=8)
> >> at
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register_GPIO_BSRRL.cpp:30
> >> #4 0xf6f88b7c in digitalWrite (pin=<optimized out>, val=<optimized
> out>)
> >> at
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../cores/arduelli/wiring_digital.h:58
> >> #5 0xf76005b6 in ffi_call_SYSV ()
> >> at
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/libffi/src/x86/sysv.S:65
> >> #6 0xf7600328 in ffi_call (cif=<optimized out>, cif@entry=0xffffb3e4,
> fn=<optimized out>, rvalue=<optimized out>, avalue=<optimized out>)
> >> at
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/libffi/src/x86/ffi.c:411
> >> #7 0xf75f8c44 in _call_function_pointer (argcount=<optimized out>,
> resmem=<optimized out>, restype=<optimized out>, atypes=<optimized out>,
> >> avalues=<optimized out>, pProc=<optimized out>, flags=<optimized
> out>)
> >> at
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/callproc.c:836
> >> #8 _ctypes_callproc (pProc=pProc@entry=0xf6f88bc3
> <t_digitalWrite(pin_t, uint32_t)>, argtuple=argtuple@entry=0xf7370b2c,
> flags=4353,
> >> argtypes=argtypes@entry=0x0, restype=restype@entry=0xf763d3dc,
> checker=checker@entry=0x0)
> >> at
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/callproc.c:1183
> >> #9 0xf75ef851 in PyCFuncPtr_call (self=0xf724bb8c, inargs=0xf7370b2c,
> kwds=0x0)
> >> at
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/_ctypes.c:3965
> >> #10 0x08066b13 in PyObject_Call (func=func@entry=0xf724bb8c,
> arg=arg@entry=0xf7370b2c, kw=kw@entry=0x0) at Objects/abstract.c:2529
> >> #11 0x080ee3af in do_call (nk=<optimized out>, na=<optimized out>,
> pp_stack=0xffffb548, func=0xf724bb8c) at Python/ceval.c:4251
> >> #12 call_function (oparg=<optimized out>, pp_stack=0xffffb548) at
> Python/ceval.c:4056
> >> #13 PyEval_EvalFrameEx (f=f@entry=0xf7172a04, throwflag=throwflag@entry=0)
> at Python/ceval.c:2679
> >> #14 0x080f1467 in fast_function (nk=<optimized out>, na=<optimized
> out>, n=3, pp_stack=0xffffb608, func=0xf7144fb4) at Python/ceval.c:4119
> >> #15 call_function (oparg=<optimized out>, pp_stack=0xffffb608) at
> Python/ceval.c:4054
> >> #16 PyEval_EvalFrameEx (f=f@entry=0xf717419c, throwflag=throwflag@entry=0)
> at Python/ceval.c:2679
> >> #17 0x080f318f in PyEval_EvalCodeEx (co=0xf71ad0f8, globals=0xf71afb54,
> locals=locals@entry=0x0, args=args@entry=0xf7370b78, argcount=2,
> >> kws=kws@entry=0xf7d58038, kwcount=kwcount@entry=0, defs=defs@entry=0x0,
> defcount=defcount@entry=0, closure=0x0) at Python/ceval.c:3265
> >> #18 0x08160676 in function_call (func=0xf7149454, arg=0xf7370b6c,
> kw=0xf7169934) at Objects/funcobject.c:526
> >> #19 0x08066b13 in PyObject_Call (func=func@entry=0xf7149454,
> arg=arg@entry=0xf7370b6c, kw=kw@entry=0xf7169934) at
> Objects/abstract.c:2529
> >> #20 0x080ef4fe in ext_do_call (nk=<optimized out>, na=0,
> flags=<optimized out>, pp_stack=0xffffb7d8, func=0xf7149454) at
> Python/ceval.c:4346
> >> #21 PyEval_EvalFrameEx (f=f@entry=0xf716d86c, throwflag=throwflag@entry=0)
> at Python/ceval.c:2718
> >> #22 0x080f318f in PyEval_EvalCodeEx (co=co@entry=0xf7452bf0,
> globals=globals@entry=0xf74c29bc, locals=locals@entry=0x0,
> >> args=args@entry=0xf7168b4c, argcount=argcount@entry=0,
> kws=0xf7168b4c, kwcount=0, defs=0x0, defcount=0, closure=0xf71b68c4)
> >> at Python/ceval.c:3265
> >> #23 0x080f13a5 in fast_function (nk=<optimized out>, na=0, n=<optimized
> out>, pp_stack=0xffffb918, func=0xf71574c4) at Python/ceval.c:4129
> >>
> ------------------------------------------------------------------------------------------
> >>
> >>
> >> On Wed, Oct 15, 2014 at 10:15 PM, Ivan Raul <supra.material@xxxxxxxxx>
> wrote:
> >> Hi,
> >>
> >> I want to help. Asif, what is the output of:
> >>
> >> python --version
> >> which python
> >> uname -a
> >>
> >> With Warm Regards, Ivan Raul
> >>
> >> On Wed, Oct 15, 2014 at 8:52 PM, Jose Granados Vergara <
> jose.granados@xxxxxx> wrote:
> >> Could you run the file command on the executable to see what it is?
> >>
> >> There are some libraries that need to be installed on Ubuntu 64 bits to
> run 32 bits (ia32-libs, gcc-multilib, g++-multilib, binutils-multiarch,
> ia32-libs-dev).
> >>
> >> As a last resource you can try 32 bits isolation by installing a 32 bit
> Ubuntu in a virtual machine.
> >>
> >> -Jose
> >> De: Asif Sardar [engr.asif.sardar@xxxxxxxxxxxxxx]
> >> Enviado: miércoles, 15 de octubre de 2014 01:48 p.m.
> >> Para: ell-i-developers@xxxxxxxxxxxxx
> >> CC: Jose Granados Vergara; Lari Lehtomäki
> >> Asunto: Re: [ell-i-developers] Re: Embedding python in emulator
> >>
> >> Yup, cross-compiling will not work, in website the procedure was
> basically cross-compiling python for target platform.
> >>
> >> Anyway, it seems to be 32-bit vs 64-bit platform issue. I separately
> compiled and linked a C++ program to embed python in it using 32-bit
> python. It compiled and linked with those CFLAGS and LDFLAGS, but the
> executable was not running reporting error: cannot execute binary file:
> Exec format error. I am using 64 bit OS Ubuntu and when I compile the above
> program using the default python I have in Ubuntu, it compiled, linked and
> worked. Also, the following errors seems to be processor architectural
> issues with binaries. I don't have enough skills with kernel-level or OS
> system level debugging issues, so if anybody else got any ideas then please
> share with us on this platform.
> >>
> >> The program is also attached to this e-mail, its very simple program to
> test embedding python in C/C++ programs.
> >>
> >> I will stick to the procedure where python calls C/C++ code in emulator
> and in return gets values from the emulator using callbacks.
> >>
> >> On Tue, Oct 14, 2014 at 8:58 PM, Pekka Nikander <pekka.nikander@xxxxxx>
> wrote:
> >> I don't think cross-compiling python for ARM is the right way. You
> will not be able to run the result anywhere.
> >>
> >> --Pekka
> >>
> >>
> >> On 2014–10–14, at 20:27 , Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx>
> wrote:
> >>
> >>> By native I meant my own laptop e.g. using gcc-linux to compile
> python. I want to use python interpretor to be called from emulator code to
> call some python functions (This will help later on for multi-purposes e.g.
> we can call python from C/C++ independently). Currently, the callbacks I
> have implemented have to pass-on information to python by actually calling
> the emulator from python itself (2-way calling). If I am able to embed
> python in emulator, it will asynchronously or independently call python
> whenever needed from C/C++ code. Actually, the test libraries are written
> in python language in Robot Framework that is why I am interested to do
> these kind of functionality to both C/C++ and python.
> >>>
> >>> I will try the steps in
> https://www.ailis.de/~k/archives/19-ARM-cross-compiling-howto.html#python
> to cross-compile the python for ARM itself. I will share the findings here!
> >>>
> >>> On Tue, Oct 14, 2014 at 8:12 PM, Jose Granados Vergara <
> jose.granados@xxxxxx> wrote:
> >>> Isn't the emulator native? why do you need to bind Python to the ARM
> binaries?
> >>>
> >>> -Jose
> >>> De: Asif Sardar [engr.asif.sardar@xxxxxxxxxxxxxx]
> >>> Enviado: martes, 14 de octubre de 2014 08:04 p.m.
> >>> Para: Jose Granados Vergara
> >>> CC: ell-i-developers@xxxxxxxxxxxxx; Lari Lehtomäki
> >>> Asunto: Re: Embedding python in emulator
> >>>
> >>> Thanks Jose,
> >>>
> >>> On native platform, its working for me as well. I think I have to
> re-compile or cross-compile 32-bit python using ARM tools e.g. using
> something like CXX=arm-none-linux-gnueabi-g++ and then rebuild robot
> framework.
> >>>
> >>> I have to compile the emulator files using arm-tool chain and I am
> including the 32-bit python in it built with native gcc toolchain.
> >>>
> >>> If anyone else has some suggestions that I will really appreciate!
> >>>
> >>> On Tue, Oct 14, 2014 at 7:58 PM, Jose Granados Vergara <
> jose.granados@xxxxxx> wrote:
> >>> This worked for me:
> >>>
> >>> something.c:
> >>>
> >>> #include "Python.h"
> >>>
> >>> int main()
> >>> {
> >>> Py_Initialize();
> >>> return 0;
> >>> }
> >>>
> >>> ~$ g++ -I/usr/include/python2.7 something.c -o something -lpython2.7
> >>>
> >>>
> >>> -Jose
> >>> De: Asif Sardar [engr.asif.sardar@xxxxxxxxxxxxxx]
> >>> Enviado: martes, 14 de octubre de 2014 07:40 p.m.
> >>> Para: ell-i-developers@xxxxxxxxxxxxx
> >>> CC: Lari Lehtomäki; Jose Granados Vergara
> >>> Asunto: Fwd: Embedding python in emulator
> >>>
> >>> Hi,
> >>>
> >>> Has anybody embedded python in C/C++ application. My platform is
> 64-bit and python binaries I am using for Runtime/emulator is 32-bit. I
> have added following flags in CFLAGS and LDFLAGS to compile and link the
> c/c++ files in emulator with (#include
> </path/32-bit-python/include/python.h>)
> >>>
> >>> CFLAGS += -I/opt/pym32/bin/include/python2.7 -fno-strict-aliasing
> -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
> >>> LDFLAGS += -L/opt/pym32/lib/python2.7/config -lpthread -ldl -lutil -lm
> -lpython2.7 -Xlinker -export-dynamic
> >>> The files compiles but breaks when linking. Can anybody help with
> following errors:
> >>>
> >>> -----------------------
> >>> ./libsystem_stm32f4discovery.a(Register.o): In function
> `Register::GPIO_Callback_Python(std::string, std::string, unsigned int,
> std::string)':
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:41:
> undefined reference to `Py_Initialize'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:43:
> undefined reference to `PyRun_SimpleStringFlags'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:44:
> undefined reference to `Py_Finalize'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:41:
> undefined reference to `Py_Initialize'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:43:
> undefined reference to `PyRun_SimpleStringFlags'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:44:
> undefined reference to `Py_Finalize'
> >>> ./libsystem_stm32f4discovery.a(system_stm32f4xx.o): In function
> `__exchange_and_add':
> >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to
> `__atomic_fetch_add_4'
> >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to
> `__atomic_fetch_add_4'
> >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to
> `__atomic_fetch_add_4'
> >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to
> `__atomic_fetch_add_4'
> >>> -----------------------
> >>>
> >>>
> >>> ---------- Forwarded message ----------
> >>> From: Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx>
> >>> Date: Mon, Oct 13, 2014 at 6:17 PM
> >>> Subject: Embedding python in emulator
> >>> To: Pekka Nikander <pekka.nikander@xxxxxx>
> >>>
> >>>
> >>> Hi Pekka,
> >>>
> >>> I am trying to include python.h in emulator/inc/register.h
> >>>
> >>> I have to include cflags and ldflags in Makefile system of emulator
> e.g.
> >>>
> >>> -I/opt/include/python2.7 -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3
> -Wall -Wstrict-prototypes
> >>> -L/opt/lib/python2.7/config -lpthread -ldl -lutil -lm -lpython2.7
> -Xlinker -export-dynamic
> >>> However, I am getting:
> >>> ----------------------------------------------------
> >>> g++ -m32 -march=i386 -m32 -march=i386 -Xlinker -export-dynamic -o
> robot_library robot_library.o stm32f4discovery.o libSPI.a "-L."
> "-L/home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../variants/stm32f4discovery"
> "-lsystem_stm32f4discovery" -lstdc++
> >>> ./libsystem_stm32f4discovery.a(Register.o): In function
> `Register::GPIO_Callback_Python(std::string, std::string, unsigned int,
> std::string)':
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:41:
> undefined reference to `Py_Initialize'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:43:
> undefined reference to `PyRun_SimpleStringFlags'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:44:
> undefined reference to `Py_Finalize'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:41:
> undefined reference to `Py_Initialize'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:43:
> undefined reference to `PyRun_SimpleStringFlags'
> >>>
> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:44:
> undefined reference to `Py_Finalize'
> >>> ./libsystem_stm32f4discovery.a(system_stm32f4xx.o): In function
> `__exchange_and_add':
> >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to
> `__atomic_fetch_add_4'
> >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to
> `__atomic_fetch_add_4'
> >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to
> `__atomic_fetch_add_4'
> >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to
> `__atomic_fetch_add_4'
> >>> collect2: error: ld returned 1 exit status
> >>> make[1]: *** [robot_library] Error 1
> >>> -----------------------------------
> >>>
> >>> Can you help me in including these flags in the Makefile system and
> link the python with it? It will help a lot, to call from c to python
> independently.
> >>>
> >>>
> >>>
> >>> --
> >>> With Best Regards,
> >>> Asif Sardar.
> >>> +358 43 8265795
> >>>
> >>>
> >>>
> >>> --
> >>> With Best Regards,
> >>> Asif Sardar.
> >>> +358 43 8265795
> >>>
> >>>
> >>>
> >>> --
> >>> With Best Regards,
> >>> Asif Sardar.
> >>> +358 43 8265795
> >>>
> >>>
> >>>
> >>> --
> >>> With Best Regards,
> >>> Asif Sardar.
> >>> +358 43 8265795
> >>
> >>
> >>
> >>
> >> --
> >> With Best Regards,
> >> Asif Sardar.
> >> +358 43 8265795
> >>
> >>
> >>
> >>
> >> --
> >> With Best Regards,
> >> Asif Sardar.
> >> +358 43 8265795
> >
>
>
--
*With Best Regards,Asif Sardar.+358 43 8265795*
Other related posts:
