Nice progress! Overall your solution sounds good to me. I think it should be possible to integrate the cpython extesion module with the emulator library, so that they are no longer separate, but I have no idea how exactly to do that. --Pekka On 2014–10–22, at 16:04 , Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx> wrote: > Hello all, > > Calling python from c-code now works. I have been studying and working on it > since more then a week, but it was good learning process. I have now good > understanding for advanced usage of "Embedding python in C" and "Extending > python in C". Both have almost same CPython-API, but usage is different. In > our case, for providing a bridge between the emulator code and python, I had > to write a c-extension module to pass on the reference of python callback > function to the emulator and using the same module to callback python > function using the reference from emulator. > > As we are loading the emulator as a shared library and the code to call > python callback using its reference and CPython-API is written in emulator, > the python is already initialized when we loaded the emulator shared library. > The only wrong thing I was doing is to acquire it in wrong context and then > trying to close it in the same python thread. After reading the python > documentation, I realized that I was not locking and unlocking the python > interpreter (Global Interpreter Lock -> GIL) to call CPython APIs. Now, using > the extension module which possess the static reference to the python > callback can be used in the CPython API in emulator. > > I had been doing the following which was a good learning process: > > 1- Using python sub-interpretor (didn't worked because all the modules, > extension modules and local and global variables are initialized again with > new state). > > 2- Using the c global variables (extern in header files) and using it as > bridge between python and emulator. > > 3- Calling the python script file emulator to store results in a file and > then read that file in python (ugly solution and very inflexible). > > 4- Calling the python callback in python script file directly using CPython > API in emulator (didn't worked to load the script file as module). > > 5- The neat and clean solution is to write a cpython extension module, > compile it, store the .so library of it in the emulator/src/ directory and > the test-scripts/emulator/ library. Then load it in python scripts or > emulator when needed. > > Any other solutions or suggestions are welcomed! > > > > On Wed, Oct 15, 2014 at 11:02 PM, Pekka Nikander <pekka.nikander@xxxxxx> > wrote: > You are not embedding a python interpreter yourself, so you shouldn't call > Py_Initialize. That's been called already. You shouldn't call Py_Finanize > neither, as that destroys the Python interpreter there is. > > You can check that with Py_IsInitialized. > > My guess is that also PyEval_ThreadsInitialized should return true, meaning > that you have the Python threads initialised. > > --Pekka > > On 2014–10–15, at 22:53 , Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx> wrote: > > > I am calling: > > > > void Register::pythonCallback() > > { > > Py_Initializa(); > > Py_Finalize(); > > } > > > > in Register.cpp through = operator in BSRR class > > On Oct 15, 2014 10:43 PM, "Pekka Nikander" <pekka.nikander@xxxxxx> wrote: > > My gut feeling is that you are somehow acquiring the Python context in a > > wrong way. > > > > It would help if you posted the relevant code from your Register.cpp. Yes, > > I did see it today, when I uncommented it, but I didn't really look at it > > from this point of view, and therefore I don't remember what there was. > > > > --Pekka > > > > On 2014–10–15, at 22:37 , Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx> > > wrote: > > > >> Hi Ivan, > >> > >> Thanks! > >> > >> python 2.7.8 (32-bit manually compiled) > >> Linux asif-Satellite-C655 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 > >> 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > >> > >> Pekka assisted me solving the issue! > >> > >> I am calling the python interpreter from Register.cpp only using ( > >> Py_Initialize() and Py_Finalize() ), still after running the rf test case > >> under gdb, I get following segmentation fault: Does that mean I have to > >> call the python interpretor in a seperate thread or so from the > >> Register.cpp class? Any guesses ... I am not calling python interpreter > >> from the main.c or so. > >> ------------------------------------------------------------------------------------------ > >> ============================================================================== > >> DigitalRead > >> ============================================================================== > >> Read high from pin :: Test Setup: Set the pin mode to output. Test... > >> GPIO:MODER:value=0 -> =40 > >> .GPIO:MODER:value=40 -> =40 > >> GPIO:BSRRL:value=0 -> =0 > >> Here we call python from Register Class > >> > >> Program received signal SIGSEGV, Segmentation fault. > >> 0x0811b581 in wait_for_thread_shutdown () at Python/pythonrun.c:1720 > >> 1720 PyObject *threading = > >> PyMapping_GetItemString(tstate->interp->modules, > >> (gdb) where > >> #0 0x0811b581 in wait_for_thread_shutdown () at Python/pythonrun.c:1720 > >> #1 Py_Finalize () at Python/pythonrun.c:416 > >> #2 0xf6f8923f in Register::GPIO_Callback_Python (this=0xf7130f78 > >> <GeneralPurposeInputOutput::GPIOB+84>) > >> at > >> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:48 > >> #3 0xf6f8a97d in Register_GPIO_BSRRL::operator= (this=0xf7130f78 > >> <GeneralPurposeInputOutput::GPIOB+84>, arg=8) > >> at > >> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register_GPIO_BSRRL.cpp:30 > >> #4 0xf6f88b7c in digitalWrite (pin=<optimized out>, val=<optimized out>) > >> at > >> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../cores/arduelli/wiring_digital.h:58 > >> #5 0xf76005b6 in ffi_call_SYSV () > >> at > >> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/libffi/src/x86/sysv.S:65 > >> #6 0xf7600328 in ffi_call (cif=<optimized out>, cif@entry=0xffffb3e4, > >> fn=<optimized out>, rvalue=<optimized out>, avalue=<optimized out>) > >> at > >> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/libffi/src/x86/ffi.c:411 > >> #7 0xf75f8c44 in _call_function_pointer (argcount=<optimized out>, > >> resmem=<optimized out>, restype=<optimized out>, atypes=<optimized out>, > >> avalues=<optimized out>, pProc=<optimized out>, flags=<optimized out>) > >> at > >> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/callproc.c:836 > >> #8 _ctypes_callproc (pProc=pProc@entry=0xf6f88bc3 <t_digitalWrite(pin_t, > >> uint32_t)>, argtuple=argtuple@entry=0xf7370b2c, flags=4353, > >> argtypes=argtypes@entry=0x0, restype=restype@entry=0xf763d3dc, > >> checker=checker@entry=0x0) > >> at > >> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/callproc.c:1183 > >> #9 0xf75ef851 in PyCFuncPtr_call (self=0xf724bb8c, inargs=0xf7370b2c, > >> kwds=0x0) > >> at > >> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Testing/Python-2.7.8/Modules/_ctypes/_ctypes.c:3965 > >> #10 0x08066b13 in PyObject_Call (func=func@entry=0xf724bb8c, > >> arg=arg@entry=0xf7370b2c, kw=kw@entry=0x0) at Objects/abstract.c:2529 > >> #11 0x080ee3af in do_call (nk=<optimized out>, na=<optimized out>, > >> pp_stack=0xffffb548, func=0xf724bb8c) at Python/ceval.c:4251 > >> #12 call_function (oparg=<optimized out>, pp_stack=0xffffb548) at > >> Python/ceval.c:4056 > >> #13 PyEval_EvalFrameEx (f=f@entry=0xf7172a04, throwflag=throwflag@entry=0) > >> at Python/ceval.c:2679 > >> #14 0x080f1467 in fast_function (nk=<optimized out>, na=<optimized out>, > >> n=3, pp_stack=0xffffb608, func=0xf7144fb4) at Python/ceval.c:4119 > >> #15 call_function (oparg=<optimized out>, pp_stack=0xffffb608) at > >> Python/ceval.c:4054 > >> #16 PyEval_EvalFrameEx (f=f@entry=0xf717419c, throwflag=throwflag@entry=0) > >> at Python/ceval.c:2679 > >> #17 0x080f318f in PyEval_EvalCodeEx (co=0xf71ad0f8, globals=0xf71afb54, > >> locals=locals@entry=0x0, args=args@entry=0xf7370b78, argcount=2, > >> kws=kws@entry=0xf7d58038, kwcount=kwcount@entry=0, > >> defs=defs@entry=0x0, defcount=defcount@entry=0, closure=0x0) at > >> Python/ceval.c:3265 > >> #18 0x08160676 in function_call (func=0xf7149454, arg=0xf7370b6c, > >> kw=0xf7169934) at Objects/funcobject.c:526 > >> #19 0x08066b13 in PyObject_Call (func=func@entry=0xf7149454, > >> arg=arg@entry=0xf7370b6c, kw=kw@entry=0xf7169934) at > >> Objects/abstract.c:2529 > >> #20 0x080ef4fe in ext_do_call (nk=<optimized out>, na=0, flags=<optimized > >> out>, pp_stack=0xffffb7d8, func=0xf7149454) at Python/ceval.c:4346 > >> #21 PyEval_EvalFrameEx (f=f@entry=0xf716d86c, throwflag=throwflag@entry=0) > >> at Python/ceval.c:2718 > >> #22 0x080f318f in PyEval_EvalCodeEx (co=co@entry=0xf7452bf0, > >> globals=globals@entry=0xf74c29bc, locals=locals@entry=0x0, > >> args=args@entry=0xf7168b4c, argcount=argcount@entry=0, kws=0xf7168b4c, > >> kwcount=0, defs=0x0, defcount=0, closure=0xf71b68c4) > >> at Python/ceval.c:3265 > >> #23 0x080f13a5 in fast_function (nk=<optimized out>, na=0, n=<optimized > >> out>, pp_stack=0xffffb918, func=0xf71574c4) at Python/ceval.c:4129 > >> ------------------------------------------------------------------------------------------ > >> > >> > >> On Wed, Oct 15, 2014 at 10:15 PM, Ivan Raul <supra.material@xxxxxxxxx> > >> wrote: > >> Hi, > >> > >> I want to help. Asif, what is the output of: > >> > >> python --version > >> which python > >> uname -a > >> > >> With Warm Regards, Ivan Raul > >> > >> On Wed, Oct 15, 2014 at 8:52 PM, Jose Granados Vergara > >> <jose.granados@xxxxxx> wrote: > >> Could you run the file command on the executable to see what it is? > >> > >> There are some libraries that need to be installed on Ubuntu 64 bits to > >> run 32 bits (ia32-libs, gcc-multilib, g++-multilib, binutils-multiarch, > >> ia32-libs-dev). > >> > >> As a last resource you can try 32 bits isolation by installing a 32 bit > >> Ubuntu in a virtual machine. > >> > >> -Jose > >> De: Asif Sardar [engr.asif.sardar@xxxxxxxxxxxxxx] > >> Enviado: miércoles, 15 de octubre de 2014 01:48 p.m. > >> Para: ell-i-developers@xxxxxxxxxxxxx > >> CC: Jose Granados Vergara; Lari Lehtomäki > >> Asunto: Re: [ell-i-developers] Re: Embedding python in emulator > >> > >> Yup, cross-compiling will not work, in website the procedure was basically > >> cross-compiling python for target platform. > >> > >> Anyway, it seems to be 32-bit vs 64-bit platform issue. I separately > >> compiled and linked a C++ program to embed python in it using 32-bit > >> python. It compiled and linked with those CFLAGS and LDFLAGS, but the > >> executable was not running reporting error: cannot execute binary file: > >> Exec format error. I am using 64 bit OS Ubuntu and when I compile the > >> above program using the default python I have in Ubuntu, it compiled, > >> linked and worked. Also, the following errors seems to be processor > >> architectural issues with binaries. I don't have enough skills with > >> kernel-level or OS system level debugging issues, so if anybody else got > >> any ideas then please share with us on this platform. > >> > >> The program is also attached to this e-mail, its very simple program to > >> test embedding python in C/C++ programs. > >> > >> I will stick to the procedure where python calls C/C++ code in emulator > >> and in return gets values from the emulator using callbacks. > >> > >> On Tue, Oct 14, 2014 at 8:58 PM, Pekka Nikander <pekka.nikander@xxxxxx> > >> wrote: > >> I don't think cross-compiling python for ARM is the right way. You will > >> not be able to run the result anywhere. > >> > >> --Pekka > >> > >> > >> On 2014–10–14, at 20:27 , Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx> > >> wrote: > >> > >>> By native I meant my own laptop e.g. using gcc-linux to compile python. I > >>> want to use python interpretor to be called from emulator code to call > >>> some python functions (This will help later on for multi-purposes e.g. we > >>> can call python from C/C++ independently). Currently, the callbacks I > >>> have implemented have to pass-on information to python by actually > >>> calling the emulator from python itself (2-way calling). If I am able to > >>> embed python in emulator, it will asynchronously or independently call > >>> python whenever needed from C/C++ code. Actually, the test libraries are > >>> written in python language in Robot Framework that is why I am interested > >>> to do these kind of functionality to both C/C++ and python. > >>> > >>> I will try the steps in > >>> https://www.ailis.de/~k/archives/19-ARM-cross-compiling-howto.html#python > >>> to cross-compile the python for ARM itself. I will share the findings > >>> here! > >>> > >>> On Tue, Oct 14, 2014 at 8:12 PM, Jose Granados Vergara > >>> <jose.granados@xxxxxx> wrote: > >>> Isn't the emulator native? why do you need to bind Python to the ARM > >>> binaries? > >>> > >>> -Jose > >>> De: Asif Sardar [engr.asif.sardar@xxxxxxxxxxxxxx] > >>> Enviado: martes, 14 de octubre de 2014 08:04 p.m. > >>> Para: Jose Granados Vergara > >>> CC: ell-i-developers@xxxxxxxxxxxxx; Lari Lehtomäki > >>> Asunto: Re: Embedding python in emulator > >>> > >>> Thanks Jose, > >>> > >>> On native platform, its working for me as well. I think I have to > >>> re-compile or cross-compile 32-bit python using ARM tools e.g. using > >>> something like CXX=arm-none-linux-gnueabi-g++ and then rebuild robot > >>> framework. > >>> > >>> I have to compile the emulator files using arm-tool chain and I am > >>> including the 32-bit python in it built with native gcc toolchain. > >>> > >>> If anyone else has some suggestions that I will really appreciate! > >>> > >>> On Tue, Oct 14, 2014 at 7:58 PM, Jose Granados Vergara > >>> <jose.granados@xxxxxx> wrote: > >>> This worked for me: > >>> > >>> something.c: > >>> > >>> #include "Python.h" > >>> > >>> int main() > >>> { > >>> Py_Initialize(); > >>> return 0; > >>> } > >>> > >>> ~$ g++ -I/usr/include/python2.7 something.c -o something -lpython2.7 > >>> > >>> > >>> -Jose > >>> De: Asif Sardar [engr.asif.sardar@xxxxxxxxxxxxxx] > >>> Enviado: martes, 14 de octubre de 2014 07:40 p.m. > >>> Para: ell-i-developers@xxxxxxxxxxxxx > >>> CC: Lari Lehtomäki; Jose Granados Vergara > >>> Asunto: Fwd: Embedding python in emulator > >>> > >>> Hi, > >>> > >>> Has anybody embedded python in C/C++ application. My platform is 64-bit > >>> and python binaries I am using for Runtime/emulator is 32-bit. I have > >>> added following flags in CFLAGS and LDFLAGS to compile and link the c/c++ > >>> files in emulator with (#include </path/32-bit-python/include/python.h>) > >>> > >>> CFLAGS += -I/opt/pym32/bin/include/python2.7 -fno-strict-aliasing > >>> -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes > >>> LDFLAGS += -L/opt/pym32/lib/python2.7/config -lpthread -ldl -lutil -lm > >>> -lpython2.7 -Xlinker -export-dynamic > >>> The files compiles but breaks when linking. Can anybody help with > >>> following errors: > >>> > >>> ----------------------- > >>> ./libsystem_stm32f4discovery.a(Register.o): In function > >>> `Register::GPIO_Callback_Python(std::string, std::string, unsigned int, > >>> std::string)': > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:41: > >>> undefined reference to `Py_Initialize' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:43: > >>> undefined reference to `PyRun_SimpleStringFlags' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:44: > >>> undefined reference to `Py_Finalize' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:41: > >>> undefined reference to `Py_Initialize' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:43: > >>> undefined reference to `PyRun_SimpleStringFlags' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:44: > >>> undefined reference to `Py_Finalize' > >>> ./libsystem_stm32f4discovery.a(system_stm32f4xx.o): In function > >>> `__exchange_and_add': > >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to > >>> `__atomic_fetch_add_4' > >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to > >>> `__atomic_fetch_add_4' > >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to > >>> `__atomic_fetch_add_4' > >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to > >>> `__atomic_fetch_add_4' > >>> ----------------------- > >>> > >>> > >>> ---------- Forwarded message ---------- > >>> From: Asif Sardar <engr.asif.sardar@xxxxxxxxxxxxxx> > >>> Date: Mon, Oct 13, 2014 at 6:17 PM > >>> Subject: Embedding python in emulator > >>> To: Pekka Nikander <pekka.nikander@xxxxxx> > >>> > >>> > >>> Hi Pekka, > >>> > >>> I am trying to include python.h in emulator/inc/register.h > >>> > >>> I have to include cflags and ldflags in Makefile system of emulator e.g. > >>> > >>> -I/opt/include/python2.7 -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 > >>> -Wall -Wstrict-prototypes > >>> -L/opt/lib/python2.7/config -lpthread -ldl -lutil -lm -lpython2.7 > >>> -Xlinker -export-dynamic > >>> However, I am getting: > >>> ---------------------------------------------------- > >>> g++ -m32 -march=i386 -m32 -march=i386 -Xlinker -export-dynamic -o > >>> robot_library robot_library.o stm32f4discovery.o libSPI.a "-L." > >>> "-L/home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../variants/stm32f4discovery" > >>> "-lsystem_stm32f4discovery" -lstdc++ > >>> ./libsystem_stm32f4discovery.a(Register.o): In function > >>> `Register::GPIO_Callback_Python(std::string, std::string, unsigned int, > >>> std::string)': > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:41: > >>> undefined reference to `Py_Initialize' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:43: > >>> undefined reference to `PyRun_SimpleStringFlags' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:44: > >>> undefined reference to `Py_Finalize' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:41: > >>> undefined reference to `Py_Initialize' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:43: > >>> undefined reference to `PyRun_SimpleStringFlags' > >>> /home/asif/Ell-i-Working-Directory/Ell-i-Software-Development/Runtime/stm32/tests/robot_library/../../emulator/src/Register.cpp:44: > >>> undefined reference to `Py_Finalize' > >>> ./libsystem_stm32f4discovery.a(system_stm32f4xx.o): In function > >>> `__exchange_and_add': > >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to > >>> `__atomic_fetch_add_4' > >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to > >>> `__atomic_fetch_add_4' > >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to > >>> `__atomic_fetch_add_4' > >>> /usr/include/c++/4.8/ext/atomicity.h:49: undefined reference to > >>> `__atomic_fetch_add_4' > >>> collect2: error: ld returned 1 exit status > >>> make[1]: *** [robot_library] Error 1 > >>> ----------------------------------- > >>> > >>> Can you help me in including these flags in the Makefile system and link > >>> the python with it? It will help a lot, to call from c to python > >>> independently. > >>> > >>> > >>> > >>> -- > >>> With Best Regards, > >>> Asif Sardar. > >>> +358 43 8265795 > >>> > >>> > >>> > >>> -- > >>> With Best Regards, > >>> Asif Sardar. > >>> +358 43 8265795 > >>> > >>> > >>> > >>> -- > >>> With Best Regards, > >>> Asif Sardar. > >>> +358 43 8265795 > >>> > >>> > >>> > >>> -- > >>> With Best Regards, > >>> Asif Sardar. > >>> +358 43 8265795 > >> > >> > >> > >> > >> -- > >> With Best Regards, > >> Asif Sardar. > >> +358 43 8265795 > >> > >> > >> > >> > >> -- > >> With Best Regards, > >> Asif Sardar. > >> +358 43 8265795 > > > > > > > -- > With Best Regards, > Asif Sardar. > +358 43 8265795
Attachment:
smime.p7s
Description: S/MIME cryptographic signature