Hi,
Great project !
Do you have any public branch where we can track your progress and
eventually comment on (dont care if it's "dirty"): it would make it
easir to build/reproduce & help you.
Cheers,
Matthieu Coudron
Le lun. 31 mai 2021 à 14:16, Parth Pratim Chatterjee
<parth27official@xxxxxxxxx> a écrit :
Hello LKL Community,
I have started working on a Google Summer of Code project that aims to
integrate LKL with ns-3’s
DCE(https://github.com/direct-code-execution/ns-3-dce) to enable host
simulation scripts to make use of the linux networking stack to make
realistic simulations of complex networks.
DCE Currently makes use of net-next-nuse-4.4.0 to extend support to Linux
kernel version 4.4, but over the years the project hasn’t kept pace with the
newer linux releases and thus I am now working towards bringing in support
for the latest linux kernel releases. LKL interested me because it currently
supports Linux 5.3 which is somewhat close to what I am trying to achieve
(support for latest kernels could be added with little more work).
I had initially come up with a solution to make this integration. I set up
all the host operations and important structures like
lkl_host_ops(threads,mutexes,semaphores,debug, etc.) and mapped LKL
networking syscalls to our socket calls made in DCE. I then loaded the
liblkl.so as a shared library in DCE and then called an initialization
function which maps all the DCE and shared linux library functions to
respective functions for host simulation scripts to run, also sets up the
netdevices and also calls lkl_start_kernel(...). This is an abstract summary
of what I specifically did to integrate LKL.
On testing the above, I found that things did build, compile and execute, but
on blocking network operations such as accept(...) the applications got
stuck. This was because the LKL kernel scheduler was not scheduling the DCE
threads and thus the client applications could not execute a corresponding
connect(...) call, which leads to the server thread to be sleeping on an
infinite timeout on schedule_timeout(..) and inet_wait_for_connect(...).
To solve the above problem, I tried to make some changes into the LKL kernel
scheduler.
I did the following things :
For every schedule_timeout, I put the current DCE calling thread to sleep for
the given timeout , call the DCE scheduler first to execute the other DCE
threads sleeping in the queue, and call the LKL kernel scheduler.
In inet_wait_for_connect, after we prepare_to_wait_exclusive(...) to put the
task in a TASK_INTERRUPTIBLE state, put to sleep the current calling DCE
thread and call the DCE scheduler.
Note : I did not change the schedule(...) function as some init calls in
start_kernel(...) required scheduling other workqueue etc., and if we did
switch the task(before start_kernel returns), we would start another DCE
thread that would use the kernel stacks which haven't been initialized yet.
After I did the above changes, after the accept call by the first DCE thread,
control switched to the second thread, and a socket(...) LKL syscall was
invoked, but this time the lkl_syscall(...) entered a deadlock on getting the
cpu lock on lkl_cpu_get(...)
This was probably because only the host thread could be holding the cpu lock
at any particular time (as far as I understood the architecture, do correct
me if I’m going wrong).
Even if we ran lkl_cpu_change_owner for every syscall, it would be
inconsistent and would obviously lead to failures but even if it doesn't
fail, LKL would guarantee it fails. LKL probably works on the idea that only
one host thread acquires the lock and thus when we manually make the previous
thread drop the lock using lkl_cpu_put, make the current thread as host using
set_thread_flag(TIF_HOST_THREAD) and then change the cpu owner, the
subsequent syscalls would fail as the cpu.count keeps increasing which has an
upper bound of 1.
A few important points to consider about DCE :
DCE uses Fibres (Light Weight Threads LWP)
DCE Task Scheduler can be found here :
https://github.com/direct-code-execution/ns-3-dce/blob/master/model/task-manager.cc#L414
DCE loads the linux kernel as a shared library
DCE makes use of libc hijacking, i.e. some libc functions are rewritten in
DCE and mapped through macros. For example : every socket(..) call in a
simulation script leads to calling dce_socket(...) which further makes a
lkl_syscall(_NR_socket,..) inside the loaded LKL library.
Would add more details as required to better understand the problem statement
It would be great if the LKL community could help me figure out possible
solutions and propose ideas on how we should approach to integrate LKL into
DCE.
Thank You,
Parth Pratim Chatterjee