Hello LKL Community,
I have started working on a Google Summer of Code project that aims to
integrate LKL with ns-3’s DCE(
https://github.com/direct-code-execution/ns-3-dce) to enable host
simulation scripts to make use of the linux networking stack to make
realistic simulations of complex networks.
DCE Currently makes use of net-next-nuse-4.4.0 to extend support to Linux
kernel version 4.4, but over the years the project hasn’t kept pace with
the newer linux releases and thus I am now working towards bringing in
support for the latest linux kernel releases. LKL interested me because it
currently supports Linux 5.3 which is somewhat close to what I am trying to
achieve (support for latest kernels could be added with little more work).
I had initially come up with a solution to make this integration. I set up
all the host operations and important structures like
lkl_host_ops(threads,mutexes,semaphores,debug, etc.) and mapped LKL
networking syscalls to our socket calls made in DCE. I then loaded the
liblkl.so as a shared library in DCE and then called an initialization
function which maps all the DCE and shared linux library functions to
respective functions for host simulation scripts to run, also sets up the
netdevices and also calls lkl_start_kernel(...). This is an abstract
summary of what I specifically did to integrate LKL.
On testing the above, I found that things did build, compile and execute,
but on blocking network operations such as accept(...) the applications got
stuck. This was because the LKL kernel scheduler was not scheduling the DCE
threads and thus the client applications could not execute a corresponding
connect(...) call, which leads to the server thread to be sleeping on an
infinite timeout on schedule_timeout(..) and inet_wait_for_connect(...).
To solve the above problem, I tried to make some changes into the LKL
kernel scheduler.
I did the following things :
1. For every schedule_timeout, I put the current DCE calling thread to
sleep for the given timeout , call the DCE scheduler first to execute the
other DCE threads sleeping in the queue, and call the LKL kernel scheduler.
2. In inet_wait_for_connect, after we prepare_to_wait_exclusive(...) to
put the task in a TASK_INTERRUPTIBLE state, put to sleep the current
calling DCE thread and call the DCE scheduler.
3. Note : I did not change the schedule(...) function as some init calls
in start_kernel(...) required scheduling other workqueue etc., and if we
did switch the task(before start_kernel returns), we would start another
DCE thread that would use the kernel stacks which haven't been initialized
yet.
After I did the above changes, after the accept call by the first DCE
thread, control switched to the second thread, and a socket(...) LKL
syscall was invoked, but this time the lkl_syscall(...) entered a deadlock
on getting the cpu lock on lkl_cpu_get(...)
This was probably because only the host thread could be holding the cpu
lock at any particular time (as far as I understood the architecture, do
correct me if I’m going wrong).
Even if we ran lkl_cpu_change_owner for every syscall, it would be
inconsistent and would obviously lead to failures but even if it doesn't
fail, LKL would guarantee it fails. LKL probably works on the idea that
only one host thread acquires the lock and thus when we manually make the
previous thread drop the lock using lkl_cpu_put, make the current thread as
host using set_thread_flag(TIF_HOST_THREAD) and then change the cpu owner,
the subsequent syscalls would fail as the cpu.count keeps increasing which
has an upper bound of 1.
A few important points to consider about DCE :
1. DCE uses Fibres (Light Weight Threads LWP)
2. DCE Task Scheduler can be found here :
https://github.com/direct-code-execution/ns-3-dce/blob/master/model/task-manager.cc#L414
3. DCE loads the linux kernel as a shared library
4. DCE makes use of libc hijacking, i.e. some libc functions are
rewritten in DCE and mapped through macros. For example : every socket(..)
call in a simulation script leads to calling dce_socket(...) which further
makes a lkl_syscall(_NR_socket,..) inside the loaded LKL library.
5. Would add more details as required to better understand the problem
statement
It would be great if the LKL community could help me figure out possible
solutions and propose ideas on how we should approach to integrate LKL into
DCE.
Thank You,
Parth Pratim Chatterjee
