Repository files navigation

A software-in-the-loop simulator for lane keeping assist system (LKAS) using Vrep and C++.If you are using this framework please refer to the following paper:

[1] S. Mohamed, S. De, K. Bimpisidis, V, Nathan, D. Goswami, H. Corporaal, and T. Basten, “IMACS: A Framework for Performance Evaluation of Image Approximation in a Closed-loop System,” in: MECO, 2019.

The controller is designed in Matlab and the control matrices\Phi,\Gamma, and control gainsK (andF) are copied to thesrc/LaneDetection_and_Control/lateralcontrol_multiple.cpp file. The LKAS controller is designed as explained in the following papers:

[2] S. Mohamed, A. U. Awan, D. Goswami, T. Basten, "Designing image-based control systems considering workload variations," in: CDC, 2019.[3] S. Mohamed, D. Zhu, D. Goswami, T. Basten, "Optimising quality-of-control for data-intensive multiprocessor image-based control systems considering workload variations," in: DSD, 2018.

An initial framework using Vrep and Matlab has been presented in:

S. Mohamed, D. Zhu, D. Goswami, T. Basten, "DASA: an open-source design, analysis and simulation framework for automotive image-based control systems," in: MCAA Annual Conference, 2019.

Tested using the following versions. If you would like to use latest versions, you might have to edit the syntax of some functions.Tested with the following versions:

• OS: Ubuntu 18.04
• g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0

1. Install dependent libraries
2. Install OpenCV
3. Install Eigen
4. Install VREP
5. Run IMACS_LKAS

It is suggested that you Forkthis repository by clicking the "fork" button in the top right. This lets you have your own working copy.

Clone the forked (or directlythis) repository:

git clone https://github.com/sajid-mohamed/cppVrepLKAS.gitcd cppVrepLKASpwd

For brevity,$(root)=pwd, i.e. the path tocppVrepLKAS is called as$(root).

The following libraries might be needed for successful execution.

sudo apt-get install libtinfo-devsudo apt-get install libjpeg-devsudo apt-get install libtiff5-devsudo apt-get install libpng-devsudo apt-get install jaspersudo apt-get install libgtk-3-devsudo apt-get install libopenblas-devsudo apt-get install -y libavcodec-devsudo apt-get install -y libavformat-devsudo apt-get install libavutil-devsudo apt-get install libswscale-devsudo apt-get install valgrind# For openCV installationsudo apt-get install cmake## qt4sudo apt-get install qt4-qmakesudo apt-get install libqt4-dev

If opencv is already installed, go to step2.2.

For detailed OpenCV installation: follow steps fromhttps://docs.opencv.org/trunk/d7/d9f/tutorial_linux_install.html.

cd $(root)/externalAppsgit clone https://github.com/opencv/opencv.gitgit clone https://github.com/opencv/opencv_contrib.git

externalApps is a folder already available when yougit clone cppVrepLKAS. It might be better to include all the external apps in this folder.

# Building OpenCV from Source Using CMakecd opencvmkdir buildcd buildcmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local .. -DOPENCV_GENERATE_PKGCONFIG=ONmakesudo make install

Note the path ofopencv*.pc file. This is thePKG_CONFIG_PATH.Typically this should be/usr/lib/pkgconfig. If you are using virtual machine this might be/usr/local/lib/pkgconfig.

In case you have errors withccache (re-)installccache as shown inFAQ 11.

If you have issues withjava, seeFAQ 12 andFAQ 13.

Make sureopencv*.pc file is in the/usr/lib/pkgconfig or change thePKG_CONFIG_PATH to point to this file.To change path follow the steps inFAQ 3 orFAQ 4 depending on whether you havesudo rights.Remember the path to youropencv installation asOPENCV_PATH.

To check ifopencv*.pc is in the/usr/lib/pkgconfig:

cd /usr/lib/pkgconfig[ -f /usr/lib/pkgconfig/opencv*.pc ] && echo "File exist"

You could also usedlocate to find the file.

If you cannot find it, you might not have donemake install in step2.1.

If the above does not work, try:

sudo apt-get install libopencv-dev

cd $(root)/externalAppsgit clone https://gitlab.com/libeigen/eigen.git

The path to eigen is needed later forEIGEN_PATH.

Download the linux version fromhttps://coppeliarobotics.com/downloads.

The license to CoppeliaSim should be obtained from their website.

If you are in an academic institution, you could simply download theEdu version.

Make sure you copy the downloaded file (e.g.CoppeliaSim_Edu_V4_0_0_Ubuntu18_04.tar.xz) to the$(root)/externalApps folder.

cp CoppeliaSim_Edu_V4_0_0_Ubuntu18_04.tar.xz $(root)/externalAppscd $(root)/externalAppsmkdir vreptar -xvf CoppeliaSim_Edu_V4_0_0_Ubuntu18_04.tar.xz -C vrep --strip-components=1

Generally for local tar file:

tar -xvf FILENAME -C FOLDER --strip-components=1

The--strip-components flag is used when a tar file would naturally expand itself into a folder, let say, like github where it examples torepo-name-master folder. Of course you wouldn’t need the first level folder generated here so--strip-components set to 1 would automatically remove that first folder for you. The larger the number is set the deeper nested folders are removed.

TheVREP_PATH is then$(root)/externalApps/vrep.

Please change the paths in the following files to the corresponding paths in your own system.

$(root)/cppVrepLKAS.pro$(root)/src/LaneDetection_and_Control/lane_detection.cpp

1. If the external applications vrep, eigen or opencv are installed to another directory than the$(root)/externalApps, change the correspondingOPENCV_PATH,PKG_CONFIG_PATH,EIGEN_PATH, andVREP_PATH incppVrepLKAS.pro to the corresponding PATH in your system.

To obtain the path, you could runpwd in a terminal opened from the corresponding folder.

2. Change line 121 oflane_detection.cpp tostring out_string = "$(root)/out_imgs";.

cd $(root)bash run.sh

The steps included in the bash scriptrun.sh are the following:

Generate theMakefile usingqmake.

cd $(root)qmake cppVrepLKAS.promake

When you want to runmake in$(root) always runqmake cppVrepLKAS.pro first.make clean cleans the files generated by thismake.

Open a vrep-scene. Vrep needs to be started before running IMACS framework.

cd $VREP_PATH./coppeliaSim.sh ../../vrep-scenes/SmallBias_report.ttt

If you are using an older version of vrep (coppeliaSim.sh is not found), please use the following command:

./vrep.sh ../../vrep-scenes/EnterCurveTest.ttt

cd $(root)# Usage: ./imacsLKAS {version} {simstep}# Minimum one argument needed. (default values) ./imacs 1 0.005./imacsLKAS 1

There might be some dialog windows opening inVREP if you are running it for the first time. This is normal.

You do not have to always close vrep to run imacs again. Juststop simulation in vrep and wait for sometime before executing./imacsLKAS 1 0.005.

In case you encounter some other errors, you can debug using the following command:

valgrind ./imacsLKAS 1

The main file:src/cpp_vrep_api/cpp_vrep_framework.cpp

• The simulation step time, the total simulation time, the initial wait time, the sampling period(s) and delay(s) are hardcoded in this file.
• If you want to add additional input arguments forimacsLKAS, you should do this here.

Controller Design:src/LaneDetection_and_Control/lateralcontrol_multiple.cpp

• The controller matrices are stored in this file.\Phi,\Gamma andK for multiple controllers are stored in the variablesm_phi_aug,m_Gamma_aug andm_K2c.
• The controller currently used is an optimal LQR control for a regulation problem where the objective is for the output to reach zero. If you want to change the type of controller you can edit the functionlateralController::compute_steering_angles().

Image Processing:src/LaneDetection_and_Control/lane_detection.cpp

• This file contains the image processing steps. Currently, we do the steps mentioned in [1]. You should be changing the functions in this file to include your own image processing algorithm.

matrix.hpp [-Wreorder]GivensQR.hpp [-Wreorder]extApiPlatform.c [-Wunused-result]

$(ROOT)/src/cpp_vrep_api/my_vrep_api.cpp:59: vrepAPI::vrepAPI(): Assertion `m_clientID != -1 && "V-REP must be started: ./vrep.sh ../../vrep-scenes/EnterCurveTest.ttt"' failed.

OpenCV Error: Assertion failed (mv && n > 0) in merge, file /home/om/hd/infra/build_opencv/opencv/modules/core/src/convert.cpp, line 327terminate called after throwing an instance of 'cv::Exception'  what():  /home/om/hd/infra/build_opencv/opencv/modules/core/src/convert.cpp:327: error: (-215) mv && n > 0 in function merge

cdsudo gedit .bash_profile

Add the export path commands in `.bash_profile' as shown below.

export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig/

Save and close.bash_profile. Then, open.bashrc.

sudo gedit .bashrc

Add the following line at the end of.bashrc.

source ~/.bash_profile

Save and close the file.Now, the paths should be exported everytime you open a terminal.

If you want to export paths permanently,Open a terminal window usingCtrl+Alt+TRun the command

gedit ~/.profile

Add the line(s) to the bottom of this file and save.

export PKG_CONFIG_PATH=/usr/lib/pkgconfig/

Log out and log in again.A safer way is to use quotes. Doing so is necessary if one or more directories in the original PATH contain spaces. E.g.:

export CLANG="$(root)/clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-18.04/bin/clang"

Alternatively, you can set the paths inpaths.sh and then source paths. Do not forget to includeexport command in the description here.

cd $(IMACSROOT)source paths.sh

Change line 43 insrc/cpp_vrep_api/cpp_vrep_framework.cpp if you want to simulate for a longer time, i.e.int simulation_time = 100; if you want to logically simulate for100 sec.

To start a straight road scene (seevrep-scenes/SmallBias_report.ttt), replace line 7 inrun.sh with the following

gnome-terminal -- sh -c 'cd externalApps/vrep; ./coppeliaSim.sh ../../vrep-scenes/SmallBias_report.ttt; exec bash'

If you are making changes to thecppVrepLKAS.pro file, you would have to run the commandmake clean first in the terminal before runningbash run.sh.

make cleanbash run.sh

Thelateralcontrol_multiple.cpp has a sample controller implemented.

You are required to adapt the codes according to the control design you choose (especially thelateralController::compute_steering_angles() function). Ideally, you should rewrite thelateralController::compute_steering_angles() function.

The current code is for sequential implementation. To enforce pipelining, you have to changesrc/cpp_vrep_api/cpp_vrep_framework.cpp:123-134.Especially, have an array of control inputs (steering_angle values) and only actuate corresponding to the sampling period.

To implement the pipelining, you would also have to amendsrc/cpp_vrep_api/cpp_vrep_framework.cpp:129 to send the corresponding steering_angle instead of the latest one. You might also have to editVrepAPI.sim_delay(time_step) accordingly.

sudo apt install -y ccachesudo /usr/sbin/update-ccache-symlinksecho 'export PATH="/usr/lib/ccache:$PATH"' | tee -a ~/.bashrcsource ~/.bashrc && echo $PATH

You can follow the tutorial here:https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-on-ubuntu-18-04.

sudo apt install default-jresudo apt install default-jdksudo update-alternatives --config javacsudo update-alternatives --config java

Choose the correctjdk version.jre does not havetools.jarUpdate the JAVA_HOME path

sudo gedit /etc/environment

At the end of this file, add the following line, making sure to replace the path with your own copied path:

JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64/bin/"

Modifying this file will set the JAVA_HOME path for all users on your system.Save the file and exit the editor.

Now reload this file to apply the changes to your current session:

source /etc/environment

If you do not havesudo access, follow the steps inFAQ 4.