NCL automatically generates network traffics for web IT environment

 NATIONAL CYBERSECURITY R&D LAB

OCTOBOT DEMO - FOR WEB IT ENV

ENV = Environment 

OVERVIEW

Experimentation/testing in a large-scale testbed environment requires a large amount of emulated traffic to ensure realistic scenario execution and better experiment results. A large number of human-generated network activities can emulate traffic from real network users. Deploying and producing a single activity from an individual user is simple, but emulating and automating it from multiple users with a wide range of activities is challenging. We designed a containerized human agent (i.e., bot) to generate a single activity. Thus, a large number of bots can be deployed and controlled by a single orchestration system. Due to the complexity and wide-range usage of container orchestration systems, we need to develop a simpler system that leverages widely-used open-source container orchestrators. So, researchers and scientists can easily use it to define and execute activity requirements with a few command lines or a single specification file.

We adopt the well-known concept on BotNet (as described above), who can launch large-scale distributed denial-of-service (DDoS) attacks by orchestrating a large number of "infected" clients (i.e., bots) for executing the attacking tasks. Our system should be able to launch large-scale human activity by orchestrating a large number of "specialized" bot who can emulate human activity tasks (e.g., launching an application, typing input, clicking a button, and so on). Unfortunately, controlling multiple specialized bots with a different set of scripts or software is more challenging than a BotNet. Because usually, the bot in BotNet has almost similar set software or script for the attacks. Our system bot is designed as an independent set of script and software with a set of configurations that is packed (i.e., containerized) for a specific task to emulate the human activity. Then, our orchestration system, called OctoBot, is capable of changing the deployment and execution of the bot and task by modifying a parameter for the activity generation such as the number of bots, type of bot, and specific task to match with the experiment and exercise scenario.

OBJECTIVES

1. Generate “normal” web traffic by simulating random web browsing activity from dozen or hundreds of users

TOPOLOGY SETUP

IT Web Logical Topology

It consists of several components:

• One Web Server (Apache Web Server)

• One Monitoring Server (Elasticsearch + Kibana + Metricbeat)

• One OctoBot Orchestrator

• Ten (10) OctoBot Worker Nodes

Simulated Physical Topology

It consists of Host machine with 11 VMs: 

• Host is used for Web Server + Monitoring Server

• 1 VM for Orchestrator or Controller

• 10 VMs for Worker Node

DEMO GUIDES

Accessing the Environment in NCL Deter Testbed

1. Access to NCL OpenStack website http://ncl.sg 

2. Login with username and password

3. After you see the dashboard, please select “Experiment” from the menu above and Click

4. Go to “Octobot” Experiment and Click “View”

5. Select VNC access by clicking the computer icon

6. It will open a new window and ask for a VNC password. Enter “password” and the desktop will appear.

7. After the desktop appears, click or open Mozilla Firefox browser.

8. Click the “Local Web Server” in the bookmark toolbar to ensure that Web server is running (Default Apache Web Page will appear)

 

9. Click the “ELK Monitoring” in the bookmark toolbar to ensure that Monitoring server is running (Default Kibana Dashboard will appear)

Starting the Traffic (Activity) Generation

1. In the same desktop, please open “Applications” > “Terminal Emulator” 

2. In the terminal window, please got to “/mnt/Octobot” directory by typing:

arisdia1@n0$ cd /mnt/Octobot

3. Access the OctoBot orchestrator by typing:

arisdia1@n0:/mnt/Octobot$ vagrant ssh octobot0

4. Inside the OctoBot orchestrator, please go to “OctoBot” directory by typing:

vagrant@controller:~$ cd OctoBot/Octo-Play

5. Access the OctoBot CLI by typing:

vagrant@controller:OctoBot/Octo-Play$ python3 main.py

6. In the OctoBot CLI, load the demo specification file by typing:

main.py:~$ loadFile web-test.yaml

7. Run the traffic generation by running the specification file:

main.py:~$ runFile web-test.yaml

8. Check the execution by verifying the web browsing bot using this command:

main.py:~$ checkStatus

9. Now only two bots are running (Bot name is started with “web-test-”)

10. Check in the “ELK Monitoring” tab of the web browser and ensure that the “Overall … total_accesses” are slightly increasing.

11. Now, let’s increase the number of bots to dramatically increase the number of web server access

12. Back to the terminal with OctoBot CLI, and check the current config:

main.py:~$ currentConfig

13. Let’s increase the number of bots into 100 by using this command and check the config again:

main.py:~$ setBotNumbers 100

main.py:~$ currentConfig

14. Save the specification file and patch the previous execution:

main.py:~$ writeFile web-test.yaml

main.py:~$ patchFile web-test.yaml

15. Check the execution by verifying the web browsing bot using this command:

main.py:~$ checkStatus

16. Now 100 bots are running (All the output may not be seen)

17. Wait for some minutes and check again in the “ELK Monitoring” tab of the web browser and ensure that the “Overall … total_accesses” are dramatically increasing