1

VMM-based System Security Assurance Ying Wang Partha Dasgupta Abstract—As the Internet been widely accepted and used, more and more critical personal information are stored on computers, security became the most significant concerns for people surfing on the Internet. Software vulnerabilities are common in computer platforms and the software flaws are often exploited to gain controls of a computer. Attacks on computers can be limited by firewall and anti-virus software. However, there is increasing ineffectiveness of these traditional malicious code detection technologies to detect and block new attacks, such as rootkits, keyboard sniffers and screen scapers. In this paper, we propose the idea of virtual machine monitor based system integrity checking, an approach to enforce system software assurance and increase system security. Our system demonstrated the advantages of implementing runtime integrity checking on VMM over traditional anti-virus software running on the potentially infected host and searching for specific symptoms of known virus. Our prototype has proven to be an effective kernel integrity monitor through experiments of all different types of rootkits and attacks, as the same time the overhead on both the memory bandwidth and CPU is remained low. Index Terms—System Security, Virtual Machine Monitor, Rootkits, Secure I/O

I. Introduction

C

RITICAL personal information stored on the computers, can be attacked, using many well known techniques such as viruses, phishing and so on [1]. Further, attackers can get control over the computers and use them to launch attacks on other computer systems [2]. According to [3], the financial impact of viruses from 1995 to 2005 has grown from $500 million to $14.2 billion worldwide. This paper explores an approach to improve system security using virtual machine monitor based security manager and kernel integrity checking. Software vulnerabilities are common in consumer and business computer platforms. Attacks on computers can be limited by firewall and anti-virus software. However, there is increasing ineffectiveness of traditional malicious code detection technologies to detect and block new attacks. Attackers, motivated by financial gains are moving to deploy rootkits, buffer overflows, pharming, keyboard sniffers, screen scrapers and other advanced attacks that current technology is unable to effectively contain. There are two groups of virus checkers, one runs at the user-level and checks the system at the user level, and the other runs at the kernel-level and checks the kernel. Both suffer from the same shortcoming, that is, malware at the user level can disable user-level checkers and malware at the kernel level can disable kernel level checkers. In addition, the rootkits are undetectable by checkers at either level. The bottom line is that a well designed malware can A. Author is with the Department of Computer Science and Engineering, Arizona State University, Tempe AZ, 85281, USA, email: [email protected] B. Author is with the Department of Computer Science and Engineering, Arizona State University, Tempe AZ, 85281, USA,

hide from detection. The user space checkers can not detect virus running as a rootkit that compromises kernels and may also uninstall the checker. A particular approach that would work better is to introduce the checking software at a level that is inaccessible by the kernel as well as the user. In our approach, we use a “Security Manager” (SM) that is a small isolated software module, running on top of a Virtual Machine Monitor (VMM) and hence is isolated from the guest operating systems. The SM performs runtime checks on the operating systems and runs security protocols. In our approach, the VMM provides the SM with the ability to directly inspect the software as well as hardware states of the guest operating systems; hence the SM can retain the maximum visibility of the guests. Because the VMM provides a high confidence barrier between the SM and the guest Virtual Machines(VMs), the SM is running on a completely different hardware protection domain, the privileged Virtual Machine(VM). The privileged VM and the SM are designed to check each other frequently, any attempt of attacking either of them will be detected by at least one of them, which enforces the trustworthiness of the SM. Further, The VMM is designed to check the privileged VM periodically, so that the privileged VM is further protected. The VMM has the control of all hardware and network access, and the SM handles all security communication to the outside world, the security of the critical transactions on the guest operating systems is enforced. Through the hierarchical runtime checking, from the VMM to the SM, from the VMs to the user applications, the overall system security can be enhanced. In this paper we show how “secure I/O” is essential to the correct operating of a security management system. The SM needs to be able to securely communicate with a human in order to perform a variety of notifications and obtain human input. Since it may be possible for a rootkit to modify the display or even the user responses, the SM cannot rely on the computer screen and the keyboard for human interactions. The “human in the loop” is essential to proper security control and this is achieved by a physically distinct hardware channel and I/O device. The secure I/O provides a trust path between the user and the machine even when the operating system, the keyboard or monitor drivers are compromised. We evaluated the effectiveness and efficiency with our prototype built on Xen with Fedora 4 core. Our experiment results show that our system can catch different types of rootkits. Our results also show that the overhead on memory bandwidth is under 0.4% and the overhead on CPU is under 0.5% for regular cases. For extreme cases, the overhead for memory bandwidth and CPU are 2.5% and 27% respectively.

2

II. Background and Related Work In this section we present background information on Virtual Machine Monitor and Rootkits, then we discuss hardware and software approaches for rootkit detection and system security enhancement. A. Virtual Machine Monitor A Virtual Machine Monitor (VMM) is a layer of software that emulates the hardware of a computer system [4]. The VMM creates an abstraction called a Virtual Machine(VM). The virtual machine is similar enough to the real hardware that operating systems and applications written for the physical hardware can run on the virtual machines. A physical machine can be thus converted to a set of virtual machines and different virtual machines load different copies of operating systems and each copy is totally isolated from the others [5]. VMMs have transitionally been used for installation management, privileged software development and testing, and logical server partitioning. Recently, as VMMs have appeared on the desktop, researchers are developing approaches based on VMM to solve security problems due to the following properties of VMMs. • Security The VMM is a very small program with limited functionality and a narrow, stable, well-defined interface to the software running above it. Unlike traditional operating systems, VMM only needs to present relatively simple abstractions, such as a virtual CPU and memory and it does not execute third-party software [6]. The simplicity of VMM promotes security and the number of vulnerabilities in the VMM is expected to be small to non-existent. • Isolation The VMM provides high isolation for each virtual machine operating under its control. In particular, a programming error or a malware in one operating system will not affect the operation of another operating system running on an independent virtual machine controlled by the same monitor [4]. • Efficiency Experience with virtual machine monitors over the past 30 years has shown the overhead of virtualization of virtualizable platforms can be made essentially negligible [6], i.e. in the 3% to 10% range [7] [8]. B. Xen Xen[8] is a virtual machine monitor for x86 that supports execution of multiple guest operating systems with unprecedented levels of performance and resource isolation [9]. The Xen VMM is an open-source project that is being developed in the computer laboratory of the University Of Cambridge, UK. In a Xen/x86 system, only the VMM (called hypervisor in xen) runs with full processor privileges (ring 0 in the x86 four-ring model). On a 32-bit x86 system, guest operating systems may use rings 1, 2 and 3. Segmentation is used

to prevent operating systems from accessing the portion of the address space that is reserved for Xen. Xen provides secure partitioning between virtual machines (known as domains in Xen terminology). Xen supports one privileged virtual machine, called domain 0, which has access to a control-interface supported by Xen. This control interface manages how the resources exported to the other virtual machines are shared. Domain 0 is a required part of any Xen-based server and runs the application software that manages the controlplane aspects of the platform [10]. We use domain 0 to run the SM. C. Rootkits Rootkits are collections of software that enable attackers who have gained administrative control of a host to hide their presence from the host’s genuine administrators [11]. Once having gained full control, rootkits begin to install several software packages, including backdoors for easy future access, deception packages and modified versions of administration utilities that conceal system modifications and refuse to counterattack any future infiltration [12]. There are two categories of rootkits; the ones that modify kernels and those do not. For the latter category, rootkits just replace some system utilities like ps, ls with modified version that hide the existence of an attacker’s processes, files or network connections. However, since the operating system kernel is not modified, these rootkits can be detected by comparing the output of the modified utilities against the output of unmodified versions [11]. For the rootkits that modify the kernels, there are several weak points of the kernel to attack. First, rootkits can be implemented as Loadable Kernel Module (LKM) s that can be loaded dynamically and can get full access to the kernel. Some rootkits even modify the data structure of kernel to delete themselves from the system module list, hence hide themselves. Second, the /dev/kmem interface can provide direct access to the kernel memory. Rootkits can make use of this interface to write into the kernel as if it were a file. Third, rootkits can redirect system call entries to their malicious functions by overwriting the system call table. And they can also add new entries to or delete entries from the table. Fourth, rootkits can use buffer overflow to overwrite the kernel’s text section directly and redirect function pointers to their code resided in the user space. Fifth, rootkits can also add hooks to /proc to make their code to be executed when a user program reads from certain /proc entries. D. Detecting Rootkits Based on the vulnerabilities addressed above, it is essential for a rootkit detector to check the following areas of a kernel. 1. Kernel’s text section. 2. The interrupt table. 3. The system call table. 4. The KLMs installed on the kernel. 5. The CPU registers. A number of software is designed to detect rootkits. The detector can be installed in three ways.

SHORT NAMES: VMM-BASED SYSTEM SECURITY ASSURANCE

3

First, the detectors run as a user program. According to symptoms of existed rootkits, this type of detectors check the corresponding weak points of the system by comparing the output from the system utilities and the output from the direct system calls. However this kind of detector can easily be defeated. The second type of detectors adds their code into the kernel by writing to the /proc/kmem interface. In this way, the designer must process the system symbols and data structure manually, which complicated the program and hence prone to errors. Also, /proc/kmem is the interface provided by the kernel; hence for rootkits resided in the kernel, it is not hard to hide themselves by providing a modified /proc/kmem. Detectors can be implemented as a KLM, and installed dynamically onto the kernel. The problem is that it does not work for systems that do not support KLM; also rootkits can modify or replace the installed module. Rootkit detection, in order to be foolproof, has to be done outside the kernel and the detector cannot be controllable from the kernel (else a rootkit can disable the detector).

network traffic is not addressed or is done differently from our design. Revirt encapsulates the target system inside a virtual machine, and then places the logging software beneath this virtual machine. Revirt logs and replays the complete instruction by instruction execution of the virtual machine. Revirt aims to log the entire system message for intrusion analysis. Plus, Revirt uses syslog of the targeted system, once the system is compromised, the log message can be manipulated. Liveware uses VMM for advanced intrusion detection on the software in the virtual machines. The Intrusion Detection System (IDS) of liveware consists an operating system interface library that provides an operating system level view of the VM by interpreting the hardware state exported by the VMM, and a policy engine that build policies and implement specific intrusion detection policies. Liveware passively checks the VMs and results in monitoring all system calls, which involves a high performance overhead. Also the police engine and the OS library may be exposed to attacks too.

E. Hardware based rootkit detector - CoPilot

Threat Model: We assume that a conventional, large operating system, coupled with users with varying skills can never be un-attackable. That is there will be viruses introduced into such system via many innovative attack procedures. We also assume that the VMM is the root of trust in the computer system and is considered to be free of viral attacks. Goal: To ensure that any introduction of viruses or rootkits is detectable so that the operating system that is used by a human is free of such malware. Approach: Run a security manager on a privileged virtual machine and use it to check the integrity of the kernels. Have kernel modules check the integrity of virus checkers running at the user level and use these checkers to ensure the integrity of application code. In this paper, we focus only on the kernel integrity checker.

Copilot[11] is a coprocessor-based kernel integrity monitor for commodity systems. Copilot is designed to detect malicious modifications to a host’s kernel by performing kernel integrity checking from a PCI card. The goal of the PCI card is to remain as independent of the potentially subverted operating system as possible. To do this, the PCI card has its own CPU and uses Direct Memory Access (DMA) to scan the physical memory of the computer looking for rootkit behavior. The Copilot PCI board also has its own network interface to communicate in a secure fashion to an administrative component[13]. Because it is a hardware-based solution, it is going to be more costly to purchase and to maintain. Further, due to the limitation of the hardware based solution, the Copilot integrity monitor only checks the integrity of the kernel; it is hard for the Copilot to couple with other software monitoring component. Plus since the PCI card can only access the memories of the system, Copilot can not detect attacks that modify the registers. F. VMM based Approaches Terra is a VMM based platform for trusted computing. It uses a trusted virtual machine monitor (TVMM) that partitions a temper-resistant hardware platform into multiple, isolated VMs. To each VM, the TVMM provides the semantics of either a general purpose hardware platform or an opaque special-purpose platform that protects the privacy and integrity of its contents. The hardware and TVMM can act as a trusted party to allow secure VMs to cryptographically identify the software they run to the remote parties. The Terra system work has not progressed much beyond its conception and several issues such as secure I/O and

III. Our Approach

A. Security Management The core of our system is the Security Manger or SM. This module, along with the VMM and the privileged VM forms the root of trust, that is, they are the trusted components of the system and are not attackable. To ensure high degree of “non-attackable” assurance, the SM and the privileged VM perform routine integrity checks on each other and hence any successful attack will be detected, unless both are compromised at the same time. Plus, the VMM checks the integrity of the kernel of the privileged VM periodically, so the trustworthiness of the trusted platform can be further improved. Our security system consists four parts, and they are the VMM, the privileged VM (or domain 0), the SM and the secure I/O subsystem. • The VMM runs one privileged VM and one or more guest VMs and isolates the privileged VM from the guest VMs. The VMM also dynamically checks the

4

of barriers against virus. AntiVirus S/W

User Apps

Guest VM1

AntiVirus S/W

User Apps

Guest VM2

1. the anti-virus software who checks the user applications; 2. the guest VM checkers who check 1; 3. the SM who checks the kernels of all VMs; 4. the prilileged VM checker who checks 3; 5. the VMM checker who checks 4;

SM

Privileged VM NAT kernel

Virtual Machine Monitor (VMM)

Hardware

B. The Secure I/O Subsystem Secure I/O driver

Secure I/O

Network Legends Runtime Checking

System component

Network Access

System Security Component

Even though the SM is the root of trust, it is possible for viruses or rootkits running on the guest VM to hinder, spoof or fool the SM by tampering with screen output or even grabbing user input to the keyboard. This vulnerability makes “secure I/O” an essential ingredient in any security infrastructure. Without secure I/O many attacks are possible, such as: • •

Fig. 1. System Architecture •

•

•

•

•

privileged VM, ensuring its kernel is never tampered. The driver to the secure I/O is placed inside the VMM, and the privileged VM and the SM have to go through the VMM to access the secure I/O. The privileged VM of the VMM runs the SM, and it has a set of correct hash values for SM and dynamically checks the SM hashes at runtime ensuring the SM is never tampered with by any viral attacks coming over the network. Through the support from the VMM, it is ensured that only the privileged VM can access the hardware devices and the networks. The guest VMs must go through the privileged VM to access the hardware and the networks. The SM is a piece of software that runs on the privileged VM. The SM can access the status of all VMs and performs runtime checking on all VMs. The SM uses a secure I/O for human interactions and runs security protocols that are crucial to ensuring confidentiality of data and validity of operations done on the computer. The secure I/O is used to provide a trusted path between the user and the machine. It is a separate keypad with display. It is wired to the computer through a distinct hardware channel. Only the trusted platform, the VMM, the privileged VM and the SM, can access this I/O device. The guest VMs run operating systems that run user applications just like any conventional system. In most cases, there will be one guest VM running one guest OS. Inside the kernels of the guest VMs, there are checkers which frequently check the integrity of the third-party anti-virus software.

With our architecture, we have built a system with layers

Suppose the SM detects a virus and notifies the user but the virus overwrites the notification on the screen. The user asks the SM to stop a corrupt guest VM, but the virus blocks the keyboard entry (or modifies it) The user updates the guest OS and now has to install the hashes in the SM, and the SM realizes that the hashes are not signed by the appropriate vendor and wants to notify the user, and the rogue OS patch interferes.

In addition installing and modifying root signatures and certificates and other user authentication keys in the SM would be compromisable without a dependable human in the loop interface. Our secure I/O system provides this key feature. IV. The trusted VMM In our approach, the VMM, the privileged VM and the SM are the roots of trust. By ”trusted VMM”, we mean that the VMM when installed will be from a legitimate provider and then the VMM is expected not to get compromised by an attacker. There are four reasons for this: •

•

•

•

The VMM is hard to penetrate, hence its trusted stature. To penetrate the VMM, an attacker has to first penetrate the VMs and then attack the VMM. Since the SM will make penetrating the VMs difficult, penetrating the VMM is even harder. The VMM is a micro-kernel and not a general purpose operating system, and does not execute thirdparty software. Its simplicity promotes security and the number of vulnerabilities in the VMM is expected to be small to non-existent. Technologies like processor privilege protection and separated memory segmentation are used in modern VMM implementations, which can greatly improve the security of the VMM. The trustworthiness of the VMM can be further ensured by hardware support as the same way as the copilot project which checks OS kernel instead of VMM.

SHORT NAMES: VMM-BASED SYSTEM SECURITY ASSURANCE

5

A. Functions

V. The Privileged VM

A.1 Isolation

Among the VMs running on the VMM, the VMM grants the administration duties to one privileged VM. This VM runs in domain 0 and runs the SM. The privileged VM is responsible for managing the physical resources through the control interface provided by the VMM for creating, shutting down, and controlling the execution of the other VMs, and connecting the VMs through virtual device interfaces.

The VMM provides secure partitioning between the VMs and ensuring each VM runs in an isolated domain. The VMM decouples the software from the hardware by forming a level of indirection between the software running in the virtual machine (layer above the VMM) and the hardware [14]. This level of indirection lets the VMM gain the control over how the VMs view and use the hardware resources, including CPU, hard disk, and memory, and also allow strong isolation between the VMs. A.2 System Inspection The VMM provides the ability to inspect the hardware state and event of the VMs. Also the VMM provides the privileged VM the ability to access the system control interface, which is used to manage and monitor the system resources and the execution of the VMs. Hence, the SM running on top of the privileged VM can inspect the hardware as well as software state and event of the VMs through the support from both the VMM and the privilege VM. A.3 Ensuring the Security of the Privileged VM The VMM does runtime checking on the integrity of the privileged VM, and the privileged VM in turn checks the integrity of the SM. Through this hierarchical integrity checking, the three components of the trusted platform can remain free of attacks. A.4 Secure I/O Access The VMM contains the driver to the system secure I/O, and provides APIs for the privileged VM and the SM to access the secure I/O. B. Prototype In our prototype, we use Xen/x86 as the trusted VMM. Xen provides secure partitioning between virtual machines through paravirtualization and provides hypercalls, the virtualized hardware interface, to the OSs running on the VMs. In addition to exporting virtualized instances of CPU, memory, network and block devices, Xen exposes a control interface to manage how these resources are shared between the running domains. Access to the control interface is restricted: it may only be used by one speciallyprivileged VM, known as domain 0. Besides the default functions provided by Xen, we add a module into Xen to check the integrity of the privileged VM. The module is configured with the hash values of the privileged VM at the time of compiling. Then when the system starts running and the privileged VM is boot up, Xen setups up a timer, which triggers the module to check the integrity of the privileged VM. For each timeout, the module hashes key components of the privileged VM and compares the result with the hashes given during the configuration. If they are not the same, a warning will be sent to the secure I/O. Detailed system integrity checking is addressed in section B.1.

A. Functions A.1 Firewall The privileged VM runs a virtual bridge which connects all the other VMs through their virtual network interfaces. And the privileged VM runs a Network Address Translation (NAT) server and the other VMs get assigned addresses via a Dynamic Host Configuration Protocol(DHCP) server. This has the immediate benefit that the privileged VM is a firewall between the network and the guest OSs. By limiting network access of the VMs, the spread of malicious code can be limited and the VMs are more resilient to attacks. This architecture also allows the SM to incorporate IDS technology, like packet filtering and inspection of client-server traffic. And it allows all client side secure transactions to be done through the SM. A.2 Ensuring the Security of the SM The privileged VM’s kernel runs a checker which frequently checks the integrity of the SM. Because the OS of the privileged VM manages all processes running on the privileged VM, the checker inside the OS can direct monitor the SM at runtime. Since the SM also does runtime checking on the privileged VM’s OS kernel, we believe this double checking ensured the security of the privileged VM and the SM. A hacker can break the SM and the hacker can break the privileged kernel, however, it is very hard for the hacker to break both of them as the same time. A.3 Hardware Access Control Through the control interface provided by the VMM, the privileged VM can create and terminate other VMs and control their associated physical memory, allocations, their access to the physical disks and network devices. More important, the privileged VM can decide which VM can access which I/O devices and with what restrictions. Hence, the privileged VM can enforce that no other VMs can access the secure I/O. B. Prototype In our prototype, the privilege VM is the Domain0 in Xen, and the kernel of Domain0 is Fedora 4. To setup a LAN for the VMs on the computer system, we add a virtual bridge to the Domain0 through brctl utility. Then we setup a NAT server and package filter through iptables and also run a DHCP server for IP assignment. And we configure Xen through the configuration file at /etc/xen/xend-

6

config.sxp and configure the other VMs through their startup configuration files, so that all the VMs are connected through the virtual bridge on Domain0 and they connect the outside network through the NAT server on Domain0. We updated the Fedora 4 kernel to insert an SM integrity checker. During the kernel booting up, the kernel creates a kernel thread, which is used to call the SM integrity checker periodically using kernel timers. The SM integrity checker gets the process list, finds the SM process, and hashes the text section of the process, then compares it with the hash given. If they are different, means someone changed the SM, and the kernel will send warning to the administrator through the secure I/O. The isolation between the secure I/O and the guest VMs ensure that the secure I/O will not be compromised by any attacks on the guest VMs. VI. Security Manager(SM)

0xc00000000 _stext

kernel text

vcpu_context { register_ctxt trap_ctxt }

CR3

_etext

kernel data

1

0xc…….

2

0xc…….

system call table

17 0xc……. 128 0xc…….

Address to the entry of the system call table

high memory

vmalloc area

The SM is a software module that is running in the privileged VM.

module 1

module n

A. Functions The functions of the SM are to perform checking on all VMs, to provide secure human machine communications and to run security protocols. A.1 VM Validation The security manager provides runtime checking on all VMs that are running. Through the support from the VMM, the VMs’ system memory as well as CPU registers and interrupts can be made visible to the SM. This has a powerful effect in ensuring the validity or genuineness of the OS. When an OS is booted for the first time, the SM computes hashes of the critical components of the OS, including OS text section, interrupt vectors, system call interfaces, kernel loadable modules and such. These hashes are stored and later verified at dynamic intervals ensuring the VMs are not tampered with. The OSs can then be augmented to do the same for application programs, and hence not only rootkits can be detected, but application viruses have a good chance of being detected. The detection scheme can also use certificates from vendors of the kernels and applications, and the fingerprints can be stored in the SM. The upgrading of the kernels or patching of a VM, would need user input to the SM, else the SM would not validate the new version of the kernels. A.2 Human Interactions In our threat model, we believe that once a virus compromised a computer, the virus can gain full control of the computer, i.e. the computer will do anything as the virus wanted, because any trusted software can be fooled by the virus somehow. To limit the actions of the virus, human in loop control is very important, since only a human-being can tell if an action of the computer is wanted or not. In our system, the SM provides secure human-machine interactions functions, and also enforces the human in loop during critical system transactions, e.g. system upgrade. Also,

Areas to be checked by the SM 0xfe0000000

Fig. 2. SM kernel checking

the user applications running on the guest VMs should go through the SM for secure human-machine interactions. A.3 Key Management All keys, certificates and such are kept in the SM. When these keys have to be used in authentication and signatures, the SM runs the code for operation and does not ever reveal the keys. In addition, some critical operations have to be done with a human validating the transaction. A.4 Secure Communications The SM handles all security credentials, security protocol endpoints, and digital signature. The SM performs secure communications and runs security protocols that are critical to ensuring confidentiality of data and validity of operations done on the computer. The SM can be used to perform any secure e-commerce applications that the user may indulge in. B. Detailed design and Prototype In our prototype, the SM is composed of the following modules, a kernel integrity checker, a secure I/O interface, a module for hashes functions, and key management functions. B.1 System Integrity Checking The kernel integrity checker is used to check the integrity of the kernels of the VMs. It first gets the hashes of the

SHORT NAMES: VMM-BASED SYSTEM SECURITY ASSURANCE

VMs, and reads the original hashes from the encrypt files, then it compares the two sets of hashes, for changed hashes and newly added modules, it sends warnings to the owner via the secure I/O and reboots the corresponding VM, for newly added VMs it just sends warnings and gets confirmation from the owner and then stores the hashes of the new VM. Four parts of the kernel are checked in our prototype. They are the kernel text, the interrupt table, the system call table, and the kernel modules. • Kernel text checking : The checker first reads kernel information, including the virtual address for stext and etext of the kernel from a configure file. Then, it gets the kernel memory page by page through the support of the VMM. After that, it computes the hash of the kernel and stores it. • IDT checking : Through the hypercalls provided by the VMM, the kernel checker can get the value of register CR3, which stores the entry of global page directory and interrupts of the virtual CPUs of the VMs. Then, the checker hashes these values and stores them. • System call table checking : The kernel checker gets the entry to the system call table through the interrupt table on its 0x80 entry. Through the entry, the checker can copy the whole system call table to the user space through the control interface provided by the VMM and hash it. • Kernel module checking : Since more and more rootkits hide themselves by modifying the module list maintained in the kernel, we adapt the approach present in [15], we scan the vmalloc area for modules rather than walking through the module list when checking the kernel modules. (For Linux kernels, all kernel modules are allocated in the vmalloc area). The checker gets the virtual address to high memory and vmalloc earlyreserve from the configuration file, through which it can get start and end points of the system vmalloc area. Then the checker scans the vmalloc area for valid ”struct module” with 0x20 bytes per step. During each step, the checker first checks if the start point is in the RAM by going through layers of the system page table and checking if the address is in the page table. If it is valid, then it casts the memory into ”struct module”, and checks if the value of the struct is reasonable. If it is a reasonable struct, then we have identified a kernel module. Once the checker finds one, it hashes the text section of the module which is indicated in the module struct and stores it. Till it reaches the end of the vmalloc area, the checker finds all kernel modules no matter the module is in the module list of the kernel or not. B.2 Hash Functions The SM provides functions related to hashes, including hash computing, reading, writing hashes from files, and comparing original and current hashes. Currently, we use md5 for hashes in our prototype; more secure hashing technology may be used for real world systems.

7

B.3 Key Management The SM generates and maintains a set of symmetric and asymmetric keys, which can be used to ensure the integrity of the stored hashes. • Key Generation When the system is installed, the SM generates a temporary symmetric key(Ktemp ) using current system time as the seed and uses it for encrypting the system hashes. After the system is booted up, the SM asks the administrator for a pass phrase through the secure I/O. Then the SM uses the pass phrase as seed to generate a symmetric key(Ks ) and a pair of public/private keys(Kpub and Kpriv ). And Kpub and Kpriv can be used for hashes encrypting/decrypting and user authentication. The Ks is used to encrypt and decrypt Kpub and Kpriv for secure storage. Or you can get a certificate (Cerf) from a Certificate Authority (CA) for the public key and the administrator, and then use the certificate to verify the validity of the pass phrase. • Key Storage When the system is on and the SM is running, the keys are stored in the SM memory, and will never be revealed. And the memory for the SM is checked by the kernel periodically. The hashes of the encrypted keys are stored in a system file or the certificate from a trusted CA is stored, and either of them can be used for user authentication. Both of them can be checked by the kernel too, any changes will be reported to the administrator. When the system reboots, the SM can get the keys by asking the administrator to input the pass phrase and then generating the keys again. To check the integrity of the keys, the SM first encrypts Kpub and Kpriv with Ks , then hashes it, and comparing the hashes with the stored system file. Or simply checks the keys with the stored certificate. • Key Revoke To revoke the keys, the administrator first logins as usual and then deletes the stored key hash file or the certificate. When the system is rebooted, the SM will ask for new pass-phase, hence generate new keys. B.4 Secure I/O APIs The SM also provides functions to read/write through the secure I/O, the serial console in our prototype. For the critical messages, the SM not only reports through the standard I/O and log files, but writes to the secure console. To ensure human in loop, the SM reads from the secure console to get input from a human. VII. Secure I/O The secure I/O is used to provide a trusted path between the user and the machine. It is a separated keypad with display. It could be as simple as a calculator with a network interface to be wired to a computer. It should be wired to the computer through a distinct hardware channel. Only the security manager can access this I/O device for humanmachine interactions. Depending the type of device used as

8

the secure I/O, Secure Shell (SSH) may be used to further protect the security of critical data going between the SM and the secure I/O. In our system, the privileged VM ensures that the secure I/O is invisible to the guest VMs, applications running on the guest VMs have no clue of the secure I/O, not mention to access it. Plus, the secure channel is between the human and the SM through a separated hardware, and the SM is the root of trust of the system, hence attackers can not break into the secure I/O without breaking into the SM, which is very hard. Only human can actually hit the keypad to send input, and a virus can not fake it. Why do we need the secure I/O? We want to ensure the human validations are really true and not faked by some rootkit. Suppose an attacker patches the VM’s kernel with a viral rootkit and informs the SM that the OS is updated, how does the SM know that this upgrade was not authorized by a human? Suppose a virus on the host purchases an item from a rogue website using the credentials of a person, and wants the SM to sign the transaction verification with the person’s private key. How does the SM know it should not do that? These are points where a human has to validate some actions that the SM can perform. In summary, the secure I/O provides trusted the input and output that can not be tampered with or recorded. Also the secure I/O can determine whether a user is physically present by checking input from the secure hardware. The secure I/O is essential to our design and it has much more applications. For example, we can use it to enforce human validation for e-commerce transactions. For each e-commerce transactions, the SM sends the relevant important data, like the amount, price, the merchant name, to the user through the secure display, the user confirms the transaction by hitting the keypad. Any virus trying to modify the content of the transaction will be caught by contrasting the transaction content appears on the host output and secure output. In our prototype, we use a serial console as the secure I/O, which is a terminal with another set of keyboard and monitor connected to the host computer through an independent serial cable. The serial port can be only accessed by the VMM and the privileged VM, other VMs are not aware the existence of the port nor can access it in any way. VIII. The Guest VMs In our system, VMs could run third party user space anti virus software, like Norton Anti Virus[16], and/or file integration checking software, like Tripwire[17]. The VMs can be configured to check the integrity of the software at runtime if the name and correct hashes of the software are given. Hence, the security of the system can be further ensured at the user application layer. In our prototype, the checker which checks the integrity of the third party software are implemented as the same as the checker for the SM stated in section V. In our system, all user applications run on the guest VMs, so that the impact of the attacks on the guest VMs

can be minimized to the privileged VM and the SM. Plus, we can further divide user applications into different categories according to their security requirement, and applications of each category run in one VM. For instance, the applications with manufacture certificates and applications dealing with critical personal information can be installed on one VM, and applications, like games, Internet surfing can be installed on another VM. Though we could not further improve the security of the VM with higher security requirement, we can ensure that the virus compromised the game VM will not compromise the critical VM, because the VMM isolated the two VMs and the SM can detect the compromising of the game VM and stop the execution of the compromised VM right way. IX. Experimental Results In this section, we present the experimental evaluation of our prototype. First, we test the effectiveness of our system against different type of rootkits and some evaluation tools. Second, we test the overhead of our prototype using different benchmarks. Our prototype runs on a 2GHz Intel-Pentium 4 PC with 512MB RAM. The privileged VM runs on a 256MB allocation of physical memory and the guest VMs run on 64MB allocation of physical memory. In our prototype, the modified Xen 3.0 for x86 is used as the VMM, the modified Fedora 4.0 is used as the privileged VM and all the guest VMs. A. Attacks We installed various rootkits to test the effectiveness of our system. All the rootkits were obtained from public sources and were modified only as necessary for configuration, or for adaptation to our kernel. Our selection of rootkits covered all types of attacks as we discussed in Rootkits subsection. Modify the System Call Table : Rootkit rkit 1.01 modifies the setuid entry in the system call table to set current user as the root. The rootkit is installed through loadable kernel module. Our system can detect it through two ways, one is checking the integrity of the system call table and the other is checking the integrity of the kernel modules depending which checking happens first. Modify the Kernel Module: Rootkit Adore-ng 0.53 deletes itself from the module list once it is installed, and it can perform process hiding and file hiding upon user’s request by modifying the process hashing table and redirecting file operating functions to its updated functions. Adore-ng is also installed through loadable kernel module. Our system can detect it through module integrity checking. However, the protection of the data of the kernel, such as processes and file list, is still an open problem. Modify the Kernel Text: Since we didn’t find any rootkit that updates kernel text directly for our Linux 2.6 system version, we developed a small tool to test our prototype, this tool can flip the bits in the given kernel virtual memory address of the given VM. Our system can detect it through

SHORT NAMES: VMM-BASED SYSTEM SECURITY ASSURANCE

either the kernel text integrity checking or the kernel module integrity checking depending on which happens first. Modify the IDT Table : We also developed a small tool which can update the given entry to the virtual IDT table of the given VM. Our system can detect it through the IDT table integrity checking. B. Performance To measure the overhead of our system, we first test its impact on system memory bandwidth with STREAM and STREAM2 benchmark, and then we test its impact over CPU with CPU intensive tasks, last we test the impact of number of VMs running on the VMM. B.1 Memory Bandwidth To test the impact of our system on host memory bandwidth, we used STREAM[18] and STREAM2[19] benchmarks which have proven to be an effective measure of PC memory bandwidth. In our experiment, we run STREAM and STREAM2 for 1000 times for both the privileged VM and the guest VM under three system settings: • The system checker is turned off; • The system checker is turned on and the checking interval is 60 seconds; • The system checker is turned on and the checking interval is 1 second. The results (table B.1) show that the privileged VM carries more penalties then the guest VM. That is because the SM checking runs on the privileged VM and uses mmap to map the VMs’ kernel memory to the user space memory of the privileged VM, which involves a lot of memory operations. Also the results show that under the extreme case, the system checking interval is set to 1 second, the system shows a 3% and 9% penalty on the guest and the privileged VM respectively, and when the system checks with 60 seconds intervals, the system shows 0.5% and 2.5% penalty on the guest and privileged VM respectively. We consider this is a reasonable penalty for both the guest VM and the privileged VM. B.2 CPU To test the impact of our system on system CPU usage, we tested the running time of a CPU intensive task, unzip and untar the Linux 2.6.12 kernel package. For each system setting, we run the task for 100 times, and each time we use system program time to count the running time of ”tar zxf”. Then we calculate the average running time for each system setting, and the penalty comparing to the system setting with no checkers running. From our experiment, the results (table B.2) show that the penalty for the CPU usage is under 0.5% and 2% for the guest and privileged VM respectively when the checking interval is 60 seconds. Under the extreme case, the checking interval is 1 second; the CPU penalty is 27% and 20% for the guest and privileged VM respectively. Also the penalty on the privileged VM is more than the penalty on the guest VM, that is because both VMs runs minimum

9

COPY

SCALE

ADD

TRIAD

FILL

COPY

DAXPY

SUM

off 60s 1s off 60s 1s off 60s 1s off 60s 1s off 60s 1s off 60s 1s off 60s 1s off 60s 1s

Privileged VM Average Penalty (MB/s) STREAM 1040.12 0.00% 1012.65 2.64% 950.02 8.66% 1045.93 0.00% 1014.58 3.00% 952.77 8.91% 3591.33 0.00% 3590.97 0.01% 3456.32 3.76% 783.96 0.00% 772.7 1.44% 730.67 6.80% STREAM2 391.98 0.00% 386.35 1.44% 365.34 6.80% 1043.02 0.00% 1013.36 2.84% 949.39 8.98% 1545.73 0.00% 1492.17 3.47% 1401.44 9.33% 1796.06 0.00% 1795.52 0.03% 1728.49 6.25%

Guest VM Average Penalty (MB/s) 919.72 917.97 900.27 921.1 920.77 901.57 3529.73 3534.67 3533.4 707.02 710.21 696.16

0.00% 0.19% 2.11% 0.00% 0.04% 2.12% 0.00% -0.14% -0.10% 0.00% -0.45% 1.54%

353.51 355.11 348.08 913.8 918.12 898.92 1356.73 1354.39 1323.45 1764.52 1767.46 1766.51

0.00% -0.45% 1.56% 0.00% -0.47% 1.63% 0.00% 0.17% 2.51% 0.00% -0.17% -0.11%

TABLE I STREAM benchmark with Different Checking Intervals

set of processes in this test, and the privileged VM runs a SM and the guest SM does not. And there is no significant difference between the running time on the privileged VM and the guest VM. B.3 Running Time To measure the impact of the number of VMs running on the VMM, we tested the running time of the SM checker with different number of VMs running. We run the check for 1000 times and count the running time of the checking in number of system ticks, then calculate the average and standard deviance of the running time and convert them into milliseconds. The results show the more VMs running, the more time needed for the checker to check. That is reasonable since the more number of VMs, the more system memory needed to be checked at each round, and more time is needed. X. Conclusion We proposed the idea of virtual machine monitor based system integrity checking, an approach to enforce system software assurance and increase system security. Our system is built with hierarchical checking from the VMM to

10

off 60 30 20 10 1

Privileged VM Average(s) Penalty 44.38071429 0.000% 45.23090909 1.916% 45.417 2.335% 45.11583333 1.656% 45.92493976 3.479% 52.90888889 19.216%

Guest VM Average(s) Penalty 44.87735294 0.00% 45.039375 0.36% 45.19 0.70% 45.587 1.58% 46.90090909 4.51% 56.81125 26.59%

TABLE II CPU Usage with Different Checking Intervals

Num of VMs 1 2 3

Average(ms) 15.728 24.180 32.956

Standard deviance 1.44 1.44 1.98

TABLE III Running Time with Different Number of VMs

the user applications with the security manager as the root. Also, we addressed the importance of the secure I/O and it’s use in secure systems. Our system demonstrates the advantages of implementing runtime integrity monitor on VMM over traditional anti-virus software running on the potentially infected host and searching for specific symptoms of known virus. Out prototype has proven to be an effective kernel integrity monitor by testing all different types of attacks, including modifying kernel executable codes, modifying system interrupt table, modifying system call table and adding or hiding malicious loadable kernel modules. Also the overhead on both the memory bandwidth and CPU is under 0.5% for 60 seconds checking intervals. References [1] Ed Shanahan, “Id thieves’s new tricks,” Reader Digest, May 2006. [2] “Why is the computer security important?,” Armor2net, http://www.armor2net.com/knowledge/computer security.htm. [3] www.computereconomics.com, “Malware report: The impact of malicious code attacks,” Computer Economics, Jan. 2006. [4] Robert P. Goldberg, “Survey of virtual machine research,” IEEE Computer, pp. 34–45, June 1974. [5] Robert P. Goldberg, Architechtural Principles for Virtual Computer Systems, Ph.D. thesis, Harvard University, 1972. [6] Tal Garfinkel, Ben Pfaff, Jim Chow, Mendel RosenBlum, and Dan Boneh, “Terra: A virtual machine-based platform for trusted computing,” in Proc. the nineteenth ACM symposium on Operating systems principles, New York, Oct. 2003, pp. 193– 206. [7] Samuel T. King, George W. Dunlap, and Peter M. Chen, “Operating system support for virtual machines,” in Proceedings of the 2003 USENIX Technical Conference, 2003. [8] Paul Barham, Boris Dragovic, Keir Fraser, Steven hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield, “Xen and the art of virtualization,” in Proc. the 19th ACM symposium on Operating systems principles, Oct. 2003. [9] “The xen virtual machine monitor,” University of Cambridge Computer Laboratory, http://www.cl.cam.ac.uk/Research/SRG/netos/xen/. [10] University of Cambridge, UK, Xen Interface manual, Xen v3.0 for x86, 2005.

[11] Jr. Nick L. Petroni, Timothy Fraser, Jesus Molina, and William A. Arbaugh, “Copilot - a coprocessor-based kernel runtime integrity monitor,” in Proceedings of the 13th USENIX Security Symposium, Aug. 2004. [12] Winfried E. Kuhnhauser, “Root kits, an operating system viewpoint,” in ACM SIGOPS, Jan. 2004, pp. 12–23. [13] James Butler and Sherri Sparks, “Windows rootkits of 2005,” Security Focus, 2005. [14] Mendel Rosenblum and Tal Garfinkel, “Virtual machine monitors: Current technology and future trends,” IEEE computer Society, pp. 39–47, May 2005. [15] “Finding hidden kernel modules (the extrem way),” http://www.phrack.org/show.php?p=61&a=3. [16] “Norton antivirus 2006,” http://www.nortonus.com/. [17] “Tripwire for servers,” http://www.tripwire.com/products/servers /index.cfm. [18] Jofn D. McCalpin, “Stream: Sustainable memory bandwidth in high performance computers,” http://www.cs.virginia.edu/stream/. [19] “Stream2,” http://www.cs.virginia.edu/stream/stream2/. [20] “Trusted computing group,” https://www.trustedcomputinggroup. org/home.