Extending Graphite Adding new system calls and configuration parameters

Session Overview • Adding new configuration parameters • Supporting new system calls – Handling system calls locally – Handling system call remotely

Configuring Graphite Configuration File

class config::Config

class Config

• Defaults to carbon_sim.cfg

• Declared in common/config/config.hpp • Parses parameters into a configuration tree

• Declared in common/misc/config.h • Caches parameter values • Used when parameter value needed at runtime

Configuration File • Arranged into sections and subsections – [perf_model] – [perf_model/l1_dcache/T1]

• Each parameter belongs to a section – [perf_model/l1_dcache/T1] cache_size = 32

• Add new sections and parameters here as needed

Adding New Parameters • Parser parses the parameters into a configuration tree • Parameter values can be searched by name • For each new parameter – Create a variable in the configuration cache to store its value – Create a function to retrieve this value – Fill in the value at initialization

Adding New Parameters /common/misc/config.h

class Config { private: int m_l1_dcache_size; public: int getL1DCacheSize() {return m_l1_dcache_size;} }

• For each new parameter – Create a variable in the configuration cache to store its value – Create a function to retrieve this value – Fill in the value at initialization

Adding New Parameters /common/misc/config.h

class Config { private: int m_l1_dcache_size; public: int getL1DCacheSize() {return m_l1_dcache_size;} }

• For each new parameter – Create a variable in the configuration cache to store its value – Create a function to retrieve this value – Fill in the value at initialization

Adding New Parameters /common/misc/config.cc

Config::Config() { m_l1_dcache_size = Sim()->getCfg()->getInt(“perf_model/l1_dcache/T1/cache_size”); }

• For each new parameter – Create a variable in the configuration cache to store its value – Create a function to retrieve this value – Fill in the value at initialization

Session Overview • Adding new configuration parameters • Supporting new system calls – Handling system calls locally – Handling system call remotely

Supporting new system calls • Application and OS have different views of the world • Simulated application: single process – Single address space – Single file descriptor table – Single pid, resource limits etc

• OS: multiple processes – Each host process has its own address space, file descriptor table etc

System Calls File Management

Memory Management

Synchronization/ Communication

open, access, read, write

mmap, munmap, brk

sigprocmask, sigsuspend, sigaction

getrlimit, nanosleep, gettid

Signal Management

Other syscalls

kill, waitpid, futex 11

Syscall Handling Machinery pin/handle_syscalls.h

• Syscall entry and exit callback functions

• syscallEnterRunModel() • thread id • Execution context • syscall standard

•syscallExitRunModel() • thread id • Execution context • syscall standard • Context modification for locally handled syscalls

Common/tile/ syscall_model.h

• Per tile syscall handling on client side • class SyscallMdl • runEnter() • runExit() • Syscall data • syscall number • scratchpads for arguments • Marshalling functions for remote syscalls

common/system/ syscall_server.h

• Part of the MCP • Handles all remote system calls

Session Overview • Adding new configuration parameters • Supporting new system calls – Handling system calls locally – Handling system call remotely

System Calls File Management

Memory Management

Synchronization/ Communication

open, access, read, write

mmap, munmap, brk

kill, waitpid, futex

sigprocmask, sigsuspend, sigaction

Signal Management

getrlimit, nanosleep, gettid

Other syscalls

Handled locally 14

Syscalls Handled Locally Core

Arguments are copied into a local buffer (if needed) On Syscall Entry

… … mov eax, 1 int $0x80 … …

Syscall Executed

On Syscall Exit

Arguments copied back into simulated memory (if needed)

Mechanism - Syscalls Handled Locally 15

Handling syscalls locally • Example: nanosleep – Two of its arguments are pointers to structures in memory – struct timespec *req (IN) • Specifies the time to sleep

– Struct timespec *rem (OUT) • Specifies remaining time if the call returns early

• Add callback functions at syscall entry and exit

Syscall Handling Machinery

pin/handle_syscalls.h

• Pin front end

common/tile/ syscall_model.h

• Client back end (per tile)

common/system/ syscall_server.h

• Server back end (MCP)

Handling syscall nanosleep pin/handle_syscall.cc void modifyNanosleepContext(CONTEXT *ctx, SYSCALL_STANDARD std); void restoreNanosleepContext(CONTEXT *ctx, SYSCALL_STANDARD std); void syscallEnterRunModel(CONTEXT *ctx, THREADID tid, SYSCALL_STANDARD std, VOID *v) { if(syscall_number == SYS_nanosleep) modifyNanosleepContext(ctx, std); }

// Code to handle other syscalls

void syscallExitRunModel(CONTEXT *ctx, THREADID tid, SYSCALL_STANDARD std, VOID *v) { if(syscall_number == SYS_nanosleep) restoreNanosleepContext(ctx, std); }

// Code to handle other syscalls

• Modify context at syscall entry

Handling syscall nanosleep pin/handle_syscall.cc void modifyNanosleepContext(CONTEXT *ctx, SYSCALL_STANDARD std); void restoreNanosleepContext(CONTEXT *ctx, SYSCALL_STANDARD std); void syscallEnterRunModel(CONTEXT *ctx, THREADID tid, SYSCALL_STANDARD std, VOID *v) { if(syscall_number == SYS_nanosleep) modifyNanosleepContext(ctx, std); }

// Code to handle other syscalls

void syscallExitRunModel(CONTEXT *ctx, THREADID tid, SYSCALL_STANDARD std, VOID *v) { if(syscall_number == SYS_nanosleep) restoreNanosleepContext(ctx, std); }

// Code to handle other syscalls

• Copy output and restore context at syscall exit

Modifying syscall context pin/handle_syscall.cc void modifyNanosleepContext(CONTEXT *ctx, SYSCALL_STANDARD std) { Tile *tile = Sim()->getTileManager()->getCurrentTile(); SyscallMdl::syscall_args_t args = syscallArgs(ctx, std); struct timesepc *req = (struct timespec*) args.arg0; struct timesepc *rem = (struct timespec*) args.arg1; if(req){ struct timespec *req_arg = (struct timespec*) tile->getSyscallMdl()-> copyArgtoBuffer(0, (IntPtr)req, sizeof(struct timespec)); PIN_SetSyscallArgument(ctx, std, 0, (ADDRINT) req_arg); } if(rem){ struct timespec *rem_arg = (struct timespec*) tile->getSyscallMdl()->copyArgtoBuffer(1, (IntPtr)rem, sizeof(struct timespec)); PIN_SetSyscallArgument(ctx, std, 1, (ADDRINT) rem_arg); } }

• Modify memory arguments to point to buffers in host address space – Copy input arguments from simulated address space to buffers

Restoring syscall context pin/handle_syscall.cc void restoreNanosleepContext(CONTEXT *ctx, SYSCALL_STANDARD std) { Tile *tile = Sim()->getTileManager()->getCurrentTile(); SyscallMdl::syscall_args_t args; tile->getSyscallMdl()->retrieveSyscallArgs(args); struct timespec *req = (struct timespec*) args.arg0; struct timespec *rem = (struct timespec*) args.arg1; if(rem) { tile->getSyscallMdl()->copyArgFromBuffer(1, (IntPtr) rem, sizeof(struct timespec)); }

}

PIN_SetSyscallArgument(ctx, std, 0, (ADDRINT)req); PIN_SetSyscallArgument(ctx, std, 1, (ADDRINT)rem);

• Restore original context

– Copy output data to the simulated address space

Session Overview • Adding new configuration parameters • Supporting new system calls – Handling system calls locally – Handling system call remotely

System Calls File Management

open, access, read, write

sigprocmask, sigsuspend, sigaction

Signal Management

Memory Management

mmap, munmap, brk

gettimeofday, uname, getrlimit, nanosleep

Other syscalls

Synchronization/ Communication

kill, waitpid, futex

Handled at the MCP 23

Syscalls Handled Centrally Core

On Syscall Entry

… … mov eax, 1 int $0x80 … …

1) Arguments are copied from simulated memory into a local buffer 2) Syscall is changed to “NOP” (getpid)

Sent to MCP

MCP Syscall Executed On Syscall Exit

1) Syscall return value received from MCP 2) Arguments copied back to simulated memory

Mechanism - Syscalls Handled Centrally at the MCP 24

Why handle syscalls remotely? • open, read, write, close – Need to be handled centrally to maintain correct file modification semantics across processes

• brk, mmap, munmap – Maintain a single simulated address space

• futex, kill – Correct communication semantics across processes

• Centralized handling is done via the MCP

Syscall Handling Machinery

pin/handle_syscalls.h

• Pin front end

common/tile/ syscall_model.h

• Client back end (per tile)

common/system/ syscall_server.h

• Server back end (MCP)

Handling syscall read (front end) pin/handle_syscall.cc void syscallEnterRunModel(THREADID tid, CONTEXT *ctx, SYSCALL_STANDARD std, VOID *v) { Tile *tile = Sim()->getTileManager()->getCurrentTile(); Intptr syscall_number = PIN_GetSyscallNumber(ctx, std); if(syscall_number == SYS_read) { SyscallMdl::syscall_args_t args = syscallArgs(ctx, std); IntPtr new_syscall = tile->getSyscallMdl()->runEnter(syscall_number, args); PIN_SetSyscallNumber(ctx, std, new_syscall); } }

// Code to handle other syscalls

• Syscall entry callback in pin/handle_syscall.c simply calls into the syscall entry function in class SyscallMdl • Replace syscall read with NOP (syscall getpid)

Syscall Handling Machinery

pin/handle_syscalls.h

• Pin front end

common/tile/ syscall_model.h

• Client back end (per tile)

common/system/ syscall_server.h

• Server back end (MCP)

Handling syscall read (client) common/tile/syscall_model.h

common/tile/syscall_model.cc

class SyscallMdl { IntPtr m_syscall_number;

IntPtr SyscallMdl::runEnter(IntPtr syscall_num, syscall_args_t &args) { m_syscall_number = syscall_num; m_send_buff.clear(); m_recv_buff.clear();

UnstructuredBuffer m_send_buff; UnstructuredBuffer m_recv_buff;

if(syscall_num == SYS_read) { return marshallReadCall(args); }

IntPtr marshallReadCall(syscall_args_t args); }

// Other class members

}

// Code for other syscalls

• SyscallMdl defines a handler function for syscall read

Handling syscall read (client) common/tile/syscall_model.cc IntPtr SyscallMdl::marshallReadCall(syscall_args_t &args) { int fd = (int) args.arg0; void *buf = (void*) args.arg1; size_t count = (size_t) args.arg2; m_send_buff getMCPTileNum(), MCP_REQUEST_TYPE, m_send_buff.getBuffer(), m_send_buff.size()); NetPacket recv_pkt = m_network->netRecv(Config::getSingleton()->getMCPTileNum(), MCP_RESPONSE_TYPE); int num_bytes; m_recv_buff >> num_bytes; char *read_buff = new char[bytes]; m_recv_buff >> make_pair(read_buf, num_bytes); Core*core = Sim()->getTileManger()->getCurrentCore(); core->accessMemory(Core::NONE, Core::WRITE, (IntPtr) buf, read_buff, num_bytes); }

return num_bytes;

• Pack syscall parameters in a network packet

Handling syscall read (client) common/tile/syscall_model.cc IntPtr SyscallMdl::marshallReadCall(syscall_args_t &args) { int fd = (int) args.arg0; void *buf = (void*) args.arg1; size_t count = (size_t) args.arg2; m_send_buff getMCPTileNum(), MCP_REQUEST_TYPE, m_send_buff.getBuffer(), m_send_buff.size()); NetPacket recv_pkt = m_network->netRecv(Config::getSingleton()->getMCPTileNum(), MCP_RESPONSE_TYPE); int num_bytes; m_recv_buff >> num_bytes; char *read_buff = new char[bytes]; m_recv_buff >> make_pair(read_buf, num_bytes); Core*core = Sim()->getTileManger()->getCurrentCore(); core->accessMemory(Core::NONE, Core::WRITE, (IntPtr) buf, read_buff, num_bytes); }

return num_bytes;

• Send a request to MCP and wait for the response

Syscall Handling Machinery

pin/handle_syscalls.h

• Pin front end

common/tile/ syscall_model.h

• Client back end (per tile)

common/system/ syscall_server.h

• Server back end (MCP)

Handling syscall read (server) common/system/syscall_server.h

common/tile/syscall_model.cc

class SyscallServer { SyscallServer(); ~SyscallServer();

void SyscallServer::handleSyscall(core_id_t core_id) { IntPtr syscall_number; m_recv_buff >> syscall_number; if(syscall_num == SYS_read) { return marshallReadCall(args); }

void handleSyscall(core_id_t core_id); IntPtr marshallReadCall(syscall_args_t args); }

// Other class members }

// Code for other syscalls

• SyscallServer defines a handler function for syscall read

Handling syscall read (server) common/system/syscall_server.cc void SyscallServer::marshallReadCall(core_id_t core_id) { int fd; size_t count; char *read_buf = (char*) m_scratch; m_recv_buff >> fd >> count; int bytes = syscall(SYS_read, fd, (void*) read_buf, count); m_send_buff fd >> count; int bytes = syscall(SYS_read, fd, (void*) read_buf, count); m_send_buff getMCPTileNum(), MCP_RESPONSE_TYPE); int num_bytes; m_recv_buff >> num_bytes; char *read_buff = new char[bytes]; m_recv_buff >> make_pair(read_buf, num_bytes); Core *core = Sim()->getTileManger()->getCurrentCore(); core->accessMemory(Core::NONE, Core::WRITE, (IntPtr) buf, read_buff, num_bytes); }

return num_bytes;

• Receive results, update simulated memory if needed

Syscall Handling Machinery

pin/handle_syscalls.h

• Pin front end

common/tile/ syscall_model.h

• Client back end (per tile)

common/system/ syscall_server.h

• Server back end (MCP)

Session Overview • Adding new configuration parameters • Supporting new system calls – Handling system calls locally – Handling system call remotely