International Business Machines Corporation
Executing system calls in isolated address space in operating system kernel

Abstract:

Embodiments of the present systems and methods may provide additional security mechanisms inside an operating system kernel itself by executing system calls in a dedicated address space to reduce the amount of shared resources that are visible to and thus exploitable by a malicious application. For example, in an embodiment, a method implemented in a computer may comprise a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor, the method may comprise: when a user process makes a system call, switching to kernel mode and using a system call page table for the user process to execute a system call handler, when the system call handler attempts to access unmapped kernel space memory, generating a page fault, and handling the page fault by determining whether the attempted access to unmapped kernel space memory is allowed.