Buffer Overflow / Memory Corruption
Buffer overflow vulnerabilities arise from bugs in memory unsafe code (i.e. C, C++). These vulnerabilities can result in arbitrary code execution. Memory corruption and software security is a full field unto itself which requires more background than can be summarized in this wiki.
Several other courses cover this topic in more detail.
CS1670 cover details of operating systems while
CS1650 focuses directly on software security.