Lecture 11: Stack, Buffer Overflow
🎥 Lecture video (Brown ID required)
💻 Lecture code
❓ Post-Lecture Quiz (due 11:59pm, Wednesday, March 8).
You will recall the stack segment of memory from earlier lectures: it is where all variables with automatic lifetime are stored. These include local variables declared inside functions, but importantly also function arguments.
Recall that in
call03.s contained a bunch of instructions referring to
%rsp, such as this implementation of the function
movl %edi, -4(%rsp) movl -4(%rsp), %eax ret
movlstores the first argument (a 4-byte integer, passed in
%edi) at an address four bytes below the address stored in register
%rsp; the second
movlinstruction takes that value in memory and loads it into register
%rsp register is called the stack pointer. It always points to the "top"
of the stack, which is at the lowest (leftmost) address current used in the stack segment. At the start of
the function, any memory to the left of where
%rsp points is therefore unused; any memory to the right
of where it points is used. This explains why the code stores the argument at addresss
%rsp - 4: it's
the first 4-byte slot available on the stack, to the left of the currently used memory.
In other words, the what happened with these instructions is that the blue parts of the picture below were added to the stack memory.
We can give names to the memory on the left and right of the address where
%rsp points in the stack.
The are called stack frames, where each stack frame corresponds to the data associated with one function
call. The memory on the right of the address pointed to be
%rsp at the point
called is the stack frame of whatever function calls
f(). This function is named the caller
(the function that calls), while
f() is the callee (the function being called).
The memory on the right of the
%rsp address at the point of
f() being called (we refer
to this as "entry
%rsp") is the caller's stack frame (red below), and the memory to its left
is the callee's stack frame.
The arguments and local variables of
f() live inside
f()'s stack frame. Subsequent
arguments (second, third, fourth, etc.) are stored at subsequently lower addresses below
call03.s for examples with more arguments), followed eventually by
any local variables in the caller.
The convention is that
%rspalways points to the lowest (leftmost) stack address that is currently used. This means that when a function declares a new local variable,
%rsphas to move down (left) and if a function returns,
%rsphas to move up (right) and back to where it was when the function was originally called.
%rsphappens in two ways: explicit modification via arithmetic instructions, and implicit modification as a side effect of special instructions. The former happens when the compiler knows exactly how many bytes a function requires
%rspto move by, and involves instructions like
subq $0x10, %rsp, which moves the stack pointer down by 16 bytes. The latter, side-effect modification happens when instruction
poprun. These instructions write the contents of a register onto the stack memory immediately to the left of the current
%rspand also modify
%rspto point to the beginning of this new data. For example,
pushq %raxwould write the 8 bytes from register
%rsp - 8and set
%rspto that address; it is equivalent to
movq %rax, -8(%rsp); subq $8, %rspor
subq $8, %rsp; movq %rax, (%rsp).
As an optimization, the compiler may choose to avoid writing arguments onto the stack. It does this for up to
six arguments, which per calling convention are held in specific registers.
call04.s shows this: the
C code we compile it from (
call04.c) is identical to the code in
Functions with more than six arguments
There is a limited number of registers in the x86-64 architecture, and you can write functions in C that
take any number of arguments! The calling convention says that the first six arguments max be passed in registers,
but that the 7th and above arguments are always passed in memory on the stack. Specifically, these
arguments go into the caller's stack frame, so they are stored above the entry
at the point where the function is called (see
As a function executes, it eventually reaches a
ret instruction in its assembly. The effect of
ret is to return to the caller (a form a control flow, as the next instruction needs to change).
But how does the processor know what instruction to execute next, and what to set
It turns out that the stack plays a role here, too. In a nutshell, each function call stores the return address as the very first (i.e., rightmost) data in the callee's stack frame. (If the function called takes more than six arguments, the return address is to the left of the 7th argument in the caller's stack frame.)
The stored return address makes it possible for each function to know exactly where to continue execution once it returns to its caller. (However, storing the return address on the stack also has some dangerous consequences, as we will see shortly.)
We can now define the full function entry and exit sequence. Both the caller and the callee have responsibilities in this sequence.
To prepare for a function call, the caller performs the following tasks:
The caller stores the first six arguments in the corresponding registers.
If the callee takes more than six arguments, or if some of its arguments are large, the caller must store the surplus arguments on its stack frame (in increasing order). The 7th argument must be stored at
(%rsp)(that is, the top of the stack) when the caller executes its
The caller saves any caller-saved registers (see last lecture's list). These are registers whose values the callee might overwrite, but which the caller needs to retain for later use.
The caller executes
callq FUNCTION. This has an effect like
pushq $NEXT_INSTRUCTION; jmp FUNCTION(or, equivalently,
subq $8, %rsp; movq $NEXT_INSTRUCTION, (%rsp); jmp FUNCTION), where
NEXT_INSTRUCTIONis the address of the instruction immediately following
To return from a function, the callee does the following:
The callee places its return value in
The callee restores the stack pointer to its value at entry ("entry
%rsp"), if necessary.
The callee executes the
retqinstruction. This has an effect like
popq %rip, which removes the return address from the stack and jumps to that address (because the instruction writes it into the special
Finally, the caller then cleans up any space it prepared for arguments and restores caller-saved registers if necessary.
Buffer overflow attacks
Now that we understand the calling convention and the stack, let's take a step back and think of some of the consequences of this well-defined memory layout. While a callee is not supposed to access its caller's stack frame (unless it's explicitly passed a pointer to an object within it), there is no principled mechanism in the x86-64 architecture that prevents such access.
In particular, if you can guess the address of a variable on the stack (either a local within the current function or a local/argument in a caller of the current function), your program can just write data to that address and overwrite whatever is there.
This can happen accidentally (due to bugs), but it becomes a much bigger problem if done deliberately by malicious actors: a user might provide input that causes a program to overwrite important data on the stack. This kind of attack is called a buffer overflow attack.
Today, we also understood in more detail how the stack segment of memory is structured and managed, and discussed how it grows and shrinks. We learned about how the compiler manages the stack pointer and how base pointers help it "unwind" the stack for debugging.
We then looked into how the very well-defined memory layout of the stack can become a danger if a program is compromised through a malicious input: by carefully crafting inputs that overwrite part of the stack memory via a buffer overflow, we can change important data and cause a program to execute arbitrary code.
In Lab 3, you will craft and execute buffer overflow attacks on a program yourself!