Stack Frames & System V AMD64 ABI Assembly

Lesson Overview & Subroutines

Modular software engineering is impossible without reusable functions and subroutines. In low-level assembly, procedures require precise orchestration of the hardware Call Stack, saving return addresses, establishing stack base frames, managing local variables, and adhering to strict platform calling conventions. In Part 10 of our curriculum, we master CALL and RET, standard function prologues/epilogues, the System V AMD64 ABI (Linux/macOS 64-bit calling convention), the 16-byte stack alignment invariant, the 128-byte Red Zone, and implement a recursive factorial function.

1. Subroutine Primitives: CALL and RET

Functions in assembly are invoked using CALL and completed using RET:

CALL target_address: Pushes the 64-bit address of the next contiguous instruction (the return address) onto the stack (sub rsp, 8; mov [rsp], rip), and assigns RIP = target_address.
RET: Pops the 64-bit address from the top of the stack directly into the Instruction Pointer (mov rip, [rsp]; add rsp, 8), resuming execution in the caller.

2. Standard Stack Frame Prologue & Epilogue

To allocate private space for local variables while preserving the caller's reference frame, functions construct a Stack Frame using the Base Pointer (RBP):

my_function:

; 1. Function Prologue

push rbp ; Save the caller's old base pointer

mov rbp, rsp ; Establish new base pointer at current top of stack

sub rsp, 32 ; Allocate 32 bytes for local variables

; 2. Function Body

mov [rbp - 8], rdi ; Store 1st incoming parameter to local variable

mov rax, [rbp - 8] ; Prepare return value in RAX

; 3. Function Epilogue

mov rsp, rbp ; Deallocate local variables

pop rbp ; Restore caller's old base pointer (or use instruction 'leave')

ret ; Return to caller

3. The System V AMD64 ABI (Calling Convention)

In 64-bit Linux, FreeBSD, and macOS, all functions (including the standard C library) adhere to the System V AMD64 ABI:

Role	Registers Assigned	Protocol / Rules
Function Arguments (1 to 6)	RDI, RSI, RDX, RCX, R8, R9	Passed in registers. 7th argument and above are pushed to the stack in reverse order.
Return Values	RAX (and RDX for 128-bit)	Primary return value in RAX. Floating point returned in XMM0.
Callee-Saved (Preserved)	RBX, RSP, RBP, R12, R13, R14, R15	If your function modifies any of these, you must push and pop them to preserve their values for the caller!
Caller-Saved (Scratch)	RAX, RDI, RSI, RDX, RCX, R8, R9, R10, R11	Can be freely overwritten by the function. Caller must save them if needed.

THE 16-BYTE STACK ALIGNMENT INVARIANT

The System V ABI mandates that RSP must be aligned to a 16-byte boundary immediately prior to executing any CALL instruction! Standard C library functions (like printf) use SSE/AVX vector instructions internally that trigger a CPU General Protection Fault (#GP / crash) if the stack is misaligned.

4. The 128-Byte "Red Zone"

The System V ABI allocates a 128-byte scratch buffer below the current stack pointer: [rsp - 1] through [rsp - 128]. Leaf functions (functions that call no other subroutines) can use this memory for local storage without decrementing RSP! Signal handlers and kernel interrupts are prohibited from overwriting this region.

5. Practical Example: Recursive Factorial in Assembly

global factorial

; Signature: uint64_t factorial(uint64_t n) -> n in RDI, return in RAX

factorial:

cmp rdi, 1

jg .recurse ; If n > 1, compute recursively

mov rax, 1 ; Base case: factorial(0) = factorial(1) = 1

ret

.recurse:

push rdi ; Save current 'n' on the stack (maintains 16-byte alignment when combined with CALL)

dec rdi ; RDI = n - 1

call factorial ; Recursively call: returns (n - 1)! in RAX

pop rdi ; Restore saved 'n'

imul rax, rdi ; RAX = (n - 1)! * n

ret

Summary & Next Step

Stack frames and calling conventions allow seamless interoperation with operating systems and C libraries. In Part 11, our series grand finale, we master Linux 64-bit System Calls, construct reusable multi-line NASM macros, perform low-level file I/O, and allocate dynamic memory directly from the Linux kernel using sys_mmap.