Part 6: x86-64 Arithmetic Instructions, Overflow & The RFLAGS Register
Lesson Overview & ALU Mechanics
Arithmetic instructions represent the core mathematical engine of any computer processor. In x86-64, the Arithmetic Logic Unit (ALU) performs integer calculations and simultaneously updates conditional status bits in the 64-bit RFLAGS register. In Part 6 of our masterclass, we explore integer addition and subtraction (ADD, SUB, ADC, SBB), signed vs. unsigned multiplication (MUL, IMUL) and division (DIV, IDIV), sign-extension mechanics (CDQ, CQO), and analyze the four critical condition flags: Carry, Zero, Sign, and Overflow.
1. Addition, Subtraction & Multi-Precision Math
The fundamental instructions for integer addition and subtraction are ADD and SUB:
add rax, 50 ; RAX = RAX + 50 (updates CF, ZF, SF, OF)
sub rbx, rcx ; RBX = RBX - RCX
inc rdx ; RDX = RDX + 1 (updates ZF, SF, OF, but preserves CF!)
dec rdi ; RDI = RDI - 1 (preserves CF!)
neg rax ; Two's complement negation: RAX = 0 - RAX
When computing with integers larger than 64 bits (such as 128-bit cryptography or 256-bit big integers), we use Add with Carry (ADC) and Subtract with Borrow (SBB), which incorporate the Carry Flag from the previous operation:
; 128-bit Addition: [RDX:RAX] + [R9:R8] -> Result in [RDX:RAX]
add rax, r8 ; Add low 64 bits. If carry occurs, CF is set to 1.
adc rdx, r9 ; Add high 64 bits PLUS the Carry Flag (CF).
2. Integer Multiplication: MUL vs. IMUL
x86-64 differentiates strictly between unsigned multiplication (MUL) and signed two's-complement multiplication (IMUL):
- Unsigned (MUL): Takes one operand. Multiplies
RAXby the operand. Because multiplying two 64-bit numbers can produce a 128-bit result, the output is split across RDX:RAX (High 64 bits in RDX, Low 64 bits in RAX). - Signed (IMUL): Extremely flexible. In addition to the single-operand form, it supports two-operand (
imul rax, rbx) and three-operand forms (imul rax, rbx, 10) that truncate to the destination register size.
3. Division and Sign Extension (CQO / CDQ)
Integer division in x86-64 requires a 128-bit dividend split across RDX:RAX. The quotient is stored in RAX, and the remainder (modulo) is stored in RDX:
; Signed 64-bit Division: RAX / RBX
mov rax, -100
mov rbx, 7
cqo ; Convert Quadword to Octoword: Sign-extends RAX into RDX:RAX
idiv rbx ; RAX = -14 (Quotient), RDX = -2 (Remainder)
FATAL TRAP WARNING: Division by Zero & Overflow
If the divisor is zero, or if the quotient exceeds the capacity of the destination register, the CPU raises an immediate hardware exception (#DE - Divide Error), which the Linux kernel translates into a crash signal (SIGFPE). For signed division, always execute CQO (or CDQ for 32-bit) beforehand!
4. The RFLAGS Status Register
Every ALU calculation updates status bits in the RFLAGS register. Understanding these flags is crucial for implementing conditional branching in assembly:
| Flag | Name | Set Condition (= 1) | Tested By |
|---|---|---|---|
| ZF | Zero Flag | Set when the mathematical result of an operation is exactly zero | JZ, JNZ, JE, JNE |
| CF | Carry Flag | Set on unsigned overflow (carry out of MSB or borrow during subtraction) | JC, JNC, JB, JAE |
| SF | Sign Flag | Set equal to the MSB of the result (1 indicates negative in two's complement) | JS, JNS |
| OF | Overflow Flag | Set on signed two's complement overflow (e.g. adding two positives yields a negative) | JO, JNO, JG, JL |
Summary & Next Step
Status flags reflect the health and sign of calculations. In Part 7, we examine bitwise logic (AND, OR, XOR, NOT, TEST), logical and arithmetic bit shifts (SHL, SHR, SAR), and bit-field manipulation techniques.
Related reading
Part 11: Linux 64-Bit System Calls, NASM Macros & Dynamic Memory Allocation
Learn low-level systems programming: Linux 64-bit SYSCALL table, multi-argument NASM %macro directives, file I/O operations, and sys_mmap memory allocation.
By Md. Harun-Or-Roshid
Part 10: Stack Frames, Procedures & The System V AMD64 ABI in Assembly
Master assembly subroutines: CALL, RET, stack frames, RBP base pointers, 16-byte stack alignment, and the official System V AMD64 ABI calling convention.
By Md. Harun-Or-Roshid
Part 9: Assembly Arrays, String Primitives & Hardware Streaming (REP MOVSB)
Master memory buffers and strings in x86-64 assembly: arrays, strides, and hardware block primitives including MOVSB, LODSB, STOSB, SCASB, and REP.
By Md. Harun-Or-Roshid
Part 8: Conditional Branching, Jump Instructions & Fast Loops in Assembly
Learn how conditional jumps work in x86-64 assembly using CMP, TEST, JE, JNE, JG, JL, JA, JB, and write high-performance loops with branch prediction.
By Md. Harun-Or-Roshid