Protections & Bypass 101: NX/ASLR, Canaries, RELRO/PIE, and Practical Exploit Strategies (Behemoth + Utumno + Maze)

Protections & Bypass 101

Modern Linux binaries ship with layers of mitigations. Winning reliably means turning that stack into a decision tree and picking the shortest stable chain (leak → ret2libc/ROP/ret2dlresolve). This guide keeps it procedural: fast survey, choose a route, then prove with tiny PoCs. You’ll end with mini-labs and a non-spoiler tie-in to OverTheWire Behemoth, Utumno, and Maze.

If you’re following my CTF series, keep these handy: Binary Exploitation Deep Dive and RE Playbook.

Table of Contents

• 1) Decision tree & mindset
• 2) Quick survey & mitigation matrix
• 3) NX/DEP: shellcode vs ret2libc/ROP
• 4) ASLR + PIE: from leaks to reliability
• 5) Stack canaries: detect, leak, or avoid
• 6) RELRO (Partial vs Full): targets & pivots
• 7) Picking chains: ret2libc, ROP, ret2dlresolve, SROP
• 8) Info-leak cookbook (fmt/PLT/GOT/UB reads)
• 9) 32-bit vs 64-bit quirks
• 10) Behemoth + Utumno + Maze tie-ins
• Appendix A — One-liners
• Appendix B — Mini-labs
• Appendix C — ROP tooling cheat-sheet
• Appendix D — Printable field checklist
• Resource Library (Videos & Reading)

1) Decision tree & mindset

Mitigation survey →
NX?
off  → shellcode ok  → still consider short ROP for reliability
on   → ret2libc / ROP / ret2dlresolve
ASLR/PIE?
no   → fixed text base → leak libc only
yes  → need pointer leak (stack/libc/PLT/GOT) → compute bases
Canary?
no   → contiguous stack overwrite
yes  → leak or avoid contiguous overwrite (off-by-one, fptr/data)
RELRO?
partial → GOT writes viable
full    → favor fptr/vtable/.fini\_array/ret2dlresolve/ROP-only
Pick shortest stable chain → align stack → assert success criteria

Principles

• Identify first, exploit second. A 10-second survey beats 2h guessing.
• Shortest stable chain > clever long chains.
• Two primitives rule: secure read (leak) and write (precise overwrite).

Further references

• CWE-119: Memory Bounds Errors
• ROP Emporium (learn ROP with puzzles)

2) Quick survey & mitigation matrix

file ./target && checksec --file=./target
readelf -hlsdW ./target | sed -n '1,120p'     # PIE, sections, RELRO
objdump -R ./target | head                    # GOT entries (Partial vs Full context)

Mitigation matrix (what it implies)

Further references

• checksec (official)
• Hardening ELF with RELRO (Red Hat)

3) NX/DEP: shellcode vs ret2libc/ROP

• NX off: shellcode still works; but ROP may be cleaner (no badchars/encoders).

• NX on: execute existing code:

  • ret2libc: system("/bin/sh") or execve.
  • ROP: set registers, call functions, or emit syscall.

Demo video (conceptual DEP→ROP) Bypass DEP using ROP Chain & Execute Shellcode (mona.py)

Although the demo targets Windows, the NX→ROP idea maps directly to Linux CTFs: you turn data execution into code reuse.

Further references

• “ret2libc explained” (short guide)
• ROPgadget • one_gadget
• ROP Emporium — start at ret2win

4) ASLR + PIE: from leaks to reliability

• Non-PIE: code base fixed; leak libc and go ret2libc.
• PIE: binary + libs move; leak any code pointer (PLT/GOT/return site) to recover bases.
• Leak sources: %p sprays, puts(ptr), wrong-sized reads/writes, sloppy logging.
• Fork-servers can allow fast brute force if entropy is tiny and crashes fast.

Further references

• LiveOverflow Binary Exploitation (playlist)
• ELF gABI (linker/relocation background)

5) Stack canaries: detect, leak, or avoid

• Detect: presence of __stack_chk_fail or prologue save/epilogue compare.
• Leak: format strings (right offset), info-leaky prints, or indirect pointer reads.
• Avoid: use non-contiguous overwrites (off-by-one NUL on saved rbp), function pointers, vtables, or heap/data pointers.

Further references

• glibc hardening notes (stack protector)

6) RELRO (Partial vs Full): targets & pivots

• Partial RELRO ⇒ GOT writable before lock-down → classic GOT overwrite (e.g., puts@got → system).

• Full RELRO ⇒ GOT RO; pivot to:

  • .fini_array/.init_array (if writable in the target, not always).
  • Function pointers / vtables / callback tables in .data/heap.
  • ret2dlresolve: forge relocation and let the dynamic loader resolve system.

Further references

• Hardening ELF Binaries using RELRO
• Hardening Executables (flags overview)

7) Picking chains: ret2libc, ROP, ret2dlresolve, SROP

• ret2libc: leak libc → compute base → call system("/bin/sh"). Minimal gadgets.
• ROP: when you must load registers or craft syscalls; keep chains short, mind 16-byte alignment on x86-64.
• ret2dlresolve: no libc leak/GOT writes? Push fake relocation/strings → have the loader resolve system.
• SROP: craft a fake signal frame to set registers in one go (needs a syscall; ret gadget).

Further references

• ret2dlresolve primer (pwntools)
• SROP (paper)
• ret2csu (universal gadget) write-up

8) Info-leak cookbook (fmt/PLT/GOT/UB reads)

• PLT→GOT leak: puts(puts@got) → compute libc base.
• Format strings: find parameter offset (%7$lx…), then %s/%hn tricks for reads/writes.
• Uninitialized reads: printing stack/structs without zeroing leaks pointers/canaries.
• Type confusion: %s on an integer treated as a pointer ⇒ read arbitrary memory.

Demo video (fmt leak in practice) PicoCTF ‘flag-leak’ — Format String Vulnerabilities

Further references

• CWE-200: Information Exposure
• RPISEC — Modern Binary Exploitation (course)
• pwn.college — Program Security module

9) 32-bit vs 64-bit quirks

• i386: args on stack; ret2libc often two pushes + ret.
• x86-64: args in registers; need gadgets for rdi, rsi, rdx, etc.
• Alignment: keep 16-byte stack alignment before call on x86-64.

Further references

• System V AMD64 psABI

10) Behemoth + Utumno + Maze tie-ins (non-spoiler)

• Behemoth: overflow instinct + SUID/env edges; expect NX + Partial RELRO → libc leak + ret2libc.
• Utumno: RE first, exploit second; the info-leak choice usually decides chain length.
• Maze: add environment/path assumptions; with Full RELRO/canaries, favor fptr/vtable or ret2dlresolve.

Further references

• OverTheWire — Behemoth • OverTheWire — Utumno • OverTheWire — Maze

Appendix A — One-liners

Mitigation survey

checksec --file ./target
readelf -hlsdW ./target | sed -n '1,120p'
objdump -R ./target | sed -n '1,40p'

Find likely leaks & sinks

strings -a ./target | egrep -i 'puts|printf|system|execve|/bin/sh' | head
objdump -d -M intel ./target | egrep 'call.*puts|call.*printf' | sed -n '1,20p'

Quick ret2libc scaffold (pwntools)

from pwn import *
elf = context.binary = ELF('./target', checksec=False)
rop = ROP(elf)
# Example: leak puts then return to main
rop.call('puts', [elf.got['puts']]); rop.call(elf.symbols['main'])

Appendix B — Mini-labs

Safety: run in your own VM. These are didactic binaries.

Lab B1 — NX on, Partial RELRO, no PIE → ret2libc

// b1.c
#include <stdio.h>
#include <unistd.h>
void vuln(){ char buf[128]; puts("name?"); read(0, buf, 512); }
int main(){ setvbuf(stdout,NULL,_IONBF,0); vuln(); }

gcc -fno-stack-protector -no-pie -o b1 b1.c
# 1) ROP: puts(puts@got) → leak; 2) return to main; 3) system("/bin/sh")

Lab B2 — PIE + canary present → leak then chain

// b2.c
#include <stdio.h>
#include <unistd.h>
int main(){ char buf[64]; puts("say:"); read(0,buf,256); printf(buf); }

gcc -fstack-protector-strong -O0 -pie -o b2 b2.c
# Format string to leak: PIE ptr + canary + libc; then aligned ROP → system

Lab B3 — Full RELRO → ret2dlresolve

// b3.c
#include <stdio.h>
#include <string.h>
int main(){ char b[64]; puts("x?"); gets(b); }

gcc -Wl,-z,relro,-z,now -no-pie -fno-stack-protector -o b3 b3.c
# Build fake .rel.plt/.dynstr/.symtab on stack; trigger resolver to call system

Lab B4 — ret2csu (universal gadget) for register setup

# Use the __libc_csu_init gadgets to populate rdi/rsi/rdx when gadgets are scarce
# Pair with ROPgadget/pwndbg to locate and script the sequence.

Appendix C — ROP tooling cheat-sheet

• Find gadgets: ROPgadget, ROP(ELF) in pwntools.
• Libc versions: ldd ./target; remote → leak addr then match by offsets (avoid guessing).
• one_gadget caveat: each gadget has constraints (stack/env/registers) — check before using.
• ret2dlresolve helpers: pwntools Ret2dlresolvePayload (version-dependent).

Further references

• pwntools docs
• ROP Emporium (challenge set)
• ROP Emporium solutions (community)

Appendix D — Printable field checklist

Resource Library (Videos & Reading)

Videos (curated)

• Bypass DEP using ROP Chain & Execute Shellcode (mona.py) — conceptual DEP→ROP demo (Windows, idea maps to Linux): YouTube
• Format String printf Vulnerabilities (PicoCTF ‘flag-leak’) — hands-on info-leak: YouTube
• Binary Exploitation playlist (LiveOverflow) — ASLR/NX/ROP series: YouTube
• DEF CON — The Rise and Fall of Binary Exploitation (Stephen Sims) — high-level context: YouTube
• Hack your grades — fun/viral context piece: YouTube
• HAVOC C2 — Demon Bypasses Windows 11 Defender — blue-team adjacent, AV-evasion perspective: YouTube

Reading / Exercises / Tools

• ROP Emporium — progressive ROP labs: Site • ret2win intro • Community solutions
• RPISEC — Modern Binary Exploitation — full course notes/labs: GitHub
• pwn.college — Program Security — structured curriculum: Module
• RELRO overview — Red Hat blog • Hardening flags: Shibumi
• Tools — checksec • ROPgadget • one_gadget
• Techniques — ret2dlresolve primer • ret2csu write-up

Final note

Treat mitigations as waypoints, not walls: survey quickly, secure a leak, pick the shortest chain, and mind alignment & stability. Pair the labs with Behemoth/Utumno/Maze boxes to turn patterns into muscle memory.

Thanks for reading!

Until next time — Otsumachi!! 💖☄️✨