Reverse Engineering Playbook: Static & Dynamic Analysis, GDB/Ghidra Workflow, and Utumno + Maze Tie-ins

Reverse Engineering Playbook

Reverse engineering (RE) is the craft of recovering intent from compiled artifacts. This guide is hands-on and workflow-first: fast triage, static structure, dynamic behavior, then tight loops in GDB and a GUI decompiler (Ghidra/Cutter). We’ll also cover anti-debug patterns, mini case studies, and a non-spoiler mapping to OverTheWire Utumno and Maze.

If you’re following my CTF series, the official hubs are: OverTheWire — Utumno and OverTheWire — Maze.

Table of Contents

• 1) Mindset & quick wins
• 2) Static analysis essentials (ELF anatomy)
• 3) Dynamic analysis essentials (tracing & observing)
• 4) Toolbelt: CLI + GUI decompilers
• 5) Calling conventions & stack frames (quick mental model)
• 6) Reading disassembly like a map
• 7) GDB cookbook (breakpoints that matter)
• 8) Anti-debug & obfuscation patterns
• 9) Production workflow / checklist
• 10) Mini case studies (5–8 lines each)
• 11) Utumno + Maze tie-ins (non-spoiler)
• Appendix A — One-liners & snippets
• Appendix B — Mini-labs (safe, local)
• Appendix C — Cheat-sheets & hotkeys
• Appendix D — Printable RE checklist
• Resource Library (Videos & Reading)

1) Mindset & quick wins

• Behavior > bytes. Treat the binary as a black box first: inputs, outputs, files, network, errors, timing.
• Triangulate. Static (what could happen) ↔ Dynamic (what does happen). Iterate.
• Name the unknowns. Unknown state? Instrument it. Unknown format? Record a sample then diff.
• Notes pay off. Keep a RE logbook: entry points, strings of interest, function IDs, what I believe vs what I saw.

Orientation video:
Reverse Engineering for People Who HATE Assembly!

Further references:

• Reverse Engineering — Introduction & Motivation (playlist)
• Self-Learning Reverse Engineering (roadmap)

2) Static analysis essentials (ELF anatomy)

• Quick metadata: file, readelf -h/l/s/d, objdump -d -M intel, strings.
• Symbols: is it stripped? Check .symtab vs .dynsym. No symbols → rely on xrefs, patterns, and data-flow.
• PLT/GOT: identify imported calls; PLT thunks are useful anchors in disassembly.
• Relocations/RELRO: understand what is writable/read-only at runtime.
• Init/fini arrays: hidden execution points; sometimes logic lives there.

Demo video:
Reverse Engineering 101 tutorial with Stephen Sims

Further references:

• System V AMD64 psABI • ELF gABI
• checksec (mitigations)

3) Dynamic analysis essentials (tracing & observing)

• Syscalls: strace -f -o s.log ./prog to see file/network/exec patterns.
• Libcalls: ltrace -f -o l.log ./prog to see printf, strcmp, fopen flows.
• Environment: run with env -i (clean env) to test assumptions; try weird PWD, LANG, TZ.
• Timing: compare behavior at different speeds (e.g., sleep 0.1 between inputs) to discover state machines.

Hands-on video:
everything is open source if you can reverse engineer (try it RIGHT NOW!)

Further references:

• strace • ltrace manpage

4) Toolbelt: CLI + GUI decompilers

• CLI: strings, hexdump -C, readelf, objdump, nm, xxd, file.
• Ghidra: decompile, rename, structure, xrefs; great for data-flow & types.
• radare2/Cutter: fast graph view, patching, scripting; Cutter gives a friendly GUI over r2.
• Diffing: build tiny clones of suspicious functions in C, then diff behavior.

Video (learning path):
Self-Learning Reverse Engineering in 2022

Further references:

• Ghidra (official) • radare2 • Cutter
• pwndbg • GEF for GDB

5) Calling conventions & stack frames (quick mental model)

• x86-64 SysV: args in rdi, rsi, rdx, rcx, r8, r9; return in rax; stack 16-byte aligned at call.
• i386 cdecl: args on stack (right→left); return in eax.
• Frames: prologue sets up rbp frame; leaf functions may omit it.

Further references:

• x86-64 psABI • i386 psABI (historical)

6) Reading disassembly like a map

• Entrypoints: main, constructors (.init_array), or custom entry wrappers.
• String-xref first: jump into callers of strcmp, fopen, system, execve, printf.
• Switches: look for jmp [table + index*8] and table regions.
• Integer parsing: strtol, sscanf, manual digit loops; check for base/overflow handling.
• Crypto/transform: loops with XOR/add/rotate; identify constants and rolling state.

Further references:

• Ghidra User Guide (decompiler basics)
• radare2 Book

7) GDB cookbook (breakpoints that matter)

• Locate hot paths: b *main, b strcmp, b read, b write, b malloc, b free.
• Follow children: set follow-fork-mode child then catch exec.
• Tame PIE: vmmap (pwndbg) to grab bases; or compute &main and mask lower page bits.
• Watch state: watch *0xADDR to catch unexpected writes; x/s, x/20gx, x/20i.
• Scripting: small .gdbinit to auto-break & pretty-print.

Deep-dive video:
PRACTICAL REVERSE ENGINEERING

Further references:

• pwndbg quickstart • GEF docs

8) Anti-debug & obfuscation patterns

• ptrace anti-debug: ptrace(PTRACE_TRACEME) or getppid() checks; bypass by patching or faking return.
• Timing: rdtsc, sleep-gates; normalize by stepping over / patching.
• Self-checksums / section hashes: identify hash loops; NOP them or re-compute.
• Packers: watch for mprotect, decompress, then a jump into RWX region; attach after unpack.

Focused video:
Reverse Engineering Anti-Debugging Techniques (with Nathan Baggs!)

Further references:

• Linux ptrace manpage
• Cracking Software with Reverse Engineering (discussion)

9) Production workflow / checklist

1. Black-box pass: run, give junk input, collect strace/ltrace, note files & strings.
2. Static skim: map imports, find string/xref hotspots, mark candidate functions.
3. Hypotheses: write what each hot function likely does.
4. Instrument: set breakpoints, record real arguments/returns; verify/adjust hypotheses.
5. Refactor view: rename functions/vars in Ghidra/Cutter; document data structures.
6. Finalize: produce a concise spec of the binary (inputs → processing → outputs/side-effects).
7. Defensive note: if you discovered insecure usage (e.g., system()), propose safer patterns.

Further references:

• Software Reverse Engineering Methodology (overview)

10) Mini case studies (5–8 lines each)

Case 1 — Password checker (string & branches)
ltrace shows strcmp(input, "SOMETHING"). In Ghidra, a loop XORs bytes of the input with a rotating key, then compares to a constant table — so it’s not plaintext. Reimplement in Python to derive the expected transformed bytes, then invert the transform to recover the secret.

Case 2 — File format probe (state machine)
strace indicates the program opens two files, reads fixed 16 bytes, rejects if magic mismatch. Disassembly reveals memcmp(buf, "UTMN\0\1", 6). The next branch computes CRC32 over chunk records; accepting path writes an output. Build a valid minimal sample to pass both checks.

Case 3 — Network puzzle (line protocol)
ltrace prints via printf("%02x ", byte) inside a loop; recv shows 64 bytes per frame. Rebuild the frame in a Python client; script the handshake (banner → nonce → response) and confirm via strace that the program transitions to the success branch.

Case study video:
Cities: Skylines II Malware — Full Reverse Engineering Analysis

11) Utumno + Maze tie-ins (non-spoiler)

• Utumno exercises static reading and small dynamic validations; expect tight loops, string ops, and input transforms.
• Maze mixes RE with exploitation instincts (watch for system(), environment assumptions, path joins).
• Your best friends: strings/xrefs, ltrace/strace, and a disciplined GDB loop with function renaming in Ghidra/Cutter.

Further references:

• OverTheWire — Utumno • OverTheWire — Maze

Appendix A — One-liners & snippets

Quick ELF survey

file ./bin && checksec --file ./bin
readelf -hlsd ./bin | sed -n '1,80p'
objdump -d -M intel ./bin | sed -n '1,120p'
strings -a -n 5 ./bin | sort -u | sed -n '1,80p'

Trace everything

strace -f -o s.log ./bin 2>&1 | head
ltrace -f -o l.log ./bin 2>&1 | head

GDB PIE base & helpful breaks (pwndbg)

# gdb -q ./bin
vmmap
break *main
break strcmp
break read
set follow-fork-mode child
run

Cutter/r2 quickstart (commands)

aa          # analyze all
afl         # list functions
pdf @ main  # print disasm of function
axt sym.imp.printf  # xrefs to printf
VV          # graph view (terminal)

Appendix B — Mini-labs (safe, local)

Lab safety: use your own VM; binaries here are benign teaching aids.

Lab B1 — Strings & xrefs → behavior sketch

// lab-strings.c
#include <stdio.h>
#include <string.h>
int main(){
  char s[64]; puts("code?"); fgets(s,sizeof(s),stdin);
  if(strncmp(s,"UTMN-",5)==0) puts("ok");
  else puts("nope");
}

Task: run strings, find "UTMN-" and confirm in Ghidra where it branches.

Lab B2 — ltrace vs strace

// lab-trace.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main(){
  char name[64]; puts("name:");
  fgets(name,sizeof(name),stdin);
  if (strstr(name,"maze")) system("echo yay");
  else puts("hmm");
}

Task: show ltrace hitting strstr and system, and strace spawning /bin/sh -c.

Lab B3 — Switch table + file format

// lab-switch.c
#include <stdio.h>
int main(int argc, char **argv){
  if(argc<2){ puts("mode?"); return 1; }
  switch(argv[1][0]){
    case 'a': puts("alpha"); break;
    case 'b': puts("bravo"); break;
    default: puts("else");
  }
}

Task: find the jump table in disassembly; explain how indices map to cases.

Lab B4 — Anti-debug taste

// lab-anti.c
#include <sys/ptrace.h>
#include <stdio.h>
#include <stdlib.h>
int main(){
  if (ptrace(0,0,0,0)==-1){ puts("debugger?"); exit(1); }
  puts("hello");
}

Task: observe behavior under GDB; patch the check (NOP) in Cutter or force return value.

Appendix C — Cheat-sheets & hotkeys

• Ghidra: R = rename, Y = retype, X = xrefs, P = function signature, D = data, G = go to.
• Cutter: G (goto), X (xrefs), right-click graph to rename & retype quickly.
• pwndbg: vmmap, telescope $rsp, context, hexdump, nearpc.

Further references:

• Ghidra Cheat Sheet • radare2 Book
• pwndbg commands

Appendix D — Printable RE checklist

Resource Library (Videos & Reading)

Videos (curated)

• Reverse Engineering for People Who HATE Assembly!
• everything is open source if you can reverse engineer (try it RIGHT NOW!)
• Self-Learning Reverse Engineering in 2022
• Learn Reverse Engineering (for hacking games)
• Cities: Skylines II Malware — Full RE Analysis
• Reverse Engineering — Introduction & Motivation (playlist)
• Cracking Software with Reverse Engineering 😳
• Reverse Engineering 101 with Stephen Sims
• Reverse Engineering Anti-Debugging Techniques (with Nathan Baggs!)
• PRACTICAL REVERSE ENGINEERING

Reading / Tools

• OverTheWire — Utumno • OverTheWire — Maze
• System V AMD64 psABI • ELF gABI
• Ghidra • radare2 • Cutter
• strace • ltrace manpage
• pwndbg • GEF

Final note

Use this as a playbook: black-box first, then lock onto hot spots with xrefs, confirm in GDB, and rename ruthlessly in your decompiler. Pair the mini-labs here with your Utumno/Maze runs to build speed and accuracy.

Thanks for reading!

Until next time — Otsumachi!! 💖☄️✨