Reverse Engineering - Taking Things Apart to Understand How They Work¶

Table of Contents¶

Introduction to Reverse Engineering - Taking Things Apart to Understand How They Work¶

What Actually Is Reverse Engineering?¶

Reverse engineering rips compiled code apart to see what it actually does You dig through the machine code running on real hardware , trace function calls , map data flows to understand the original logic and algorithms when developers ship binaries without source

Why Bother Learning This?¶

Legacy codebases need understanding Companies keep old proprietary software running for years , but when the original developers leave and documentation vanishes , reverse engineering becomes the only way to maintain , debug , or extend these critical systems that businesses depend on daily

Understanding Legacy Software: Old programs nobody remembers how they work You inherit some proprietary tool at work , reverse engineering lets you figure out what it actually does without the original source code that got lost years ago

Interfacing with Closed-Source Software: Need to work with third-party libraries Sometimes you need to integrate with closed-source APIs or libraries , understanding their internal behavior through reverse engineering helps you build proper interfaces and workarounds

Learning How Computers Actually Work: Assembly forces you to understand low-level details When you reverse engineer , you're forced to understand CPU registers , memory layouts , calling conventions , and system calls , making you a better programmer who actually knows what happens under the hood

Debugging Without Source: Third-party crashes happen all the time Ever had a vendor library crash with no source available , reverse engineering lets you trace the problem to the exact instruction causing the fault and understand why it happens

Preservation and Modification: Old software needs fixes sometimes Want to patch security holes in abandoned software or add features to programs no longer maintained , reverse engineering gives you the knowledge to modify binaries directly

It's Just Plain Cool: Assembly puzzles are satisfying to solve There's something deeply satisfying about staring at raw machine code and slowly piecing together what the original programmer intended , like solving a complex technical puzzle

Core Principles - The Mindset That Actually Works¶

Patience is not optional Reverse engineering takes time , you'll spend hours staring at assembly trying to figure out what a single function does , that's normal , don't rush it , the first time you reverse a function and finally understand it , that feeling is worth the hours

Start Simple , Stay Systematic: Break down complex binaries into manageable pieces Don't try to understand a 100 , 000 line binary all at once , find the entry point , follow the control flow , understand one function at a time , one basic block at a time if needed

Tools Help , But Understanding Matters: Decompilers save time but need assembly knowledge Decompilers are amazing and save hours , but if you don't understand assembly , you'll get lost when the decompiler output doesn't make sense , learn the fundamentals first

Pattern Recognition: Assembly patterns become familiar over time After reversing enough code , you'll start recognizing patterns like "oh , this is a string comparison" or "this looks like a loop" or "that's definitely calling malloc" , the more patterns you know , the faster you work

Document Everything: You'll forget what you learned yesterday Take notes constantly , comment your findings , draw diagrams , future you will thank present you when you need to remember how that custom protocol works

It's Okay to Get Stuck: Everyone gets stuck sometimes Everyone hits code they can't figure out , take a break , ask questions , look for similar patterns , you'll figure it out eventually

Linux Advantages - Why I Prefer Linux for RE¶

Linux makes reverse engineering easier Linux has native ELF format support , open source tools , and direct system call access that makes analyzing binaries much more straightforward than Windows alternatives

Native ELF Format: ELF is well-documented and tool-supported Linux binaries are ELF format , it's well-documented , tools support it well , and it's easier to work with than Windows PE format in my experience

Open Source Tool Ecosystem: Free tools are everywhere on Linux You've got GDB , Radare2 , Ghidra , and tons of other excellent free tools , Windows has good tools too , but Linux has more free options

Scripting Everything: Python and Bash are built in Bash and Python are built in , most RE tools have Python APIs , you can automate everything , want to analyze 100 binaries , write a script

Direct System Call Access: strace and ltrace work natively Want to see what syscalls a program makes , strace it , want to see library calls , ltrace it , these tools are native to Linux and incredibly useful

Package Management: apt install gets you set up fast apt install gdb radare2 ghidra and you're done , no hunting for installers , no dealing with Windows UAC prompts , just works

The Terminal Is Your Friend: Command line tools make you efficient Once you get comfortable with command line tools , you'll work faster , GUI tools are great , but knowing how to do things from the terminal makes you more efficient

CPU Architecture and Assembly Fundamentals¶

Programming Foundation: Understanding CPU architecture is crucial for reverse engineering. For practical C programming examples that demonstrate these low level concepts , see the C Programming Cheatsheet.

x86/x64 Architecture Overview¶

CPU Components¶

ALU (Arithmetic Logic Unit): Performs arithmetic and logical operations
CU (Control Unit): Directs operations using control signals
Registers: High-speed storage locations
Cache: Fast memory close to CPU
MMU (Memory Management Unit): Handles virtual-to-physical address translation

Von Neumann vs Harvard Architecture¶

Von Neumann: Shared memory for data and instructions (x86 uses this)
Harvard: Separate memories for data and instructions (some embedded systems)

Registers¶

General Purpose Registers (64-bit) - Your New Best Friends¶

These are the registers you'll see constantly. Learn them , love them , they're your friends.

# x86-64 general purpose registers
RAX - Accumulator
# Holds return values from functions
# Used for arithmetic operations
# First operand and result for many instructions

RBX - Base register
# Often used for pointers
# Sometimes used for array base addresses
# Callee-saved (function must preserve it)

RCX - Counter
# Used in loops (DEC/JNZ patterns)
# Used for shift counts
# Caller-saved (can be modified by called functions)

RDX - Data register
# Second operand in arithmetic
# I/O port numbers
# Division operations (holds high bits)
# Caller-saved

RSI - Source Index
# Source pointer in string operations
# Second function argument (System V calling convention)
# Caller-saved

RDI - Destination Index
# Destination pointer in string operations
# First function argument (System V calling convention)
# Caller-saved

RBP - Base Pointer (frame pointer)
# Points to current stack frame
# Used to access local variables and arguments
# Callee-saved

RSP - Stack Pointer
# Points to top of stack
# Changes as you push/pop
# Automatically managed by CPU

R8-R15 - Extended registers
# More registers added in x86-64
# R8-R11: Caller-saved (can be modified)
# R12-R15: Callee-saved (must be preserved)

Pro tip: In 32-bit mode , these are EAX , EBX , ECX , etc. Same registers , just 32-bit instead of 64-bit. The 32-bit versions are still accessible as the lower 32 bits of the 64-bit registers (e.g. , mov eax, 5 clears the upper 32 bits of RAX).

Special Purpose Registers - The Important Ones¶

These registers control execution flow and CPU state:

# Instruction Pointer
RIP - Points to next instruction to execute
# This is THE most important register
# When RIP points somewhere, that's where execution goes
# Jump instructions modify RIP
# Call instructions push current RIP, then modify RIP
# Return instructions pop RIP from stack

# Flags Register
RFLAGS - Status flags
# ZF (Zero Flag): Set if result is zero
# CF (Carry Flag): Set if unsigned overflow
# SF (Sign Flag): Set if result is negative
# OF (Overflow Flag): Set if signed overflow
# PF (Parity Flag): Even parity
# AF (Auxiliary Flag): BCD operations

# Conditional jumps depend on these flags
cmp rax, rbx  # Sets flags based on comparison
je label      # Jump if ZF set (equal)
jg label      # Jump if !ZF && !SF && !OF (greater, signed)

Segment Registers (Mostly Historical):

# CS - Code Segment (where code is)
# DS - Data Segment (where data is)
# SS - Stack Segment (where stack is)
# ES, FS, GS - Extra segments

# In 64-bit Linux, segments are mostly ignored
# Flat memory model - everything in one address space
# Still present for compatibility, but not usually relevant

Register Usage in Functions - The Calling Convention¶

This is crucial. When you see a function call , you need to know where arguments go and where return values come from.

System V AMD64 Calling Convention (Linux/macOS):

# Function arguments (in order):
# 1st: RDI
# 2nd: RSI
# 3rd: RDX
# 4th: RCX
# 5th: R8
# 6th: R9
# More than 6? On the stack

# Return value: RAX (or RDX:RAX for 128-bit returns)

# Callee-saved registers (function must preserve):
# RBX, RBP, R12, R13, R14, R15
# If function uses these, it must restore original values

# Caller-saved registers (function can modify):
# RAX, RCX, RDX, RSI, RDI, R8, R9, R10, R11, RSP
# Caller must save these if needed after function call

Why This Matters:

# Function call example:
mov rdi, 10      # First argument = 10
mov rsi, 20      # Second argument = 20
call add_numbers # Call function
# Result in rax

# Inside function:
add_numbers:
    mov rax, rdi  # Load first argument
    add rax, rsi  # Add second argument
    ret           # Return (value in rax)

Real Example: When you see call printf , you know: - RDI = format string - RSI = first argument - RDX = second argument - etc. - Return value (number of chars printed) in RAX

This pattern recognition is key to understanding function calls in reverse engineering.

x86/x64 Architecture Deep Dive - The Fun Stuff (Really, I Promise)¶

Alright , time to get into the weeds. Don't worry , it's not as scary as it sounds. Think of the CPU like a chef in a kitchen - it has ingredients (registers) , instructions (what to do) , and decides what to cook next (RIP).

Control Flow and Instruction Pointer (RIP) - Where Are We Going?¶

The RIP register is basically the GPS of your CPU. It always knows where it's going next. Without it , your CPU would be lost , wandering around memory like "uh , what do I do now?"

Here's how it works in plain English:

Linear Execution (The Normal Case): Imagine reading a book page by page. That's what RIP does normally - it just goes to the next instruction. You execute instruction 1 , then instruction 2 , then instruction 3. Simple , right?

mov rax, 5      ; RIP points here
add rax, 10     ; After mov, RIP automatically points here
mov rbx, rax    ; After add, RIP automatically points here

The CPU automatically increments RIP after each instruction. It's like your eye automatically moving to the next word as you read.

Control Flow Instructions (Changing Direction): But sometimes you want to skip ahead , go back , or jump around. That's when you use control flow instructions:

jmp label       ; "Hey RIP, stop being linear, go to label instead!"
call function   ; "Save where we are, then jump to function"
ret             ; "Go back to where we came from"
je label        ; "If things are equal, jump to label, otherwise continue"

real world analogy: Think of it like a choose-your-own-adventure book. Normally you read page by page (linear execution). But sometimes the book says "if you chose to fight the dragon , go to page 50" (conditional jump). That's what RIP does - it follows these instructions.

Why this matters for reverse engineering: When you're reading assembly , you need to understand where execution is going. If you see jne label , you need to know: 1. What condition is being checked? (Usually right before the jump) 2. Where does execution go if the condition is true? 3. Where does it go if false?

This is how you understand program logic in reverse engineering.

The Stack and its Registers (RSP, RBP) - Memory's Filing Cabinet¶

The stack is like a filing cabinet, but you can only put things on top and take things from the top. It's a LIFO (Last-In, First-Out) structure. Like a stack of plates - you put plates on top, and you take plates from the top. You can't grab a plate from the middle.

RSP (Stack Pointer) - "Where's the top of the stack?"

RSP always points to the top of the stack. But here's the weird part - the stack grows DOWNWARDS in memory. I know, it's backwards from how you'd think. It's like building a tower that goes into the ground instead of into the sky.

push rax    ; Put rax on stack, RSP moves DOWN (decrements)
pop rbx     ; Take from stack, RSP moves UP (increments)

Why backwards? Historical reasons, mostly. CPUs had limited memory, so they put stacks at high addresses and had them grow down. This way stacks and heaps (which grow up) could meet in the middle. Genius, right?

RBP (Base Pointer) - "Where's my stuff?"

RBP is like a bookmark. When you enter a function, you save where the stack was, then use RBP as a reference point. This lets you access local variables and function arguments easily.

Think of it like this: You walk into a room (function) and put a marker on the floor (set RBP). Now you know "my bag is 3 steps from the marker" (local variable at [rbp-8]). Even if you move around the room (RSP changes), you can always find your bag using the marker.

; Function prologue - setting up the room
push rbp        ; "Remember where the last room's marker was"
mov  rbp, rsp   ; "Put marker here - this is OUR room now"
sub  rsp, 16    ; "Make space for our stuff (16 bytes for locals)"

; Now we can use:
; [rbp-8] = first local variable
; [rbp-16] = second local variable
; [rbp+8] = first function argument
; [rbp+16] = second function argument

; Function epilogue - cleaning up
mov  rsp, rbp   ; "Clean up our stuff"
pop  rbp        ; "Restore the previous room's marker"
ret             ; "Leave the room"

The Stack Frame Layout (What's Actually in Memory):

When you're inside a function, the stack looks like this (growing downward):

High addresses (old stuff):
  [rbp+24] - Third argument (if any)
  [rbp+16] - Second argument  
  [rbp+8]  - First argument
  [rbp]    - Saved RBP (old base pointer)

  [rbp-8]  - First local variable
  [rbp-16] - Second local variable

Low addresses (newer stuff):
  [rsp]    - Top of stack (current position)

Practical reverse engineering tip: When you see mov rax, [rbp-8], you know it's accessing a local variable. When you see mov rax, [rbp+8], you know it's accessing a function argument. This pattern recognition is gold for understanding code.

Flags Register (RFLAGS) - The CPU's Mood Ring¶

RFLAGS is like the CPU's mood ring - it tells you how the last operation went. Did it succeed? Fail? Overflow? These flags control conditional jumps, which is how programs make decisions.

ZF (Zero Flag) - "Did we get zero?"

Set to 1 if the result is zero, 0 otherwise. This is used CONSTANTLY in assembly.

cmp rax, rbx    ; Compare rax and rbx (sets ZF if equal)
je label        ; Jump if equal (ZF is 1)
jne label       ; Jump if not equal (ZF is 0)

# Other instructions that set ZF:
test rax, rax   ; Sets ZF if rax is zero
sub rax, 5      ; Sets ZF if result is zero

Real example: Want to check if a password is correct? Compare entered password with stored password. If ZF is set (equal), password is correct.

SF (Sign Flag) - "Is it negative?"

Set to 1 if result is negative (most significant bit is 1), 0 otherwise.

mov rax, -5
test rax, rax   ; SF is now 1 (negative)
js label        ; Jump if signed (negative)
jns label       ; Jump if not signed (positive or zero)

CF (Carry Flag) - "Did we overflow (unsigned)?"

For unsigned arithmetic, CF tells you if there was overflow.

mov rax, 0xFFFFFFFFFFFFFFFF  ; Max 64-bit value
add rax, 1                   ; Overflow! CF is set
jc label                     ; Jump if carry (overflow occurred)

OF (Overflow Flag) - "Did we overflow (signed)?"

For signed arithmetic, OF tells you if there was overflow.

mov rax, 0x7FFFFFFFFFFFFFFF  ; Max positive signed 64-bit
add rax, 1                   ; Signed overflow! OF is set
jo label                     ; Jump if overflow

Why flags matter:

Every conditional statement in high level code becomes flags + conditional jumps in assembly:

if (x == 0) {
    do_something();
}

Becomes:

cmp rax, 0      ; Compare x with 0, sets ZF
je do_something ; If zero flag set, jump to do_something

Understanding flags is crucial for understanding program logic in reverse engineering. You'll see cmp and test everywhere - they're setting up these flags for conditional jumps.

Pro tip: The flags register has other flags too (PF for parity, AF for BCD), but ZF, CF, SF, and OF are the ones you'll see 95% of the time. Focus on those first.

Practical Assembly Example: C to Assembly - See It in Action¶

Alright , let's stop talking theory and see this stuff actually work. This is where it all clicks - when you see high level code become assembly.

Simple C Program - Let's Start Easy¶

Let's take the simplest possible function and see what the compiler does with it:

// compile with: gcc -S -O0 -m32 -o example.s example.c
int add_numbers(int a, int b) {
    int result;
    result = a + b;
    return result;
}

This is as simple as it gets. Two numbers go in , one number comes out. Let's see what GCC makes of it.

Compile it:

gcc -S -O0 -m32 example.c -o example.s
# -S = output assembly
# -O0 = no optimization (makes it easier to read)
# -m32 = 32-bit (smaller, easier to understand)

Disassembly Analysis - Line by Line, Let's Figure This Out¶

Here's what GCC generated. I'll explain every line:

add_numbers:
    ; --- Function Prologue - Setting Up Shop ---
    push   ebp                ; Save old base pointer
    ; "Hey, remember where the caller's stack frame was"
    ; We're about to set up our own, so save this

    mov    ebp, esp           ; Set new base pointer
    ; "This is OUR stack frame now. ebp points to it."
    ; Now we can use [ebp+offset] to access arguments and locals

    ; --- Function Body - The Actual Work ---
    mov    edx, [ebp+8]       ; Load first argument 'a'
    ; Arguments are on the stack ABOVE the saved ebp
    ; [ebp+4] = return address (pushed by call instruction)
    ; [ebp+8] = first argument 'a'
    ; [ebp+12] = second argument 'b'

    mov    eax, [ebp+12]      ; Load second argument 'b'
    ; Put it in eax because we're going to add to it

    add    eax, edx           ; eax = eax + edx
    ; So: eax = b + a = a + b
    ; Result is now in eax (which is the return register)

    ; --- Function Epilogue - Cleaning Up ---
    pop    ebp                ; Restore old base pointer
    ; "Thanks for holding our space, caller. Here's your ebp back."

    ret                       ; Return to caller
    ; Pops the return address and jumps there
    ; Return value is already in eax, so we're done!

Wait, why [ebp+8] and [ebp+12]?

Here's the stack layout when we're inside the function:

High addresses:
  [ebp+12] = second argument (b)
  [ebp+8]  = first argument (a)
  [ebp+4]  = return address (pushed by 'call')
  [ebp]    = saved ebp (pushed by prologue)

  [ebp-4]  = could be local variable (we don't have any here)

Low addresses (current top of stack):
  [esp] = top of stack

The call instruction pushes the return address (4 bytes in 32-bit), then jumps to the function. The function prologue pushes ebp (another 4 bytes). So: - First argument is at ebp + 8 (skip saved ebp and return address) - Second argument is at ebp + 12 (skip first argument too)

Why is the result in EAX?

That's just the calling convention. In 32-bit x86, return values go in EAX. Always. No exceptions. Well, unless it's a struct or something large, but for integers, EAX.

Let's trace through an actual call:

int sum = add_numbers(10, 20);

The calling code looks like:

push 20           ; Second argument (pushed first, remember stack is LIFO)
push 10           ; First argument
call add_numbers  ; Push return address, jump to function
; After function returns:
; eax contains the result (30)
mov [sum], eax    ; Store result in variable

64-bit Version (Because We Live in the Modern World):

In 64-bit, things are different - arguments go in registers first:

; 64-bit version of add_numbers
add_numbers:
    push rbp              ; Save old base pointer
    mov  rbp, rsp         ; Set new base pointer

    ; Arguments come in registers!
    mov  eax, edi         ; First argument 'a' is in RDI
    add  eax, esi         ; Second argument 'b' is in RSI
    ; Result in eax (well, rax, but eax is lower 32 bits)

    pop  rbp              ; Restore base pointer
    ret                   ; Return

The calling code:

mov edi, 10      ; First argument
mov esi, 20      ; Second argument
call add_numbers ; Call function
; Result in rax (eax)

Why this matters for reverse engineering:

When you're looking at assembly, you need to: 1. Recognize function prologue/epilogue patterns 2. Understand where arguments come from (registers in 64-bit, stack in 32-bit) 3. Know that return values are in EAX/RAX 4. Understand stack layout to find locals and arguments

Once you can read this simple function, you can read any function. It's all just variations on this theme.

Common variations you'll see:

; Function with local variables
my_function:
    push rbp
    mov  rbp, rsp
    sub  rsp, 32        ; Allocate 32 bytes for locals
    ; ... use [rbp-8], [rbp-16], etc. for locals ...
    mov  rsp, rbp       ; Clean up locals
    pop  rbp
    ret

; Function that calls other functions
my_function:
    push rbp
    mov  rbp, rsp
    ; ... do stuff ...
    call other_function ; Call another function
    ; ... more stuff ...
    pop  rbp
    ret

; Optimized function (no frame pointer!)
; Compiler uses RSP directly (faster)
my_function:
    ; No prologue needed!
    ; Use [rsp+offset] instead of [rbp+offset]
    ; ... code ...
    ret

Pro tip: Once you recognize these patterns, reverse engineering gets way easier. You'll see push rbp; mov rbp, rsp and think "ah, function start" without even thinking about it.

Introduction to Dynamic Analysis with GDB - Your New Best Friend¶

Static analysis is great - reading code , understanding structure. But sometimes you need to see what the program actually DOES. That's where dynamic analysis comes in , and GDB (GNU Debugger) is your weapon of choice.

GDB is like... a time machine for code. You can pause execution at any point , look around , change things , then continue. It's incredibly powerful once you get the hang of it.

Essential GDB Commands - The Ones You'll Actually Use¶

Let me show you the commands that matter. There are hundreds of GDB commands , but you'll use maybe 20 regularly. Here are the essentials:

Starting GDB:

# Basic start
gdb ./my_binary

# Start with arguments
gdb --args ./my_binary arg1 arg2 arg3

# Attach to running process (super useful!)
gdb -p $(pidof my_binary)

# Load core dump
gdb ./my_binary core

Setting Breakpoints - Where to Stop:

# Break at function
(gdb) break main
(gdb) break function_name

# Break at address
(gdb) break *0x4005c0

# Break at line (if you have source)
(gdb) break file.c:42

# Conditional breakpoint (this is gold!)
(gdb) break main if argc > 1
(gdb) break malloc if $rdi > 1024

# Temporary breakpoint (auto-deletes after first hit)
(gdb) tbreak main

Running and Controlling Execution:

# Start program
(gdb) run
(gdb) run arg1 arg2  # With arguments

# Continue execution
(gdb) continue       # Or just 'c'
(gdb) continue 5     # Continue and ignore breakpoint 5 times

# Step through code
(gdb) step           # Step into function calls (s)
(gdb) stepi          # Step one instruction (si)
(gdb) next           # Step over function calls (n)
(gdb) nexti          # Step over one instruction (ni)
(gdb) finish         # Run until function returns (fin)

Examining State - What's Going On:

# Registers
(gdb) info registers  # All registers (i r)
(gdb) p $rax          # Print specific register
(gdb) p $rsp          # Stack pointer value

# Memory
(gdb) x/20gx $rsp     # Examine 20 8-byte values in hex
(gdb) x/10i $rip      # Examine 10 instructions
(gdb) x/s $rdi        # Examine as string
(gdb) x/wx 0x601000   # Examine word at address

# Variables (if debug symbols present)
(gdb) print variable_name
(gdb) print *pointer
(gdb) print array[5]

Understanding the 'x' Command - Your Memory Inspector:

The x command is confusing at first , but super powerful:

x/[count][format][size] [address]

Format:
  o = octal
  x = hex
  d = decimal
  u = unsigned decimal
  t = binary
  c = character
  s = string
  i = instruction

Size:
  b = byte
  h = halfword (2 bytes)
  w = word (4 bytes)
  g = giant (8 bytes)

Examples:

(gdb) x/20gx $rsp     # 20 8-byte values in hex from stack pointer
(gdb) x/10i $rip      # 10 instructions from instruction pointer
(gdb) x/s 0x400500    # String at address
(gdb) x/16bx $rdi     # 16 bytes in hex from RDI
(gdb) x/wd 0x601000   # 1 word (4 bytes) in decimal

Viewing Code:

# Disassemble current function
(gdb) disas
(gdb) disas main      # Disassemble specific function
(gdb) disas 0x4005c0 0x400600  # Disassemble address range

# Assembly around current instruction
(gdb) x/10i $rip      # 10 instructions around RIP

# With source (if available)
(gdb) list            # Show source code
(gdb) list 10,20      # Lines 10-20

Stack Traces - Where Did We Come From:

(gdb) bt               # Backtrace (backtrace, whereami)
(gdb) bt 10            # Last 10 frames
(gdb) frame 2          # Switch to frame 2 (f 2)
(gdb) info frame       # Info about current frame
(gdb) up               # Move up stack
(gdb) down             # Move down stack

Modifying Execution - Change Things on the Fly:

# Change register values
(gdb) set $rax = 42
(gdb) set $rdi = 0xdeadbeef

# Change memory
(gdb) set {int}0x601000 = 123
(gdb) set {char[10]}0x601000 = "hello"

# Modify variables (if debug symbols)
(gdb) set variable = 100

Practical GDB Session Example:

Here's what a typical reverse engineering session looks like:

$ gdb ./mystery_binary
(gdb) break main
Breakpoint 1 at 0x4005c0
(gdb) run
Starting program: ./mystery_binary

Breakpoint 1, 0x00000000004005c0 in main ()
(gdb) disas
Dump of assembler code for function main:
=> 0x00000000004005c0 <+0>:     push   rbp
   0x00000000004005c1 <+1>:     mov    rbp,rsp
   0x00000000004005c4 <+4>:     mov    edi,0x4006a4
   0x00000000004005c9 <+9>:     call   0x400490 <puts@plt>
...
(gdb) x/s 0x4006a4
0x4006a4:       "Hello, World!"
(gdb) continue
Continuing.
Hello, World!
[Inferior 1 (process 12345) exited normally]

Analyzing Crashes - When Things Go Wrong:

This is where GDB really shines. When a program crashes , GDB can tell you exactly what happened.

# Enable core dumps
ulimit -c unlimited

# Run program (let it crash)
./my_binary

# Load crash dump
gdb ./my_binary core

# GDB will automatically show you:
# - Where it crashed (instruction)
# - Signal that killed it (SIGSEGV, etc.)
# - Register state
# - Stack trace

# Then investigate:
(gdb) info registers
(gdb) x/20gx $rsp      # See stack
(gdb) bt               # See call stack
(gdb) x/i $rip         # See instruction that crashed

Understanding the Crash:

When you see:

Program received signal SIGSEGV, Segmentation fault.
0x00000000004005c0 in main ()

It means: - SIGSEGV = segmentation fault (accessed invalid memory) - 0x4005c0 = address where it crashed - main() = function where it crashed

Then investigate:

(gdb) x/i $rip         # What instruction?
(gdb) info registers     # What were the register values?
(gdb) x/20gx $rsp       # What's on the stack?
(gdb) bt                # How did we get here?

GDB Scripting - Automate the Boring Stuff:

You can write scripts to automate GDB:

# Create .gdbinit file
cat > .gdbinit << 'EOF'
define myfunc
  printf "RAX: 0x%x\n", $rax
  printf "RDI: 0x%x\n", $rdi
  x/10i $rip
end
EOF

# Now you can use:
(gdb) myfunc

Pro Tips That Will Save Your Sanity:

Use aliases: define s = stepi saves typing
Set disassembly flavor: set disassembly-flavor intel (Intel syntax is more readable)
Use GEF or Peda: Enhanced GDB with better visuals (gef, peda)
History: Use up/down arrows to repeat commands
Tab completion: GDB has it! Type info reg and press tab
Help: help command shows help for any command

Common Mistakes (I've Made These):

Forgetting arguments: run doesn't use command line args , use run arg1 arg2
Stepping too far: Use finish to get out of functions you accidentally stepped into
Getting lost: bt and frame N to navigate the call stack
Assembly confusion: set disassembly-flavor intel makes it way clearer

The Reality:

GDB has a learning curve. It's not the friendliest tool. But once you get comfortable with it , there's no substitute. It's like learning to drive stick - annoying at first , but then you wonder how people drive automatics.

Start simple: set breakpoints , examine registers , step through code. Build from there. Before you know it , you'll be debugging like a pro.

Memory Organization - Where Your Data Actually Lives¶

Memory layout might seem boring , but understanding it is crucial for reverse engineering. When you see an address like 0x601000 , you need to know what that means. Is it code? Data? Stack? Heap? Let's figure it out.

Memory Layout (Linux Process) - The Big Picture¶

Every Linux process gets its own virtual address space. Think of it like your own private universe - your process thinks it owns all of memory , but really it's sharing physical RAM with other processes. The OS handles the translation.

Here's what that address space looks like:

High addresses (0x7fffffffffff):
┌─────────────────────────────────┐
│   Kernel Space                   │  ← You can't touch this (OS stuff)
│   (Not accessible to user)       │
├─────────────────────────────────┤
│   Stack                          │  ← Grows DOWNWARD (weird, I know)
│   - Local variables              │    Starts high, grows down
│   - Function parameters           │    RSP points here
│   - Return addresses             │
│   - Temporary data               │
├─────────────────────────────────┤
│   (unused/gap)                   │
├─────────────────────────────────┤
│   Heap                           │  ← Grows UPWARD
│   - malloc() allocations         │    Starts low in heap area
│   - Dynamic memory               │    Grows up toward stack
│   - Your custom allocations      │
├─────────────────────────────────┤
│   BSS (Block Started by Symbol) │  ← Uninitialized globals
│   - Global variables = 0         │    All zeros initially
│   - Static uninitialized        │
├─────────────────────────────────┤
│   Data                           │  ← Initialized globals
│   - Global variables with values │    Your constants, strings
│   - Static variables             │
├─────────────────────────────────┤
│   Text/Code                      │  ← Your executable code
│   - Machine instructions         │    Read-only, executable
│   - Function code                │    RIP reads from here
│   - Constants (sometimes)        │
└─────────────────────────────────┘
Low addresses (0x400000):

The Stack - Your Function's Workspace

The stack is like a scratchpad. Every time you call a function , it gets its own area on the stack (a "stack frame") for local variables and arguments.

void my_function(int x, int y) {
    int local1 = 10;      // Lives on stack
    int local2 = 20;      // Lives on stack
    char buffer[100];     // Lives on stack
    // When function returns, all this is gone
}

Key points: - Grows DOWNWARD (high addresses to low) - Fast (just move RSP) - Limited size (usually 8MB default) - Automatic cleanup (when function returns) - Stack overflow = crash (you ran out of stack space)

The Heap - Dynamic Memory

The heap is where you put things that need to stick around or vary in size.

void my_function() {
    int *array = malloc(1000 * sizeof(int));  // Lives on heap
    // This memory stays until you free() it
    // Even after function returns!
    free(array);
}

Key points: - Grows UPWARD (low addresses to high) - Flexible size (can grow/shrink) - Slower than stack (involves system calls) - Must manually free (memory leaks if you forget!) - Can fragment (gaps between allocations)

BSS and Data Sections - Global Variables

int global_uninitialized;        // Goes in BSS (starts as 0)
int global_initialized = 42;     // Goes in Data section
static int static_var = 100;     // Goes in Data section
const char *message = "Hello";  // Goes in Data section

BSS: - Block Started by Symbol - Uninitialized global variables - All zeros by default - Doesn't take space in executable (just a note "I need this much")

Data: - Initialized global/static variables - Actual values stored in executable - Read-write (can modify globals)

Text/Code Section - Your Instructions

This is where your actual code lives. The CPU reads instructions from here.

int main() {
    return 42;  // This code lives in .text section
}

Key points: - read only (usually , unless self-modifying code) - Executable (CPU runs instructions from here) - Shared between processes (if same binary loaded) - RIP reads from here

How to See This in Action:

# View process memory layout
cat /proc/$(pidof my_program)/maps

# Output looks like:
# 00400000-00401000 r-xp 00000000 08:01 123456 /path/to/binary  (Text/Code)
# 00600000-00601000 r--p 00000000 08:01 123456 /path/to/binary  (Data)
# 00601000-00602000 rw-p 00001000 08:01 123456 /path/to/binary  (BSS)
# 7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0               (Stack)
# 7ffff7ffd000-7ffff7ffe000 rw-p 00000000 00:00 0               (Heap)

# Or in GDB:
(gdb) info proc mappings

Reading the Maps:

First column: Address range
Second column: Permissions (r=read , w=write , x=execute , p=private)
Third column: Offset in file
Fourth column: Device
Fifth column: Inode
Last column: File/description

Practical Reverse Engineering:

When you see an address in assembly:

mov rax, [0x601000]  ; What is this?

Check the maps: - 0x400000-0x401000 = Code section (instructions) - 0x600000-0x601000 = Data section (global variables) - 0x601000-0x602000 = BSS section (uninitialized globals) - 0x7fff... = Stack area - Heap addresses vary

So 0x601000 is probably a global variable in the BSS section.

Virtual Memory - The Illusion That Makes Everything Work¶

Here's the thing: when your program accesses memory at address 0x400000 , that's not actually where it is in physical RAM. The OS creates an illusion - virtual memory.

How It Works:

Your Program Sees:
Virtual Address: 0x400000 → "This is my code"

OS Translation Layer:
Virtual 0x400000 → Physical 0x12345000 (actual RAM)

CPU:
Reads from Physical 0x12345000 → Gets your code

Why This Exists:

Isolation: Each process thinks it owns all memory
Security: Processes can't access each other's memory
Simplicity: Programs don't need to know where memory actually is
Efficiency: Physical memory can be swapped to disk

Pages - The Building Blocks:

Memory is divided into pages (usually 4KB on x86/x64):

Address Space:
┌─────────┬─────────┬─────────┐
│  Page 1 │  Page 2 │  Page 3 │  (Each is 4KB)
└─────────┴─────────┴─────────┘

Pages can be: - Present: In physical RAM - Swapped: On disk (slow to access) - Not present: Never allocated

Page Tables - The Translation Book:

The OS maintains page tables that map virtual addresses to physical addresses. The CPU's MMU (Memory Management Unit) uses these to translate addresses automatically.

You don't need to understand page tables deeply for reverse engineering , but know that: - Each process has its own page tables - Translation happens automatically - Virtual addresses you see in assembly are per-process

Address Space Layout Randomization (ASLR) - Why Addresses Change

ASLR randomizes where things are loaded in memory. This means:

# Run 1:
$ ./binary
main() is at 0x4005c0

# Run 2:
$ ./binary
main() is at 0x4012a0  # Different address!

How ASLR Works:

OS randomizes base address of code , stack , heap , libraries
Makes addresses unpredictable between runs
Each process gets different random offsets

Checking ASLR:

# Check if ASLR is enabled
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled
# 1 = conservative (stacks , mmaps)
# 2 = full (everything randomized)

# View actual addresses
cat /proc/$(pidof program)/maps

For Reverse Engineering:

With ASLR: Addresses change each run (use offsets , not absolute addresses)
Without ASLR: Addresses are consistent (easier to debug)
PIE (Position Independent Executable): Code can load at any address

Practical Impact:

When debugging: - Use offsets from base address , not absolute addresses - Or disable ASLR: echo 0 | sudo tee /proc/sys/kernel/randomize_va_space - Or use GDB (which disables ASLR for debugged processes)

Pro Tips:

Memory regions tell you what you're looking at:
Code section = instructions
Data section = initialized globals
BSS = uninitialized globals
Stack = local variables
Heap = dynamic allocations
Permissions matter:
r-x = read , execute (code)
rw- = read , write (data , stack , heap)
r-- = read only (constants)
Use /proc/PID/maps to understand layout:
```
cat /proc/$(pidof program)/maps | less
```
In GDB , use info proc mappings for the same info

Understanding memory layout helps you understand what addresses mean when you're reverse engineering. It's one of those foundational things that makes everything else make sense.

Assembly Language Basics - Learning to Read the Language of the CPU¶

Okay , time to learn assembly. Don't panic. It's not as hard as people make it sound. Think of it like learning a new language - at first everything looks like gibberish , but once you know the words (instructions) , it starts making sense.

Assembly is just a human-readable representation of machine code. Instead of 0x48 0x89 0xc3 (raw bytes) , we write mov rbx, rax (assembly). Much better , right?

Instruction Format - The Grammar Rules¶

Every assembly instruction follows a pattern:

[label:] mnemonic [operands] [; comment]

Breaking it down: - Label: A name for this location (optional). Other code can jump here. - Mnemonic: The actual instruction name (like mov, add, jmp) - Operands: What the instruction operates on (registers, memory, constants) - Comment: Human-readable notes (starts with ;)

Examples:

main:              ; Label named "main"
    mov rax, 5      ; Mnemonic: mov, Operands: rax and 5
    add rax, 10     ; Add 10 to rax
    ret             ; Return (no operands needed)

; Comments can be on their own line too
; This does nothing, just a note

The beauty of assembly: It's straightforward. mov rax, 5 means "move the value 5 into register rax." No ambiguity , no hidden behavior. What you see is what you get.

Data Movement Instructions - Moving Stuff Around¶

This is the most common thing you'll see. Programs are constantly moving data from place to place.

Register to Register:

mov rdx, rax    ; Copy rax into rdx
                ; After: rdx = value of rax, rax unchanged

Immediate to Register:

mov rax, 0xdeadbeef  ; Put constant into register
mov rax, 42          ; Decimal works too
mov rax, -10         ; Negative numbers work

Memory to Register (Loading):

mov rax, [rbx]      ; Load value from memory address in rbx
                    ; Like: rax = *rbx in C

mov rax, [rbx+8]    ; Load from rbx + 8
                    ; Like: rax = *(rbx + 8) in C

mov rax, [rbp-16]   ; Load from stack (local variable)
                    ; Like: rax = local_var in C

Register to Memory (Storing):

mov [rbx], rax      ; Store rax into memory at address in rbx
                    ; Like: *rbx = rax in C

mov [rbp-8], rax    ; Store into stack (local variable)
                    ; Like: local_var = rax in C

Important note about brackets: - mov rax, rbx = move register to register - mov rax, [rbx] = move value FROM memory address in rbx - mov [rbx], rax = move value TO memory address in rbx

The brackets mean "dereference this address." Like pointers in C.

Common mistake: Forgetting brackets when you need them , or including them when you don't. This is assembly's version of segfault city.

Arithmetic Instructions - Math Time¶

The CPU can do math. Surprise! Here's how:

Basic Arithmetic:

add rax, rbx        ; rax = rax + rbx
                    ; Add rbx to rax, store result in rax

sub rax, rbx        ; rax = rax - rbx
                    ; Subtract rbx from rax

mul rbx             ; rdx:rax = rax * rbx (unsigned)
                    ; Result is 128 bits! High bits in rdx, low in rax
                    ; Warning: rdx gets overwritten!

div rbx             ; rax = (rdx:rax) / rbx, rdx = remainder
                    ; Dividend must be in rdx:rax
                    ; Quotient in rax, remainder in rdx
                    ; Both rdx and rax get overwritten!

Multiplication and division are weird: - mul and div use specific registers - They're destructive (overwrite registers) - They work with 128-bit values (rdx:rax)

Increment/Decrement:

inc rax             ; rax++ (add 1)
dec rax             ; rax-- (subtract 1)

; These are faster than add/sub with 1
; Compilers use them a lot

Negation:

neg rax             ; rax = -rax
                    ; Two's complement negation
                    ; Like: rax = 0 - rax

Logic Instructions - Bit Twiddling¶

These work on individual bits. Super useful for flags , masks , and bit manipulation.

Bitwise Operations:

and rax, rbx        ; rax = rax & rbx (bitwise AND)
                    ; Both bits must be 1 for result to be 1

or rax, rbx         ; rax = rax | rbx (bitwise OR)
                    ; Either bit can be 1 for result to be 1

xor rax, rbx        ; rax = rax ^ rbx (bitwise XOR)
                    ; Bits differ = 1, same = 0
                    ; Note: xor rax, rax = 0 (fast way to zero register!)

not rax             ; rax = ~rax (bitwise NOT)
                    ; Flip all bits

The test Instruction - Checking Without Changing:

test rax, rax       ; Like "and rax, rax" but doesn't store result
                    ; Only sets flags
                    ; Commonly used: test rax, rax sets ZF if rax is zero
                    ; Equivalent to: if (rax == 0)

test rax, 1         ; Check if lowest bit is set (odd number)
                    ; Sets ZF if bit is 0

Shifts - Moving Bits Around:

shl rax, 1          ; Shift left by 1 (multiply by 2)
                    ; Like: rax = rax << 1
                    ; Leftmost bit goes to CF, rightmost becomes 0

shr rax, 1          ; Shift right by 1 (logical, unsigned divide by 2)
                    ; Like: rax = rax >> 1 (unsigned)
                    ; Rightmost bit goes to CF, leftmost becomes 0

sar rax, 1          ; Shift right by 1 (arithmetic, signed divide by 2)
                    ; Like: rax = rax >> 1 (signed)
                    ; Sign bit is preserved (rightmost goes to CF)
                    ; Leftmost bit stays same (sign extension)

sal rax, 1          ; Shift arithmetic left (same as shl)

Why shifts matter: - shl rax, 1 = multiply by 2 (faster than mul) - shr rax, 1 = divide by 2 (faster than div) - shl rax, 3 = multiply by 8 (2^3)

Compilers love using shifts for powers of 2. It's way faster.

Control Flow Instructions - Making Decisions¶

This is where assembly gets interesting. Without control flow , code would just run straight through. Boring.

Unconditional Jumps:

jmp label           ; Jump to label (always happens)
                    ; Like: goto label in C

jmp rax             ; Jump to address in rax
                    ; Indirect jump (where does rax point?)
                    ; Common in function pointers, switch statements

Conditional Jumps - The Decision Makers:

These jump based on flags set by previous instructions:

# Equality checks
je label            ; Jump if equal (ZF == 1)
jne label           ; Jump if not equal (ZF == 0)

# Signed comparisons (for signed integers)
jg label            ; Jump if greater (signed)
jge label           ; Jump if greater or equal (signed)
jl label            ; Jump if less (signed)
jle label           ; Jump if less or equal (signed)

# Unsigned comparisons (for unsigned integers)
ja label            ; Jump if above (unsigned)
jae label           ; Jump if above or equal (unsigned)
jb label            ; Jump if below (unsigned)
jbe label           ; Jump if below or equal (unsigned)

How They Work:

cmp rax, rbx        ; Compare rax with rbx, sets flags
je label            ; If equal (ZF set), jump to label
                    ; Otherwise, continue to next instruction

# Equivalent to:
# if (rax == rbx) goto label;

Common Pattern:

cmp rax, 0          ; Compare with zero
je zero_case        ; If zero, handle zero case
                    ; Otherwise continue...

Function Calls:

call function       ; Call a function
                    ; 1. Push return address (next instruction)
                    ; 2. Jump to function
                    ; When function returns, execution continues here

ret                 ; Return from function
                    ; Pop return address from stack
                    ; Jump to that address

The Call/Ret Dance:

call my_function    ; Push address of "mov rax, 0" onto stack
                    ; Jump to my_function
mov rax, 0          ; When my_function returns, execution continues here

my_function:
    ; Do stuff
    ret             ; Pop address from stack, jump there
                    ; This brings us back to "mov rax, 0"

Stack Operations - The Filing System¶

The stack is super important. Here's how you use it:

Push/Pop:

push rax            ; Push rax onto stack
                    ; Equivalent to:
                    ;   sub rsp, 8
                    ;   mov [rsp], rax
                    ; But push is faster (single instruction)

pop rbx             ; Pop from stack into rbx
                    ; Equivalent to:
                    ;   mov rbx, [rsp]
                    ;   add rsp, 8
                    ; But pop is faster

Stack Pointer Manipulation:

sub rsp, 16         ; Allocate 16 bytes on stack
                    ; Make space for local variables

add rsp, 16         ; Deallocate 16 bytes
                    ; Clean up local variables

Why sub/add instead of just push/pop? - When you need multiple bytes (like a struct or array) - When you want to allocate once , use multiple times - More efficient than pushing individual values

Complete Function Example:

my_function:
    ; Prologue
    push rbp        ; Save caller's base pointer
    mov rbp, rsp    ; Set our base pointer
    sub rsp, 32     ; Allocate 32 bytes for locals

    ; Function body
    mov [rbp-8], 10     ; Local variable 1
    mov [rbp-16], 20    ; Local variable 2
    ; ... do work ...

    ; Epilogue
    mov rsp, rbp    ; Deallocate locals
    pop rbp         ; Restore caller's base pointer
    ret             ; Return

Pro Tips for Reading Assembly: 1. Follow the data: Where does data come from? Where does it go? 2. Recognize patterns: Function prologues , loops , conditionals 3. Understand the stack: Locals are negative offsets from RBP , arguments are positive 4. Watch flags: cmp and test set flags for conditional jumps 5. Track registers: Which register holds what value?

Common Patterns You'll See:

Loop Pattern:

mov rcx, 10         ; Counter
loop_start:
    ; ... loop body ...
    dec rcx         ; Decrement counter
    jnz loop_start  ; Jump if not zero

If/Else Pattern:

cmp rax, 0
je if_zero          ; If zero
                    ; Else case here
jmp done
if_zero:
    ; Zero case here
done:

Switch Statement Pattern:

cmp rax, 0
je case0
cmp rax, 1
je case1
cmp rax, 2
je case2
jmp default

Or with jump table:

jmp [rax*8 + jump_table]  ; Indirect jump through table

The Reality: Assembly isn't magic. It's just instructions. Each one does one thing. When you put them together , you get programs. Start by understanding individual instructions , then see how they combine into patterns. Before you know it , you'll be reading assembly like it's Python (okay , maybe not that easy , but you'll get there).

Function Calls and Stack Frames - The House of Cards¶

Functions are the building blocks of programs. Understanding how they work is crucial. Let me break down what actually happens when you call a function - it's more interesting than you'd think.

Calling Convention - The Rules Everyone Follows¶

A calling convention is like a contract. "Hey function , I'm going to put your arguments HERE , and you're going to put your return value THERE. Agreed?" Everyone follows these rules , or everything breaks.

The Function Call Sequence - Step by Step:

Let's say you have:

int result = add_numbers(10, 20);

Here's what actually happens:

; Step 1: Prepare arguments
mov rdi, 10         ; First argument goes in RDI (64-bit System V)
mov rsi, 20         ; Second argument goes in RSI

; Step 2: Call the function
call add_numbers    ; This does TWO things:
                    ;   1. Push return address (address of next instruction)
                    ;   2. Jump to add_numbers

; Step 3: (After function returns)
; Result is now in RAX
mov [result], rax   ; Store result in variable

What call Actually Does:

The call instruction is sneaky - it does two things: 1. Pushes return address: Saves where to come back to 2. Jumps: Goes to the function

It's like leaving a breadcrumb trail so you can find your way back.

call function
; Is equivalent to:
push $+5            ; Push address of next instruction
jmp function        ; Jump to function

Inside the Function - Setting Up Shop:

When the function starts , it needs to set up its own workspace:

add_numbers:
    ; PROLOGUE - Setting up our workspace
    push rbp            ; Save caller's base pointer
                        ; "Hey caller , I'm borrowing your base pointer location"
                        ; "I'll give it back when I'm done , promise"

    mov rbp, rsp        ; Set our own base pointer
                        ; "This is OUR workspace now"

    sub rsp, 16         ; Allocate space for local variables
                        ; "I need 16 bytes for my stuff"
                        ; (RSP moves down , stack grows)

    ; FUNCTION BODY - Do the actual work
    mov eax, edi        ; Load first argument (from RDI)
    add eax, esi        ; Add second argument (from RSI)
                        ; Result is in EAX (return register)

    ; EPILOGUE - Clean up and leave
    mov rsp, rbp        ; Deallocate locals
                        ; "Clean up our workspace"

    pop rbp             ; Restore caller's base pointer
                        ; "Here's your base pointer back , caller"

    ret                 ; Return to caller
                        ; "Pop return address , jump back home"

Why This Prologue/Epilogue Pattern?

Every function follows this pattern. It's standardized. Here's why:

Saving RBP: The caller might be using RBP. We can't just overwrite it. So we save it , use it , then restore it.
Setting our RBP: Having a stable reference point makes accessing arguments and locals easy.
Allocating space: Local variables need somewhere to live. The stack is perfect for this.

Stack Frame Layout - What's Actually in Memory¶

When you're inside a function , the stack looks like a layered cake. Let me show you exactly what's where:

High addresses (what was pushed first):
┌─────────────────────────────────────┐
│   [rbp+24] - 4th argument (if any)  │  ← Arguments
│   [rbp+16] - 3rd argument           │
│   [rbp+8]  - 2nd argument            │
│   [rbp]    - Saved RBP              │  ← From caller's frame
│   [rbp-8]  - 1st local variable     │  ← Our locals
│   [rbp-16] - 2nd local variable     │
│   [rbp-24] - 3rd local variable     │
│   ...                                │
│   [rsp]    - Top of stack (current) │  ← Where RSP points
└─────────────────────────────────────┘
Low addresses (what was pushed last):

Wait , where's the return address?

Good catch! The return address is between the arguments and the saved RBP , but in 64-bit with registers for arguments , you might not see it on the stack unless there are more than 6 arguments.

In 32-bit (stack-based arguments):

High:
  [ebp+12] - 2nd argument
  [ebp+8]  - 1st argument
  [ebp+4]  - Return address (pushed by call)
  [ebp]    - Saved EBP (pushed by prologue)
  [ebp-4]  - 1st local
Low:

In 64-bit (register-based arguments):

High:
  [rbp]    - Saved RBP
  [rbp-8]  - 1st local
  [rbp-16] - 2nd local
  ...
  [rsp]    - Top of stack
Low:

(Return address is on stack but you don't access it directly - ret handles it)

Practical Example - Tracing Through a Call:

Let's trace through calling printf("Hello , %d\n" , 42):

; Calling code prepares arguments
lea rdi, [format_string]  ; First arg: format string pointer
mov esi, 42               ; Second arg: integer to print
call printf               ; Call printf

; Stack just before call:
;   ... (caller's stack frame) ...
;   (return address will go here)

; Inside printf (simplified):
printf:
    push rbp              ; Save caller's RBP
    mov  rbp, rsp         ; Set our RBP
    sub  rsp, 32          ; Allocate locals (printf needs space for formatting)

    ; RDI = format string pointer
    ; RSI = 42
    ; ... do formatting work ...

    mov rsp, rbp          ; Clean up
    pop rbp               ; Restore caller's RBP
    ret                   ; Return to caller

Understanding the Stack Frame:

The stack frame is like a function's personal workspace. It contains: - Arguments: Passed from caller - Return address: Where to go back to - Saved registers: Any registers we need to preserve - Local variables: Our temporary data - Temporary space: For calculations

real world Reverse Engineering:

When you see assembly like:

mov rax, [rbp+8]    ; You know: accessing first argument
mov [rbp-16], rax   ; You know: storing to local variable

This pattern recognition is gold. Once you see it , you can quickly understand: - What arguments functions take - What local variables they use - How they manipulate data

Common Variations:

Function with no locals:

simple_function:
    push rbp
    mov  rbp, rsp
    ; No sub rsp needed - no locals!
    ; ... work ...
    pop  rbp
    ret

Function with many arguments (more than 6 in 64-bit):

; First 6 arguments in registers (RDI , RSI , RDX , RCX , R8 , R9)
; 7th+ arguments on stack
my_function:
    push rbp
    mov  rbp, rsp
    mov  rax, [rbp+16]   ; 7th argument (skip saved RBP and return address)
    ; ... work ...
    pop  rbp
    ret

Optimized function (no frame pointer):

optimized_function:
    ; No prologue! Faster but harder to debug
    ; Use [rsp+offset] directly
    mov rax, [rsp+8]      ; Access argument
    ; ... work ...
    ret

Pro Tips:

Function prologues are your friends: They mark function boundaries clearly
Negative offsets = locals: [rbp-8] is almost always a local variable
Positive offsets = arguments: [rbp+8] is often an argument
RSP changes: It moves around during function execution , RBP stays stable
Epilogue mirrors prologue: Whatever you push , you pop. Whatever you allocate , you deallocate.

The Beauty of This System:

Every function is independent. It has its own stack frame , its own locals , its own space. Functions don't interfere with each other. It's elegant , really. And once you understand it , reverse engineering gets way easier because you can follow the data flow through function calls.

System Calls - Talking to the Operating System¶

Alright , here's where programs interact with the outside world. Your program runs in user space , but sometimes it needs to do things that only the OS can do - read files , write to network , create processes. That's what system calls are for.

Think of it like this: Your program is a customer in a restaurant (user space). The OS is the kitchen (kernel space). You can't go into the kitchen yourself , but you can ask the waiter (system call) to bring you food (operate on files , network , etc.).

Linux System Calls - The Interface¶

System calls are the only way your program talks to the kernel. Everything else - file I/O , network , process management - goes through system calls.

How System Calls Work:

Put syscall number in RAX: Tells kernel what you want
Put arguments in registers: RDI , RSI , RDX , RCX , R8 , R9
Execute syscall instruction: Hands control to kernel
Kernel does the work: Performs the operation
Return value in RAX: Success/error code

The syscall Instruction:

mov rax, 1          ; Syscall number 1 = write
mov rdi, 1          ; First arg: file descriptor (1 = stdout)
mov rsi, message    ; Second arg: buffer pointer
mov rdx, len        ; Third arg: length
syscall             ; Make the system call
                    ; Kernel takes over, does the work
                    ; Returns with result in RAX

Important System Call Numbers (x86-64):

# These are the numbers you'll see in RAX before syscall:
0   - read          # Read from file descriptor
1   - write         # Write to file descriptor
2   - open          # Open a file
3   - close         # Close a file descriptor
4   - stat          # Get file status
9   - mmap          # Map memory
12  - brk           # Change data segment size
57  - fork          # Create child process
59  - execve        # Execute program
60  - exit          # Exit process
61  - wait4         # Wait for child process
62  - kill          # Send signal to process

You don't need to memorize all of these. Just know that: - Low numbers (< 100) = common operations - The exact number depends on architecture and OS version - Tools like strace will show you the syscall names

Common System Calls - The Ones You'll See Most¶

File Operations - Reading and Writing:

# open() - Open a file
mov rax, 2          ; open syscall
lea rdi, [filename] ; First arg: pathname
mov rsi, 0          ; Second arg: flags (O_RDONLY = 0)
mov rdx, 0          ; Third arg: mode (not needed for read-only)
syscall
; Returns file descriptor in RAX (or -1 on error)

# read() - Read from file
mov rax, 0          ; read syscall
mov rdi, fd         ; First arg: file descriptor
mov rsi, buffer     ; Second arg: buffer pointer
mov rdx, size       ; Third arg: bytes to read
syscall
; Returns bytes read in RAX (or -1 on error)

# write() - Write to file
mov rax, 1          ; write syscall
mov rdi, fd         ; First arg: file descriptor (1 = stdout)
mov rsi, buffer     ; Second arg: buffer pointer
mov rdx, size       ; Third arg: bytes to write
syscall
; Returns bytes written in RAX (or -1 on error)

# close() - Close file descriptor
mov rax, 3          ; close syscall
mov rdi, fd         ; First arg: file descriptor
syscall
; Returns 0 on success, -1 on error

Complete Example - Writing "Hello , World!\n":

section .data
    message db 'Hello , World!' , 10  ; String + newline
    len equ $ - message              ; Length calculation

section .text
    global _start

_start:
    ; write(stdout , message , len)
    mov rax, 1          ; write syscall
    mov rdi, 1          ; stdout
    mov rsi, message    ; Message pointer
    mov rdx, len        ; Message length
    syscall

    ; exit(0)
    mov rax, 60         ; exit syscall
    mov rdi, 0          ; Exit code 0
    syscall

Process Operations - Creating and Managing Processes:

# fork() - Create child process
mov rax, 57         ; fork syscall
syscall
; Returns: In parent = child PID , In child = 0 , Error = -1

# execve() - Execute program
mov rax, 59         ; execve syscall
lea rdi, [path]     ; First arg: pathname
lea rsi, [argv]     ; Second arg: argument array
lea rdx, [envp]     ; Third arg: environment array
syscall
; Never returns if successful (replaces current process)
; Returns -1 on error

# exit() - Terminate process
mov rax, 60         ; exit syscall
mov rdi, 0          ; Exit code
syscall
; Process terminates , never returns

Why This Matters for Reverse Engineering:

When you see syscall in assembly , you know the program is: - Interacting with the OS - Doing I/O operations - Creating processes - Something "real world"

Identifying System Calls:

# Pattern to recognize:
mov rax, SYSCALL_NUMBER
mov rdi, arg1
mov rsi, arg2
mov rdx, arg3
syscall
; Result in rax

Using strace to See System Calls:

Instead of reading assembly , you can just watch what syscalls a program makes:

# See all system calls
strace ./program

# Filter for specific syscalls
strace -e trace=open , read , write ./program

# Save to file
strace -o trace.log ./program

real world Example:

When you run a program , here's what system calls it might make:

$ strace ./hello_world
execve("./hello_world" , ["./hello_world"] , ...) = 0
brk(NULL)                   = 0x1234000
access("/etc/ld.so.preload" , R_OK) = -1 ENOENT
openat(AT_FDCWD , "/lib/x86_64-linux-gnu/libc.so.6" , O_RDONLY|O_CLOEXEC) = 3
read(3 , "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\205\2\0\0\0\0\0"..., 832) = 832
mmap(NULL , 2031616 , PROT_READ|PROT_EXEC , MAP_PRIVATE|MAP_DENYWRITE , 3 , 0) = 0x7f1234567890
...
write(1 , "Hello , World!\n" , 14) = 14
exit_group(0)                    = ?
+++ exited with 0 +++

Understanding the Output:

execve: Load the program
openat: Open libraries
mmap: Map libraries into memory
write: Actually write "Hello , World!"
exit_group: Exit program

Common Syscall Patterns You'll See:

File Reading Pattern:

mov rax, 2          ; open
lea rdi, [filename]
mov rsi, 0
syscall
mov r12, rax        ; Save file descriptor

mov rax, 0          ; read
mov rdi, r12
mov rsi, buffer
mov rdx, 1024
syscall

mov rax, 3          ; close
mov rdi, r12
syscall

Network Operations:

# socket() - Create socket
mov rax, 41         ; socket syscall
mov rdi, 2          ; AF_INET
mov rsi, 1          ; SOCK_STREAM
mov rdx, 0          ; Protocol
syscall
mov r12, rax        ; Save socket fd

# connect() - Connect to server
mov rax, 42         ; connect syscall
mov rdi, r12        ; Socket fd
lea rsi, [sockaddr] ; Address structure
mov rdx, 16         ; Address length
syscall

# Then read/write on socket fd

Pro Tips:

Syscall numbers vary: x86-64 has different numbers than x86-32. Don't hardcode them.
Use strace: Much easier than reading assembly to understand what a program does
Error handling: Most syscalls return -1 on error , check errno for details
File descriptors: 0 = stdin , 1 = stdout , 2 = stderr , 3+ = opened files
Syscall vs libc: Most programs use libc functions (printf , fopen) which internally call syscalls

The Reality:

You won't see raw syscalls in most binaries - they use libc functions. But understanding syscalls helps you understand what those libc functions actually do. And sometimes you'll see syscalls directly (especially in malware , hand-written assembly , or optimized code).

Quick Reference:

Want to see what syscalls a program makes? strace
Want to see what libraries it uses? ltrace
Want to understand syscall numbers? /usr/include/x86_64-linux-gnu/asm/unistd_64.h
Want to see syscall definitions? man 2 syscall_name

System calls are the bridge between user programs and the kernel. Understanding them helps you understand what programs actually do.

ARM Architecture¶

ARM Registers¶

# General purpose registers
R0-R12 - General purpose
R13 (SP) - Stack pointer
R14 (LR) - Link register
R15 (PC) - Program counter
CPSR - Current program status register

ARM Instruction Set¶

# Data movement
MOV R0, R1          ; Move R1 to R0
LDR R0, [R1]        ; Load from memory
STR R0, [R1]        ; Store to memory

# Arithmetic
ADD R0, R1, R2      ; R0 = R1 + R2
SUB R0, R1, R2      ; R0 = R1 - R2

# Branches
B label             ; Branch to label
BL function         ; Branch with link (call)
BX LR               ; Return

ARM vs x86 - The Key Differences:

ARM is everywhere in mobile and embedded. Phones, tablets, IoT devices - they're all ARM. Understanding ARM is crucial for mobile pentesting and embedded reverse engineering.

Register Differences: - ARM has 16 registers (R0-R15) vs x86's many - No dedicated instruction pointer register (PC is R15, but you don't access it directly) - Link register (LR/R14) stores return address automatically on calls - Current Program Status Register (CPSR) holds flags and processor state

Instruction Set Differences: - ARM uses fixed 32-bit instructions (mostly) - Load/Store architecture - can't operate directly on memory - Must load data into registers first, then operate - This is called "load/store" architecture

ARM Calling Convention:

# Function call in ARM
BL function_name    ; Branch with link - saves return address in LR
; Function prologue
PUSH {R4-R7, LR}    ; Save registers and return address
; Function body
POP {R4-R7, PC}     ; Restore and return (PC = LR)

ARM Assembly Example:

; Simple function: int add(int a, int b)
add_numbers:
    ADD R0, R0, R1  ; R0 = R0 + R1 (first arg + second arg)
    BX LR           ; Return (branch to address in LR)

; Calling code:
MOV R0, #10         ; First argument
MOV R1, #20         ; Second argument
BL add_numbers      ; Call function
; Result in R0

ARM Endianness: ARM can be little-endian or big-endian, but most modern ARM is little-endian (like x86).

ARM vs Thumb Mode: ARM has two instruction sets: - ARM mode: 32-bit instructions - Thumb mode: 16-bit instructions (more compact)

You switch between them with BX instruction.

Why ARM Matters for Reverse Engineering: - Mobile apps are ARM (Android/iOS) - Embedded devices are ARM - IoT devices are ARM - Understanding ARM lets you reverse engineer the majority of consumer electronics

ARM Tools:

# Cross-compilation
sudo apt install -y gcc-arm-linux-gnueabi

# Disassemble ARM binary
arm-linux-gnueabi-objdump -d binary

# Emulate ARM code
qemu-arm-static ./arm_binary

ARM Common Patterns:

# Function prologue (ARM)
PUSH {R4-R7, LR}    ; Save registers
SUB SP, SP, #16     ; Allocate stack space

# Function epilogue
ADD SP, SP, #16     ; Deallocate stack
POP {R4-R7, PC}     ; Restore and return

# Load effective address
LDR R0, [R1, #8]    ; R0 = *(R1 + 8)

ARM is different from x86, but the concepts are the same. Once you understand registers, memory, and functions, you can reverse engineer ARM just like x86.

Endianness¶

Little Endian vs Big Endian¶

# Little endian (x86): LSB first
# 0x12345678 stored as: 78 56 34 12

# Big endian: MSB first
# 0x12345678 stored as: 12 34 56 78

# Check system endianness
#include <stdio.h>
int main() {
    int x = 1;
    char *p = (char*)&x;
    if (*p) printf("Little endian\n");
    else printf("Big endian\n");
}

Why Endianness Matters in Reverse Engineering:

When you're looking at memory or files, you need to know the endianness. Get it wrong, and you'll misinterpret all your data. It's like reading a book backwards - you'll understand the words but not the story.

x86 is Little Endian:

# Memory address: 0x1000
# Value: 0x12345678
# In memory (little endian):
# 0x1000: 78
# 0x1001: 56
# 0x1002: 34
# 0x1003: 12

ARM can be either: - Most ARM is little endian now - Some embedded systems are big endian - Always check with file command or by examining known values

Network Byte Order: - Network protocols use big endian - Called "network byte order" - Functions like htonl(), ntohl() convert between host and network byte order

Practical Example: When reversing a file format or network protocol, you might see:

# File header
# Offset 0: Magic bytes (big endian)
# Offset 4: File size (little endian on x86)

Identifying Endianness:

# Look for known values
# If you see 0x12345678 as 78 56 34 12, it's little endian
# If you see 0x12345678 as 12 34 56 78, it's big endian

# Check with hexdump
hexdump -C file | head

Common Mistakes: - Assuming everything is little endian (x86 bias) - Not checking endianness when analyzing unknown formats - Mixing endianness in the same data structure

Tools to Handle Endianness:

# Python struct module
import struct
# '<' = little endian, '>' = big endian
value = struct.unpack('<I', data)[0]  # Little endian 32-bit

# In GDB
(gdb) x/4xb address  # Examine as bytes
# Manually determine endianness

The Bottom Line: Always verify endianness when analyzing unknown data. Don't assume - check. It's saved me countless hours of confusion.

Note: For more C programming examples and best practices, refer to the C Programming Cheatsheet.

CPU Modes and Rings¶

x86 Protection Rings¶

# Ring 0: Kernel mode (highest privilege)
# Ring 1: Device drivers
# Ring 2: Device drivers
# Ring 3: User applications (lowest privilege)

# Privilege escalation
# System calls transition from ring 3 to ring 0

Why Rings Matter: In x86, the CPU has 4 privilege levels (rings). Ring 0 is the most privileged - the kernel runs here. User applications run in ring 3, the least privileged. This prevents user code from crashing the system or accessing hardware directly.

System Calls - Crossing the Boundary: When your program calls write(), it triggers a system call. The CPU switches from ring 3 to ring 0, the kernel handles the request, then switches back. This is how user programs safely access system resources.

Privilege Escalation: In security contexts, privilege escalation means getting code to run in a higher privilege ring. Exploits often aim to execute code in ring 0 (kernel mode) from ring 3 (user mode).

ARM Exception Levels¶

# EL0: User applications
# EL1: OS kernel
# EL2: Hypervisor
# EL3: Secure monitor

ARM Security Model: ARM has exception levels instead of rings. EL0 is user mode, EL1 is kernel mode. Higher levels (EL2, EL3) handle virtualization and secure boot.

TrustZone (ARM Security Extensions): ARM has TrustZone - basically two worlds: - Normal world: Regular apps and OS - Secure world: Trusted execution environment

TrustZone protects sensitive operations like cryptography, secure boot, and DRM.

Why This Matters for Reverse Engineering: Understanding privilege levels helps you understand: - What code can do what - Security boundaries - Where exploits need to go - How system calls work

Real World Impact: When reversing malware, you might see code trying to: - Escalate privileges (ring 3 → ring 0) - Bypass TrustZone protections - Execute in secure world

Checking Current Privilege Level:

# In Linux, check if you're root (effective UID 0)
id

# In kernel code, check current ring
# But you usually can't directly access this from user space

The Bottom Line: Privilege levels create security boundaries. Understanding them helps you understand what code is allowed to do and how exploits work around these restrictions.

Cache and Memory Hierarchy¶

CPU Cache Levels¶

# L1 Cache: Smallest, fastest (32KB-64KB per core)
# L2 Cache: Medium size/speed (256KB-512KB per core)
# L3 Cache: Largest, shared among cores (up to 32MB)

# Cache lines: 64 bytes typically
# Cache coherence protocols (MESI)

Why Cache Matters in Reverse Engineering: Modern CPUs have multiple cache levels. Understanding cache can help you understand performance characteristics and sometimes identify timing-based vulnerabilities.

Cache Lines: Data is moved in 64-byte chunks called cache lines. If you access one byte, the whole line gets loaded. This is why accessing nearby data is fast - it's already in cache.

Cache Coherence (MESI): - Modified: This core modified the data - Exclusive: Only this core has the data - Shared: Multiple cores have read-only copies - Invalid: Data is stale

For Reverse Engineering: Cache behavior can affect timing attacks. Some crypto implementations leak information through cache timing. Understanding cache helps you understand performance-sensitive code.

Memory Access Patterns¶

# Temporal locality: Recently accessed data likely accessed again
# Spatial locality: Nearby data likely accessed soon

# Cache-friendly code
# Process data in contiguous blocks
# Avoid cache thrashing

Temporal Locality: If you access data now, you'll likely access it again soon. Good for loops, frequently used variables.

Spatial Locality: If you access address X, you'll likely access X+1, X+2, etc. soon. Good for arrays, sequential processing.

Cache-Friendly Code Patterns:

// Good: Process array sequentially
for (int i = 0; i < 1000; i++) {
    sum += array[i];  // Accesses nearby memory
}

// Bad: Jump around randomly
for (int i = 0; i < 1000; i++) {
    sum += array[random_indices[i]];  // Cache misses everywhere
}

In Reverse Engineering: When you see code that processes data in strange patterns, it might be trying to avoid cache timing attacks or defeat prefetching. Understanding cache helps you understand why code is structured a certain way.

SIMD Instructions¶

SSE/AVX Instructions¶

# SSE (128-bit)
MOVAPS xmm0, xmm1   ; Move packed single precision
ADDPS xmm0, xmm1    ; Add packed singles

# AVX (256-bit)
VMOVAPS ymm0, ymm1  ; Move packed
VADDPS ymm0, ymm0, ymm1 ; Add packed

# Use for parallel processing
# Graphics, crypto, scientific computing

What is SIMD? SIMD (Single Instruction, Multiple Data) lets you process multiple values with one instruction. Instead of adding two numbers, you can add eight numbers at once. This is huge for performance in graphics, crypto, and scientific computing.

SSE Registers: - xmm0-xmm15: 128-bit registers (16 bytes) - Can hold 4 floats, 2 doubles, or various integer types

AVX Registers: - ymm0-ymm15: 256-bit registers (32 bytes) - Can hold 8 floats, 4 doubles, etc.

Common SIMD Operations:

# Floating point
ADDPS xmm0, xmm1    ; Add 4 floats
MULPS xmm0, xmm1    ; Multiply 4 floats
SQRTPS xmm0, xmm1   ; Square root of 4 floats

# Integer
PADDD xmm0, xmm1    ; Add 4 integers
PMULLD xmm0, xmm1   ; Multiply 4 integers

# Shuffles and permutations
SHUFPS xmm0, xmm1, 0x1B  ; Shuffle floats
PSHUFD xmm0, xmm1, 0x1B  ; Shuffle integers

Why This Matters in Reverse Engineering: When you see SIMD instructions, you know the code is optimized for performance. This often indicates: - Graphics processing (games, video) - Cryptographic operations (AES, etc.) - Scientific computing - Image processing

Identifying SIMD Usage:

# Look for SIMD instructions in disassembly
objdump -d binary | grep -E "(xmm|ymm|ps|pd|ss|sd)"

# Check for SIMD-related strings
strings binary | grep -i "simd\|sse\|avx"

# In debugger, see SIMD registers
(gdb) info registers xmm0

SIMD in Crypto: Many crypto algorithms use SIMD for parallel processing: - AES encryption can process multiple blocks - Hash functions can process multiple chunks - This makes crypto faster but also more complex to reverse

The Bottom Line: SIMD instructions show where performance matters. Understanding them helps you understand what parts of the code are critical for speed.

CPU Microarchitecture¶

Pipeline Stages¶

# Fetch: Get instruction from memory
# Decode: Interpret instruction
# Execute: Perform operation
# Memory: Access memory if needed
# Writeback: Write result to register

# Branch prediction
# Out-of-order execution
# Speculative execution

Understanding the Pipeline: Modern CPUs don't execute instructions one at a time. They use a pipeline - like an assembly line. While one instruction is being executed, the next is being decoded, and the one after that is being fetched.

Branch Prediction: CPUs guess which way branches will go. If they guess wrong, the pipeline stalls and performance drops. This is why predictable branches are important.

Out-of-Order Execution: CPU can execute instructions in different order than written, as long as the final result is correct. This maximizes performance but makes debugging harder.

Speculative Execution: CPU executes instructions before knowing if they're needed (like guessing branch outcomes). If wrong, results are discarded. This is how Spectre works.

Hazards¶

# Data hazards: Dependencies between instructions
# Control hazards: Branches
# Structural hazards: Resource conflicts

# Mitigation techniques
# Forwarding, branch prediction, etc.

Data Hazards: When instructions depend on each other's results. Like:

add rax, rbx    ; Produces result in RAX
add rcx, rax    ; Needs result from previous instruction

Control Hazards: Branches cause uncertainty. CPU doesn't know which instruction to fetch next until branch is resolved.

Structural Hazards: Two instructions need the same resource at the same time (like both needing the ALU).

Why This Matters in Reverse Engineering: Understanding microarchitecture helps you understand: - Why certain code patterns are faster - How timing attacks work (Spectre, Meltdown) - Why compilers arrange code certain ways - Performance characteristics of different algorithms

In Practice: When you see code that looks inefficient but runs fast, it's probably optimized for the pipeline. When you see code that avoids certain patterns, it might be defending against timing attacks.

Performance Considerations¶

Instruction Latency and Throughput¶

# Different instructions have different costs
# ADD: 1 cycle latency
# MUL: 3-4 cycle latency
# DIV: 20+ cycle latency

# Throughput: Instructions per cycle

Understanding Latency vs Throughput: - Latency: How long one instruction takes (cycles from start to finish) - Throughput: How many instructions you can start per cycle

Why This Matters in Reverse Engineering: When you see code that looks inefficient, it might be optimized for throughput. Understanding this helps you understand why compilers make certain choices.

Common Instruction Latencies (x86-64):

# Arithmetic
ADD/SUB: 1 cycle
MUL (32-bit): 3 cycles
MUL (64-bit): 4 cycles
DIV (32-bit): 20-30 cycles
DIV (64-bit): 40-60 cycles

# Memory operations
MOV from register: 0 cycles (register rename)
MOV from memory: 4-5 cycles (cache hit)
MOV from memory: 100+ cycles (cache miss)

# Branches
Conditional jump: 1-2 cycles (predicted)
Conditional jump: 15-20 cycles (mispredicted)

Throughput Considerations: - Modern CPUs can execute multiple instructions per cycle - But dependencies limit this - Understanding this helps explain why some code is structured oddly

Optimization Techniques¶

# Loop unrolling
# Instruction scheduling
# Register allocation
# Memory alignment

Loop Unrolling:

// Original loop
for (int i = 0; i < 100; i++) {
    sum += array[i];
}

// Unrolled (reduces loop overhead)
for (int i = 0; i < 100; i += 4) {
    sum += array[i];
    sum += array[i+1];
    sum += array[i+2];
    sum += array[i+3];
}

Instruction Scheduling: Compilers reorder instructions to avoid dependencies and maximize parallelism.

Register Allocation: Choosing which variables go in which registers to minimize memory accesses.

Memory Alignment: Data aligned to cache line boundaries (64 bytes) for faster access.

Why This Matters: Understanding optimizations helps you understand why decompiled code looks different from source. It also helps you identify hand-written assembly vs compiler-generated code.

Identifying Compiler Optimizations: - Look for unrolled loops - Check for unusual instruction ordering - See if variables are kept in registers - Notice aligned data structures

The Bottom Line: Performance considerations explain why code looks the way it does. Understanding them helps you reverse engineer more effectively.

Linux Environment Setup¶

Essential Tool Installation¶

Core Analysis Tools¶

# Update system
sudo apt update && sudo apt upgrade -y

# Basic utilities
sudo apt install -y build-essential git curl wget python3 python3-pip

# Binary analysis essentials
sudo apt install -y binutils gdb radare2 strace ltrace
sudo apt install -y binwalk foremost scalpel
sudo apt install -y hexedit bless

Getting Started: Tools matter , but don't get overwhelmed. Start with GDB and objdump , they're free and powerful.

Package Management: Apt makes this easy , just run the commands above. No hunting for downloads or dealing with dependencies.

Advanced Tools¶

# Decompilers and disassemblers
sudo apt install -y ghidra
wget -O ida_free.deb https://out7.hex-rays.com/files/idafree80_linux.run
chmod +x ida_free.deb && ./ida_free.deb

# Python libraries for RE
pip3 install pefile capstone unicorn keystone-engine ropgadget
pip3 install angr pwntools

# Network analysis
sudo apt install -y wireshark tshark tcpdump scapy

Ghidra is Amazing: NSA made it free , it's basically IDA Pro but without the price tag. Takes time to learn , but worth it.

Python Libraries: These let you automate reverse engineering tasks. Capstone for disassembly , Angr for symbolic execution.

Specialized Tools¶

# Malware analysis
sudo apt install -y yara clamav
pip3 install floss

# Firmware tools
sudo apt install -y qemu-system qemu-user-static
git clone https://github.com/devttys0/binwalk
cd binwalk && sudo python3 setup.py install

# Cryptography
sudo apt install -y john hashcat
pip3 install cryptography pycrypto

YARA Rules: Write custom rules to find malware patterns. Like regex but for binaries.

QEMU for Emulation: Run ARM binaries on x86 , or analyze firmware in a safe environment.

Development Environment¶

Custom Kernel for Analysis¶

# Install kernel headers
sudo apt install -y linux-headers-$(uname -r)

# Build custom kernel with debug symbols
sudo apt install -y kernel-package
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.0.tar.xz
tar xf linux-5.15.0.tar.xz
cd linux-5.15.0
make menuconfig  # Enable debug options
make -j$(nproc)
sudo make modules_install
sudo make install

Kernel Debugging: Sometimes you need to understand kernel modules or drivers. This lets you build a kernel with debug symbols.

Time Consuming: Building a kernel takes hours , but gives you full source-level debugging of kernel code.

Virtual Machine Setup¶

# Install KVM/QEMU
sudo apt install -y qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
sudo systemctl enable libvirtd
sudo systemctl start libvirtd

# Create analysis VM
virt-install --name analysis-vm \
  --ram 4096 \
  --disk path=/var/lib/libvirt/images/analysis-vm.qcow2,size=20 \
  --vcpus 2 \
  --os-type linux \
  --os-variant ubuntu20.04 \
  --network bridge=virbr0 \
  --graphics none \
  --console pty,target_type=serial \
  --location 'http://archive.ubuntu.com/ubuntu/dists/focal/main/installer-amd64/' \
  --extra-args 'console=ttyS0,115200n8 serial'

Safe Analysis: Run suspicious binaries in VMs. If they break out , only the VM is affected.

Headless Setup: No GUI needed for analysis , serial console works fine for command line tools.

Tool Configuration¶

GDB Setup¶

# Install GEF (GDB Enhanced Features)
wget -q -O- https://github.com/hugsy/gef/raw/master/scripts/gef.sh | sh

# Create .gdbinit
cat > ~/.gdbinit << 'EOF'
source ~/.gef.py
set disassembly-flavor intel
set pagination off
set confirm off
EOF

GEF Makes GDB Better: Adds heap analysis , ROP gadgets , and better formatting. Makes debugging way easier.

Intel Syntax: Most people prefer Intel syntax over AT&T. This config sets it automatically.

Radare2 Configuration¶

# Create radare2 config
mkdir -p ~/.config/radare2
cat > ~/.config/radare2/radare2rc << 'EOF'
e asm.arch = x86
e asm.bits = 64
e asm.syntax = intel
e scr.color = true
EOF

Radare2 Defaults: Set your preferred architecture and syntax. Colors make output readable.

Scriptable Power: Radare2 has a full scripting language. Automate complex analysis tasks.

Ghidra Setup¶

# Run Ghidra
cd /opt/ghidra
./ghidraRun

# Configure for headless analysis
java -cp "ghidra.jar:." ghidra.app.util.headless.AnalyzeHeadless /tmp/project -import /path/to/binary -scriptPath /path/to/scripts -postScript AnalyzeBinary.java

Headless Mode: Analyze binaries from command line. Great for automation and CI/CD pipelines.

Scripting: Java-based scripting lets you extend Ghidra's capabilities.

File Analysis and Triage¶

Initial File Assessment¶

File Type Identification¶

# Basic file type
file binary

# Detailed ELF analysis
readelf -a binary

# Check for UPX packing
upx -t binary

# Entropy analysis (high entropy = compressed/encrypted)
ent binary

# Hash verification
md5sum binary
sha256sum binary

First Things First: When you get a mysterious binary , start here. File type tells you everything.

ELF Deep Dive: Readelf gives you the full ELF structure. Sections , symbols , everything.

Binary Properties¶

# Architecture and compilation details
objdump -f binary

# Check if stripped
nm binary | head -10  # Empty output = stripped

# Library dependencies
ldd binary

# Check for debug symbols
objdump -g binary | head

# File size and sections
ls -lh binary
size binary

Stripped Binaries: No symbols means harder analysis. You'll see this in production software.

Library Dependencies: Shows what the binary needs to run. Missing libs = won't work.

Content Extraction¶

String Analysis¶

# Extract ASCII strings
strings binary

# Unicode strings
strings -e l binary  # UTF-16LE
strings -e b binary  # UTF-16BE

# Minimum string length
strings -n 8 binary

# With offsets
strings -t x binary

Strings Are Gold: Error messages , URLs , file paths - they give away functionality.

Unicode Matters: Modern software uses Unicode. Don't miss those strings.

Binary Data Carving¶

# Extract embedded files
binwalk -e binary

# Carve specific file types
foremost -t all -i binary -o carved/

# Manual carving with dd
dd if=binary of=extracted.bin bs=1 skip=0x1000 count=0x2000

Embedded Files: Binaries often contain images , configs , or other data.

Manual Carving: When tools fail , dd is your friend. Calculate offsets carefully.

Archive Analysis¶

# Check for archives
file binary | grep -i archive

# Extract if ZIP
unzip binary 2>/dev/null

# Extract if TAR
tar -xf binary

# Extract if RAR
unrar x binary

Hidden Archives: Sometimes binaries are just archives with executable bits set.

Extraction Tools: Each format needs its own tool. Know your formats.

Understanding Binary Protections - What Defenses Are Enabled?¶

Modern compilers add various protections to binaries. Understanding what's enabled helps you understand how the binary works.

Checking Stack Protection:

# Stack canary (prevents stack overflow)
readelf -s binary | grep __stack_chk_fail
# If found, stack canary is enabled

# How it works:
# Compiler inserts random value before return address
# Before return, checks if value changed
# If changed, program aborts (overflow detected)

Stack Canaries: Like a tripwire on your stack. Overflow touches it , program dies.

Why It Matters: Tells you if the binary was compiled with security in mind.

Checking Executable Protection:

# NX/DEP (No Execute / Data Execution Prevention)
readelf -l binary | grep GNU_STACK
# Shows stack permissions
# RW- means non-executable (good)

# What it means:
# Code sections executable
# Data sections not executable
# Prevents executing data as code

NX Bit: Old exploit technique was to execute shellcode on stack. NX stops that.

Modern Default: Most compilers enable NX now. Old binaries might not have it.

Checking ASLR:

# ASLR (Address Space Layout Randomization)
readelf -d binary | grep PIE
# If PIE (Position Independent Executable), ASLR enabled

# What it means:
# Base addresses randomized each run
# Makes addresses unpredictable

PIE Binaries: Can load anywhere in memory. Makes exploitation harder.

Position Dependent: Old binaries load at fixed addresses. Easier to attack.

Understanding Function Safety:

# Check what string functions are used
objdump -d binary | grep -E "(strcpy|strcat|sprintf|gets|scanf)"
# Older unsafe functions might be present

# Modern code uses safer alternatives:
# strcpy -> strncpy or strlcpy
# sprintf -> snprintf
# gets -> fgets

Unsafe Functions: These are buffer overflow waiting to happen. Modern code avoids them.

Safe Alternatives: Bounds checking prevents overflows. Good coding practice.

Other Protection Mechanisms:

# RELRO (Relocation Read-Only)
readelf -l binary | grep RELRO
# Makes relocation tables read-only after linking

# Fortify Source
objdump -d binary | grep -i fortify
# Compiler adds runtime checks for buffer operations

# Safe Linking
readelf -d binary | grep BIND_NOW
# Immediate binding, prevents lazy linking attacks

RELRO Levels: Partial RELRO allows some writes. Full RELRO locks everything down.

Fortify Source: Adds _chk versions of dangerous functions. Like strcpy_chk.

Why This Matters: Understanding protections helps you: - Understand how the binary was compiled - Know what security features are active - Better understand runtime behavior - These aren't vulnerabilities - they're defensive measures

Real World Impact: When you see protections enabled , you know the developers cared about security. When disabled , you know where to look for bugs.

Advanced Identification Techniques¶

Memory Layout Identification¶

Process Memory Mapping¶

# View process memory layout
cat /proc/$(pidof binary)/maps

# Identify memory regions
# 00400000-00401000 r-xp 00000000 08:01 1234567 /path/to/binary  # Text segment
# 00600000-00601000 r--p 00000000 08:01 1234567 /path/to/binary  # Read-only data
# 00601000-00602000 rw-p 00001000 08:01 1234567 /path/to/binary  # Data segment
# 7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0               # Stack
# heap_start-heap_end rw-p 00000000 00:00 0                     # Heap

Heap Analysis¶

# Identify heap chunks (glibc)
# malloc_chunk structure:
#   prev_size (8 bytes)
#   size (8 bytes) - includes flags
#   fd/next (8 bytes) - forward pointer
#   bk/prev (8 bytes) - backward pointer
#   user data

# Check heap metadata
gdb ./binary
(gdb) heap chunks
(gdb) heap bins

Stack Frame Identification¶

# Stack frame structure
# High addresses
#   Function arguments
#   Return address (8 bytes)
#   Saved RBP (8 bytes) - frame pointer
#   Local variables
# Low addresses

# Identify stack frames
gdb ./binary
(gdb) bt  # Backtrace
(gdb) info frame
(gdb) x/20gx $rsp  # Examine stack

Memory Corruption Detection¶

# Check for heap corruption
# Look for corrupted chunk headers
# Size field manipulation
# Pointer corruption

# Stack corruption
# Return address overwrite
# Frame pointer corruption
# Local variable corruption

Assembly Instruction Identification¶

Instruction Categories¶

# Data Movement
mov, push, pop, lea, xchg

# Arithmetic
add, sub, mul, div, inc, dec, neg

# Logic
and, or, xor, not, test

# Control Flow
jmp, je/jne, jg/jl, call, ret

# String Operations
movs, stos, lods, scas, cmps

# System Calls
syscall, int 0x80, sysenter

Calling Convention Identification¶

# System V AMD64 (Linux)
# Arguments: RDI, RSI, RDX, RCX, R8, R9
# Return: RAX
# Callee-saved: RBX, RBP, R12-R15
# Caller-saved: RAX, RCX, RDX, RSI, RDI, R8-R11, RSP

# Microsoft x64
# Arguments: RCX, RDX, R8, R9
# Return: RAX
# Callee-saved: RBX, RBP, RDI, RSI, RSP, R12-R15
# Caller-saved: RAX, RCX, RDX, R8-R11

# Identify by prologue/epilogue
push rbp        # Save frame pointer
mov rbp, rsp    # Set new frame pointer
sub rsp, size   # Allocate locals
...             # Function body
mov rsp, rbp    # Restore stack pointer
pop rbp         # Restore frame pointer
ret             # Return

Anti-Disassembly Techniques¶

# Opaque predicates
# Always true/false conditions
# jmp short $+2 + nop

# Control flow flattening
# Single dispatcher block
# State machine based control flow

# Instruction overlapping
# Different interpretations of same bytes

Gadget Identification¶

# ROP gadgets
ropper -f binary

# Common gadgets
pop rax; ret
add rax, rbx; ret
jmp rax

# Gadget quality
# No bad characters
# Useful instructions
# Clean return

System Call Identification¶

Linux System Call Table¶

# x86-64 syscall numbers
# 0: read
# 1: write
# 2: open
# 3: close
# 4: stat
# 5: fstat
# 6: lstat
# 7: poll
# 8: lseek
# 9: mmap
# 10: mprotect
# 11: munmap
# 12: brk
# 13: rt_sigaction
# 14: rt_sigprocmask
# 15: rt_sigreturn
# 16: ioctl
# 17: pread64
# 18: pwrite64
# 19: readv
# 20: writev
# 21: access
# 22: pipe
# 23: select
# 24: sched_yield
# 25: mremap
# 26: msync
# 27: mincore
# 28: madvise
# 29: shmget
# 30: shmat
# 31: shmctl
# 32: dup
# 33: dup2
# 34: pause
# 35: nanosleep
# 36: getitimer
# 37: alarm
# 38: setitimer
# 39: getpid
# 40: sendfile
# 41: socket
# 42: connect
# 43: accept
# 44: sendto
# 45: recvfrom
# 46: sendmsg
# 47: recvmsg
# 48: shutdown
# 49: bind
# 50: listen
# 51: getsockname
# 52: getpeername
# 53: socketpair
# 54: setsockopt
# 55: getsockopt
# 56: clone
# 57: fork
# 58: vfork
# 59: execve
# 60: exit
# 61: wait4
# 62: kill
# 63: uname
# 64: semget
# 65: semop
# 66: semctl
# 67: shmdt
# 68: msgget
# 69: msgsnd
# 70: msgrcv
# 71: msgctl
# 72: fcntl
# 73: flock
# 74: fsync
# 75: fdatasync
# 76: truncate
# 77: ftruncate
# 78: getdents
# 79: getcwd
# 80: chdir
# 81: fchdir
# 82: rename
# 83: mkdir
# 84: rmdir
# 85: creat
# 86: link
# 87: unlink
# 88: symlink
# 89: readlink
# 90: chmod
# 91: fchmod
# 92: chown
# 93: fchown
# 94: lchown
# 95: umask
# 96: gettimeofday
# 97: getrlimit
# 98: getrusage
# 99: sysinfo
# 100: times
# 101: ptrace
# 102: getuid
# 103: syslog
# 104: getgid
# 105: setuid
# 106: setgid
# 107: geteuid
# 108: getegid
# 109: setpgid
# 110: getppid
# 111: getpgrp
# 112: setsid
# 113: setreuid
# 114: setregid
# 115: getgroups
# 116: setgroups
# 117: setresuid
# 118: getresuid
# 119: setresgid
# 120: getresgid
# 121: getpgid
# 122: setfsuid
# 123: setfsgid
# 124: getsid
# 125: capget
# 126: capset
# 127: rt_sigpending
# 128: rt_sigtimedwait
# 129: rt_sigqueueinfo
# 130: rt_sigsuspend
# 131: sigaltstack
# 132: utime
# 133: mknod
# 134: uselib
# 135: personality
# 136: ustat
# 137: statfs
# 138: fstatfs
# 139: sysfs
# 140: getpriority
# 141: setpriority
# 142: sched_setparam
# 143: sched_getparam
# 144: sched_setscheduler
# 145: sched_getscheduler
# 146: sched_get_priority_max
# 147: sched_get_priority_min
# 148: sched_rr_get_interval
# 149: sched_getaffinity
# 150: sched_setaffinity
# 151: sched_getattr
# 152: sched_setattr
# 153: getcpu
# 154: gettid
# 155: tkill
# 156: time
# 157: futex
# 158: sched_setaffinity
# 159: io_setup
# 160: io_destroy
# 161: io_getevents
# 162: io_submit
# 163: io_cancel
# 164: get_thread_area
# 165: lookup_dcookie
# 166: epoll_create
# 167: epoll_ctl_old
# 168: epoll_wait_old
# 169: remap_file_pages
# 170: getdents64
# 171: set_tid_address
# 172: restart_syscall
# 173: semtimedop
# 174: fadvise64
# 175: timer_create
# 176: timer_settime
# 177: timer_gettime
# 178: timer_getoverrun
# 179: timer_delete
# 180: clock_settime
# 181: clock_gettime
# 182: clock_getres
# 183: clock_nanosleep
# 184: exit_group
# 185: epoll_wait
# 186: epoll_ctl
# 187: tgkill
# 188: utimes
# 189: vserver
# 190: mbind
# 191: set_mempolicy
# 192: get_mempolicy
# 193: mq_open
# 194: mq_unlink
# 195: mq_timedsend
# 196: mq_timedreceive
# 197: mq_notify
# 198: mq_getsetattr
# 199: kexec_load
# 200: waitid
# 201: add_key
# 202: request_key
# 203: keyctl
# 204: ioprio_set
# 205: ioprio_get
# 206: inotify_init
# 207: inotify_add_watch
# 208: inotify_rm_watch
# 209: pset
# 210: migrate_pages
# 211: openat
# 212: mkdirat
# 213: mknodat
# 214: fchownat
# 215: futimesat
# 216: newfstatat
# 217: unlinkat
# 218: renameat
# 219: linkat
# 220: symlinkat
# 221: readlinkat
# 222: fchmodat
# 223: faccessat
# 224: pselect6
# 225: ppoll
# 226: unshare
# 227: set_robust_list
# 228: get_robust_list
# 229: splice
# 230: tee
# 231: sync_file_range
# 232: vmsplice
# 233: move_pages
# 234: utimensat
# 235: epoll_pwait
# 236: signalfd
# 237: timerfd_create
# 238: eventfd
# 239: fallocate
# 240: timerfd_settime
# 241: timerfd_gettime
# 242: accept4
# 243: signalfd4
# 244: eventfd2
# 245: epoll_create1
# 246: dup3
# 247: pipe2
# 248: inotify_init1
# 249: preadv
# 250: pwritev
# 251: rt_tgsigqueueinfo
# 252: perf_event_open
# 253: recvmmsg
# 254: fanotify_init
# 255: fanotify_mark
# 256: prlimit64
# 257: name_to_handle_at
# 258: open_by_handle_at
# 259: clock_adjtime
# 260: syncfs
# 261: sendmmsg
# 262: setns
# 263: getcpu
# 264: process_vm_readv
# 265: process_vm_writev
# 266: kcmp
# 267: finit_module
# 268: sched_setattr
# 269: sched_getattr
# 270: renameat2
# 271: seccomp
# 272: getrandom
# 273: memfd_create
# 274: kexec_file_load
# 275: bpf
# 276: execveat
# 277: userfaultfd
# 278: membarrier
# 279: mlock2
# 280: copy_file_range
# 281: preadv2
# 282: pwritev2
# 283: pkey_mprotect
# 284: pkey_alloc
# 285: pkey_free
# 286: statx
# 287: io_pgetevents
# 288: rseq
# 289: pkey_mprotect

System Call Tracing¶

# Trace system calls
strace ./binary

# Filter specific syscalls
strace -e trace=network ./binary

# Count syscalls
strace -c ./binary

# Attach to running process
strace -p $(pidof binary)

System Call Identification in Assembly¶

# Direct syscall
mov rax, 1      ; write syscall
mov rdi, 1      ; stdout
mov rsi, msg    ; buffer
mov rdx, len    ; length
syscall

# libc wrapper
call write@plt

# Identify syscall patterns
# RAX = syscall number
# Arguments in registers
# syscall instruction

Library Identification¶

Dynamic Library Analysis¶

# List loaded libraries
ldd binary

# Detailed library information
readelf -d binary

# Library search paths
ldconfig -p

# Check library versions
objdump -p binary | grep NEEDED

Static Library Analysis¶

# Extract from archives
ar -t lib.a

# List symbols in static library
nm lib.a

# Check for static linking
file binary | grep statically

Library Function Identification¶

# Find function usage
objdump -d binary | grep call

# PLT/GOT analysis
objdump -R binary  # Relocations
objdump -s -j .plt binary  # Procedure Linkage Table

# Import table
readelf -r binary

Custom Library Detection¶

# Embedded libraries
strings binary | grep -i lib

# Library loading patterns
# dlopen, dlsym, dlclose

# Custom library paths
# LD_LIBRARY_PATH
# /etc/ld.so.conf

Function Identification¶

Function Boundary Detection¶

# Function prologue patterns
push rbp
mov rbp, rsp
sub rsp, size

# Function epilogue patterns
mov rsp, rbp
pop rbp
ret

# Identify functions
nm binary | grep -E " [Tt] "  # Text symbols

Function Signature Analysis¶

# Standard library functions
strcmp, strcpy, printf, malloc, free

# Identify by calling convention
# Arguments passed in registers/stack
# Return value in RAX

# Function fingerprinting
# Control flow patterns
# Register usage patterns

Virtual Function Table Identification¶

# C++ vtables
# Array of function pointers
# Located in read-only data section

# RTTI information
# Type information
# Inheritance hierarchy

Inline Function Detection¶

# Compiler optimizations
# Functions expanded inline
# No separate symbol

# Identify by code patterns
# Repeated instruction sequences
# Parameter passing elimination

Data Structure Identification¶

Common Data Structures¶

# Arrays
# Contiguous memory blocks
# Same data type elements

# Linked lists
# Node structures with pointers
# next/prev pointers

# Hash tables
# Key-value pairs
# Hash function + collision handling

# Trees
# Node structures with child pointers
# Binary trees, red-black trees, etc.

Structure Field Identification¶

# Offset calculations
# mov rax, [rbx+offset]

# Structure padding
# Compiler alignment rules
# sizeof calculations

# Bit field identification
# Bit manipulation operations
# Masking and shifting

String and Buffer Identification¶

# Null-terminated strings
# ASCII/Unicode detection
# String table analysis

# Buffer operations
# memcpy, memset patterns
# Length calculations

Global Variable Identification¶

# .data section analysis
# Initialized globals

# .bss section analysis
# Uninitialized globals

# GOT/PLT references
# External variable access

Code Pattern Recognition¶

Common Idioms¶

# Loop constructs
# for: init, condition, increment
# while: condition check, body
# do-while: body, condition

# Conditional statements
# if-else chains
# switch-case tables

# Function pointer usage
# Indirect calls
# Callback mechanisms

Compiler-Specific Patterns¶

# GCC optimizations
# Loop unrolling
# Function inlining
# Tail call optimization

# Clang/LLVM patterns
# Different optimization choices
# Debug information differences

Anti-Analysis Pattern Detection¶

# Junk code insertion
# NOP sleds
# Unreachable code

# Control flow obfuscation
# Opaque predicates
# Control flow flattening

# Data obfuscation
# Encrypted strings
# Polymorphic code

Cross-Reference Analysis¶

Function Call Graph¶

# Build call graph
# Who calls whom
# Call hierarchies

# Identify entry points
# Exported functions
# Callback registrations

Data Flow Analysis¶

# Variable usage tracking
# Data dependencies
# Taint analysis

# Register liveness
# Variable lifetime analysis

Control Flow Analysis¶

# Basic block identification
# Control flow graphs
# Dominance analysis

# Loop detection
# Nested loop identification
# Loop invariants

Static Analysis Tools¶

objdump Deep Dive¶

Your First Tool: Objdump is the Swiss Army knife of binary analysis. It's built into Linux and always works.

Disassembly:

# Full disassembly
objdump -d binary

# Intel syntax (easier to read)
objdump -d -M intel binary

# Specific function only
objdump -d --start-address=0x4005c0 --stop-address=0x400600 binary

# With source lines if debug info exists
objdump -d -S binary

Symbol Table Analysis:

# All symbols in binary
nm binary

# Dynamic symbols (imports/exports)
nm -D binary

# Undefined symbols (what it needs)
nm -u binary

# Demangle C++ names
nm -C binary

Section Analysis:

# Section headers overview
objdump -h binary

# Specific section content
objdump -s -j .text binary

# Relocation entries
objdump -r binary

Real World Use: When you need quick disassembly without fancy GUIs, objdump delivers. It's fast and reliable.

readelf Comprehensive Usage¶

ELF Expert: Readelf knows everything about ELF files. Use it when objdump isn't enough.

ELF Header Analysis:

# File header details
readelf -h binary

# Program headers (runtime segments)
readelf -l binary

# Section headers (link-time sections)
readelf -S binary

# Symbol table
readelf -s binary

Dynamic Analysis:

# Dynamic section (libraries, relocations)
readelf -d binary

# Relocation details
readelf -r binary

# Version information
readelf -V binary

Debugging Information:

# Debug sections
readelf -w binary

# String table dump
readelf -p .strtab binary

Practical Tip: Readelf shows the ELF structure clearly. Essential for understanding binary layout.

Radare2 Static Analysis¶

Command Line Power: Radare2 is incredibly powerful but has a steep learning curve. Worth it for scripting.

Basic Analysis:

# Open and auto-analyze
r2 -A binary

# List all functions
afl

# List strings
iz

# Cross-references to function
axt @ function_name

Code Analysis:

# Disassemble main function
pdf @ main

# Generate control flow graph
agf @ main > main.dot
dot -Tpng main.dot -o main.png

# Analyze function variables
afv @ main

Binary Information:

# File info summary
i

# Entry point address
ie

# Imports and exports
ii
iE

Scripting Advantage: Radare2's scripting lets you automate complex analysis tasks.

Ghidra Static Analysis¶

NSA's Gift: Ghidra is free IDA Pro. It's amazing but takes time to learn. The decompiler alone is worth it.

Project Setup:

# Launch Ghidra
ghidraRun

# Create new project
File -> New Project -> Non-Shared Project

# Import your binary
File -> Import File

# Auto-analyze (takes time)
Analysis -> Auto Analyze

Code Analysis:

# Find main function
Window -> Symbol Tree -> Functions -> main

# Decompile to C-like code
Right-click function -> Decompile

# Find cross-references
Right-click symbol -> References -> Find References to

Script Analysis:

# Run analysis scripts
Window -> Script Manager
Select script -> Run

# Custom Java scripts
public class AnalyzeBinary extends GhidraScript {
    @Override
    public void run() throws Exception {
        println("Analysis started");
        // Your analysis code here
    }
}

Learning Curve: Ghidra is powerful but overwhelming at first. Start with simple binaries and build up.

Dynamic Analysis with GDB¶

GDB Basics¶

Getting Started: GDB is your debugger. It's powerful but takes practice. Start simple.

Starting and Running:

# Start GDB
gdb binary

# Run with arguments
run arg1 arg2

# Attach to running process
gdb -p $(pidof binary)

# Load core dump
gdb binary core

Breakpoints:

# Break at function
break main
break printf

# Break at address
break *0x4005c0

# Break at line (if source available)
break file.c:10

# Conditional breakpoint
break main if argc > 1

# Hardware breakpoint
hbreak main

Execution Control:

# Continue execution
continue

# Step into function
step

# Step over function
next

# Step one instruction
stepi

# Continue until return
finish

# Run until address
until *0x400600

Memory Analysis¶

Examining Memory:

# Print variable
print variable
print $rax

# Print string
print (char*) $rdi

# Examine memory
x/s $rsp          # String at stack pointer
x/10i $rip        # 10 instructions from instruction pointer
x/20wx $rsp       # 20 words in hex from stack
x/50cb $rbx       # 50 bytes in char from rbx

Memory Layout:

# Stack frame info
info frame

# Registers
info registers

# Current stack
x/20gx $rsp

# Heap analysis
info proc mappings

Advanced GDB Features¶

Watchpoints:

# Watch variable change
watch global_var

# Watch memory location
watch *0x601000

# Read watchpoint
rwatch *0x601000

# Access watchpoint
awatch *0x601000

Reverse Debugging:

# Enable recording
record

# Reverse step
reverse-step

# Reverse continue
reverse-continue

# Reverse to previous breakpoint
reverse-next

GDB Scripting:

# Python scripting in GDB
python
import gdb

def on_stop(event):
    print("Program stopped")
    gdb.execute("info registers")

gdb.events.stop.connect(on_stop)
end

GDB with GEF¶

GEF Makes GDB Better: GEF adds heap analysis, ROP gadgets, and better formatting. Install it.

GEF Commands:

# Enhanced context
context

# Heap analysis
heap chunks
heap bins

# ROP gadgets
ropgadget

# Pattern creation
pattern create 100

# Checksec equivalent
checksec

Exploit Development:

# Find offset
pattern offset input_pattern

# Shellcode testing
shellcode search x86/linux execve

# Format string helper
fmtstr_payload 8 {"A"*8 + p64(system_addr)}

Disassembly and Decompilation¶

IDA Pro Usage¶

Basic Disassembly¶

# Open binary
idaq64 binary

# Auto-analysis
# Wait for analysis to complete

# Navigate functions
View -> Open subviews -> Functions

# Cross-references
View -> Open subviews -> Cross-references

Graph View¶

# Control flow graph
Space (toggle graph/text)

# Zoom and navigation
Mouse wheel, drag

# Function calls
Double-click on call

Decompilation¶

# Hex-Rays decompiler
View -> Open subviews -> Pseudocode

# Decompile function
Right-click in disassembly -> Decompile

Ghidra Decompilation¶

Decompiler Window¶

# Open decompiler
Window -> Decompiler

# Decompile specific function
Select function in listing -> Decompile

# Edit decompiled code
Right-click in decompiler -> Edit Function

Analysis Options¶

# Configure analysis
Analysis -> Analyze All Open
Analysis -> One Shot -> Function ID

# Script decompilation
decompileFunction(function)

Binary Ninja¶

Basic Usage¶

# Open binary
./binaryninja binary

# Linear view
View -> Linear

# Graph view
View -> Graph

# Hex view
View -> Hex

Analysis¶

# Function analysis
Tools -> Analysis -> Analyze All Functions

# String analysis
Tools -> Strings

# Cross-references
Right-click symbol -> References

Custom Disassembly Scripts¶

Python with Capstone¶

from capstone import *

# Initialize disassembler
md = Cs(CS_ARCH_X86, CS_MODE_64)

# Disassemble code
code = b"\x55\x48\x89\xe5\x89\x7d\xfc\x8b\x45\xfc"
for i in md.disasm(code, 0x1000):
    print(f"0x{i.address:x}:\t{i.mnemonic}\t{i.op_str}")

Radare2 Scripting¶

# Radare2 script
#!/bin/r2p
# Analyze all functions
aaa

# Print function list
afl

# Disassemble main
pdf @ main

# Find strings
iz

Binary Patching and Modification¶

Hex Editing¶

Direct Binary Surgery: Hex editing lets you modify binaries at the byte level. It's like editing DNA - one wrong change and everything breaks.

Basic Patching:

# Open in hex editor
hexedit binary

# Or use xxd
xxd binary > binary.hex
# Edit binary.hex
xxd -r binary.hex > patched_binary

Common Patches:

# NOP out instructions (x86)
# Change JZ to JNZ: 74 -> 75
# Change JE to JNE: 84 -> 85

# Remove password check
# Find password comparison
# Replace with unconditional jump

Radare2 Patching¶

Radare2 Makes It Easy: Radare2's patching commands let you modify binaries without hex editors. Much safer and more precise.

Write Operations:

# Open for writing
r2 -w binary

# Write assembly
wa nop @ 0x4005c0

# Write hex
wx 9090 @ 0x4005c0

# Write string
w hello @ 0x601000

Advanced Patching:

# Insert code
# Create space first
r2 -w binary
s 0x4005c0
wai nop
wai mov rax, 0xdeadbeef

Binary Instrumentation¶

Runtime Modification: Instrumentation lets you modify program behavior at runtime. Like adding sensors to a car engine while it's running.

Function Hooking:

# Using Frida
frida-trace -f binary -i "open"

# Hook with custom script
frida binary -l hook.js

Library Injection:

# LD_PRELOAD hooking
gcc -shared -fPIC hook.c -o hook.so
LD_PRELOAD=./hook.so ./binary

Binary Reconstruction¶

ELF Surgery: Sometimes you need to modify the ELF structure itself. Entry points, interpreters, library paths - all changeable.

ELF Modification:

# Change entry point
readelf -h binary
# Use patchelf
patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 binary
patchelf --set-rpath /custom/path binary

Section Manipulation:

# Add section
objcopy --add-section .custom=data.bin binary patched_binary

# Remove section
objcopy --remove-section .comment binary stripped_binary

Understanding Obfuscation - When Code Hides Its Intent¶

Some software tries to make reverse engineering harder. Code obfuscation, packing, anti-debugging - these techniques exist. Understanding them is part of reverse engineering. Let's talk about what you'll encounter and how to handle it.

Why Code Gets Obfuscated¶

People obfuscate code for different reasons: - Protection of intellectual property - Don't want competitors to understand algorithms - Anti-tampering - Prevent modification of software - License enforcement - Make it harder to bypass checks - Sometimes just because - Some developers obfuscate everything (not always necessary)

The reality: Good obfuscation makes reverse engineering take longer, but it doesn't make it impossible. It's an arms race, and understanding what you're up against helps.

Packed Binaries - Compression and Encryption¶

Packed binaries are compressed or encrypted executables that unpack themselves at runtime. The actual code is hidden until execution.

Detecting Packing:

# Check file entropy (high = likely compressed/encrypted)
ent binary

# UPX is common
upx -t binary

# Check for unusual section names
readelf -S binary | grep -E "(packed|upx|crypt)"

# Strings output might be minimal
strings binary | wc -l  # Very few strings = likely packed

UPX Unpacking: UPX is common and often easy to unpack:

# Try automatic unpacking
upx -d binary

# If that fails, manual unpacking
# Run in debugger, find OEP (Original Entry Point)
# Dump memory after unpacking

Manual Unpacking Process:

# 1. Run packed binary in debugger
gdb ./packed_binary

# 2. Set breakpoint on common unpack functions
(gdb) break malloc
(gdb) break VirtualAlloc  # Windows
(gdb) run

# 3. When unpacking completes, find OEP
(gdb) info proc mappings  # See where code was unpacked

# 4. Dump unpacked code
(gdb) dump binary memory unpacked.bin start_addr end_addr

Understanding Packing: Most packers follow similar patterns: 1. Small unpacker stub (the code you see initially) 2. Unpacker decompresses/decrypts main code 3. Unpacker transfers control to original entry point (OEP) 4. Program runs normally

Finding the OEP: - Look for jumps to code sections - Find where unpacker completes - Look for return addresses on stack - Pattern matching: unpackers often have signatures

Code Obfuscation - Making Assembly Confusing¶

Obfuscation transforms code to be harder to understand while maintaining functionality.

Control Flow Flattening: This makes all code paths look similar by using a dispatcher:

# Normal code:
if (x > 0) {
    do_a();
} else {
    do_b();
}

# Flattened version:
state = 0;
while (state != EXIT) {
    switch(state) {
        case 0: state = (x > 0) ? STATE_A : STATE_B; break;
        case STATE_A: do_a(); state = EXIT; break;
        case STATE_B: do_b(); state = EXIT; break;
    }
}

Recognizing Control Flow Flattening: - Look for large switch statements or jump tables - Single dispatcher block that routes to different states - All functions look similar structurally - State variable that controls flow

Deobfuscation Strategy: 1. Identify the dispatcher 2. Map state values to original code paths 3. Reconstruct control flow graph 4. Simplify by removing dispatcher logic

Opaque Predicates: These are conditions that always evaluate the same way but look like they could go either way:

# Opaque predicate - always true
mov rax, 5
mul rax      # rax = 25
mov rbx, rax
add rbx, 10  # rbx = 35
cmp rbx, 30  # Always > 30, so always true
jge real_code
junk_code:   # Never executed
    nop
    nop
real_code:
    # Actual code here

Identifying Opaque Predicates: - Conditions that always evaluate the same way - Junk code that's never executed - Complex arithmetic that simplifies to constant - Use symbolic execution or runtime analysis to identify

Instruction Substitution: Replacing simple operations with equivalent complex ones:

# Normal: mov rax, 5
# Obfuscated:
mov rax, 10
shr rax, 1   # rax = 5

# Or:
xor rax, rax
add rax, 3
add rax, 2   # rax = 5

Recognizing Substitutions: - Look for equivalent instruction sequences - Use pattern matching to find common substitutions - Simplify during analysis

String Obfuscation¶

Strings often give away functionality, so they're frequently obfuscated.

Encrypted Strings:

# Strings encrypted at compile time
# Decrypted at runtime when needed
# Look for decryption functions called before string usage

Finding String Decryption:

# Find where strings are used
objdump -d binary | grep -A 10 "call.*decrypt"

# Or in debugger:
(gdb) break strcmp
(gdb) run
# When strcmp called, examine arguments
(gdb) x/s $rdi  # First argument (decrypted string)

XOR Encoding (Common): Simple XOR is often used:

// Encrypted: "Hello" XOR 0x42
char encrypted[] = {0x2a, 0x26, 0x2c, 0x2c, 0x2f};
for (int i = 0; i < 5; i++) {
    encrypted[i] ^= 0x42;  // Decrypt
}

Identifying XOR: - Look for XOR loops before string operations - Common key values: 0x00-0xFF - Check if same key used for all strings

Anti-Debugging Techniques¶

Some software tries to detect if it's being debugged.

Common Detection Methods:

# ptrace check (Linux)
mov rax, 101      # ptrace syscall
xor rdi, rdi      # PTRACE_TRACEME
syscall
# If returns -1, debugger detected

# Timing checks
rdtsc             # Read timestamp
# Compare execution time
# Debugger slows things down

Recognizing Anti-Debug: - Calls to ptrace / IsDebuggerPresent - Timing checks using RDTSC - Exception handling for breakpoints - Checksums of code sections

Dealing with Anti-Debug:

# Patch out detection code
# Find ptrace calls
objdump -d binary | grep -A 5 "call.*ptrace"

# Replace with NOPs
# Or patch return value to 0

Note: We're not bypassing security here - we're understanding how programs work. If you're reverse engineering your own software or have permission, these techniques help you understand the code flow.

Virtual Machine Obfuscation¶

Advanced obfuscation: code is converted to bytecode for a custom VM.

How It Works: 1. Original code compiled to custom bytecode 2. VM interpreter executes bytecode 3. Makes reverse engineering much harder

Recognizing VM Obfuscation: - Look for interpreter loop (big switch/case) - Bytecode patterns in data sections - No direct assembly equivalent - Execution appears to follow VM instruction set

Analyzing VM Obfuscation: - Identify VM instruction set - Understand bytecode format - Reverse VM interpreter - Decompile bytecode to higher level

This is Advanced: VM obfuscation is serious obfuscation. Takes significant effort to reverse. But it's possible - everything is, given enough time.

Practical Tips for Obfuscated Code¶

Use All Tools: - Static analysis first (what can you see?) - Dynamic analysis (run it, see what happens) - Hybrid approach (combine both)

Focus on Behavior: - Don't get lost in obfuscation - Understand what the code does, not just how it's obfuscated - Runtime behavior tells you a lot

Pattern Recognition: - Obfuscation often has patterns - Once you understand one function, others are similar - Look for common obfuscation techniques

Automation: - Write scripts to deobfuscate patterns - Use symbolic execution for opaque predicates - Batch process similar obfuscated code

Take Breaks: - Obfuscated code is mentally exhausting - Come back with fresh eyes - Sometimes the solution appears after a break

Bypassing Anti-Reversing¶

Ptrace Bypass¶

# Disable ptrace
echo 0 > /proc/sys/kernel/yama/ptrace_scope

# Or patch the binary
# Find ptrace call
objdump -d binary | grep ptrace

# NOP it out

Anti-Debug Patches¶

# Remove debugger checks
# Find IsDebuggerPresent equivalent
grep -r "IsDebuggerPresent" .

# Patch to always return false

Obfuscation Removal¶

String Decryption¶

# Find decryption function
# Set breakpoint on string usage
# Run and dump decrypted strings

Control Flow Deobfuscation¶

# Use symbolic execution
angr
# Or manual analysis

Packers and Crypters¶

UPX Unpacking¶

# Check if packed
upx -t binary

# Unpack
upx -d binary

# Manual unpacking
# Find OEP (Original Entry Point)
# Dump memory at OEP

Custom Packers¶

# Identify packer
strings binary | grep -i packer

# Generic unpacking
# Run in debugger
# Find OEP
# Dump process memory

Code Obfuscation¶

Deobfuscation Techniques¶

# Control flow flattening
# Identify dispatcher
# Reconstruct control flow

# Instruction substitution
# Pattern matching
# Replace with canonical forms

Tool-Assisted Deobfuscation¶

# Use de4dot for .NET
de4dot binary.exe

# Use unconfuser for Java
# Manual analysis for custom obfuscation

Understanding Software Behavior - What Does This Actually Do?¶

The Fun Part: You've got a binary, and you want to know what it does. Not breaking it - just understanding. Here's how to figure out what this program actually does.

Initial Assessment: When you first load a binary, you need to understand what you're dealing with. Is it a GUI app? A server? A command line tool? What libraries does it use?

File Type and Architecture:

# First, what are we dealing with?
file mystery_binary

# Get detailed info
readelf -h mystery_binary

# Check what it links against
ldd mystery_binary  # Shared libraries
objdump -p mystery_binary | grep NEEDED  # Dynamic dependencies

Understanding the Entry Point: Every program has to start somewhere. Finding where execution begins helps you understand the flow.

# Find the entry point
readelf -h binary | grep Entry

# Disassemble from entry point
objdump -d --start-address=$(readelf -h binary | grep Entry | awk '{print $NF}') binary | head -50

Common Entry Point Patterns: - _start function - sets up environment, calls main - main function - your actual program logic - Library initialization - if statically linked, there's setup code

Behavioral Analysis - Watch It Run¶

Sometimes the best way to understand a program is to... just run it and see what happens. With monitoring, of course.

System Call Tracing:

# See what syscalls it makes
strace ./mystery_binary

# Filter for specific operations
strace -e trace=file ./mystery_binary  # File operations only
strace -e trace=network ./mystery_binary  # Network operations

# Save output for analysis
strace -o trace.log ./mystery_binary

Library Call Tracing:

# See what library functions it calls
ltrace ./mystery_binary

# Useful for understanding API usage
ltrace -e malloc+free ./mystery_binary  # Memory operations

File System Monitoring:

# See what files it accesses
strace -e trace=open,openat,read,write ./binary 2>&1 | grep -E "(open|read|write)"

# Or use inotify
inotifywait -m -r /tmp &
./binary

Network Monitoring:

# Capture network traffic
tcpdump -i any -w capture.pcap &
./binary
killall tcpdump

# Analyze later
tcpdump -r capture.pcap -A

Static Code Analysis - Reading the Code¶

Once you know what it does behaviorally, dive into the code to understand how it does it.

Function Identification:

# List all functions (if symbols present)
nm binary | grep -E " [Tt] "

# Find main function
objdump -d binary | grep -A 20 "<main>:"

# Look for interesting functions
strings binary | grep -i "function_name"

Following the Call Chain: Start from main, follow function calls, understand the flow.

# In GDB, set breakpoints on functions
gdb ./binary
(gdb) break main
(gdb) run
(gdb) step  # Step into functions
(gdb) next  # Step over functions

Understanding Data Flow:

# Find where variables are used
# Look for mov instructions storing values
objdump -d binary | grep -B 5 -A 5 "mov.*rax"

# String usage
strings binary | while read str; do
    grep -r "$str" binary  # Find where it's referenced
done

Dynamic Analysis - Runtime Investigation¶

Running the program in a debugger lets you see what's actually happening.

Setting Breakpoints:

# Break at function entry
gdb ./binary
(gdb) break main
(gdb) break function_name

# Break at specific address
(gdb) break *0x4005c0

# Conditional breakpoints (super useful)
(gdb) break main if argc > 1

Examining Runtime State:

# When stopped at breakpoint
(gdb) info registers  # See all register values
(gdb) print $rax  # Print specific register
(gdb) x/20gx $rsp  # Examine stack (20 8-byte values in hex)
(gdb) x/s $rdi  # Examine as string
(gdb) info locals  # Local variables (if debug info)

Tracing Execution:

# Record execution (reverse debugging)
(gdb) record
(gdb) continue
# ... program runs ...
(gdb) reverse-step  # Step backwards!
(gdb) reverse-continue  # Go back to previous breakpoint

Understanding Control Flow¶

One of the hardest parts of reverse engineering is understanding how the program makes decisions.

Identifying Conditionals:

# Compare and jump patterns
cmp rax, rbx     # Compare two values
je label         # Jump if equal (zero flag set)
jne label        # Jump if not equal
jg label         # Jump if greater (signed)
jl label         # Jump if less (signed)

Recognizing Loops:

# Loop pattern
mov rcx, 10      # Counter
loop_start:
    # ... loop body ...
    dec rcx       # Decrement counter
    jnz loop_start  # Jump if not zero

# While loop pattern
while_start:
    cmp rax, 0
    je loop_end
    # ... loop body ...
    jmp while_start
loop_end:

Switch Statements:

# Jump tables for switch statements
# Usually you'll see:
cmp rax, N
ja default_case
jmp [rax*8 + jump_table_base]  # Jump to table entry

Understanding Data Structures¶

When you reverse engineer, you'll encounter data structures. Here's how to recognize them.

Arrays:

# Array access pattern
mov rax, [rbx + rcx*8]  # rax = array[rcx]  (8-byte elements)
# rbx = array base
# rcx = index
# 8 = element size

Structures:

# Structure field access
mov rax, [rbx+0x10]  # Access field at offset 0x10
mov rax, [rbx+0x18]  # Access field at offset 0x18
# rbx = structure pointer
# Offsets tell you field positions

Linked Lists:

# List traversal
mov rbx, [rbx+8]  # rbx = rbx->next
test rbx, rbx
jnz loop_start    # Continue while not NULL

real world Example: Reversing a Simple Program¶

Let's reverse engineer something simple to see the process in action.

The Program:

// Let's say this is what we're reversing (we don't know this)
int calculate_sum(int *array, int length) {
    int sum = 0;
    for (int i = 0; i < length; i++) {
        sum += array[i];
    }
    return sum;
}

What We See in Disassembly:

calculate_sum:
    push rbp
    mov rbp, rsp
    sub rsp, 16          # Allocate locals
    mov [rbp-4], 0       # sum = 0
    mov [rbp-8], 0       # i = 0
loop:
    mov eax, [rbp-8]     # eax = i
    cmp eax, [rbp+16]    # compare i with length
    jge done             # if i >= length, done
    mov eax, [rbp-8]     # eax = i
    cdqe                 # sign extend
    mov rdx, rax         # rdx = i
    mov rax, [rbp+24]    # rax = array pointer
    add rax, rdx         # rax = array + i
    mov eax, [rax]       # eax = array[i]
    add [rbp-4], eax     # sum += array[i]
    add [rbp-8], 1       # i++
    jmp loop
done:
    mov eax, [rbp-4]     # return sum
    leave
    ret

How to Understand This: 1. Function prologue: push rbp; mov rbp, rsp - standard setup 2. Local variables: [rbp-4] and [rbp-8] - two integers 3. Loop: compare, jump, increment - classic loop pattern 4. Array access: [rax] where rax = array base + index 5. Return: value in eax (return register)

Tips for Understanding Assembly: - Name things: rename [rbp-4] to sum, [rbp-8] to i - Follow the flow: where does control go? - Understand the data: what's being stored where? - Look for patterns: loops, conditionals, function calls

Firmware and Embedded Analysis¶

Embedded Systems Are Everywhere: IoT devices, routers, smart appliances - they all run firmware. Understanding firmware helps you work with these devices. Not breaking them - understanding how they work.

Firmware Extraction: First step: get the firmware out of the device. Hardware or software methods work.

Hardware Methods:

# Chip-off
# Remove flash chip
# Read with programmer

# JTAG
# Connect JTAG debugger
# Dump memory

# UART/Serial
# Connect to console
# Extract via bootloader

Software Methods:

# Firmware updates
# Download official firmware
# Extract with binwalk

# Network extraction
# TFTP/HTTP capture during update

Firmware Analysis¶

Once You Have Firmware: Extract file systems and analyze the binaries inside.

File System Extraction:

# SquashFS
unsquashfs firmware.bin

# JFFS2
mount -t jffs2 firmware.bin /mnt/firmware

# CramFS
cramfsck firmware.bin
mount -t cramfs firmware.bin /mnt/firmware

Binary Analysis:

# Cross-compilation tools
sudo apt install -y gcc-arm-linux-gnueabi

# Analyze ARM binaries
file binary  # ARM ELF

# Emulate with QEMU
qemu-arm-static -L /usr/arm-linux-gnueabi/ ./binary

Embedded Systems Reversing¶

Different Architectures: Embedded systems use various CPUs. ARM is common, but MIPS, AVR, and others exist.

Architecture-Specific Analysis:

# MIPS analysis
objdump -d -m mips binary

# ARM analysis
arm-none-eabi-objdump -d binary

# AVR analysis
avr-objdump -d binary

Hardware Interaction: Embedded code interacts with hardware directly. Look for peripheral access.

# GPIO analysis
# Look for hardware control code

# Peripheral access
# I2C, SPI, UART code

IoT and Embedded Device Analysis¶

Why Reverse Firmware: - Understanding device functionality - Creating compatible software - Fixing bugs in unsupported devices - Learning embedded systems

Finding Interesting Things:

# Default credentials (if hardcoded)
strings firmware | grep -i password

# API keys or tokens
strings firmware | grep -E "[0-9a-f]{32,}"

# Network configurations
strings firmware | grep -E "(192\.168|10\.|172\.)"

# Device identifiers
strings firmware | grep -i "device\|model\|version"

Update Mechanism Understanding:

# How does firmware update work?
# Look for update functions
strings firmware | grep -i "update\|upgrade\|firmware"

# Update validation logic
# Signature verification?
# Version checking?

The Goal: Understanding how the device works, not exploiting it. You might want to: - Build compatible software - Understand undocumented features - Recover lost configuration methods - Learn how embedded systems work

Protocol and Format Reversing - Understanding Data Structures¶

Data Formats Are Everywhere: You've got files and network traffic that need understanding. Legacy formats, proprietary protocols - they all have structure. Here's how to figure them out.

File Format Reversing: Start with the basics. What type of file is this? Magic bytes tell the story.

Identifying File Format:

# Check file type
file unknown_file

# Look at magic bytes (first few bytes)
hexdump -C unknown_file | head -5

# Common magic bytes:
# PNG: 89 50 4E 47
# GIF: 47 49 46 38
# ZIP: 50 4B 03 04
# ELF: 7F 45 4C 46

Analyzing Structure: Once you know the type, dig into the structure. Look for patterns in the hex dump.

# Look for patterns in hex
hexdump -C file | less

# Find repeating structures
# Look for fixed-size chunks
# Identify headers/footers

Creating a Parser: When you understand the format, write code to parse it. This lets you work with the data.

import struct

def parse_custom_format(filename):
    with open(filename, 'rb') as f:
        # Read header (example: 4-byte magic, 4-byte version, 8-byte size)
        magic = f.read(4)
        version = struct.unpack('<I', f.read(4))[0]  # Little-endian uint32
        size = struct.unpack('<Q', f.read(8))[0]  # Little-endian uint64

        # Read data based on structure you discovered
        data = f.read(size)

        return {
            'magic': magic,
            'version': version,
            'size': size,
            'data': data
        }

Network Protocol Reversing¶

Network Traffic Holds Secrets: Software talks to servers using protocols. Sometimes those protocols aren't documented. Capture and analyze to understand.

Capturing Traffic: First step: capture the conversation. Use tcpdump or Wireshark.

# Capture network packets
tcpdump -i any -w protocol.pcap port 12345

# Or use Wireshark GUI
wireshark

# Analyze captured data
tcpdump -r protocol.pcap -A

Understanding Protocol Structure: Look at the packets. What patterns do you see? Fixed headers? Variable lengths?

# Look for patterns in packets
# Fixed headers?
# Variable-length fields?
# Encryption?
# Checksums?

# Use Wireshark's "Follow TCP Stream" to see conversations

Identifying Message Boundaries: How do you know where one message ends and another begins?

Length prefixes?
Delimiters?
Fixed-size messages?
Keep-alive messages?

Reverse Engineering Protocol: Write code to parse what you captured. Scapy is great for this.

# Parse captured packets
from scapy.all import *

packets = rdpcap('protocol.pcap')

for packet in packets:
    if packet.haslayer(Raw):
        data = packet[Raw].load

        # Analyze structure
        # Look for repeating patterns
        # Identify fields based on context
        print(hexdump(data))

Binary Protocol Analysis¶

Protocols Have Patterns: Most protocols follow common patterns. Length prefixes, TLV structures, fixed formats.

Common Patterns: - Length-prefixed fields: first bytes indicate length - Type-length-value (TLV): type byte, length bytes, value - Fixed structures: always same size - Tagged unions: type discriminator followed by data

Tools for Protocol Analysis:

# Scapy (Python) - protocol manipulation
python3 -c "from scapy.all import *; help(IP)"

# Wireshark dissectors - create custom parsers
# Can write Lua dissectors to decode protocols

# binwalk - find embedded data structures
binwalk unknown_file

Archive Format Reversing¶

Archives Hide Files: ZIP, TAR, custom formats - they all store multiple files. Understand the structure to extract them.

Common Archive Formats:

# ZIP format
unzip -l archive.zip  # List contents
unzip -p archive.zip file.txt  # Extract specific file

# TAR format
tar -tf archive.tar  # List contents
tar -xf archive.tar  # Extract

# Custom formats
# Look for file headers within archive
# File count? File table? Individual files?

Understanding Archive Structure: Hex dump the archive. Look for metadata about files.

# Hexdump to see structure
hexdump -C archive | head -50

# Look for:
# - File count (usually at start)
# - File table (list of files with offsets/sizes)
# - Individual files (after table)

# Example ZIP structure:
# Local file header
# File data
# Central directory (at end)
# End of central directory record

Database Format Reversing¶

Databases Store Structured Data: SQLite is common, but custom formats exist. Reverse engineer to understand the schema.

SQLite Databases:

# SQLite has well-documented format
sqlite3 database.db ".schema"  # See structure
sqlite3 database.db ".tables"  # List tables
sqlite3 database.db "SELECT * FROM table;"

# But what if it's a custom format?
hexdump -C custom.db | head -20

Custom Database Formats: Analyze the binary structure. Look for headers, tables, records.

# Analyze structure
with open('custom.db', 'rb') as f:
    # Read header
    header = f.read(64)  # Usually has metadata

    # Look for table definitions
    # Usually stored as structures

    # Find data records
    # Usually fixed-size or length-prefixed

Configuration File Formats¶

Configs Store Settings: INI, JSON, binary configs - they all need parsing. Reverse engineer binary ones.

Common Config Formats:

# INI files
# [section]
# key=value

# JSON
# {"key": "value"}

# Binary configs
# Need to reverse engineer structure
hexdump -C config.bin

Reverse Engineering Config Formats: Change settings, compare files. Map changes to config values.

# Change config, compare binary
# Make small change, see what bytes change
# Map bytes to config values

# Example:
# Original config has "port=80"
# Change to "port=443"
# Compare binaries, find changed bytes
# Those bytes likely store port number

Image Format Reversing¶

Images Have Structure: PNG, JPEG, custom formats - understand the layout to read/write them.

Understanding Image Formats:

# PNG format (well documented)
# IHDR chunk: width, height, bit depth, color type
# IDAT chunks: compressed image data
# IEND chunk: end marker

# If custom format:
hexdump -C image.custom | head -20

Creating Custom Format Readers: Write code to parse the custom format once you understand it.

def read_custom_image(filename):
    with open(filename, 'rb') as f:
        magic = f.read(4)
        width = struct.unpack('<I', f.read(4))[0]
        height = struct.unpack('<I', f.read(4))[0]
        pixel_data = f.read(width * height * 3)  # RGB

        # Convert to standard format
        from PIL import Image
        img = Image.frombytes('RGB', (width, height), pixel_data)
        return img

Understanding Cryptography in Binaries - When Software Uses Crypto¶

Crypto Shows Up Everywhere: Programs encrypt data, verify integrity, authenticate users. Understanding crypto usage is key to reverse engineering. Not breaking it - understanding how it works.

Finding Crypto in Code: Start by identifying what crypto libraries and functions are used.

Identifying Cryptographic Code:

# What crypto libraries are linked?
ldd binary | grep -i crypto
# Might show: libcrypto, libssl, custom crypto libs

# What crypto functions are imported?
objdump -T binary | grep -i crypto
# Shows functions like: AES_encrypt, RSA_public_encrypt, SHA256_Init

Finding Crypto Constants:

# Look for algorithm names in strings
strings binary | grep -E "(AES|RSA|SHA|MD5|DES|Blowfish|Twofish)"

# Look for magic numbers
# AES S-box values, RSA key structures, etc.

Understanding Crypto Usage:

# Where is crypto called?
objdump -d binary | grep -B 5 -A 10 "call.*crypto"

# What's the context?
# Encryption before network send?
# Decryption after file read?
# Hashing for integrity check?

Common Cryptographic Patterns¶

Hashing Functions (Integrity Checks):

# MD5, SHA1, SHA256 usage
# Usually you'll see:
call SHA256_Init
call SHA256_Update  # Multiple times for data
call SHA256_Final   # Get hash result

Symmetric Encryption (Data Protection):

# AES encryption/decryption
call AES_set_encrypt_key
call AES_encrypt
# Or AES_cbc_encrypt for CBC mode

Asymmetric Encryption (Key Exchange):

# RSA operations
call RSA_public_encrypt
call RSA_private_decrypt
# Or RSA_sign/RSA_verify for signatures

Understanding Encryption Workflows¶

Data Encryption Pattern: 1. Generate or load key 2. Initialize encryption algorithm 3. Encrypt data 4. Store/transmit encrypted data

In Reverse Engineering:

# Find where encryption happens
# Trace key generation/loading
# Understand encryption mode (CBC, CTR, GCM, etc.)
# Map encrypted data flow

Key Management:

# Where do keys come from?
# Hardcoded? (strings binary | grep -E "[0-9a-f]{32,}")
# Derived? (PBKDF2, key derivation functions)
# Exchanged? (RSA, Diffie-Hellman)

Hardcoded Keys and Secrets¶

Finding Hardcoded Keys:

# Look for long hex strings
strings binary | grep -E "[0-9a-f]{32,}"

# Common key lengths:
# AES-128: 16 bytes (32 hex chars)
# AES-256: 32 bytes (64 hex chars)
# RSA keys: Much longer (2048+ bits)

Why Keys Might Be Hardcoded: - Convenience (not secure, but common) - License validation - Configuration encryption - Sometimes accidental (should use key derivation)

Understanding Key Usage:

# In debugger, see when keys are used
gdb ./binary
(gdb) break AES_set_encrypt_key
(gdb) run
# When breakpoint hits, examine key parameter
(gdb) x/16bx $rdi  # First 16 bytes of key (AES-128)

Custom Cryptographic Algorithms¶

Custom Crypto Happens: Sometimes software uses custom algorithms instead of standard ones.

Identifying Custom Crypto:

# Look for:
# - Non-standard operations
# - Complex bit manipulation
# - Substitution boxes (S-boxes)
# - Permutation networks
# - Custom round functions

Reverse Engineering Custom Crypto: 1. Identify algorithm structure 2. Understand operations 3. Map input/output 4. Understand key schedule 5. Reimplement if needed

Common Custom Crypto Patterns: - XOR-based encryption (weak but simple) - Substitution-permutation networks - Feistel networks - Stream ciphers

Understanding Hash Functions¶

MD5/SHA Usage:

# Often used for:
# - File integrity checks
# - Password hashing (older systems)
# - Checksums
# - Deduplication

# Identify by constants:
# MD5: 0x67452301, 0xefcdab89, 0x98badcfe, 0x10325476
# SHA1: Similar but different constants

Hash Functions in Code:

# MD5 pattern:
call MD5_Init
call MD5_Update  # Data chunks
call MD5_Final   # Get hash

# Result is usually compared or stored
cmp hash_result, expected_hash

Digital Signatures¶

RSA Signatures:

# Signing:
call RSA_sign
# Creates signature of data

# Verification:
call RSA_verify
# Checks signature validity

Understanding Signature Verification: - Often used for license validation - Software update verification - Code integrity checks

Practical Crypto Analysis Workflow¶

Step 1: Identify Crypto Usage

# Find crypto libraries
ldd binary
strings binary | grep -i crypto

Step 2: Understand Context

# What's being encrypted/hashed?
# Trace data flow
# Find encryption/decryption points

Step 3: Understand Keys

# Where do keys come from?
# Hardcoded? Derived? Exchanged?
# When are they used?

Step 4: Map the Flow

# Document the crypto workflow
# Input -> [Encryption/Hashing] -> Output
# Understanding helps you work with the data

The Goal: Not to break crypto (that's cryptanalysis, different field), but to understand how the software uses it. This helps you: - Understand data formats (encrypted vs plaintext) - Work with encrypted data if needed - Understand authentication mechanisms - Interface with crypto-protected features

Network Protocol Reversing¶

Protocol Analysis¶

Packet Capture¶

# Capture traffic
tcpdump -i eth0 -w capture.pcap

# Filter by protocol
tcpdump -i eth0 tcp port 80 -w http.pcap

Protocol Dissection¶

# Wireshark analysis
wireshark capture.pcap

# Tshark scripting
tshark -r capture.pcap -T fields -e tcp.srcport -e tcp.dstport

Custom Protocol Reversing¶

State Machine Analysis¶

# Identify protocol states
# Message formats
# Error handling

Grammar Inference¶

# Learn protocol format
# Generate parser
# Fuzz testing

Protocol Implementation¶

Client/Server Creation¶

# Scapy for protocol testing
from scapy.all import *

# Craft custom packets
packet = IP(dst="target")/TCP(dport=1234)/Raw(load="data")
send(packet)

Protocol Fuzzing¶

# Protocol-aware fuzzing
# State model fuzzing
# Grammar-based fuzzing

Advanced Analysis Techniques¶

When Basic Analysis Isn't Enough: Sometimes you need advanced techniques. Symbolic execution, binary instrumentation, automated analysis - these help with complex binaries.

Symbolic Execution¶

Execute with Symbols: Instead of concrete values, use symbols. Find paths, solve constraints, understand complex logic.

Angr Usage:

import angr

# Load binary
proj = angr.Project('binary')

# Create state
state = proj.factory.entry_state()

# Symbolic execution
simgr = proj.factory.simgr(state)
simgr.explore(find=lambda s: b'flag' in s.posix.dumps(1))

# Get flag
print(simgr.found[0].posix.dumps(1))

Constraint Solving:

# Solve for input
solver = state.solver
solution = solver.eval(input_var)

Binary Instrumentation¶

Runtime Modification: Insert code into running binaries. Trace execution, hook functions, analyze behavior.

DynamoRIO:

# Instrument binary
drrun -c client.dll -- binary args

# Custom instrumentation
# Memory tracing
# Function hooking

Intel PIN:

# PIN tool
pin -t tool.so -- binary

# Instruction counting
# Memory access tracing

Code Coverage Analysis¶

What Code Runs: Understand which parts of the binary execute. Find dead code, missed paths, optimization opportunities.

GCOV:

# Compile with coverage
gcc -fprofile-arcs -ftest-coverage binary.c

# Run and analyze
./a.out
gcov binary.c

Dynamic Coverage:

# AFL coverage
afl-showmap -o coverage ./binary < input

# GDB coverage
# Custom scripts

Automated Analysis¶

Scale Analysis: Automate reverse engineering tasks. Process many binaries, find patterns, generate reports.

Binary Analysis Frameworks:

# Binary Ninja API
from binaryninja import *

# Load and analyze
bv = BinaryViewType.get_view_of_file('binary')
bv.update_analysis()

# Function analysis
for func in bv.functions:
    print(func.name)

Custom Analysis Tools:

# IDA Python
import idaapi

# Function enumeration
for ea in Functions():
    print(GetFunctionName(ea))

# Cross-reference analysis
for xref in XrefsTo(ea):
    print(xref.type, hex(xref.frm))

Automation and Scripting¶

Scale Your Analysis: Reverse engineering one binary is fine, but what about 100? Automation lets you process binaries at scale, find patterns across codebases, and generate reports automatically.

Python Automation¶

Script Everything: Python is perfect for RE automation. Libraries like pefile, capstone, and angr let you build powerful analysis tools.

Binary Analysis Scripts:

import pefile
import capstone

# PE analysis
pe = pefile.PE('binary.exe')
for section in pe.sections:
    print(section.Name.decode(), hex(section.VirtualAddress))

# Disassembly
md = capstone.Cs(capstone.CS_ARCH_X86, capstone.CS_MODE_64)
for i in md.disasm(code, 0x1000):
    print(f"{i.address:x}: {i.mnemonic} {i.op_str}")

Automated Vulnerability Scanning:

# Find common vulnerabilities
def scan_binary(filename):
    # Check for unsafe functions
    unsafe_funcs = ['strcpy', 'sprintf', 'gets']
    # Check for protections
    # Generate report
    return report

GDB Automation¶

Extend GDB: GDB's Python API lets you create custom commands and automate debugging tasks.

Custom Commands:

import gdb

class AnalyzeHeap(gdb.Command):
    def __init__(self):
        super().__init__("analyze_heap", gdb.COMMAND_USER)

    def invoke(self, arg, from_tty):
        # Heap analysis code
        gdb.execute("info proc mappings")
        # More analysis...

AnalyzeHeap()

Automated Debugging:

# Script to find buffer overflows
def find_overflow():
    # Set watchpoints
    # Run with fuzz input
    # Catch crashes
    # Report findings

Bash Automation¶

Command Line Power: Bash scripts let you chain tools together for automated analysis pipelines.

Analysis Pipeline:

#!/bin/bash

analyze_binary() {
    local binary=$1

    echo "=== File Analysis ==="
    file "$binary"

    echo "=== Strings ==="
    strings "$binary" | head -20

    echo "=== Functions ==="
    nm "$binary" 2>/dev/null | head -20

    echo "=== Disassembly ==="
    objdump -d "$binary" | head -50
}

analyze_binary "$1"

Batch Processing:

# Analyze all binaries in directory
for binary in *; do
    if file "$binary" | grep -q "ELF"; then
        echo "Analyzing $binary"
        ./analyze.sh "$binary" > "${binary}.analysis"
    fi
done

Tool Integration¶

Connect Everything: Modern RE involves multiple tools. Integration lets you combine their strengths.

Ghidra Scripting:

import ghidra.app.script.GhidraScript;

public class AnalyzeScript extends GhidraScript {
    @Override
    public void run() throws Exception {
        println("Starting analysis");

        // Analysis code
        FunctionIterator functions = currentProgram.getFunctionManager().getFunctions(true);
        for (Function function : functions) {
            println("Function: " + function.getName());
        }
    }
}

Radare2 Scripting:

# Radare2 script
#!/bin/bash

r2 -i - <<EOF
aaa
afl
iz
EOF

CI/CD Integration:

# Automated analysis in build pipeline
# Check for vulnerabilities
# Generate security reports
# Block insecure builds

Binary Analysis Tools Deep Dive¶

Tool Comparison Matrix¶

Tool	Platform	Cost	Best For	Limitations
IDA Pro	Windows/Linux/macOS	Commercial	Professional RE	Expensive
Ghidra	cross platform	Free	NSA-developed, powerful	Steep learning curve
Binary Ninja	cross platform	Commercial	Cloud-based, modern UI	Subscription model
Radare2	cross platform	Free	command line, scripting	Complex interface
Cutter	cross platform	Free	Radare2 GUI frontend	Less mature
Hopper	macOS	Commercial	macOS binaries	macOS only

Advanced Ghidra Features¶

Function Signature Analysis¶

# Apply function signatures
File -> Apply Function Signatures

# Create custom signatures
Tools -> Function ID -> Create Function Signature

# Signature libraries
# Download and apply FLIRT signatures

Version Tracking¶

# Compare binary versions
File -> Open -> Version Tracking

# Diff binaries
Tools -> Diff Binary

# Patch analysis
# Track changes between versions

Collaboration Features¶

# Shared projects
File -> New Project -> Shared Project

# Multi-user analysis
# Real-time collaboration

Radare2 Advanced Usage¶

Visual Mode¶

# Enter visual mode
V

# Navigate
h/j/k/l - movement
p - toggle views
q - quit visual mode

# Graph navigation
:ag main

Emulation¶

# Emulate code
aei  # Initialize ESIL VM
aeim  # Initialize memory
aeip  # Initialize program counter

# Step emulation
aes  # Step
aec  # Continue

Custom Plugins¶

# Radare2 plugin in Python
from r2lang import *

def r2plugin():
    def hello():
        print("Hello from plugin")

    return {"name": "hello", "commands": {"hello": hello}}

Binary Ninja Features¶

Intermediate Languages¶

# MLIL (Medium Level IL)
# HLIL (High Level IL)
# LLIL (Low Level IL)

# Switch views
View -> IL Views -> HLIL

Scripting API¶

from binaryninja import *

# Load binary
bv = BinaryViewType.get_view_of_file('binary')

# Analyze function
func = bv.get_function_at(0x4005c0)
for block in func.basic_blocks:
    print(block)

Cutter (Radare2 GUI)¶

Interface Overview¶

# Main panels
# Disassembly, Decompiler, Hex, Strings, Functions

# Graph view
# Control flow graphs
# Call graphs

Analysis Automation¶

# Auto-analysis
File -> New -> Load Binary -> Analyze

# Custom analysis scripts
# Python scripting integration

Hopper Disassembler¶

macOS Binary Analysis¶

# Open binary
File -> Open -> Executable

# Decompilation
Right-click -> Decompile

# Cross-references
Right-click -> References

Scripting¶

# Hopper Python scripting
doc = Document.getCurrentDocument()
proc = doc.getCurrentProcedure()

# Analyze procedure
for instr in proc.getInstructions():
    print(instr)

Specialized Tools¶

BinDiff (Binary Diffing)¶

# Compare binaries
# IDA plugin or standalone
# Function matching
# Difference visualization

Diaphora (Binary Diffing)¶

# Open source binary diffing
# IDA plugin
# Advanced matching algorithms

YARA for Binary Analysis¶

# Create YARA rules
rule suspicious_strings {
    strings:
        $a = "cmd.exe"
        $b = "powershell"
    condition:
        $a or $b
}

# Scan binary
yara rule.yar binary

Cloud-Based Analysis¶

REMnux (Malware Analysis)¶

# Boot REMnux VM
# Pre-installed tools
# Malware analysis environment

FLARE-VM (Windows Malware)¶

# Windows VM with RE tools
# Chocolatey package manager
# Automated tool installation

Performance Optimization¶

Large Binary Analysis¶

# Selective analysis
# Focus on specific sections
# Use faster tools for triage

# Memory management
# Close unused views
# Use 64-bit tools

Automation for Scale¶

# Batch processing scripts
# Parallel analysis
# Result aggregation

real world Reverse Engineering Workflows - How I Actually Do It¶

Let me share some real workflows I've used. These aren't case studies about breaking things - these are about understanding things. The techniques are the same, but the mindset is different.

Workflow 1: Understanding a Legacy command line Tool¶

The Situation: You've got this old utility program. Nobody remembers who wrote it, but it's critical for some process. The source code is long gone. You need to understand what it does and how it works.

Step 1: Initial Triage

# What are we dealing with?
file mystery_tool
# ELF 64-bit LSB executable, x86-64

# What does it link to?
ldd mystery_tool
# Shows it uses libc, maybe some other libs

# What strings are in it?
strings mystery_tool | head -20
# Might show usage messages, error messages, file paths

Step 2: Run It and Observe

# Try running it
./mystery_tool
# Maybe it shows usage, maybe it errors

# Try with arguments
./mystery_tool --help
./mystery_tool -h

# See what it does
strace ./mystery_tool arg1 arg2 2>&1 | tail -50
# This tells you what files it reads, what syscalls it makes

Step 3: Understand the Logic

# Find main function
objdump -d mystery_tool | grep -A 30 "<main>:"

# Or in GDB
gdb ./mystery_tool
(gdb) disas main

# Follow function calls
# What functions does main call?
# What do those functions do?

Step 4: Document Findings As you understand things, write them down: - What arguments does it take? - What files does it read/write? - What's the algorithm? - What's the output format?

The Result: You now understand the tool. You can document it, modify it, or reimplement it if needed.

Workflow 2: Reversing a Game's Save File Format¶

The Situation: Old game, save files are in some custom format. You want to create a save editor or convert saves between versions.

Step 1: Get Sample Files

# Collect multiple save files
# Different game states
# Different versions if possible

# Compare files
diff save1.dat save2.dat | head -20
# See what changed

Step 2: Hexdump Analysis

# Look at structure
hexdump -C save1.dat | head -30

# Look for patterns
# Fixed-size headers?
# Repeated structures?
# Length prefixes?

Step 3: Modify and Test

# Make small changes in game
# Save file
# Compare before/after

# Change one thing at a time
# Map bytes to game data

Step 4: Write Parser

# Once you understand format
def parse_save_file(filename):
    with open(filename, 'rb') as f:
        # Parse based on your discoveries
        header = f.read(64)
        # ... parse structure ...
        return save_data

The Result: You can read and write the save format. You understand how the game stores data.

Workflow 3: Understanding a Proprietary Protocol¶

The Situation: You've got some software that talks to a server using a custom protocol. The protocol isn't documented. You want to interface with it or create a compatible implementation.

Step 1: Capture Traffic

# Run the software
# Capture network traffic
tcpdump -i any -w protocol.pcap port 12345

# Or use Wireshark GUI
wireshark &
# Filter for the port/protocol

Step 2: Analyze Packets

# Look at packet structure
tcpdump -r protocol.pcap -A -c 10

# Look for patterns
# Fixed headers?
# Variable length?
# Encryption?

Step 3: Map Actions to Packets

# Do action A in software
# See what packet is sent
# Do action B
# Compare packets

# Build up understanding:
# Action X -> Packet format Y

Step 4: Reverse the Protocol

# Reconstruct protocol format
class ProtocolParser:
    def parse_message(self, data):
        # Based on your analysis
        message_type = data[0]
        length = struct.unpack('>H', data[1:3])[0]
        payload = data[3:3+length]
        return {'type': message_type, 'data': payload}

The Result: You understand the protocol. You can communicate with the server or create compatible clients.

Workflow 4: Modifying a Binary (Ethically)¶

The Situation: You've got software you own/control. You want to modify its behavior. Maybe change a configuration limit, maybe patch a bug, maybe add a feature.

Step 1: Understand What to Change

# Find the code you want to modify
# In disassembly or debugger
gdb ./binary
(gdb) break function_name
(gdb) run
# Understand current behavior

Step 2: Plan the Modification

# Current code:
cmp rax, 100
jle continue
# Limit is 100

# Want to change to 200
# Need to change the comparison

Step 3: Apply Patch

# Using hex editor or radare2
r2 -w binary
# Navigate to address
# Write new instructions

Step 4: Test

# Run modified binary
./patched_binary
# Verify behavior changed correctly

The Result: Binary behaves as desired. You've successfully modified it.

Workflow 5: Understanding an Algorithm¶

The Situation: Program does some calculation. You want to understand the algorithm. Maybe it's a compression algorithm, maybe it's a game mechanic, maybe it's a data transformation.

Step 1: Identify the Algorithm Function

# Find where calculation happens
# Maybe it's called from multiple places
# Trace execution flow

Step 2: Understand Input/Output

# In debugger, see what goes in
(gdb) break algorithm_function
(gdb) run
(gdb) print $rdi  # Input parameter
(gdb) continue
(gdb) print $rax  # Output/return value

Step 3: Trace Through Logic

# Step through algorithm
(gdb) stepi
# Watch registers change
# Understand data transformations

Step 4: Reimplement

// Recreate algorithm based on understanding
int algorithm(int input) {
    // Your implementation based on reverse engineering
    return result;
}

The Result: You understand and can reimplement the algorithm. Maybe even improve it.

Common Patterns I've Learned¶

Start Broad, Get Specific: - Don't dive into details immediately - Understand overall flow first - Then dive into specific functions

Use Multiple Approaches: - Static analysis gives you structure - Dynamic analysis gives you behavior - Combine both for complete picture

Document As You Go: - Write notes constantly - Draw diagrams - You'll forget things otherwise

Test Your Understanding: - Try to predict what code will do - Run it and see if you're right - If wrong, revise understanding

Break Complex Things Down: - Don't try to understand everything at once - One function at a time - One basic block at a time if needed

Tools That Actually Help¶

For Static Analysis: - Ghidra - Amazing decompiler, free - Radare2 - Powerful, scriptable - objdump/readelf - Always useful

For Dynamic Analysis: - GDB - The classic, still great - strace/ltrace - See what program does - Wireshark - Network analysis

For Understanding: - Your brain - Most important tool - Pen and paper - Draw diagrams - Documentation - Write everything down

The Reality of Reverse Engineering¶

It Takes Time: Don't expect to understand a binary in an hour. Complex programs take days or weeks to fully understand.

You'll Get Stuck: Everyone does. That's normal. Take breaks. Ask questions. You'll figure it out.

Pattern Recognition Helps: The more you reverse, the faster you get. You start recognizing patterns.

Tools Are Great, But Understanding Is King: Decompilers save time, but you need to understand assembly. Tools help, they don't replace understanding.

It's Satisfying: When you finally understand that confusing piece of code? That feeling is worth it.

These workflows show the practical side of reverse engineering. It's not about breaking things - it's about understanding things. That's what makes reverse engineering valuable and interesting.

Resources and Further Reading¶

Tools Documentation¶

Official Docs¶

GDB manual: https://sourceware.org/gdb/current/onlinedocs/gdb/
Radare2 book: https://radare.gitbooks.io/radare2book/
Ghidra docs: https://ghidra-sre.org/
IDA Pro docs: https://hex-rays.com/products/ida/support/idadoc/
Nightmare : https://guyinatuxedo.github.io

Reverse Engineering - Taking Things Apart to Understand How They Work¶

Table of Contents¶

Foundations¶

Setup and Environment¶

File Analysis¶

Identification Techniques¶

Static Analysis¶

Dynamic Analysis¶

Disassembly and Decompilation¶

Binary Modification¶

Understanding Obfuscation¶

Protocols and Formats¶

Cryptography¶

Firmware and Embedded¶

Workflows and Methodology¶

Advanced Techniques¶

Automation¶

Tools Deep Dive¶

Resources¶

Introduction to Reverse Engineering - Taking Things Apart to Understand How They Work¶

What Actually Is Reverse Engineering?¶

Why Bother Learning This?¶

Core Principles - The Mindset That Actually Works¶

Linux Advantages - Why I Prefer Linux for RE¶

CPU Architecture and Assembly Fundamentals¶

x86/x64 Architecture Overview¶

CPU Components¶

Von Neumann vs Harvard Architecture¶

Registers¶

General Purpose Registers (64-bit) - Your New Best Friends¶

Special Purpose Registers - The Important Ones¶

Register Usage in Functions - The Calling Convention¶

x86/x64 Architecture Deep Dive - The Fun Stuff (Really, I Promise)¶

Control Flow and Instruction Pointer (RIP) - Where Are We Going?¶

The Stack and its Registers (RSP, RBP) - Memory's Filing Cabinet¶

Flags Register (RFLAGS) - The CPU's Mood Ring¶

Practical Assembly Example: C to Assembly - See It in Action¶

Simple C Program - Let's Start Easy¶

Disassembly Analysis - Line by Line, Let's Figure This Out¶

Introduction to Dynamic Analysis with GDB - Your New Best Friend¶

Essential GDB Commands - The Ones You'll Actually Use¶

Memory Organization - Where Your Data Actually Lives¶

Memory Layout (Linux Process) - The Big Picture¶

Virtual Memory - The Illusion That Makes Everything Work¶

Assembly Language Basics - Learning to Read the Language of the CPU¶

Instruction Format - The Grammar Rules¶

Data Movement Instructions - Moving Stuff Around¶

Arithmetic Instructions - Math Time¶

Logic Instructions - Bit Twiddling¶

Control Flow Instructions - Making Decisions¶

Stack Operations - The Filing System¶

Function Calls and Stack Frames - The House of Cards¶

Calling Convention - The Rules Everyone Follows¶

Stack Frame Layout - What's Actually in Memory¶

System Calls - Talking to the Operating System¶

Linux System Calls - The Interface¶

Common System Calls - The Ones You'll See Most¶

ARM Architecture¶

ARM Registers¶

ARM Instruction Set¶

Endianness¶

Little Endian vs Big Endian¶

CPU Modes and Rings¶

x86 Protection Rings¶

ARM Exception Levels¶

Cache and Memory Hierarchy¶

CPU Cache Levels¶

Memory Access Patterns¶

SIMD Instructions¶

SSE/AVX Instructions¶

CPU Microarchitecture¶

Pipeline Stages¶

Hazards¶

Performance Considerations¶

Instruction Latency and Throughput¶

Optimization Techniques¶

Linux Environment Setup¶

Essential Tool Installation¶

Core Analysis Tools¶

Advanced Tools¶