# File system race conditions

## What Are Race Conditions and Why Do They Occur

Race conditions are timing-dependent bugs that occur when multiple processes or operations access shared resources concurrently without proper synchronization. In file systems, these happen because file operations are not atomic - they consist of multiple steps that can be interrupted.

#### How Race Conditions Work in Operating Systems

**Non-Atomic Operations**: File operations like "check permissions then open file" are multiple system calls that can be interrupted between steps.

**Process Scheduling**: The OS can interrupt any process between system calls, creating timing windows for exploitation.

**Shared Resources**: Multiple processes can access the same files simultaneously without coordination.

**Time Windows**: Small gaps between operations create opportunities for interference by other processes.

#### The TOCTOU Problem (Time of Check, Time of Use)

```
Program A:                    Attacker:
1. access("/tmp/file", R_OK)  
2. [INTERRUPTED] -----------> rm /tmp/file; ln -s /etc/passwd /tmp/file
3. open("/tmp/file", O_RDWR)  [Now opens /etc/passwd instead]
```

#### Why Race Conditions Are Hard to Detect

**Timing Dependent**: Only occur under specific timing conditions that may be rare in normal operation.

**Intermittent**: May work correctly most of the time, failing only under specific load conditions.

**Platform Specific**: Depend on CPU speed, system load, and process scheduling policies.

**Appear Random**: Often dismissed as "weird bugs" rather than recognized as security issues.

#### Common Misconceptions

* "Fast operations can't be raced" - Even microsecond gaps can be exploited with proper timing
* "Only affects legacy programs" - Modern applications frequently contain race conditions
* "Only theoretical attacks" - Race conditions are actively exploited in real-world attacks

## Race Condition Detection and Analysis

#### Understanding Detection Challenges

Race conditions are notoriously difficult to detect because:

* **Timing Sensitivity**: They only manifest under specific timing conditions
* **Observer Effect**: Monitoring tools can change timing and mask the bug
* **Load Dependency**: System load affects whether races occur
* **False Negatives**: Tests may pass even when vulnerabilities exist

#### Basic Detection Using Standard Tools

**Step 1: System Call Analysis with strace**

Monitor file system operations to identify TOCTOU patterns:

```bash
# Basic strace analysis for file operations
strace -f -e trace=file -T program 2>&1 | tee strace_output.txt

# Look for access/stat followed by open patterns
grep -E "(access|stat)" strace_output.txt > checks.txt
grep -E "(open|openat)" strace_output.txt > opens.txt

# Manual review for common file paths
echo "Files checked for permissions:"
awk -F'"' '/access|stat/ {print $2}' checks.txt | sort -u

echo "Files opened:"
awk -F'"' '/open/ {print $2}' opens.txt | sort -u
```

**Step 2: Using ltrace for Library Call Analysis**

Monitor library function calls that may contain race conditions:

```bash
# Monitor file-related library calls
ltrace -e 'access,stat,fopen,open' program 2>&1 | tee ltrace_output.txt

# Look for patterns where file checks precede file usage
grep -B2 -A2 "access\|stat" ltrace_output.txt
```

**Step 3: File System Monitoring with inotifywait**

Track file system events during program execution:

```bash
# Monitor /tmp directory for file creation/access patterns
inotifywait -m /tmp -e create,open,access --format '%T %w%f %e' --timefmt='%H:%M:%S' &
INOTIFY_PID=$!

# Run target program
./vulnerable_program

# Stop monitoring
kill $INOTIFY_PID

# Review the timing of file operations for potential race windows
```

**Step 4: Process Monitoring with ps and lsof**

Observe file descriptors and process behavior:

```bash
# Monitor open file descriptors during execution
PID=$(pgrep vulnerable_program)
watch -n 0.1 "lsof -p $PID"

# Check for temporary file usage patterns
lsof -p $PID | grep "/tmp"
```

#### Advanced Detection with Specialized Tools

**Using Valgrind's Helgrind for Race Detection**

Helgrind can detect some types of race conditions:

```bash
# Run program under Helgrind to detect races
valgrind --tool=helgrind ./program

# Focus on file-related race conditions
valgrind --tool=helgrind --track-lockorders=no ./program 2>&1 | grep -i "file\|race"
```

**Using AddressSanitizer with ThreadSanitizer**

For programs you can compile:

```bash
# Compile with ThreadSanitizer
gcc -fsanitize=thread -g program.c -o program_tsan

# Run to detect race conditions
./program_tsan 2>&1 | grep -A5 -B5 "WARNING: ThreadSanitizer"
```

**Static Analysis with Cppcheck**

For source code analysis:

```bash
# Check for TOCTOU vulnerabilities
cppcheck --enable=all --inconclusive program.c 2>&1 | grep -i "time.*check.*use\|toctou\|race"

# Look for file operation patterns
cppcheck --enable=all program.c 2>&1 | grep -i "access.*open\|stat.*fopen"
```

**Using Flawfinder for Static Analysis**

```bash
# Scan for race condition vulnerabilities
flawfinder --html program.c | grep -A3 -B3 "race\|TOCTOU"

# Focus on file operations
flawfinder program.c | grep -E "(access|stat|fopen|open)" -A2 -B2
```

## Race Condition Techniques

#### Time of Check, Time of Use (TOCTOU)

**Classic TOCTOU Detection:**

```bash
# Monitor for access() followed by open() pattern
strace -e trace=access,open program 2>&1 | grep -B1 -A1 "access.*succeeded"
```

**TOCTOU with File Permissions:**

```bash
# Identify programs checking file permissions before access
strace -e trace=access,stat,open program 2>&1 | \
  awk '/access.*F_OK/ {check=$0; getline; if(/open/) print "POTENTIAL TOCTOU:", check, $0}'
```

#### Temporary File Race Conditions

**Detecting Predictable Temporary Files:**

```bash
# Monitor temporary file creation patterns
strace -e trace=open,creat program 2>&1 | grep "/tmp\|/var/tmp"

# Use lsof to see open temporary files
lsof | grep "/tmp" | grep "$(pgrep program)"
```

**Finding Insecure Temporary Operations:**

```bash
# Check for mktemp usage (secure)
ltrace -e mktemp program

# Check for insecure temp file creation
strace program 2>&1 | grep -E "open.*O_CREAT.*tmp"
```

#### Symbolic Link Race Conditions

**Monitoring for Symlink Vulnerabilities:**

```bash
# Track symbolic link operations
strace -e trace=readlink,symlink,lstat program 2>&1

# Monitor file operations that might be symlink-exploitable  
inotifywait -m /tmp -e create,modify | grep -v "\.tmp\."
```

**Detecting Directory Traversal Risks:**

```bash
# Check for directory operations before file access
strace -e trace=openat,chdir,fchdir program 2>&1 | grep -B2 -A2 "chdir"
```

#### File Descriptor Race Conditions

**Monitoring File Descriptor Usage:**

```bash
# Track file descriptor operations
strace -e trace=open,close,dup,dup2 program 2>&1

# Monitor concurrent access to same files
lsof -r 0.1 +D /tmp | grep program
```

## Real-World Race Condition Examples

#### Example 1: Backup Script TOCTOU

**Detection:**

```bash
# Analyze backup script behavior
strace -e trace=file backup_script 2>&1 | grep -E "(access|stat).*open" -B1 -A1
```

**Pattern Recognition:**

* Script checks file permissions with `access()`
* Later opens file with elevated privileges
* Time gap allows symlink attack

#### Example 2: Log Rotation Vulnerability

**Analysis with auditd:**

```bash
# Monitor file operations during log rotation
auditctl -w /var/log -p wa -k logrotate
ausearch -k logrotate | grep -A3 -B3 "syscall=open"
```

**Using journalctl for service analysis:**

```bash
# Monitor systemd service file operations
journalctl -u logrotate.service -f &
strace -p $(pgrep logrotate) -e trace=file
```

#### Example 3: Service Initialization Race

**Monitoring with systemd:**

```bash
# Check service startup file operations
systemd-analyze plot > bootup.svg
systemctl status --no-pager -l service_name

# Monitor file creation during service start
inotifywait -m /tmp -e create &
systemctl restart vulnerable_service
```

## Exploitation Testing

#### Manual Race Condition Testing

**Basic TOCTOU Test:**

```bash
# Create initial file
echo "safe content" > /tmp/testfile

# Monitor for program access
strace -e trace=access program &

# When access() is called, quickly replace file
rm /tmp/testfile; ln -s /etc/passwd /tmp/testfile
```

**Timing-Based Testing:**

```bash
# Use watch to monitor file operations
watch -n 0.1 'ls -la /tmp/target*'

# In another terminal, run the vulnerable program repeatedly
for i in {1..100}; do ./vulnerable_program; done
```

#### Automated Testing with Existing Tools

**Using race-condition-exploit.py (if available):**

```bash
# Generic TOCTOU exploitation framework
python race-exploit.py --target /tmp/vulnerable_file --link /etc/passwd
```

**Using fswatch for cross-platform monitoring:**

```bash
# Monitor file system changes
fswatch -o /tmp | xargs -I {} echo "File change detected at $(date)"
```

## Key Operational Considerations

#### Success Indicators

* Predictable file operations by privileged programs
* Temporary files created in world-writable directories
* TOCTOU patterns detected in strace output
* Successful symlink manipulation during race windows
* File content modification through timing attacks

#### Common Failure Points

* Atomic file operations preventing race conditions
* Proper file locking mechanisms (flock, fcntl)
* Unpredictable file names or secure temporary directories
* Fast execution leaving no exploitable timing window
* File system monitoring detecting manipulation attempts

#### Detection Tool Limitations

**Important Notes:**

* These detection methods may produce false positives
* Manual verification is required for all findings
* System load and timing affect detection reliability
* Tools may alter program timing and mask vulnerabilities
* Not all race conditions are detectable through static analysis

#### Best Practices for Analysis

1. **Combine multiple detection methods** for comprehensive coverage
2. **Manual review** of strace/ltrace output is essential
3. **Test under various load conditions** to trigger race windows
4. **Focus on privileged programs** for high-impact vulnerabilities
5. **Verify findings** through controlled exploitation attempts

Race conditions require precise timing and deep understanding of program behavior, but can provide significant privilege escalation opportunities when successfully identified and exploited.
