File system race conditions

What Are Race Conditions and Why Do They Occur

Race conditions are timing-dependent bugs that occur when multiple processes or operations access shared resources concurrently without proper synchronization. In file systems, these happen because file operations are not atomic - they consist of multiple steps that can be interrupted.

How Race Conditions Work in Operating Systems

Non-Atomic Operations: File operations like "check permissions then open file" are multiple system calls that can be interrupted between steps.

Process Scheduling: The OS can interrupt any process between system calls, creating timing windows for exploitation.

Shared Resources: Multiple processes can access the same files simultaneously without coordination.

Time Windows: Small gaps between operations create opportunities for interference by other processes.

The TOCTOU Problem (Time of Check, Time of Use)

Program A:                    Attacker:
1. access("/tmp/file", R_OK)  
2. [INTERRUPTED] -----------> rm /tmp/file; ln -s /etc/passwd /tmp/file
3. open("/tmp/file", O_RDWR)  [Now opens /etc/passwd instead]

Why Race Conditions Are Hard to Detect

Timing Dependent: Only occur under specific timing conditions that may be rare in normal operation.

Intermittent: May work correctly most of the time, failing only under specific load conditions.

Platform Specific: Depend on CPU speed, system load, and process scheduling policies.

Appear Random: Often dismissed as "weird bugs" rather than recognized as security issues.

Common Misconceptions

• "Fast operations can't be raced" - Even microsecond gaps can be exploited with proper timing

• "Only affects legacy programs" - Modern applications frequently contain race conditions

• "Only theoretical attacks" - Race conditions are actively exploited in real-world attacks

Race Condition Detection and Analysis

Understanding Detection Challenges

Race conditions are notoriously difficult to detect because:

• Timing Sensitivity: They only manifest under specific timing conditions

• Observer Effect: Monitoring tools can change timing and mask the bug

• Load Dependency: System load affects whether races occur

• False Negatives: Tests may pass even when vulnerabilities exist

Basic Detection Using Standard Tools

Step 1: System Call Analysis with strace

Monitor file system operations to identify TOCTOU patterns:

# Basic strace analysis for file operations
strace -f -e trace=file -T program 2>&1 | tee strace_output.txt

# Look for access/stat followed by open patterns
grep -E "(access|stat)" strace_output.txt > checks.txt
grep -E "(open|openat)" strace_output.txt > opens.txt

# Manual review for common file paths
echo "Files checked for permissions:"
awk -F'"' '/access|stat/ {print $2}' checks.txt | sort -u

echo "Files opened:"
awk -F'"' '/open/ {print $2}' opens.txt | sort -u

Step 2: Using ltrace for Library Call Analysis

Monitor library function calls that may contain race conditions:

# Monitor file-related library calls
ltrace -e 'access,stat,fopen,open' program 2>&1 | tee ltrace_output.txt

# Look for patterns where file checks precede file usage
grep -B2 -A2 "access\|stat" ltrace_output.txt

Step 3: File System Monitoring with inotifywait

Track file system events during program execution:

# Monitor /tmp directory for file creation/access patterns
inotifywait -m /tmp -e create,open,access --format '%T %w%f %e' --timefmt='%H:%M:%S' &
INOTIFY_PID=$!

# Run target program
./vulnerable_program

# Stop monitoring
kill $INOTIFY_PID

# Review the timing of file operations for potential race windows

Step 4: Process Monitoring with ps and lsof

Observe file descriptors and process behavior:

# Monitor open file descriptors during execution
PID=$(pgrep vulnerable_program)
watch -n 0.1 "lsof -p $PID"

# Check for temporary file usage patterns
lsof -p $PID | grep "/tmp"

Advanced Detection with Specialized Tools

Using Valgrind's Helgrind for Race Detection

Helgrind can detect some types of race conditions:

# Run program under Helgrind to detect races
valgrind --tool=helgrind ./program

# Focus on file-related race conditions
valgrind --tool=helgrind --track-lockorders=no ./program 2>&1 | grep -i "file\|race"

Using AddressSanitizer with ThreadSanitizer

For programs you can compile:

# Compile with ThreadSanitizer
gcc -fsanitize=thread -g program.c -o program_tsan

# Run to detect race conditions
./program_tsan 2>&1 | grep -A5 -B5 "WARNING: ThreadSanitizer"

Static Analysis with Cppcheck

For source code analysis:

# Check for TOCTOU vulnerabilities
cppcheck --enable=all --inconclusive program.c 2>&1 | grep -i "time.*check.*use\|toctou\|race"

# Look for file operation patterns
cppcheck --enable=all program.c 2>&1 | grep -i "access.*open\|stat.*fopen"

Using Flawfinder for Static Analysis

# Scan for race condition vulnerabilities
flawfinder --html program.c | grep -A3 -B3 "race\|TOCTOU"

# Focus on file operations
flawfinder program.c | grep -E "(access|stat|fopen|open)" -A2 -B2

Race Condition Techniques

Time of Check, Time of Use (TOCTOU)

Classic TOCTOU Detection:

# Monitor for access() followed by open() pattern
strace -e trace=access,open program 2>&1 | grep -B1 -A1 "access.*succeeded"

TOCTOU with File Permissions:

# Identify programs checking file permissions before access
strace -e trace=access,stat,open program 2>&1 | \
  awk '/access.*F_OK/ {check=$0; getline; if(/open/) print "POTENTIAL TOCTOU:", check, $0}'

Temporary File Race Conditions

Detecting Predictable Temporary Files:

# Monitor temporary file creation patterns
strace -e trace=open,creat program 2>&1 | grep "/tmp\|/var/tmp"

# Use lsof to see open temporary files
lsof | grep "/tmp" | grep "$(pgrep program)"

Finding Insecure Temporary Operations:

# Check for mktemp usage (secure)
ltrace -e mktemp program

# Check for insecure temp file creation
strace program 2>&1 | grep -E "open.*O_CREAT.*tmp"

Symbolic Link Race Conditions

Monitoring for Symlink Vulnerabilities:

# Track symbolic link operations
strace -e trace=readlink,symlink,lstat program 2>&1

# Monitor file operations that might be symlink-exploitable  
inotifywait -m /tmp -e create,modify | grep -v "\.tmp\."

Detecting Directory Traversal Risks:

# Check for directory operations before file access
strace -e trace=openat,chdir,fchdir program 2>&1 | grep -B2 -A2 "chdir"

File Descriptor Race Conditions

Monitoring File Descriptor Usage:

# Track file descriptor operations
strace -e trace=open,close,dup,dup2 program 2>&1

# Monitor concurrent access to same files
lsof -r 0.1 +D /tmp | grep program

Real-World Race Condition Examples

Example 1: Backup Script TOCTOU

Detection:

# Analyze backup script behavior
strace -e trace=file backup_script 2>&1 | grep -E "(access|stat).*open" -B1 -A1

Pattern Recognition:

• Script checks file permissions with access()

• Later opens file with elevated privileges

• Time gap allows symlink attack

Example 2: Log Rotation Vulnerability

Analysis with auditd:

# Monitor file operations during log rotation
auditctl -w /var/log -p wa -k logrotate
ausearch -k logrotate | grep -A3 -B3 "syscall=open"

Using journalctl for service analysis:

# Monitor systemd service file operations
journalctl -u logrotate.service -f &
strace -p $(pgrep logrotate) -e trace=file

Example 3: Service Initialization Race

Monitoring with systemd:

# Check service startup file operations
systemd-analyze plot > bootup.svg
systemctl status --no-pager -l service_name

# Monitor file creation during service start
inotifywait -m /tmp -e create &
systemctl restart vulnerable_service

Exploitation Testing

Manual Race Condition Testing

Basic TOCTOU Test:

# Create initial file
echo "safe content" > /tmp/testfile

# Monitor for program access
strace -e trace=access program &

# When access() is called, quickly replace file
rm /tmp/testfile; ln -s /etc/passwd /tmp/testfile

Timing-Based Testing:

# Use watch to monitor file operations
watch -n 0.1 'ls -la /tmp/target*'

# In another terminal, run the vulnerable program repeatedly
for i in {1..100}; do ./vulnerable_program; done

Automated Testing with Existing Tools

Using race-condition-exploit.py (if available):

# Generic TOCTOU exploitation framework
python race-exploit.py --target /tmp/vulnerable_file --link /etc/passwd

Using fswatch for cross-platform monitoring:

# Monitor file system changes
fswatch -o /tmp | xargs -I {} echo "File change detected at $(date)"

Key Operational Considerations

Success Indicators

• Predictable file operations by privileged programs

• Temporary files created in world-writable directories

• TOCTOU patterns detected in strace output

• Successful symlink manipulation during race windows

• File content modification through timing attacks

Common Failure Points

• Atomic file operations preventing race conditions

• Proper file locking mechanisms (flock, fcntl)

• Unpredictable file names or secure temporary directories

• Fast execution leaving no exploitable timing window

• File system monitoring detecting manipulation attempts

Detection Tool Limitations

Important Notes:

• These detection methods may produce false positives

• Manual verification is required for all findings

• System load and timing affect detection reliability

• Tools may alter program timing and mask vulnerabilities

• Not all race conditions are detectable through static analysis

Best Practices for Analysis

1. Combine multiple detection methods for comprehensive coverage

2. Manual review of strace/ltrace output is essential

3. Test under various load conditions to trigger race windows

4. Focus on privileged programs for high-impact vulnerabilities

5. Verify findings through controlled exploitation attempts

Race conditions require precise timing and deep understanding of program behavior, but can provide significant privilege escalation opportunities when successfully identified and exploited.