gf (Gf-Muster) Cheat Sheet¶
Überblick¶
gf ist ein Wrapper um grep, mit dem Sie schnell nach Mustern in Dateien mit vordefinierten Mustersets suchen können. Es ist besonders nützlich für Sicherheitsjagd, Sicherheitsforschung und Codeanalyse. Das Tool kommt mit integrierten Mustern für allgemeine Schwachstellen und Sicherheitsfragen und unterstützt kundenspezifische Muster-Erstellung für spezialisierte Jagdszenarien.
RECHT **Key Features*: Vordefinierte Sicherheitsmuster, benutzerdefinierte Musterunterstützung, schnelle grep-basierte Suche, Integration mit anderen Sicherheitswerkzeugen, erweiterte Musterbibliothek und automatisierungsfreundliche Ausgabe.
Installation und Inbetriebnahme¶
Zur Installation¶
```bash
Install gf via Go¶
go install github.com/tomnomnom/gf@latest
Verify installation¶
gf --help
Check version¶
gf --version
Install to specific location¶
GOBIN=/usr/local/bin go install github.com/tomnomnom/gf@latest
Update to latest version¶
go install github.com/tomnomnom/gf@latest ```_
Binärinstallation¶
```bash
Download latest release for Linux¶
wget https://github.com/tomnomnom/gf/releases/latest/download/gf-linux-amd64.tgz tar -xzf gf-linux-amd64.tgz sudo mv gf /usr/local/bin/
Download for macOS¶
wget https://github.com/tomnomnom/gf/releases/latest/download/gf-darwin-amd64.tgz tar -xzf gf-darwin-amd64.tgz sudo mv gf /usr/local/bin/
Download for Windows¶
wget https://github.com/tomnomnom/gf/releases/latest/download/gf-windows-amd64.zip unzip gf-windows-amd64.zip
Move gf.exe to PATH¶
Verify installation¶
gf --help ```_
Installation des Paketmanagers¶
```bash
Homebrew (macOS/Linux)¶
brew install gf
Arch Linux (AUR)¶
yay -S gf-bin
Snap package¶
sudo snap install gf
Build from source¶
git clone https://github.com/tomnomnom/gf.git cd gf go build -o gf main.go sudo mv gf /usr/local/bin/ ```_
Musterinstallation und Setup¶
```bash
Create gf configuration directory¶
mkdir -p ~/.gf
Install default patterns¶
git clone https://github.com/tomnomnom/gf.git /tmp/gf-repo cp /tmp/gf-repo/examples/*.json ~/.gf/
Install community patterns¶
git clone https://github.com/1ndianl33t/Gf-Patterns.git /tmp/gf-patterns cp /tmp/gf-patterns/*.json ~/.gf/
Install additional pattern collections¶
git clone https://github.com/dwisiswant0/gf-secrets.git /tmp/gf-secrets cp /tmp/gf-secrets/*.json ~/.gf/
List available patterns¶
gf -list
Verify pattern installation¶
ls -la ~/.gf/
Set up environment variables¶
echo 'export GF_PATTERNS_PATH=~/.gf' >> ~/.bashrc source ~/.bashrc ```_
Benutzerdefinierte Muster Erstellung¶
```bash
Create custom pattern directory¶
mkdir -p ~/.gf/custom
Create basic custom pattern¶
cat > ~/.gf/custom/api-keys.json << 'EOF' { "flags": "-HnriE", "patterns": [ "api[-]?key['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}", "secret[-]?key['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}", "access[-]?token['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}", "auth[-]?token['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}" ] } EOF
Create advanced custom pattern with multiple flags¶
cat > ~/.gf/custom/sql-injection.json << 'EOF' { "flags": "-HnriE", "patterns": [ | | | | "\b(union | select | insert | update | delete | drop | create | alter | exec | execute)\b.\b(from | where | order | group | having)\b", | | | | "'\s(or|and)\s*'?\d+'?\s*=\s*'?\d+'?", | | | | "\b(sleep | benchmark | waitfor)\s*\(", | | | | | | | | "\b(load_file | into\s+outfile | into\s+dumpfile)\b", | | | | | | | | "\b(information_schema | mysql | sys | performance_schema)\.", | | | | | | | | "\b(concat | group_concat | char | ascii | substring | mid | left | right)\s*\(" | | | | ] } EOF
Create XSS detection pattern¶
cat > ~/.gf/custom/xss-advanced.json << 'EOF' { "flags": "-HnriE", "patterns": [ "<\s*script[^>]>.?</\s*script\s*>", "javascript\s*:", | | | | "on(load | error | click | mouseover | focus | blur | change | submit)\s*=", | | | | | | | | "<\s*(img | iframe | object | embed | applet | meta | link | style)\s+[^>]>", | | | | | | | | "\b(alert | confirm | prompt | eval | setTimeout | setInterval)\s\(", | | | | | | | | "document\.(write | writeln | cookie | location | referrer)", | | | | | | | | "window\.(location | open | close | focus | blur)", | | | | | | | | "\b(innerHTML | outerHTML | insertAdjacentHTML)\s*=" | | | | ] } EOF
Create file inclusion pattern¶
cat > ~/.gf/custom/file-inclusion.json << 'EOF' { "flags": "-HnriE", "patterns": [ | | | | "\b(include | require | include_once | require_once)\s*\([^)]*\\(", | | | | | | | | "\\b(file_get_contents | readfile | fopen | file)\\s*\\([^)]*\\\)", | | | | "\.\.[\/\\]", | | | | "\b(etc\/passwd | etc\/shadow | boot\.ini | win\.ini)", | | | | | | | | "\b(proc\/self\/environ | proc\/version | proc\/cmdline)", | | | | "\b(WEB-INF|META-INF)\/", | | | | "\b(php:\/\/ | file:\/\/ | data:\/\/ | expect:\/\/)" | | | | ] } EOF
Create command injection pattern¶
cat > ~/.gf/custom/command-injection.json << 'EOF' { "flags": "-HnriE", "patterns": [ | | | | "\b(system | exec | shell_exec | passthru | eval | popen | proc_open)\s*\(", | | | | | | | | "\b(cmd | command | sh | bash | powershell | pwsh)\s*\(", | | | | "[;&|`$(){}\[\]]", | | | | "\b(cat | ls | dir | type | echo | whoami | id | uname | pwd)\b", | | | | | | | | "\b(nc | netcat | telnet | wget | curl | ping | nslookup)\b", | | | | "\\x[0-9a-fA-F]{2}", | | | | "\b(base64 | hex | url)\s*(encode | decode)" | | | | ] } EOF
Verify custom patterns¶
gf -list | grep custom ```_
Grundlegende Verwendung und Befehle¶
Einfache Mustersuche¶
```bash
Search for basic patterns in files¶
gf xss file.txt gf sqli *.php gf secrets config.json
Search in multiple files¶
gf api-keys *.js *.json *.yaml
Search recursively in directories¶
gf passwords -r /path/to/directory
Search with case sensitivity¶
gf -i secrets file.txt
Search and show line numbers¶
gf -n xss index.html
Search and show file names only¶
gf -l sqli *.php
Search and show context lines¶
gf -A 3 -B 3 secrets config.js
Search with custom flags¶
gf -E -i -n passwords *.txt ```_
Erweiterte Mustersuche¶
```bash
Combine multiple patterns¶
gf xss file.html && gf sqli file.php
Search with output redirection¶
gf secrets *.js > found_secrets.txt
Search and count matches¶
gf api-keys *.json | wc -l
Search with grep options¶
gf -v false-positive *.log
Search with custom delimiters¶
gf --delimiter=":" secrets file.txt
Search with pattern exclusions¶
| | | | gf secrets *.js | grep -v "test | example" | | | |
Search with multiple file types¶
find . -name ".js" -o -name ".json" | xargs gf api-keys
Search with time-based filtering¶
find . -mtime -7 -name "*.log" | xargs gf errors ```_
Integration von Pipeline¶
```bash
Use with other tools¶
cat urls.txt | gf xss echo "test.php?id=1" | gf sqli
Chain with curl¶
curl -s https://example.com | gf secrets
Use with subfinder and httpx¶
| | | | subfinder -d example.com | httpx | gf endpoints | | | |
Combine with waybackurls¶
| | | | echo "example.com" | waybackurls | gf xss | | | |
Use with paramspider¶
python3 paramspider.py -d example.com | gf sqli
Chain with nuclei¶
gf secrets *.js | nuclei -t exposures/
Use with meg¶
meg / domains.txt | gf secrets
Combine with hakrawler¶
| | | | echo "https://example.com" | hakrawler | gf endpoints | | | | ```_
Advanced Vulnerability Hunting¶
Benutzerdefinierte Jagd Workflows¶
```python
!/usr/bin/env python3¶
Advanced vulnerability hunting with gf patterns¶
import subprocess import json import os import threading import time from concurrent.futures import ThreadPoolExecutor, as_completed from pathlib import Path
class GfVulnerabilityHunter: def init(self, max_workers=10): self.max_workers = max_workers self.results = [] self.lock = threading.Lock() self.patterns_dir = os.path.expanduser("~/.gf")
def get_available_patterns(self):
"""Get list of available gf patterns"""
try:
result = subprocess.run(
['gf', '-list'],
capture_output=True, text=True
)
if result.returncode == 0:
patterns = result.stdout.strip().split('\n')
return [p.strip() for p in patterns if p.strip()]
else:
return []
except Exception as e:
print(f"Error getting patterns: {e}")
return []
def search_pattern(self, pattern, target_files, options=None):
"""Search for specific pattern in target files"""
if options is None:
options = []
try:
# Build gf command
cmd = ['gf'] + options + [pattern] + target_files
result = subprocess.run(
cmd,
capture_output=True, text=True,
timeout=300
)
search_result = {
'pattern': pattern,
'target_files': target_files,
'matches': [],
'match_count': 0,
'success': result.returncode == 0,
'error': result.stderr if result.returncode != 0 else None
}
if result.stdout:
matches = result.stdout.strip().split('\n')
search_result['matches'] = [m for m in matches if m.strip()]
search_result['match_count'] = len(search_result['matches'])
return search_result
except subprocess.TimeoutExpired:
return {
'pattern': pattern,
'target_files': target_files,
'matches': [],
'match_count': 0,
'success': False,
'error': 'Search timeout'
}
except Exception as e:
return {
'pattern': pattern,
'target_files': target_files,
'matches': [],
'match_count': 0,
'success': False,
'error': str(e)
}
def hunt_vulnerabilities(self, target_directory, patterns=None, file_extensions=None):
"""Hunt for vulnerabilities using multiple patterns"""
if patterns is None:
patterns = self.get_available_patterns()
if file_extensions is None:
file_extensions = ['*.js', '*.php', '*.py', '*.java', '*.jsp', '*.asp', '*.aspx', '*.json', '*.xml', '*.yaml', '*.yml', '*.conf', '*.config', '*.ini', '*.env']
# Find target files
target_files = []
for ext in file_extensions:
try:
result = subprocess.run(
['find', target_directory, '-name', ext, '-type', 'f'],
capture_output=True, text=True
)
if result.returncode == 0:
files = result.stdout.strip().split('\n')
target_files.extend([f for f in files if f.strip()])
except Exception as e:
print(f"Error finding files with extension {ext}: {e}")
if not target_files:
print("No target files found")
return []
print(f"Found {len(target_files)} target files")
print(f"Hunting with {len(patterns)} patterns")
# Search patterns concurrently
with ThreadPoolExecutor(max_workers=self.max_workers) as executor:
# Submit search tasks
future_to_pattern = {}
for pattern in patterns:
future = executor.submit(
self.search_pattern,
pattern,
target_files,
['-H', '-n', '-r']
)
future_to_pattern[future] = pattern
# Process results
for future in as_completed(future_to_pattern):
pattern = future_to_pattern[future]
try:
result = future.result()
with self.lock:
self.results.append(result)
if result['success'] and result['match_count'] > 0:
print(f"✓ {pattern}: {result['match_count']} matches")
else:
print(f"- {pattern}: No matches")
except Exception as e:
print(f"✗ Error with pattern {pattern}: {e}")
return self.results
def analyze_results(self):
"""Analyze hunting results and categorize findings"""
analysis = {
'total_patterns': len(self.results),
'successful_patterns': 0,
'total_matches': 0,
'high_risk_findings': [],
'medium_risk_findings': [],
'low_risk_findings': [],
'pattern_summary': {}
}
# Risk categorization
high_risk_patterns = [
'sqli', 'xss', 'rce', 'lfi', 'rfi', 'xxe', 'ssti',
'secrets', 'api-keys', 'passwords', 'tokens'
]
medium_risk_patterns = [
'debug', 'errors', 'endpoints', 'upload', 'redirect',
'cors', 'csrf', 'idor', 'path-traversal'
]
for result in self.results:
if result['success']:
analysis['successful_patterns'] += 1
analysis['total_matches'] += result['match_count']
# Categorize by risk
pattern = result['pattern'].lower()
risk_level = 'low'
if any(hrp in pattern for hrp in high_risk_patterns):
risk_level = 'high'
analysis['high_risk_findings'].append(result)
elif any(mrp in pattern for mrp in medium_risk_patterns):
risk_level = 'medium'
analysis['medium_risk_findings'].append(result)
else:
analysis['low_risk_findings'].append(result)
# Pattern summary
analysis['pattern_summary'][result['pattern']] = {
'matches': result['match_count'],
'risk_level': risk_level,
'success': result['success']
}
return analysis
def generate_report(self, output_file='gf_vulnerability_report.json'):
"""Generate comprehensive vulnerability hunting report"""
analysis = self.analyze_results()
report = {
'scan_info': {
'timestamp': time.strftime('%Y-%m-%d %H:%M:%S'),
'tool': 'gf (Gf patterns)',
'total_patterns_used': analysis['total_patterns'],
'successful_patterns': analysis['successful_patterns'],
'total_matches': analysis['total_matches']
},
'risk_summary': {
'high_risk_findings': len(analysis['high_risk_findings']),
'medium_risk_findings': len(analysis['medium_risk_findings']),
'low_risk_findings': len(analysis['low_risk_findings'])
},
'detailed_findings': {
'high_risk': analysis['high_risk_findings'],
'medium_risk': analysis['medium_risk_findings'],
'low_risk': analysis['low_risk_findings']
},
'pattern_summary': analysis['pattern_summary'],
'raw_results': self.results
}
# Save report
with open(output_file, 'w') as f:
json.dump(report, f, indent=2)
print(f"\nVulnerability Hunting Report:")
print(f"Total patterns used: {analysis['total_patterns']}")
print(f"Successful patterns: {analysis['successful_patterns']}")
print(f"Total matches found: {analysis['total_matches']}")
print(f"High risk findings: {len(analysis['high_risk_findings'])}")
print(f"Medium risk findings: {len(analysis['medium_risk_findings'])}")
print(f"Low risk findings: {len(analysis['low_risk_findings'])}")
print(f"Report saved to: {output_file}")
return report
def create_custom_pattern(self, name, patterns, flags="-HnriE", description=""):
"""Create custom gf pattern"""
pattern_data = {
"flags": flags,
"patterns": patterns
}
if description:
pattern_data["description"] = description
pattern_file = os.path.join(self.patterns_dir, f"{name}.json")
try:
with open(pattern_file, 'w') as f:
json.dump(pattern_data, f, indent=2)
print(f"Custom pattern '{name}' created: {pattern_file}")
return True
except Exception as e:
print(f"Error creating custom pattern: {e}")
return False
Usage example¶
if name == "main": # Create hunter instance hunter = GfVulnerabilityHunter(max_workers=5)
# Create custom patterns
hunter.create_custom_pattern(
"crypto-keys",
[
"-----BEGIN [A-Z ]+-----",
"-----END [A-Z ]+-----",
"ssh-rsa [A-Za-z0-9+/=]+",
"ssh-ed25519 [A-Za-z0-9+/=]+",
"ecdsa-sha2-nistp[0-9]+ [A-Za-z0-9+/=]+"
],
description="Detect cryptographic keys and certificates"
)
hunter.create_custom_pattern(
"cloud-secrets",
[
"AKIA[0-9A-Z]{16}", # AWS Access Key
"AIza[0-9A-Za-z\\-_]{35}", # Google API Key
"sk_live_[0-9a-zA-Z]{24}", # Stripe Live Key
"sk_test_[0-9a-zA-Z]{24}", # Stripe Test Key
"xox[baprs]-[0-9]{12}-[0-9]{12}-[0-9a-zA-Z]{24}" # Slack Token
],
description="Detect cloud service secrets and API keys"
)
# Hunt for vulnerabilities
target_dir = "/path/to/target/directory"
results = hunter.hunt_vulnerabilities(
target_dir,
patterns=['secrets', 'xss', 'sqli', 'api-keys', 'crypto-keys', 'cloud-secrets'],
file_extensions=['*.js', '*.php', '*.py', '*.json', '*.yaml']
)
# Generate report
report = hunter.generate_report('vulnerability_hunting_report.json')
```_
Automatisierte Musterverwaltung¶
```bash
!/bin/bash¶
Automated gf pattern management and updates¶
GF_PATTERNS_DIR="\(HOME/.gf" BACKUP_DIR="\)HOME/.gf-backup" TEMP_DIR="/tmp/gf-patterns-update"
Function to backup existing patterns¶
backup_patterns() { echo "Backing up existing patterns..." mkdir -p "\(BACKUP_DIR" | | | | cp -r "\)GF_PATTERNS_DIR"/* "$BACKUP_DIR/" 2>/dev/null | | true | | | | echo "Backup completed: $BACKUP_DIR" }
Function to update patterns from repositories¶
update_patterns() { echo "Updating gf patterns from repositories..."
mkdir -p "$TEMP_DIR"
cd "$TEMP_DIR"
# Update from main repository
echo "Updating from tomnomnom/gf..."
git clone https://github.com/tomnomnom/gf.git main-repo
| | | | cp main-repo/examples/*.json "$GF_PATTERNS_DIR/" 2>/dev/null | | true | | | |
# Update from community patterns
echo "Updating from 1ndianl33t/Gf-Patterns..."
git clone https://github.com/1ndianl33t/Gf-Patterns.git community-patterns
| | | | cp community-patterns/*.json "$GF_PATTERNS_DIR/" 2>/dev/null | | true | | | |
# Update from secrets patterns
echo "Updating from dwisiswant0/gf-secrets..."
git clone https://github.com/dwisiswant0/gf-secrets.git secrets-patterns
| | | | cp secrets-patterns/*.json "$GF_PATTERNS_DIR/" 2>/dev/null | | true | | | |
# Update from additional sources
echo "Updating from additional pattern sources..."
# Nuclei templates patterns
git clone https://github.com/projectdiscovery/nuclei-templates.git nuclei-templates
| | | | find nuclei-templates -name ".yaml" -exec grep -l "gf:" {} \; | head -10 | while read template; do | | | | pattern_name=\((basename "\)template" .yaml) echo "Extracting pattern from \(template..." # Extract gf patterns from nuclei templates (simplified) | | | | grep -A 5 "gf:" "\)template" | grep -E "pattern | regex" | sed 's/.: //' > "\(GF_PATTERNS_DIR/nuclei-\)pattern_name.txt" 2>/dev/null | | true | | | | done
# Clean up
cd "$HOME"
rm -rf "$TEMP_DIR"
echo "Pattern update completed"
}
Function to validate patterns¶
validate_patterns() { echo "Validating gf patterns..."
local invalid_patterns=()
for pattern_file in "$GF_PATTERNS_DIR"/*.json; do
if [ -f "$pattern_file" ]; then
pattern_name=$(basename "$pattern_file" .json)
# Check JSON syntax
if ! jq . "$pattern_file" > /dev/null 2>&1; then
echo "❌ Invalid JSON syntax: $pattern_name"
invalid_patterns+=("$pattern_name")
continue
fi
# Check required fields
if ! jq -e '.patterns' "$pattern_file" > /dev/null 2>&1; then
echo "❌ Missing 'patterns' field: $pattern_name"
invalid_patterns+=("$pattern_name")
continue
fi
# Test pattern with gf
if ! gf -list | grep -q "^$pattern_name$"; then
echo "❌ Pattern not recognized by gf: $pattern_name"
invalid_patterns+=("$pattern_name")
continue
fi
echo "✅ Valid pattern: $pattern_name"
fi
done
if [ ${#invalid_patterns[@]} -gt 0 ]; then
echo "Found ${#invalid_patterns[@]} invalid patterns"
return 1
else
echo "All patterns are valid"
return 0
fi
}
Function to create comprehensive pattern collection¶
create_comprehensive_patterns() { echo "Creating comprehensive pattern collection..."
# Web vulnerabilities
cat > "$GF_PATTERNS_DIR/web-vulns-comprehensive.json" << 'EOF'
{ "flags": "-HnriE", "patterns": [ | | | | "\b(union | select | insert | update | delete | drop | create | alter | exec | execute)\b.\b(from | where | order | group | having)\b", | | | | "<\s*script[^>]>.?</\s*script\s>", "javascript\s*:", | | | | "on(load | error | click | mouseover | focus | blur | change | submit)\s*=", | | | | | | | | "\b(system | exec | shell_exec | passthru | eval | popen | proc_open)\s*\(", | | | | | | | | "\b(include | require | include_once | require_once)\s*\([^)]*\$", | | | | "\.\.[\/\\]", | | | | "\b(etc\/passwd | etc\/shadow | boot\.ini | win\.ini)", | | | | | | | | "\b(proc\/self\/environ | proc\/version | proc\/cmdline)", | | | | "\b(WEB-INF|META-INF)\/", | | | | "\b(php:\/\/ | file:\/\/ | data:\/\/ | expect:\/\/)" | | | | ] } EOF
# API and secrets
cat > "$GF_PATTERNS_DIR/api-secrets-comprehensive.json" << 'EOF'
{ "flags": "-HnriE", "patterns": [ "api[-]?key['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}", "secret[-]?key['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}", "access[-]?token['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}", "auth[-]?token['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}", "bearer['\"]?\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}", "password['\"]?\s*[:=]\s*['\"]?[^'\"\s]{8,}", "passwd['\"]?\s*[:=]\s*['\"]?[^'\"\s]{8,}", "pwd['\"]?\s*[:=]\s*['\"]?[^'\"\s]{8,}", "AKIA[0-9A-Z]{16}", "AIza[0-9A-Za-z\-]{35}", "sk_live[0-9a-zA-Z]{24}", "sk_test_[0-9a-zA-Z]{24}", "xox[baprs]-[0-9]{12}-[0-9]{12}-[0-9a-zA-Z]{24}", "-----BEGIN [A-Z ]+-----", "-----END [A-Z ]+-----" ] } EOF
# Endpoints and parameters
cat > "$GF_PATTERNS_DIR/endpoints-comprehensive.json" << 'EOF'
{ "flags": "-HnriE", "patterns": [ | | | | "\/[a-zA-Z0-9_-]+\.(php | asp | aspx | jsp | do | action)", | | | | "\/api\/[a-zA-Z0-9_\/-]+", "\/v[0-9]+\/[a-zA-Z0-9_\/-]+", "\?[a-zA-Z0-9_-]+=[a-zA-Z0-9_-]+", "&[a-zA-Z0-9_-]+=[a-zA-Z0-9_-]+", | | | | "\/(admin | administrator | management | manager | login | signin | auth | dashboard | panel)", | | | | | | | | "\/(upload | download | file | files | document | documents | attachment | attachments)", | | | | | | | | "\/(backup | bak | old | tmp | temp | test | dev | staging | prod | production)", | | | | | | | | "\/(config | configuration | settings | setup | install | installation)", | | | | | | | | "\/(debug | error | exception | log | logs | trace | stack)" | | | | ] } EOF
# Configuration and sensitive files
cat > "$GF_PATTERNS_DIR/sensitive-files.json" << 'EOF'
{ "flags": "-HnriE", "patterns": [ | | | | "\.(env | config | conf | ini | yaml | yml | json | xml | properties | cfg)\(", | | | | | | | | "\\.(key | pem | p12 | pfx | jks | keystore | crt | cer | der)\)", | | | | | | | | "\.(sql | db | sqlite | mdb | bak | backup | dump)\(", | | | | | | | | "\\.(log | logs | trace | debug | error)\)", | | | | | | | | "\.(git | svn | hg | bzr)\/", | | | | | | | | "\.(DS_Store | thumbs\.db | desktop\.ini)\(", | | | | | | | | "\\b(web\\.config | app\\.config | machine\\.config)\)", | | | | | | | | "\b(htaccess | \.htpasswd | httpd\.conf | nginx\.conf)\(", | | | | | | | | "\\b(id_rsa | id_dsa | id_ecdsa | id_ed25519)\)", | | | | | | | | "\b(known_hosts | authorized_keys | ssh_config)$" | | | | ] } EOF
echo "Comprehensive patterns created"
}
Function to test patterns¶
test_patterns() { echo "Testing gf patterns..."
# Create test files
mkdir -p /tmp/gf-test
cat > /tmp/gf-test/test.php << 'EOF'
EOF
cat > /tmp/gf-test/test.js << 'EOF'
const apiKey = "AIzaSyAbCdEfGhIjKlMnOpQrStUvWxYz12345678"; const token = "xoxb-123456789012-123456789012-abcdefghijklmnopqrstuvwx"; document.innerHTML = userInput; eval(userCode); window.location = userUrl; EOF
cat > /tmp/gf-test/config.json << 'EOF'
{ "database": { "password": "dbpassword123", "connection": "mysql://user:pass@localhost/db" }, "aws": { "access_key": "AKIAIOSFODNN7EXAMPLE", "secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" } } EOF
# Test patterns
local test_patterns=("secrets" "xss" "sqli" "api-keys" "endpoints")
for pattern in "${test_patterns[@]}"; do
echo "Testing pattern: $pattern"
if gf -list | grep -q "^$pattern$"; then
result=$(gf "$pattern" /tmp/gf-test/* 2>/dev/null | wc -l)
echo " Found $result matches"
else
echo " Pattern not found"
fi
done
# Clean up
rm -rf /tmp/gf-test
echo "Pattern testing completed"
}
Function to show pattern statistics¶
show_pattern_stats() { echo "Gf Pattern Statistics" echo "===================="
local total_patterns=$(gf -list | wc -l)
echo "Total patterns: $total_patterns"
local json_patterns=$(find "$GF_PATTERNS_DIR" -name "*.json" | wc -l)
echo "JSON patterns: $json_patterns"
local txt_patterns=$(find "$GF_PATTERNS_DIR" -name "*.txt" | wc -l)
echo "TXT patterns: $txt_patterns"
echo ""
echo "Pattern categories:"
| | | | gf -list | grep -E "(xss | sqli | rce | lfi | rfi)" | wc -l | xargs echo " Web vulnerabilities:" | | | | | | | | gf -list | grep -E "(secret | key | token | password)" | wc -l | xargs echo " Secrets and credentials:" | | | | | | | | gf -list | grep -E "(endpoint | url | param)" | wc -l | xargs echo " Endpoints and parameters:" | | | | | | | | gf -list | grep -E "(debug | error | log)" | wc -l | xargs echo " Debug and logging:" | | | |
echo ""
echo "Recent patterns (last 7 days):"
find "$GF_PATTERNS_DIR" -name "*.json" -mtime -7 -exec basename {} .json \; | sort
}
Main execution¶
case "${1:-help}" in "backup") backup_patterns ;; "update") backup_patterns update_patterns validate_patterns ;; "validate") validate_patterns ;; "create") create_comprehensive_patterns ;; "test") test_patterns ;; "stats") show_pattern_stats ;; "help"|*) echo "Gf Pattern Management Script" | | | | echo "Usage: $0 {backup | update | validate | create | test | stats | help}" | | | | echo "" echo "Commands:" echo " backup - Backup existing patterns" echo " update - Update patterns from repositories" echo " validate - Validate pattern syntax and functionality" echo " create - Create comprehensive pattern collection" echo " test - Test patterns with sample data" echo " stats - Show pattern statistics" echo " help - Show this help message" ;; esac ```_
Integration von Sicherheitswerkzeugen¶
Bug Bounty Automation¶
```bash
!/bin/bash¶
Bug bounty automation with gf patterns¶
DOMAIN="\(1" OUTPUT_DIR="bug-bounty-\)(date +%s)"
if [ -z "$DOMAIN" ]; then
echo "Usage: $0
echo "Starting bug bounty automation for: $DOMAIN" echo "Output directory: $OUTPUT_DIR"
mkdir -p "$OUTPUT_DIR"
Subdomain discovery¶
echo "Discovering subdomains..." subfinder -d "\(DOMAIN" -silent > "\)OUTPUT_DIR/subdomains.txt" amass enum -d "\(DOMAIN" -passive >> "\)OUTPUT_DIR/subdomains.txt" assetfinder --subs-only "\(DOMAIN" >> "\)OUTPUT_DIR/subdomains.txt"
Remove duplicates¶
sort "\(OUTPUT_DIR/subdomains.txt" | uniq > "\)OUTPUT_DIR/unique_subdomains.txt" SUBDOMAIN_COUNT=\((wc -l < "\)OUTPUT_DIR/unique_subdomains.txt") echo "Found $SUBDOMAIN_COUNT unique subdomains"
Probe for live hosts¶
echo "Probing for live hosts..." cat "\(OUTPUT_DIR/unique_subdomains.txt" | httpx -silent > "\)OUTPUT_DIR/live_hosts.txt" LIVE_COUNT=\((wc -l < "\)OUTPUT_DIR/live_hosts.txt") echo "Found $LIVE_COUNT live hosts"
Crawl and gather URLs¶
echo "Crawling and gathering URLs..." cat "\(OUTPUT_DIR/live_hosts.txt" | hakrawler -depth 3 -plain > "\)OUTPUT_DIR/crawled_urls.txt" cat "\(OUTPUT_DIR/live_hosts.txt" | waybackurls >> "\)OUTPUT_DIR/crawled_urls.txt" cat "\(OUTPUT_DIR/live_hosts.txt" | gau >> "\)OUTPUT_DIR/crawled_urls.txt"
Remove duplicates and filter¶
| | | | sort "\(OUTPUT_DIR/crawled_urls.txt" | uniq | grep -E "\\.(php | asp | aspx | jsp | do | action)" > "\)OUTPUT_DIR/interesting_urls.txt" | | | | URL_COUNT=\((wc -l < "\)OUTPUT_DIR/interesting_urls.txt") echo "Found $URL_COUNT interesting URLs"
Extract parameters¶
echo "Extracting parameters..." | | | | cat "\(OUTPUT_DIR/crawled_urls.txt" | unfurl keys | sort | uniq > "\)OUTPUT_DIR/parameters.txt" | | | | PARAM_COUNT=\((wc -l < "\)OUTPUT_DIR/parameters.txt") echo "Found $PARAM_COUNT unique parameters"
Hunt for vulnerabilities with gf¶
echo "Hunting for vulnerabilities with gf patterns..."
XSS hunting¶
echo "Hunting for XSS..." cat "\(OUTPUT_DIR/crawled_urls.txt" | gf xss > "\)OUTPUT_DIR/xss_candidates.txt" XSS_COUNT=\((wc -l < "\)OUTPUT_DIR/xss_candidates.txt") echo "Found $XSS_COUNT XSS candidates"
SQL injection hunting¶
echo "Hunting for SQL injection..." cat "\(OUTPUT_DIR/crawled_urls.txt" | gf sqli > "\)OUTPUT_DIR/sqli_candidates.txt" SQLI_COUNT=\((wc -l < "\)OUTPUT_DIR/sqli_candidates.txt") echo "Found $SQLI_COUNT SQL injection candidates"
LFI hunting¶
echo "Hunting for LFI..." cat "\(OUTPUT_DIR/crawled_urls.txt" | gf lfi > "\)OUTPUT_DIR/lfi_candidates.txt" LFI_COUNT=\((wc -l < "\)OUTPUT_DIR/lfi_candidates.txt") echo "Found $LFI_COUNT LFI candidates"
RCE hunting¶
echo "Hunting for RCE..." cat "\(OUTPUT_DIR/crawled_urls.txt" | gf rce > "\)OUTPUT_DIR/rce_candidates.txt" RCE_COUNT=\((wc -l < "\)OUTPUT_DIR/rce_candidates.txt") echo "Found $RCE_COUNT RCE candidates"
SSRF hunting¶
echo "Hunting for SSRF..." cat "\(OUTPUT_DIR/crawled_urls.txt" | gf ssrf > "\)OUTPUT_DIR/ssrf_candidates.txt" SSRF_COUNT=\((wc -l < "\)OUTPUT_DIR/ssrf_candidates.txt") echo "Found $SSRF_COUNT SSRF candidates"
Secrets hunting¶
echo "Hunting for secrets..." cat "\(OUTPUT_DIR/crawled_urls.txt" | gf secrets > "\)OUTPUT_DIR/secrets_candidates.txt" SECRETS_COUNT=\((wc -l < "\)OUTPUT_DIR/secrets_candidates.txt") echo "Found $SECRETS_COUNT secrets candidates"
API keys hunting¶
echo "Hunting for API keys..." cat "\(OUTPUT_DIR/crawled_urls.txt" | gf api-keys > "\)OUTPUT_DIR/api_keys_candidates.txt" API_KEYS_COUNT=\((wc -l < "\)OUTPUT_DIR/api_keys_candidates.txt") echo "Found $API_KEYS_COUNT API keys candidates"
Generate summary report¶
cat > "$OUTPUT_DIR/summary.txt" << EOF Bug Bounty Automation Summary ============================ Domain: $DOMAIN Date: $(date)
Discovery Results: - Subdomains: $SUBDOMAIN_COUNT - Live hosts: $LIVE_COUNT - URLs crawled: \((wc -l < "\)OUTPUT_DIR/crawled_urls.txt") - Interesting URLs: $URL_COUNT - Parameters: $PARAM_COUNT
Vulnerability Candidates: - XSS: $XSS_COUNT - SQL Injection: $SQLI_COUNT - LFI: $LFI_COUNT - RCE: $RCE_COUNT - SSRF: $SSRF_COUNT - Secrets: $SECRETS_COUNT - API Keys: $API_KEYS_COUNT
Next Steps: 1. Manual verification of vulnerability candidates 2. Automated testing with nuclei or other tools 3. Report writing and submission EOF
echo "" echo "Bug bounty automation completed!" echo "Results saved in: \(OUTPUT_DIR" echo "Summary:" cat "\)OUTPUT_DIR/summary.txt" ```_
CI/CD Integration¶
```yaml
.github/workflows/gf-security-scan.yml¶
name: Gf Pattern Security Scan
on: push: branches: [ main, develop ] pull_request: branches: [ main ] schedule: - cron: '0 2 * * *' # Daily scan at 2 AM
jobs: gf-security-scan: runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v4
with:
go-version: '1.19'
- name: Install gf
run: |
go install github.com/tomnomnom/gf@latest
# Install patterns
mkdir -p ~/.gf
git clone https: //github.com/tomnomnom/gf.git /tmp/gf-repo
cp /tmp/gf-repo/examples/*.json ~/.gf/
git clone https: //github.com/1ndianl33t/Gf-Patterns.git /tmp/gf-patterns
cp /tmp/gf-patterns/*.json ~/.gf/
# Verify installation
gf -list
- name: Scan for secrets
run: |
mkdir -p scan-results
# Scan for various types of secrets
| | | | gf secrets . > scan-results/secrets.txt 2>/dev/null | | touch scan-results/secrets.txt | | | | | | | | gf api-keys . > scan-results/api-keys.txt 2>/dev/null | | touch scan-results/api-keys.txt | | | | | | | | gf passwords . > scan-results/passwords.txt 2>/dev/null | | touch scan-results/passwords.txt | | | |
# Count findings
SECRETS_COUNT=$(cat scan-results/secrets.txt | wc -l)
API_KEYS_COUNT=$(cat scan-results/api-keys.txt | wc -l)
PASSWORDS_COUNT=$(cat scan-results/passwords.txt | wc -l)
echo "SECRETS_COUNT=$SECRETS_COUNT" >> $GITHUB_ENV
echo "API_KEYS_COUNT=$API_KEYS_COUNT" >> $GITHUB_ENV
echo "PASSWORDS_COUNT=$PASSWORDS_COUNT" >> $GITHUB_ENV
# Generate summary
echo "Secrets found: $SECRETS_COUNT" > scan-results/summary.txt
echo "API keys found: $API_KEYS_COUNT" >> scan-results/summary.txt
echo "Passwords found: $PASSWORDS_COUNT" >> scan-results/summary.txt
- name: Scan for vulnerabilities
run: |
# Scan for common web vulnerabilities in code
| | | | gf xss . > scan-results/xss.txt 2>/dev/null | | touch scan-results/xss.txt | | | | | | | | gf sqli . > scan-results/sqli.txt 2>/dev/null | | touch scan-results/sqli.txt | | | | | | | | gf lfi . > scan-results/lfi.txt 2>/dev/null | | touch scan-results/lfi.txt | | | | | | | | gf rce . > scan-results/rce.txt 2>/dev/null | | touch scan-results/rce.txt | | | |
# Count findings
XSS_COUNT=$(cat scan-results/xss.txt | wc -l)
SQLI_COUNT=$(cat scan-results/sqli.txt | wc -l)
LFI_COUNT=$(cat scan-results/lfi.txt | wc -l)
RCE_COUNT=$(cat scan-results/rce.txt | wc -l)
echo "XSS_COUNT=$XSS_COUNT" >> $GITHUB_ENV
echo "SQLI_COUNT=$SQLI_COUNT" >> $GITHUB_ENV
echo "LFI_COUNT=$LFI_COUNT" >> $GITHUB_ENV
echo "RCE_COUNT=$RCE_COUNT" >> $GITHUB_ENV
# Add to summary
echo "XSS patterns found: $XSS_COUNT" >> scan-results/summary.txt
echo "SQL injection patterns found: $SQLI_COUNT" >> scan-results/summary.txt
echo "LFI patterns found: $LFI_COUNT" >> scan-results/summary.txt
echo "RCE patterns found: $RCE_COUNT" >> scan-results/summary.txt
- name: Upload scan results
uses: actions/upload-artifact@v3
with:
name: gf-scan-results
path: scan-results/
- name: Comment PR with results
if: github.event_name == 'pull_request'
uses: actions/github-script@v6
with:
script: |
const fs = require('fs');
const summary = fs.readFileSync('scan-results/summary.txt', 'utf8');
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `## Gf Security Scan Results\n\n\`\`\`\n${summary}\n\`\`\``
});
- name: Fail if critical findings
run: |
TOTAL_CRITICAL=$((SECRETS_COUNT + API_KEYS_COUNT + PASSWORDS_COUNT))
if [ "$TOTAL_CRITICAL" -gt "0" ]; then
echo "Critical security findings detected!"
echo "Secrets: $SECRETS_COUNT"
echo "API Keys: $API_KEYS_COUNT"
echo "Passwords: $PASSWORDS_COUNT"
exit 1
fi
```_
Leistungsoptimierung und Fehlerbehebung¶
Leistung Tuning¶
```bash
Optimize gf for different scenarios¶
Fast scanning with basic patterns¶
gf -l secrets *.js
Thorough scanning with context¶
gf -A 3 -B 3 api-keys *.json
Memory-efficient scanning for large files¶
find . -name "*.log" -exec gf -l errors {} \;
Parallel processing for multiple patterns¶
patterns=("secrets" "xss" "sqli" "api-keys") for pattern in "\({patterns[@]}"; do gf "\)pattern" *.php > "results_$pattern.txt" & done wait
Performance monitoring¶
time gf secrets large_file.txt
Resource usage monitoring¶
!/bin/bash¶
monitor_gf_performance() { local pattern="\(1" local files="\)2" local output_file="gf-performance-$(date +%s).log"
echo "Monitoring gf performance for pattern: $pattern"
# Start monitoring
{
echo "Timestamp,CPU%,Memory(MB),Files_processed"
while true; do
if pgrep -f "gf.*$pattern" > /dev/null; then
local cpu=$(ps -p $(pgrep -f "gf.*$pattern") -o %cpu --no-headers)
local mem=$(ps -p $(pgrep -f "gf.*$pattern") -o rss --no-headers | awk '{print $1/1024}')
echo "$(date +%s),$cpu,$mem,processing"
fi
sleep 1
done
} > "$output_file" &
local monitor_pid=$!
# Run gf
gf "$pattern" $files
# Stop monitoring
kill $monitor_pid 2>/dev/null
echo "Performance log saved: $output_file"
}
Usage¶
monitor_gf_performance "secrets" "*.js *.json *.yaml" ```_
Probleme bei der Fehlerbehebung¶
```bash
Troubleshooting script for gf¶
troubleshoot_gf() { echo "Gf Troubleshooting Guide" echo "======================="
# Check if gf is installed
if ! command -v gf &> /dev/null; then
echo "❌ gf not found in PATH"
echo "Solution: Install gf using 'go install github.com/tomnomnom/gf@latest'"
return 1
fi
echo "✅ gf found: $(which gf)"
# Check patterns directory
if [ ! -d "$HOME/.gf" ]; then
echo "❌ Patterns directory not found: $HOME/.gf"
echo "Solution: Create directory and install patterns"
echo "mkdir -p ~/.gf"
echo "git clone https://github.com/tomnomnom/gf.git /tmp/gf && cp /tmp/gf/examples/*.json ~/.gf/"
return 1
fi
echo "✅ Patterns directory found: $HOME/.gf"
# Check available patterns
local pattern_count=$(gf -list 2>/dev/null | wc -l)
if [ "$pattern_count" -eq 0 ]; then
echo "❌ No patterns available"
echo "Solution: Install pattern files in ~/.gf/"
return 1
fi
echo "✅ Found $pattern_count patterns"
# Test basic functionality
echo "Testing basic gf functionality..."
echo "test api_key = 'abc123'" > /tmp/gf_test.txt
if gf secrets /tmp/gf_test.txt > /dev/null 2>&1; then
echo "✅ Basic functionality test passed"
else
echo "❌ Basic functionality test failed"
echo "Solution: Check pattern files and gf installation"
rm -f /tmp/gf_test.txt
return 1
fi
rm -f /tmp/gf_test.txt
echo "Troubleshooting completed successfully"
}
Common error solutions¶
fix_common_gf_errors() { echo "Common Gf Error Solutions" echo "========================"
echo "1. 'pattern not found'"
echo " Solution: Check available patterns with 'gf -list'"
echo " Install missing patterns in ~/.gf/"
echo ""
echo "2. 'no such file or directory'"
echo " Solution: Check if target files exist"
echo " Use absolute paths or verify current directory"
echo ""
echo "3. 'permission denied'"
echo " Solution: Check file permissions"
echo " Use 'chmod +r filename' to make files readable"
echo ""
echo "4. 'grep: invalid option'"
echo " Solution: Check pattern file syntax"
echo " Verify JSON format with 'jq . pattern.json'"
echo ""
echo "5. 'no matches found' (potential false negatives)"
echo " Solution: Try different patterns or create custom ones"
echo " Use verbose mode with grep flags: gf -v pattern files"
echo ""
echo "6. 'too many matches' (performance issues)"
echo " Solution: Use more specific patterns"
echo " Filter results with additional grep: gf pattern files | grep specific"
}
Run troubleshooting¶
troubleshoot_gf fix_common_gf_errors ```_
Ressourcen und Dokumentation¶
Offizielle Mittel¶
- gf GitHub Repository - Hauptrepository und Dokumentation
- Pattern Beispiele - Offizielle Mustersammlung
- Anmerkungen löschen - Aktuelle Updates und Änderungen
- Bug Reports - Problemverfolgung und Fehlerberichte
Gemeinschaftsmittel¶
- Gf-Patterns Collection - Gemeinschaftsmuster-Repository
- Gf-Secrets Patterns - Spezialisierte geheime Detektionsmuster
- Bug Bounty Patterns - Bug bounty Jagdmuster
- Security Research Patterns - Nuclei Vorlagenmuster
Integrationsbeispiele¶
- Automation Scripts - Tom Hudsons Sicherheitsautomatisierung
- Bug Bounty Workflows - Die Methodik des Bug Hunter
- Security Testing Pipelines - DevSecOps Integration
- Pattern Development Guide - Erstellung von benutzerdefinierten Mustern