httprobe

Overview

httprobe is a lightweight Go utility that takes a list of domains and tests each one to see if it’s responding to HTTP or HTTPS requests. It’s particularly useful during the reconnaissance phase of penetration testing to identify which domains have active web servers. The tool is part of the ProjectDiscovery toolkit and is commonly found in Kali Linux.

httprobe efficiently handles large lists of domains and provides multiple options for customization, including support for custom ports, timeout configuration, and header injection.

Installation

# Using Go
go install -v github.com/tomnomnom/httprobe@latest

# On Kali Linux (pre-installed)
httprobe --version

# Or compile from source
git clone https://github.com/tomnomnom/httprobe
cd httprobe
go build -o httprobe
sudo mv httprobe /usr/local/bin/

Basic Usage

Simple Domain Probing

Command	Description
httprobe < domains.txt	Probe domains from file (stdin)
echo "example.com" | httprobe	Probe a single domain via pipe
httprobe -h	Display help information
httprobe -v	Show verbose output

Input Format

httprobe expects one domain per line:

example.com
google.com
github.com
stackoverflow.com

Basic Examples

# Probe domains from a file
cat domains.txt | httprobe

# Probe with verbose output
cat domains.txt | httprobe -v

# Probe single domain
echo "example.com" | httprobe

# Combine with other tools
subfinder -d example.com -silent | httprobe

Common Options

Port Configuration

Option	Description	Example
-p	Specify custom ports	httprobe -p 8080,8443
-c	Concurrency (threads)	httprobe -c 50
--prefer-https	Prioritize HTTPS results	httprobe --prefer-https

Timeout and Protocol Settings

Option	Description	Example
-t	Timeout in milliseconds	httprobe -t 5000
--skip-save	Don’t save results	httprobe --skip-save
-s	Stdin mode (default)	httprobe -s

Examples with Options

# Probe with custom timeout (10 seconds)
cat domains.txt | httprobe -t 10000

# Probe with increased concurrency (100 threads)
cat domains.txt | httprobe -c 100

# Probe custom ports
cat domains.txt | httprobe -p 8080,8443,9000

# Probe with HTTPS preference and custom timeout
cat domains.txt | httprobe --prefer-https -t 5000

# Combine verbose and concurrency
cat domains.txt | httprobe -v -c 50

Advanced Usage

Integration with Reconnaissance Workflows

# Get subdomains and probe for HTTP/HTTPS
subfinder -d example.com -silent | httprobe

# Use with assetfinder
assetfinder example.com | httprobe

# Combine with amass for comprehensive discovery
amass enum -d example.com -silent | httprobe

# Multi-domain enumeration with httprobe
for domain in example.com test.com sample.com; do
  subfinder -d $domain -silent | httprobe
done

Output Processing

# Save results to file
cat domains.txt | httprobe > live_hosts.txt

# Count results
cat domains.txt | httprobe | wc -l

# Filter for HTTPS only
cat domains.txt | httprobe | grep "^https://"

# Filter for HTTP only
cat domains.txt | httprobe | grep "^http://" | grep -v "^https://"

# Check specific status codes with curl
cat live_hosts.txt | xargs -I {} curl -o /dev/null -s -w "{} -> %{http_code}\n" {}

Large-Scale Scanning

# High concurrency for large lists
cat large_domain_list.txt | httprobe -c 200 -t 3000

# Store output with timestamp
cat domains.txt | httprobe > results_$(date +%Y%m%d_%H%M%S).txt

# Parallel processing with GNU parallel
cat domains.txt | parallel --pipe --block 10M httprobe -c 50

# Resume scanning if interrupted
comm -23 <(sort domains.txt) <(sort results.txt | cut -d: -f1 | sort) | httprobe >> results.txt

Common Workflows

Web Application Reconnaissance

# Discover live web servers across target domain
subfinder -d target.com -silent | httprobe -c 100 -t 5000

# Identify web server versions
cat live_hosts.txt | while read url; do
  curl -I -s $url | grep -i "server:"
done

# Extract title tags
cat live_hosts.txt | while read url; do
  curl -s $url | grep -oP '<title>\K[^<]+' || echo "No title: $url"
done

Bug Bounty Reconnaissance

# Enumerate all subdomains and probe
assetfinder -subs-only example.com | httprobe | tee -a scope.txt

# Verify live targets before testing
cat scope.txt | httprobe --prefer-https > verified_targets.txt

# Quick HTTP status check
while read url; do
  status=$(curl -o /dev/null -s -w "%{http_code}" $url)
  echo "$url -> $status"
done < scope.txt

Corporate Network Assessment

# Probe internal domain list
cat internal_domains.txt | httprobe -c 50

# Identify web services on custom ports
cat hosts.txt | httprobe -p 8080,8443,9000,3000

# Log detailed discovery results
cat hosts.txt | httprobe -v > discovery_$(date +%s).log 2>&1

Output Format

httprobe returns the full URL (including protocol) for accessible services:

http://example.com
https://example.com
http://subdomain.example.com
https://api.example.com

Practical Examples

Example 1: Basic Domain Enumeration

# Create domain list
echo -e "google.com\ngithub.com\nstackoverflow.com" > targets.txt

# Probe domains
httprobe < targets.txt

# Expected output:
# http://google.com
# https://google.com
# http://github.com
# https://github.com

Example 2: Integration with Subfinder

# Full reconnaissance pipeline
subfinder -d target.com -silent | \
  httprobe -c 100 -t 5000 --prefer-https | \
  tee target_hosts.txt

# Count results
wc -l target_hosts.txt

Example 3: Custom Port Scanning

# Test common development ports
cat hosts.txt | httprobe -p 8000,8080,8443,3000,5000,9000

# Results will include:
# http://host1:8080
# https://host1:8443

Example 4: Performance Testing

# Large-scale scanning with optimal settings
time cat 10000_domains.txt | \
  httprobe -c 200 -t 3000 > results.txt

# Monitor progress
cat 10000_domains.txt | httprobe -v -c 100 | tee progress.log

Performance Tips

• Concurrency: Increase with -c flag; 50-100 is typical, 200+ for very large lists
• Timeout: Reduce timeout with -t for faster scanning; 3000-5000ms is reasonable
• Output Buffering: Redirect to file to see results as they complete
• Memory: httprobe is lightweight; suitable for resource-constrained environments

Troubleshooting

Issue	Solution
No results	Increase timeout: httprobe -t 10000
Slow scanning	Increase concurrency: httprobe -c 100
Connection refused	Check network connectivity and firewall rules
Empty input	Verify domain list format (one per line)
Port conflicts	Try alternative ports with -p option

Security Considerations

• Authorization: Only probe domains/networks you own or have explicit permission to test
• Rate Limiting: Respect server limits; use appropriate concurrency settings
• Logging: Monitor and log all discovery activities for audit trails
• False Positives: Verify results with manual inspection before further testing

Related Tools

• subfinder: Subdomain enumeration
• assetfinder: Asset discovery
• amass: Advanced OSINT subdomain enumeration
• curl/wget: HTTP client utilities for detailed investigation
• nmap: Network scanning and port discovery