GoSpider is a high-performance, security-focused web crawler engineered specifically for penetration testers, bug bounty hunters, and security researchers. This comprehensive guide covers installation, basic usage, advanced parameters, and ethical considerations.
What is GoSpider?
GoSpider is a fast, powerful, and lightweight web crawler written in Go that specializes in discovering endpoints, parameters, and hidden content on web applications. It's specifically designed for security researchers, penetration testers, and bug bounty hunters who need to map out the attack surface of web applications efficiently.
Unlike traditional crawlers that focus on content indexing, GoSpider is built with security testing in mind. It can:
- Discover hidden endpoints and directories
- Extract JavaScript files and analyze them for endpoints
- Parse robots.txt and sitemap.xml files
- Find form actions and parameterized URLs
- Identify potential injection points and attack vectors
- Handle JavaScript-heavy single-page applications (SPAs)
Key Benefits & Advantages
Lightning Fast
Written in Go with concurrent processing, GoSpider can crawl thousands of pages per minute while maintaining low memory usage.
Intelligent Discovery
Automatically extracts parameters, endpoints, and potential vulnerabilities without manual intervention.
JavaScript Support
Can execute and analyze JavaScript to discover dynamically generated content and endpoints.
Smart Filtering
Built-in blacklist/whitelist functionality to focus on relevant content and ignore noise.
CLI Friendly
Designed for command-line usage with easy integration into security toolchains and automation scripts.
Security Focused
Built specifically for security testing with features like parameter extraction and vulnerability pattern recognition.
Installation Steps
Install Go Programming Language
GoSpider requires Go 1.16 or higher. Install Go from the official website:
# Ubuntu/Debian sudo apt update && sudo apt install golang-go # macOS (with Homebrew) brew install go # Windows Download from https://golang.org/dl/
Install GoSpider via Go
Use the go install command to download and compile GoSpider:
go install github.com/jaeles-project/gospider@latest
This will install GoSpider to your $GOPATH/bin directory.
Add to PATH (if not already)
Ensure your Go bin directory is in your system PATH:
# Add to your shell profile (~/.bashrc, ~/.zshrc, etc.) export PATH=$PATH:$(go env GOPATH)/bin # Reload your shell source ~/.bashrc
Verify Installation
Test that GoSpider is properly installed:
gospider --help
You should see the help menu with all available options.
Basic Crawling Examples
Simple Single URL Crawl
Start with the most basic command to crawl a single website:
gospider -s https://example.com
Basic Crawl with Common Options
A more practical basic crawl with reasonable settings:
gospider -s https://example.com -c 5 -d 3 --blacklist ".(jpg|jpeg|png|css|js|ico)"
Save Output to File
Always save your results for later analysis:
gospider -s https://example.com -o results.txt
Understanding Basic Output
GoSpider output includes different types of discovered content:
- [url] - Discovered URLs and endpoints
- [form] - Form actions and parameters
- [javascript] - JavaScript files found
- [robots] - Entries from robots.txt
- [sitemap] - URLs from sitemap.xml
- [linkfinder] - Links extracted from JavaScript
Advanced Parameters & Arguments
GoSpider offers extensive configuration options for advanced crawling scenarios. Here's a comprehensive breakdown of the most important arguments:
| Argument | Description | Example |
|---|---|---|
-s, --site |
Starting URL(s) for crawling | -s https://example.com |
-c, --concurrent |
Number of concurrent threads (default: 1) | -c 10 |
-d, --depth |
Maximum crawl depth (default: 1) | -d 5 |
--blacklist |
Regex pattern to exclude URLs | --blacklist ".(jpg|jpeg|png|css|js)" |
--whitelist |
Regex pattern to include only matching URLs | --whitelist ".*api.*" |
--other-source |
Enable crawling from additional sources | --other-source |
--include-subs |
Include subdomains in the crawling scope | --include-subs |
--delay |
Delay between requests in milliseconds | --delay 1000 |
--user-agent |
Custom User-Agent string for requests | --user-agent "MyCrawler/1.0" |
--proxy |
HTTP/HTTPS proxy for requests | --proxy http://127.0.0.1:8080 |
--timeout |
Request timeout in seconds (default: 10) | --timeout 30 |
--headless |
Enable headless browser mode for JavaScript execution | --headless |
--no-headless |
Disable headless browser mode (faster but no JS execution) | --no-headless |
-o, --output |
Output file path for results | -o crawl_results.txt |
--debug |
Enable debug mode for verbose output | --debug |
Advanced Crawling Example
Here's a comprehensive command that combines multiple advanced options:
gospider -s https://target.com \ -c 15 -d 6 \ --blacklist ".(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico|pdf|svg|txt|mp4|webm)" \ --other-source --include-subs \ --headless --delay 500 \ --user-agent "Mozilla/5.0 (Security Research)" \ -o comprehensive_crawl.txt
Parameter Discovery Pipeline
The real power of GoSpider comes when combined with other security tools in a pipeline for parameter discovery. Here's the complete workflow:
Complete Parameter Discovery Command
gospider -s https://www.hackthissite.org/ -c 10 -d 5 \ --blacklist ".(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico|pdf|svg|txt)" \ --other-source | gf allparam | sed 's/=.*/=/' \ | grep -Eo 'https://[^&]+' \ | awk '!seen[$0]++' \ | grep "^https://www.hackthissite.org/"
Pipeline Breakdown
- GoSpider Crawl: Discovers all URLs with parameters from the target
- gf allparam: Filters only URLs containing query parameters using Gf patterns
- sed 's/=.*/=/': Normalizes parameter values to just the parameter name
- grep -Eo 'https://[^&]+': Extracts clean URLs up to the first parameter
- awk '!seen[$0]++': Removes duplicate URLs while preserving order
- Final grep: Ensures results are scoped to the target domain only
Ethics & Legal Considerations
Always Obtain Proper Authorization
Never scan or test systems you don't own or have explicit written permission to test. Unauthorized scanning can result in:
- Legal consequences including fines and imprisonment
- Being blocked or blacklisted from networks
- Damage to your professional reputation
- Violation of terms of service agreements
Safe Learning Environments
Use these tools only on:
- Your own systems and applications
- Intentionally vulnerable training platforms (HackTheBox, TryHackMe, OWASP WebGoat)
- Authorized penetration testing engagements with written scope
- Bug bounty programs with clear rules of engagement
Summary
GoSpider is an essential tool for modern web security research, providing fast and comprehensive crawling capabilities specifically designed for parameter discovery and attack surface mapping.
- What it is: A fast, security-focused web crawler written in Go
- Key benefits: Speed, JavaScript support, intelligent discovery, and CLI integration
- Installation: Simple Go-based installation with minimal dependencies
- Basic usage: Start with simple commands and gradually add complexity
- Advanced features: Extensive configuration options for professional security testing
- Parameter discovery: Powerful pipeline integration for finding injection points
- Ethics: Always obtain authorization and use responsibly
Remember: With great power comes great responsibility. Use GoSpider ethically and legally to contribute positively to web security.