Cyber Services / All lessons / CEH — Ethical Hacking / GoSpider Professional Web Crawling

GoSpider Professional Web Crawling

Module 121 of 130 · 5 min read · Free to read

MODULES ↓ Command blocks have a COPY button

GoSpider is a high-performance, security-focused web crawler engineered specifically for penetration testers, bug bounty hunters, and security researchers. This comprehensive guide covers installation, basic usage, advanced parameters, and ethical considerations.

What is GoSpider?

GoSpider is a fast, powerful, and lightweight web crawler written in Go that specializes in discovering endpoints, parameters, and hidden content on web applications. It's specifically designed for security researchers, penetration testers, and bug bounty hunters who need to map out the attack surface of web applications efficiently.

Unlike traditional crawlers that focus on content indexing, GoSpider is built with security testing in mind. It can:

  • Discover hidden endpoints and directories
  • Extract JavaScript files and analyze them for endpoints
  • Parse robots.txt and sitemap.xml files
  • Find form actions and parameterized URLs
  • Identify potential injection points and attack vectors
  • Handle JavaScript-heavy single-page applications (SPAs)

Key Benefits & Advantages

Lightning Fast

Written in Go with concurrent processing, GoSpider can crawl thousands of pages per minute while maintaining low memory usage.

Intelligent Discovery

Automatically extracts parameters, endpoints, and potential vulnerabilities without manual intervention.

JavaScript Support

Can execute and analyze JavaScript to discover dynamically generated content and endpoints.

Smart Filtering

Built-in blacklist/whitelist functionality to focus on relevant content and ignore noise.

CLI Friendly

Designed for command-line usage with easy integration into security toolchains and automation scripts.

Security Focused

Built specifically for security testing with features like parameter extraction and vulnerability pattern recognition.

Installation Steps

Install Go Programming Language

GoSpider requires Go 1.16 or higher. Install Go from the official website:

# Ubuntu/Debian
sudo apt update && sudo apt install golang-go

# macOS (with Homebrew)
brew install go

# Windows
Download from https://golang.org/dl/

Install GoSpider via Go

Use the go install command to download and compile GoSpider:

go install github.com/jaeles-project/gospider@latest

This will install GoSpider to your $GOPATH/bin directory.

Add to PATH (if not already)

Ensure your Go bin directory is in your system PATH:

# Add to your shell profile (~/.bashrc, ~/.zshrc, etc.)
export PATH=$PATH:$(go env GOPATH)/bin

# Reload your shell
source ~/.bashrc

Verify Installation

Test that GoSpider is properly installed:

gospider --help

You should see the help menu with all available options.

Basic Crawling Examples

Simple Single URL Crawl

Start with the most basic command to crawl a single website:

gospider -s https://example.com

Basic Crawl with Common Options

A more practical basic crawl with reasonable settings:

gospider -s https://example.com -c 5 -d 3 --blacklist ".(jpg|jpeg|png|css|js|ico)"

Save Output to File

Always save your results for later analysis:

gospider -s https://example.com -o results.txt

Understanding Basic Output

GoSpider output includes different types of discovered content:

  • [url] - Discovered URLs and endpoints
  • [form] - Form actions and parameters
  • [javascript] - JavaScript files found
  • [robots] - Entries from robots.txt
  • [sitemap] - URLs from sitemap.xml
  • [linkfinder] - Links extracted from JavaScript

Advanced Parameters & Arguments

GoSpider offers extensive configuration options for advanced crawling scenarios. Here's a comprehensive breakdown of the most important arguments:

Argument Description Example
-s, --site Starting URL(s) for crawling -s https://example.com
-c, --concurrent Number of concurrent threads (default: 1) -c 10
-d, --depth Maximum crawl depth (default: 1) -d 5
--blacklist Regex pattern to exclude URLs --blacklist ".(jpg|jpeg|png|css|js)"
--whitelist Regex pattern to include only matching URLs --whitelist ".*api.*"
--other-source Enable crawling from additional sources --other-source
--include-subs Include subdomains in the crawling scope --include-subs
--delay Delay between requests in milliseconds --delay 1000
--user-agent Custom User-Agent string for requests --user-agent "MyCrawler/1.0"
--proxy HTTP/HTTPS proxy for requests --proxy http://127.0.0.1:8080
--timeout Request timeout in seconds (default: 10) --timeout 30
--headless Enable headless browser mode for JavaScript execution --headless
--no-headless Disable headless browser mode (faster but no JS execution) --no-headless
-o, --output Output file path for results -o crawl_results.txt
--debug Enable debug mode for verbose output --debug

Advanced Crawling Example

Here's a comprehensive command that combines multiple advanced options:

gospider -s https://target.com \
  -c 15 -d 6 \
  --blacklist ".(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico|pdf|svg|txt|mp4|webm)" \
  --other-source --include-subs \
  --headless --delay 500 \
  --user-agent "Mozilla/5.0 (Security Research)" \
  -o comprehensive_crawl.txt

Parameter Discovery Pipeline

The real power of GoSpider comes when combined with other security tools in a pipeline for parameter discovery. Here's the complete workflow:

Complete Parameter Discovery Command

gospider -s https://www.hackthissite.org/ -c 10 -d 5 \
  --blacklist ".(jpg|jpeg|gif|css|tif|tiff|png|ttf|woff|woff2|ico|pdf|svg|txt)" \
  --other-source | gf allparam | sed 's/=.*/=/' \
  | grep -Eo 'https://[^&]+' \
  | awk '!seen[$0]++' \
  | grep "^https://www.hackthissite.org/"

Pipeline Breakdown

  1. GoSpider Crawl: Discovers all URLs with parameters from the target
  2. gf allparam: Filters only URLs containing query parameters using Gf patterns
  3. sed 's/=.*/=/': Normalizes parameter values to just the parameter name
  4. grep -Eo 'https://[^&]+': Extracts clean URLs up to the first parameter
  5. awk '!seen[$0]++': Removes duplicate URLs while preserving order
  6. Final grep: Ensures results are scoped to the target domain only

Ethics & Legal Considerations

Always Obtain Proper Authorization

Never scan or test systems you don't own or have explicit written permission to test. Unauthorized scanning can result in:

  • Legal consequences including fines and imprisonment
  • Being blocked or blacklisted from networks
  • Damage to your professional reputation
  • Violation of terms of service agreements

Safe Learning Environments

Use these tools only on:

  • Your own systems and applications
  • Intentionally vulnerable training platforms (HackTheBox, TryHackMe, OWASP WebGoat)
  • Authorized penetration testing engagements with written scope
  • Bug bounty programs with clear rules of engagement

Summary

GoSpider is an essential tool for modern web security research, providing fast and comprehensive crawling capabilities specifically designed for parameter discovery and attack surface mapping.

  • What it is: A fast, security-focused web crawler written in Go
  • Key benefits: Speed, JavaScript support, intelligent discovery, and CLI integration
  • Installation: Simple Go-based installation with minimal dependencies
  • Basic usage: Start with simple commands and gradually add complexity
  • Advanced features: Extensive configuration options for professional security testing
  • Parameter discovery: Powerful pipeline integration for finding injection points
  • Ethics: Always obtain authorization and use responsibly

Remember: With great power comes great responsibility. Use GoSpider ethically and legally to contribute positively to web security.

Modules (130) ← CEH — Ethical Hacking