What is it?
hakrawler is a Go web crawler designed for easy, quick discovery of endpoints and assets within a web application. It can be used to discover:
- Related domains
The goal is to create the tool in a way that it can be easily chained with other tools such as subdomain enumeration tools and vulnerability scanners in order to facilitate tool chaining, for example:
assetfinder target.com | hakrawler | some-xss-scanner
- Unlimited, fast web crawling for endpoint discovery
- Fuzzy matching for domain discovery
- robots.txt parsing
- sitemap.xml parsing
- Plain output for easy parsing into other tools
- Accept domains from stdin for easier tool chaining
- SQLMap-friendly output format
- Want more? Submit a feature request!
- hakluke wrote the tool
- cablej cleaned up the code
- delic made the code much cleaner
- hoenn made the code even cleanerer
- codingo and prodigysml/sml555, my favourite people to hack with. A constant source of ideas and inspiration. They also provided beta testing and a sounding board for this tool in development.
- tomnomnom who wrote waybackurls, which powers the wayback part of this tool
- s0md3v who wrote photon, which I took ideas from to create this tool
- The folks from gocolly, the library which powers the crawler engine
- oxffaa, who wrote a very efficient sitemap.xml parser which is used in this tool
- Install Golang
- Run the command below
go get github.com/hakluke/hakrawler
- Run hakrawler from your Go bin directory. For linux systems it will likely be:
Note that if you need to do this, you probably want to add your Go bin directory to your $PATH to make things easier!
Note: multiple domains can be crawled by piping them into hakrawler from stdin. If only a single domain is being crawled, it can be added by using the -domain flag.
hakrawler -url bugcrowd.com -depth 1