Back to Cookbook

Web Discovery Noise Filter (ffuf & gobuster)

Eliminate wildcard 200s and handle 429 rate limiting

Content discovery explodes with false positives when targets return soft-404 pages (200 for everything) or wildcard DNS responses. This recipe calibrates baselines, applies response filters (-fs/-fw/--exclude-length), and tunes request rates to avoid WAF throttling.

House RecipeWork5 min

INGREDIENTS

PROMPT

Create a skill called "Web Discovery Noise Filter". Inputs I will provide: - Target URL pattern (including FUZZ position) and any required headers - Sample responses for a known good path and a random nonexistent path - Constraints: max requests/second (if provided) and whether WAF throttling is expected Task: 1) Determine whether the target returns soft-404/wildcard responses. 2) Propose ffuf/gobuster commands with calibrated filters (size/words/lines). 3) Provide safe rate-limiting knobs (threads, rate, delay) to avoid 429/DoS. 4) Output should include a short "baseline record" I can paste into my engagement notes.

What this fixes

Common symptoms:

  • ffuf shows thousands of `200` responses (soft-404 / wildcard behavior)
  • gobuster/ffuf vhost fuzzing returns results for almost every entry
  • Scans trip rate limits or WAF throttling (`429 Too Many Requests`)

Prerequisites

  • `ffuf` and/or `gobuster` installed
  • A known base URL and (if needed) a known Host header / vhost pattern
  • A small "calibration" wordlist (10–50 entries) for fast baselining

Steps and commands

  1. Calibrate the baseline (soft-404 detection):
  • Request a random, nonexistent path and note status + size + words:

`curl -sk -o /dev/null -w "%{http_code} %{size_download}\n" https://target.tld/this-should-not-exist-$(date +%s)`

  1. ffuf directory fuzzing with explicit filters:
  • Match all, filter baseline response size:

`ffuf -u https://target.tld/FUZZ -w words.txt -mc all -fs -t 20 -rate 50 -o ffuf.json -of json`

  • If size varies but word count is stable, filter by words:

`ffuf -u https://target.tld/FUZZ -w words.txt -mc all -fw `

  • Let ffuf auto-calibrate (when appropriate):

`ffuf -u https://target.tld/FUZZ -w words.txt -ac`

  1. gobuster directory enumeration with length exclusion:

`gobuster dir -u https://target.tld -w words.txt --exclude-length -t 20`

  1. Handle 429 rate limiting / WAF throttling:
  • Reduce concurrency and add pacing:

`ffuf ... -t 10 -rate 10`

  • If sporadic 429s persist, add a delay:

`ffuf ... -p 0.2`

  1. Export results in a report-friendly format:

`ffuf ... -o ffuf.md -of md`

Expected outputs

  • A reduced, high-signal hitlist (paths or vhosts)
  • ffuf JSON/CSV/MD output suitable for review and reporting
  • Notes documenting your filter baselines (size/word/line)

Common errors and troubleshooting

  • Everything is 200
  • Use `-mc all` and filter by size/words/lines, or enable `-ac`.
  • Confirm the body differs meaningfully (not just reflected input).
  • Many false positives on DNS/VHost fuzzing
  • Test a random subdomain first and filter based on the wildcard response.
  • 429 Too Many Requests
  • Reduce `-t` and `-rate` until 429 disappears.
  • Coordinate with the client if the target is behind aggressive WAF policy.

References

  • https://github.com/ffuf/ffuf/wiki
  • https://hackertarget.com/gobuster-tutorial/
  • https://codingo.com/posts/2020-08-29-everything-you-need-to-know-about-ffuf/

Example inputs

  • URL: https://target.tld/FUZZ
  • Wordlist: /usr/share/seclists/Discovery/Web-Content/common.txt
  • Baseline size: 1534
Tags:#pentesting#web#reconnaissance#ffuf#gobuster#fuzzing#troubleshooting