How to Find All Pages on a Website
Short answer: start with the site’s XML sitemap (try /sitemap.xml, or look for a Sitemap: line in /robots.txt). Then crawl the site from the home page to catch pages the sitemap leaves out. For a site you own, add Search Console and your server logs. No single method finds everything, so combine two or three and remove duplicates.
What each method finds, and what it misses
| Method | Finds | Misses |
|---|---|---|
| XML sitemap | Pages the owner chose to list | Pages left out of the sitemap; may include stale URLs |
| Crawler | Every page reachable through links | Orphan pages; links that appear only after JavaScript (for simple crawlers) |
| Google site: search | A sample of indexed pages | Non-indexed pages; not exhaustive by Google’s own docs |
| Search Console | Google’s view of your own site | Only your verified sites; 1,000 example rows per status |
| Server logs / analytics | Every URL that was requested or visited | Only your own site; pages nobody visited |
| Web archives and SEO tools | Old and externally linked URLs | Recent pages; results vary by tool |
1. Read the XML sitemap
Most sites publish a sitemap for search engines. Try these addresses:
https://example.com/sitemap.xml https://example.com/sitemap_index.xml (Yoast, Rank Math) https://example.com/wp-sitemap.xml (WordPress core) https://example.com/robots.txt (look for "Sitemap:" lines)
Each <loc> tag holds one URL. Big sites split their URLs over several sitemaps (each holds up to 50,000 URLs) listed in a sitemap index, so open each child file. To pull the URLs out with one command:
curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//'
Can’t find it? The free sitemap finder reads robots.txt and tries the common paths for you, then reads up to five sitemap files and tests 30 of the URLs.
2. Crawl the site
A crawler starts at the home page, follows every internal link, and repeats, the way a search engine bot does. It finds pages the owner forgot to put in the sitemap. It can’t find pages that nothing links to.
Our free XML sitemap generator works as a quick crawler. It follows links from the home page of one site, up to 500 pages, and respects robots.txt. It shows the pages it found that can be indexed, plus a list of pages it left out with the reason (redirect, error, noindex, or a canonical pointing elsewhere). It does not run JavaScript, so links that only appear after scripts run are missed. For larger sites, use a desktop crawler such as Screaming Frog or Sitebulb.
3. Search Google with site:
site:example.com site:example.com/blog/ site:example.com inurl:product
This shows pages Google has indexed, on any public site. Treat it as a sample. Google’s own docs say the operator “doesn’t necessarily return all the URLs that are indexed” and the list “is not always exhaustive.” Narrow it by folder to see more of each section.
4. Use Search Console (your own site)
In Search Console, the Pages report (Indexing → Pages) groups every URL Google knows into indexed and not indexed, with a reason for each. Click a status to see example URLs and export them. Google limits each list to 1,000 rows, so on a big site it’s a sample too. The Performance report adds the pages that got search impressions.
5. Check server logs and analytics (your own site)
Your server logs record every URL that was requested, including pages with no links and pages only bots visit. Analytics lists every page that got a visit. Both are good at finding pages the other methods miss, such as old landing pages and campaign URLs.
6. Look at web archives and SEO tools
The Wayback Machine at web.archive.org can list archived URLs for a domain, which helps find old or removed pages. SEO tools such as Ahrefs and Semrush keep their own index of the web and can list a domain’s pages that have traffic or backlinks.
Turn the lists into one clean inventory
- Merge and dedupe. Normalize http vs https, www vs non-www and trailing slashes so one page isn’t counted three times.
- Group parameter URLs.
?sort=,?utm_source=and filter parameters can turn a few hundred pages into thousands of near-copies. - Sort by folder (
/blog/,/products/) to see the site’s structure. - Check status codes. A URL that redirects or returns 404 isn’t a live page.
- Compare sources. A page in your sitemap or logs that the crawl never reached has no internal links: an orphan page. A live, indexable page missing from your sitemap should be added.
Frequently asked questions
What is the fastest way to see all pages on a website?
Open its XML sitemap, usually at /sitemap.xml or listed in /robots.txt. It is the list the owner publishes for search engines. It can miss pages, so crawl the site too if you need everything.
Can I find all pages on a website I don’t own?
Yes, the public ones. The sitemap, a crawl from the home page and Google’s site: search all work on any public site. Pages behind a login, and pages nothing links to, stay hidden.
Why does site:example.com show fewer pages than the site has?
Google says the site: operator does not necessarily return all indexed URLs, and the list is not exhaustive. Use it as a sample, not a count.
How do I find hidden or orphan pages?
Compare lists. A URL that is in the sitemap, your server logs or analytics but not found by a crawl has no internal links pointing to it. That is an orphan page.
How many pages can Search Console list?
The Page indexing report shows up to 1,000 example URLs per status, and you can export them. For a full list of your own site, combine your sitemap with a crawl.