Duplicate URLs in a Sitemap: Exact Repeats and Near-Duplicates
Short answer: there are two kinds. Exact repeats (the same URL listed twice) are mostly harmless (for a sitemap index, Search Console counts a duplicate URL once in its discovered pages), but they point to a buggy generator. Near-duplicates are the real problem: /page and /page/, http and https, www and non-www, or parameter variants of one page. Google treats each as a separate URL, and only one can be the canonical. Keep exactly one version per page, the one that returns 200 and names itself as canonical.
Exact repeats
The same <loc> appears twice, often because a page belongs to two categories or two plugins both write sitemap entries. Find them in one line:
curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*' | sort | uniq -d
For a sitemap index, run it over the child sitemaps together, because the same URL can sit in two different files. Fix the generator so each URL is written once.
Don’t confuse this with Search Console’s “Invalid XML: too many tags”. That error means one entry contains a tag twice, such as two <loc> tags. See sitemap parsing errors.
Near-duplicates: one page, several URLs
https://example.com/shoes https://example.com/shoes/ <- trailing slash variant http://example.com/shoes <- protocol variant https://www.example.com/shoes <- host variant https://example.com/shoes?sort=price <- parameter variant https://example.com/Shoes <- case variant
If several of these return 200, Google sees duplicates and picks one canonical itself. If they redirect, the sitemap is listing redirects. Either way, only one belongs in the sitemap. Sort the list and look for URLs that differ only by slash, case, host or parameters.
How to fix it
- Pick the canonical form for every page: one protocol, one host, one slash style, no tracking or sort parameters.
- Make the site agree: 301-redirect the other forms and give each page a self-referencing canonical. See trailing slashes and www vs non-www.
- Make the sitemap agree: output only that form, once. Google’s sitemap guide says to list the URLs you want in search results, which is the canonical one.
- Resubmit and check the Page indexing report with the sitemap filter set to “All submitted pages”.
The free sitemap checker flags mixed trailing slashes and mixed-case paths across your sitemap, and tests 30 URLs for redirects and canonicals pointing elsewhere. More on the canonical side in non-canonical pages in a sitemap.
Frequently asked questions
Does listing the same URL twice cause an error?
An exact repeat is mostly harmless: for a sitemap index, Search Console counts a duplicate URL once in the discovered pages. It is still a sign that the generator is buggy, so fix it.
What about /page and /page/ both in the sitemap?
Those are two different URLs to Google. Only one can be the canonical, so the other entry is a non-canonical or redirecting URL. List only the version that returns 200.
What does "Invalid XML: too many tags" mean?
That is a different problem: a single entry has a tag twice, such as two <loc> tags in one <url>. Keep one of each.
How do I count duplicates quickly?
Extract the <loc> values and sort them: curl -s URL | grep -o "<loc>[^<]*" | sort | uniq -d prints every URL that appears more than once.