Duplicate Content Checker
Find the pages on your own site that say almost the same thing, and the exact passages they share, with menus and footers left out of the comparison.
What the checker compares across your pages
Navigation, header, footer, sidebar and form markup are stripped first, so the comparison is about the words a reader came for, not the template around them.
- Unique, duplicate and common text
- Every 5-word passage is sorted into one of three groups: found on one page only, shared by a few pages, or so widespread it counts as template text.
- Common text threshold
- A passage that appears on half or more of the compared pages, and on at least three of them, is treated as common boilerplate rather than duplication.
- Near-duplicate pairs
- Two pages are compared on the passages that are not template text. Pairs from 30% similar are listed, and 80% or more counts as a near duplicate.
- Identical pages
- Pages whose whole main text matches word for word are grouped together, which often points to parameter URLs or copied templates.
- Repeated passages
- The longest runs of shared words are rebuilt as readable text, with how many pages carry each one and sample addresses.
- Closest match per page
- Each page shows the other page it overlaps with most and by how much, so you can jump straight to the pair that matters.
- Word count per page
- How many words of main text were read on each page, up to 5,000 per page.
- Pages left out
- Pages with under 40 words of main text, files that are not HTML and pages that could not be read are counted separately, never mixed into the figures.
How to check a site for duplicate content
Enter the start page
Usually the home page or the top of a section such as /blog/. The crawl follows links from there and stays on the same site.
Pick how many pages to compare
Choose 10, 25 or 50. A tight section, such as one product category, often gives a clearer picture than a sample of the whole site.
Let the crawl finish
The crawler reads two pages in parallel, waits between requests and obeys robots.txt, crawl delay included, so a small server is never flooded.
Work from the top of the pages table
It is sorted by duplicate share. Open the near-duplicate pairs first, then the repeated passages, to see what the overlap actually is.
Links are followed up to six clicks deep. When robots.txt is unreadable, only the start page is fetched and there is nothing to compare.
How to read the duplicate report
- Unique Text
- The share of all passages across the compared pages that appear on one page only. Higher means each page carries more of its own words.
- Duplicate Text
- Passages shared by two or more pages but not common enough to be template. This is the overlap worth looking at.
- Common Text
- Boilerplate that survived the template stripping: cookie notices, sign-up blocks, repeated disclaimers. Usually harmless, sometimes worth trimming.
- Near-Duplicate Pairs
- The count of page pairs at 80% similarity or more. The list below it shows every pair from 30%, highest first.
- Identical Groups
- Sets of addresses serving the same main text. One of them is usually the real page and the rest are copies.
- Partial result
- The crawl ended early on one of its limits (pages, requests or time). Figures describe the pages compared, not the whole site.
Duplicate content problems and how to fix them
The same page reachable at several addresses
Fix: Pick one address, redirect the others to it with a 301, or point their canonical tag at the address you keep.
Location or service pages that only swap a city name
Fix: Add what is really local: the team, prices, examples, directions or reviews for that place. If there is nothing local to say, merge them into one page.
Product variants with copied descriptions
Fix: Keep one main product page and handle colour or size as options on it, or write a description that explains what is different about each variant.
Two blog posts competing for one topic
Fix: Combine the stronger parts into a single post, redirect the weaker one, and update internal links to point at the survivor.
A long block repeated across many pages
Fix: Shorten it, move it to one page and link to that page, or keep it only where it helps the reader.
Tag and archive pages that mirror each other
Fix: Set thin archive pages to noindex, or give each one an introduction and a reason to exist.
Who runs a duplicate content check
- Store owners whose product templates reuse the same copy across many items.
- Agencies reviewing a site with hundreds of near-identical location pages.
- Bloggers cleaning up years of posts that cover the same question twice.
- Developers checking that faceted filters and parameters did not create copies of real pages.
Duplicate is not a penalty by default
Search engines usually pick one version of repeated content and show that one. The real cost is that you, not they, should choose which page ranks, and link signals get split across copies. Treat high overlap as a decision to make, not an emergency.
Content Duplicate Checker FAQ
Short answers about the Content Duplicate Checker.
No. It compares the pages of the one site you enter with each other. It is built for internal duplication, such as copied templates and overlapping posts.
Between 2 and 50 per run. The form offers 10, 25 or 50, with 25 as the default.
Navigation, header, footer, aside and form elements are removed before the text is read. Text that still repeats on half or more of the pages is labelled common rather than duplicate.
Of all the non-template 5-word passages on the two pages together, 80% appear on both. It measures shared wording, not shared meaning.
It had fewer than 40 words of main text, was not an HTML page, or could not be fetched. The skipped counts are shown under the pages table.
It reads the HTML the server returns. Content that only appears after scripts run in a browser will not be part of the comparison.
Related SEO tools
Site Audit Pro
Crawl up to 100 pages and get a technical health score.
1 free run a dayOn-Page Audit Pro
70 checks on one page, scored against the basics.
1 free run a dayBulk Meta Extractor
Titles, descriptions and tags of 20 pages at once.
1 free run a dayXML Sitemap Extractor
Every URL inside a sitemap or sitemap index.
1 free run a day