How to run an AI-assisted SEO competitor analysis that actually works

You can now do in 20 minutes what used to take a full afternoon. Feed two Semrush exports into Claude or ChatGPT, and you’ll get a polished competitor analysis – complete with topic clusters, gap tables, and prioritized briefs.

The output looks convincing. The tables are clean. The recommendations sound confident.

That’s the problem. AI can organize and summarize data quickly, but it can’t make strategic decisions. Without the right workflow, prompts, and validation, you risk acting on insights that sound right but lack depth.

Used correctly, though, AI can surface meaningful patterns – revealing differences in topical depth, content coverage, and authority signals that influence search visibility.

Here’s a walkthrough of a real two-competitor analysis using Claude and Semrush data, showing how to turn fast AI outputs into a reliable strategy. You’ll get a repeatable workflow, tested prompts, and a validation checklist to catch common mistakes, along with a clear sense of where to trust AI — and where to rely on your judgment.

AI won’t run a competitor analysis for you. But it can compress the manual work — clustering, pattern matching, and synthesis — so you can focus on interpreting intent, validating opportunities, and deciding what’s worth pursuing.

Note: The sites in this analysis are real but anonymized. Site Y is our client, while Competitors A and B are direct competitors in the same niche. The data is from real Semrush exports pulled in early 2026.

Start with data, not a prompt

Whenever possible, start by exporting data from your SEO tool. Don’t ask an AI assistant to guess what an SEO tool can tell you.

Otherwise, you assume your AI assistant is a measurement tool. Although it isn’t, it’ll try its best to respond to your request. This often looks like plausible-sounding traffic estimates, keyword lists, and competitive assessments that are partially or entirely fabricated.

Here’s what we exported and why each piece matters.

Export 1: Organic Research > Pages (top 100 by estimated traffic)

This report tells you which pages are winning. Key columns include the URL, estimated traffic per page, number of ranking keywords per page, the intent breakdown (commercial, informational, navigational, transactional), and the traffic change column that shows momentum.

For example, a page pulling 14,500 visits from 1,632 keywords is a different asset from a page pulling 400 visits from 12 keywords. The intent split tells you why that traffic matters.

Export 2: Organic Research > Positions (top 100 keywords by traffic)

This export tells you which keywords are winning. Key columns here are keyword and position, search volume, keyword difficulty , search engine results page (SERP) features (image packs, video carousels, and People Also Ask), and keyword intent tags.

Instead of telling you which URLs perform best, this report reveals which search queries drive the most traffic. You need both reports for a complete picture.

The export checklist 

For each competitor and for your own site, pull:

• Semrush Organic Research > Pages, top 50-100, sorted by traffic.
• Semrush Organic Research > Positions, top 100-500, sorted by traffic.
• Semrush Keyword Gap report (optional).
• Screaming Frog crawl with URLs, titles, H1s, word count, crawl depth, and internal links. This optional report adds structural context (like how deep pages are buried in the site architecture) that the Semrush exports don’t include.

Conduct a 20-minute competitive review

Next, feed your exports into your AI assistant. Ask it to do three things: classify, cluster, and compare.

Topic taxonomy (per site)

Here’s the prompt I used:

I'm going to give you a Semrush Organic Pages export for a website. Each row is a URL with its estimated organic traffic, number of ranking keywords, and intent breakdown.

Please:
1. Assign each URL to a topic category (e.g., "Product - Roof Racks," "Editorial - Buying Guides," "Support - Technical," "Category - Inventory")
2. Assign a page type: Homepage, Product Page, Category Page, Editorial/Guide, Blog Post, Support/Info, Landing Page, or Other
3. Create a summary table showing: topic category, number of pages, total traffic, and dominant intent

Rules:
- Base classifications on the URL path and any context available. Do NOT guess traffic numbers or keyword data. Use only what's in the export.
- If a URL is ambiguous, flag it as "needs manual review" rather than guessing.
- Group similar topics (e.g., don't create separate categories for "off-road accessories" and "off-road bumper kits." Cluster them).
- After classifying, list any URLs where you're less than 80% confident in the classification. I'll verify those manually.

Here's the data:
[PASTE PAGES EXPORT]

For Site Y, Claude identified seven topic clusters across 100 pages. Here’s the summary:

Topic Cluster	Pages	Traffic	Dominant intent
Homepage/Brand	3	14,651	Mixed (commercial and informational)
Buying guides and comparisons	25	~10,600	Informational and commercial
Roof racks and cargo (product)	2	~5,100	Commercial and transactional
Bumpers and armor (product)	38	~2,300	Commercial
Installation and how-to content	4	~1,300	Informational
Inventory/Category	4	~540	Transactional
Other (brand, manufacturer, thin)	24	~1,300	Mixed

Even before comparing competitors, this taxonomy tells a story. Our client’s organic traffic is driven more by editorial content (buying guides and comparisons) than by all product pages combined.

In fact, a single buying guide pulled 7,336 visits on its own, and the top product page drove 5,021. That editorial strength is both a strategic asset and a vulnerability, since editorial rankings can be more volatile than product page rankings.

Competitor comparison

Once you’ve created a taxonomy for each site, use this prompt to compare them:

I now have topic taxonomies for three competing sites in the same niche. I'm going to give you the summary tables for all three.

Please:
1. Build a comparison table showing how each site's traffic distributes across topic categories
2. Identify each site's "content strategy signature": what type of content drives the majority of their organic traffic
3. Flag any categories where one site dominates and the others are weak or absent
4. Note the traffic concentration: what percentage of each site's total traffic comes from their top 3 pages

Rules:
- Use only the data provided. Do not estimate or infer traffic for categories not present in a site's export.
- If a category doesn't exist for a site, mark it "Not present" rather than zero. We don't know if they have content there, only that it doesn't appear in their top 100.

Site Y taxonomy:
[PASTE]

Competitor A taxonomy:
[PASTE]

Competitor B taxonomy:
[PASTE]

When we used this prompt, Claude revealed three completely different strategies from the same niche.

	Site Y	Info/support pages (60 of the top 100)	Competitor B
Content strategy	Editorial-led	Utility/support-led	Product page-led
Top content type	Buying guides and comparisons	Info/support pages (60 of top 100)	Product pages and category pages
Non-homepage hero page	Tow capacity and fitment calculator (7,336 visits)	Bolt pattern lookup guide (1,245 visits)	Off-road bumper category (3,200 visits)
Traffic concentration (top three)	75.3%	81.2%	71.8%
Estimated traffic (top 100)	35,681	7,017	11,093
Momentum	Growing (+1,743 net)	Flat (-264 net)	Declining (-1,525 net)

Manually developing this comparison could require hours of spreadsheet work between categorizing 300 URLs, building pivot tables, and trying to spot patterns across three tabs. But Claude did it in minutes.

The pattern recognition alone (three completely different strategies from three sites selling in the same market) is genuinely valuable output.

The numbers show that Site Y pulls five times the organic traffic of Competitor A and three times that of Competitor B, despite all three competing in the same space.

Competitor A’s second-highest traffic page is a bolt pattern guide on a support subdomain. Competitor B is losing ground fast, with its top category page dropping by 1,184 visits.

If you’re running a competitive analysis and you don’t spot patterns like these, you’re missing the strategic story behind the data.

Apply human judgment

If you were to stop after generating the clusters and comparison chart, you’d have a plausible-looking competitive analysis. But the AI-generated output needs human intervention before you make any strategic decisions.

Check the classifications

Spot-check 10-15% of classifications by visiting the URLs. Correct the taxonomy, and then re-run the comparison. This turns an 85% accurate first draft into one with 95% or higher accuracy.

The “confidence flag” line in the prompt (“list any URLs where you’re less than 80% confident”) saves you from having to guess which ones to check. If you skip this step, the misclassifications can distort your entire competitive profile.

For example, when I checked Claude’s page classifications against the actual live pages, roughly 15% needed correction. It tagged a product comparison page as a blog post. It classified a regional landing page as a category page. And it lumped an FAQ page into the “Other” category even though it served as the site’s primary buyer’s guide for a specific product line.

These misclassifications were the kind of accidental calls that come from categorizing URLs by path structure alone, without seeing the page content. For example, if a URL path says /blog/best-off-road-accessories/, AI assistants will call it a blog post even if the page functions as a commercial comparison guide.

Consider the intent

AI assistants can surface data points in seconds, but they can’t make strategic calls for you. Interpreting the data requires understanding your client’s business model, their authority level, and their content capacity.

I’ve seen teams burn an entire content sprint on high-volume informational keywords that drove plenty of traffic and zero leads. If the intent doesn’t match your business goals, the volume is irrelevant.

For example, Competitor A’s second-highest-traffic page is a bolt pattern lookup guide, pulling 1,245 visits per month. Claude flagged this as a content strategy gap for Site Y, since our client had no equivalent utility content.

While this is technically correct, it’s strategically misleading. The bolt pattern guide targets purely informational intent. So, the page builds authority and earns links, but it’s not a commercial driver.

While it can be helpful to create utility content like this, it should be a steady background effort, not a priority sprint. The commercially relevant gaps (product categories, buying guides) come first.

Use this prompt fix:

For each opportunity you flag, check the intent breakdown from the Semrush data. 
If more than 60% of the traffic is informational or navigational intent, flag it separately as "authority builder, not direct conversion driver" so I can prioritize accordingly.

Compare the SERP reality vs. the ranking position

AI assistants work from the position numbers and volume data in your SEO reports. They don’t know what the SERP looks like.

For example, Claude saw that Site Y ranks Position 3 for “off-road roof rack” (22,200 monthly searches, driving 1,443 visits) and treated it as a straightforward optimization opportunity. Push the page to position one, and capture more traffic. Simple.

But in reality, the SERP is packed with rich features: popular products, an image pack, and People Also Ask. The traditional organic blue links appear barely above the fold on desktop and well below the fold on mobile.

Ranking in position one likely wouldn’t deliver the traffic increase you’d normally expect from a 22,200-volume keyword because the SERP features absorb most of the clicks.

For your top five or 10 priority keywords, do a manual SERP check. If the page is dominated by shopping carousels and video results, then a traditional organic push may not be the right play. Instead, a product feed optimization or video content strategy might be more effective.

---

Do a gap analysis

Your SEO tool already has a keyword gap report. But a raw list of missing keywords isn’t a strategy.

Use it as a starting point. Then, let AI clusteri those gaps into themes, tiering them by intent and business relevance and turning raw gap data into prioritized actions.

Start with the tool data

We pulled two Semrush Keyword Gap reports comparing Site Y against both competitors. They revealed:

• Missing keywords: 217 keywords where both competitors rank and Site Y doesn’t appear at all. Combined search volume ~49,700/month.
• Weak keywords: 106 keywords where Site Y ranks but gets outperformed by both competitors. Combined search volume: ~33,650/month.

Feed the gap data to AI

Use this prompt with your AI assistant:

I'm going to give you two Semrush Keyword Gap reports:
1. MISSING: keywords where both competitors rank and Site Y doesn't
2. WEAK: keywords where Site Y ranks but competitors outrank us

Each row includes: keyword, intent tags, search volume, keyword difficulty, CPC, and the ranking position for each site.

Please:
1. Cluster the keywords into thematic groups (e.g., "bumpers," "roof racks," "overlanding gear," "light bar kits," "torque specs/fitment"). A keyword can only belong to one cluster.
2. For each cluster, provide: number of keywords, total search volume, dominant intent, and average keyword difficulty.
3. Separate the clusters into tiers based on intent:
  - Tier 1 (Commercially relevant): Clusters with predominantly commercial or transactional intent that align with the site's core product/service offering
  - Tier 2 (Adjacent commercial): Clusters that are commercially relevant to the broader market but may not be the site's primary product focus
  - Tier 3 (Authority builders): Clusters with primarily informational or navigational intent that build topical authority but are unlikely to drive direct conversions
  Note: I will review the tier assignments and adjust based on business model fit. AI should make its best guess and flag any clusters where the tier assignment is uncertain.
4. Within each tier, sort by combined search volume
5. Flag any keywords that are branded competitor terms (e.g., a competitor's product or brand name). These are generally not pursuable gaps
6. For the WEAK keywords, separate into "close wins" (Site Y in positions 1-10) vs. "long shots" (Site Y in positions 50+)

Rules:
- Use ONLY the keywords in these exports. Do not suggest keywords not present in the data.
- If intent data is missing or ambiguous, mark it "verify manually" rather than guessing.
- Do not invent search volume or ranking data. If a field is empty, say "not available."

MISSING keywords:
[PASTE]

WEAK keywords:
[PASTE]

When we used this prompt with Claude, clear thematic clusters emerged from the 217 missing keywords:

Cluster	Keywords	Combined volume	Dominant intent	Claude’s tier
Bumpers / skid plates	30+	~12,000/mo	Commercial	1
Roof racks / cargo systems	10+	~8,000/mo	Commercial	1
Winches (for sale)	15+	~5,500/mo	Transactional	1
LED light bar kits	12+	~3,200/mo	Commercial	1
Overlanding gear / overlanding accessories	10+	~2,800/mo	Commercial	1
Torque specs / installation guides	8+	~1,500/mo	Informational	3
Branded competitor terms	6+	~1,200/mo	Navigational	Skip

Correct AI’s priorities

This step determines where you spend the next quarter’s content budget, so human judgment is essential.

If you let an AI assistant set your content priorities based purely on search volume and intent labels, you’ll end up chasing someone else’s market instead of dominating your own. Volume is seductive, but business alignment is what drives revenue.

For example, Claude clustered 323 keywords and tiered them by intent in minutes. But it assigned bumpers/skid plates (~12,000/month volume) the same priority as overlanding gear (~2,800/month) because it doesn’t know what Site Y sells.

Without our human override, we may have built our content calendar around the wrong cluster.

Cluster	Claude’s tier	Corrected tier	Reasoning
Overlanding gear / overlanding accessories	1	1: Core business	Directly aligned with Site Y’s primary product line. These are the keywords that drive qualified buyers.
Bumpers / skid plates	1	2: Adjacent	High volume, commercially relevant to the broader market, and Site Y stocks some of these products. Worth targeting through editorial/guide content over time, but not the priority sprint.
Roof racks / cargo systems	1	2: Adjacent	Related to what Site Y does, but not the core offering.
Winches (for sale)	1	2: Adjacent	Transactional intent is appealing, but these are a different product category.
LED light bar kits	1	2: Adjacent	Related market, but not core inventory.
Torque specs / installation guides	3	3: Authority	Informational content that builds topical relevance. Steady background effort.
Branded competitor terms	Skip	Skip	Can’t realistically win these anytime soon.

Identify small pushes that make big differences

Next, find the low-effort opportunities with the biggest payoffs.

For example, from 106 weak keywords, we separated 17 close wins where Site Y already ranks in positions one through 10. These have real potential:

Keyword	Volume	Site Y Position	Best Competitor Position	Gap
overlanding accessories	1,600	3	1	2 positions
overlanding gear	720	3	1	2 positions
overlanding roof rack	720	4	1	3 positions
overlanding accessory kit	590	3	1	2 positions
overlanding storage system	390	3	1	2 positions
overland vehicle accessories	320	3	1	2 positions
overland accessories	260	3	1	2 positions
overlanding cargo rack	210	3	1	2 positions

Site Y sits at position three across virtually every “overlanding” variant, while Competitor A holds position one. These are optimization opportunities. A focused push toward better on-page targeting, internal linking adjustments, and content updates incorporating “overlanding” language more explicitly could flip several of these to position one or two.

That’s a different action than writing a new page. Claude would have defaulted to the latter if we hadn’t split the data into close wins and long shots.

Factor in authority context

As a final validation step, pull the backlink profiles for your competitors.

When we did this, we found that both had relatively thin link profiles. Competitor B had 199 backlinks with an average page authority score of just 1.1 (on Semrush’s 0-100 scale), while Competitor A had 128 backlinks, averaging a 3.1 authority score. The highest quality links for both came from the same handful of overlanding and off-road vehicle publications.

The most-linked pages and the top organic pages barely overlapped for either competitor. Only the homepages appeared in both lists.

Competitor B’s top backlinks pointed to product pages, while its top organic traffic came from category pages. Competitor A’s best links came from editorial features, while their organic traffic was dominated by the homepage and a support page.

This tells us their organic rankings are driven more by topical relevance and on-page SEO than by direct link equity to individual pages. It means the keyword gaps we identified are likely winnable through content and optimization rather than requiring a major link building campaign.

Turn the gap analysis into a brief

Use your competitor analysis to draft a content brief with AI. Input this prompt:

Based on the gap analysis we ran, [DESCRIBE PRIORITY CLUSTER] emerged as a priority. Draft a content brief for optimizing the existing presence and/or creating a new page to capture this cluster.

Include:
1. Primary and secondary target keywords (from our data only)
2. Recommended page type and format (based on what's currently ranking for these terms)
3. Content structure with suggested H2s
4. Content elements the ranking competitors include that our page should match or exceed
5. Estimated word count range based on competing content

Then, in a separate section called "Differentiation: For Human Review," suggest 3 possible angles that would make this page genuinely different from what already ranks. These are suggestions for me to evaluate, not final decisions.

Before finalizing the brief, cross-reference the target keywords against Site Y's existing pages export. Flag any existing pages that already rank for or target similar keywords. These are potential cannibalization risks that need to be resolved before creating new content.

Rules:
- Do not fabricate competitor content details. Base element recommendations on what we know from our data (URLs, page types, keyword footprints)
- If you need information you don't have (e.g., actual competitor page content), say "manual review needed: [specific thing to check]" rather than guessing

From this prompt, Claude drafted a clean brief with target keywords from our data, recommended format (long-form guide with product integration), and an H2 structure.

It also performed a cannibalization check. Because we added a cross-reference line to the prompt, Claude flagged that Site Y already had a related page pulling 838 visits. If we’d created a new page without checking, it would have competed with the existing page. That one line in the prompt saved us from unnecessary internal competition.

But the differentiation section needed human input. Only someone who knows Site Y’s brand voice and customer objections could pick the right angle from these suggested options:

• First-hand testing and review angle: Site Y installs and tests these products, so they can show real usage via trail tests, installation photos, and customer experiences.
• Comparison angle: What’s the difference between overlanding versus off-road? This directly addresses the keyword overlap we noticed in the gap data.
• Buyer qualification angle: Who needs overlanding gear versus who would be fine with standard off-road accessories?

The experience signals (actual trail tests, customer stories, installation details) also need substantial human oversight. This is where Google’s emphasis on experience, expertise, authoritativeness, and trustworthiness meets practical execution. If you don’t have genuine first-hand experience to draw on, no amount of keyword optimization will close that gap.

Run through a validation checklist

Before you act on any AI-assisted competitor analysis, go through this checklist to prevent the most common errors.

Data validation

• Base all analysis on tool exports (Semrush, Ahrefs, Screaming Frog), not AI-generated estimates.
• Check for export dates (if data is older than 90 days, recent algorithm updates or market shifts may have changed the picture).
• Use a meaningful sample size (top 50+ pages per competitor, not just top 10).
• Include both Pages and Positions exports.

Classification validation

• Spot-check 10-15% of the AI assistant’s page type and topic classifications against live pages.
• Correct any misclassifications and re-run the comparison.
• Check whether AI created overly granular or overly broad categories.
• Verify that pages on subdomains or unusual URL structures were classified correctly.

Intent validation

• Check intent tags (not just search volume) on all flagged opportunities.
• Separate commercially relevant gaps from informational and authority-building gaps.
• Verify intent interpretation with a manual SERP check on your top three to five priority keywords.
• Make a conscious decision to pursue, defer, or skip high-volume informational keywords.

Prioritization validation

• Confirm your AI assistant’s priority ranking aligns with your business goals, not just search volume.
• Check whether the product or service matches what you sell if a cluster looks like tier one based on volume alone.
• Determine if opportunities are achievable given site authority and content resources.
• Confirm no opportunities are branded competitor terms you can’t realistically win.
• Check whether a gap is better addressed by optimizing existing content versus creating new content.

Brief validation

• Choose a differentiation angle for AI-generated briefs (not just keywords and structure).
• Verify the recommended content format matches what ranks in SERPs.
• Confirm the brief doesn’t target keywords that your own site already ranks for.
• Identify E-E-A-T signals and determine what original content the page needs that AI can’t generate.

The shift to AI-assisted SEO competitor analysis

AI tools have changed where you spend your time when conducting a competitor analysis. The data gathering, clustering, cross-referencing, and initial synthesis that used to consume most of your time? AI handles that efficiently.

Instead, AI assistants free up thinking time. Now, you can spend that time on the parts that determine whether your analysis leads to results: interpreting intent, validating classifications, and making strategic calls about what’s worth pursuing and what’s a distraction.