There are various factors you could possibly need to have to discover all of the URLs on a web site, but your exact intention will ascertain That which you’re seeking. As an example, you might want to:

Discover each and every indexed URL to investigate difficulties like cannibalization or index bloat
Accumulate present-day and historic URLs Google has viewed, especially for web page migrations
Uncover all 404 URLs to recover from article-migration problems
In each circumstance, just one Software won’t Provide you almost everything you'll need. Regretably, Google Lookup Console isn’t exhaustive, in addition to a “web site:case in point.com” lookup is proscribed and difficult to extract facts from.

Within this article, I’ll walk you through some applications to construct your URL listing and before deduplicating the information employing a spreadsheet or Jupyter Notebook, based on your web site’s size.

Previous sitemaps and crawl exports
For those who’re searching for URLs that disappeared with the live web site a short while ago, there’s a chance a person with your workforce might have saved a sitemap file or simply a crawl export before the alterations were being created. When you haven’t already, check for these documents; they might frequently present what you need. But, should you’re looking at this, you almost certainly did not get so lucky.

Archive.org
Archive.org
Archive.org is an invaluable Device for Search engine optimization jobs, funded by donations. When you look for a website and choose the “URLs” possibility, you are able to accessibility as many as ten,000 outlined URLs.

Even so, Here are a few limits:

URL limit: You'll be able to only retrieve approximately web designer kuala lumpur ten,000 URLs, which happens to be insufficient for bigger internet sites.
Excellent: Lots of URLs may be malformed or reference resource files (e.g., visuals or scripts).
No export option: There isn’t a constructed-in approach to export the checklist.
To bypass The shortage of an export button, make use of a browser scraping plugin like Dataminer.io. Even so, these constraints indicate Archive.org might not give a whole Resolution for more substantial web-sites. Also, Archive.org doesn’t point out irrespective of whether Google indexed a URL—however, if Archive.org uncovered it, there’s a very good chance Google did, too.

Moz Pro
While you may usually use a hyperlink index to locate external sites linking to you personally, these applications also explore URLs on your web site in the method.


The way to use it:
Export your inbound backlinks in Moz Pro to secure a brief and simple listing of goal URLs from a internet site. Should you’re dealing with a huge website, consider using the Moz API to export details over and above what’s workable in Excel or Google Sheets.

It’s vital that you Take note that Moz Professional doesn’t validate if URLs are indexed or discovered by Google. Even so, because most web-sites utilize the identical robots.txt regulations to Moz’s bots since they do to Google’s, this process frequently works effectively being a proxy for Googlebot’s discoverability.

Google Research Console
Google Lookup Console delivers various valuable sources for making your listing of URLs.

Hyperlinks stories:


Comparable to Moz Professional, the Hyperlinks area provides exportable lists of target URLs. However, these exports are capped at one,000 URLs each. You'll be able to use filters for precise internet pages, but since filters don’t apply towards the export, you could ought to trust in browser scraping instruments—limited to five hundred filtered URLs at a time. Not best.

Effectiveness → Search Results:


This export offers you an index of web pages getting look for impressions. When the export is restricted, You need to use Google Search Console API for larger sized datasets. Additionally, there are free Google Sheets plugins that simplify pulling additional comprehensive information.

Indexing → Web pages report:


This section delivers exports filtered by concern type, although they are also minimal in scope.

Google Analytics
Google Analytics
The Engagement → Internet pages and Screens default report in GA4 is an excellent source for accumulating URLs, with a generous limit of 100,000 URLs.


Better still, you may utilize filters to create distinct URL lists, proficiently surpassing the 100k Restrict. By way of example, if you need to export only web site URLs, observe these methods:

Stage one: Include a phase for the report

Step 2: Simply click “Create a new section.”


Step 3: Determine the segment that has a narrower URL pattern, for instance URLs containing /site/


Note: URLs present in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they supply useful insights.

Server log documents
Server or CDN log data files are Probably the final word Software at your disposal. These logs seize an exhaustive list of every URL path queried by consumers, Googlebot, or other bots during the recorded time period.

Issues:

Details dimension: Log information might be massive, so many sites only keep the last two months of data.
Complexity: Examining log information can be complicated, but many equipment are available to simplify the procedure.
Merge, and great luck
When you’ve collected URLs from every one of these resources, it’s time to combine them. If your internet site is sufficiently small, use Excel or, for much larger datasets, resources like Google Sheets or Jupyter Notebook. Be certain all URLs are continually formatted, then deduplicate the list.

And voilà—you now have an extensive list of latest, old, and archived URLs. Very good luck!