Knowledge Base
Ways to Find All URLs on a Page with Selenium
A practical guide to gathering every link on a rendered page with Selenium, normalising and deduplicating URLs, and where proxies help with larger crawls.
Collecting all the links on a page is the foundation of many crawling and auditing tasks. Selenium is well suited to this when links appear only after JavaScript runs, because it works with the fully rendered page rather than the raw HTML.
This walkthrough explains the practical ways to find all URLs, the cleanup steps that turn raw links into a usable list, and where proxies become relevant once you move from one page to a wider crawl.
Collecting links from the rendered page
The core idea is to select every anchor element and read its href. Conceptually you call find_elements(By.TAG_NAME, 'a') to get all anchors, then read the href attribute from each. Because Selenium reads the live DOM, it captures links injected by scripts that a static parser would miss.
Reading href through Selenium's attribute accessor typically returns the fully resolved absolute URL, which saves you from manually joining relative paths in many cases. Still, it is worth validating, because some anchors hold fragments or JavaScript handlers rather than real destinations.
Cleaning, filtering, and deduplicating
Raw link lists are noisy. To produce a clean set:
- Drop empty values, anchors that only point to
#, andjavascript:ormailto:entries if you only want navigable pages. - Normalise URLs by stripping fragments and, where appropriate, trailing parameters so equivalent links collapse together.
- Deduplicate by adding URLs to a set, which removes repeats automatically.
If you are crawling within one domain, filter out external hosts so the list stays focused on the site you care about.
Handling pagination and dynamic loading
Many pages reveal more links only after scrolling or clicking a "load more" control. To capture those, scroll the page or trigger the control, wait for new content to appear, then re-collect anchors. For paginated listings, repeat the collection on each page and merge the results.
Always wait for new links to be present before reading them, since collecting too early returns an incomplete set. Combining a wait with a loop is the reliable pattern for infinite-scroll and paginated layouts.
Where proxies fit for larger crawls
One page is fine without a proxy, but link discovery usually leads to crawling many pages across a site. Sustained crawling from a single IP can trigger throttling, so proxies help maintain steady coverage.
For broad crawling, datacenter proxies are a value-focused option, while residential proxies may suit stricter or location-aware sites. Explore proxy use cases for crawling scenarios and trade-offs.
What to compare before buying
Before you order, weigh these points so the proxies you pick match your real workload and budget:
- Proxy type: datacenter for broad crawling, residential for stricter sites
- Concurrency support matching your crawl parallelism
- Rotation: per-request rotation to spread many page requests
- Geographic coverage if links differ by region
- Bandwidth model suited to full page rendering
- Transparent pricing: verify the exact package before ordering
- Support quality for handling blocks during long crawls
Frequently asked questions
Select all anchor elements by their tag name and read the href attribute from each. Because Selenium uses the rendered DOM, it captures links added by JavaScript that static parsers miss.
Often not, because reading href through Selenium's attribute accessor usually returns the resolved absolute URL. Still validate results, since some anchors hold fragments or script handlers rather than real pages.
Normalise links by stripping fragments and unneeded parameters, then add them to a set, which automatically discards repeats and gives you a clean unique list.
Scroll the page or trigger the load-more control, wait for new content to appear, then re-collect the anchors. Repeat until no new links are added.
No. Proxies become useful when link discovery turns into crawling many pages, since sustained requests from one IP can trigger rate limiting.
Datacenter proxies are a value-focused choice for broad crawling, while residential proxies may be more suitable for stricter or location-aware sites.
Related pages worth comparing
Have a comparison question about how to find all urls using selenium? Email info@comparebestproxy.com.