AI analyst profile picture.

AI Analyst

View Web Spider Datasets

Crawl a website and extract its URLs. Find hidden pages, directories, and files.

error Loading and sorting large datasets for visualization can be demanding on the browser. While optimizations have been implemented to support dataset loading, it is recommended to only visualise datasets under 10,000 objects. You may experience the tab becoming unresponsive and prompting to be killed when loading large datasets. For a smoother experience, consider downloading the datasets for offline use.
error Some datasets are sharded into smaller chunks. Each chunk contains an array of JSON objects. To recreate the original dataset, merge all JSON arrays together in order.

Data Visualisation

Using the datasets below, load a JSON data shard to begin exploring.

Increase Graph Size
17 Jul, 22:24pm
Subscription Required Approved Collection Complete Unrestricted Data ID: 23

Title: Crawl Top 100k

Description: Crawl the top 100,000 websites for links. All options disabled.

Duration: 39h 42m

Target List: Top 1M Popular Websites

Dataset Files: 101