You All know Google used to crawl web pages and index them into there database of millions and trillions of pages they use a great software called spider to perform this Process this spider will index all the web pages in the internet with great speed like the same way i coded out a simple mini web crawler crawls the specific webpages and get out there links and displays it, i used PHP and Jquery to perform this actions when a people types the url and clicks crawl the software crawls the whole web page and displays the link present in it and you can see the demo here and download the script for free
