What the hell are these bots or robots?   The robots exclusion protocol or simply robots.txt, is a standard used by most websites to communicate with web crawlers and other web robots. Yes, robots will be taking over the world, yeah sort of. Ever heard of HAL 9000?   The standard specifies how to tell the … Continue reading Robots.txt File, Sitemap.xml and the XML schema