Robots.txt File, Sitemap.xml and the XML schema

What the hell are these bots or robots?   The robots exclusion protocol or simply robots.txt, is a standard used by most websites to communicate with web crawlers and other web robots. Holy fuck, so now the robots are taking over the world, yeah sort of. Ever heard of HAL 9000?   The standard specifies how … Continue reading Robots.txt File, Sitemap.xml and the XML schema