Miha I didn't know The Wayback Machine blocks access to website snapshots if robots.txt blocks web crawlers. So this old website got shut down, the domain name gets parked by a domain squatter that set the robots.txt to ban all web crawlers. Upon archive request, Wayback Machine checks the domain, reads robots.txt, then effectively blocks you from accessing the snapshots.
😀 Tom That sucks. I should think that the robots.txt shouldn't be retroactive but only apply while it is published.
7y, 3d reply
Login or register your account to reply