Miha I didn't know The Wayback Machine blocks access to website snapshots if robots.txt blocks web crawlers. So this old website got shut down, the domain name gets parked by a domain squatter that set the robots.txt to ban all web crawlers. Upon archive request, Wayback Machine checks the domain, reads robots.txt, then effectively blocks you from accessing the snapshots.
Login or register your account to reply
😀 Tom That sucks. I should think that the robots.txt shouldn't be retroactive but only apply while it is published.
8y, 5w reply