I have never even see documentation say, "before calling http://client.download(url), make a call to robots.txt and see if you are allowed to make that call"- unless of course you are an actual spider & will be downloading *every* page. https://x.com/mistersql/status/970686880147701761
Post
March 6, 2018