Post

@ChrisShort This is a bigger deal for companies that sell content (newspapers, etc). For all the individuals, imho, updating robots.txt is a symbolic gesture. Most ordinary people are writing content on someone else's machine (like here! or comments on youtube, etc) governed by someone elses Terms of Service