Interesting. I know you have had this in mind for some time. You're right that it is overdue since this will prevent a number of possible well-known attack vectors against the server.

I don't suppose you could persuade Stone to stop allowing every search engine and spider from ransacking and recording every last post we make here. They get permission from the robots.txt file on the server. Currently it remains empty meaning that any search engine has permission to permanently datamine every last thing we post here. Our handles, our comments, the articles we respond to, all of it available to put in some database to be mined for years or decades into the future.

LF: robots.txt

How about just restricting it to allowing the search engines and spiders to see the Articles only?

By comparison, we can look at the same file over at 4um and see that they do restrict the permission of search engines and spiders to scan and mine their content.

User-agent: * Disallow: /cgi-bin/latestcomments.cgi Disallow: /cgi-bin/latestarticles.cgi Disallow: /cgi-bin/readart.cgi

In plain words, it says that all search engines are allowed (first line) but that they cannot indext the Latest Comments page, the Latest Articles page, and they cannot do any reading and scanning of articles (the readart.cgi line).

Stone has allowed these search engines to datamine us all for years now, all in the hope that he might attract some new users. Well, it hasn't worked. We're the same old crew we've been since the days of LP.

If Stone wants to have a search engine that can help us locate old posts or old articles here, he could enable just one search engine to index the site. A privacy-oriented search like DuckDuckGo would be a good choice.

Restricting these search engines would also reduce bandwidth and CPU load on the server, although this would be somewhat marginal. It is clear that search engines like Google do spider and ransack this site about once an hour. That's about how long it takes for our posts and articles here at LF to show up in their search engine.

BTW, I feel embarrassed that I noticed a bit ago that all my visited links here at LF had suddenly turned to unvisited. That is because you changed us from HTTP to HTTPS and all of those old visited pages were thus now considered as unvisited pages. I just didn't realize why and didn't inspect my URL bar.

So thanks a lot for making me feel like a luser. Which would make you the BOFH.     : )

Yes, it is interesting. It was very easy to do. The setup programs took care of everything. I didn't have to make any apache modifications myself.

We've discussed this before. It's Stone's decision, but there are 2 things I'd respond with:

1) Having the articles and comments added to the search engine could help bring in new people.

2) The Robots file is only advisory in nature. It does not forcibly prohibit or restrict access to the site by search engines. They are as free to visit the site as much as any human is able to, and mine whatever it finds regardless of what is put into the robots file. Normally, yes, they would abide by it as it's the convention, but any company or organization that has any motive at all to ignore the robots file can do so at their whim. In fact, paying attention to the robots file requires a little MORE effort in the spider programming than does simply ignoring it.

hehe... I am guessing within an hour you'll be caught up.

1) Having the articles and comments added to the search engine could help bring in new people.

But it hasn't brought in new people. It could serve to flag LF as a hate site with the usual SPLC types and brand all users as haters. So there is at least as much downside as upside.

And where are all these new users? They aren't here and they haven't joined in all this time that LF and its users have been exposed to this extensive datamining.

2) The Robots file is only advisory in nature. It does not forcibly prohibit or restrict access to the site by search engines.

True enough. But the major search engines do not index where they are forbidden by robots.txt. They respect this because they know how easy it is to block their spider IP addresses if they refuse to honor the restrictions of the robots.txt.

I know this is not your decision; it is Stone's. But I have raised and will continue to raise it.

There is no upside to this rampant data mining. There are considerable potential downsides to it. I would not complain if Stone would simply restrict it to allowing only DDG to index the site.

hehe... I am guessing within an hour you'll be caught up.

Yeah, you do have those server logs to inspect.

I always laughed my ass off at BOFH. Hard to believe they actually turned it into a series of books but more power to them if they can monetize it.

Thanks,I am so ignorant of this stuff that I have been assuming ever since the first day I logged on to any web site that they were all encrypted.

My motto seems to be,"When you are going to be wrong,but wrong BIG!"

I know this is not your decision; it is Stone's. But I have raised and will continue to raise it.

Stone wants his pearls of wisdom propagated to the stars!

Is there any doubt that extraterrestrial spiders are gobbling up his every keystroke?

Stone wants his pearls of wisdom propagated to the stars!

Yeah, well some of those pearls of wisdom are ones that he pulls down himself. Like his recent Hip Hip Hooray thread about banning you for a week.

Ipso facto. He isn't proud of some of his posts when he pulls them down.