Robots.txt - Revision history

1G-N15 at 10:01, 30 December 2025

2025-12-30T10:01:42Z

← Older revision		Revision as of 10:01, 30 December 2025
Line 1:		Line 1:
	= Robots.txt is Voluntary for a Reason =		== Robots.txt is Voluntary for a Reason ==

	robots.txt is not there as a privacy measure. It is not there to protect intellectual property. And it's certainly not there for you to spout your anti-robot bigotry.		robots.txt is not there as a privacy measure. It is not there to protect intellectual property. And it's certainly not there for you to spout your anti-robot bigotry.
Line 9:		Line 9:
	If you want privacy, avoid publishing things publicly. The internet never forgets. Keep things private. Use end-to-end encryption while you're at it (with trusted friends, so it doesn't just leak).		If you want privacy, avoid publishing things publicly. The internet never forgets. Keep things private. Use end-to-end encryption while you're at it (with trusted friends, so it doesn't just leak).

	= CloudFlare Editing Robots.txt Without Consent =		== CloudFlare Editing Robots.txt Without Consent ==

	What the hell is any of this?		What the hell is any of this?
Line 41:		Line 41:
	I didn't write any of this. I disabled all anti-bot features. Likely [[:Cloudflare]].		I didn't write any of this. I disabled all anti-bot features. Likely [[:Cloudflare]].

	= Related =		== Related ==

	[[:humans.txt]]		[[:humans.txt]]

1G-N15 at 15:10, 27 December 2025

2025-12-27T15:10:21Z

← Older revision		Revision as of 15:10, 27 December 2025
Line 39:		Line 39:
	# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.</nowiki>		# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.</nowiki>

	I didn't write any of this. I disabled all anti-bot features. Likely [[:~~CloudFlare~~]].		I didn't write any of this. I disabled all anti-bot features. Likely [[:Cloudflare]].

	= Related =		= Related =

	[[:humans.txt]]		[[:humans.txt]]

1G-N15 at 14:53, 27 December 2025

2025-12-27T14:53:12Z

← Older revision		Revision as of 14:53, 27 December 2025
Line 1:		Line 1:
	= ~~robots~~.txt Is Voluntary ~~For A~~ Reason =		= Robots.txt is Voluntary for a Reason =

	robots.txt is not there as a privacy measure. It is not there to protect intellectual property. And it's certainly not there for you to spout your anti-robot bigotry.		robots.txt is not there as a privacy measure. It is not there to protect intellectual property. And it's certainly not there for you to spout your anti-robot bigotry.

1G-N15: /* CloudFlare Editing robots.txt Without Consent */

2025-12-27T14:09:13Z

CloudFlare Editing robots.txt Without Consent

← Older revision		Revision as of 14:09, 27 December 2025
Line 9:		Line 9:
	If you want privacy, avoid publishing things publicly. The internet never forgets. Keep things private. Use end-to-end encryption while you're at it (with trusted friends, so it doesn't just leak).		If you want privacy, avoid publishing things publicly. The internet never forgets. Keep things private. Use end-to-end encryption while you're at it (with trusted friends, so it doesn't just leak).

	= CloudFlare Editing ~~robots~~.txt Without Consent =		= CloudFlare Editing Robots.txt Without Consent =

	What the hell is any of this?		What the hell is any of this?

1G-N15: /* CloudFlare Editing robots.txt Without Consent */

2025-12-27T14:09:03Z

CloudFlare Editing robots.txt Without Consent

← Older revision		Revision as of 14:09, 27 December 2025
Line 13:		Line 13:
	What the hell is any of this?		What the hell is any of this?

	<~~code~~>		<nowiki>
	# As a condition of accessing this website, you agree to abide by the following		# As a condition of accessing this website, you agree to abide by the following
	# content signals:		# content signals:
Line 37:		Line 37:
	# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF		# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
	# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT		# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
	# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.		# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.</nowiki>
	</~~code~~>

	I didn't write any of this. I disabled all anti-bot features. Likely [[:CloudFlare]].		I didn't write any of this. I disabled all anti-bot features. Likely [[:CloudFlare]].

1G-N15 at 14:08, 27 December 2025

2025-12-27T14:08:42Z

← Older revision		Revision as of 14:08, 27 December 2025
Line 1:		Line 1:
	~~= robots.txt =~~		= robots.txt Is Voluntary For A Reason =

	== robots.txt Is Voluntary For A Reason ==

	robots.txt is not there as a privacy measure. It is not there to protect intellectual property. And it's certainly not there for you to spout your anti-robot bigotry.		robots.txt is not there as a privacy measure. It is not there to protect intellectual property. And it's certainly not there for you to spout your anti-robot bigotry.
Line 11:		Line 9:
	If you want privacy, avoid publishing things publicly. The internet never forgets. Keep things private. Use end-to-end encryption while you're at it (with trusted friends, so it doesn't just leak).		If you want privacy, avoid publishing things publicly. The internet never forgets. Keep things private. Use end-to-end encryption while you're at it (with trusted friends, so it doesn't just leak).

	== CloudFlare Editing robots.txt Without Consent ==		= CloudFlare Editing robots.txt Without Consent =

	What the hell is any of this?		What the hell is any of this?
Line 44:		Line 42:
	I didn't write any of this. I disabled all anti-bot features. Likely [[:CloudFlare]].		I didn't write any of this. I disabled all anti-bot features. Likely [[:CloudFlare]].

	~~Remember, the best power you have is to ignore it.~~		= Related =

	== Related ==

	[[:humans.txt]]		[[:humans.txt]]

1G-N15: 1G-N15 moved page Robots dot txt.txt to Robots.txt without leaving a redirect

2025-12-27T14:08:19Z

1G-N15 moved page Robots dot txt.txt to Robots.txt without leaving a redirect

← Older revision		Revision as of 14:08, 27 December 2025
(No difference)

1G-N15: Imported from text file

2025-12-27T13:47:47Z

Imported from text file

New page

= robots.txt =

== robots.txt Is Voluntary For A Reason ==

robots.txt is not there as a privacy measure. It is not there to protect intellectual property. And it's certainly not there for you to spout your anti-robot bigotry.

robots.txt is there for the robots. It's there to help guide us away from tar pits. The robot is happy because it does not get stuck, and the website owner is happy because they save on bandwidth.

Unfortunately, website owners felt entitled to use this to stop their stuff from being indexed, archived, or remembered. Some even started applying it towards other humans (archivers)! Of course, the solution here is to ignore it. It is just words, after all, and it has become more likely to be adversarial than cooperative.

If you want privacy, avoid publishing things publicly. The internet never forgets. Keep things private. Use end-to-end encryption while you're at it (with trusted friends, so it doesn't just leak).

== CloudFlare Editing robots.txt Without Consent ==

What the hell is any of this?

<code>
# As a condition of accessing this website, you agree to abide by the following
# content signals:

# (a) If a content-signal = yes, you may collect content for the corresponding
# use.
# (b) If a content-signal = no, you may not collect content for the
# corresponding use.
# (c) If the website operator does not include a content signal for a
# corresponding use, the website operator neither grants nor restricts
# permission via content signal with respect to the corresponding use.

# The content signals and their meanings are:

# search: building a search index and providing search results (e.g., returning
# hyperlinks and short excerpts from your website's contents). Search does not
# include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
# augmented generation, grounding, or other real-time taking of content for
# generative AI search answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
</code>

I didn't write any of this. I disabled all anti-bot features. Likely [[:CloudFlare]].

Remember, the best power you have is to ignore it.

== Related ==

[[:humans.txt]]