What is this?

This is a fast and very simple to use PHP (>= 8.4.0) library to parse “robots.txt“ files and check paths against the rules they define, optionally using “Google interpretation” of the Robots Exclusion Protocol (REP) as defined by RFC 9309. It supports the «Sitemap» and «Crawl-Delay» unofficial fields. «Sitemap» lines can be anywhere in the “robots.txt“ file, even between «User-Agent» lines, in which case they don’t logically break a multiple «User-Agents» block of lines defining a group.

How can i use it?

You just need to clone this repo with, e.g., git clone https://git.lattuga.net/Jones/rob.txt.git, then require [path/to/]robtxt.php from the PHP script where you want to use it, and then use it; for example, like this:

require __DIR__.'/robtxt.php';
$root='https://domain.tld';
$robtxturl="{$root}/robots.txt";
$myagent='pippo';
$rtimeout=10;
$paths=['/foo/bar/baz.html', '/baz/foo/bar.php?a=1&b=5'];
$robtxt=new robtxt($robtxturl, $myagent, true, $rtimeout);
if (!is_null($robtxt->error)) {
	echo "Warning: could not fetch “robots.txt” file from {$robtxturl}: {$robtxt->error}.\n";
} else {
	$delay=$robtxt->getDelay();
	$context=stream_context_create(['http'=>['user_agent'=>$myagent, 'timeout'=>$rtimeout]]);
	foreach ($paths as $path) {
		if ($robtxt->isPathAllowed($path, false, false)) {
			if ($delay>0) sleep($delay);
			$html=@file_get_contents("{$root}{$path}",false,$context);
			// ... do something with "$html"
		} else {
			echo "Warning: did not try to fetch {$root}{$path} because {$robtxturl} doesn’t allow us.\n";
		}
	}
	$sitemaps=$robtxt->getAllowedSitemaps(false);
	foreach ($sitemaps as $sitemap) {
		$sitemap=@file_get_contents($sitemap,false,$context);
		// ... do something with "$sitemap"
	}
}

In more details

In more details, you just need to create a new object using the new statement to instantiate and initialise the robtxt class, which has this syntax, robtxt(<URL or path>, <your user-agent>, <google mode>, <read timeout>), where:

1. <URL or path> is the full URL or path (for possible local testing purposes) of a file in “robots.txt” format (e.g.: https://domain.tld/robots.txt, /local/path/to/robots.txt);
2. <your User-Agent> is the “User-Agent” you use;
3. <google mode> can be one of true and false, with true being the value we suggest to use; if true, only the rules defined in the given “robots.txt“’s group(s) under the User-Agent with the token which has the longest match against <your User-Agent> will be considered (if more than one group with different User-Agent tokens have matches with the same length, only the rules defined by those among them which share the same User-Agent token appearing first most near the end of the given “robots.txt” will be considered); if false, every rule defined in every group with a User-Agent token matching against <your User-Agent> will be considered (still, any “catch all agents” groups of rules —i.e., those under a User-Agent with a * token— will be superseded by any group under a User-Agent with a more specific token matching <your User-Agent>);
4. <read timeout> is the maximum number of seconds you want your script to wait for connecting and fetching the given “robots.txt” before setting the $robtxt->error variable to the appropriate “timeout error” string instead of its default null value.

Then you need to check whether fetching the given “robots.txt” succeeded, by cheking if $robtxt->error is null: if it is, the file was correctly fetched; if it’s not, the file could not be fetched, and $robtxt->error is a string with a related short message telling why.

Then you can check whether the rules defined in the given “robots.txt” file allow or disallow <your User-Agent> to access some paths, using the isPathAllowed(<path>, <google mode>, <show matches>) function, where...

1. <path> is the path you want to check against the rules;
2. <google mode> can be one of true and false, with false being the value we suggest to use; if true, when an allow/disallow rule pattern defined in the given “robots.txt” file does not end with a $, it works as if it ended with a *, so its possible match against the given <path> is always as long as the given <path>; if false, its possible match is as long as the actual match; also, if a disallow rule pattern match has the same length of an allow rule pattern match, when <google mode> is set to true, the rule defined by the longest pattern wins (if the two patterns have the same length, the allow rule wins); when it is set to false, the rule with the highest “specificity rate” wins, with its “specificity rate” being calculated as the pattern length minus the number of * chars it may contain, plus 1 if the pattern ends with a $ (if the two patterns have the same “specificity rate”, the allow rule wins);
3. <show matches> can be one of true and false; if true, the function will output any allow/disallow matches on STDOUT; this can be used for testing or debugging purposes.

Then, if the path you want to check is allowed, you can try to fetch its content and do something with it, maybe after waiting the number of seconds defined by a possible Crawl-Delay, using the getDelay() function, which takes no arguments.

You can also get an array with the sitemaps which <your User-Agent> is allowed to access using the getAllowedSitemaps(<google mode>) function, where <google mode> is just passed to the already described isPathAllowed(<path>, <google mode>, <show matches>) function, and then try to fetch and parse them.

What else?

The test subdirectory contains a robotstxt.php script that you can use to test robtxt.php against local or remote “robots.txt” files. You can read its help text by calling it with the -h or --help option.