Hello, I am trying to limit referrer spam from urls that look like the first line below. The referrals I am interested in are mostly like the second line:
My site is on a shared hosting account, so my tools are limited to mod_rewrite in the htaccess file. I tried adding the following mod-rewrite rule to catch any referrer url with
http//(anything except '/').(1-4 alphabetic chars, may be followed by '/'):
The filter.php file records the referrer information in a log file, so I can check if any legitimate request might have been caught. I can then modify the htaccess rules to make the necessary corrections. The file also displays a page with an apology, and a button that the visitor can press to enter the site.
I have a couple of issues though, which I was hoping to get some advice on.
Is there a way to redirect referrer spam, and log it, in such a way that Awstats will not count it as a visitor?
Also, the mod_rewrite rule does not seem to be catching anything. I still see the same types of referrals in my Awstats, and my log file stays empty. However after testing my rule from a Google link, it worked fine, including logging the referrer (I changed my rule to detect 'https', then changed it back):
Here is the code for my filter.php:
UPDATE:
I cought a little bugger!
Not quite sure when it started to work, but I remember the last thing that was added to the mod_rewrite rule was the ending "/?".
However, these visits are still recorded in Awstats 'Links from an external page'. Are they counted in the stats for unique visitor? If so, I guess the original question still remains. Is there anything I can tweak in the Awstats configuration file? I have an awstats.mydomain.conf file in folder home/tmp/awstats.
It crossed my mind to change the rewrite rule to 'Fail, but redirect to this page first':
It seemed like too dumb of an idea to even try.
UPDATE:
Apologies I was not sure if this should be posted as a different question.
The variable %{HTTP_REFERER} can return two different strings for the same domain:
Why the difference?
Thanks!
Code:
http://referralspam.ru
http://www.domain.com/viewtopic.phphttp//(anything except '/').(1-4 alphabetic chars, may be followed by '/'):
Code:
RewriteCond %{HTTP_REFERER} ^http://[^/]*\.[a-zA-Z]{1,4}/?$ [NC]
RewriteRule .* /system/services/referrer/filter.php?ref=%{HTTP_REFERER} [L]I have a couple of issues though, which I was hoping to get some advice on.
Is there a way to redirect referrer spam, and log it, in such a way that Awstats will not count it as a visitor?
Also, the mod_rewrite rule does not seem to be catching anything. I still see the same types of referrals in my Awstats, and my log file stays empty. However after testing my rule from a Google link, it worked fine, including logging the referrer (I changed my rule to detect 'https', then changed it back):
Code:
2016-01-06 21:36:29 Referrer redirected by htaccess: https://www.google.com
Code:
<?php
// file: filter.php
// prmissions: 644
$no_error = @ require($_SERVER['DOCUMENT_ROOT']."/../system/services/referrer/filter.inc");
?>
Code:
<?php
// File: filter.inc
define ('HOME_FOLDER', $_SERVER['DOCUMENT_ROOT']."/../system/services/referrer/");
define ('LOGFILE', HOME_FOLDER.'referrer.log');
$backgrImg = "/system/services/referrer/g3sf15.gif";
$textColorNormal = "#ffd700";
$link = "http://www.mywebsite.com";
if(isset($_GET['ref'])) {
$referrer = "Referrer redirected by htaccess: ".strip_tags(trim($_GET['ref']));
}
else if(isset($_SERVER['HTTP_REFERER'])){
$referrer = "Referrer not redirected by htaccess: ".strip_tags(trim($_SERVER['HTTP_REFERER']));
}
else{
$referrer = "Direct access from: ".strip_tags(trim($_SERVER['REMOTE_ADDR']));
}
$time = date('Y-m-d H:i:s', $_SERVER['REQUEST_TIME']);
$str = $time." ".$referrer.PHP_EOL;
error_log($str, 3, LOGFILE); // append to log
?>
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name=viewport content="width=device-width, initial-scale=1">
<title>Referral filter</title>
<style>
body {
color: <?php echo $textColorNormal ?>;
font-size: 100%;
font-weight: bold;
}
ul{
list-style-type: none;
}
.button_container a {
border-bottom: 1px solid #777777;
border-left: 1px solid #000000;
border-right: 1px solid #333333;
border-top: 1px solid #000000;
background-color: gray;
color: <?php echo $textColorNormal ?>;
height: 1em;
padding: 0.2em;
width: 2em;
text-decoration: none;
}
</style>
</head>
<body background="<?php echo $backgrImg ?>" >
<div id="google_translate_element"></div>
<h2>
<ul>
<li>Apologies, the site that directed you here may have</li>
<li>been accidentally flagged as a bad site.</li>
<li><p></p></li>
<li>Please click on the button to enter.<li>
<li><p></p></li>
<li><div class="button_container"><a href="<?php echo $link ?>">Enter</a></div></li>
</ul>
</h2>
<script type="text/javascript">
function googleTranslateElementInit() {
new google.translate.TranslateElement({pageLanguage: 'en', layout:
google.translate.TranslateElement.InlineLayout.SIMPLE}, 'google_translate_element');
}
</script>
<script type="text/javascript"
src="//translate.google.com/translate_a/element.js?cb=googleTranslateElementInit"></script>
</body>
</html>I cought a little bugger!
Code:
2016-01-06 21:51:06 Referrer redirected by htaccess: http://vesta-lada.net/
2016-01-06 21:51:06 Referrer redirected by htaccess: http://vesta-lada.net/
2016-01-06 21:51:07 Referrer redirected by htaccess: http://vesta-lada.net/However, these visits are still recorded in Awstats 'Links from an external page'. Are they counted in the stats for unique visitor? If so, I guess the original question still remains. Is there anything I can tweak in the Awstats configuration file? I have an awstats.mydomain.conf file in folder home/tmp/awstats.
It crossed my mind to change the rewrite rule to 'Fail, but redirect to this page first':
Code:
RewriteRule .* /system/services/referrer/filter.php?ref=%{HTTP_REFERER} [F, L]UPDATE:
Apologies I was not sure if this should be posted as a different question.
The variable %{HTTP_REFERER} can return two different strings for the same domain:
Code:
http://www.domain_name.com
http://domain_name.comThanks!
Last edited:
