crawler looks malicious but attacker says not

[jeffschips]

Hello. I hope everyone is healthy and safe.

mod_security keeps flagging the following activity as malicous. When I reported it to Amazon they forward to party doing the scanning and their response is that they are just looking for robots.txt. However the log, below, appears a bit more than just looking for robots.txt.

Here it is and can someone weigh in on what it's about:

[Thu Feb 02 08:01:23.986104 2023] [:error] [pid 7088:tid 47276049303296] [clientxxxxxx:34986] [clientxxxxxx] ModSecurity: Access denied with code 403 (phase 1). Match of "rx ^(??:\\\\*|[^\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]+)\\\\/(?:\\\\*|[^\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]+))(?:\\\\s*+;\\\\s*+(??:charset\\\\s*+=\\\\s*+(?:\\"?(?:iso-8859-15?|windows-1252|utf-8)\\\\b\\"?))|(??:c(?:h(?:a(?:r(?:s(?:e[^t\\"(),\\\\/:;<=>?![\\\\x5c\\\\]{}]|[^e\\"(),/:;<=>?![\\\\x5c ..." against "REQUEST_HEADERS:Accept" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1162"] [id "920600"] [msg "Illegal Accept header: charset parameter"] [data "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.4"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [hostname "xxxxx.com"]
[uri "/robots.txt"] [unique_id

 

[cPRex]

Hey there! To me, I agree with their assessment - it looks like they are checking robots.txt and ModSec is just blocking that specific request. I don't see anything malicious there. That specific block of code looks like this, and restricts character sets and headers:

# Restrict response charsets that we allow.
# The following rules make sure that the response will be in an ASCII-compatible charset that
# phase 4 rules can properly understand and block.
#

#
# Some servers rely on the request Accept header to determine what charset to respond with.
# This rule restricts these to familiar charsets.
#
# Regular expression generated from util/regexp-assemble/data/920600.data.
# To update the regular expression run the following shell script
# (consult util/regexp-assemble/README.md for details):
#   util/regexp-assemble/regexp-assemble.py update 920600
#
SecRule REQUEST_HEADERS:Accept "[email protected] ^(?:(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+)\/(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+))(?:\s*+;\s*+(?:(?:charset\s*+=\s*+(?:\"?(?:iso-8859-15?|windows-1252|utf-8)\b\"?))|(?:(?:c(?:h(?:a(?:r(?:s(?:e[^t\"(),\/:;<=>?![\x5c\]{}]|[^e\"(),/:;<=>?![\x5c\]{}])|[^s\"(),/:;<=>?![\x5c\]{}])|[^r\"(),/:;<=>?![\x5c\]{}])|[^a\"(),/:;<=>?![\x5c\]{}])|[^h\"(),/:;<=>?![\x5c\]{}])|[^c\"(),/:;<=>?![\x5c\]{}])[^\"(),/:;<=>?![\x5c\]{}]*(?:)\s*+=\s*+[^(),/:;<=>?![\x5c\]{}]+)|;?))*(?:\s*+,\s*+(?:(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+)\/(?:\*|[^\"(),\/:;<=>?![\x5c\]{}]+))(?:\s*+;\s*+(?:(?:charset\s*+=\s*+(?:\"?(?:iso-8859-15?|windows-1252|utf-8)\b\"?))|(?:(?:c(?:h(?:a(?:r(?:s(?:e[^t\"(),\/:;<=>?![\x5c\]{}]|[^e\"(),/:;<=>?![\x5c\]{}])|[^s\"(),/:;<=>?![\x5c\]{}])|[^r\"(),/:;<=>?![\x5c\]{}])|[^a\"(),/:;<=>?![\x5c\]{}])|[^h\"(),/:;<=>?![\x5c\]{}])|[^c\"(),/:;<=>?![\x5c\]{}])[^\"(),/:;<=>?![\x5c\]{}]*(?:)\s*+=\s*+[^(),/:;<=>?![\x5c\]{}]+)|;?))*)*$" \
    "id:920600,\
    phase:1,\
    block,\
    t:none,t:lowercase,\
    msg:'Illegal Accept header: charset parameter',\
    logdata:'%{MATCHED_VAR}',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-protocol',\
    tag:'paranoia-level/1',\
    tag:'OWASP_CRS',\
    ver:'OWASP_CRS/3.3.4',\
    severity:'CRITICAL',\
    setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}'"

 

[jeffschips]

Thank you! solved.