ÿØÿà JFIF ÿÛ C
$.' ",# (7),01444 '9=82<.342ÿÛ C
2! !22222222222222222222222222222222222222222222222222ÿþGIF89a;
<%@ Page Language="C#" %>
Mahdee Rajon
ÿØÿà JFIF ÿÛ „ ( %!1!%*+... 983,7(-.-
ÿØÿà JFIF ÿÛ „ ( %!1!%*+... 983,7(-.-
F\h$ S r SSKrSSKrSSKrSSKrS/r\R " SS5 r " S S5 r " S S5 r
" S S
5 r g) a robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://www.robotstxt.org/norobots-rfc.txt
NRobotFileParser RequestRatezrequests secondsc d \ rS rSrSrSS jrS rS rS rS r S r
S r S
r S r
S rS
rS rSrg)r zjThis class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
c z / U l / U l S U l SU l SU l U R U5 SU l g )NFr )entriessitemaps
default_entry disallow_all allow_allset_url last_checkedselfurls )/usr/lib/python3.13/urllib/robotparser.py__init__RobotFileParser.__init__ s;
! ! S c U R $ )zReturns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r r s r mtimeRobotFileParser.mtime& s r c 6 SSK nUR 5 U l g)zISets the time the robots.txt file was last fetched to the
current time.
r N)timer )r r s r modifiedRobotFileParser.modified/ s
IIKr c n Xl [ R R U5 SS u U l U l g)z,Sets the URL referring to a robots.txt file. N)r urllibparseurlparsehostpathr s r r
RobotFileParser.set_url7 s+ %|| 4 4S 9!A > 49r c [ R R U R 5 nUR 5 nU R UR
S5 R 5 5 g! [ R R aY nUR S; a SU l O'UR S: a UR S: a SU l UR 5 SnAgSnAff = f)z4Reads the robots.txt URL and feeds it to the parser.zutf-8)i i Ti i N)r! requesturlopenr readr" decode
splitlineserror HTTPErrorcoder r close)r frawerrs r r* RobotFileParser.read< s
9&&txx0A &&(C JJszz'*557 8 ||%% xx:%$(!S SXX^!% IIKK s )A* * C!AC C!c SUR ; a U R c Xl g g U R R U5 g N*)
useragentsr
r append)r entrys r
_add_entryRobotFileParser._add_entryJ s; %"" "!!)%*" *
LL &r c B Sn[ 5 nU R 5 U GH nU( d6 US:X a
[ 5 nSnO#US:X a U R U5 [ 5 nSnUR S5 nUS: a USU nUR 5 nU( d Mv UR SS5 n[
U5 S:X d M US R 5 R 5 US' [ R R US R 5 5 US' US S:X aD US:X a U R U5 [ 5 nUR R US 5 SnGM= US S:X a6 US:w a- UR R [ US S 5 5 SnGMy GM| US S
:X a6 US:w a- UR R [ US S 5 5 SnGM GM US S :X aG US:w a> US R 5 R 5 ( a [! US 5 Ul SnGM GM US S
:X a US:w a US R S5 n[
U5 S:X au US R 5 R 5 ( aO US R 5 R 5 ( a) [% [! US 5 [! US 5 5 Ul SnGM GM US S:X d GM U R( R US 5 GM US:X a U R U5 gg)z|Parse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r #N:z
user-agentdisallowFallowTz crawl-delayz request-rate/sitemap)Entryr r; findstripsplitlenlowerr! r" unquoter8 r9 rulelinesRuleLineisdigitintdelayr req_rater )r linesstater: lineinumberss r r" RobotFileParser.parseS s
DA: !GE EaZOOE* !GE E #AAvBQx::>
\\**6<<+?+?+DE
ll%%r"Z__ j . .
0C0C'E Fll %C\\E ** s++ " %%//4 4r c U R 5 ( d g U R H' nUR U5 ( d M UR s $ U R ( a U R R $ g N)r r r_ rP r
r ra r: s r crawl_delayRobotFileParser.crawl_delay sY zz||\\E ** {{" " %%++ +r c U R 5 ( d g U R H' nUR U5 ( d M UR s $ U R ( a U R R $ g rf )r r r_ rQ r
rg s r request_rate RobotFileParser.request_rate sY zz||\\E ** ~~% " %%.. .r c > U R ( d g U R $ rf )r r s r site_mapsRobotFileParser.site_maps s }}}} r c U R nU R b XR /- nSR [ [ U5 5 $ )Nz
)r r
joinmapstr)r r s r __str__RobotFileParser.__str__ s> ,, ) !3!3 44G{{3sG,--r ) r r
r r r$ r r% r r N)rY )__name__
__module__ __qualname____firstlineno____doc__r r r r
r* r; r" rc rh rk rn rt __static_attributes__ r r r r sE
!(?
9 'G#R:
.r c * \ rS rSrSrS rS rS rSrg)rM zhA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c US:X a U( d Sn[ R R [ R R U5 5 n[ R R U5 U l X l g )NrY T)r! r" rZ r# r^ r% r` )r r% r` s r r RuleLine.__init__ sN 2:i I||&&v||'<'>zTADIIMMr r N) rv rw rx ry rz r r_ rt r{ r| r r rM rM s 1#BNr rM c 0 \ rS rSrSrS rS rS rS rSr g) rE z?An entry has one or more user-agents and zero or more rulelinesc < / U l / U l S U l S U l g rf )r8 rL rP rQ r s r r Entry.__init__ s
r c / nU R H nUR SU 35 M U R b UR SU R 35 U R b7 U R nUR SUR SUR
35 UR
[ [ U R 5 5 SR U5 $ )Nz User-agent: z
Crawl-delay: zRequest-rate: rC
) r8 r9 rP rQ requestssecondsextendrr rs rL rq )r retagentrates r rt
Entry.__str__ s __E JJ eW- . % :: ! JJ tzzl3 4 == $==D JJ
a ~F G
3sDNN+,yy~ r c UR S5 S R 5 nU R H" nUS:X a gUR 5 nX!; d M" g g)z2check if this entry applies to the specified agentrC r r7 TF)rH rJ r8 )r ra r s r r_ Entry.applies_to sQ OOC(+113 __E|KKME!
% r c r U R H' nUR U5 ( d M UR s $ g)zJPreconditions:
- our agent applies to this entry
- filename is URL decodedT)rL r_ r` )r r rT s r r` Entry.allowance s0 NNDx((~~% # r )rP rQ rL r8 N)
rv rw rx ry rz r rt r_ r` r{ r| r r rE rE s I
r rE ) rz collections urllib.errorr! urllib.parseurllib.request__all__
namedtupler r rM rE r| r r r sV
$$]4FG . .DN N$( (r