erin
/
ferret
1
1
Fork 0
Code Issues 8 Pull Requests Packages Projects Releases Wiki Activity

Follow robots.txt #2

New Issue
Open
opened 2023-07-26 03:45:44 +00:00 by erin · 0 comments
erin commented 2023-07-26 03:45:44 +00:00
Owner

Make the crawler obey entries in robots.txt.

Make the crawler obey entries in `robots.txt`.
erin added the
bug
 label 2023-07-26 03:45:44 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
Labels
Clear labels   
bug

Something is not working   
duplicate

This issue or pull request already exists   
enhancement

New feature   
help wanted

Need some help   
invalid

Something is wrong   
question

More information is needed   
wontfix

This won't be fixed
No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: erin/ferret#2
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?