14 Commits (f3422a494910ec729e87c97fa370107f2c759c4d)

All Branches

Search

Author	SHA1	Message	Date
~erin	f3422a4949	Filetype filtering	2023-07-25 20:25:49 -04:00
~erin	ed53ec320e	Fix panic, preventing full crawling	2023-07-25 20:11:29 -04:00
~erin	c141135847	Sanitize HTML?	2023-07-25 19:54:05 -04:00
~erin	758c16b78b	Justfile	2023-07-25 19:47:39 -04:00
~erin	561ef2dfb4	Test database	2023-07-25 19:28:17 -04:00
~erin	541deca5f6	Add README	2023-07-25 19:27:56 -04:00
~erin	326a6b8042	Should crawl pages after a certain age	2023-07-25 18:33:38 -04:00
~erin	57684c037e	Fix recursive crawling	2023-07-25 18:16:45 -04:00
~erin	159164674e	Crawl through URLs via allowlist	2023-07-25 17:36:41 -04:00
~erin	892dc35f74	Crawl all at once, test cacache	2023-07-25 16:58:25 -04:00
~erin	86f6d1a631	Fetch pages in sitemap	2023-07-25 16:11:09 -04:00
~erin	bc062a39f3	Use basic configuration file	2023-07-25 15:24:14 -04:00
~erin	03f3cfd5c0	config changes	2023-07-25 14:59:47 -04:00
~erin	971b217a94	Basic project setup	2023-07-25 14:52:19 -04:00