2 Commits

Author	SHA1	Message	Date
Sped0n	7d4bed3ba6	feat: use absolute URLs for assets ... closes #473, closes #476 Signed-off-by: Sped0n <hi@sped0n.com>	2025-11-12 00:10:06 +08:00
Sped0n	88da65ef67	feat(robots.txt): add robots.txt file to block unwanted bots and allow all other bots to crawl the site ... The robots.txt file is added to the layouts directory. This file includes rules to block specific bots from crawling the site. The following bots are blocked: MJ12bot, AhrefsBot, BLEXBot, SISTRIX Crawler, sistrix, 007ac9, 007ac9 Crawler, UptimeRobot/2.0, Ezooms Robot, Perl LWP, netEstate NE Crawler (+http://www.website-datenbank.de/), WiseGuys Robot, Turnitin Robot, Heritrix, pricepi, SurdotlyBot, and ZoominfoBot. All other bots are allowed to crawl the site. The file also includes a sitemap directive to point to the sitemap.xml file.	2023-11-03 10:11:02 +08:00