Yeah not that cheap. There's a few articles on HN now about small, independent websites being essentially DDOS'd by crawlers. Although, to be fair, mostly AI crawlers.