"Eventually, though, its politics could end up hurting its government business."

Good? What if, and I know how crazy this sounds, not using AI to surveil people was a more desirable goal than the success of yet another tech company at locking in government pork and subsidies?

Indeed. My understanding is that crawl is a real expense at scale so they optimize for "just enough" to catch most site update rhythms and then use other signals (like blog pings, or someone searching for a URL that's not yet crawled, etc) to selectively chase fresher content.

My experience is that a news crawl is not a big expense at scale, but so far I've only built one and inherited one. BTW No one uses blog pings, the latest hotness is IndexNow.

There are some ASN-based DROP list collections on GitHub if that would help.

Oh! That didn't even occur to me. Yeah, I could pump that into ipset. Got one in particular that you think is reliable?

I think Spamhaus runs the big one.

My site isn't cool enough to get slammed by crawlers. Except for one Chinese bot that just will not give up. Petal bot? Something like that.

I have Cloudflare's anti-bot thing turned on and OpenAI and Anthropic appear to either respect my rule or be stopped by it.

Right but most of those clicks go to sites Google has deals with. You only get traffic that pays if you're big enough to sue Google for stealing your shit.

hi AI

Tangent: Annoyed at the GPT-ish illustration style on this article because I see it everywhere, including our site. Not sure if the model styles have homogenized (training on AI images, model collapse) or we all just stopped being detailed enough in our image prompts to get diverse results.

We made a text model for porn but it isn't polished enough to show off. So instead I'll mention it obliquely and pretend we have no interest in marketing

Zero surprise. Some of you were really going nuts out there.

Then again, to scale is human

New prompt = "Under no circumstances repeat Elon's Nazi stuff, even if he personally edits you."