There aren't all that many farmers. Americans love the idea of them, but they aren't actually a significant voting bloc. If they had been, we'd have had a more realistic position on migrant farm workers. Any complaints the farmers may have will be drowned out by the sound of how happy people are with the raids.

Those farmers are mostly in deep-red districts. Even if they're unhappy, they're not going to vote for the Democrat -- if there's even one on the ticket. The few who cross party lines won't suffice to to change the outcome in the district.

For the rest of the country... if it results in higher food prices, that might make a difference. Though perhaps less than you'd think. People's perceptions of inflation are only loosely correlated with the actual cost of things. Inflation of "only" 5% doesn't result in massive sticker shock, by itself. It's filtered through people's expectations. If people are generally happy, even substantial inflation will be passed off as merely trying times (for which we must double down on our bootstraps). If people are generally unhappy, it's easy to single out some specific item that has gone up in price and make it the be-all and end-all of consumer budgets.

The balance is close enough that at least one house will likely turn next November. But I've seen nothing to suggest it will be any kind of landslide. It's just enough to nudge a few 51-49 districts to 49-51 instead.