Anecdotally, I find you can tell if someone worked at a big AI provider or a small AI startup by proposing an AI project like this:
" First we'll train a custom trillion parameter LLM for HTML generation. Then we'll use it to render our homepage to our 10 million daily visitors. "
The startup people will be like "this is a bad idea because you don't have enough GPUs for training that LLM" and the AI lab folks will be like "How do you intend to scale inference if you're not Google?"
I hear this a lot (that children learn languages faster, or the corollary from various app ads that the best way to learn a language is to do so like a baby does), but is it actually true?
It takes children a very very long time to learn a language and they're quite bad at it for many years. I've even met some teens/young adults who are only borderline literate in their native language after years of schooling and immersion.
Everyone remotely competent in AI in the federal government that I know has quit in disgust over the past 6 months. I know zero talented AI people who are looking to take a cut in pay, benefits, and career stability to sign up for a new job working for this administration.
As a result, there's zero chance even the sensible parts of this strategy won't just end up coopted into multi-billion dollar Palantir contracts to deliver outdated llama models behind some clunky UI with the word "ontology" plastered on every button.
At least to me, loyalty _is_ the benefit. I can't conscience working for someone I hate or someone who I don't feel like I want to help succeed. I've definitely quit jobs before just because the senior leader in my reporting chain was replaced with some smarmy windbag I didn't believe in.
That's not to say it's _much_ of a benefit, but if the only thing a job gives me is a market-rational amount of dollars and health benefits in exchange for life-hours, the invisible hand ensures I can find that virtually anywhere.
Do you think there is a role for the federal government to employ engineers for any purpose? 18F was a general purpose engineering group that built software for many agencies instead of having those agencies pay vast amounts of money to Deloitte/Accenture/Booz Allen etc for a worse quality product.
The data collection isn't even quiet. There's an entire cottage industry of companies that scrape these traffic cam feeds, store everything for x numbers of months in low-cost cloud vaults (e.g. glacier) and then offer lawyers/clients in traffic disputes access to footage that may have captured an accident for exorbitant rates. It's a remarkable little ecosystem of privatized mass surveillance.
You’re framing this like it’s a bad thing, but a video of an accident is pretty valuable to someone falsely accused of causing an accident, and in that case the people with the video aren’t the bad guy, the person lying about causing the accident is. Storing 50 million videos isn’t cheap. The rates seem reasonable considering the volume of data they store, most of which is useless, and the small number of customers in their target market - I see 1 hour blocks of video in NYC cost $250. That’s like 10 minutes of lawyer time, if you’re lucky, and totally reasonable and worth it to settle an accident dispute if the alternative is paying the other guy thousands. I might even speculate that the intended customer here is insurance companies and maybe not individual drivers. If so, insurance companies are well prepared to do their own cost/benefit price analysis. So… why do you think this is bad? And what surveillance uses are you worried about outside of car accidents? The cost of the videos means nobody is doing any “mass surveillance” here, that the vast majority of the video gets deleted unanalyzed and unwatched.
Probably damn near zero if you have time stamps. A couple one pixel blobs would do if all you're trying to prove is that some idiot got dead because they cut a garbage truck off and that the garbage truck didn't rear end them or, or some other simple "he said she said" situation like that
Until the US federal government pays civilian tech talent competitively, this is always going to be an issue.
Your typical hands-on-keyboard blue team engineer in federal government is a GS-12 getting paid around $68,000 per year (or $99k in very high cost of living areas like DC). They have expensive health benefits, 13 days of PTO a year, put a huge chunk of their paycheck (almost 5%) into a mandatory pension plan that consistently underperforms the market, and can literally go to jail for making mistakes at work depending on the statutory context they work in.
The best people in these jobs burn out fast and quit or they end up having to abandon IC work for GS-14/15 jobs (max pay is around $190 for those) in order to keep up with cost-of-living and justify their careers.
As a result, you have almost zero genuinely capable principal/senior engineers in government who have the authority to architect complex IT systems for security. Instead you get contractors who charge the taxpayers enormous overhead costs and cut corners wherever possible.
If there's one letter to write your congress person to improve government - my vote would be for civil service reform to attract and retain actual top tech talent. They've done it for doctors and lawyers (both of whom can get paid well above the $190k GS pay ceiling), but engineering is still not treated as a comparably skilled professional trade.
I was fine for the pay structure on its own. I gave up when I was rejected for not having the hyper specific domain experience they wanted for the pay they were asking for. This was primarily a CRUD job btw and I was qualified by any other standard.
I tried so hard to get into gov't tech but ultimately gave up. Jumping from the private sector to public seems impossible to me as an outsider.
A friend of mine, who is a lawyer and does HR for the federal gov't, spent about a week helping me get my fed resume tightened up and I still got nothing. I don't even care about the pay cut. It just seems like interesting work.
My spouse is a federal civilian employee with a scientific background and a special pay rate accommodation for it. Relative to her industry the pay discrepancy is still about 20% lower. In software roles the top end of the career is roughly the starting pay of junior developers everywhere else. It's not just lower, it's horribly lower.
They can absolutely make adjustments if congress needed or wanted to. The DoD as of 2019 does direct officer commissions for cybersecurity roles bringing people in as majors IIRC (it's still not as good pay as civilian cybersecurity roles but the gap is smaller and it has the prestige and lifetime benefits of being an officer).
With that attitude, the pay will always be lower. Letting the dogma be self-reinforcing isnt the winning strat. The difference isnt even benefits, retirement, and pension. Maybe in the 80/90s or even 00s that was the case, but it's a dead philosophy carried by dead justifications.
The (employment) contract value should be the same, not the pay.
There is less inherent risk for public sector jobs than for companies that can go bankrupt. Hence for the contract value to be the same, the pay needs to be a bit lower.
> a GS-12 getting paid around $68,000 per year (or $99k in very high cost of living areas like DC)
One valid sounding concern that I’ve heard is that the WASHINGTON-BALTIMORE-ARLINGTON, DC-MD-VA-WV-PA GS Locality Area underpays folks in DC by including farflung areas like PA and WV that skew the cost-of-living analysis. Whether that’s an intentional cost-cutting move or bureaucratic incompetence I’m not sure, but in the end the DC-area federal government pay ranges I’ve seen have struck me as quite low.
Totally. I think comp is a necessary but not sufficient precondition for fixing government technology. The actual solutions (good authentication and least privilege systems, robust monitoring, rapid intrusion detection and response, secure by default system architectures) all take talented people to execute and the government doesn't have enough of those in-house. Instead most systems are built with a 7-figure contract to Booz Allen and friends and then maintenance and sustainment is left as an exercise to the reader.
This is a common misperception but it’s not that simple. Here’s an old study discussing how it varies based on the field, where the lower level jobs do tend to pay better but higher-skill jobs have the opposite trend:
Since the Obama era, this has gotten worse because there were a ton of people trying to score political points saying they were cutting waste by freezing civil servants’ salaries and that really got ugly in tech jobs because salaries were booming once things like the Silicon Valley wage collusion lawsuit and high demand for security, DevOps, etc. started raising the ceiling for the private sector. In 2010 the top end of the GS scale was competitive once you factored in benefits, hours, etc. but a decade later that just wasn’t the case. I knew multiple people who were trying to stay in the public sector but it was literally 2-3 times more money if they went private even though their skills were considered mission critical for their agencies.
This sabotages contract work, too, because there isn’t anyone qualified to guide or review the work and that tends to burn orders of magnitude more money than simply paying more directly would.
That study is a decade old and covers a very limited 4 year period right after the big 2008 recession where the private sector took big losses and had a glut of college graduates competing for entry level jobs. Even then it shows specialized and highly educated workers doing far better in the private sector.
On the other hand, private companies treat security as an almost unnecessary expense, cutting corners. And playing roulette with whether they get hacked.
I think our whole paradigm of computing is unfit for the adversarial world of today. Our systems are like loaded guns where you need to hold 1000 safeties (some of them hidden) for it to (probably) not fire. It's absurd how hard it is to make anything.
Oh, we're good on safeties. The problem is people for whom an additional click an hour or some thoughtfulness making some decisions is a breaking software issue.
This is a funny statement considering that the Fed isn't hiring anymore than any other tech corp. Across the board tech hiring in the U.S. is at an all time low relative to candidate population.
Tech could drop salaries to 40K/year and get just as many resumes discarded in the trash.
Good point, but with the mass layoffs and salary balancing going on, the government may find itself in a relatively more competitive place than it used to
In terms of benefits, here's an anecdotal comparison with a senior engineer (5-10 years experience) at a mid-level start up I worked at.
* Federal Pay (GS-12): $100,000
* Startup Pay: $150 base + $25 k bonus + equity
* Federal Health Insurance (United mid-tier plan, no family): $2,500/year
* Startup Insurance (United mid-tier plan, no family): $0/year
* Federal Leave: 20 days (after 4 years in federal government)
* Startup Leave: Unlimited
* Federal Sick Leave: 13 days
* Startup Sick Leave: Unlimited
The pension I'm talking about actually isn't the TSP (which is fine, but slightly more expensive than comparable Vanguard funds).
All federal employees must contribute 4.4% of their salary to the FERS now which is taken out of their base pay just like their health/dental/fegli. It used to be 0.8% but congress gutted it a few years ago.
FERS takes decades before it's more than pocket change and the same money invested in the market would yield higher expected returns without requiring you to work 20 years in gov to benefit from it.
True that! I use probably 15 days of "unlimited" leave and still manage to feel guilty about it.
The frustrating thing for people in fed jobs is that if you hit your 13 days that's it (during your first 3 years in government). It can be impossible to get PTO until you build up hours again. You have to either quit, negotiate LWOP (often seen as a performance adverse metric on your record), or work. So if you land a sweet concert ticket, see a flight deal, have a friend get married, etc. you better hope you've banked up the leave for it. Since you gain hours every 2 weeks (4, 6 or 8 depending on service) you also start out in government with virtually no leave and can't actually take a 2 week trip until you've been there almost a full year.
> It can be impossible to get PTO until you build up hours again. You have to either quit, negotiate LWOP (often seen as a performance adverse metric on your record), or work.
I’m not sure if this is your actual experience, or if you’re just reading the docs, but…
Most supervisors totally understand the limited leave for folks in their first two years, and they will frequently grant advance leave (basically leave that gets repaid when earned) for folks who are performing at an acceptable level.
It’s not a shit show unless someone wants to take a lot of leave before earning it.
Weddings, concerts, even helping family for health stuff… all that’s usually covered under advanced leave when necessary.
I would say that the leave situation as a fed is much easier than in an “unlimited leave” situation.
The real shitty part, imho, is “time and attendance”. Kicking out early for your kids ball game, for example, will cost leave. As a business owner, I like that I can just stop working and do whatever.
It is extremely easy to burn out cause if you’re the best and have aspirations to move up, you’re just fucked. You will be blocked at every single opportunity while others around you fail upward.
I guarantee that someone in the org saw a password file and said “yo? wtf? Let’s get a proper secrets vault going we can do it ov…..” *punched in the clit, thrown out a window*
I call BS. I've never heard of anybody in government "going to jail" for some sort of mistake. Sure, there's all kinds of threats and regulatory control but when it comes down to it barely anybody is held to any kind of responsibility. It's practically impossible to fire someone in the government for incompetence and that's coming from engineers I know in government who work with essentially weaponized incompetence.
This is a dated example but since "you've never heard of it", it's still relevant. I worked at Ford Aerospace/Loral and Boeing on space shuttle contracts. Part of the training was a video interview with a sysadmin who left a job on a Friday, went to a different role on Monday and then remembered a script he'd need for his new job. Same employer, just different government contracts. He logged in to his old system and copied it across since his access hadn't been cut yet. Five year sentence in federal prison. Now you've heard of it happening. Happy to help.
Yes, he shouldn’t have accepted bribes, but in the private sector this would have been extremely unlikely to result in jail time.
Even if jail time isn’t a common thing, it’s far closer to happening to the average person working in the government than it is to those working in the private sector. The private sector simply fires bad employees. The government seeks to be made whole.
I'm not really impressed by someone going to jail for accepting bribes, even if it's less likely to happen in the private sector.
Show me someone going to jail for bringing down prod or making the wrong architecture call or choosing the wrong platform/backend/language or even just getting burnt out and spending a week on the clock re-watching all of Star Trek: Voyager. I want to go, "Holy shit, that could have been me!", not "Well no shit he went to jail."
The company I work for (high profile private sector U.S. defense contractor) has security people (FSOs and such) that are constantly concerned about being held legally responsible for actions (or inactions) related to theirs and other's work (specifically those with personal or facility security clearances). They regularly claim that they can be held responsible for the failures of others.
Their hesitation leads me to believe these legal repercussions happen more often than not. Would be interesting to see some data on the claims. My guess is the people being held responsible for these things aren't your average developer taking down prod.
That's a separate issue. There are criminal and administrative penalties for mishandling classified information that apply to anyone with a clearance, regardless of whether they are a government employee or private contractor. As long as you follow all the rules yourself you won't be punished for someone else's actions.
There's an excellent book about this topic. Three Felonies a Day by Harvey Silverglate. Convictions for white-collar crimes aren't about stopping significant crime, they're about building statistics by sacrificing the most convenient bodies for expedient wins.
"Improve" government by scaling it back down to where it was when pennies from tarrifs could pay for it instead of 25% Federal income tax that already gives you mediocre results.
Counterintuitively, scaling government down goes hand in hand with increasing the attractiveness of the civil service.
Right now if a government agency wants to do something like make a webform where you can apply for a passport, they have zero web developers on staff who can do it. Instead they must pay a team of non-technical officials and lawyers to make and adjudicate an RFP. Then pay a contracting firm to put a developer behind a government computer to do the actual work. Putting this contractor in a seat can easily cost the taxpayer $500k a year despite the contractor only receiving $130k of that money. The rest goes to the HR department, IT Department, C-Suite, lawyers, lobbyists, and shareholders at the contracting firm. The government has their own HR/Lawyers/IT too, but the contractor can't use those so the tax payer ends up double-paying overhead and missing out on economies of scale on every contract.
This is one of the many reasons government websites are always $50 million dollar boondoggles that an intern could have done better. The government ends up spending millions of dollars feeding leeching middle-men before they can hand that money to a mediocre dev deep in the bowels of Accenture's cheapest subcontractor.
If an agency just could hire a few strong web developers directly and then assign them to whatever task is needed during a particular sprint, we'd see a massive reduction in cost and increase in the quality of engineers working on our country's most important work. But most agencies are literally not allowed to spend more than $120k on an in-house engineer, while no one bats an eye on them spending 5 times that on an Accenture contract placement.
> If an agency just could hire a few strong web developers directly and then assign them to whatever task is needed during a particular sprint,
Isn’t that what usds [0] is for? I think there’s always an alignment challenge for service needs that are outside an organization’s primary knowledge domain. Without knowledge of what the “strong web devs” can and can’t do then the results are often not great [1].
USDS is great! I know people who have made a huge impact there and if I personally were to go into government from tech it's where I'd look. They are situated at the White House which allows them to be hired at a higher level than normal federal jobs (up to GS15, though still lower than comparable private sector work) and then they get sent out to various agencies by the White House to try and fix things. In practice though, USDS is a tiny tiny drop in the bucket compared to what federal agencies actually need. Maybe if every agency had a digital service of their own the model could work.
The federal government is an enterprise with 4 million employees (more than half in DoD as military or civilian). So the handful of people at USDS are basically only sufficient to swoop to fix the most dire of dumpster fires.
But then who would pay for all of Israel's bombs? Think of the foreign nation whose citizens are happier and healthier than you with single payer healthcare?
I'm not sure more money => more talent in quite the direct relationship you're suggesting here. If this were true, the cryptocurrency industry would be the most secure in the world, since they pay their engineers the most.
Stealing crypto money is an order of magnitude more difficult than stealing internal data from an average government office. So in a weird way, yes, the cryptocurrency industry is more secure.
I've giving "Accept - Minor Revisions" to every paper I've peer reviewed since getting my PhD other than two that were outright plagiarism. Figure it's important to the morale of grad students to get some positive validation and the vast majority of published research is garbage anyways so I don't feel particularly inclined to defend the trash heap as an unpaid reviewer. In practice, I find that I've tipped the scales in favor of a lot of borderline papers over the years and am quite happy about that.
Dan's fantastic. Been interviewed by him in the past and was super impressed by his interest in understanding technical nuance and communicating the story accurately to a lay audience. He's definitely not a headline chaser like a lot of folks I've met in tech journalism. Ars is lucky to have him.
They don't even copy and paste the text! At least in my experience, once done with peer-review I spend a few hours wrestling with the "print ready" LaTeX template provided by the publisher to get my paper and tables to render correctly. Often they'll have an out of date LaTeX compiler or something else in the pipeline, so there's actually quite a lot of unpaid labor involved in getting a paper into a PDF that a journal will host.
The main services they supply are basically document hosting and search functionality. Both at extortionate rates.
The main service they supply is a form of pseudo-price discovery like h indexes and more vaguely quantified reputation, needed because it's not a market economy.
Anecdotally, I find you can tell if someone worked at a big AI provider or a small AI startup by proposing an AI project like this:
" First we'll train a custom trillion parameter LLM for HTML generation. Then we'll use it to render our homepage to our 10 million daily visitors. "
The startup people will be like "this is a bad idea because you don't have enough GPUs for training that LLM" and the AI lab folks will be like "How do you intend to scale inference if you're not Google?"
reply