It’s not common, but it does happen. Andy Weir, author of “The Martian” and “Project Hail Mary”, originally gave his work away for free online on his website. He only self-published to Kindle (for the lowest possible price setting, 99 cents) because some of his fans didn’t know how or didn’t want to manually install his home-rolled ePubs on their devices, and begged him for the Amazon/Kindle distribution.
The US federal government is in yet another shutdown right now, so how would some sub-agency even know if there were an unplanned outage, who would report it, and who would try to fix it?
Actually, “op and crew” were the Plaintiffs (well, really the Petitioners, to be pedantic) who already sued the VA for this database in federal court (SDNY), and won that multi-year lawsuit, and even won our attorneys fees too. If you had checked our website, you’d see we even posted the court papers online for free, from both sides — and the judge’s order in our favor, of course.
I think there’s a way you could do this ethically, but personally I believe that the way you are implementing this is unethical. Obviously you are much more familiar with what you’re doing than I am, and I’d be happy to be convinced otherwise. But I think at the very least you could work on your messaging about this but there are a few things I noticed and raised yellow flags:
1. Primarily, you seem to mention winning your lawsuit and legal battle with the VA a lot to justify what you’re doing, but just because what you’re doing is legal does not make it ethical. I do not believe the FOIA was ever intended to be used to expose an API for accessing veterans’ medical records, and even if it did intend to do that for some reason, it would still be wrong to make that data completely available to the public.
2. Your framing of your legal battles with the VA gives me the impression that you are seeking to be vindictive or spiteful with your Fax-API. It’s hard to believe you are doing this because you care about Veterans when you’re actively forcing the VA to expend what I imagine must be considerable time and resources to comply with your requests. Maybe you do have a justifiable vendetta against them but I don’t see how this makes them any better.
3. Some of this data is so recent and potentially damaging that there seems no way to justify making it accessible to the public because of how it benefits genealogy or historians.
All that said I do genuinely believe that this could be an amazing resource for historians, genealogists, and people wanting to learn more about their deceased relatives. I just personally believe that when dealing with such sensitive data you have an obligation to treat it more carefully than you seem to be, even if you are justifying it to yourself as “just” making it 1000x easier to access and telling the world exactly how to do it through legal means.
Yes, a terrible fire, although there are efforts ongoing to restore some of those files, even reading the data from charred papers and edges with newer technology.
However, these particular files (benefits claims files, or C-Files) are a different type of file and never burned. Better yet, they often have some parts of the veteran’s OMPF that were copied *into* the C-File, to establish eligibility for those benefits — copies that were made before the fire! In other words, these files could serve as partial backups…
We love MuckRock! And we made the original FOIA request to the VA for this dataset via MuckRock’s platform. You can see the actual screenshot in the “Reclaiming These Records” legal papers section. They also get a shoutout in our colophon for indirectly inspiring the FOIA-by-fax-via-web method, although I believe their site uses e-mail, including interfacing with agency FOIA portals when possible.
Indeed. This is ABSOLUTELY not the first time we’ve dealt with a government agency (at the local, state, or federal levels) providing a copy of a public dataset to Ancestry.com and not to the general public. Our taxpayer-funded data keeps ending up solely behind a $300/year paywall. It’s not fair.
(Also, the stripped-down version of BIRLS that has been on the Ancestry website for a while now is much smaller and older.)
Ancestry has a somewhat smaller copy of the BIRLS database online, covering just the years 1850-2010 [1], and it seems to have been published on their website in 2011:
https://www.ancestry.com/search/collections/2441/
Our data set from the VA contains data through mid-2020, and was turned over to us in 2022 after undergoing extensive double-checking by the agency, including through non-public VA sources, to confirm the veterans really were all deceased. There's a paper showing the agency's methodology on our site, which we FOIAed from them.
There are a significant number of deceased veterans whose data is *not* included in the BIRLS database, because they (or their family/heirs) simply did not have any contact with the VA concerning benefits in or after the 1970s, which is when the database was first starting to be built. That is, their files almost always still exist on a warehouse shelf somewhere, but they weren't active any time in the past fifty years so they didn't get pulled and indexed into the database. You can still make a FOIA request to the agency asking for one of those files, but the VA will have a lower chance of successfully finding the file, and it usually will take longer for you to get a response.
[1] 1850 is very likely an approximation. While there are certainly deceased veterans listed in the BIRLS database who had birthdates or deathdates in the mid nineteenth century and/or service in the late nineteenth century, they are relatively few. Many of them are actually veterans with likely birthdates or deathdates in the twentieth century whose data seems to have been initially recorded by the VA with a two digit year of birth or death or enlistment/entry, and then assigned to the wrong two digit prefix, causing an incorrect four digit year of birth or death or year of entry/enlistment into service.
In other words, the VA's historic data is very messy and is a great example of an actual Y2K issue.
Yes, that’s on purpose. SSNs of *deceased* people are public, not private. They are never reused. They are available under FOIA from other sources as well, such as the SSDMF.
The processes for getting these very particular records (C-Files, as opposed to something like an OMPF or other better known military records) has been horrendously broken for years. They were almost completely inaccessible from this specific agency (the VBA, inside the VA) their entire existence. Only 5% of the files have been turned over to NARA, even for records that are very old.
And even now, the “processes” to get the records, as defined by a 58+-year-old law (FOIA) are not really being followed. An agency refusing to process any FOIA requests except by fax (!) is insane, in this day and age. But more specifically, it’s against the law. A letter AND an e-mail are supposed to work. Hence our use of a fax API on this website…
Furthermore, the “requirement” that a FOIA requester must hand-sign the paperwork is absolutely made up by this agency. Hence our signature widget on this website…
Point being, if they’re going to shamelessly ignore or misinterpret the federal law, we are going to just jump through those hoops and say no, we want the files, please do your jobs.
The VA worked to confirm that everyone in this dataset is deceased, in order to satisfy the judge’s order, and produced an internal document about how they did it — which we then FOIAed and posted online too. (It’s up on the site, next to the legal paperwork.) The veterans and their SSNs are believed to have been deceased prior to mid-2020, checked by the VA’s internal datasets as well as public data sets such as the SSDMF. And SSNs of deceased people are *not private*, since they are never reused. The Social Security Administration also makes copies of all deceased peoples’ original SS-5 applications available to the public under FOIA.
The VA is obligated to follow the law as it relates to open records. Broadly speaking, America actively chooses to treat vets poorly. Call your Congressional rep (who directly controls policy that controls funding) versus taking your feelings out on your fellow citizen working for little or no pay to encourage government accountability as it relates to their legal obligations.
So you would prefer I sue the VA?! They tried to preserve my families' privacy and reduce risk to current heirs like me. No, doxxing the dead cause "letter of the law matters" for some stupid side project like this is far from commendable. I will bet those of us with a lot of family in there, especially recently deceased, will feel very different than you or the CEO of doxx the dead.
Thanks. The original data set, as provided by the VA, has all sorts of data errors and oddities in it. The major ones involving surnames include the inconsistent use of apostrophes in names like O’BRIEN, often written as O BRIEN, and/or vice versa — or the inconsistent formatting of MC and MAC names like MCMAHON as MC MAHON, and/or vice versa. There are also some names where the VA includes an errant dash, not meant to be a hyphen, and other mistakes, as well.
So we try our best to help a user find the veteran even with the dirty data we have. For example, there is code here (using a common NPM package) to convert a user’s potential typed accent marks to a non-accented version of the same letter. In compound surnames we will also break up the surname on a space or a hyphen and search both parts, but not if a surname part is three letters or fewer. It’s imperfect but we have to work with the data we’ve got and can’t and shouldn’t normalize or clean the underlying file.
>names like MCMAHON as MC MAHON, and/or vice versa.
My mom has a Van name and it's hell trying to use government and insurance websites, because they'll take the space out or add one in irregardless of what you use when signing up and then fail to find the account when doing a lookup for things like password resets or for activating the account that they created for her.
It'd seem logical that some sort of fuzzy matching for apostrophe and spaces would be built in, but I've yet to find a government site where that's the case.
