As another extremely simple idea, in my personal experience, just using regular Lua table syntax for serialization and then pre-compiling it with luac so it could be loaded quickly via dofile(), produced results just as fast for loading it in as using lua-protobuf. (I don't remember the difference between writing out Lua tables vs. lua-protobuf because my use case needed to read the data more often than generate it, but it must have not been big enough, if any, otherwise I would probably remember it.) I was loading gigabytes of data for large batch processing.
We exactly did the first half in my previous job, because JSON was really painful to use as is (and none of JSON5 or HOCON or whatever were popular enough at that time). The second half was substantially different due to our idiosyncratic requirements though.
For what it’s worth, luac has been deprecated now.
[EDIT] sorry folks! Must have misunderstood something I read a while back. Trying to dig it up now. But I could’ve sworn I’d read somewhere that this at least wasn’t suggested, and I thought it was also removed from the build in a 5.4 patch. Will circle back if I find what I’m looking for.
Where did you see that? I'm skeptical of that claim because I know some embedded uses of Lua strongly utilize pre-compiling to Lua bytecode so they can keep their hardware as slim as possible. I know some will even strip out the Lua compiler from the binary they ship to shrink the profile even more. Also, many video games like to pre-compile their shipped Lua scripts to reduce their load times.
I know that the Lua team has internal private repositories, and that luac is developed in a separate repo for them. I've seen occasional reports on the Lua mailing list that luac.c is forgotten or not updated in non-official releases. That is because those source drops didn't go through the full official release process which includes merging in from their luac repo. Maybe you are confusing these intermediate source drops with deprecation? If there is deprecation, I would like to see the details on that. I presume they would be introducing some kind of replacement that addresses all the real world use cases that rely on the abilities of pre-compiling to Lua bytecode.
> But I could’ve sworn I’d read somewhere that this at least wasn’t suggested
Maybe you’re thinking about the use of untrusted bytecode? Loading it is strongly discouraged because that’s insecure - Lua has nothing like a bytecode verifier.
Are you asking if there are special numbers or strings or some other kind of plain data that would glitch out the interpreter when they're read back in? That would be an impressive failure of a programming language.
Yes it's safe as long as you're serializing correctly (which isn't very hard).
By definition you can't trust that "untrusted data" has been serialized correctly.
Lua has strong sandboxing capabilities (i.e. ability to limit and control the environment visible from a chunk of code), but the Lua authors years ago explicitly disclaimed the ability to sandbox untrusted code. The compiler and runtime are not bug free. They don't have the resources, notwithstanding that compared to any other non-formally verified implementation their track record is pretty decent, even compared to past and current efforts from Sun, Microsoft, Mozilla, and Google. If you want to run untrusted Lua code, Lua sandboxing should be just the first of multiple line of defense, just as modern web browsers rely on various operating system mechanisms to constrain breakouts.
> By definition you can't trust that "untrusted data" has been serialized correctly.
Please read the comment again. They said the person loading the data is the same person that serialized it. The data came from an untrusted source prior to being serialized.
For example, consider a guestbook program. People send it untrusted text and then the program serializes it into a database. Reading the database back is safe.
By definition you can't trust that "untrusted data" has been serialized correctly.
Lua has strong sandboxing capabilities (i.e. ability to limit and control the environment visible from a chunk of code), but the Lua authors years ago explicitly disclaimed the ability to sandbox untrusted code. The compiler and runtime are not bug free. They don't have the resources, notwithstanding that compared to any other non-formally verified implementation their track record is pretty decent, even compared to past and current efforts from Sun, Microsoft, Mozilla, and Google. If you want to run untrusted Lua code, Lua sandboxing should be just the first of multiple line of defense, just as modern web browsers rely on various operating system mechanisms (process separation, filesystem to constrain breakouts.
