Sandboxing Java Code, a Modern Approach

When I started work on this game engine, back in 2016, I'd been planning to eventually use a Java SecurityManager to allow code from  mods to run in a reasonably safe manner, but Oracle has chosen to not only deprecate the SecurityManager, but in Java 17, they've started removing vital parts of it from the base libraries of Java.

So, I've changed my design plans for the scope of sandboxing that will be used and I've been wracking my brain for an alternative.

For the past couple of weeks, during my spare time, I've been experimenting with a strange, but surprisingly viable alternative: a custom ClassLoader that analyzes the Java bytecode of each class it loads, before it loads it, and then totally refuses to load the class if it makes use of any forbidden classes, fields or methods, by throwing a variation of a LinkageError.  This completely prevents the offending class from being able to ever run code.

The nice thing I was able to do was make the error very informative.  I'm using a bytecode manipulation library called ASM for the analysis, so I'm able to produce an entire list of forbidden classes, fields and methods that the refused class was compiled to make use of.

The only downside I'm seeing to this approach is the extra work involved in deciding which classes of the JVM are safe to be used by untrusted code, which can be quite time-consuming and will have to be re-evaluated with each new release of Java.

So, while my original plan was to totally sandbox all mods, except for the bootstrap mod, now I'm debating.  Currently, my short-term plans involve introducing sandboxed, programmable, wired furniture items to the game, which will allow the end-user to upload their own code, to govern how those wired items behave.  There will also be the opportunity to build interactive furniture items, fairly soon. When the user interacts with these, a user interface mostly driven by the code they've uploaded will be displayed.

At this point, my new sandboxing code is between 50%-70% done, so you may see this new feature become available soon.

In the long term, I really have to make up my mind about the sandboxing of mods, but I'm leaning toward using my new sandboxing code to do the job.  The other alternative is to not bother, leaving mods with the freedom to do as they please.  Likely I'll be doing both, by making this an optional feature that can be disabled by whatever developer decides to make use of the game engine.