libobs, frontend: Reject non-default modules in obs-plugins directory

[cg2121]

Description

If a third party plugin is in the OBS main directory, and OBS is not in portable mode, reject the plugin.

Motivation and Context

Replaces #12290

How Has This Been Tested?

Loaded third party plugin in main directory and made sure it wasn't loaded.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[PatTheMav]

I'm afraid we need a more holistic solution to untangle the mess OBS Studio has navigated itself into and not a solution (particularly as the plugin location for portable mode also needs to change).

What needs to happen is that module loading (as an initialisation unit) needs to be split up into two phases:

• In the first phase all plugins from a hard-coded module path are loaded. This path cannot be changed and is platform specific.
• In the second phase all plugins from the module directory array are loaded.

That establishes an architectural foundation to even start separating core modules from plugins and allows the next steps:

1. Core modules would move into a directory like obs-core to clearly distinguish it from the legacy obs-plugins directory. This would only change the hardcoded path used in the first loading phase.

2. The code for the first phase is changed to only load the modules from the compile time constant and will not glob the contents of the obs-core directory. If no module for a corresponding module name is found, it's a critical error and program execution is aborted.

   a. The code for the first phase can now be enhanced to also use fingerprinting/hashing to ensure the modules are not replaced, but this is now entirely decoupled from third party module loading

3. The code for the second phase is changed (in conjunction with step 1) to also implicitly use the legacy obs-plugins directory for discovering and loading plugins. Any plugin found in this directory is marked in the plugin manager and a non-skippable notification is presented to users. This change would go in OBS Studio in one major version, warning about a removal of that location for the next.

4. In the next major version support for the obs-plugins directory is removed, only the "new" default paths are added. This change as well as the one before it only affect the second phase of module loading, as core module loading is already decoupled from it via step 1.

This would ensure that:

• Core module loading is finally decoupled from third party module loading, allowing code changes to that part of module loading without affecting plugins
• The location of core modules becomes decoupled even from legacy plugins
• Legacy plugins are still discovered and loaded for the duration of a major version of OBS to allow users to replace/update their plugins and plugin developers to release corresponding updates
• Legacy plugins are not automatically removed and thus updating them is entirely under control of users and scene collections don't break automatically during the transition period
• With the pending PR for the portable plugin location change this would allow a transition path for both scenarios

I am aware that this does not fix the issue of a new version of a plugin (designed to work from the program data directory on Windows) not being loaded because an older version still exists in the obs-plugins directory, but there is no good automatic fix for that.

Plugin developers will have to do the same thing that OBS Studio has to do when the changes above are implemented:

• The installer/updater will have to check for "old files" and remove/move them accordingly
• If a user chooses to only download a ZIP file of a plugin or OBS Studio itself, it is their responsibility to clean this up
• A knowledge base/wiki entry/release note entry should exist so those users know what they have to do to fix the situation if they encounter it

[matoi974]

Please consider that some people may intentionally replace first-party plugins (say, to troubleshoot older or newer hardware encoder implementations). If there are going to be separate folders for first-party and third-party plugins, if a plugin is present in the folder meant for third-party plugins, third party plugin should be loaded instead of the first-party plugin of the same name.

[PatTheMav]

Please consider that some people may intentionally replace first-party plugins (say, to troubleshoot older or newer hardware encoder implementations). If there are going to be separate folders for first-party and third-party plugins, if a plugin is present in the folder meant for third-party plugins, third party plugin should be loaded instead of the first-party plugin of the same name.

Replacement of core modules has never been and will not be considered a supported use case. That it is possible to do so right now is a loophole of the module loading system and it is unfortunate that it hasn't been fixed earlier.

That's also why at some point the core modules might also be checked against hash values or other fingerprints to give the project and its volunteers the choice of whether they have the capacity to support users that have actively tampered with an installation.

[Fenrirthviti]

While this is a step in the right direction, we are not prepared to make a hard cut like this. There needs to be a transition period as well, as this is effectively the primary method that users install plugins with. This is going to need a lot more discussion and proper design/planning to handle correctly, and Pat's explanation above is what I would like to see. As such, going to close this for now until the groundwork is in place first.