Split repo into multiple projects?

[jothepro]

Inspired by @a4z's thoughts I'd like to propose splitting the single repo up into multiple sub-projects to separate the different components into self-contained codebases. This helps clean up the repo structure.

When analysing the current repo I could isolate the following self-contained components:

• djinni parser/generator commandline tool
• support lib & integration-tests (depends on djinni parser/generator)
• android example (depends on djinni parser/generator)
• ios example (depends on djinni parser/generator)

The main benefits would be:

• ✅ The generator/parser repo would be a lot cleaner & just contain the sbt project + documentation on how to use the cli
• ✅ The support-lib repo just contains cpp/obj gluecode sources & could easily be published to dependency-managers like conan, hunter.
• ✅ The iOS/Android examples would be closer to reality as they have to include the djinni-cli binary as external utility like any other project has to. They could easily serve as templates for a new project (just fork and modify to your needs).
• ✅ More examples (e.g. CMake-based iOS/Android projects) could easily be added in separate repos without having to open PRs in the 'main' repo everytime.

Downsides & things that need further discussion:

• ❌ The generator/parser & the support-lib have separate lifecycles, that may (theoretically) lead to incompatibilities ("upsidaisy I am using the generator in version X but it has features that are not supported in support-lib version Y"). In reality I don't expect changes that are big enough to often cause this kind of trouble.
• ❌ The documentation is split up: The generator usage-instructions are in one repo, the support-lib instructions in the other repo. -- A github-page that provides a general-purpose documentation for all djinni stuff may be a solution for this.
• ❌ ❌ ❌ The parser/generator repo looses tests for QA: Currently there is only integration-testing going on, no unit-tests for the generator itself. This means that all integration tests depend on the support-lib and therefore will have to be executed in the support-lib repo. To have any kind of QA we would have to write entirely new unit- or integration-tests for the parser/generator.

I am looking forward to your thoughts on this wall of text ☺️

[a4z]

Thanks a lot for this well through thought summary @jothepro ! excellent!!

Ad downsides, I think we can manage them, and maybe even turn them into strength

• different life cycle can, in long term view, be solved with versioning, see protobuf for example, so it should be possible to not end up with undefined behavior but rather nice error messages if generated code and support lib do not match. We will have to think about that, but I am sure this is doable with a reasonable effort.

• documentation can be solved, there are tools like https://antora.org/ that are designed for situation like this, or , create a 'virtual master repo' that checks out some sub repos, makes the doc and pushs it to a gh page.

• Parser generator tests, this is text in text out. so the main test might be comparing generated text. Changes have of course to trigger examples to be executed. The fact that those are in a different repo means just that we need to think about how to connect them, and how to ensure that all examples (since they can be seen as system tests) are executed before a release.

so all this is solvable with effort, question is, is it worth the effort , and I think yes.
Worst case might be that, until everything is automated, we need to do some things manual, like run examples, but at the end I think it will not be super complicated to have some (thing like) if this than that ....

As mentioned, I was pretty long unsure if it would be worth the work, to split the repos, but seeing your listing, I would vote for yes. Since it makes absolute sense.
And it will provide a lot of useful side effects, experimenting with different generator implementations, adding new language bindings, working with just examples ...
If we manage to have everything well structured, plus helpers on how to get the parts you need on a machine, it will make much more fun to work with the single components

[jothepro]

FYI I'm starting to experiment with splitting up the repo.
The first step is to extract the parster/generator into one repo: https://github.com/jothepro/djinni-generator
As soon as I have established some basic integration testing (IDL-file in, generated file out comparision) I am going to propose moving the repo to this organisation.

[a4z]

awesome!

[jothepro]

Update:

Here is my first take on adding integration-tests for the generator: https://github.com/jothepro/djinni-generator/blob/master/src/it/scala/djinni/GeneratorIntegrationTest.scala

It tests if a given djinni-IDL-file produces the expected output. For that I'm calling the cli from within scalaTest and then read the created files from the filesystem & compare them to the expectations, which are stored as files in the repo

I'm not yet finished with the tests and the overall repo structure, but I'd love to hear your feedback about this approach! :)

[a4z]

This looks already pretty good !
Think it is time to start something similar with the support lib

[jothepro]

I want to create the following new repositories in cross-language-cpp:

• djinni-documentation - contains documentation about dijnni & publishes it as github page. Contains logic to pull the docs from all other repos & combines & renders it to a searchable documentation about all djinni stuff.
• djinni-generator - djinni generator code, usage documentation of the CLI, documentation of the djinni-IDL
• djinni-support-lib - C++ support library, docs on how to integrate it into your project
• djinni-intellij-plugin - Code of the IntelliJ plugin.
• djinni-vscode-plugin - the vscode plugin (from here. I'd like to ask the owner if he would be ok with moving the repo to our organization or if he's fine with us maintaining a mirror here.
• android-example - (refreshed) example on how to use djinni & C++ in an Android project
• ios-example - (refreshed) example on how to use djinni & C++ in an iOS project

Other cool stuff may be added in the future, e.g. I am dreaming of djinni-cmake, a neat little support-lib that wraps calling djinni in cmake (see #10)

If you are ok with my proposed structure I will create all repos, fill them with a placeholder Readme that informs about our migration process and open a migration-issue in each project. (Plus I will open my first MR for djinni-generator 💪)

Once migration to the new structure is finished, I'd like to move the documentation from djinni-documentation to djinni, overwriting the contents of the current fork. The djinni-repo is then supposed to be the entrypoint that bundles & publishes the documentation & links to the other repos & resources.

I'm sure my proposed repo-list is not yet complete. There still is some things in the repo that I don't know where to put... I guess we'll have to figure that out on the go. :)

All I need for now is your go for my plans, @a4z, @bojanin. If you'd like to contribute in the migration, please tell me on what repository you'd like to help. Let's rock! 🎸

[a4z]

Cool @jothepro , please feel free to make the suggested changes, and thanks a lot for taking initiative and doing the required working steps!