Recently, we came up with the feature to support Creative Builder rendering on the server using NodeJs. Up until then, everything from designing the creatives to rendering for export happened within the browser (Roimatic App). To utilize server resources and to make export more efficient, a server side rendering logic was necessary. Since both the modules accessed the same javascript files to run, a clean way to share these set of files among both the repositories was needed, which could also make the development and deployment of both repositories easy.

Options we considered

npm

• npm is the most popular way to share modules among projects. But we had a couple of reasons to not go with bundling the common files as an npm package.
• Adding an npm package would mean that we had to host the package on our private server (kind of like the DeltaX Nuget server for c#)
• Maintaining code in 3 different components would be difficult since almost any change in feature would result in a change in the submodule and subsequent changes in other repositories which imports it.

bit

• Using bit components would result in a big mono repo.
• Versioning would also be a little complex since each and every component needs to have a version and we need to maintain it.

git-submodules

• The reason we chose git submodules is that it fitted in our scenario really well without having that extra overhead of creating/maintaing packages. The features it offered could make the development process of both the repositories smooth.

As you can see, a common repository ‘CreativeBuilderTools’ was created and added as a submodule to 2 projects - ‘Roimatic App’ and ‘CB Node Server’.

Key features of submodules

• Submodules can be added to any repository and we can access/modify the files as if it were native to that repository.
• Whole commit history of the submodule is available to the parent project.
• The submodule can be updated either by checking out the submodule folder or we can directly update from the parent repository as well.
• In addition to serving the purpose of sharing code, submodules can also be used to break down large/complex projects into simpler ones and then linking them together.
• For instance, when we built our node server app, it required the font files to physically be available in order to load them. Google fonts have their own repository and hosting all font files on bitbucket after downloading did not make sense. Hence, we added the Google fonts repository as a submodule to node server app. This ensures that our bitbucket source is clean and the font files are loaded when we build the app.

Let’s now see how we can actually work with submodules.

Adding a submodule

We can add a submodule by the following command

git submodule add {url} {path_to_submodule}

path_to_submodule is an optional parameter used to rename the submodule or change the folder path to the submodule. If not provided, the submodule will be added at root level and with the same name as the repository.

Once we add a submodule, git will create a ‘.gitmodules’ file with the details of all submodules added. For e.g, on adding the submodule ‘CreativeBuilderTools’ to any repo, we have

We can also configure the .gitmodules file to change which branch of the submodule should the parent track by adding branch field in the submodule section.

Cloning a submodule

On just cloning a project with submodules, all project files along with the reference to the submodule will be cloned. BUT the actual files in the submodule will not be present. In order to pull the submodule files along with the parent one, a flag ‘–recurse-submodules’ has to be added at the end of clone command.

git clone {url} --recurse-submodules

If you have already cloned without adding the flag, you can just update the submodule files manually by

git submodule update --remote --init --recursive

’–remote’ here specifies that the submodule is present remotely.

’–init’ specifies that if any new submodule is added, we need to fetch that too.

’–recursive’ specifies that if there submodules within submodules, then recursively fetch all the nested files.

Code changes in submodules.

Working with submodules is very similar to how you would make changes to any git repository. Modifying a submodule file is totally possible from the parent project too. After making the changes, you can navigate to the submodule folder, then stage > commit > push like you would normally do. The only catch is you cannot commit the changes directly.

When you checkout the submodule folder, as you can see, the parent project points to a detached head. There is no branch associated with that commit. So if you directly try to commit, your changes will not be pushed to any branch on remote. Therefore, you need to make sure that there is a branch checked out before you commit from the parent project.

Pulling latest submodule changes.

There may be changes in the submodule that you may need to pull into your project. The changes may have been made directly in the submodule repo or by some other project who has the same submodule. Either way, there are 3 ways in which you can pull latest changes.

The first approach is to manually checkout the submodule folder. From there you can fetch and pull just like you do on any other repository.

You can also fetch the submodule files from the root of parent repo using the ‘update’ command as specified earlier.

git submodule update --remote --init --recursive

There is also a way if you do not want to write 2 separate command to fetch project as well as the submodule files. You can add a flag ‘–recurse-submodules’ at the end of git pull to pull latest submodule changes along with the latest parent changes.

git pull origin master --recurse-submodules

If you do not want to add the flag everytime you pull, you can set the git config submodule.recurse to true. This will add the flag ‘–recurse-submodules’ to all your future git commands (except git clone).

You can also set aliases to shorten the commands. For e.g, ‘git spull’ can be an alias for ‘git pull –recurse-submodules’

Let’s understand the workflow with an example

• Consider that we are supporting a new export format ‘WebP’ images for CreativeBuilder. To make these changes from Roimatic.App, we need the CBTools submodule checked out.
• Checkout a feature branch > make the required changes > stage > commit and push.
• This will create a feature branch on CBTools bitbucket repository for only the submodule changes.
• We can merge after verifying changes.
• Since we need the changes to support Webp on server as well, just checkout the server repository.
• Run the ‘git submodule update’ command to pull latest WebP changes of submodule.
• As soon as you pull latest submodule, you can see a change within the CB Node Server repo stating that the latest commit has been changes. We need to commit this as well.

• Now, both the parent repos, Roimatic App and CB Node Server, are updated with the latest changes in the submodule, CreativeBuilderTools.

Final Words

Git submodules can be a powerful tool. If you design the repositories properly, you can use submodules in your project efficiently. You need to be sure that submodules is a correct solution for your problem as we did in our case. Generally speaking, submodules can be used with projects that are interlinked/an extension of one another. The submodule can be worked upon as an individual entity as well as an extension of an existing project. The parameters of your situation decide wether submodules are a good option for you or not.

Additional notes for commands related to submodules.