Setting up our monorepo with Turborepo, pnpm, React Native, Expo, and NestJS

Overview

In this post we will go over our project setup and build. We will be building both a mobile app using React Native with Expo and an API service using NestJS.  For the purpose of this project I have chosen to use a monorepo where both the mobile app and service code will live.  Within our monorepo, we will modularize our artifacts from the start.  This makes it easier to reuse shared code amongst our different applications, to split up CI/CD, and if we ever want to split into a multi-repo, we will have clear boundaries already in place.  

Turborepo and pnpm - A winning combination

To enable a fast build with multiple artifacts all living in the same repo, Turborepo and pnpm are some of the top choices.  

pnpm is a replacement for npm with a faster build that also takes up less disk space due the the way it shares package files.  Its workspace feature provides the ability to make one build artifact depend on another artifact in the same repo without the need to publish it as an npm package first.

Turborepo helps speed up and scale monorepo development by both parallelizing and caching the results of different CI steps so that you only build what has been changed.  It does so by building a dependency graph between artifacts and CI tasks which you can tune as needed.  As your project gets larger and larger this is a place where the time savings can really add up.  These caches can even be configured to be stored remotely which is useful in an ephemeral CI environment.


Our shared artifacts

Common convention is to place all shared artifacts go in a top-level directory called packages.  The project we will be referencing has the following shared packages:

packages/eslint-config

Lint config for react-based projects.

packages/eslint-config-backend

Lint config for backend projects.

packages/tour-common

Shared types across all layers.  For example, shared types returned from the tour-service apis.

packages/tsconfig

Shared typescript config.


Our application artifacts

Another common conventions is to place all of our applications go in the apps directory.  The build artifacts in the apps directory will not be made as dependencies for any artifacts in the repo, however they will most likely depend on one or more artifacts from the packages directory.  The autotoor project will have the following two apps:

apps/tour-service

The tour-service NestJS backend service.

apps/mobile

The autotour mobile application


Installing pnpm

Installing pnpm: https://pnpm.io/installation


Since I am using a mac I installed using brew:


brew install pnpm


Installing Turborepo

Since we have pnpm installed, we can use it to install turbo

pnpm install turbo --global


Other ways to install are detailed here: https://turbo.build/repo/docs/installing


A deep-dive into the build

In our top level package.json file there are a few key things to note:
  1. turbo is included as a dev dependency in the devDependencies section.
  2. Nearly all of the scripts point to turbo.  This allows turbo to call the relevant scripts in the other packages in the repo and manage the caching of all of their results.
  3. Some of the scripts use a --filter option.  This is used to limit which dependency tree is considered when building with a given command.  More details on filtering are available here.
  4. There is a pnpm section with some peer dependency rules.  This is used to suppress some warnings about missing peer dependencies. We have these warnings since we need to use hoisted node modules in order for our react-native project to work properly.

package.json

{
  "private": true,
  "name": "@autotoor/monorepo",
  "scripts": {
    "clean": "turbo run clean && rm -rf node_modules",
    "dev": "turbo dev",
    "dev:mobile": "turbo dev --filter=\"{./apps/mobile}...\"",
    "dev:tour": "turbo dev --filter=\"{./apps/tour-service}...\"",
    "lint": "turbo lint",
    "test": "turbo test",
    "build": "turbo build",
    "build:mobile": "turbo build --filter=\"...{./apps/mobile}\"",
    "build:tour": "turbo build --filter=\"...{./apps/tour}\"",
    "build:docker:tour": "docker build -t tour-service . -f ./apps/tour-service/docker/Dockerfile"
  },
  "devDependencies": {
    "turbo": "^1.10.7",
    "typescript": "^4.9.5"
  },
  "pnpm": {
    "peerDependencyRules": {
      "ignoreMissing": [
        "@babel/*",
        "expo-modules-*",
        "typescript"
      ]
    }
  }
}


Defining the pnpm Workspace

In pnpm, an entire project which has a pnpm-workspace.yaml file at its root is referred to as a workspace.  Individual build artifacts that can be shared in that workspace are referred to as packages.  Each package has its own package.json at its root.  The pnpm-workspace.yaml file is used to define the workspace root as well as which directories to look in for packages to include/exclude from the workspace.  All packages included in the workspace are available to be imported into other packages in the workspace.

Below is the pnpm-workspace.yaml from our project.  Some things to note:

  1. By including 'apps/*' the * denotes that only packages that are direct children of the apps directory will be included.  In our repo, this is where we will be putting our applications.
  2. By including 'packages/**' the ** denotes that packages in any subdir of packages will be included. In our repo, this is where we will be putting our artifacts that are to be imported by one-another and by our applications. 

pnpm-workspace.yaml

packages:
  - 'apps/*'
  - 'packages/**'

Defining the build pipeline with Turborepo

We use Turborepo to define our build pipeline.  While pnpm refers to the whole project as a workspace and each of the individual build artifacts as packages, Turborepo calls the whole project a workspace and also calls the individual artifacts workspaces as well. The turbo.json file is where our Turborepo config lives.  This is where we use the pipeline directive to define all of the tasks in our build pipeline: lint, test, build, dev, clean.

Turborepo matches the task names to script names in the individual packages/workspaces in the project workspace.  For example, in our repo, every package has a build script.  Calling turbo build from the root of our workspace will result in the build script in each of our packages being run.  The benefit of using Turborepo to do so is that it will build a dependency graph across all of our packages and then use it to parallelize the build of the different packages whenever possible.  It will also cache the results of each task execution and only run the ones need to change each time the task is called.  For example calling turbo build  if no code has changed since the last time it was called will not do anything.

In order to know which tasks to run and when, turborepo allows the definition of dependencies using the dependsOn directive (in depth explanation can be found here).  If you look at our turbo.json file below you will notice that sometimes our dependsOn contains "build" and other times it contains "^build".  Why is that? 

A dependency task name without the caret symbol "^" preceding it indicates that it is the name of a task in the same workspace.  For example, you will notice in the definition of the test task it depends on "build" without a caret, so that is essentially saying that running the test task in a workspace requires that that workspace's build task has been run first.

When a dependency task is preceded by the caret symbol "^" this indicates that the name of the task it precedes is the name of a task in a workspaces that the current workspace depends on.  For example the build task in the pipeline defined below depends on "^build" which means that the build task for each workspace requires that the build task of any workspace it depends on to be run first.  This ensures that all dependencies are built in order.

Turborepo then uses dependsOn directive to build out a dependency graph which enables it to know which tasks can be parallelized and which need to run serially.  It also uses this to determine what needs to be rebuilt when a workspace has changes vs using a cached value.

turbo.json

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "lint": {
      "outputs": []
    },
    "test": {
      "dependsOn": ["build"],
      "inputs": ["**/*.{ts,tsx,js,jsx}"]
    },
    "build": {
      "dependsOn": ["^build"],
      "outputs": [
        ".next/**",
        "build/**",
        "dist/**",
        "node_modules/.cache/metro/**"
      ]
    },
    "dev": {
      "cache": false,
      "persistent": true
    },
    "clean": {
      "cache": false
    }
  }
}

Module hoisting to support react-native

Since we are using react-native, we need a hoisted node_modules.  This is because when it is not hoisted, symlinks are used and react-native has traditionally had some issues with using those (Note: that there is a somewhat recent react-native update that should enable react native to work with symlinks).  The hoisting is defined via a .npmrc file as shown below.

.npmrc
node-linker=hoisted


Including one package in another

The packages that we make available in our pnpm workspace can then be included into other packages.  The excerpt below is from the package.json file from our library of shared api types, tour-common.  Note that we are including the @autotoor/eslint-config-backend and @autotoor/tsconfig packages.  Since pnpm by default will link packages from the current workspace if they match the declared range, and since we always want the latest version of our workspace dependencies, we can set the range to be either "workspace:*" or even just "*" which means we always want the latest version of a package.  Using "workspace:*" is generally considered safer since it not attempt to download the package from the package registry if it is not found in the workspace and instead return an error.  However, since we have prefixed all of our packages with "@autotoor", it is very unlikely they will be found on the web somewhere else, so using "*" here works just fine.  For more information about pnpm workspace package resolution, take a look at the documentation here.

packages/tour-common/package.json
{
  "name": "@autotoor/tour-common",
  "version": "1.0.0",
  "description": "Common Tour Types",
  "files": [
    "dist"
  ],
  "main": "dist/index.js",
  "typings": "dist/index.d.ts",
  "scripts": {
    "type-check": "tsc",
    "lint": "NODE_OPTIONS=\"--max-old-space-size=5120\" eslint --ext .ts,.tsx src",
    "build": "tsc -b",
    "clean": "rm -rf .turbo && rm -rf node_modules && rm -rf dist",
    "test": "NODE_ENV=test jest --passWithNoTests"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@autotoor/eslint-config-backend": "*",
    "@autotoor/tsconfig": "*",
    "jest": "29.5.0",
    "ts-jest": "29.1.0",
    "typescript": "^5.0.4"
  }
}

In the snippet below from the package.json file from our tour-service app you can see that workspace package dependencies are pulled in the same way

apps/tour-service/package.json
{
  "name": "tour-service",
  ...
  "dependencies": {
    "@autotoor/tour-common": "*",
    "@nestjs/axios": "^2.0.0",
    ...
  },
  "devDependencies": {
    "@autotoor/eslint-config-backend": "*",
    "@autotoor/tsconfig": "*",
    "@nestjs/cli": "^9.0.0",
    ...
  }
}

Turbo build tasks vs package-level scripts

In our turbo.json file referenced above, we defined 5 tasks: lint, test, build, dev, clean.  By default when we run tasks using Turborepo, it will run the package-levels scripts with same name. Note that we have matching scripts (in bold in the file excerpt below) named lint, test, build, dev, and clean in the package.json our tour-service app.  These will get run when the corresponding Turborepo tasks are run.
Note that the tour-service has some additional scripts (not in bold): format, start, start:debug, start:prod, test:watch, test:cov, and test:debug.  These will not be run by Turborepo since they are not configured as task, however they can still be run at the package-level via pnpm.

apps/tour-service/package.json
{
  "name": "tour-service",
  ...
  "scripts": {
    "build": "nest build",
    "clean": "rm -rf .turbo && rm -rf node_modules && rm -rf dist",
    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
    "start": "nest start",
    "dev": "nest start --watch",
    "start:debug": "nest start --debug --watch",
    "start:prod": "node dist/main",
    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
    "test": "jest",
    "test:watch": "jest --watch",
    "test:cov": "jest --coverage",
    "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand"
  },
  ...
}


Is a monorepo right for your project?

For a demo project like this a monorepo is a great choice since all of the code can be located in one place, but does that hold true for a real project?  Like anything in software, the answer is "It depends...".  At the start of a project, a monorepo is probably the easiest choice.  However, when your company and the number of services you have grows, so does your repo, along with the number of people competing for merges into your main branch.  This can lead to a whole new set of problems.  Even if you do decide you need separate repos for different services or teams, it may still be useful to produce multiple artifacts out of each of those separate repos.  For example, the tour-service could publish an npm with some common types or client utils to be imported by other service repos that it also then depends on itself.  In such a case, using the monorepo setup along with pnpm and turbo in each of the separate service repos would still provide a lot of value.


Next

Up next, I will start by digging into the NextJS tour-service application which provides the API consumed by our mobile app.


Get the code

All the code we will be discussing and more is available in the autotoor-app repo available here: https://github.com/autotoor/autotoor-app


Comments

Post a Comment

Popular posts from this blog

Node.js Microservice with NestJS: Part 3 - NestJS Dependency Injection

An Introduction to Dependency Injection in NestJS NestJS is a powerful framework for building efficient, reliable, and scalable server-side applications. One of the core concepts in NestJS is Dependency Injection (DI) , which is built on the Inversion of Control (IoC) design pattern. In this third installment on building a microservice using  NestJS , we'll explore what dependency injection is, how it's implemented in NestJS , and why it's crucial for building maintainable applications. We'll also provide practical examples  of how we use it in our tour-service application to illustrate these concepts and break down important topics like decorators, constructors, modules, application context, and services. For more details about the service we are building and what we have done so far, check out the following posts: Node.js Microservice with NestJS: Part 1 - Service Overview  - An overview of the microservice we will be building Node.js Microservice with NestJS: Part ...
Read more

REST APIs with Controllers - Node.js Microservice with NestJS: Part 4

Building REST APIs with Controllers in NestJS NestJS is a progressive Node.js framework that builds on top of Express and uses TypeScript , making it a fantastic choice for developers seeking a structured and scalable approach to building backend applications. In this fourth installment of the series of posts on building a Node.js microservice with NestJS , we'll explore how NestJS helps developers create declarative REST APIs using controllers, along with powerful features like decorators, Data Transfer Objects (DTOs) , validation, and type transformation. If you're familiar with JAX-RS in Java, you'll notice a similar approach to structuring APIs. For more details about the service we are building and what we have done so far, check out some of the prior posts on the topic: Node.js Microservice with NestJS: Part 1 - Service Overview  - An overview of the microservice we will be building Node.js Microservice with NestJS: Part 2 - Using the NestJS CLI  - Provisioning o...
Read more

Constructor-Based vs Property-Based Dependency Injection in NestJS - Which is Better?

Image
Constructor-Based vs Property-Based Dependency Injection in NestJS - Which is Better? Using constructor-based injection is generally considered better than property-based injection in NestJS (and in many other frameworks) for several key reasons.  Lets look at a few reasons why and some scenarios where  property-based injection  may be more appropriate. Immutability and Readability Constructor-based injection ensures that dependencies are provided at the time of object creation, making them immutable . This means that the dependencies cannot be reassigned after the object is instantiated. With property-based injection , dependencies are injected later, making them mutable. This can lead to side effects if the dependencies are modified during the object's lifetime. Constructor-based injection makes it clear what dependencies the class needs at the time of instantiation, improving code readability. Example: private readonly landmarkService : LandmarkService ; constructo...
Read more