Over the past several weeks I've been working on building an ideal developer environment that I can use to build and launch a variety of my side projects. Because this environment will only be used by me, expect a small number of sessions/users, and since I'm a cheapass, I wanted the environment to solve the following problems:
- Provide both an API server and web server for my web apps
- Be easily deployable
- Support both front-end and backend development with all the latest tools
- Enable future decomposition and modular design
- Ensure I only need to pay for one server/instance/dyno/etc for all my web apps
So let's go through what the architecture and folder structure looks like.
This project contains a couple of key folders. The
app directory contains all of our front-end and UI code. Each folder within it represents a different side project that will be assigned its own subdomain. The
server directory contains all of our server-side code that Nodejs will use to launch our web server. It contains our API server and hosts our apps using express. The
configs directory contains all of our custom configs for each environment (developer, production, etc). Finally, the
build-utils directory contains all of our build tasks and will house any other build instructions. So broken down, the contents within these directories will look something like this:
I'm not going to go into the details of each file, the tools I'm using, or their responsibilities. That'll further discussed in a different post. Instead, I'll be spending time going over the build system, how this environment enables me to work efficiently, and how it is easily deployed out to production.
This project uses a combination of
webpack. I am using gulp for task management and webpack for asset bundling for client-side files.
Because this project contains both a node web server and the front-end code to be used by the browser, I am leveraging both systems to handle each respectively. Every build will bring all the relevant assets into the
dist directory. This includes both the server code, cient code, and any environment configs. More on that later.
I am using gulp instead of npm scripts because I want to create more complex tasks. My Gulp tasks are responsible for transpiling all of my non-client side code, copying files to the respective directory, and any tasks that help improve the developer workflow. Gulp is helping to ensure that I can use the latest tools to streamline my developer workflow - I can write all of my code (on both the frontend and backend) in ES6, my local web server restarts automatically, any files I change are recompiled as needed, and etc.
Webpack is specifically used for asset management for my front-end UI code. Thus, it is responsible for transpiling, bundling, and minifying + uglifying any code within the
app directory and output the results into the
dist directory. Webpack is especially good at asset bundling, which is why we're using this over setting up complex Gulp tasks.
As mentioned earlier, every build will bring all the deployable, relevant code into the
dist directory. This includes the server code, client code, and any environment configs.
When I want to deploy to production, I would only deploy this
dist directory and any other files required by the production server. In this case, because I will be deploying to Heroku, I will also want to deploy the
package.json. Heroku will respect the
Procfile and use it determine how to run the app. So in the
Procfile, you'll see the following code:
web: node dist/index.js
Deploying to Heroku using Codeship
From this point on, you can choose to deploy to Heroku directly from your local codebase, but I want to make this as streamlined and easy to deploy as possible, so I will be leverging Codeship, a CI/CD platform. Why Codeship and not Travis CI, Circle CI, or etc? Cause Codeship's free tier allows for private repo's and has Gitlab integration (cause I use Gitlab's free unlimited private repos...because I'm cheap).
I'm not going to go over how to set up a project on Codeship, you can find out how to set up your job here. I will be setting up the build job like so
These commands will make sure I am running my build job against Node v8, install all the
gulp-cli on the machine, and run the build job. As I mentioned earlier, when the build runs, everything gets compiled into the
dist directory and I want Codeship to deploy the following:
I only need to include these files because once the files are deployed on Heroku, Heroku's Node.js buildpack will run
yarn install before running the
Procfile. In Heroku I have already specified the
NODE_ENV environment variable to production. When that is the case, yarn will only install the production
node_modules required for the project.
IMPORTANT Because yarn will only install the production dependencies when
NODE_ENV=production, make sure that Codeship does not do this. Since we want Codeship to run the build job, we need it to install our
devDependencies as well. To make sure that happens, I set a
NODE_ENV environment variable on Codeship to
To make sure that I only deploy those four key things, Heroku supports something called a
.slugignore. This file acts like a
.gitignore file and enables you to specify which files/directories to exclude from the deployment.
.slugignore will handle all file names and regex's recursively, so if you input something like
index.js, it will exclude all JS files or JS files with the name
index.js respectively. So make sure you are careful with how you specify the files to exclude from the deployment.
Use Heroku's Platform API for deployment
You may notice that Codeship has Heroku deployment integration, as documented here. However, we will not be using this. If you deploy using Codeship's built-in Heroku deployment (or even when you deploy from your own local environment), it performs the deployment through Heroku's GIT deployment instructions. This is great and easy most of the time, but keep in mind that it not only respects the
.slugignore file, but the
.gitignore file as well.
Most likely, you have included the
/dist directory in your
.gitignore to ensure that it does not get pushed to your Git repo, but as a result, it will also not deploy your directory to Heroku! So how do you solve this?
Heroku provides a Platform API that is optimized for non-interactive CI setups. Thankfully, Codeship has open-sourced some of their deployment scripts for custom builds, one of which supports the Heroku Platform API. You can find that script here and their other scripts here! If you follow the instructions on the script, you'll be able to deploy to Heroku through Codeship with the exact files that you want! As you can see, I am using both the deploy script as well as the check url script to make sure the website returns a successful 200 after deploying.
This deployment problem solves (2) of my list above. Once all this setup is done, whenever I commit code to the master branch, Codeship will automatically pull the code, run the build, and auto deploy to Heroku.
Alright now where's the code?
But what about (4)? Well, in its current state, it has already lended itself to prepare for modular design. If I want to host the front-end code for one of my apps in a different GIT repo, deploy them to a CDN, or simply want to separate my front-end and server code so that I don't have to restart my Heroku instance every time I make a UI change, the code/folder structure will allow me to easily pull these out as needed with little hiccups.
However, this environment is still a work in progress for me and I'm still in the process of optimizing certain aspects of the architecture. I had to do a lot of research to figure out how to get Codeship + Gulp + Webpack + Heroku to play well together. Some similar scenarios posted on StackOverflow or any existing documentation and blog posts did not provide an ideal solution that I desired. It was all kind of hacky. As a result, I wrote this to help some people who are trying to solve similar problems (e.g. deploying a
public folder to Heroku when it's part of a
.gitignore, how to make sure your deployment doesn't install unneeded dependencies, how to make one Express app support multiple subdomains, etc.)
Expect the sample code to be released in the near future, stay tuned...