Software Engineer, Ember hacker, Ruby lover, NixOS user.
7,235 words
https://ghedam.at @ghedamat

An introduction to nix-shell

actions

Short version

On June 26th I gave a short talk at our Toronto Elixir Meetup, this post is an extended version of that talk.

If you are in a rush, you can skim through the slides here:


Long version

👋 A word of warning, there is a fair amount of hand-waving ahead. Suggestions and corrections are welcome and encouraged! Feel free to reach out to me on twitter or via email!

Definition: nix-shell

nix-shell — start an interactive shell based on a Nix expression

The command nix-shell will build the dependencies of the specified derivation, but not the derivation itself.
It will then start an interactive shell in which all environment variables defined by the derivation path have been set to their corresponding values, and the script $stdenv/setup has been sourced. This is useful for reproducing the environment of a derivation for development.

Extract from the Nix manual

Let me paraphrase

Running nix-shell will start an interactive bash shell, in the current working directory. The packages required (we'll see shorty how to specify them) will be downloaded but not installed globally. Instead the shell will have its ENV set appropriately so that all the packages in the shell definition are available.

You can test this by typing

env
...
# lots of stuff

nix-shell -p ripgrep
env
...
# lots of stuff but with a bunch of new things!

Some simple nix-shell uses

Install a package without installing it globally

nix-shell --packages
# or
nix-shell -p

Starts a nix-shell that has the package available in its $PATH

$ which rg
rg not found

$ nix-shell -p ripgrep
[nix-shell:~]$ which rg
/nix/store/rw24lqk4ls1b90k1jj0j1ld05kgqb8ac-ripgrep-11.0.2/bin/rg

Run a command in a nix-shell

Building on the above, you can temporarily add a package and immediately use it

$ nix-shell -p ripgrep --run "rg foo"

Creating your first shell.nix

The nix-shell command receives an optional argument for a .nix file. By default if invoked with no arguments nix-shell will first look for a file named shell.nix and then for one named default.nix.

This .nix file has to contain the definition of a derivation, the standard library offers a special derivation function mkShell specifically for this purpose (although the more general stdenv.mkDerivation can still be used).

Derivations are the building blocks of a Nix system, from a file system view point. The Nix language is used to describe such derivations. (cit. Nix Pills)

Here's a basic shell, it provides only the buildInputs attribute, that is, the list of packages to make available in your shell.

# simple.nix
with (import <nixpkgs> {});
mkShell {
  buildInputs = [
    ripgrep
  ];
}
$ nix-shell simple.nix
[nix-shell:~]$ rg foo
# ... 

You can also provide the shellHook attribute to customize the bash shell being spawned.

# hooks.nix
with (import <nixpkgs> {});
mkShell {
  shellHook = ''
    alias ll="ls -l"
    export FOO=bar
  '';
}
$ nix-shell
[nix-shell:~]$ echo $FOO
bar

Using nix-shell for development

Where nix-shell really shines for me is in its ability to provide uniform and shareable configuration for development environments in virtually any language.

In this section I'll provide a few examples of shell.nix configuration files for different programming languages. You can imagine these shell derivations as a drop in replacement for what usually is done with language-specific version managers like Rvm, nvm and asdf but, as we'll see, this approach is beyond just managing language versions.

A Python example

I am not a Python dev, but from my experimentation the support for Python feels quite "native" in Nix, one can create a custom Python build for the shell, and add the desired dependencies. A lot of Python version and packages are already available in the main nixpkgs package tree.

The following example is lifted from the NixOS Wiki

# python.nix
with (import <nixpkgs> {});
let
  my-python-packages = python-packages: with python-packages; [
    pandas
    requests
    # other python packages you want
  ];
  python-with-my-packages = python3.withPackages my-python-packages;
in
mkShell {
  buildInputs = [
    python-with-my-packages
  ];
}

Here's also a recent blog post about using Python on Nix.

A Rust example

For Rust mozilla has been providing a shell.nix to get you started

# rust.nix
with import <nixpkgs> {};
let src = fetchFromGitHub {
      owner = "mozilla";
      repo = "nixpkgs-mozilla";
      rev = "9f35c4b09fd44a77227e79ff0c1b4b6a69dff533";
      sha256 = "18h0nvh55b5an4gmlgfbvwbyqj91bklf1zymis6lbdh75571qaz0";
   };
in
with import "${src.out}/rust-overlay.nix" pkgs pkgs;
stdenv.mkDerivation {
  name = "rust-env";
  buildInputs = [
    # Note: to use use stable, just replace `nightly` with `stable`
    latest.rustChannels.nightly.rust

    # Add some extra dependencies from `pkgs`
    pkgconfig openssl
  ];

  # Set Environment Variables
  RUST_BACKTRACE = 1;
}

The interesting thing here is that the Mozilla Nix overlay is fetched as part of the shell derivation, this shows how shells are not limited to a single source for packages.

The recommended approach for NodeJS, Ruby, Elixir

My current understanding is that in the Nix way a package and its dependencies are "reproducible", the final derivation we build is always gonna be the same because the inputs will always be the same.
Languages like ruby, js and others don't "naturally" provide this guarantee but the Nix ecosystem has produced a few ways to work around this problem

For Ruby this pattern is implemented using bundix:
bundix runs against your Gemfile and generates a Nix expression that includes all the Ruby dependencies used in your project

With that you can define a nix-shell that will have all the dependencies available and can effectively avoid using bundler (the Ruby package manager) in your workflow.

In practice:

Given a Ruby project with a Gemfile you can:

  • run bundix -l
  • source the generated gemset.nix in your shell.nix
# bundix.nix
with (import <nixpkgs> {});
let
  gems = bundlerEnv {
    name = "your-package";
    inherit ruby;
    gemdir = ./.;
  };
in mkShell {
  buildInputs = [gems ruby];
}

Similar solutions exist for other languages, for example Node has yarn2nix.

A more generic approach for interpreted languages

While the previous approach has some really good advantages I personally found that for personal projects, and for my team at work, a less Nix-y solution has been working better.

The strategy that I have been using is to override the environment variables that the package manager provides and force the installation of packages to happen locally to the directory in which the shell is being used.

Here's an example for a NodeJS shell.

# node.nix
with (import <nixpkgs> {});
mkShell {
  buildInputs = [
    nodejs-12_x
    yarn
  ];
  shellHook = ''
      mkdir -p .nix-node
      export NODE_PATH=$PWD/.nix-node
      export NPM_CONFIG_PREFIX=$PWD/.nix-node
      export PATH=$NODE_PATH/bin:$PATH
  '';
}

And here's a slightly bigger one that I've used for Ruby on Rails development

# ruby.nix
with (import <nixpkgs> {});
mkShell {
  buildInputs = [
    nodejs-12_x
    ruby
    yarn
    gnumake
    gcc
    readline
    openssl
    zlib
    libiconv
    postgresql_11
    pkgconfig
    libxml2
    libxslt
  ];
  shellHook = ''
    mkdir -p .nix-gems

    export GEM_HOME=$PWD/.nix-gems
    export GEM_PATH=$GEM_HOME
    export PATH=$GEM_HOME/bin:$PATH
    export PATH=$PWD/bin:$PATH

    gem list -i ^bundler$ -v 1.17.3 || gem install bundler --version=1.17.3 --no-document
    bundle config build.nokogiri --use-system-libraries
    bundle config --local path vendor/cache
  '';
}

I've also talked about how to do this in Elixir in a separate blog post.

At a high level, this tecnique is very similar regardless of the programming language:

  1. Identify the ENV variable that determine the installation paths for packages and executables
  2. Override them to be local to $PWD
  3. Extend $PATH to include the installation path for binaries (so that things like npm install -g work)

Tips for sharing shell.nix

Use a specific "Nix channel"

A "trick" that I have found useful is being able to import from a different channel within a Nix derivation, this is often useful if in your shell.nix you want to install packages from a more recent version. In the following example I'm using the unstable channel while my host system <nixpkgs> are version 20.03.

with (import (fetchTarball https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz) {});
mkShell {
  buildInputs = [
    git-up
  ];
}

Pinning to a specific <nixpkgs> SHA.

When sharing a shell.nix it can be helpful to "pin" the <nixpkgs> version. This guarantees that regardless of the nix-channel used on the system everyone gets exactly the same Nix packages.

This is done by specifying a commit SHA directly from Github.

with (import (fetchTarball https://github.com/nixos/nixpkgs/archive/8531aee99f4907bd255545eb94468e52a79a44f1.tar.gz) {});
mkShell {
  buildInputs = [
    git-up
  ];
}

This guarantees that so long as you specify all the dependencies, and don't accidentally rely on something coming from the OS, every user will get the same setup.

This tutorial also offers a good explanation.

Extending a shared shell.nix

As soon as we started using a shared shell.nix at work it became clear that there was a need to customize the some aspects of the shell on a per-user basis.

The solution I resorted to is check if a local.nix is present and if so expect that file to provide an attributeSet with two attributes: inputs and hooks.
These attributes are merged with the ones provided by the shell.nix that is checked in into your git repository.

# shell.nix
with (import <nixpkgs> {});
let
  basePackages = [ ripgrep ];
  localPath = ./local.nix;
  inputs = basePackages
    ++ lib.optional (builtins.pathExists localPath) (import localPath {}).inputs;

  baseHooks = ''
    alias ll="ls -l"
  '';

  shellHooks = baseHooks
    + lib.optionalString (builtins.pathExists localPath) (import localPath {}).hooks;

in mkShell {
  buildInputs = inputs;
  shellHook = shellHooks;
}
# local.nix
{ pkgs ? import <nixpkgs> {} }:
{
  inputs = [ pkgs.curl ];
  hooks = ''
    alias ghedamat="mattia"
  '';
}

Cross platform nix-shell

Nix works both on MacOS and Linux but there are some dependencies that are platform specific.
The following example shows how these can be accounted for in your configurations

# cross.nix
with (import <nixpkgs> {});
let
  basePackages = [
    ripgrep
  ];

  inputs = basePackages
    ++ lib.optional stdenv.isLinux inotify-tools
    ++ lib.optionals stdenv.isDarwin (with darwin.apple_sdk.frameworks; [
        CoreFoundation
        CoreServices
      ]);

in mkShell {
  buildInputs = inputs;
}

A complete development setup: nix-shell and docker

nix-shell works great to configure dependencies but does not really solve for services. Often your development environment will require one or more databases (I often need PostgreSQL and Redis running). Such services can be installed at the system level but project-based isolation is in my opinion preferred.

While I found that nix-shell is a much better development environment than docker I do think that running services is what the latter excels at.

The solution then becomes: use both! docker-compose for services and nix-shell to run code!

I wrote a previous post on how at Precision Nutrition we implemented this hybrid approach. If you are interested I encourage you to read it and let me know what you think!

Customizing the shell

Extending nix-shell to allow for more customization (i.e. using zsh or not have to type nix-shell every time) is beyond the scope of this post but I will leave a few pointers here for the interested reader.

  • nix-shell --run zsh is a simple workaround that allows you to change the $SHELL from bash
  • direnv can be used to take this a step further and "load" the nix-shell ENV without spawning a new shell
  • lorri is another project that aims at replacing nix-shell by extending it.

Recap

  • nix-shell allows you to define development environments for pretty much any language in a consistent way, it makes also easy to support different versions of the same language!
  • Adding shell.nix to your project can be used to ensure that everyone on the team has the same configuration and is also a great way to help new contributors get setup quickly.
  • In my experience, combining docker and nix-shell for projects that require databases or other services, is the way to go!

Thanks for reading!

So, tell me about Nix

Loom

Nix has been around the block for a while but recently, both from outside and from within the Nix community, I've seen several efforts to make Nix more beginner friendly.

I have been using NixOS for a year or so, and I had myself to go through what was a somewhat confusing and effortful onboarding process. I was lucky enough to have friends like @shazow to help me on the way but I still feel I have many gaps in my understanding.

I'm writing this post is to collect my thoughts on Nix and present a few of the things that I found helpful during my Nix journey.

My hope is that they can also help others learn and use what I consider being the next step when it comes to developer ergonomics both on MacOS and Linux.

In the resources section below you will find more detailed and precise explanations so feel free to skip there!

Where to start?

The first confusing thing about Nix is that the name is used to refer to a few things:

  • Nix the package manager
  • Nix the programming language
  • NixOS the operating system

I'll briefly cover what I like and how I use each one of these, leaving some resources at the end of this post for readers that are interested in a deeper exploration.

What I love about Nix, the package manager

Nix can be used like brew on MacOS or apt, yum, emerge and many others on Linux. It can be installed locally for a single user or globally on any Unix based system (including MacOS).

One you have installed Nix you can do stuff like

$ nix-env -iA curl

and in Nix will download and install the package you selected and all its dependecies.
It will be there for you to use until you decide to delete it.

There is a lot to be said about how Nix works and all the features that it brings to package management, a notable one being reproducible builds that guarantee that a given version of a package will always be the same, including all its dependencies. Some of the links at the end of this post will allow you to explore this topic.

One thing that is unique to Nix is the ability to download and run the package without the need to install it globally, you can instead spawn a new bash shell with the packages you are interested in

$ nix-shell -p curl
[nix-shell:~]$ curl -I http://nixos.org
# ...

once you leave this shell curl will not be available anymore.

nix-shells can do this an much more, they also allow you to define isolated development environments, and share them with other developers ensuring that they can quickly spawn a bash shell using exactly the same dependecies you have.

nix-shell is a complex topic though (and I have much left to learn myself), I will probably cover it more in a future post, for now you can see a small example here.

What I love about NixOS, the Operating System

Quoting directly from the NixOS website:

In NixOS, the entire operating system — the kernel, applications, system packages, configuration files, and so on — is built by the Nix package manager from a description in a purely functional build language. The fact that it’s purely functional essentially means that building a new configuration cannot overwrite previous configurations. Most of the other features follow from this.

Unlike most Operating Systems NixOS allows users to describe their desired system configuration and then leave it to Nix to apply said configuration an make all the required changes.

There are no configuration files to edit, copy, backup outside of the .nix ones in /etc/nixos.

Here's an example of the full configuration (minus the hard-drive part) required to provision a PostgreSQL server.

{ config, pkgs, ... }:

{
  # Include the results of the hardware scan.
  imports = [
    ./hardware-config.nix
  ];

  environment.systemPackages = with pkgs; [
    curl
    wget
  ];

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;

  networking.hostName = "postgres-server"; # Define your hostname.
  networking.firewall.enable = false;
  networking.interfaces.enp6s18.useDHCP = true;

  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.users.ghedamat = {
    isNormalUser = true;
    extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
    shell = pkgs.zsh;
  };

  programs.zsh.enable = true;
  programs.zsh.enableCompletion = true;
  programs.zsh.promptInit = ""; # Clear this to avoid a conflict with oh-my-zsh

  # postgres config
  services.postgresql = {
    enable = true;
    package = pkgs.postgresql_11;
    enableTCPIP = true;
  };

  system.stateVersion = "20.03"; 
}

There's a lot going on here and much more to say but it is hard to describe, as a long time Linux user and system administrator, how much joy the ability to declare the state of my system configuration brings me.

Oh and there's one more thing!

Thanks to the magic of the Nix package manager you can also rollback safely to any previous configuration you ever ran on your machine.

What I love about the Nix ecosystem

The Nix community created other tools the follow the same philosophy:

  • home-manager allows you manage your user configuration. You can use it to configure git, zsh, install packages only for your user and much more.
  • NixOps allows you to provision and manage remote machines. Think like terraform plus chef but done the nix way
  • nix-darwin is NixOS but on MacOS!

home-manager has completely changed how I manage my dotfiles, now I have a single place to configure all my user-level applications across different machines. As an example, this is my ThinkPad config.

With NixOPS I am managing my laptop, my desktop and all my development VMs. I'll try to cover this setup in a future blog post, but you can get a sense for it here.

What about the Nix language?

The Nix language is what powers this all. Truth to be told, I can't say that I have a solid understanding of it even after using Nix for quite a while.

The good news is that I feel that's OK. Even with my limited proficiency I have been able to manage several systems: laptops, desktops and VMS. All my dotfiles are now managed in Nix too and I also moved over our entire development environments at work to use nix-shell.

My advice is to not get intimidated and learn it more as you need it.

Getting started

If you looking to give this a go, whether you are on MacOS or Linux my advice is:

1) install Nix for your user
2) use nix-shell to try a few packages
3) start using home-manager to manage your installed packages, and maybe try to move a few of your dotfiles over
4) start using nix-shell to define per-project development environments

Some resources

Here's some of the resources I found the most helpful getting started with Nix:

Finally a thing that can be intimidating at first but works super well is reading the source. Once you get the basics it's surprisingly approachable.

And last but not least have a look at other people nixfiles. It was a great way for me to get started. If you are curious these are mine and shazow's nixfiles.

Conclusion

Hope this will help you or maybe even convince you to join the Nix community! Feel free to tweet at me for any feedback or questions.

We also have a very small Toronto meetup and discord channel. Reach out if you wanna join us!

Bonus - NixTO online meetup

On Thursday June 25th we are running our first online Toronto Nix meetup.

Regardless of where you are, come join us!

100 Days of meditation

3 headed monkey

This is a personal post about my relationship with meditation and my life journey so far with respect to this practice. I'm not sure if it will be helpful to anyone else, but I wanted to push myself to reflect a bit on this subject so here we are.

The journey so far

I discovered meditation ~10 years ago, before moving to Canada I was struggling with anxiety and emotional balance and my therapist at the time suggested that I should give it a try.

My journey started attending some groups organized by a counselor and through him I ended up taking by-weekly classes for two years with Andrea Cappellari (link in Italian). Andrea taught both Shamata (concentration) and Vipassana (insight) meditation. What really stuck with me of his approach was that, even if all his techings where heavily rooted in Tibetan buddishm, he was always keeping a tecnical attitude and regularly stressing that no religious beliefs were required to rip the benefits of the practice.

One thing that Andrea always pushed his students to do was to pratice at home, and that was a thing that I invariably failed to do regularly. I would practice every few days, especially when feeling down or anxious but quickly stop and go back to my usual routine and favourite addictions (mostly cigarettes and drinks).

In those years I also attended a few meditation retreats and, while I always loved the experience, coming back to "normal" life afterwards always lead to the same outcome. I would come home and stop meditating entirely for months! Some way of compensating for the excessive effort maybe..

Moving to Canada didn't help, a new life to build, a job to find, a VISA to obtain, finding a house, having my partner move from Italy and eventually, over time, rebuilding a life from scratch took a lot of energy and also worked as a very good excuse to not look closely at the idea of resuming with meditation.

For the first few weeks, when I was here alone and completely lost, I found good help in attending a meditation group of the shambala tradition, but quickly stopped going once I got busy with a full time job and had enough friends to go out with and explore my new city.

Once things start rolling they can get pretty fast, before I knew it we owned a house, a dog, two jobs, friends and effectively had built a new life in Canada (I do love and will be always greatful to this country and what it gave me), but meditation was entirely gone. Just something that I would think about every once in a while.

During these years I tried a few times to use Headspace to get back into the habit. While I think it's a great app it never worked for me because I was still approaching meditation as a way to feel better. Hence I would meditate only when feeling particularly bad or anxious.

As the year passed my lifestyle changed substantially, I gave up smoking (although I still miss it every day) and later (~3 years ago) also had to give up driking for health reasons. Aside from the occasional use of pot (legal in Canada), mostly CBD for pain management, I found myself leading a life free of most stimulants. 35 year old me looks very fondly at 20 year old me, but I don't think I could survive more than 2 days of what I used to do to my body back then!

Something changed

Late in 2019 I found myself battling again with anxiety, even though my conditions are not particularly serious I started observing how they affected my mood. Chronic pain is a bitch, aside from the annoyance of the suffering (and I consider my pain fairly low most days), there are two things that I found really hit me.
First I became more sensible to days in which the pain comes back, especially after feeling well for a while, when the pain returns I have feelings of anger, sadness and my mood is way less stable.
Second, and more pertinent to the topic, my anxiety started coming back, I find myself worrying that the pain will come back, or that other things will start causing me pain. I'm not sure about others but my mind can always find a way to get anxious about something and now I had the perfect excuse.

I said I realized this was happening in 2019 but, if I'm honest about it, the anxiety never left, getting sick just gave it a head start and now I was just catching up with what happened.

What to do now?

I'm an engineer, that approach to life is embedded with my sense of self, but it's also very much how I tackle problems.
This means that when faced with an issue my usual approach is to study. Around this time I ran in to The Craving Mind by Jud Brewer and immediately started reading.

Even if I had already encountered a lot of the concepts that Jud Brewer talks about (i.e. The power of habit), even if I was already familiar with Jon Kabat-Zinn and even if this is what Andrea always referred to in his lessons, the connection between my anxiety patterns and how meditation could help untangle them had never been as clear.

Unwinding Anxiety is an headspace-like app that was developed to put into pratice the ideas from the book and I decided to give it a shot.

The parallel between the method the app uses and what we do with our clients at work (maybe I'll talk about my experience with PN coaching too at some point) are stirking and that probably gave me faith in the methodology, given that it worked so well for my nutrition and training habits in the past.

The program guides you to bring mindfulness to you anxiety, to look at your patterns and learn to observe yourself as things happen. Interestingly enough, there's very little "traditional mediatation".

Did I finally start meditating?

Going through the first few weeks of the course really worked, things kept happening but I felt less of a victim of my patterns. After a while though something changed and my interest shifted. I was feeling better but I wanted to understand more of the why.
Not why in a scientific sense but why from an experiential standpoint.
What is there, what is this mind that is doing this to me and how does it work?

And this is when I started going back to Andrea's teachings (well Buddhism techings to be precise). The following is a great and imprecise oversemplifications but it conveys my current understanding.
The mind is a muscle, and like other muscles it can be trained. Through concentration practice we can stabilize the mind, once the mind is stable the mind can be used to observe, traditionally observasion can be of sensations, feelings, the mind itself and all phenomena. The mind can then see itself and see what is real.

I wanted some answers and I had a way to go get them. Not to feel better, not to run away, but to face them.

So I started sitting, once again with the help of a book: Genuine Happines by Allan Wallace. I know very little about him but he was Andrea's teacher and I bought the first copy of this book many years ago in Italy, keeping it on a shelf, without ever really trying to put it into practice until now.

The book guides you through a series of different buddhist meditations, and this happens over the course of several months. I will not claim that this is on par with having a teacher and a group with you but so far it's working out allright.

So that's what I'm doing, I started 108 days ago and I'm slowing making my way through, 25 minutes a day, most days I feel I can barely keep my mind on the object of meditation, other days it seems to work out.

I still get anxious. I still get angry. I still mess up regularly as I'm sure my partner (bless her), my friends and my collagues can testify.

But it's been a ton of fun!

Here we are

I don't know if my experience is unique or common but this is what has been true for me so far. It's really hard to do something if my goal is feeling better because as soon as I do I lose the motivation.

I believe that the combination of having a different reason and removing all factors that were hindering my practice (mostly smoking and drinking) were the successful combination that got me here.

For the record. I still feel like I "suck" at the pratice itself. The mind is still a wild monkey, jumping all over the place, observing it without getting involved only happens for a few instants, the rest of the time... I become the monkey. The difference with my past attempts is that this time I keep showing up.

I'm very curious to see what happens next. Will I be able to keep it up or will I pick up old habits again? How is travel (when that is a thing we do again) gonna affect it? How are bigger life changes gonna change it? I don't know. And I'm looking forward to finding out.

I'm not in the business of giving advice but I'm gonna leave a thing for my future self.

Even if you fail. Be kind. If you fuck up it doesn't mean that you're a fuck up.

A nix-shell for developing Elixir

Developing with Elixir requires a fair amount of configuration

you need

  • Erlang
  • Elixir
  • Hex package manager

And usually you want all these to be locked at a specific version.

There are several solutions out there, the most popular probably being https://github.com/asdf-vm/asdf
but for Nix fans, how do we setup a nix-shell and do this the "nix way"?

The Nix community seems to reccomend to nixify your projects, examples exist for

I've been using a different pattern with most of my projects to achieve this, and it works for both Ruby, Elixir and many other languages.

The trick is find the env variables that allow us to "isolate" locally the dependency installation, in our example these are MIX_HOME and HEX_HOME.

By setting these two variables to the local directory within the nix-shell we allow mix to install the packages locally instead of trying to add them to the nix-store path (/nix...) that is readonly.

Summing it up

Here's the shell.nix file that I use in my elixir projects

with import <nixpkgs> {};
let
  # define packages to install with special handling for OSX
  basePackages = [
    gnumake
    gcc
    readline
    openssl
    zlib
    libxml2
    curl
    libiconv
    elixir_1_9
    glibcLocales
    nodejs-12_x
    yarn
    postgresql
  ];

  inputs = basePackages
    ++ lib.optional stdenv.isLinux inotify-tools
    ++ lib.optionals stdenv.isDarwin (with darwin.apple_sdk.frameworks; [
        CoreFoundation
        CoreServices
      ]);

  # define shell startup command
  hooks = ''
    # this allows mix to work on the local directory
    mkdir -p .nix-mix
    mkdir -p .nix-hex
    export MIX_HOME=$PWD/.nix-mix
    export HEX_HOME=$PWD/.nix-hex
    export PATH=$MIX_HOME/bin:$PATH
    export PATH=$HEX_HOME/bin:$PATH
    export LANG=en_US.UTF-8
    export ERL_AFLAGS="-kernel shell_history enabled"
  '';

in mkShell {
  buildInputs = inputs;
  shellHook = hooks;
}

save this in your project directory as shell.nix then type nix-shell and you will be in a bash shell that has mix and elixir ready to be used!

Bonus: Add gigalixir support

https://www.gigalixir.com is one of the simplest ways to deploy Elixir applications.
Similarly to Heroku, you subscribe, create hjan account, install the cli and you are off to the races.

The problem is that our nix-shell does not allow us to run pip install gigalixir and install the gigalixir command line utility.

Setting up python in nix requires a little more configuration.

First we need to add Python to our nix-shell and ensure that our python install has the pip package manager, a detailed explaination can be found here https://nixos.wiki/wiki/Python.

In short you can do something like this

  my-python-packages = python-packages: with python-packages; [
    pip
    setuptools
  ];

  python-with-my-packages = pkgs.python3.withPackages my-python-packages;

Then, like we did for Elixir, we have to force Python to
install its stuff locally, we do this by setting PIP_PREFIX and PYTHONPATH;

 alias pip="PIP_PREFIX='$(pwd)/_build/pip_packages' \pip"
 export PYTHONPATH="$(pwd)/_build/pip_packages/lib/python3.7/site-packages:$PYTHONPATH"

The resulting shell.nix file is then as follows

with import <nixpkgs> {};
let
  my-python-packages = python-packages: with python-packages; [
    pip
    setuptools
  ];

  python-with-my-packages = pkgs.python3.withPackages my-python-packages;

  # define packages to install with special handling for OSX
  basePackages = [
    gnumake
    gcc
    readline
    openssl
    zlib
    libxml2
    curl
    libiconv
    elixir_1_9
    glibcLocales
    nodejs-12_x
    yarn
    postgresql
    inotify-tools
    python-with-my-packages
  ];


  inputs = basePackages
    ++ lib.optional stdenv.isLinux inotify-tools
    ++ lib.optionals stdenv.isDarwin (with darwin.apple_sdk.frameworks; [
        CoreFoundation
        CoreServices
      ]);

  # define shell startup command
  hooks = ''
    export PS1='\n\[\033[1;32m\][nix-shell:\w]($(git rev-parse --abbrev-ref HEAD))\$\[\033[0m\] '

    # this allows python to work locally
    alias pip="PIP_PREFIX='$(pwd)/_build/pip_packages' \pip"
    export PYTHONPATH="$(pwd)/_build/pip_packages/lib/python3.7/site-packages:$PYTHONPATH"
    unset SOURCE_DATE_EPOCH

    # this allows mix to work on the local directory
    mkdir -p .nix-mix
    mkdir -p .nix-hex
    export MIX_HOME=$PWD/.nix-mix
    export HEX_HOME=$PWD/.nix-hex
    export PATH=$MIX_HOME/bin:$PATH
    export PATH=$HEX_HOME/bin:$PATH
    export LANG=en_US.UTF-8
    export PATH=$PATH:$(pwd)/_build/pip_packages/bin
    export ERL_AFLAGS="-kernel shell_history enabled"
  '';

in mkShell {
  buildInputs = inputs;
  shellHook = hooks;
}

At this point we can type nix-shell and then pip install gigalixir and we are ready to go!

Notes:

The full code can be found here in this gist

After publishing this post I found this one that presents a simpler approach and with a shorter shell configuration, I updated my notes above to use mkShell and a nicer syntax that uses lib.optionals to load the MacOS only packages

Another good resource I only found today is here

Speedy Development environments with Nix and Docker

this is a cross post from PrecisionNutrition's tech blog https://tech.precisionnutrition.com

The problem

Our stack at PN is composed by one Rails monolith and a plethora of EmberJS frontend apps.

We also use PostgreSQL, Redis and NGINX.

This means that in order to run our development stack each engineer has to at a minimum run one Rails server, two databases and a web server, usually they will also be running at least one frontend application and possibly a REPL to interact with the Rails server and maybe some tests.

The commonly used solution

In the "good old days" developers used to run everything locally on their work laptop, this was done using a bunch of tools:

  • rbenv - managing Ruby version
  • nvm - managing NodeJS versions
  • homebrew - managing system packages (like PostgreSQL, C libraries, Redis)

But running everything locally has several problems:

  • It required each dev to manage their own environment and dependencies
  • It is very hard to have different versions of certain dependencies (i.e database) installed at the same time (one may work on multiple projects or want to experiment with a newer version of some dependency)
  • A system update can easily break the dev environment, causing lost productivity

Docker saves the day (or not?)

In recent years the dev community moved to containerized solutions.

'Containerization' usually means docker, although alternatives do exist.

handwaving ahead - The idea behind Docker is to use the "host" system kernel but package applications and dependencies into a single "blob". This blob can be pushed to the cloud and downloaded for later use. This guarantees that when the application is the deployed no dependencies have to be installed on the remote system because they are all contained within the "docker image" that will be run.

Although initially intended to make deployments easier, the dev community rallied around Docker and extended its use case beyond deployments to include local development.

Docker Compose is typically used to orchestrate the various containers needed for local development. A docker-compose.yml configuration is defined, which includes all services as well as the main application.

To run a Ruby On Rails app similar to ours you would need a docker-compose file that configures a PostgreSQL server, a Redis server, an NGINX server and a Linux image with all the dependencies for the Rails app. Here's one of the many blogposts on how to do this.

The only real difference between this setup and the image that gets deployed to production is that the development image is usually configured to "mount" the code directory from the host system, allowing developers to edit their code locally and have it reload within the docker image.

Issues with Docker

The main issue we found with using Docker locally is that docker filesharing is extremely slow, especially on MacOS. The interwebs have plenty of resources to address this problem but these approaches simply mitigate rather than resolve the underlying performance issue.

Docker performance is pretty bad for Rails development but it's even worse for front-end apps that require a gazillion files to be loaded and written (cough cough webpack).
Poor Docker performance usually leads developers to give up on Docker for their frontend - and return to a painfully slow backend development process.

Nix-shells - A better way?

Surely there must be a way to use docker for what it is good at (running services like databases) and have a way to manage dev dependencies without having to manually install them like in the "good old days".

Enter Nix. Nix is "The Purely Functional Package Manager", you can imagine it as an alternative to Homebrew or apt, yum, etc.

Nix works both on MacOS and Linux and allows userspace installation of packages.

But for our use case the best part of nix is nix-shell.

A nix-shell is a bash console that is loaded starting from the host terminal but is initialized with a pre-defined set of packages which are downloaded the first time you run the shell. The packages are then instantly available for later use. Think about it as a bundle install or a npm install but for your OS dependencies.
nix-shells work in isolation, this means that the dependencies available inside the shell cannot leak out to your host system. Nix achieves this by using a symlink structure and by manipulating your bash PATH.
If you are curious on how this works try to issue echo $PATH when you start the example shell below.

For example

A shell.nix file with the following contents

let
  basePackages = [
    ruby
  ];

  hooks = ''
    mkdir -p .nix-gems
    export GEM_HOME=$PWD/.nix-gems
    export GEM_PATH=$GEM_HOME
    export PATH=$GEM_HOME/bin:$PATH
    export PATH=$PWD/bin:$PATH
  '';

in
  pkgs.stdenv.mkDerivation {
    name = "your-shell-name";
    buildInputs = basePackages;
    shellHook = hooks;
    hardeningDisable = [ "all" ];
  }

can be invoked by simply running nix-shell in the current directory, you will be moved to a new bash shell that has Ruby installed for you!

An important thing to note is that a nix-shell is just another bash shell, there is no virtualization happening, the only difference is that the nix-shell has access to more dependencies that come from the shell configuration.

The consequence of this is that the shell is not like a docker container and will not run services for you, services are still system level processes.

A small note about packaged dependencies

Sometimes your project will require to install packages that are not available on Nix.
An example of this can be ruby gems that you install with gem install or node packages installed with npm install -g.

The Nix ecosystem offers a few solutions for this problem but the shell.nix file we included above shows a simple trick that we found works well.

By setting some exports (i.e GEM_PATH) we manipulate the install paths for RubyGems so that all gems installations are local to the shell. Normally RubyGems would try to install these globally and because Ruby was installed by Nix the commands would fail.

Our solution: use Nix and Docker together

The solution we went with at PN is to take the best of docker and nix-shell and use each one where it shines.

This means using docker to run our databases and NGINX and using nix-shell to manage the dependencies and run ruby and node.

Our main Rails application then ships with

  • a docker-compose.yml that configures PostgreSQL, Redis, NGINX
  • a shell.nix that gives the user a nix-shell with the right version of Ruby, NodeJS, OpenSSL etc

The development workflow then becomes

  • start docker-compose
  • run nix-shell and from there start bundle exec rails s or bundle exec rails c or any other process you might need to run

This also works great for our EmberJS applications and allows us to avoid using nvm while retaining native performance.

Taking this further

After doing this for a few months and enjoying the greatly improved development speed we decided to take this further and build some more automation around this.

pndev was born - pndev is a command-line tool that automates our dev workflows and will likely be the subject of a future blog post.

Resources

Here are some useful resources to get you started with development in a nix-shell

Some caveats

  • Apple does their best to mess up 3rd party installs so installing Nix on MacOS is a bit more complicated than one would like
  • Nix is somewhat difficult programming language to learn but writing nix-shells is fairly easy