For a while now I’ve been wondering about Nix Flakes, what they are and how they are going to change how we use Nix.
This post is a summary of my understanding so far based on research (see resources below) and some light experimentation.
Disclamer: I am quite new to flakes and by no means a Nix expert, so if something in this article is inaccurate or wrong please reach out and I’ll fix it!
Warning: The word flakes appears over 30 times in this blog post.
Nix Flakes are an upcoming feature of the Nix package manager.
Flakes allow to define inputs (you can think of them as dependencies) and outputs of packages in a declarative way.
You will notice similarities to what you find in package definitions for other languages (Rust crates, Ruby gems, Node packages) - like many language package managers flakes also introduce dependency pinning using a lockfile
.
The main benefits of flakes are:
Reproducibility: Even if the Nix language is purely functional it is currently not possible to ensure that the same derivation will yield the same results. This is because a derivation can have undeclared external dependencies such as local configuration, $NIX_PATH
, command-line arguments and more. Flakes tackle this issue by making every dependency explicit as well as pinning (or locking) their versions.
Standardization: Every Flake implements a schema
and defines clear inputs
and outputs
(see below for more details). This allows for better composability as well as avoiding the need for niv.
Discoverability: Because all outputs are declared, it is possible to know what is exposed by a flake (i.e which packages).
On top of the above there are other practical benefits for day to day Nix use.
Nix Flakes add another layer of caching that was not possible before: the evaluation of Nix expressions.
A very practical result of this change is that nix-shell
gets a whole lot faster. The first run will take the usual time but any subsequent one will be practically instant!
nix
commandAnother change that been paired with the addition of flakes is a whole new set of features for the nix
command.
I will not cover them all (see NixOS wiki) but here’s a few that I have been using:
nix build
and nix run
nix build
replaces nix-build
and extends it in interesting ways, it can be used to build local as well as remote flakes.
$ nix build .# <- defaultApp in local flake
$ nix build github:Mic92/nix-ld#nix-ld <- specifies output in remote flake
$ nix build github:Mic92/nix-ld <- defaultPackage in remote flake
nix run nipkgs#XXX
replaces nix-shell -p XXX --run XXX
$ nix run nixpkgs#jq <- runs the defaultApp from the `jq package`
You can use this to run also remote packages (same as with nix build
).
$ nix run 'github:nixos/nixpkgs/nixpkgs-unstable#jq'
# ^^ this could be any repository, not only nixpkgs
nix shell
or nix develop
?There are now two separate commands to replace nix-shell
although I expect to be mostly using nix develop
The main difference is that nix shell nixpkgs#hello
will spawn a bash shell with the hello
executable in the $PATH
but no development dependencies while nix develop nixpkgs#hello
will add those too.
I will provide a more detailed example of how to build a nix-shell
using flakes later in this post.
nix flake ...
Nix Flakes have a dedicated whole set of subcommands, the two that I expect to use the most are
nix flake show
which lists all the outputs provided by a flake.
nix flake lock
which allows to manipulate the lockfile, in particular this is used to update the pinned versions of dependencies. i.e. nix-flake lock --update-input nixpkgs
.
nix flake new
can be used to create a new flake, it also supports templates!
For example:
$ nix flake new . -t templates#rust
Most of the information I found claims that flakes should be pretty stable at this point. That said, note that this is still flagged as an experimental feature.
Enable the required features in your configuration.nix
.
{ pkgs, ... }: {
nix = {
package = pkgs.nixUnstable; # or versioned attributes like nix_2_4
extraOptions = ''
experimental-features = nix-command flakes
'';
};
}
Enable the required features for your user.
$ nix-env -f '<nixpkgs>' -iA nixUnstable # ensure you are running unstable
$ mkdir -p ~/.config/nix
$ echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf
The NixOS Wiki has a dedicated page with more details.
Nix Flakes are declared as an “attribute set” that respects a predefined schema
.
There are 3 top level attributes and various sub-attributes:
{
description = "I am an optional description";
# inputs contains all the dependencies of the flake
inputs = {
nixpkgs.url = "github:NixOs/nixpkgs" # this is set by default
flake-utils.url = "github:numtide/flake-utils" # this is another flake
};
# outputs is a function that has the inputs are passed as parameters
outputs = { self, nixpkgs, flake-utils }:
packages = ... # packages exposed by this flake, used by nix build
defaultPackage = ... # package called when nix build has no arguments
apps = ... # apps exposed by this flake, used by nix run
defaultApp = ... # app called when nix run has no arguments
devShell = ... # this is the definition of the nix-shell
... # there's much more, see the NixOS wiki for details
}
}
devShell
As said above, nix develop
replaces nix-shell
.
Here’s an example of a basic development shell with ripgrep
:
{
description = "A basic devShell";
outputs = { self, nixpkgs }:
let pkgs = nixpkgs.legacyPackages.x86_64-linux;
in {
devShell.x86_64-linux = pkgs.mkShell {
buildInputs = with pkgs; [ ripgrep ];
shellHook = ''
echo "shell with ripgrep"
'';
};
};
}
Run it and you’ll see the lockfile being created.
$ nix develop
warning: creating lock file '/home/ghedamat/DEV/OSS/ghedamat/flakes-playground/basic-shell/flake.lock'
shell with ripgrep
Run it again and you’ll notice a faster startup as well as no changes to the lockfile.
$ nix develop
shell with ripgrep
You might have also noticed that I had to specify the “architecture”, this is because one of the goals of flakes is to return the same output regardless of the environment they are evaluated in.
This is achieved by making the output an attribute set with values for each architecture. For more details see this serokell.io blog post.
Running nix flake show
shows:
$ nix flake show
path:/home/ghedamat/DEV/OSS/ghedamat/flakes-playground/basic-shell?narHash=sha256-waOoDEnxQM7fdvfFtDWYMd+jQRkwA1BrohBE37rUjYs=
└───devShell
└───x86_64-linux: development environment 'nix-shell'
To avoid having to repeat our declaration for each architecture we intend to support we can use eachDefaultSystem
from flake-utils .
{
description = "A basic devShell using flake-utils each";
inputs.flake-utils.url = "github:numtide/flake-utils";
outputs = { self, nixpkgs, flake-utils }:
flake-utils.lib.eachDefaultSystem (system:
let pkgs = nixpkgs.legacyPackages.${system}; in
rec {
devShell = pkgs.mkShell {
buildInputs = with pkgs; [ ripgrep ];
shellHook = ''
echo "shell with ripgrep"
'';
};
}
);
}
Using nix flake show
we can see more systems:
$ nix flake show
path:/home/ghedamat/DEV/OSS/ghedamat/flakes-playground/flake-utils-each-shell?narHash=sha256-KHDsbeusyxmw+BVMcIMnYJvKFwUXjf7Jiv+S7TaJo+Q=
└───devShell
├───aarch64-darwin: development environment 'nix-shell'
├───aarch64-linux: development environment 'nix-shell'
├───i686-linux: development environment 'nix-shell'
├───x86_64-darwin: development environment 'nix-shell'
└───x86_64-linux: development environment 'nix-shell'
shell.nix
It is possible to import existing shell.nix
files using flakes so that you can use nix develop
. This allows for better code organization as well as having some users rely on flakes while others can keep using nix-shell
in the same project.
Let’s start from the shell.nix
for NodeJS found in my nix-shell post.
with (import <nixpkgs> {});
mkShell {
buildInputs = [
nodejs-12_x
yarn
];
shellHook = ''
mkdir -p .nix-node
export NODE_PATH=$PWD/.nix-node
export NPM_CONFIG_PREFIX=$PWD/.nix-node
export PATH=$NODE_PATH/bin:$PATH
'';
}
We first need to update this file to take pkgs
as an optional parameter. This is required because using flakes environment dependent globals like <nixpkgs>
(that is based on $NIX_PATH
) are disabled.
{ pkgs ? import <nixpkgs> {} }:
with pkgs;
mkShell {
buildInputs = [
nodejs-12_x
yarn
];
shellHook = ''
mkdir -p .nix-node
export NODE_PATH=$PWD/.nix-node
export NPM_CONFIG_PREFIX=$PWD/.nix-node
export PATH=$NODE_PATH/bin:$PATH
'';
}
With this change we can write our flake.nix
{
description = "A devShell that imports shell.nix";
inputs.flake-utils.url = "github:numtide/flake-utils";
outputs = { self, nixpkgs, flake-utils }:
flake-utils.lib.eachDefaultSystem (system:
let pkgs = nixpkgs.legacyPackages.${system}; in
rec {
# we pass `pkgs` directly to shell.nix
devShell = import ./shell.nix { inherit pkgs; };
}
);
}
I have not tested it yet but it is possible to have shell.nix
either rely on niv
or on the nixpkgs
passed by the flake.
I am looking forward to using flakes more in my personal and work projects.
In particular I want to explore if and how I can use flakes with my nixops setup as well as starting to migrate the shell.nix
files that we use at work to support flakes.
If you want to dive deeper here are the references I used to write this post:
Usual shoutout to @shazow for reviewing this article as well as pointing out that, regarding reproducibility, there should be ways to depends on the environment safely. He also suggested this blog post series on “content addressable Nix” that (if my understanding is correct) will be the next step after flakes.
]]>Continuing on the trend of things that I can talk your ear off about… Productivity tools and systems!
Over the years I’ve collected several techniques and practices that I use regularly to help me stay focused and meet my goals.
In this post I’m gonna go over them and provide links and pointers to go deeper if you are interested.
For many years, the feelings of scarcity (specifically not having enough time) and of being controlled by my to-do list have lead me to research systems and tools to become more effective.
I like to think in terms of effectiveness rather than productivity. Effectiveness is leverage, it’s “choosing the right something” versus being just very efficient at doing things that are not need in the first place.
Effectiveness is a means to an end rather than an end itself. It’s a tool. By becoming more effective I’ve been able to enjoy more and do more of the things that are meaningful to me.
It’s an ongoing process, and it has already lead to more enjoyment, focus, and tranquility.
My days are organized using Bullet Journaling (in short BuJo). In essence this is a way to organize information, todos, notes, and appointments using a single notebook.
Everything that you need to do is written down, notes that you take during the day are written down, tasks in the future are written down. While simple, this is incredibly freeing because I never have to worry about remembering anything. I just open the journal and it’s all there.
When I started doing this the main benefit that I found was that, because it’s on paper, BuJo forces a constant reflection on the tasks that I carry over into the next day (I manually have to copy them).
In the last few years Bullet Journaling has grown a lot in popularity, now there’s a book (worth a quick read), a community, apps etc. The core information is still available for free.
After years of doing it on paper I now do it digitally, it requires more discipline (copy-paste is almost free) but it fits better in my PKM (see below). If you are just getting started I would suggests to do paper for at least a few months.
I’m admittedly a bit OCD about my calendar, my time is almost entirely blocked off and most of it is not for meetings.
Blocking off time on my calendar allows me to ensure I will have time to do things I care about the most. During my Monthly and Weekly reviews (see below) I block off time in line with my goals. This also forces me to accept the things that I am saying NO to.
I mostly don’t decide in advance the specific activities I will do in a block of time but I know the “category” they belong to. I block off work time, slots for meetings, slots for family, meditation, workouts, seeing friends, going for walks, relax etc.
I have a shared calendar with my partner and personal appointments also block time in my work calendar automatically.
This article describes well how Marc Andreessen does this.
Either in the evening before going to bed or in the morning before starting my day I follow a quick daily planning ritual.
This allows me to reflect on what I just accomplished and set realistic expectations for the day to come.
Every evening I track my mood for the day and I write a few words on what happened. I use this information in my Monthly Review (see below) to understand how I am doing, find patterns and get ideas on what changes I can make.
Initially I tried to track with a lot of granularity the people I met and the things I did but I found the overhead to be too much so I scaled back.
The next 3 sections are all dedicated to practices that I adopted from Sebastian Marshall and the folks at Ultraworking.
Every Sunday afternoon/evening I spend 30-60 minutes looking at the week that just ended and at the one that is about to start.
I ask myself these 3 questions while I look at my calendar, my email inbox, my tasks.
Based on the answers I organize my calendar for the week and as ideas pop I prepare notes for the meetings I’m going to have.
Weekly reviews are the practice with the highest return of this list. Every Sunday I get so much clarity and calm by answering these 3 questions.
Episode #27 of the Ultraworking podcast goes into the details of this process, Sebastian also wrote about the anatomy of a perfect week.
Every Month I spend 2h or so looking at my past month, reflecting on what happened (using the mood data mentioned above among other things), looking ahead of what I want and need to accomplish in the following month and finally coming up with a list of things that I am saying YES to and a (usually longer list) of things that I am saying NO to.
I then schedule time for all these items on my calendar to ensure that I will get them done.
In order to get inspiration and be able to select items effectively I’ve been using the monthly review template from Ultraworking.
This process helps me clarify what I can realistically commit to in the month ahead and decide what’s important. Like the Weekly Review it provides me with a lot of clarity and tranquility.
Most months I don’t achieve everything but over time I’m getting closer and closer to realistic goals and accomplishing what I want. Learning to not over-commit and schedule personal time in advance has been the big benefit of this practice.
Also, looking at the previous month and reflecting on how that went allows me to notice trends, if I have more than a few bad days in a month it’s a sign that something needs to be looked at.
Back in university I started using the pomodoro technique for my studies. I found it helpful but something was missing.
Work Cycles takes the pomodoro idea a step further.
I won’t go deep into the details but they are a very effective way to plan work session, define work blocks, maintain focus and recognize when things are going off track.
The Ultraworking marketing page does not lie, I can easily get twice as much done when I do cycles. And most importantly it’s much easier for me to get into flow and stay there.
Ultraworking also has an online community called The Work Gym, it’s a permanent Zoom room filled with other productivity nerds. There we do Work Cycles at the same cadence. On top of the social accountability and support it also forces me to take breaks between cycles, regardless of how in the zone I am.
For a long time I iterated on how to organize my notes. Turns out that in the last few years this has become a quite popular topic and there’s a lot of ideas, courses, articles available.
What has worked best for me is building my own version of Tiago Forte’s Second Brain.
There’s a lot in the whole methodology (Tiago has a very popular course and a book on the way). In terms of impact, my advice is start from P.A.R.A.
In short, take digital notes, organize your them based on “actionability” (which project is the note gonna be useful for) and don’t worry about tagging, categorizing etc. Notes can always be moved later and you always have full text search.
Get clear on:
Organize files, notes, todo’s across all your tools using the exact same structure. This way, when you switch tool, you know where to find everything.
The reason why PARA has been useful is that it helps me be clear on what projects I’m currently working on and when I switch between them I can easily get back to where I left.
An interesting realization, after trying so many apps for so many years, is that most of this requires very little tech.
As with many other things, there’s no substitute for practice. No need to search the perfect system, text files and spreadsheets might be all I need!
Over the last 20 years I’ve iterated through a lot of note taking apps, todo apps etc. At last, I am very happy to be back to plain-text markdown files.
The app is really well made and hackable, the community is friendly and it’s priced very fairly. It uses local text files. so even if it Obsidian were to disappear your notes will be left untouched.
I use Obsidian for Bullet Journaling, Weekly Reviews and for all text notes in my PARA system.
I’ve already talked about calendar management above and there’s not much to add, I have a work and a personal calendar, they are linked so that I can have a single view between the two.
My google drive(s) - work and personal - follow the PARA structure.
Since reading Digital Productivity Coach I’ve been practicing inbox zero. It takes me < 10 minutes a day (admittedly I don’t get a ton of useful email).
Finally, Google Sheet is how all the Ultraworking systems are delivered, it works great!
Exists is a nice little app that I use to track my mood, do light journaling and monitor a few other stats.
I always liked the idea of quantified self but these days my tracking is very minimal.
Instapaper is a simple and clear read later app that I use to read all my articles. It’s connected to my Obsidian setup so that highlights are then saved as notes that I can search in my PKM.
I want to thank all the creators that built and shared their ideas and systems. In particular the folks at Ultraworking, Forte Labs and Obsidian have really made a difference in my day-to-day. Although over the years I’ve paid for courses and services from all the authors linked in this article most of the content and resources are available free of charge.
I believe this shows how it’s possible to build successful business while being generous and allow everyone to benefit.
Here’s also some of my favorite links on these topics:
]]>How do you know someone is a meditator? Don’t worry they’ll tell you. OK, that was not the original joke but you get the idea.
If you know me you probably know that I really enjoy meditation. Because of this people end up asking me advice on how to get started and how to build a practice.
Usually the questions are some variation of:
I’m far from an expert. I consider myself a lay practitioner, I have indeed been meditating on and off for the last ~12 years and after a long struggle I have been maintaining a daily practice for the last 2 or so.
What I will share below is my experience, I realise that some of my suggestions are not in line with mainstream advice. Everyone’s journey is different. As usual I hope this will be helpful.
When I turned to meditation my goal was to reduce suffering, I was in emotional pain and I wanted it to stop. Later, to establish a regular practice I found understanding my deeper motivation to be the key (I talked a bit about this already in my previous post 100 days of meditation).
This is also at the center of one of the resources I’ll talk more about below called “The Mind Illuminated” (in short TMI) where reviewing your motivation is suggested as the first step of every session.
One of my favourite reflections on importance of motivation is this article from my colleague Craig Weller.
Your motivation doesn’t have to be deep (i.e I want to awaken), simply being clear about what brings you to the practice will allow you to go through the inevitable moments of resistance and discomfort.
I know this Path is hard. I know this Path is hard. But there’s something harder than this Path. What’s harder than this Path is not having a Path!
- Lisa Wahpepah
If you are quite new to meditation it can be daunting to start. There are so many schools, techniques, apps, courses, opinions, philosophies to pick from.
My journey started with something called “concentration practice”. The goal is to focus the mind on an object and let distractions fall in the background of your awareness. I’ve mostly focused on the breath but sounds and sights can work as well.
The steps are quite simple (to describe at least):
It’s not required, but I’ve found guided meditations very helpful at the beginning. The guidance helps you recognise that you are distracted and brings your attention back to the meditation object.
Every major app has introductory guided meditations to get you started and most of them use the breath as the base.
A couple of years ago I started studying the teachings of Shinzen Young (more below) and learned his approach based on “noting practice”. I won’t be able to explain it in detail here but the claim (and apparently also the result of Shinzen’s research) is that this particular flavor of noting practice can be a great starting point for new meditators.
I got to this practice later so I can’t vouch for that but I can say that I find Shinzen’s flavor of mindfulness freeing and easy to bring into my day to day activities compared to sitting with the breath.
The tiny version of noting is:
Here’s a video explanation as well as a guided meditation from Shinzen.
Another practice that can be a great starting point is Loving Kindness meditation (also known as “Metta”).
This type of meditation has been recognised as beneficial both by spiritual traditions and modern neuroscience. It develops compassion, reduces stress (among other things) and it might be easier for newcomers. Focusing on the pleasant can give faster access to enjoyment and create a positive feedback loop that will help reinforce your practice.
There are many variations of Metta, what I usually do is repeating the following in my mind:
I target specific people as I do this. Starting from myself, my loved ones, acquaintances, people I’m having difficulties with and finally all life.
Tasshin Fogleman as a lot of resources on this practice, start from this post.
The Mind Illuminated also has a chapter dedicated to this practice.
The Mind Illuminated puts daily practice as the first stage of the meditator’s path. And I would agree, the shift between “I meditate every once in a while” to “I meditate daily” has been quite marked for me.
I want to stress that if you are just starting, It’s OK to not have a daily practice (it took me 10 years to build one!), and it’s OK and important to explore different techniques and traditions. You don’t have to commit and go all in immediately. In fact, if you are new to this, I would not even recommend it. Start slow and explore. Focus on finding what you enjoy.
They say that your calendar shows your priorities. Decide that meditating is going to be a priority for you, set time aside on your calendar every day. Be ready to make the changes you have to make to accommodate this in your life.
Doing it first thing in the morning has worked very well for me, if you do it in the morning then it’s done, and you don’t need to think about it again. Plus, it will likely improve your day!
You don’t need a community to practice and in fact I know a lot of people that do meditations mostly alone. For me though, finding a community has made a big difference.
I started meditating back in Italy 2-3 times a week with a teacher that was giving community classes. That allowed me to deepen my practice, ask questions and learn from other’s experiences.
Thanks to the Covid-19 pandemic (weird words I realize) a lot of groups went online and it’s now possible to find meditation groups as well as meditation retreats.
Since 2020 I’ve been meditating most days of the week with an online Zen Buddhist group called Hollow Bones Zen in their Virtual Zendo and I credit this for a lot of my success with establishing a daily practice.
Common advice is that 5-10 minutes a day is enough to start. I did not find this to be the case. I hit a threshold when I started to meditate consistently at least 25-30 minutes a day.
Something shifted and I the impact of the practice started to manifest outside the cushion. I and the people around me noticed the difference.
Most importantly I started to deeply enjoy the practice.
To do stillness meditation you do not need to sit on the floor on a cushion. A chair or even standing or laying down are perfectly valid positions.
The goal is to find a pose the allows your body to enter a state of deep rest while maintaining alertness.
If you can though, I would encourage you to try using a cushion as that’s been the most successful position for me.
Sitting even only for 30 minutes can be painful if you don’t have good posture so it’s worth researching a bit how to sit. Stephanie Nash has a wonderful pdf with detailed instructions that I recommend.
I have 4 groups of teachings that I refer to most often.
The Mind Illuminated is a very comprehensive manual. It will guide you through all the steps, from how to sit to very advanced concentration and awareness stages.
You can sit (quite literally) with this book for years and it might be all you need.
If you like structure and are committed to establishing a daily practice I would definitely give this a go. You don’t have to read the full thing either (I myself have yet to go beyond stage 4 of 9), the first few chapters will give you a pretty solid foundation and an idea of the suggested path.
Shinzen’s Unified Mindfullness system takes a comprehensive approach and organizes a whole spectrum of meditation traditions and techniques.
Starting from “See Hear Feel” that I mentioned above it also covers many other practices.
If this sounds appealing to you I would suggest to start from:
Shinzen also runs monthly online retreats that I highly recommend, there are 5 sessions spread across one weekend. You can attend any number of them for a very modest price.
You can find them here Home Practice Program.
I mentioned Alan Wallace in my previous post 100 days of meditation. His book Genuine Happiness offers a good overview of different techniques in the tibetan tradition as well as a path that can be cultivated over several months.
In particular the explanations around the concentration practice (shamata) is worth reading if you are getting started as well as the chapters on Metta and Bodhicitta.
Zen does not give a lot of meditation instructions and there is almost no guidance provided during sitting periods. That said I got a lot of value out of Zen teachings. In particular An Introduction to Zen Training has some great pointers on how to sit and focus on the breath. I personally found the lack of guidance to be somewhat liberating. It helped me avoid getting stuck in thinking technique is better than another one or that I have to find the perfect one.
The founder of the Zen group I sit with defined Zen practice beautifully:
Sit down, concentrate, realize the true depth of your mind, celebrate
Compared to 12 years ago there are now a LOT of meditation apps, and that’s great. I got value out of them at different points in my life and they helped people close to me.
The main risk that I see with apps is the “gym membership effect”, if you want to start meditating you don’t need to buy an app, you need to meditate!
With that caveat out of the way here’s a few I would suggest:
There are also more experimental things that I’ve tried over the years that I wanna mention. I am a nerd after all and I like trying new things:
Relax, Look for the joy, Observe, Let it come - Let it be - Let it go
- Culadasa
It’s hard to grasp at first but even in concentration practice effort does not help. Ultimately the goal is to enjoy yourself.
This does not mean to be happy or in bliss the whole time, most sessions in fact you will spend thinking about stuff. And you can still enjoy it!
Some of the best advice I got was to celebrate every time I realize I’m distracted.
As I said above, it took me 10+ years to get here, and I know I’m just getting started, and I OK with that, I also hope this won’t discourage you from starting. My practice was helpful and beneficial from the first day, it’s just more so now!
Please reach out and let me know if this was useful, I enjoy talking about all this and I would love to hear your story!
Deep gratitude to all the teachers, mentors and friends that I met on the way. This continues to be one of the most rewarding parts of my life. Thanks also to @rcyeske for his feedback on this post.
]]>A successful software project has many parts and many people contributing to it. What I want to focus on here is the role that a software engineer can play when they are leading the tech function.
At PN we call this role “solutions engineer”, it is sometimes called “project lead” or “lead engineer”.
My goal is to clarify my own thinking on this topic as well as giving insipiration to others that are holding a similar role or are interested in taking it.
As the engineers in charge of the project a lot of things are our responsibility. Because we are familiar with the requirements as well as the implementation often we have the most accurate picture and that is a big opportunity for contributing beyond writing code.
We are partners and allies with product managers and designers in ensuring the success of the project.
Fulfilling the expectations of this role means enabling the team to deliver “good work” defined as:
I believe that shipping is the most important thing. Shipping means delivering the feature in a way that satisfies the requirements of the customer as defined by product.
Shipping requires to balance code quality and maintainability with meeting deadlines. In my experience shipping early gives the greater benefit because it reduced the time to new information.
This is probably one of the most difficult parts of our work. During every phase of the project product and design will almost invariably ask for more scope, other engineers (and often ourselves) will push for higher quality and/or a solve that is more future-proof.
The temptation to add scope is always strong: if we paint ourselves in a corner mistakes will be painful. I find that reminding yourself of the end goal - delivering value to customers - helps me to to keep perspective.
The biggest lever that we have to ensure that projects ship on time is cutting scope, it’s not the only option but to me it’s the most effective.
Cutting scope allows us to hone in on what’s the core value proposition for the customer and focusing on delivering that first and putting it in front of them as soon as possible.
The goal is to ship fast, quickly testing what’s being coded and ideally getting the feature in front of real users even before the planned deadline.
Our role then is to help inform product about where it’s safe to cut scope and flag items that could lead to delays and change in timelines.
I believe in feature flags vs feature branches. Even if they can make the code a bit messy and require cleanup, in my experience they are a fantastic tool. They allow us to have new code be as close as possible to production and avoid the pain of long running branches.
As the engineers in charge, we should be as familiar to the feature as much as the product owner if not more. Engineers are not “just doing” what is requested in the ticket, they are integral part of building the product and they have a unique perspective because they can see it from ideation to delivery.
I found that timeboxing projects and keeping the scope small works best. A project should never take more than 6-8weeks to be in front of users.
Once again, the goal is to DELIVER VALUE to customers as quickly as possible so the team can test its hypothesis.
The second benefit is that timeboxing projects to a small size reduces the amount of time spent doing the “wrong thing” - if the project goes “off track” it will do so only for a few weeks.
When you start doing this it might feel like this is an impossible ask, surely some problems can’t be solved in 6 weeks. And you are right, they can not, and that’s OK. What is always possible in my experience is breaking down a problem into smaller problems that CAN be solved in 6 weeks. Sometimes this will mean releasing something only internally, other times it will mean shipping alpha to a small subset of trusted users.
As the lead engineer we can work with the product managers to break down tasks and identify what parts of the solution fit into the timebox.
A big part of the job is making technical decisions, here are some of the things I usually think about:
We want to understand where the risk is and work with product to help shape the requirements in a way that minimizes it while maximizing value for our customers.
Projects can be broken down in 3 parts:
The solutions engineer can help in different ways during each one of these.
Here are some questions that I think through when working with product/design on project scoping and planning:
This is a list of things I think about and do when I have to plan and deliver a project. I encourage you to build your own list based on your needs.
I find helpful to plan the release in advance and consider a few things:
Cleaning up is important, part of this process produces some craft that requires to be dealt with.
It’s helpful to ensure that tickets to remove feature flags are ready and scheduled for after delivery.
It’s also important to leave the project in a state that will allow us or someone else to continue work on it later on. Usually the end of the project is when we are the most clear about what we could have done differently and what needs fixing. It’s also when there is no time to do so. Leave your future self (or someone else) notes/tickets on what can be improved.
I hope this post helps you in your engineering roles, whatever they are. Everyone is different and everywhere is different and you will need to find your way to solve the problems that are unique to your situation. These things have worked for me and the teams I’ve been a part of in the last few years.
If you read this and want to discuss these ideas feel free to reach out!
People smarter and more experienced than me have written a lot on this topic, here’s some of my favorites pics:
Many thanks to @typeoneerror, @lukegalea and @benjamintmoss for their feedback on this article.
]]>For the last few weeks I’ve been playing a bit with Nx
- the new library for Numeric computation in Elixir.
This post is about how I got it running on Elixir 1.12 using niv
.
Nx is still offered as a developer build and because of that it ends up having somewhat stringent requirements, in particular the latest version released a few days ago requires the most recent Erlang and Elixir.
The nix community does a great job at keeping the nixpkgs-unstable
channel up to date but Elixir 12 was not available yet.
Nudged by my friend Ben I decided to see what it would take to rely on community supported packages.
To start, I did some googling. I looked for a community repository that offered up to date derivations for Elixir packages.
nix-beam is exactly what I needed. It exposes several derivations for BEAM (Erlang virtual machine).
To list them I ran the following command, as instructed in the README of nix-beam
.
$ nix-env -qaP -f https://github.com/jechol/nix-beam/archive/master.tar.gz
pkg.v23_3.elixir.v1_10_4 elixir-1.10.4_erlang-23.3
pkg.v23_3.elixir.v1_11_4 elixir-1.11.4_erlang-23.3
pkg.v23_3.elixir.v1_12_0 elixir-1.12.0_erlang-23.3
pkg.v24_0.elixir.v1_12_0 elixir-1.12.0_erlang-24.0
erlang.v23_3 erlang-23.3
erlang.v24_0 erlang-24.0
pkg.v23_3.rebar3 rebar3-3.15.1_erlang-23.3
pkg.v24_0.rebar3 rebar3-3.15.1_erlang-24.0
niv
to install ElixirI then had to add nix-beam
as a source of derivations for my nix-shell.
In most of my nix-shell configurations I use niv
.
niv
allows me to “pin” the version of nixpkgs
and also makes it easy to add other external dependencies.
$ nix-shell -p niv --run "niv init"
$ nix-shell -p niv --run "niv add jechol/nix-beam"
note that by default
niv init
will track the current nixpkgs-stable
Then I changed my shell.nix
to use the package sources generated by niv
.
# shell.nix
{ sources ? import ./nix/sources.nix }:
let
pkgs = import sources.nixpkgs { };
beam = import sources.nix-beam { };
inputs = [
...
pkgs.inotify-tools # example of nixpkgs package
beam.pkg.v24_0.elixir.v1_12_0 # package from the nix-beam repository
];
in
pkgs.mkShell {
buildInputs = inputs;
}
When I ran nix-shell
I noticed that after downloading some pre-built derivations a compilation process started. My laptop was about to build Erlang from source!
This is because packages from external sources are not part of the nixpkgs cache, nix-beam
only contains the derivation definition after all.
Thankfully the author of nix-beam
provides a hosted cache using cachix.
To use it and save a lot of compilation time first I had to install cachix
:
$ nix-env -iA cachix -f https://cachix.org/api/v1/install
and then add the provided cache for nix-beam
$ cachix use jechol
Then, I was able to run nix-shell
again and no compilation was required!
Note: if you use NixOS you might need to add your user as a
trustedUser
, that is required for thecachix use
part to work.
I hope this post helps you when you find yourself needing a dependency (Elixir or not) that is not available in the main package tree.
Funny enough, the nixpkgs Pull Request mentioned at the beginning of this post was merged a few hours after I solved the issue and Elixir 12 is now in the nixpkgs-unstable
channel.
Because of that, a simpler solution would now be to add a niv
source to track unstable
and install Elixir using that source.
Also of note, most of the community maintained derivations do not offer caches in cachix
but often they also don’t require a big build phase as the ones I was installing in this case.
My opinion is that Home Manager provides a radically better way to manage a user’s environment for both packages and dotfiles, effectively allowing you to take a configuration as code approach.
The goal of this post is to show you how to get started with Home Manager from scratch, install packages, port over some existing configuration and explore more advanced features.
This content does not require nor assumes knowledge of the Nix language, although you might find helpful to read some of the resources linked at the end of the post.
If you want to know more about Nix you can also read this post
Importantly by following this strategy you will be able to port your existing configuration incrementally and do it at your own pace.
One of the things that I’ve always found cumbersome of setting up a new laptop/desktop/server has been managing my user-level configuration and packages.
I’m referring to things like git
, vim
, tmux
, ssh
, window manager and terminal settings for my desktop machines and in general applications that are only required for my current user.
Over the years I tried several solutions for my configuration:
dotfiles
repository, sometimes managed with bash scripts like homeshickFor apps management I’ve used Homebrew on MacOS or my distro’s package manager and always installed packages globally.
After moving to NixOS I’ve finally landed on the Nix solution to this problem: Home Manager.
Home Manager allows you to use Nix’s declarative approach to manage your user-level configuration and packages. It works on any *nix system supported by Nix, including MacOS.
To get started you’ll need two things: Nix and Home Manager.
If you are not on NixOS the first thing you need to install is the Nix package manager.
Run the following command (you will need sudo
access).
curl -L https://nixos.org/nix/install | sh
Once complete follow the instructions to load nix
in the current shell or restart your terminal.
Test that everything worked by typing.
nix
You should see a help message.
Detailed instructions (and warnings) are available on the Home Manager homepage.
If you followed the instructions above to install nix
you will now be on the unstable
channel of the nix package tree (the one that has the most up to date packages).
The next step then is to add the appropriate channel for Home Manager.
nix-channel --add https://github.com/nix-community/home-manager/archive/master.tar.gz home-manager
nix-channel --update
Change the command above as required if you are tracking a different nix channel.
Now we can install home-manager
by typing
nix-shell '<home-manager>' -A install
You can test that everything worked by typing
home-manager
Congratulations, you are ready to go!
As you have seen in the install prompt, by default Home Manager initializes a new configuration in
$HOME/.config/nixpkgs/home.nix
that should look roughly like this
{ config, pkgs, ... }:
{
# Let Home Manager install and manage itself.
programs.home-manager.enable = true;
# Home Manager needs a bit of information about you and the
# paths it should manage.
home.username = "ghedamat";
home.homeDirectory = "/home/ghedamat";
# This value determines the Home Manager release that your
# configuration is compatible with. This helps avoid breakage
# when a new Home Manager release introduces backwards
# incompatible changes.
#
# You can update Home Manager without changing this value. See
# the Home Manager release notes for a list of state version
# changes in each release.
home.stateVersion = "21.05";
}
The first step is to move this configuration to a git repository, I prefer to have it in a different location and use symlinks to expose it to Home Manager.
Run:
mv ~/.config/nixpkgs ~/nixfiles
cd ~/nixfiles
git init
git add .
git commit -m 'getting started with Home Manager'
cd ~/.config
ln -s ~/nixfiles nixpkgs
Then test that you can apply the configuration.
home-manager switch
We have not added anything so this should be a no-op.
Home Manager will let you know that it’s “reusing lastest profile generation”.
First let’s see how we can use Home Manager to install packages for our user. In this example we’ll add tmux
.
Edit ~/nixfiles/home.nix
as follows:
home.homeDirectory = "/home/ghedamat";
+ # Packages to install
+ home.packages = [
+ # pkgs is the set of all packages in the default home.nix implementation
+ pkgs.tmux
+ ];
+
# This value determines the Home Manager release that your
Then run
home-manager switch
And now try it out
tmux
A great place to search for packages is https://search.nixos.org/packages, make sure you pick the right “channel”, if you are following this guide it will be unstable
.
home-manager switch
business?home-manager switch
is how you “activate” your configuration.
Home Manager will evaluate your configuration, build the result and atomically switch your old configuration with the new one.
This also means that it’s possible to see all old configurations
$ home-manager generations
2021-04-03 01:39 : id 2 -> /nix/store/5v5gnkq7ddkikdpghxlp039zfzxib1x1-home-manager-generation
2021-04-02 21:41 : id 1 -> /nix/store/rjzjszmwfrhmwzvqxhgy4l2a4rrr2xma-home-manager-generation
And you can rollback to older versions as well.
# copy the path from the command above and add /activate
/nix/store/rjzjszmwfrhmwzvqxhgy4l2a4rrr2xma-home-manager-generation/activate
...
# this will create a new generation
Creating profile generation 3
...
You can then switch again to re-apply your changes to go back to the current version of home.nix
.
$ home-manager switch
dotfile
So far we saw how to use Home Manager to install packages for our user, but a perhaps more important use case is manage our user configuration.
First we’ll see how to take an existing configuration file and make it part of Home Manager.
The simplest way to do this is to use the home.file
option.
Assume that you have a ~/.vimrc
with the following contents:
call plug#begin()
Plug 'LnL7/vim-nix'
call plug#end()
First let’s move it in our nixfiles repo
mv ~/.vimrc ~/nixfiles/vimrc
You can then edit ~/nixfiles/home.nix
as follows
pkgs.tmux
];
+ # Raw configuration files
+ home.file.".vimrc".source = ./vimrc;
+
# This value determines the Home Manager release that your
And run home-manager switch
again.
Now, let’s check what happened
$ ls -l ~/.vimrc
lrwxrwxrwx 1 ghedamat ghedamat 69 Apr 3 05:38 /home/ghedamat/.vimrc -> /nix/store/ynhkrdrc7hzrqqkq42iiqzb81bz8gaqc-home-manager-files/.vimrc
/.vimrc
is now a symlink to a file in the “Nix store”, the place where all the nix things go. Without concering ourself with details, the thing to notice is that if you now change the contents of ~/nixfiles/vimrc
and re-run home-manager switch
Home Manager will detect the changes, create a new version of .vimrc
in the Nix store and update the symlink.
$ echo "hello nix" > ~/nixfiles/vimrc
$ home-manager switch
$ ls -l ~/.vimrc
lrwxrwxrwx 1 ghedamat ghedamat 69 Apr 3 05:47 /home/ghedamat/.vimrc -> /nix/store/dsq0da2y4p7w67imwnd95crv4k35d6qb-home-manager-files/.vimrc
It is true that managing configuration in this way will add a step every time you want to edit your vimrc
. I believe that this tradeoff is worth it even if you were to decide to not use any other feature offered by Home Manager as you now have a reliable and consistent way to manage all your configuration files and packages.
Using the home.file
configuration option is my preferred way to port existing configuration files . Once that’s done though Home Manager has much more to offer.
Home Manager comes with a large amount of pre-defined configuration modules
for a variety of applications (full list on github).
These modules allow you to use a consistent syntax (the Nix language) to configure every application regardless of the output format that each program requires (ini, yaml, custom…).
By using modules you will get type safety guarantees about your configuration as it will be now written in Nix and the modules can specify types for each option you pass. This also means that you can use the Nix language to add logic (i.e conditionals and functions) as well as the ability to compose your configuration as you would with any other functional program written in Nix.
The downside is that you have to learn at least a small part of the Nix language (mostly how to write sets
, which are similar to maps
and hash
in other languages).
Once you have identified a module you are interested in, all the options available are listed in the Home Manager Manual.
git
configLet’s see an example with porting over our ~/.config/git/config
to Home Manager.
# current contents of ~/.config/git/config
[user]
email = ghedamat@gmail.com
name = ghedamat
[alias]
st = "status"
Edit home.nix
as follows (you can find the full list of options for programs.git
here)
home.file.".vimrc".source = ./vimrc;
+ # Git config using Home Manager modules
+ programs.git = {
+ enable = true;
+ userName = "ghedamat";
+ userEmail = "ghedamat@gmail.com";
+ aliases = {
+ st = "status";
+ };
+ };
+
# This value determines the Home Manager release that your
Let’s try to apply this change
$ home-manager switch
Existing file '/home/ghedamat/.config/git/config' is in the way of '/nix/store/9k4rq2d247qcs9n12f324wg9b7120i57-home-manager-files/.config/git/config'
Please move the above files and try again or use -b <ext> to move automatically.
Ah! Home Manager noticed that we already have that file in the system and will not override it with the one it generates. Neat!
The fix is simple:
rm ~/.config/git/config
home-manager switch
We can verify that the file is now generated by Home Manager (notice the content is slightly different)
$ cat ~/.config/git/config
[alias]
st = "status"
[user]
email = "ghedamat@gmail.com"
name = "ghedamat"
The author of Home Manager recommends to start with a single home.nix
file and I would definitely agree. As you learn more about the Nix language you’ll find about all the different ways to structure your code.
Later, you might want to learn about using imports to break down your configuration into multiple files. A more advance approach is to build your own Nix modules.
I might decide to cover these in a future post.
There is much more to discuss and cover but this should be enough for a short introductory tutorial that aims at getting you started.
In summary:
home.file
for most thingsI also want to leave a big thank you to the Home Manager contributors and the Nix community at large, these tools have greatly improved my workflow.
Feel free to reach out with any comments/questions/suggestions and correction about what’s written in this article.
My hope is to see more people adopting Home Manager and Nix :)
“Let me tell you how much I dislike configuring dev machines: I use Macs but haven’t installed dev tools since 2011. I got tired of OS X releases breaking them so I do all of my dev on external machines or VMs. I tried Home Manager and it ended working better than I expected.” @elucid
“I see how it is a better idea that just hand configuring everything poorly like I do now.” @benjamintmoss
Here’s a list of other blog posts and resources that will help you get started:
Many thanks to @shazow and @eviltrout for their feedback on this article.
]]>I’ve been setting up my development environment like described in this post since when I started working from home 6 years ago. It’s been great for me and many on my team and I’ve been recommending this to friends for quite a while but I’ve never formalized it.
I want to thank my colleague Ben who got interested and nudged me into writing this :)
What I’m about to describe has additional complexity compared to running everything on a laptop, as most people do, but I think the trade-offs are well worth it.
Don’t run your development environment on your laptop. Use the laptop as your client (Browser, Slack, Zoom, VSCode) and run your code on another machine (ideally running Linux) that you can connect to with a low latency. This allows you to split the load across multiple computers and separate different load types to different environments.
Below I’ll explain in detail how I’ve been doing it.
Have you ever been on a Zoom call, trying to pair with a colleague and experienced super slow build times? Is your Macbook attempting to fly off your desk when your tests are running? Do you have to reboot your machine regularly?
If the answer is no, I’m really happy for you! If it’s yes, I can guarantee you’re not alone.
Most developers seem to be working on laptops, for good reasons. You can move them around, you can change spot during your day, you can take them with you when you leave the house. But, because of the tradeoffs laptop producers have to make, their performance often is not enough to support our daily workload. Especially at a time when everyone has video conferencing and chat apps opened at all times.
Running a dev stack in 2020 is resource intensive, If you are like my team at PrecisionNutrition you probably have:
On top of this, your computer is likely running:
No wonder your having some performance issues.
I’ve previously written about how these days my go-to solution for every project is to use Docker to run services (i.e databases) and use Nix to run the code so that I can have native performance. While this has been a big improvement, informal testing among my team at PN has shown that - particularly of macOS users - build times and general performance are still an issue on laptops.
The “thin-client” idea is by no means new, but compared to the 80s the big difference is that now gigabit ethernet and high speed wifi are commonly available.
This makes implementing a low latency client-server experience quite affordable.
To implement this approach you will need 3 things:
Whatever you prefer to use as a desktop driver is up to you, this can be your laptop or a desktop computer. It can run macOS, Linux or Windows.
You will likely need to buy this, or if you’re like me and keep a lot of old hardware, you can probably re-purpose an old system you have sitting in storage. The good news is that this machine does not need to be super powerful. I use a 4 year old Intel NUC Skull Canyon and it still outperforms a modern Macbook Pro.
If you are looking for something to buy the newest NUC models look pretty sweet.
Whatever you use, all it has to do is run a modern Linux install, you won’t need monitors or other peripherals once the setup is done.
Your primary goal here is minimize latency. You will be typing on a remote terminal (possibly even a remote editor) and even medium latency can be noticeable. A gigabit network switch or router to connect server and client, alternatively get some fast WiFi but I would recommend plugging in the server directly onto your router.
You found a PC sitting in your basement, or you bought a new fancy NUC. Perfect, here’s what to do next.
If you are not used to Linux this will require some learning, read some guides, be patient, ask for help. If this the case, even though I’m an big NixOS fan, the Linux distribution I recommend to start with is Ubuntu Server LTS; it’s supported well and editors know how to interact with it (see later for notes about VSCode).
If you are used to doing development only on macOS, you will have to figure out what are the equivalent Linux packages for your development dependencies.
Generally speaking in Ubuntu you will end up needing something that looks like this:
sudo apt install build-essential libxml2-dev libz-dev libxslt-dev
Unfortunately this will vary based on your programming language and project so be ready for some googling. If you were to find this too painful it might also be interesting to consider switching to nix-shell.
Once the base setup is done, make sure to enable ssh (apt install openssh-server
) so you can start connecting to the server from your client’s terminal app.
Find your router’s DHCP settings page and make sure you instruct it to always assign the same IP to your server, note it down as you’ll need soon. This is important as we will configure your client to resolve an internal domain to your server in the next step.
Note: there are more complex alternatives here, I personally recommend running your own DNS server within your home network but I’ll probably cover this in a future post.
We now need to configure your client so that it will be able to resolve your server, the simplest way to achieve this is to edit resolv.conf.
# the following works on macOS and Linux
sudo vim /etc/hosts
# assuming your server ip is 192.168.1.144
192.168.1.144 myserver myserver.mydomain.local
If you wanna learn more about basic networking on Linux I recommend this fantastic zine from Julia Evans.
At this point you should be able to open a terminal on your client and
ssh myuser@myserver
# then run
python3 -m http.server 8000
if you start an http on your server you should be able to visit http://myserver:8000 in your browser and see it running.
From now on every time you need to run any command it will be done over ssh onto your server. To make this a bit easier I recommend using tmux. tmux
is a terminal-multiplexer, it allows to manage a remote session with multiple terminals. This way you don’t have to open a new ssh connection for every terminal and the session can stay active even after you disconnect from it. This has other interesting side-effects, i.e. it will allow you to keep your server running even when your laptop goes to sleep or is disconnected from the network.
Once you go back to work in the morning you just have to ssh onto the server, reconnect your tmux session and you’re ready to work.
I almost never shutdown my server and my typical tmux session has several terminals open at all times:
If you have never used tmux the tmux 2 book is a great resource, I’ve also heard really good things about The Tao of tmux.
Warning: when running your webserver make sure they are listening on 0.0.0.0 and not on localhost only otherwise you won’t be able to visit http://myserver:YOURPORT
If you are a user of terminal editors like VIM on Emacs you are pretty much set. Use tmux to run your commands and your editor. With the DNS resolution setup correctly, use your browser from your client machine and that’s it.
What about “nicer” editors though? Your code now lives on the server so the editor has to as well or does it?
There are a few solutions I’ve used over the years and YMMV depending on your editor of choice.
If you are a VSCode user you’re in luck, there is a great remote ssh extension that allows you to run a remote code
session directly from your local client.
VSCode will also take care of installing itself on the remote system. Opening terminals within the session will give you terminals on the remote system and even the Language Server will work properly because all commands are run remotely.
This is the one that “should” always work. Install and configure samba on the server, share the directories that have your code and mount them on your client. If you are on a high speed connection even functionality like quick file search should work pretty well.
The main problem with this solution is that you will not get Language Server working without also installing some dependencies on your client. The editor is running locally after all.
The X server is what is used to run graphical applications on Linux. The trick is, it does not need to be running on the same machine that is running the graphical application.
If you are running an X compatible server on your client you can “forward” it to the server and the server applications will know to use that when launched. This is a bit mind-bending because your client is now acting as the graphical server for your server…
If you are on linux you likely have X already running, on macOS you will need to install XQuartz while on Windows you will need something like MobaXterm.
The simplest way to get forwarding to work is by using SSH tunneling.
You can try it out like this:
ssh -Y myuser@myserver
sudo apt install x11-apps
xeyes
The xeyes
app is running on the server but the output is forwarded to your client. You could install the Ubuntu version of VSCode and run it the same way.
This is not incredibly efficient but depending on your connection speed and your tolerance for latency it might do the trick.
A more performant option is to allow the X server running on your client to accept connections from your server directly. This is a bit tricky to do depending on your setup. As an example on Linux this is achieved by passing the option to -listen tcp
when the process is started.
Once you have this sorted you can on your client allow insecure connections to your X server by issuing xhost +
.
This has some security implications that you should research but I consider it generally safe within my private home network.
Finally in your ssh connection on the server you have to “export” the “DISPLAY” environment variable so that it knows how to connect to the X instance running on your client.
example:
ssh myuser@myserver
export DISPLAY=192.168.1.12 # this is the IP of your client
code
I find that over wired gigabit this solution has basically no latency but as mentioned it requires some extra work to get going.
Some resources you might find useful if this is the way you want to go:
Props to Luke Galea for showing me this one.
Admittedly there are some steps involved into setting up an environment of this kind and you will need to commit some time but my experience has been incredibly positive.
A thing that I find myself doing every day is switching from my desktop computer to my laptop seamlessly. This also allows me to work form a fairly under-powered 12 inch laptop with no issues.
My current setup is in fact more complex than the one explained in this article and I plan to cover it in a future blog post. The main change I made (with lots of help from luke) is that the server now hosts an hypervisor that can run multiple virtual machines. This allows me, among other things, to have my own DNS server, simplified local testing on mobile devices, snapshotting of my virtual machines and much more.
Aside from the obvious one of having to learn how to do all this and having to buy a server, this setup does not work for every use case:
nix-shell
and VSCode play nicely when using the SSH extension option.👋 As always, suggestions and corrections are welcome and encouraged! Feel free to reach out to me on twitter or via email [ghedamat at gmail] and let me know what you think!
Many thanks to @typeoneerror and @benjamintmoss for their feedback on this article.
]]>On June 26th I gave a short talk at our Toronto Nix Meetup, this post is an extended version of that talk.
If you are in a rush, you can skim through the slides here:
👋 A word of warning, there is a fair amount of hand-waving ahead. Suggestions and corrections are welcome and encouraged! Feel free to reach out to me on twitter or via email!
nix-shell
nix-shell — start an interactive shell based on a Nix expression
The command nix-shell will build the dependencies of the specified derivation, but not the derivation itself. It will then start an interactive shell in which all environment variables defined by the derivation path have been set to their corresponding values, and the script $stdenv/setup has been sourced. This is useful for reproducing the environment of a derivation for development.
Running nix-shell
will start an interactive bash
shell, in the current working directory. The packages required (we’ll see shorty how to specify them) will be downloaded but not installed globally. Instead the shell will have its ENV
set appropriately so that all the packages in the shell definition are available.
You can test this by typing
env
...
# lots of stuff
nix-shell -p ripgrep
env
...
# lots of stuff but with a bunch of new things!
nix-shell
usesnix-shell --packages
# or
nix-shell -p
Starts a nix-shell
that has the package available in its $PATH
$ which rg
rg not found
$ nix-shell -p ripgrep
[nix-shell:~]$ which rg
/nix/store/rw24lqk4ls1b90k1jj0j1ld05kgqb8ac-ripgrep-11.0.2/bin/rg
nix-shell
Building on the above, you can temporarily add a package and immediately use it
$ nix-shell -p ripgrep --run "rg foo"
shell.nix
The nix-shell
command receives an optional argument for a .nix
file. By default if invoked with no arguments nix-shell
will first look for a file named shell.nix
and then for one named default.nix
.
This .nix
file has to contain the definition of a derivation, the standard library offers a special derivation function mkShell
specifically for this purpose (although the more general stdenv.mkDerivation
can still be used).
Derivations are the building blocks of a Nix system, from a file system view point. The Nix language is used to describe such derivations. (cit. Nix Pills)
Here’s a basic shell, it provides only the buildInputs
attribute, that is, the list of packages to make available in your shell.
# simple.nix
with (import <nixpkgs> {});
mkShell {
buildInputs = [
ripgrep
];
}
$ nix-shell simple.nix
[nix-shell:~]$ rg foo
# ...
You can also provide the shellHook
attribute to customize the bash
shell being spawned.
# hooks.nix
with (import <nixpkgs> {});
mkShell {
shellHook = ''
alias ll="ls -l"
export FOO=bar
'';
}
$ nix-shell
[nix-shell:~]$ echo $FOO
bar
nix-shell
for developmentWhere nix-shell
really shines for me is in its ability to provide uniform and shareable configuration for development environments in virtually any language.
In this section I’ll provide a few examples of shell.nix
configuration files for different programming languages. You can imagine these shell derivations as a drop in replacement for what usually is done with language-specific version managers like Rvm, nvm and asdf but, as we’ll see, this approach is beyond just managing language versions.
I am not a Python dev, but from my experimentation the support for Python feels quite “native” in Nix, one can create a custom Python build for the shell, and add the desired dependencies. A lot of Python version and packages are already available in the main nixpkgs
package tree.
The following example is lifted from the NixOS Wiki
# python.nix
with (import <nixpkgs> {});
let
my-python-packages = python-packages: with python-packages; [
pandas
requests
# other python packages you want
];
python-with-my-packages = python3.withPackages my-python-packages;
in
mkShell {
buildInputs = [
python-with-my-packages
];
}
Here’s also a recent blog post about using Python on Nix.
For Rust mozilla has been providing a shell.nix
to get you started
# rust.nix
with import <nixpkgs> {};
let src = fetchFromGitHub {
owner = "mozilla";
repo = "nixpkgs-mozilla";
rev = "9f35c4b09fd44a77227e79ff0c1b4b6a69dff533";
sha256 = "18h0nvh55b5an4gmlgfbvwbyqj91bklf1zymis6lbdh75571qaz0";
};
in
with import "${src.out}/rust-overlay.nix" pkgs pkgs;
stdenv.mkDerivation {
name = "rust-env";
buildInputs = [
# Note: to use use stable, just replace `nightly` with `stable`
latest.rustChannels.nightly.rust
# Add some extra dependencies from `pkgs`
pkgconfig openssl
];
# Set Environment Variables
RUST_BACKTRACE = 1;
}
The interesting thing here is that the Mozilla Nix overlay is fetched as part of the shell derivation, this shows how shells are not limited to a single source for packages.
My current understanding is that in the Nix way a package and its dependencies are “reproducible”, the final derivation we build is always gonna be the same because the inputs will always be the same.
Languages like ruby
, js
and others don’t “naturally” provide this guarantee but the Nix ecosystem has produced a few ways to work around this problem
For Ruby this pattern is implemented using bundix
:
bundix
runs against your Gemfile
and generates a Nix expression that includes all the Ruby dependencies used in your project
With that you can define a nix-shell
that will have all the dependencies available and can effectively avoid using bundler
(the Ruby package manager) in your workflow.
Given a Ruby project with a Gemfile
you can:
bundix -l
gemset.nix
in your shell.nix
# bundix.nix
with (import <nixpkgs> {});
let
gems = bundlerEnv {
name = "your-package";
inherit ruby;
gemdir = ./.;
};
in mkShell {
buildInputs = [gems ruby];
}
Similar solutions exist for other languages, for example Node has yarn2nix
.
While the previous approach has some really good advantages I personally found that for personal projects, and for my team at work, a less Nix-y solution has been working better.
The strategy that I have been using is to override the environment variables that the package manager provides and force the installation of packages to happen locally to the directory in which the shell is being used.
Here’s an example for a NodeJS shell.
# node.nix
with (import <nixpkgs> {});
mkShell {
buildInputs = [
nodejs-12_x
yarn
];
shellHook = ''
mkdir -p .nix-node
export NODE_PATH=$PWD/.nix-node
export NPM_CONFIG_PREFIX=$PWD/.nix-node
export PATH=$NODE_PATH/bin:$PATH
'';
}
And here’s a slightly bigger one that I’ve used for Ruby on Rails development
# ruby.nix
with (import <nixpkgs> {});
mkShell {
buildInputs = [
nodejs-12_x
ruby
yarn
gnumake
gcc
readline
openssl
zlib
libiconv
postgresql_11
pkgconfig
libxml2
libxslt
];
shellHook = ''
mkdir -p .nix-gems
export GEM_HOME=$PWD/.nix-gems
export GEM_PATH=$GEM_HOME
export PATH=$GEM_HOME/bin:$PATH
export PATH=$PWD/bin:$PATH
gem list -i ^bundler$ -v 1.17.3 || gem install bundler --version=1.17.3 --no-document
bundle config build.nokogiri --use-system-libraries
bundle config --local path vendor/cache
'';
}
I’ve also talked about how to do this in Elixir in a separate blog post.
At a high level, this tecnique is very similar regardless of the programming language:
ENV
variable that determine the installation paths for packages and executables$PWD
$PATH
to include the installation path for binaries (so that things like npm install -g
work)shell.nix
A “trick” that I have found useful is being able to import from a different channel within a Nix derivation, this is often useful if in your shell.nix
you want to install packages from a more recent version. In the following example I’m using the unstable
channel while my host system <nixpkgs>
are version 20.03
.
with (import (fetchTarball https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz) {});
mkShell {
buildInputs = [
git-up
];
}
<nixpkgs>
SHA.When sharing a shell.nix
it can be helpful to “pin” the <nixpkgs>
version. This guarantees that regardless of the nix-channel
used on the system everyone gets exactly the same Nix packages.
This is done by specifying a commit SHA directly from Github.
with (import (fetchTarball https://github.com/nixos/nixpkgs/archive/8531aee99f4907bd255545eb94468e52a79a44f1.tar.gz) {});
mkShell {
buildInputs = [
git-up
];
}
This guarantees that so long as you specify all the dependencies, and don’t accidentally rely on something coming from the OS, every user will get the same setup.
This tutorial also offers a good explanation.
shell.nix
As soon as we started using a shared shell.nix
at work it became clear that there was a need to customize the some aspects of the shell on a per-user basis.
The solution I resorted to is check if a local.nix
is present and if so expect that file to provide an attributeSet
with two attributes: inputs
and hooks
.
These attributes are merged with the ones provided by the shell.nix
that is checked in into your git repository.
# shell.nix
with (import <nixpkgs> {});
let
basePackages = [ ripgrep ];
localPath = ./local.nix;
inputs = basePackages
++ lib.optional (builtins.pathExists localPath) (import localPath {}).inputs;
baseHooks = ''
alias ll="ls -l"
'';
shellHooks = baseHooks
+ lib.optionalString (builtins.pathExists localPath) (import localPath {}).hooks;
in mkShell {
buildInputs = inputs;
shellHook = shellHooks;
}
# local.nix
{ pkgs ? import <nixpkgs> {} }:
{
inputs = [ pkgs.curl ];
hooks = ''
alias ghedamat="mattia"
'';
}
nix-shell
Nix works both on MacOS and Linux but there are some dependencies that are platform specific. The following example shows how these can be accounted for in your configurations
# cross.nix
with (import <nixpkgs> {});
let
basePackages = [
ripgrep
];
inputs = basePackages
++ lib.optional stdenv.isLinux inotify-tools
++ lib.optionals stdenv.isDarwin (with darwin.apple_sdk.frameworks; [
CoreFoundation
CoreServices
]);
in mkShell {
buildInputs = inputs;
}
nix-shell
and docker
nix-shell
works great to configure dependencies but does not really solve for services. Often your development environment will require one or more databases (I often need PostgreSQL and Redis running). Such services can be installed at the system level but project-based isolation is in my opinion preferred.
While I found that nix-shell
is a much better development environment than docker
I do think that running services is what the latter excels at.
The solution then becomes: use both! docker-compose
for services and nix-shell
to run code!
I wrote a previous post on how at Precision Nutrition we implemented this hybrid approach. If you are interested I encourage you to read it and let me know what you think!
Extending nix-shell
to allow for more customization (i.e. using zsh
or not have to type nix-shell
every time) is beyond the scope of this post but I will leave a few pointers here for the interested reader.
nix-shell --run zsh
is a simple workaround that allows you to change the $SHELL
from bash
direnv
can be used to take this a step further and “load” the nix-shell
ENV without spawning a new shelllorri
is another project that aims at replacing nix-shell by extending it.nix-shell
allows you to define development environments for pretty much any language in a consistent way, it makes also easy to support different versions of the same language!shell.nix
to your project can be used to ensure that everyone on the team has the same configuration and is also a great way to help new contributors get setup quickly.docker
and nix-shell
for projects that require databases or other services, is the way to go!Thanks for reading!
Nix has been around the block for a while but recently, both from outside and from within the Nix community, I’ve seen several efforts to make Nix more beginner friendly.
I have been using NixOS for a year or so, and I had myself to go through what was a somewhat confusing and effortful onboarding process. I was lucky enough to have friends like @shazow to help me on the way but I still feel I have many gaps in my understanding.
I’m writing this post is to collect my thoughts on Nix and present a few of the things that I found helpful during my Nix journey.
My hope is that they can also help others learn and use what I consider being the next step when it comes to developer ergonomics both on MacOS and Linux.
In the resources section below you will find more detailed and precise explanations so feel free to skip there!
The first confusing thing about Nix is that the name is used to refer to a few things:
I’ll briefly cover what I like and how I use each one of these, leaving some resources at the end of this post for readers that are interested in a deeper exploration.
Nix can be used like brew
on MacOS or apt
, yum
, emerge
and many others on Linux. It can be installed locally for a single user or globally on any Unix based system (including MacOS).
One you have installed Nix you can do stuff like
$ nix-env -iA curl
and Nix will download and install the package you selected and all its dependencies. It will be there for you to use until you decide to delete it.
There is a lot to be said about how Nix works and all the features that it brings to package management, a notable one being reproducible builds that guarantee that a given version of a package will always be the same, including all its dependencies. Some of the links at the end of this post will allow you to explore this topic.
One thing that is unique to Nix is the ability to download and run the package without the need to install it globally, you can instead spawn a new bash
shell with the packages you are interested in
$ nix-shell -p curl
[nix-shell:~]$ curl -I http://nixos.org
# ...
once you leave this shell curl
will not be available anymore.
nix-shell
s can do this an much more, they also allow you to define isolated development environments, and share them with other developers ensuring that they can quickly spawn a bash
shell using exactly the same dependencies you have.
nix-shell
is a complex topic though (and I have much left to learn myself), I will probably cover it more in a future post, for now you can see a small example here.
Quoting directly from the NixOS website:
In NixOS, the entire operating system — the kernel, applications, system packages, configuration files, and so on — is built by the Nix package manager from a description in a purely functional build language. The fact that it’s purely functional essentially means that building a new configuration cannot overwrite previous configurations. Most of the other features follow from this.
Unlike most operating systems NixOS allows users to describe their desired system configuration and then leave it to Nix to apply said configuration and make all the required changes.
There are no configuration files to edit, copy, backup outside of the .nix
ones in /etc/nixos
.
Here’s an example of the full configuration (minus the hard-drive part) required to provision a PostgreSQL server.
{ config, pkgs, ... }:
{
# Include the results of the hardware scan.
imports = [
./hardware-config.nix
];
environment.systemPackages = with pkgs; [
curl
wget
];
# Use the systemd-boot EFI boot loader.
boot.loader.systemd-boot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;
networking.hostName = "postgres-server"; # Define your hostname.
networking.firewall.enable = false;
networking.interfaces.enp6s18.useDHCP = true;
# Define a user account. Don't forget to set a password with ‘passwd’.
users.users.ghedamat = {
isNormalUser = true;
extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
shell = pkgs.zsh;
};
programs.zsh.enable = true;
programs.zsh.enableCompletion = true;
programs.zsh.promptInit = ""; # Clear this to avoid a conflict with oh-my-zsh
# postgres config
services.postgresql = {
enable = true;
package = pkgs.postgresql_11;
enableTCPIP = true;
};
system.stateVersion = "20.03";
}
There’s a lot going on here and much more to say but it is hard to describe, as a long time Linux user and system administrator, how much joy the ability to declare the state of my system configuration brings me.
Oh and there’s one more thing!
Thanks to the magic of the Nix package manager you can also rollback safely to any previous configuration you ever ran on your machine.
The Nix community created other tools the follow the same philosophy:
home-manager has completely changed how I manage my dotfiles
, now I have a single place to configure all my user-level applications across different machines. As an example, this is my ThinkPad config.
With NixOPS I am managing my laptop, my desktop and all my development VMs. I’ll try to cover this setup in a future blog post, but you can get a sense for it here.
The Nix language is what powers this all. Truth to be told, I can’t say that I have a solid understanding of it even after using Nix for quite a while.
The good news is that I feel that’s OK. Even with my limited proficiency I have been able to manage several systems: laptops, desktops and VMS. All my dotfiles are now managed in Nix too and I also moved over our entire development environments at work to use nix-shell
.
My advice is to not get intimidated and learn it more as you need it.
If you looking to give this a go, whether you are on MacOS or Linux my advice is:
1) install Nix for your user
2) use nix-shell
to try a few packages
3) start using home-manager to manage your installed packages, and maybe try to move a few of your dotfiles
over
4) start using nix-shell
to define per-project development environments
Here’s some of the resources I found the most helpful getting started with Nix:
Finally a thing that can be intimidating at first but works super well is reading the source. Once you get the basics it’s surprisingly approachable.
And last but not least have a look at other people nixfiles. It was a great way for me to get started. If you are curious these are mine and shazow’s nixfiles.
Hope this will help you or maybe even convince you to join the Nix community! Feel free to tweet at me for any feedback or questions.
We also have a very small Toronto meetup and discord channel. Reach out if you wanna join us!
On Thursday June 25th we are running our first online Toronto Nix meetup.
Regardless of where you are, come join us!
]]>This is a personal post about my relationship with meditation and my life journey so far with respect to this practice. I’m not sure if it will be helpful to anyone else, but I wanted to push myself to reflect a bit on this subject so here we are.
I discovered meditation ~10 years ago, before moving to Canada I was struggling with anxiety and emotional balance and my therapist at the time suggested that I should give it a try.
My journey started attending some groups organized by a counselor and through him I ended up taking by-weekly classes for two years with Andrea Cappellari (link in Italian). Andrea taught both Shamata (concentration) and Vipassana (insight) meditation. What really stuck with me of his approach was that, even if all his techings where heavily rooted in Tibetan buddishm, he was always keeping a tecnical attitude and regularly stressing that no religious beliefs were required to rip the benefits of the practice.
One thing that Andrea always pushed his students to do was to pratice at home, and that was a thing that I invariably failed to do regularly. I would practice every few days, especially when feeling down or anxious but quickly stop and go back to my usual routine and favourite addictions (mostly cigarettes and drinks).
In those years I also attended a few meditation retreats and, while I always loved the experience, coming back to “normal” life afterwards always lead to the same outcome. I would come home and stop meditating entirely for months! Some way of compensating for the excessive effort maybe..
Moving to Canada didn’t help, a new life to build, a job to find, a VISA to obtain, finding a house, having my partner move from Italy and eventually, over time, rebuilding a life from scratch took a lot of energy and also worked as a very good excuse to not look closely at the idea of resuming with meditation.
For the first few weeks, when I was here alone and completely lost, I found good help in attending a meditation group of the shambala tradition, but quickly stopped going once I got busy with a full time job and had enough friends to go out with and explore my new city.
Once things start rolling they can get pretty fast, before I knew it we owned a house, a dog, two jobs, friends and effectively had built a new life in Canada (I do love and will be always greatful to this country and what it gave me), but meditation was entirely gone. Just something that I would think about every once in a while.
During these years I tried a few times to use Headspace to get back into the habit. While I think it’s a great app it never worked for me because I was still approaching meditation as a way to feel better. Hence I would meditate only when feeling particularly bad or anxious.
As the year passed my lifestyle changed substantially, I gave up smoking (although I still miss it every day) and later (~3 years ago) also had to give up driking for health reasons. Aside from the occasional use of pot (legal in Canada), mostly CBD for pain management, I found myself leading a life free of most stimulants. 35 year old me looks very fondly at 20 year old me, but I don’t think I could survive more than 2 days of what I used to do to my body back then!
Late in 2019 I found myself battling again with anxiety, even though my conditions are not particularly serious I started observing how they affected my mood. Chronic pain is a bitch, aside from the annoyance of the suffering (and I consider my pain fairly low most days), there are two things that I found really hit me. First I became more sensible to days in which the pain comes back, especially after feeling well for a while, when the pain returns I have feelings of anger, sadness and my mood is way less stable. Second, and more pertinent to the topic, my anxiety started coming back, I find myself worrying that the pain will come back, or that other things will start causing me pain. I’m not sure about others but my mind can always find a way to get anxious about something and now I had the perfect excuse.
I said I realized this was happening in 2019 but, if I’m honest about it, the anxiety never left, getting sick just gave it a head start and now I was just catching up with what happened.
I’m an engineer, that approach to life is embedded with my sense of self, but it’s also very much how I tackle problems. This means that when faced with an issue my usual approach is to study. Around this time I ran in to The Craving Mind by Jud Brewer and immediately started reading.
Even if I had already encountered a lot of the concepts that Jud Brewer talks about (i.e. The power of habit), even if I was already familiar with Jon Kabat-Zinn and even if this is what Andrea always referred to in his lessons, the connection between my anxiety patterns and how meditation could help untangle them had never been as clear.
Unwinding Anxiety is an headspace-like app that was developed to put into pratice the ideas from the book and I decided to give it a shot.
The parallel between the method the app uses and what we do with our clients at work (maybe I’ll talk about my experience with PN coaching too at some point) are stirking and that probably gave me faith in the methodology, given that it worked so well for my nutrition and training habits in the past.
The program guides you to bring mindfulness to you anxiety, to look at your patterns and learn to observe yourself as things happen. Interestingly enough, there’s very little “traditional meditation”.
Going through the first few weeks of the course really worked, things kept happening but I felt less of a victim of my patterns. After a while though something changed and my interest shifted. I was feeling better but I wanted to understand more of the why. Not why in a scientific sense but why from an experiential standpoint. What is there, what is this mind that is doing this to me and how does it work?
And this is when I started going back to Andrea’s teachings (well Buddhism techings to be precise). The following is a great and imprecise oversemplifications but it conveys my current understanding. The mind is a muscle, and like other muscles it can be trained. Through concentration practice we can stabilize the mind, once the mind is stable the mind can be used to observe, traditionally observation can be of sensations, feelings, the mind itself and all phenomena. The mind can then see itself and see what is real.
I wanted some answers and I had a way to go get them. Not to feel better, not to run away, but to face them.
So I started sitting, once again with the help of a book: Genuine Happines by Allan Wallace. I know very little about him but he was Andrea’s teacher and I bought the first copy of this book many years ago in Italy, keeping it on a shelf, without ever really trying to put it into practice until now.
The book guides you through a series of different buddhist meditations, and this happens over the course of several months. I will not claim that this is on par with having a teacher and a group with you but so far it’s working out allright.
So that’s what I’m doing, I started 108 days ago and I’m slowing making my way through, 25 minutes a day, most days I feel I can barely keep my mind on the object of meditation, other days it seems to work out.
I still get anxious. I still get angry. I still mess up regularly as I’m sure my partner (bless her), my friends and my collagues can testify.
But it’s been a ton of fun!
I don’t know if my experience is unique or common but this is what has been true for me so far. It’s really hard to do something if my goal is feeling better because as soon as I do I lose the motivation.
I believe that the combination of having a different reason and removing all factors that were hindering my practice (mostly smoking and drinking) were the successful combination that got me here.
For the record. I still feel like I “suck” at the pratice itself. The mind is still a wild monkey, jumping all over the place, observing it without getting involved only happens for a few instants, the rest of the time… I become the monkey. The difference with my past attempts is that this time I keep showing up.
I’m very curious to see what happens next. Will I be able to keep it up or will I pick up old habits again? How is travel (when that is a thing we do again) gonna affect it? How are bigger life changes gonna change it? I don’t know. And I’m looking forward to finding out.
I’m not in the business of giving advice but I’m gonna leave a thing for my future self.
Even if you fail. Be kind. If you fuck up it doesn’t mean that you’re a fuck up.
]]>