This is an update on how I am getting along with nixos/nix since last time. At that point I had only installed nixos, added a few essential packages and even started editing and generating this blog on the new system. Now I want to play around with bleeding edge Haskell code and not pay a heavy price in terms of system configuration. If I can continue on from there to develop production code I will consider the experiment a success and may migrate to using nixos full-time.
Update 07–09–2016: nix continues to evolve. So much so that a blog post like this or even the nix wiki often provides out-of-date information. I am working on an updated post about nix, but until then and after, I recommend you rely on the nix manual for authoritatve informantion about nixos and nix. If you encounter problems head straight the github repository for nix and check for related issues.
Nixos is a linux distribution that uses the nix package manager to manage your system, while nix is a package manager that can be used on a variety of systems, perhaps as a replacement for homebrew on mac, or in addition to apt-get on ubuntu. There is also a cloud management tool called nixops that is very interesting in its own right.
One of the big draws of nix is the idea that it sandboxes everything. That means every package you install keeps track of all its inputs and outputs in /nix/store. Nix manages your environment by adding and removing packages from the various paths that make up your environment. Conflicting packages can easily coexist on the system, since every package can specify all its dependencies in complete ignorance of other packages. Some Haskell devs have moved to nix to avoid so called “cabal hell”. Cabal sandboxes, especially the newer version, work fine for many users, but nix also covers code outside of Haskell and makes it easier, in my opinion at least, to support multiple versions of ghc on your dev box.
I have never gotten the hang of running multiple ghc versions on my ubuntu dev box and have never seen a solution that looked easy to manage. Frankly, I am a coward, since I use my dev box to produce production code, and I don’t want to break anything besides my own code. Of course breaking things is probably the best way to learn. Nix promised to make it easy for the cowardly and fearless me to work on the same box. That’s a very enticing claim to test.
So far the learning curve for nix has been steep, though maybe less steep than learning Haskell. Before doing some more serious hacking, here are some suggestions that a “normal” Haskell developer like me might wish to consider to make the learning curve that much easier.
The documentation for nix is fairly complete, but sometimes seems written by experts, for experts. Fortunately there are a few good introductory blog pieces around, many of them written by devs who appear to be heavily involved with the nix project. Here are a few in no special order that I found helpful:
Special mention goes to a couple of blog posts by Ollie Charles, which convinced me to finally invest some serious time in nix.
It is also worth highlighting a few of the most helpful spots in the offical docs.
The installation command nix-env -i lets you refer to packages either through their “attribute path” or their package name. The latter is very slow due to the way the query tool currently works.
As a result, for me
nix-env -iA nixpkgs.haskellPackages.SomeHaskellPkgis usually dramatically faster than
nix-env -i haskell-some-haskell-pkgWhere to find these attribute paths? You can do nix-env -qaP to get the attribute paths. For a while I was saving the output to a file and searching for packages there until I learned the next trick.
It is as easy as
nix-env -iA nixpkgs.nix-replHaving the repl available does a number of things for you. First, since nix uses the nix expression language for all its build recipes, nix-repl is like ghci for that language. You can play around with simple expressions to get used to the language. That will come in handy later.
nix-repl is also handy for inspecting available packages. Try this…
$ nix-repl
nix-repl> nixpkgs = import <nixpkgs>
nix-repl> :a builtins
nix-repl> attrNames nixpkgs
builtins.attrNames pkgs
[ "AgdaSheaves" "AgdaStdlib" "DisnixWebService" "LASzip" "MPlayerPlugin"
"OVMF" "PPSSPP" "PatolineEnv" "R" "SDL" "SDL2" "SDL2_gfx" "SDL2_image"
"SDL2_mixer" "SDL2_net" "SDL_gfx" "SDL_image" "SDL_mixer" "SDL_net" "SDL_sound"
"SDL_ttf""TotalParserCombinators" "VisualBoyAdvance" "Xaw3d" "ZopeInterface"
"_915resolution" "a2jmidid" "a52dec" "aacskeys" "aalib" "aangifte2006"
"aangifte2007" "aangifte2008" "aangifte2009" "aangifte2010" "aangifte2011"
"aangifte2012" "aangifte2013" "abc" "abc-verifier" "abcPatchable" "abcde"
"abduco" "abiword" ... ]A total of 4430 packages appear in the output on my system. Let’s hunt for versions of ghc that are available.
nix-repl> attrNames nixpkgs.haskellP<TAB>
nix-repl> builtins.attrNames nixpkgs.haskell
nixpkgs.haskell
nixpkgs.haskellPackages
nixpkgs.haskellPackages_ghc6104
nixpkgs.haskellPackages_ghc6123
nixpkgs.haskellPackages_ghc704
nixpkgs.haskellPackages_ghc722
nixpkgs.haskellPackages_ghc742
nixpkgs.haskellPackages_ghc763
nixpkgs.haskellPackages_ghc783
nixpkgs.haskellPackages_ghc783_no_profiling
nixpkgs.haskellPackages_ghc783_profiling
nixpkgs.haskellPackages_ghcHEAD
nix-repl> exitThis shows all the versions of ghc available including the latest bleeding edge version. The default is currrently 7.63 and ghcHEAD corresponds to an unreleased version of ghc. Many of these also have binary cached versions available, which install much faster and are exactly what you would get if you built them yourself.
You can also hunt specifically for Haskell packages. Let’s see what’s available for ghc 7.6.3.
$ nix-repl
nix-repl> nixpkgs.haskellPackages_ghc763.<TAB>
nixpkgs.haskellPackages_ghc763.ACVector
nixpkgs.haskellPackages_ghc763.AES
nixpkgs.haskellPackages_ghc763.Agda
nixpkgs.haskellPackages_ghc763.Allure
nixpkgs.haskellPackages_ghc763.BNFC
nixpkgs.haskellPackages_ghc763.BlogLiterately
nixpkgs.haskellPackages_ghc763.Boolean
nixpkgs.haskellPackages_ghc763.BoundedChan
nixpkgs.haskellPackages_ghc763.CCdelcont
nixpkgs.haskellPackages_ghc763.Cabal
nixpkgs.haskellPackages_ghc763.Cabal_1_16_0_3
nixpkgs.haskellPackages_ghc763.Cabal_1_18_1_3
nixpkgs.haskellPackages_ghc763.Cabal_1_20_0_2
...
nixpkgs.haskellPackages_ghc763.zippers
nixpkgs.haskellPackages_ghc763.zlib
nixpkgs.haskellPackages_ghc763.zlibBindings
nixpkgs.haskellPackages_ghc763.zlibConduit
nix-repl> exitIf you are using the default version of ghc, currently 7.6.3, you can omit reference to the compiler version and install a package like zippers this way.
nix-env -iA nixpkgs.haskellPackages.zippersYour nix installation already contains a complete copy of all the nix packages under ~/.nix-defexpr/channels/nixpkgs/. This is called a channel, which bundles the ability to build all packages from source with the potential to install prebuilt binaries. These binaries are called a substitute in nix parlance and are a pure win if they are available, since your built-from-source package would be identical.
You can also have your own complete set of nix packages to play with. Just fork clone the repo over at https://github.com/NixOS/nixpkgs.
One reason for forking nixpkgs is that you can update the repo to obtain the very latest versions. It’s perfectly ok to install packages either from your forked repo or your nix channel.
If you want to run different versions of ghc there is another reason to fork. I have a fork stored at /timsears/code/nixpkgs. I can open nix-repl again and point to the fork.
nix-repl> mynixpkgs = import /home/timsears/code/nixpkgs {}
nix-repl> mynixpkgs.haskell
mynixpkgs.haskell
mynixpkgs.haskellPackages
mynixpkgs.haskellPackages_ghc6104
mynixpkgs.haskellPackages_ghc6123
mynixpkgs.haskellPackages_ghc704
mynixpkgs.haskellPackages_ghc722
mynixpkgs.haskellPackages_ghc742
mynixpkgs.haskellPackages_ghc763
mynixpkgs.haskellPackages_ghc783
mynixpkgs.haskellPackages_ghc783_no_profiling
mynixpkgs.haskellPackages_ghc783_profiling
mynixpkgs.haskellPackages_ghcHEADRight now the list is the same as the one from my channel. But a couple of weeks ago, ghc 7.83 was only available through my forked repo.
Either way, this is cool. I would not dare to support so many versions of ghc on my ubuntu dev box.
Everything to do with building packages ends up in /nix/store. When you first poke around in there it is a little bewildering. Do take a few nix pills and learn what’s what. The .drv files are what nix calls derivations. These files relate all the input and output files connected to building a package. All those inputs and outputs are isolated from each other and stored in their own subdirectory under /nix/store. Any dependencies of installed packages are there too, along with dependencies of dependencies, and so forth. You may have several versions of the same package if two downstream packages need different versions. It is probably not too surprising that all the outputs are there, but what about the inputs? Why keep them? Nix keeps references to everything needed to build and reproduce a build exactly. Having nix know the inputs essentially provides the proof that the output will be the same. Nix can even locate binary caches of those builds if available elswhere. That’s the benefit of purity. Nix starts to look very much like a memoized Haskell function.
There at least three “locations” to install packages in nixos: the system, user-profile and per-project.
You can add packages to /etc/configuration.nix as I have done here. This makes them available to all user profiles. There’s probably only one real user on your dev box, but this is still a useful option. For example, you may want to have multiple user profiles to segregate packages that don’t play nice, but you still don’t want to reinstall packages like emacs everywhere. If you prefer not to edit this file you can create a custom set of packages that every user profile can install. I won’t talk about this further here, but I am interested to hear other nix users’ comments on easy ways to manage sets of common packages.
You can also install packages in your default profile nix-env -iA nixpkgs.somePackage and that’s fine. Eventually you will need to manage groups of packages that do not play nice, such as different versions of ghc and their corresponding libraries. Let’s create two new user profiles. In so2013 we will run ghc 7.42 and in the profile letitbleed we will run ghcHEAD which corresponds to an unreleased ghc 7.9.
Nix was designed by folks unafraid of long pathnames, so here’s how you are supposed to do it.
$ nix-env --switch-profile /nix/var/nix/profiles/per-user/timsears/so2013
$ nix-env -q --confirm it's an empty profile
$ nix-env -iA nixpkgs.haskellPackages_ghc742.ghcRun ghc --version to confirm that it worked.
Let’s add an old favorite, hmatrix, to the profile. It’s known to work with ghc 7.4.
$ nix-env -iA nixpkgs.haskellPackages.hmatrixYou should now be able to see something like this.
$ nix-env -q
ghc-7.4.2-wrapper
haskell-hmatrix-ghc7.4.2-0.16.0.6We can now repeat the process to set up our letitbleed profile.
$ nix-env --switch-profile /nix/var/nix/profiles/per-user/timsears/letitbleed
$ nix-env -iA nixpkgs.haskellPackages_ghcHEAD.ghc
$ nix-env -iA nixpkgs.haskellPackages_ghcHEAD.hmatrixThe system responds with
installing `haskell-hmatrix-ghc7.9.20141106-0.16.0.6-shared'
... output omitted ...
Creating package registration file:
/nix/store/d0wccnkgjgxikh1l4hs9c3s4jnyqfzgq-haskell-hscolour-ghc7.9.20141106-1.20.3-shared/lib/ghc-7.9.20141106/package.conf.d/hscolour-1.20.3.conf
ghc-pkg: ghc no longer supports single-file style package databases (/nix/store/d0wccnkgjgxikh1l4hs9c3s4jnyqfzgq-haskell-hscolour-ghc7.9.20141106-1.20.3-shared/lib/ghc-7.9.20141106/package.conf.d/hscolour-1.20.3.installedconf) use 'ghc-pkg init' to create the database with the correct format.
[qbuilder for `/nix/store/vvf2ml647ms6c07885i190ay8b5gznhb-haskell-hscolour-ghc7.9.20141106-1.20.3-shared.drv' failed with exit code 1
cannot build derivation `/nix/store/0b0dgss2dpl0vnfqmvdnbr66n4vzk3g7-haskell-hmatrix-ghc7.9.20141106-0.16.0.6-shared.drv': 1 dependencies couldn't be built
error: build of `/nix/store/0b0dgss2dpl0vnfqmvdnbr66n4vzk3g7-haskell-hmatrix-ghc7.9.20141106-0.16.0.6-shared.drv' failedBoom! Looks like the dependency hscolour does not install, confirming we have indeed arrived on the bleeding edge. We’ll leave that for another day.
When building your own code with nix you can hide your system and user profiles to build code in a more or less clean environment using nix-build and nix-shell. You don’t really install packages, you declare dependencies. I will get into more detail in a later post.
As long as this post turned out to be, it was still too short to cover how to hack on a more realistic project. For that we need to cover cabal2nix and be able to read and modify nix expressions. We will also need to point nix at our own source code and prevent it from doing crazy things like copying our entire git repo into the nix store.