MISC

Manually Installed and Source Compiled

an easier way to source-compile programs and stuff

why?

I run Debian Stable, which is great for the most part. However, there are some programs that I want to have a newer version of than what’s in the Debian repos — and the easiest way to install those is by compiling them myself from source.

However, compiling from source can be complex and fiddly, which makes it ripe for automation! misc is my submission into the wide and varied world of software building software. I use it and not other solutions for the following reasons:

• It’s written in bash and extended in bash, the lingua franca of Unix-ish operating systems. I could’ve done it in POSIX sh, but I really wanted to have arrays.
• Because it’s written in bash, it’s usable anywhere bash is installed. So most places.
• It doesn’t make too many assumptions. It only barely installs dependencies, and really is just a wrapper around the ol’ ./configure, make, make install workflow we all know and love.
• It’s (going to be) extensible to cover uses I haven’t thought of yet.

usage

Basic usage is simply running misc PLAN, where PLAN is a file in $MISC_PLAN_ROOT (default: $XDG_CONFIG_HOME/misc) that details how to install a program. There’s no differentiation between install and update, and there’s currently no method of removing a program (in my experience, uninstalling is more bespoke than installing and would require greater complexity).

Programs are installed under $MISC_INSTALL_PREFIX, which is $HOME/.local by default. I like putting programs under the $HOME/.local prefix for a few reasons:

• They don’t require sudo to install
• $HOME/.local already includes $HOME/.local/share, which is $XDG_DATA_HOME and already includes a bunch of stuff
• It’s pretty easy to add $HOME/.local/bin and the like to $PATH.

Of course, you can change $MISC_INSTALL_PREFIX to whatever you like, either in your shell’s initiation file or in $MISC_CONFIG, or $XDG_CONFIG_HOME/misc/misc.sh by default.

misc.sh

In this section, I’ll discuss all the environment variables that misc itself uses. Plans use a lot of environment variables too, but those are in the following section (Plans).

• $MISC_CONFIG defines the location of misc’s configuration file. This is a shell script that’s sourced before building anything. Default: $XDG_CONFIG_HOME/misc/misc.sh.
• $MISC_REPO_ROOT is the directory that all programs are downloaded or cloned to. Default: $XDG_DATA_HOME/misc.
• $MISC_PLAN_ROOT is the directory where misc will search for plan files. Default: $XDG_CONFIG_HOME/misc. There might be a better place for these but until I can think of one this is where it’ll be.

The following variables are used when processing plans, but they’re misc-specific. You might want to reference these in your plans.

• $MISC_PLAN: The name of the program to build and install or upgrade.
• $MISC_FULL_PLAN: The full path of the plan file; formed from $MISC_PLAN_ROOT/$MISC_PLAN.
• $MISC_REPO: The local directory holding the source of the program being built. Most tasks should be performed within this directory.
• $MISC_INSTALL_PREFIX: The installation PREFIX variable that’s invoked when makeing the program. It can also be used in $CONFIGURE_ARGS and the like, which see.
• $MISC_DONE: A sigil variable indicating when building is done. If this variable is non-empty, misc will detect the build/install as done—otherwise, it will run the default build/install flow. The following functions automatically set $MISC_DONE; of course you can set it yourself in a plan.
  • repo_pull
  • repo_ready
  • make
  • make_install
  • configure

misc invocation

misc takes a number of command-line arguments. What follows is the output you’ll get when you run misc -h.

misc: install programs from source
--- Usage:
    misc -h
    misc [-d] [-n] [-r DIR] [-p DIR] REPO ...
--- Parameters:
    REPO    A program to pull from upstream, build, and install locally.
--- Flags:
    -h      Show this help and exit.
    -d      DRY RUN: Don't do anything, just print what would happen.
    -n      NUKE: Delete REPO, then re-download and build.
--- Options:
    -r DIR  Change the misc repo directory to DIR
              (default: $MISC_REPO_ROOT).
    -p DIR  Change the directory to search for install plans
              (default: $MISC_PLAN_ROOT).
    -i DIR  Change the PREFIX to use when installing programs
              (default: $MISC_INSTALL_PREFIX).

Plans

“Plans” are what misc calls the instruction files to install programs. There are some examples in the examples/ subfolder of this repo.

Plans are simply bash source files defining some variables and optionally specifying build/install steps. There is a default build process that misc will use if it does not detect any build steps being defined in the plan file. Below are the variables and functions in the misc environment at build-time; of course since plans are just bash files, you can define your own or really do anything you want.

Plan variables

• $SOURCE: The source URL of the program’s file or repository.
• $CHECKOUT: The Git checkout to clone. Default: master.
• $CONFIGURE_ARGS: Arguments to pass to the ./configure script.
• $MISC_DEPENDENCIES: Dependencies to install via misc before installing the planned program.
• $APT_DEPENDENCIES: Depenedencies to install via apt before installing the planned program.

Plan functions

There are a number of functions defined in misc that plans inherit into their environments. You can use all of these in plan files, though this is not an exhaustive list. This is, however, all the functions that I think you might find useful in a plan file.

Utility functions

• die CODE MESSAGE...: output MESSAGE to standard error and exit with CODE.
• run COMMAND...: log COMMAND, then execute it if not in dry-run mode.
• cmd COMMAND...: the same as run, but uses command instead of eval.
• with_repo COMMAND...: cd to $MISC_REPO, then execute COMMAND.

Building and installation functions

• install_deps: install the dependencies in $MISC_DEPENDENCIES and $APT_DEPENDENCIES.
• repo_pull [CHECKOUT]: pull the repo from $SOURCE, a git repository. If CHECKOUT is given, use that branch; otherwise use master.
• repo_download_extract: download $SOURCE using curl, then untar it. This expects the source URL to be a .tar.gz file.
• repo_ready [TARGET...]: ready the local source directory by running git -dxf (if applicable) and make clean. If TARGETs are given, use those as the make targets instead of clean.
• repo_nuke: Delete the repository altogether before re-downloading or cloning it.

Plan convenience aliases

Since misc and plan files are simply shell scripts, you could just write the direct commands to download, extract, and install a given program. However, then there’s little reason to use misc — so I’ve included a few thin wrapper functions around common commands that operate in the $MISC_REPO directory and specify a prefix of $MISC_INSTALL_PREFIX.

• make [ARG...], git [ARG...], tar [ARG...]: These run their commands with a -C $MISC_REPO argument for ease of use. If you want to use the raw commands, _make _git, and _tar are defined for those.
• make_and_install [TARGET...]: First runs make with the supplied TARGETs, then runs make install. Note that make is the make defined above, that is, the misc-ified one.
• configure [ARG...]: This runs ./configure in the $MISC_REPO directory. If called with ARGs, it uses those; otherwise it uses the arguments defined in $CONFIGURE_ARGS.

caveats & todos

As you can see from these variables and functions, misc is currently optimized for a ./configure; make; make install-style build system on a Debian-derived host. Again, since a plan file is just a shell script, you can run any commands you like in there—and it’ll still take advantage of misc’s standardized prefix and build root. See the keepassxc example for an example using the cmake system, and lieer for a python-based system.

Of course, it’d be ideal to have these vagaries somewhat abstracted-away by misc, so at some point in the future I’ll include some sort of “add-on” system where a user will be able to define further functions to fit alternative build systems. I’ll also want to figure out a way to generalize dependency installation across distributions, since the current solution (apt install hardcoded into the script) is brittle.

If you have any ideas about these todos or how to solve the problems I’m currently facing, feel free to [email me][mailto:git@acdw.net?subject=misc] with your ideas.

license

misc and all other code in this repository is licensed under the terms of the ISC license. See COPYING for details.