American Fuzzy Lop

Fuzzing Julia used to be impractical due to the significant start-up time of the Julia VM, rendering typical fuzzing approaches inefficient and calling for specialized tools such as TypeFuzz.jl. However, start-up latency has been improving lately, and AFL now has some features to cope with slow binaries.

Quick start

Working with AFL 2.52b, the naive approach of using afl-gcc and afl-g++ to compile Julia by setting the CC and CXX build variables, results in a functioning julia binary. This only instruments those parts of the compiler that have been written in C/C++; code written in Julia is not covered.

To trap interesting errors, I built with LLVM_ASSERTIONS=1 and applied the following patch to at least detect errors in the Julia parts of the compiler:

--- a/julia/src/gf.c
+++ b/julia/src/gf.c
@@ -280,10 +280,11 @@ jl_code_info_t *jl_type_infer(...)
 jl_printf(JL_STDERR, "Internal error: encountered unexpected error in runtime:\n");
 jl_static_show(JL_STDERR, jl_current_exception());
 jl_printf(JL_STDERR, "\n");
 jlbacktrace(); // written to STDERR_FILENO
 linfo_src = NULL;
+abort();

Running the instrumented binary with afl-fuzz crashes the AFL forkserver. This is due to an incompatibility with the LD_BIND_NOW linker option that AFL uses. It can be disabled by specifying LD_BIND_LAZY=1 before running the fuzzer. Furthermore, Julia preallocates quite a bit of memory, and interesting test-cases might take a while due to infer and compile, so I specify -m 2048 to use up to 2G per process and -t 5000 to let them run for up to 5 seconds:

$ LD_BIND_LAZY=1 afl-fuzz -i inputs/ -o outputs/
                          -m 2048 -t 5000 -- julia @@
afl-fuzz 2.52b by <lcamtuf@google.com>
[+] You have 16 CPU cores and 2 runnable tasks (utilization: 12%).
[+] Try parallel jobs - see docs/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #4.
[*] Checking core_pattern...
[*] Checking CPU scaling governor...
[*] Setting up output directories...
[*] Scanning 'inputs'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[+] Deferred forkserver binary detected.
[*] Attempting dry run with 'id:000000,orig:binary_trees.jl'...
[*] Spinning up the fork server...
[+] All right - fork server is up.
    len = 1324, map size = 12396, exec speed = 249839 us
[+] All test cases processed.
[+] All set and ready to roll!

Sadly, performance is pretty low: between 0.2 and 2 executions per second, respectively executions that error (not crash) and succeed.

Let’s try and optimize that.

Only instrument the compiler

A first trivial optimization is to only instrument code that matters. Specifically, compiling dependencies (such as LLVM and OpenBLAS) with the system compiler bumps the speed up a notch:

julia$ make -C deps
julia$ make CC=afl/afl-gcc CXX=afl/afl-g++

Fail hard and fast

Fuzzing code results in lots of invalid sources getting processed by the Julia compiler. Each of these result in a nice exception, which relies on expensive stack unwinding to provide back traces. We can disable this functionality by building with DISABLE_LIBUNWIND=1.

Another costly element of errors is displaying them, as this code is not part of the precompiled system image. This functionality, courtesy of the display_error function in julia/base/client.jl, is easily disabled by preloading a file that redefines the relevant methods:

$ cat fuzz.jl
Base.display_error(er, bt) = println("ERROR")
$ julia -L fuzz.jl -e "error()"
ERROR

The above also applies to deprecation warnings, but those can be disabled more easily by passing --depwarn=no to Julia.

Deferred fork server

Much of the time launching Julia is spent setting-up libraries, initializing code generation, etc. This recurring cost can be avoided by using a deferred fork server, where AFL starts the binary at a manually-specified point during execution. In the case of Julia, this would be right before loading the input file:

--- a/julia/ui/repl.c
+++ b/julia/ui/repl.c
@@ -89,6 +89,10 @@ static NOINLINE int true_main(int argc, char *argv[])
     jl_function_t *start_client = jl_base_module ?
         (jl_function_t*)jl_get_global(jl_base_module, jl_symbol("_start")) : NULL;
 
+#ifdef __AFL_HAVE_MANUAL_CONTROL
+  __AFL_INIT();
+#endif
+
     if (start_client) {

This approach relies on the binary being compiled with AFL’s LLVM instrumenter, available as afl-clang-fast and afl-clang-fast++ for compiling respectively C and C++ code, again specified using the CC and CXX build variables. Note that this instrumentation seems incompatible with OpenBLAS, so if you were to instrument dependencies you would need to compile that library using a regular compiler.

Use of the deferred fork server necessitated several changes. For one, the symbols AFL defines need to be exported globally:

--- a/julia/src/julia.expmap
+++ b/julia/src/julia.expmap
@@ -1,5 +1,6 @@
 {
   global:
+    __afl*;

Furthermore, despite the __afl_manual_init symbol that enables the deferred fork server being part of the Julia binary, AFL does not pick it up and defaults to regular forking. I did not debug this issue, and instead patched AFL to unconditionally enter deferred mode:

--- a/afl/afl-fuzz.c
+++ b/afl/afl-fuzz.c
@@ -6950,7 +6950,7 @@ EXP_ST void check_binary(u8* fname) {
-  if (memmem(f_data, f_len, DEFER_SIG, strlen(DEFER_SIG) + 1)) {
+  if (1) {
 
     OKF(cPIN "Deferred forkserver binary detected.")

You should now see the following in the afl-fuzz output:

[+] Deferred forkserver binary detected.

Note that the AFL documentation suggests putting the __AFL_INIT call before the initialization of important threads, as they don’t survive the fork. The patch above does not do that, as threads are spawned by restore_signals as well as different module initializers. To be safe, I built with JULIA_THREADS=0, configured LLVM with ENABLE_THREADS unset and executed with OPENBLAS_NUM_THREADS=1 to reduce possible issues. Still, your mileage might vary and you might have to move the initialization point further upwards.

Multithreaded fuzzing

With the above fixes, we go from 0.2-2 executions/s to 3-30; still considered slow but a significant improvement nonetheless. Luckily, AFL scales pretty well and can be easily executed across all threads of a system. This brings the final execution speed to around 500 per second.

Now let’s focus on improving fuzzing quality.

Instrumenting Julia code

AFL can more accurately measure the impact of changes to inputs if all relevant code is instrumented. Nowadays, part of the Julia compiler is written in Julia, and compiled during initial bootstrap of the julia binary. Julia code is compiled with LLVM, so we can re-use the LLVM-based instrumentation passes that are part of AFL. However, Julia uses its own (heavily patched) version of LLVM, so we need to recompile AFL and link against Julia’s LLVM library:

afl/llvm_mode$ PATH=julia/usr/tools:$PATH make
# building the tests will fail

In order to use this pass, which is now linked against Julia’s LLVM, we need a compatible build of Clang that can load the instrumentation pass. We can do so by rebuilding Julia’s copy of LLVM with the BUILD_LLVM_CLANG variable set:

julia$ make -C deps BUILD_LLVM_CLANG=1

# we can now run AFL's tests
afl/llvm_mode$ PATH=julia/usr/tools:$PATH make

AFL needs to be modified to expose its instrumentation pass:

--- a/afl/llvm_mode/afl-llvm-pass.so.cc
+++ b/afl/llvm_mode/afl-llvm-pass.so.cc
@@ -177,6 +177,10 @@
+namespace llvm {
+Pass *createAFLCoveragePass() {
+  return new AFLCoverage();
+}
+};

The following changes make Julia use these instrumentation passes:

--- a/julia/src/Makefile
+++ b/julia/src/Makefile
@@ -95,6 +95,8 @@
 endif # USE_LLVM_SHLIB == 1
 endif

+LLVMLINK += afl/afl-llvm-pass.so

--- a/julia/src/jitlayers.cpp
+++ b/julia/src/jitlayers.cpp
@@ -32,6 +32,7 @@
 namespace llvm {
     extern Pass *createLowerSimdLoopPass();
+    extern Pass *createAFLCoveragePass();
 }

@@ -230,6 +231,8 @@ void addOptimizationPasses(...)
+    PM->add(createAFLCoveragePass());
 }

Now build Julia while pointing AFL to the compatible clang binary:

julia$ export AFL_CC=julia/usr/tools/clang
julia$ export AFL_CXX=julia/usr/tools/clang++
julia$ make CC=afl/afl-clang-fast CXX=afl/afl-clang-fast++

For some reason, compiled Julia code uses AFL’s afl_prev_loc TLS variable differently then how it is defined in the contained binary (instead using __emutls_v.__afl_prev_loc). I worked around this issue by changing the definition to a regular variable, since we’re not using threads anyway:

--- a/afl/llvm_mode/afl-llvm-rt.o.c
+++ b/afl/llvm_mode/afl-llvm-rt.o.c
@@ -52,7 +52,7 @@
-__thread u32 __afl_prev_loc;
+u32 __afl_prev_loc;

--- a/afl/llvm_mode/afl-llvm-pass.so.cc
+++ b/afl/llvm_mode/afl-llvm-pass.so.cc
@@ -101,8 +101,7 @@ bool AFLCoverage::runOnModule(Module &M) {
 GlobalVariable *AFLPrevLoc = new GlobalVariable(
-M, Int32Ty, false, GlobalValue::ExternalLinkage, 0, "__afl_prev_loc",
-0, GlobalVariable::GeneralDynamicTLSModel, 0, false);
+M, Int32Ty, false, GlobalValue::ExternalLinkage, 0, "__afl_prev_loc");

Make sure to rebuild all of Julia after applying this patch.

We can now compare the generated code with and without instrumentation:

--- julia-original     -e "@code_llvm sin(1.)"
+++ julia-instrumented -e "@code_llvm sin(1.)"
 L4:                                               ; preds = %top
 ; Location: special/trig.jl:32
 ; Function <; {
 ; Location: float.jl:452
-  %4 = fcmp uge double %2, 0x3E50000000000000
+  %10 = load i32, i32* @__afl_prev_loc
+  %11 = load i8*, i8** @__afl_area_ptr
+  %12 = xor i32 %10, 56933
+  %13 = getelementptr i8, i8* %11, i32 %12
+  %14 = load i8, i8* %13
+  %15 = add i8 %14, 1
+  store i8 %15, i8* %13
+  store i32 28466, i32* @__afl_prev_loc
+  %16 = fcmp uge double %8, 0x3E50000000000000
 ;}

Dictionary for Julia code

A dictionary makes it easier for AFL to explore verbose grammars such as programming languages. I created a dictionary for Julia, with most keywords, important operators, and common syntax such as values and definitions. You can use this dictionary by passing -x julia.dict to afl-fuzz.

Test-case reduction

Although AFL provides a test-case minimizer (afl-tmin), I decided to look into a independent tool. C-Reduce is such a tool, blindly reducing test cases based on a user-specified condition. As such, it is more flexible, and independent from the complicated set-up from above.

Using C-Reduce is fairly simple: provide a script whose return code indicates the state of the current test case, and run it over some files. I’ve created some scripts that improve integration with Julia and its code loading mechanism: maleadt/creduce_julia. For more information, refer to the repository’s README, or have a look at this Discourse thread.

Demo

After running AFL for about a day on a 16-core machine, afl-whatsup reported the following:

$ afl-whatsup -s outputs/
status check tool for afl-fuzz by <lcamtuf@google.com>

Summary stats
=============

       Fuzzers alive : 16
      Total run time : 13 days, 9 hours
         Total execs : 3 million
    Cumulative speed : 42 execs/sec
       Pending paths : 2023 faves, 62710 total
  Pending per fuzzer : 126 faves, 3919 total (on average)
       Crashes found : 1 locally unique

We found a crash! The actual input is written in one of the crashes subdirectories, and contained the following code:

abstract type BTree end

mutable struct Empty <: BTree
end

mutable struct Node <: BTree
  info
  left::BTree
  right::BTree
end

function make(val, d)
  if d == 0
    Node(val, Empty(), Empty())
  else
    nval = val * 2
    Node(val, make(nval-1, d-1), make(nval, d-1))
  end
end

check(t::Empty) = 0
check(t::Node) = t.info + check(t.left) - check(t.right)

function loop_depths(d, min_depth, max_depth)
  for i = 0:div(max_depth - d, 2)
    niter = 1 << (max_depth - d + min_depth)
    c = 0
    for j = 1:niter
      c += check(make(i, d)) + check(make(-i, d))
    end,#@printf("%i\t trees of depth %i\t check: %i\n", 2*niter, d, c)
    d += 2
  end
end

const N=10
min_depth = 4
max_depth = N
stretch_depth = max_depth + 1

let c = check(make(0, stretch_depth))
  #@printf("stretch tree of depth %i\t check: %i\n", stretch_depth, c)
end

long_lived_tree = make(0, max_depth)

loop_depths(min_depth, min_depth, max_depth)

As you may notice, I used the BaseBenchmark’s shootout benchmarks as inputs to AFL. Pasting the reduced code in a Julia REPL indeed crashes the compiler:

Internal error: encountered unexpected error in runtime:
BoundsError(a=Array{Core.Compiler.NewNode, (0,)}[], i=(3,))

signal (4): Illegal instruction

To reduce this code, I check-out maleadt/creduce_julia and use the crashing code as the initial input. We need to make sure that the run script successfully catches the error condition; in this case it is sufficient to grep for encountered unexpected error in runtime:

--- a/creduce_julia/run
+++ b/creduce_julia/run
@@ -30,4 +30,4 @@ JULIA_LOAD_PATH=$TEMP \
-julia main.jl |& grep "example error message"
+julia main.jl |& grep "encountered unexpected error in runtime"

creduce_julia$ ./run
Internal error: encountered unexpected error in runtime:

creduce_julia$ echo $?
0

Finally, we run the reduce script to kick-off the process. It will automatically work in parallel on all available cores:

creduce_julia$ ./reduce
===< 30926 >===
running 8 interestingness tests in parallel
...
===================== done ====================

pass statistics:
  method pass_balanced :: parens-inside worked 1 times and failed 2 times
  method pass_indent :: final worked 1 times and failed 0 times
  method pass_clex :: rm-toks-3 worked 1 times and failed 61 times
  method pass_lines :: 4 worked 2 times and failed 50 times
  method pass_lines :: 1 worked 2 times and failed 50 times
  method pass_clex :: rename-toks worked 2 times and failed 3 times
  method pass_blank :: 0 worked 2 times and failed 0 times
  method pass_indent :: regular worked 2 times and failed 0 times
  method pass_lines :: 8 worked 2 times and failed 50 times
  method pass_lines :: 10 worked 2 times and failed 50 times
  method pass_lines :: 6 worked 2 times and failed 50 times
  method pass_lines :: 3 worked 2 times and failed 50 times
  method pass_clex :: rm-toks-2 worked 2 times and failed 64 times
  method pass_lines :: 2 worked 2 times and failed 50 times
  method pass_balanced :: parens-to-zero worked 2 times and failed 6 times
  method pass_clex :: rm-toks-4 worked 3 times and failed 49 times
  method pass_clex :: rm-toks-13 worked 3 times and failed 18 times
  method pass_lines :: 0 worked 15 times and failed 78 times
  method pass_clex :: rm-toks-1 worked 16 times and failed 68 times

The above took about 2 minutes on an 4-core workstation. Doing some manual clean-up, we end up with the issue as reported in JuliaLang/julia#30062:

for a = 1 end, b += 2

Future work

Although a tool like AFL isn’t ideal for fuzzing a compiler as done in this blogpost, the technique and its set-up can be used to fuzz other libraries and tools that are written in Julia (eg. HTTP.jl or Images.jl). That would of course require productionisation of this proof-of-concept and its many workarounds.

A hypothetical AFL.jl package would:

• build AFL’s instrumentation pass against an LLVMBuilder toolchain
• provide native functionality a la python-afl
• conditionally instrument code (Julia has some great tools for that)

Fuzzing the compiler itself could also be improved:

• deferring the fork server even more: there’s significant initialization done by Julia code, and the manual initialization point could move right before the input eval loop
• persistent fuzzing: having AFL input new test cases without restarting the process. Although Julia is far from stateless, AFL could be modified to verify crashes in a fresh process.

Here’s the material from those talks:

• CUDAnative.jl talk (high-level): slides
• CUDAnative.jl workshop (hands-on): notebook (HTML, slides)
• LLVM.jl lightning talk: notebook (HTML, slides)

In this post I’ll explain how to use Clang’s address sanitizer (or ASAN) with Julia. This is somewhat tricky, as the Julia compiler uses LLVM for code generation purposes. Long story short, this implies that all instances of LLVM (ie. the one Julia is compiled with, and the one used for code generation) have to match up exactly for the instrumentation to work as expected.

LLVM toolchain

We’ll start by building a toolchain to compile Julia with. As mentioned before, all LLVM instances in play have to match up exactly for instrumentation to work, so we’ll use Julia’s build infrastructure to generate us an LLVM toolchain.

Start by checking-out Julia, and creating an out-of-tree build directory:

$ git clone https://github.com/JuliaLang/julia
$ cd julia
$ make O=configure sanitize_toolchain

This build will need to provide clang, so create a Make.user containing BUILD_LLVM_CLANG=1. In addition, LLVM does not build its sanitizers with autotools, so add override LLVM_USE_CMAKE=1 to that file as well. And because that triggers LLVM bug #23649, also add USE_LLVM_SHLIB=0

Now execute make install-llvm from the deps subfolder. When it finishes, check if binaries have been written to usr/bin (due to what’s probably a bug in LLVM’s build scripts), and move them to usr/tools if they have.

Sanitized Julia

Now that we have a working toolchain, we’ll use it to compile a sanitized version of the Julia compiler and libraries. Start by creating a new out-of-tree build directory using make O=configure sanitize. But this time, our Make.user will be significantly more complex:

TOOLCHAIN=$(BUILDROOT)/sanitize_toolchain/usr/tools

# use our new toolchain
USECLANG=1
override CC=$(TOOLCHAIN)/clang
override CXX=$(TOOLCHAIN)/clang++
export ASAN_SYMBOLIZER_PATH=$(TOOLCHAIN)/llvm-symbolizer

# enable ASAN
override SANITIZE=1
override LLVM_SANITIZE=1

# autotools doesn't have a self-sanitize mode
override LLVM_USE_CMAKE=1

# make the GC use regular malloc/frees, which are intercepted by ASAN
override WITH_GC_DEBUG_ENV=1

# default to a debug build for better line number reporting
override JULIA_BUILD_MODE=debug

Now kick-off the build using make from the sanitize build directory. Barring any memory issues triggered during system image generation, this should yield a sanitized julia binary and system image.

Running the test-suite

The test-suite is a beast, and because ASAN keeps track of a lot of information it easily takes over 128GiB of memory to run it to completion. Instead, we’ll tune ASAN to consume less memory at the expense of accuracy and report detail.

Julia however already configures default ASAN options, which we need to copy when specifying a different set. Do so by defining the ASAN_OPTIONS environment variable and assigning it the value of detect_leaks=0:allow_user_segv_handler=1:fast_unwind_on_malloc=0:malloc_context_size=2. This copies aforementioned default values, and caps backtrace collection.

Using CUDA packages

If you thought all that was convoluted, prepare for some more. ASAN uses so-called shadow memory to store information about memory allocations. There is a correspondence between regular memory addresses and their shadow counterpart, and this mapping is fixed in order to keep the instrumentation overhead low. Sadly, the default shadow memory location overlaps with fixed memory allocated by CUDA (presumably for its unified virtual address space).

Because the shadow memory is fixed, we need to patch both instances of LLVM (easiest to add a patch to llvm.mk) and have it pick a different shadow offset:

--- lib/Transforms/Instrumentation/AddressSanitizer.cpp
+++ lib/Transforms/Instrumentation/AddressSanitizer.cpp
@@ -359,7 +359,7 @@
       if (IsKasan)
         Mapping.Offset = kLinuxKasan_ShadowOffset64;
       else
-        Mapping.Offset = kSmallX86_64ShadowOffset;
+        Mapping.Offset = kDefaultShadowOffset64;
     } else if (IsMIPS64)
       Mapping.Offset = kMIPS64_ShadowOffset64;
     else if (IsAArch64)
--- projects/compiler-rt/lib/asan/asan_mapping.h
+++ projects/compiler-rt/lib/asan/asan_mapping.h
@@ -146,7 +146,7 @@
 #  elif SANITIZER_IOS
 #    define SHADOW_OFFSET kIosShadowOffset64
 #  else
-#   define SHADOW_OFFSET kDefaultShort64bitShadowOffset
+#   define SHADOW_OFFSET kDefaultShadowOffset64
 #  endif
 # endif
 #endif

Note that you might need to redefine a different macro for your platform.

Sanitizing older versions of Julia

If you want to sanitize older versions of Julia, before the switch to LLVM 3.9, there’s yet other issues: only LLVM 3.9 is compatible with recent versions of glibc, while the CMake build system of LLVM 3.7 doesn’t export all necessary public symbols. You can work around these issues by using a sufficiently old system, and overriding the LLVM version to 3.8 (by specifying override LLVM_VER=3.8.1 in the Make.user of both build directories) or preventing it from generating a shared library (by specifying USE_LLVM_SHLIB=0 in the Make.user of the final Julia build).

bmaptool create

Although bmaptool features a create option, that mode of operation is mostly useless as it relies on the file sparseness to extract the unused blocks. However, sparseness has completely different semantics! Quoting Wikipedia:

When reading sparse files, the file system transparently converts metadata representing empty blocks into “real” blocks filled with zero bytes at runtime.

When creating a bmap file based on image sparseness and subsequently writing that image to a device, the previously sparse blocks will be skipped, resulting in non-zero bytes when reading from those blocks.

bmaptool-scan

I’ve written a small script which scans for the actual unused blocks, i.e., based on the underlying file system instead of the image sparseness. These blocks are truly unused, which means we can safely skip writing them when copying an image to a device.

Usage

# modprobe loop
# bmaptool-scan --bmap something.img | tee something.bmap
Found 1 partition(s) in image
- processing 1.7G partition at 1.0M into the image

<?xml version="1.0"?>
<bmap version="2.0">
 ...
</bmap>

There’s two program modes available: --bmap and --sparse. The former generates a bmap file, as can be seen above. The --sparse option punches holes in the input image. This can subsequently be used to generate a bmap file using bmaptool create, or just to optimize storage of the image.

Note that you need to run the script as root, and you need quite some Perl packages. See the README for more details.

Limitations

As the script relies on parsing the image partition table and probing any partition found using file system specific tools, there are a few limitations:

• Implementation wise: adding more file systems. Currently only ext4 is supported, which should be trivial to extend to the entire ext family. Adding support for other file systems means adding code to dump and parse the unallocated blocks within the partition.
• Nonstandard partition layouts (hybrid images, overlapping partitions, …): these might break the script, or might even be unsafe to write selectively. Use at your own risk!

I started from a nice-looking example I found online, which features an inexpensive aluminum enclosure from eBay and a compact switching power supply to get rid of the external charger.

I used the same aluminum case and power supply as the aforementioned project, but rather than using cables to connect the Pi’s I/O I milled a piece of MDF¹ to neatly expose the original connectors, together with the audio jacks and a reset switch:

The reset switch and power rocker are wired to a Mausberry shutdown circuit. This nifty small device allows for a clean shut-down, by sending a shutdown signal (over GPIO) to the Pi when the power button is flipped. A user-space daemon monitors the GPIO pin, and issues a halt command when requested. Finally, when the Pi has shut down the Mausberry circuit cuts the power, indicated by a LED connected to the Pi’s VCC.

Because the case is fully closed (ie. no ventilation holes) I added a DS1820 temperature sensor, wired to one of the remaining GPIO ports. Under normal load the temperature remains at a comfortable 30°C, and even when stressing the system it stays below 35°C.

As a final tweak, I added a 1mF capacitor to the output of the power supply. This turned out to be necessary because of some bouncy switches in my house making the power supply fluctuate by almost 2V, which in turn caused the Pi to reboot. A higher-quality PSU would probably not have had this issue…

Bill of Materials

Software

I initially looked at ready-to-use distributions such as Volumio (or the related RuneAudio), but both are just too damn messy behind the scenes and did not turn out to be reliable enough for general use. Instead, I decided to configure a generic Raspbian installation. Having wasted enough time in the past on getting rid of the useless packages provided by the official Raspbian, I bootstrapped a minimal version using the unattended netinstaller. The end result is a minimal Raspbian-based jessie which boots in 5 seconds (2s kernel + 3s userspace).

The audio-part is taken care of by Mopidy, a heavyweight but flexible Python-based mpd-like audio server². To make it all user-friendly, I’m currently using the Mopify web client linked to a premium Spotify account.

Some random notes to close with:

• The HifiBerry DAC+ isn’t detected on the current 3.18.0 kernel in Raspbian (the HifiBerry test images use 3.18.7), so for the time being you’ll need to run the horror that is rpi-update.
• There seems to be a mismatch in volume scale handling between some components used by Mopidy’s alsamixer, so you’ll need to use a software mixer to get useful volume control.
• The rpi-updated kernel seems to boot significantly faster than its Raspbian counterpart (presumably because it doesn’t use an initrd).

For this exploit, I am assuming local user access where you can drop a file and change environment flags. This is a hefty requirement; many people would argue local user access means game over already.

That said, the procedure described below is an interesting way of getting to root, and is undocumented as far as I could find. It also works on all recent distributions I’ve tested, regardless of security features such as SELinux.

But first, some context. Skip this if you already know about LD_PRELOAD and how it can’t be used for setuid binaries like sudo.

Context

The famous LD_PRELOAD variable is a powerful yet dangerous way of influencing how dynamic libraries are loaded on most Linux systems. It instructs the dynamic linker to load certain libraries before all others, which makes it possible to override or hook functions. Although many benign use cases exist, the feature can also be misused: imagine some library hooking the C runtime’s read function and spying on everything you type.

To prevent such shenanigans affecting your entire system, the LD_PRELOAD environment flag is mostly ignored when executing binaries with the setuid or setgid bits set. These bits allow an application to change its user identity to whoever owns the binary being executed. For example:

int main()
{
  if (setuid(0))
    perror("setuid failed");
  else
    system("/usr/bin/whoami");

  return 0;
}

If we compile and run this, the setuid call to switch our user identity to user 0 (root) is obviously blocked off. However, if we make root own this binary and toggle the setuid bit, all is allowed:

# chown root test
# chmod 4755 test
$ ./test
root

This mechanism is what powers applications such as sudo. Needless to say, it would be bad if an attacker could influence the behaviour of said applications through means of LD_PRELOAD, which is why these and other environment variables are ignored in such circumstances.

Abusing the cache

Not to have you enter your password over and over again, sudo comes with a credential cache which remembers that you’ve entered your credentials already. Quoting the manual:

Once a user has been authenticated, a record is written containing the uid that was used to authenticate, the terminal session ID, and a time stamp. The user may then use sudo without a password for a short period of time.

By saving the terminal session ID, an attacker cannot hijack your authenticated sudo session from another terminal. That means if we want to abuse an unlocked cache, we’d need to do it from the very same terminal where sudo is being used.

Waiting for a signal

Although we cannot use LD_PRELOAD for hijacking library calls within sudo (as it is a setuid binary), we can hook onto a user-level call and check whether the cache is unlocked. When the user calls an application in which we’ve hooked a certain library call, and the user has recently used (and unlocked) sudo within the same terminal session, we can assume control:

$ application
  ├ preload hooked.so
  └ call foobar()
    └ hooked::foobar()
      ├ is sudo unlocked? nope...
      └ call real::foobar()

$ sudo application
  ├ unlocks the credential cache
  ├ call setuid
  └ exec application

$ application
  ├ preload hooked.so
  └ call foobar()
    └ hooked::foobar()
      ├ is sudo unlocked? yes!
      └ game over

This completely circumvents the existing LD_PRELOAD protections, because it never attempts to preload when executing a setuid binary, but only targets regular applications executed afterwards.

Implementation

Ideally, checking the credential cache would happen quickly after spawning a new application. For the demonstration below, I’ve picked the open call in libc. Using dlsym we look up the address of the original function, and wrap the call before returning:

static int (*open_real)(const char *, int, ...) = NULL;

int open(const char *path, int flags, ...) {
  // Load the real 'open' function
  if (!open_real)
    open_real = (int (*)(const char *, int, ...))
                dlsym(RTLD_NEXT, "open");

  // TODO: check if sudo is unlocked

  // Wrap call
  va_list ap;
  mode_t mode;
  va_start(ap, flags);
  mode = va_arg(ap, mode_t);
  va_end(ap);
  return open_real(path, flags, mode);
}

Now for the crucial part: check whether sudo is unlocked. This can be accomplished by running sudo -n true, which indicates failure when a password is required. I’ve chosen to implement this using a quick fork and exec, while the main thread waits for its child to finish:

pid_t child = fork();
if (child == 0) {
  int null = open("/dev/null", O_WRONLY);
  dup2(null, STDERR_FILENO);

  execlp("sudo", "sudo", "-n", "true", NULL);

  exit(-1);
} else {
  int exit_code;
  if (wait_for(child, 10 /*ms*/, &exit_code) == 0) {
    if (exit_code == 0) {
      printf("I am ");
      fflush(stdout);
      execlp("sudo", "sudo", "whoami", NULL);
    }
  }
}

Note that these code fragments are for demonstrative purposes only, and lack crucial bits of functionality in order to work.

What this means

As said before, this exploits requires local user access, at which point many other options exist¹. Still, I think this post describes an interesting path to root, in particular because it targets a design ‘decision’ rather than an ordinary vulnerability.

Because of this, and the fact that LD_PRELOAD is particularly hard to detect (just have a look @haxelion’s recent blogpost), this privilege escalation seems hard to mitigate. The safest path forward would be to either disable LD_PRELOAD or the caching mechanism in sudo, but that might break certain use-cases.

In the end, it all is a matter of securing your local accounts, because if they get breached by a determined hacker it is just a matter of time before your entire system is compromised. This vulnerability might just give him an extra weapon of choice.

The most common problem with tracking BitTorrent connections is the high volume of them, causing internal state tables to overflow. I bumped into another issue, where certain incoming RST-flagged packets got blocked, much like how this forum post details. This thoroughly messed with my torrent client, which got stuck with many closed connections thinking they were still valid.

I ultimately decided to get rid of connection tracking for BitTorrent traffic altogether. By marking packages NOTRACK in the respective raw tables, we instruct iptables not to track these packages. However, this also means losing some helpful functionality. conntrack and NAT helper will simply not work, and we’ll need to handle ICMP errors ourselves. Luckily, BitTorrent traffic is not state-full (as opposed to, for example, FTP), so we don’t need any helpers.

The following snippet lists the required iptables commands (usable with ip6tables too):

# BitTorrent "server"
SERVER="40000"
for PROTO in tcp udp; do
  # incoming traffic
  iptables -t raw -I PREROUTING -p $PROTO \
           --dport ${SERVER} -j NOTRACK
  iptables -I INPUT -i eth0 -p $PROTO \
           --dport ${SERVER} -j ACCEPT

  # outgoing traffic (replies)
  iptables -t raw -I OUTPUT -p $PROTO \
           --sport ${SERVER} -j NOTRACK
  iptables -I OUTPUT -p $PROTO \
           --sport ${SERVER} -j ACCEPT                               
done

# BitTorrent "client"
CLIENT="40001:40999"
for PROTO in tcp udp; do
  # outgoing traffic
  iptables -t raw -I OUTPUT -p $PROTO \
           --sport ${CLIENT} -j NOTRACK
  iptables -I OUTPUT -p $PROTO \
           --sport ${CLIENT} -j ACCEPT                               

  # incoming traffic (replies)
  iptables -t raw -I PREROUTING -p $PROTO \
           --dport ${CLIENT} -j NOTRACK
  iptables -I INPUT -i eth0 -p $PROTO \
           --dport ${CLIENT} -j ACCEPT
done

Recall that due to marking all packages NOTRACK, iptables will not be able to tie ICMP replies to the corresponding connection, and as a result treat those packages as invalid. To resolve this, we should explicitly allow such control messages:

for TYPE in destination-unreachable source-quench \
            ttl-exceeded parameter-problem; do
  iptables -I INPUT -p icmp \
           --icmp-type $TYPE -j ACCEPT
  iptables -I OUTPUT -p icmp \
           --icmp-type $TYPE -j ACCEPT
done
for TYPE in destination-unreachable packet-too-big \
            ttl-exceeded parameter-problem; do
  ip6tables -I INPUT -p icmpv6 \
            --icmpv6-type $TYPE -j ACCEPT
  ip6tables -I OUTPUT -p icmpv6 \
            --icmpv6-type $TYPE -j ACCEPT
done

If required, you could still tighten these rules up a bit. Some ideas:

• restrict outgoing traffic with --uid-owner
• limit the allowed ICMP datagrams based on the original IP payload (for instance, using the u32 extension)

Hardware

Opening up the clicker, the main chip appears to be a Nordic Semiconductor NRF 24LE1E. Some details about that chip:

• nRF24LE1 SoC
• 32-pin 5x5 QFN package
• Single package MCU, RF unit & flash memory

That last characteristic is already a big difference with the RCRF-01 model, where the flash memory was stored off-package on an SPI-addressable EEPROM.

Wiring

The datasheet¹ of our SoC reveals an SPI programming interface. If we pull the PROG pin high and pulse RESET, certain pins get reconfigured as an SPI slave².

Conveniently, most of these pins are exposed through large contacts accessible via the battery compartment. Mapping them using a continuity tester, we can find out how they are connected, and connect it to some SPI-compatible tool such as the Bus Pirate:

Only the RESET pin is not exposed (it is hard-wired to VCC using a 4.7kΩ resistor), but as it turned out you don’t really have to pulse this pin as detailed in the datasheet; rebooting the device while PROG is active suffices.

Connecting the PROG pin directly to the power supply permanently pulls it high; this is fine for programming purposes.

As the SoC works between 1.9V and 3.6V (by default the clicker is equipped with 3V CR2032 batteries), and has a low power consumption, I wired it directly to the Bus Pirate’s integrated power supply (remember to enable that peripheral).

Lastly, I also connected the blue AUX connector to the RESET pin of the MCU so I could accurately pulse it, but as explained before that wasn’t really necessary.

Software

SPI is a simple interface, but allows for different modes selecting clock polarity and phase. By default, the nRF24LE1’s SPI slave starts in mode 0³. I used the following settings for the Bus Pirate:

• speed: 125 KHz
• clock polarity: idle low
• output clock edge: active to idle
• input sample phase: middle
• chip select: active low
• output type: normal (high bit = 3.3V)

As an exercise to get to know the Bus Pirate better, I decided to use the binary interface and script everything up. However, as soon as I got some code running, I discovered that the RDISMB (Read DISable Main Block) bit in the Flash Status Register is set. As the name suggests, this prevents reading from main memory through an external interface such as SPI:

Disable flash main block access from external interfaces:

• 0xFF: Flash main block accessible from external interfaces

• Other value: No read/erase/write of flash main block from external interfaces. Only read of info page

Can only be changed once by SPI command RDISMB. Can only be reset by SPI command ERASE ALL

And the relevant entry at address 0x23 in the Info Page:

0x00000000 (00000)   005a4330 54383231 02173e4f 775f6092
0x00000010 (00016)   26b14f0d 0affffff ff82ffff ffff0000
0x00000020 (00032)   ffffff00 ffffffff ffffffff ffffffff

I haven’t found a way around this protective bit. It is obviously impossible to change its setting in the FSR, just as writing the underlying Info Page entry doesn’t do anything. The only apparent way to clear this bit is to issue the ERASE ALL command, but that kinda defeats the purpose…

As a last resort, I tried passively sniffing the SPI bus when executing the code, but the internal SPI bus doesn’t seem to be exposed when operating normally.

Wrapping up

In conclusion, it seems like Nordic has upped their game and protected their code better. Circumventing the RDISMB restriction doesn’t seem very easy. For example, Bunnie’s UV hack wouldn’t apply because the info page is stored in the main flash. Racing to access memory before the FSR is initialized can’t work because RDISMB is enabled by default.

A possible path forward would be to gain access to the firmware upgrade code path, if that even exists (I assume so because of the exposed SPI connectors). Or trigger unintended behaviour by altering values in the writeable part of the Info Page. A more far fetched alternative is to try to glitch the processor when issuing an ERASE ALL command. But that’s taking it to a whole new level…

I’ve put the code I’ve developed on Github. It’s a comprehensive example of how to use the Bus Pirate’s binary SPI mode, and how to contact the nRF24LE1. Although it’s a non-modular piece of code, it is pretty well documented and easy to read (even though it’s written in Perl – don’t let that scare you!).

Note from 2018: Much has happened since 2015, and as a result this blog post has become pretty stale. Check-out CUDAnative.jl for more details about the current state of affairs.

My work allows for code such as:

using CUDA

# define a kernel
@target ptx function kernel_vadd(a, b, c)
    i = blockId_x() + (threadId_x()-1) * numBlocks_x()
    c[i] = a[i] + b[i]

    return nothing
end

# set-up
dev = CuDevice(0)
ctx = CuContext(dev)
cgctx = CuCodegenContext(ctx, dev)

# create some data
dims = (3, 4)
a = round(rand(Float32, dims) * 100)
b = round(rand(Float32, dims) * 100)
c = Array(Float32, dims)

# execute!
len = prod(dims)
@cuda (len, 1) kernel_vadd(CuIn(a), CuIn(b), CuOut(c))

# verify
@show a+b == c

# tear-down
destroy(cgctx)
destroy(ctx)

which is pretty neat I think :-)

I’ll start by giving a quick description of the modifications. Jump to the bottom of this post for usage instructions.

Overview

Compiling Julia for GPUs requires support at multiple levels. I’ve tried to avoid touching too much of core compiler; as a consequence most functionality is part of the CUDA.jl package. This should make it easier to maintain and eventually merge the code.

All of the relevant repositories are hosted at my Github page, and contain README and TODO files. If you have any questions though, feel free to contact me.

Julia compiler

Using the NVPTX back-end of LLVM, I have modified the Julia compiler so that it can generate PTX assembly. A non-exhaustive list of modifications:

• @target macro for annotating functions with target information
• per-target compiler state (module, pass manager, etc)
• diverse small changes to generate suitable IR
• exported functions for accessing the PTX code

Most of the code churn comes from using an address-preserving bitcast, which is already being upstreamed thanks to Valentin Churavy.

CUDA.jl support package

Generating PTX assembly is only one part of the puzzle: hardware needs to be configured, code needs to be uploaded, etc. This functionality is exposed through the CUDA runtime driver, which already was conveniently wrapped in the CUDA.jl package.

I have extended this package with functionality required for GPU code generation, and developed user-friendly wrappers which should make it easier to interact with PTX code:

• @cuda macro for invoking GPU kernels
• automatic argument management
• lightweight on-device arrays
• improved API consistency
• many new features

The significant part is obviously the @cuda macro, allowing for seamless execution of kernel functions on your GPU. The macro compiles the kernel you’re calling to PTX assembly, and generates code for interacting with the driver (creating a module, uploading code, managing arguments, etc).

The argument management is also pretty interesting. In function of the argument type, it generates type conversions and/or memory operations in order to mimic Julia’s pass-by-sharing convention. For example, if you pass an array to a kernel, @cuda will automatically up- and download it when required¹.

Most functionality of @cuda is built using staged functions, and thus only executes once without a recurring runtime cost. This means that it should be possible to reach the same average performance of a traditional, precompiled CUDA application :-)

GPU Ocelot emulator

I have also forked the GPU Ocelot project, which is a research project providing a dynamic compilation framework (read: emulator) for CUDA hardware. By extending API support calls and fixing certain bugs, you can use this as a drop-in replacement for libcuda.so, fully compatible with CUDA.jl.

In practice, I used this emulator for everyday development on a system without an NVIDIA GPU, while testing happened on real hardware.

Limitations

The code is far from production ready: it is not cross-platform (Linux only), several changes should be discussed with upstream, and only a restricted subset of the language is supported. Most notable shortcomings:

• cannot alloc in PTX mode (which breaks most of the language)
• can only pass bitstypes, arrays or pointers
• standard library functionality is unavailable, because it lives in another LLVM module

In short: unless you’re only using relatively simple kernels with non-complex data interactions, this code is not yet usable for you.

Usage

Even though all code is pretty functional and well-maintained, you need some basic development skills to put the pieces together. Don’t expect a polished product!

Julia compiler

Compile the modified compiler from source, using LLVM 3.5:

$ git clone https://github.com/maleadt/julia.git
$ cd julia
$ make LLVM_VER=3.5.0

Optionally, make sure Julia is not broken (this does not include GPU tests):

$ make LLVM_VER=3.5.0 testall

Note: the compiler will require libdevice to link kernel binaries. This library is only part of recent CUDA toolkits (version 5.5 or greater). If you use an older CUDA release, you will need to get a hold of these files. Afterwards, you can point Julia to them using the NVVMIR_LIBRARY_DIR environment variable.

GPU Ocelot emulator

If you don’t have any suitable CUDA hardware, you can use GPU Ocelot:

$ git clone --recursive https://github.com/maleadt/gpuocelot.git
$ cd gpuocelot
$ $PAGER README.md
$ CUDA_BIN_PATH=/opt/cuda-5.0/bin \
  CUDA_LIB_PATH=/opt/cuda-5.0/lib \
  CUDA_INC_PATH=/opt/cuda-5.0/include \
  python2 build.py --install -p $(realpath ../julia/usr)

Note: this probably will not build flawlessly. You’ll need at least the CUDA toolkit² (headers and tools, not the driver), gcc 4.6, scons, LLVM 3.5 and Boost. Check the README!

Now if you load CUDA.jl and it doesn’t find libcuda.so, it will look for libocelot.so instead:

$ ./julia
  > using CUDA
  > CUDA_VENDOR
    "Ocelot"

CUDA.jl support package

Installing packages is easy³ (just make sure you use the correct julia binary):

$ ./julia
  > Pkg.clone("https://github.com/maleadt/CUDA.jl.git")

Optionally, but recommended, test GPU support:

$ ./julia
  > Pkg.test("CUDA")

What now?

You tell me! I think this work can be a good start for future GPU endeavours in Julia land, even without most code being directly re-usable. For me at least it has been a very interesting project, but it’s in the hands of the community now.

In search for a good protection measure, I decided to avoid exposure of critical services through a layer of port knocking. Traditionally, one relies on special network traffic (sequence of connection attempts, specific packet with encrypted payload, …) to temporarily open up additional ports. However, generating such traffic is not user friendly (ie. requires some tool or script) and it can be hard getting it through strict firewalls.

I wanted to knock ports using some protocol or traffic, which is:

• Secure: it should be hard to discover or sniff the knocking sequence;
• User-friendly: not relying on auxiliary tools;
• Firewall-friendly.

My solution to this is a simple HTML/PHP form, served over password- protected HTTPS. No firewall would block such traffic, and it is both extremely user-friendly and very secure.

Implementation

The bottom part of the system is the firewall, allowing for certain ports to be opened upon an external signal. I’m using FireHOL, a lightweight iptables-based firewall with a user-friendly syntax:

interface eth0 internet not "${UNROUTABLE_IPS}"
    policy drop
    protection strong 10/sec 10

    server ssh accept with knock admin

The important part here is with knock $NAME, which will build a special chain called knock_$NAME. Adding rules to this chain will allow access to the ssh service, deleting them will revoke access (keeping established connections intact).

Next up is some HTML form allowing the user to select a set of services to unlock. The forms also provides an IP field, which can be useful when unlocking from a different address (for example, your phone).

<form action="<?=htmlentities($_SERVER['PHP_SELF']); ?>"
      method="post">
  <table>
    <tr>
      <td><label for='service[]'>Service:</label></td>
      <td><select name="service">
        <option value="admin">administrative</option>
      </select></td>
    </tr>

    <tr>
      <td><label for='source'>Source IP:</label></td>
      <td><input type="text" name="source"
                 value="<?=(isset($_POST['source'])
                            ? $_POST['source']
                            : $_SERVER['REMOTE_ADDR'])?>">
      </td>
    </tr>

    <tr>
      <td><label for='duration'>Duration:</label></td>
      <td><input type="text" name="duration"
                 value="<?=(isset($_POST['duration'])
                            ? $_POST['duration']
                            : 15)?>">
      </td>
    </tr>

    <tr>
      <td></td>
      <td><input type="submit" name="knock"
                 value="Knock"></td>
    </td>
  </table>
</form>

In my browser, this form looks like this: {% image php-port-knocker/form.png alt:“Screenshot of form” title:“State-of-the-art UI design” %}

Within the same PHP file, we process the data submitted by the user:

<?php
if (isset($_POST['knock'])) {
  // Build command line
  $error = NULL;
  $command = ["/usr/bin/sudo", "/usr/local/sbin/knock"];
  if (!empty($_POST['source']))
    $command = 
      array_merge($command,
                  ["-s", escapeshellarg($_POST['source'])]);
  if (!empty($_POST['duration']))
    $command =
      array_merge($command,
                  ["-d", escapeshellarg($_POST['duration'])]);
  if (!empty($_POST['service']))
    array_push($command,
               escapeshellarg($_POST['service']));
  else
    $error = "Missing service...";

  // Perform call
  if (!$error) {
    $output = [];
    $rv = 0;
    exec(join(" ", $command), $output, $rv);
    if ($rv != 0)
      $error = join('', $output);
  }

  if ($error)
    echo("<p>Error: " . join('', $output) . ".</p>");
  else
    echo("<p>Service knock completed successfully.</p>");
}
?>

Even though the form is only privately accessible, password protected and served strictly over HTTPS, we still take some basic security precautions by checking and escaping all user inputs.

As another layer of security, I do not allow the PHP script to invoke iptables directly. Instead, I have written an auxiliary knock Perl script which takes care of adding and removing rules to iptables:

use strict;
use warnings;

use POSIX;
use Getopt::Std;

# Parse user-input
my $usage = "Usage: $0 [-s SOURCE] [-d DELAY] SERVICE\n";
my %options;
getopts("s:d:",\%options) || die $usage;
my $service = shift || die "Missing service.\n$usage";
die "Invalid service.\n" unless ($service =~ m/^[a-zA-Z]+$/);
my $delay = $options{d} || 15;
die "Invalid delay.\n" unless ($delay =~ m/^[0-9]+$/);

my @flags;
if (defined $options{s}) {
        push(@flags, ("-s", $options{s}));
}
push(@flags, ("-j", "ACCEPT"));

defined (my $kid = fork) or die "Cannot fork: $!\n";
my $mode;
if ($kid) {
    $mode = "-A";
} else {
    daemonize();
    sleep($delay);
    $mode = "-D";
}

system("/sbin/iptables", $mode, "knock_$service", @flags);
exit 0;

sub daemonize {
    POSIX::setsid or die "setsid: $!";
    chdir "/";
    umask 0;
    foreach (0 .. (POSIX::sysconf (&POSIX::_SC_OPEN_MAX) || 1024)) {
        POSIX::close $_
    }
    open(STDIN, "</dev/null");
    open(STDOUT, ">/dev/null");
    open(STDERR, ">&STDOUT");
}

By using such a script, a hypothetical malicious user can only control a limited subset of iptables commands, restricted to knock_* chains. Note that we use the list-form of Perl’s system() to prevent command injection, and take care of sanitizing user input yet again.

Lastly, the webserver needs to be able to call this script with elevated permissions. Adding www-data to the sudo group (or equivalent) would not be wise, as that would allow an attacker to gain root through the webserver user. Instead, we still use sudo, but put a very specific rule in /etc/sudoers.d/knock allowing the webserver to use sudo for our script but nothing else:

www-data    ALL=NOPASSWD: /usr/local/sbin/knock

Conclusion

Port knocking is a powerful tool for hiding services, and combined with some simple scripts it is easy to use from an ordinary browser. Security is tricky though, so it is best to hide the interface from public access.

In a quest to secure my personal server, I wanted to get a notification for each SSH login. I found some solutions online, but they weren’t waterproof (relying on per-user .profile modifications) or were a bit too hackish for my taste (actively tailing auth.log).

Solution

It’s essential that any user logging in cannot externally prevent the notification from being sent. The best solution I found is to modify the SSH server configuration, and force execution of a wrapper script which will generate the notification. The ForceCommand option does exactly that:

Forces the execution of the command specified by ForceCommand, ignoring any command supplied by the client and ~/.ssh/rc if present.

Within the wrapper script, we can use the (relatively undocumented) SSH_CONNECTION environment variable to get a hold of some details about the current connection. We will use that to make our notification a bit more interesting. Finally, we hand control back to the user, spawning the command asked for (saved in SSH_ORIGINAL_COMMAND), or the user’s default shell:

#!/bin/sh

SHELL=$(getent passwd $USER | cut -d: -f7)
IP=$(echo $SSH_CONNECTION | cut -d " " -f 1)
HOSTNAME=$(dig -x $IP +short)
if [ -z "${HOSTNAME}" ]; then
        HOSTNAME=$IP
fi
LOCALHOST=$(hostname)

/usr/local/bin/pushmessage \
        "SSH login on $LOCALHOST" \
        "User ${USER} has logged in from ${HOSTNAME}"

${SSH_ORIGINAL_COMMAND-$SHELL}

I saved this script in /usr/local/sbin/ssh-wrapper, made it executable, and activated it by adding ForceCommand /usr/local/sbin/ssh-wrapper to /etc/ssh/sshd_config.

Note that my wrapper script makes use of pushmessage, a small script which delivers the actual notification:

#!/bin/sh

if [ "$#" != 2 ]; then
    echo "Usage: $0 TITLE MESSAGE" >&2
    exit 1
fi

APIKEY="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
IDEN="XXXXXXXXXXXXXXXXXXXXXX"

curl https://api.pushbullet.com/api/pushes -u ${APIKEY}: \
    --output /dev/null --silent --max-time 5 \
    -X POST -d device_iden=${IDEN} \
    -d type=note -d title="$1" \
    -d body="$2"

The APIKEY and IDEN variables are personalised, and contain respectively your Pushbullet access token and the device identification. The former you can find on your Pushbullet account page, and in order to get a suitable device identifier you’ll need to inquire the API:

$ curl --silent -u ${APIKEY}: \
  https://api.pushbullet.com/api/devices | python -m json.tool
{
    "devices": [
        {
            ...,
            "iden": "XXXXXXXXXXXXXXXXXXXXXX"
        }
    ],
    "shared_devices": []
}

Limitations

• Slowdown: on a normal day, a pushmessage invocation takes about 500 ms. This is a significant slowdown of the — already pretty slow — SSH login. Backgrounding the API call solves this, but can get you spammed.
• Too many invocations: the wrapper script is called on each login, even if the SSH connection was already established (ie. when multiplexing connections using ControlMaster). This is especially noticeable when using tab completion for the scp command: every <TAB> will trigger a notification.
• Cluttered process tree: the ssh-wrapper script keeps on existing in the process hierarchy because it spawns the user shell. It should be possible to avoid this (eg. by execveing the shell instead).

References

1. SSH Login Notifications using Pushbullet
2. SSH Login Meldung aufs Handy per Pushbullet
3. Raspberry Pi: Push Benachrichtigung bei SSH Anmeldung
4. Wie wird eine Pushover Benachrichtigung auf dem Raspberry Pi bei Zugriff per SSH versendet um Einbrüche zu erkennen?

By forking an existing theme, all the boilerplate was already taken care of. I picked Lanyon for its looks and simplicity, and managed to get a basic blog up and running in minutes – almost as advertised.

The resulting set-up is surprisingly functional! Using the built-in web server, launched using the command jekyll serve, you can easily preview local changes. If you want to launch your browser as soon as Jekyll has launched, you can use the following GNU Make rules (or extract the blob between define EXPECT_SERVE and endef, substituting double $’s with single ones):

SHELL := /bin/bash

define EXPECT_SERVE
set timeout 5
spawn jekyll serve
expect {
    -re {Server address:\s+(.+)$$} {
        set address $$expect_out(1,string)
    }
}
expect "Server running..."
catch {
    exec xdg-open $$address
}
interact
endef
export EXPECT_SERVE

.PHONY: serve
serve:
    expect <( echo "$$EXPECT_SERVE" )