Intro

Before you read this be sure to look at:

  1. Phoenix Advanced Deployment
  2. Exrm Docs

And if you really have a ton of time learn ye some Erlang

Deploying your Elixir App onto ARM-Linux device with Exrm

I had a Raspberry Pi 1 running OSMC on my local network for quite a while now and I trust I should be able to host my Phoenix application on that box as well. In Erlang land you need a release to run your app on a remote host. Elixir has Exrm lib for building releases for mix projects. Phoenix Docs describe deploy process thoroughly but it works for systems with Erlang pre-installed. You are on your own if you want to cross-compile your release from Intel to ARM architecture.

I failed to do build it manually on OSX so I’ve used erlang-minimal with help of Erlang Embedded docs. This post this post goes over details on how to get minimal runtime up and running.

In essence, you must have a working elrang runtime environment to get your app working, then OpenSSL lib must be present to be able to deal with crypto.

After the installation run, you will get the erlang build installed in /usr/lib/erlang/ next copy that folder to a convenient location /Volumes/Disk/erlang

That folder can be used for all further deploys to most ARM architectures.

Then tell Exrm about specific libs before running release. Libs and erts of Erlang built for Pi (/Volumes/Disk/erlang see Configure relx).

These are the steps:

Generate a new Application

$ mix help phoenix.new --no-brunch

Check if it works locally

$ cd pi
$ mix phoenix.server

Install exrm

# in mix.exs

defp deps do
  [ ..., {:exrm, "~> 1.0.3"}]
end
$ mix deps.get

Configure relx

$ mkdir -p rel
$ echo '{include_erts, "/Volumes/Disk/erlang"}.' >> rel/relx.config
$ echo '{system_libs, "/Volumes/Disk/erlang/lib"}.' >> rel/relx.config

Replace /Volumes/Disk/erlang with a path to the erts extracted from the pi

Compile the app for prod and build a release

$ MIX_ENV=prod mix phoenix.digest &&\
  MIX_ENV=prod mix compile &&\
  MIX_ENV=prod mix release

Test it:

$ scp rel/pi/releases/0.0.1/pi.tar.gz tv:~
$ ssh tv
$ cd ~ && mkdir -p app && tar -xzf pi.tar.gz -C ./app

For details see Phoenix on Pi repo

Testing requests with generator functions ES6 style

I was implementing an async retrier while refactoring some callback hell. And while testing it I realized that there’s a simple way to make your generators wait.

'use strict';

function wait(done) {
  setTimeout(done, 1000);
}

That’s it. Function above is perfectly yieldable and will suspend your generator.

So in practice your test would look like this.

'use strict';

function wait(time) {
  return function wait(done) {
    setTimeout(done, time);
  };  
}

describe('Retry', () => {
  it('retries 3 failed att', function* times3() {
    // tell mocha to wait longer
    this.timeout(8000);

    let total = 3;
    function* req() {
      while (true) {
        // our awesome wait function
        yield wait(1000);
        if (total-- === 1) {
          return 'done';
        }
        throw new Error('test');
      }
    }
    const result = yield retry(req, { times: 4 });
    assert.equal(result, 'done');
  });
});

I use co-mocha for generator tests, hence function* times3 callback.

Wait function also works in Chrome console in case you want to check it out (don’t forget to use strict).

Sorting Server IPs By a Property in the Hash

Apparently you can sort Redis lists by referencing other data structures in your storage.

More on Redis Sort command here.

Problem described here is an inherited design and doesn’t reflect my engineering opinion. I just had to make it work in given ecosystem.

Problem:

Given a list of hardware servers we need to pull the least busy one from the pool.

Pool in our case is a sorted set of worker ips which is sorted by CPU load in descending order (least busy first). Thus queuer can just take first worker ip in the array and know that’s the least busiest worker.

Infrastructure:

We need a Redis DB running and be accessible by other servers to update information about themselves.

Will use a Redis Set to store our server ips. And Redis Hashes to store CPU utilization and some other info about the server.

Data Structures

$ redis-cli keys "server*"
1) "server:worker:10.0.1.2"
2) "server:worker:10.0.1.3"
3) "server_ips"

Here we have two workers and one server_ips set

Worker

$ redis-cli hgetall "server:worker:10.0.1.3"
1) "cpu"
2) "0.38"
3) "host"
4) "10.0.1.3"
5) "errors"
6) "0"
7) "type"
8) "worker"

In JSON that would be

{
  "cpu": "0.38",
  "host": "10.0.1.2",
  "errors": "0",
  "type": "worker"
}

{
  "cpu": "0.50",
  "host": "10.0.1.3",
  "errors": "0",
  "type": "worker"
}

IpSet

$ redis-cli smembers "server_ips"
1) "10.0.1.3"
2) "10.0.1.2"

In JSON that would be

["10.0.1.3", "10.0.1.2"]

Actual Sorting

127.0.0.1:6379> SORT server_ips BY server:worker*->cpu LIMIT 0 1 DESC
1) "10.0.1.2"

Since cpu load of our “10.0.1.2” worker is less than “0.50” it’s considered the least busy one.

It would be awesome to get entire hash back when you do the sorting, but you can only use GET command for hash property.

You can do pretty cool things with if you read the docs.

Loading Images as binary with Webworker built from the function

Few months ago I had to read binary image data from s3 with XHRHttpRequest and then parse ExIf header to determine its orientation.

I had to load and parse numerous images asynchronously. Tried executing it all in one thread; no fun. Browser starts freaking out once I hit 4 images.

Just loading binary image data is a problem.

I am not going to describe the basics of Webworkers in this post. Html5 Rocks explained it with much better English in 2010. Please read that linked article, your future self will thank you for investing time in this.

Goal:

Load and parse images in separate threads and then pass their metadata to the main thread.

My biggest problems with default Webworker loader are versioning and maintenance of separate files. I don’t like extra dependencies in my code unless I must support unstable versions of browsers.

How:

You can create a Webworker with just a blob of javascript.

var blob = new Blob([worker])
var url = URL.createObjectURL(blob)
this.worker = new window.Worker(url)

Worker here is actually a string I load from browserify module. Now there’s a problem with maintaining actual text of the worker function. Thank God there’s a way to pull in body of the function in javascript.

Try following in the console you’ll get the result.

var worker = function(){
 // body of the function…..
};
var worker_str = worker.toString();

Please note that there’s a context problem. When you load this to webworker as a blob you’ll start getting weird messages about undefined references. Try calling self.postMessage

I immediately invoke the wrapper function to keep the context.

var worker = function(){
 // body of the function..
};
var worker_str = '('+worker.toString()+')(this);';

Final version with xhr calls minus ExIf parser looks like so:

var cached, worker;

cached = null;

worker = function(self) {
  var init, onload, onprogress, xhr;
  xhr = null;
  init = function(url) {
    xhr = new XMLHttpRequest();
    xhr.open('GET', url, true);
    xhr.responseType = 'arraybuffer';
    xhr.onprogress = onprogress;
    xhr.onload = onload;
    xhr.send();
  };
  onprogress = function(evt) {
    if (xhr.status !== 200) {
      return self.close();
    }
    self.postMessage({
      name: 'progress',
      val: {
        loaded: evt.loaded,
        total: evt.total
      }
    });
  };
  onload = function(evt) {
    self.postMessage({
      name: 'onload',
      val: {
        status: xhr.status,
        orientation: xhr.getResponseHeader('x-amz-meta-orientation'),
        array: xhr.response
      }
    }, [xhr.response]);
    self.close();
  };
  self.onmessage = function(evt) {
    init(evt.data);
  };
};

module.exports = function() {
  return cached != null ? cached : cached = '(' + worker.toString() + ')(this);';
};

And to utilize that module I just require the module and call it in the view:

// some backbone jazz
window.URL = window.URL || window.webkitURL;
this.url = options.url
blob = new Blob([worker])
url = URL.createObjectURL(blob)
this.worker = new window.Worker(url)
this.worker.addEventListener('message', this.onMessage)
this.worker.postMessage(this.url)
// rest of the backbone magic

Now your webworker is passing messages essential to keep main thread informed of the loading and work process. See onload, onprogress.

When loading is done it simply kills itself self.close().

Important thing to notice that you should transfer objects not copy them (see about transferrable objects Html5 Rocks).

self.postMessage({
  name: 'onload',
  val: {
    status: xhr.status,
    orientation: xhr.getResponseHeader('x-amz-meta-orientation'),
    array: xhr.response
  }
// this last argument allows us to transfer data from the worker to the main thread
}, [xhr.response]);

And here’s what we do with the image when worker is done:

onload: function(val) {
  if (val.status == 200) {
    value = val.orientation
    this.angle = this.interpritAngleFromHeaders(value)
    blob = new Blob([val.array], {type:'image/jpeg'})
    this.image = new Image()
    this.image.onload = this.complete
    this.image.src = window.URL.createObjectURL(blob)
  } else {
    this.$el.removeClass('progress')
    this.render(-1)
  }
}

Same blob work and ObjectURL.

Given two-dimensional array now walk it spirally

One of those brain-twisters you might be asked if you apply for Software Engineer position at a decent company.

The interviewer goes to the white board and draws a 2d array.

var array = [
    [1,2,3],
    [4,5,6],
    [7,8,9]
];

Write a function that walks that array in spiral and outputs:

'1,2,3,6,9,8,7,4,5';

How to walk it spirally (over engineered)

Like always with programming there are tons of ways to tackle a problem. However, only efficient solutions counts.

Here’s my take on this one:

  1. Shave off sides of the 2d matrix(array)
  2. Append contents of the side to result array
var array = [
    [1,2,3],
    [4,5,6],
    [7,8,9]
];
// shave 1 (top)
var result = [1,2,3];

var array = [
    [4,5,6],
    [7,8,9]
];
// shave 2 (right)
var result = [1,2,3,6,9];

var array = [
    [4,5],
    [7,8]
];
// shave 3 (bottom)
var result = [1,2,3,6,9,8,7];

var array = [
    [4,5]
];
// shave 4 (left)
var result = [1,2,3,6,9,8,7,4];

var array = [
    [5]
];
// shave 5 (top)
var result = [1,2,3,6,9,8,7,4,5];

var array = [
    []
];

return result.join(',');

“Hold ye horses sire… How the hell did you shave sides off?”, I hear you cry.

The answer is that we have to transpose the matrix every time before we take off one side. But not the first time!

Basically we need to turn that two-dimensional array -90° and take the top off (shave).

In any case, we’ve just identified two main functions here

  • transpose to turn the matrix
  • spiral to shave off top row
[1,2,3],
[4,5,6],
[7,8,9]
// -90 =>
[3,6,9],
[2,5,8],
[1,4,7]

Let’s write some tests for these functions

var M = require('../lib/matrix');
describe('Matrix', function() {

  it('walks spiral', function(){
    var result = M.spiral([
      [1,2,3],
      [4,5,6],
      [7,8,9]
    ]);
    expect(result).toEqual('1,2,3,6,9,8,7,4,5');
  });

  it('transposes 3x3', function() {
    var result = M.transpose([
      [1,2,3],
      [4,5,6],
      [7,8,9]
    ]);
    expect(result).toEqual([
      [3,6,9],
      [2,5,8],
      [1,4,7]
    ]);
  });

  it('transposes 2x3', function(){
    var result = M.transpose([
      [4,5,6],
      [7,8,9]
    ]);
    expect(result).toEqual([
      [6,9],
      [5,8],
      [4,7]
    ]);
  });
});

We would have to use while loops in spiral to mimic recursion as JavaScript does not yet fully support tail call optimisation. Read more about es6.

Code below will make our tests pass.

function spiral(array) {
  var width = array[0].length;
  var height = array.length;
  var row, index=0;
  var result = new Array(width*height);
  // or
  //var result=[];

  // mutate original array
  // use it as a stack

  // recursion goes here in es6

  while(array.length) {
    row = array.shift();
    while(row.length) {
      result[index++] = row.shift();
      // or
      // result.push(row.shift());
    }
    array = transpose(array);
  }
  return result.join(',');
}

function transpose(array) {

  var column = array[0];
  if (!column) return array;

  var columns = column.length;
  var rows = array.length;
  var result = new Array(columns);
  // or
  // var result = [];
  var columnIndex,
      resultColumnIndex,
      rowIndex;
  
  // turn matrix
  // [o,o,x]
  // [o,o,x]
  // into
  // [x,x]
  // [o,o]
  // [o,o]

  for(resultColumnIndex = 0, columnIndex = columns -1; columnIndex >= 0; columnIndex--, resultColumnIndex++) {
    result[resultColumnIndex] = new Array(rows);
    // or
    // result.push([]);
    for(rowIndex = 0; rowIndex < rows; rowIndex++) {
      result[resultColumnIndex][rowIndex] = array[rowIndex][columnIndex];
    }
  }
  return result;
}

module.exports = {
  transpose: transpose,
  spiral: spiral
};

Awesome, right? Well, NO.

How to really walk two-dimensional array spirally

Obviously, the solution above is over engineered and most likely you won’t get a job :(. It’s fun to play with matrixes but when it comes to real life you should never use code described above in production.

The spiral problem is solved by keeping track of the coordinates for the traversed two-dimensional array.

  • identify coordinates topIndex, bottomIndex, leftIndex, rightIndex
[top ,o,o,o,  o]
[left,o,o,o,  o]
[o   ,o,o,o,rgt]
[btm ,o,o,o,  o]
  • keep a reference of a current side being shaved off
function spiral(array) {
  var rows = array.length;
  var columns = array[0].length;
  var topIndex = 0;
  var bottomIndex = rows - 1;
  var leftIndex = 0;
  var rightIndex = columns - 1;

  var result = [];

  var side = 'top';
  var i;

  while(topIndex <= bottomIndex && leftIndex <= rightIndex) {
    
    if (side === 'top') {
      for(i = leftIndex; i <= rightIndex; i++) {
        result.push(array[topIndex][i]);
      }
      topIndex++;
      side = 'right';
      continue;
    }
    if (side === 'right') {
      for(i = topIndex; i <= bottomIndex; i++) {
        result.push(array[i][rightIndex]);
      }
      rightIndex--;
      side = 'bottom';
      continue;
    }
    if (side === 'bottom') {
      for(i = rightIndex; i >= leftIndex; i--) {
        result.push(array[bottomIndex][i]);
      }
      bottomIndex--;
      side = 'left';
      continue;
    }
    if (side === 'left') {
      for(i = bottomIndex; i >= topIndex; i--) {
        result.push(array[i][leftIndex]);
      }
      leftIndex++;
      side = 'top';
      continue;
    }
  }

  return result.join(',');

}

Function above runs 80% faster than overengineered solution, in which you need to rotate arrays.

spiral x 882,768 ops/sec ±0.80% (84 runs sampled)