Intro

Before you read this be sure to look at:

  1. Phoenix Advanced Deployment
  2. Exrm Docs

And if you really have ton of time then learn ye some Erlang

If you write Elixir and if you question things, then at some point you realize that you need to learn some of Erlang with its OTP.

Elixir is like a Coffee-Script for the JavaScript world but with much better support and you almost never need to touch Erlang.

Until you want to do some cool stuff like cross-compiled deploys.

Deploying your Elixir App onto ARM-Linux device with Exrm

I have a Raspberry Pi 1 running OSMC a linux system. I want to run Phoenix application on that box just for fun.

Exrm and Phoenix Docs describe deploy process thoroughly but it works for systems with Erlang pre-installed.

In order to run an app on a system without

1
erl
binaries you’d have to build cross compiled version of erlang specifically for that system (in my case that’s ARM-linux OSMC).

I failed to do build it manually on OSX. I had to install

1
erlang-minimal
with help of Erlang Embedded guys.

Just follow this post to install erlang on your Pi.

You will get erlang build installed in

1
/usr/lib/erlang/
you can copy that folder to convenient location
1
/Volumes/Disk/erlang

That folder is useful for all further deploys to any Pi

You still would need to tell Exrm about specific libs before running release. Libs and erts of Erlang built for Pi (

1
/Volumes/Disk/erlang
see
1
Configure relx
).

So these are the steps:

Generate a new Application

$ mix help phoenix.new --no-brunch

Check if it works locally

$ cd pi
$ mix phoenix.server

Install exrm

# in mix.exs
defp deps do
  [ ..., {:exrm, "~> 1.0.3"}]
end
$ mix deps.get

Configure relx

$ mkdir -p rel
$ echo '{include_erts, "/Volumes/Disk/erlang"}.' >> rel/relx.config
$ echo '{system_libs, "/Volumes/Disk/erlang/lib"}.' >> rel/relx.config

Replace /Volumes/Disk/erlang with a path to the erts extracted from the pi

Compile the app for prod and build a release

$ MIX_ENV=prod mix phoenix.digest &&\
  MIX_ENV=prod mix compile &&\
  MIX_ENV=prod mix release

Test it:

$ scp rel/pi/releases/0.0.1/pi.tar.gz tv:~
$ ssh tv
$ cd ~ && mkdir -p app && tar -xzf pi.tar.gz -C ./app

For details see Phoenix on Pi repo

Please dm me on twitter Andrew Shatnyy in case there’re questions. English is my second language and I am sure this guide could make no sense to a native speaker :)

Testing requests with generator functions ES6 style

I was implementing an async retrier while refactoring some callback hell. And while testing it I realized that there’s a simple way to make your generators wait.

'use strict';

function wait(done) {
  setTimeout(done, 1000);
}

That’s it. Function above is perfectly yieldable and will suspend your generator.

So in practice your test would look like this.

'use strict';

function wait(time) {
  return function wait(done) {
    setTimeout(done, time);
  };  
}

describe('Retry', () => {
  it('retries 3 failed att', function* times3() {
    // tell mocha to wait longer
    this.timeout(8000);

    let total = 3;
    function* req() {
      while (true) {
        // our awesome wait function
        yield wait(1000);
        if (total-- === 1) {
          return 'done';
        }
        throw new Error('test');
      }
    }
    const result = yield retry(req, { times: 4 });
    assert.equal(result, 'done');
  });
});

I use co-mocha for generator tests, hence

1
function* times3
callback.

Wait function also works in Chrome console in case you want to check it out (don’t forget to use strict).

Sorting Server IPs By a Property in the Hash

Apparently you can sort Redis lists by referencing other data structures in your storage.

More on Redis Sort command here.

Problem described here is an inherited design and doesn’t reflect my engineering opinion. I just had to make it work in given ecosystem.

Problem:

Given a list of hardware servers we need to pull the least busy one from the pool.

Pool in our case is a sorted set of worker ips which is sorted by CPU load in descending order (least busy first). Thus queuer can just take first worker ip in the array and know that’s the least busiest worker.

Infrastructure:

We need a Redis DB running and be accessible by other servers to update information about themselves.

Will use a Redis Set to store our server ips. And Redis Hashes to store CPU utilization and some other info about the server.

Data Structures

$ redis-cli keys "server*"
1) "server:worker:10.0.1.2"
2) "server:worker:10.0.1.3"
3) "server_ips"

Here we have two workers and one server_ips set

Worker

$ redis-cli hgetall "server:worker:10.0.1.3"
1) "cpu"
2) "0.38"
3) "host"
4) "10.0.1.3"
5) "errors"
6) "0"
7) "type"
8) "worker"

In JSON that would be

{
  "cpu": "0.38",
  "host": "10.0.1.2",
  "errors": "0",
  "type": "worker"
}

{
  "cpu": "0.50",
  "host": "10.0.1.3",
  "errors": "0",
  "type": "worker"
}

IpSet

$ redis-cli smembers "server_ips"
1) "10.0.1.3"
2) "10.0.1.2"

In JSON that would be

["10.0.1.3", "10.0.1.2"]

Actual Sorting

127.0.0.1:6379> SORT server_ips BY server:worker*->cpu LIMIT 0 1 DESC
1) "10.0.1.2"

Since cpu load of our “10.0.1.2” worker is less than “0.50” it’s considered the least busy one.

It would be awesome to get entire hash back when you do the sorting, but you can only use GET command for hash property.

You can do pretty cool things with if you read the docs.

##Loading Images as binary with Webworker built from the function

Few months ago I had to read binary image data from s3 with XHRHttpRequest and then parse ExIf header to determine its orientation.

I had to load and parse numerous images asynchronously. Tried executing it all in one thread; no fun. Browser starts freaking out once I hit 4 images.

Just loading binary image data is a problem.

I am not going to describe the basics of Webworkers in this post. Html5 Rocks explained it with better English in 2010. Please read that linked article. Your future self with thank you for investing your time in this.

###Goal: load and parse images in separate threads and then later pass them (with metadata) to the main thread.

My biggest problem with default Webworker loader is versioning and maintenance of separate files. I don’t like extra dependencies unless I need to support unstable versions of browsers.

###How: You can create a Webworker with just a blob of javascript.

var blob = new Blob([worker])
var url = URL.createObjectURL(blob)
this.worker = new window.Worker(url)

Worker here is actually a string I load from browserify module. Now there’s a problem with maintaining actual text of the worker function. Thank God there’s a way to pull in body of the function in javascript.

Try following in the console you’ll get the result.

var worker = function(){
 // body of the function…..
};
var worker_str = worker.toString();

Please note that there’s a context problem. When you load this to webworker as a blob you’ll start getting weird messages about undefined references. Try calling

1
self.postMessage

I immediately invoke the wrapper function to keep the context.

var worker = function(){
 // body of the function..
};
var worker_str = '('+worker.toString()+')(this);';

Final version with xhr calls minus ExIf parser looks like so:

var cached, worker;

cached = null;

worker = function(self) {
  var init, onload, onprogress, xhr;
  xhr = null;
  init = function(url) {
    xhr = new XMLHttpRequest();
    xhr.open('GET', url, true);
    xhr.responseType = 'arraybuffer';
    xhr.onprogress = onprogress;
    xhr.onload = onload;
    xhr.send();
  };
  onprogress = function(evt) {
    if (xhr.status !== 200) {
      return self.close();
    }
    self.postMessage({
      name: 'progress',
      val: {
        loaded: evt.loaded,
        total: evt.total
      }
    });
  };
  onload = function(evt) {
    self.postMessage({
      name: 'onload',
      val: {
        status: xhr.status,
        orientation: xhr.getResponseHeader('x-amz-meta-orientation'),
        array: xhr.response
      }
    }, [xhr.response]);
    self.close();
  };
  self.onmessage = function(evt) {
    init(evt.data);
  };
};

module.exports = function() {
  return cached != null ? cached : cached = '(' + worker.toString() + ')(this);';
};

And to utilize that module I just require the module and call it in the view:

// some backbone jazz
window.URL = window.URL || window.webkitURL;
this.url = options.url
blob = new Blob([worker])
url = URL.createObjectURL(blob)
this.worker = new window.Worker(url)
this.worker.addEventListener('message', this.onMessage)
this.worker.postMessage(this.url)
// rest of the backbone magic

Now your webworker is passing messages essential to keep main thread informed of the loading and work process. See

1
onload, onprogress
.

When loading is done it simply kills itself

1
self.close()
.

Important thing to notice that you should transfer objects not copy them (see about transferrable objects Html5 Rocks).

self.postMessage({
  name: 'onload',
  val: {
    status: xhr.status,
    orientation: xhr.getResponseHeader('x-amz-meta-orientation'),
    array: xhr.response
  }
// this last argument allows us to transfer data from the worker to the main thread
}, [xhr.response]);

And here’s what we do with the image when worker is done:

onload: function(val) {
  if (val.status == 200) {
    value = val.orientation
    this.angle = this.interpritAngleFromHeaders(value)
    blob = new Blob([val.array], {type:'image/jpeg'})
    this.image = new Image()
    this.image.onload = this.complete
    this.image.src = window.URL.createObjectURL(blob)
  } else {
    this.$el.removeClass('progress')
    this.render(-1)
  }
}

Same blob work and ObjectURL.

Given two-dimensional array now walk it spirally

One of those brain-twisters you might be asked if you apply for Software Engineer position at a decent company.

The interviewer goes to the white board and draws a 2d array.

var array = [
    [1,2,3],
    [4,5,6],
    [7,8,9]
];

Write a function that walks that array in spiral and outputs:

'1,2,3,6,9,8,7,4,5';

How to walk it spirally (over engineered)

Like always with programming there are tons of ways to tackle a problem. Only efficient solutions counts.

Here’s my take on this one:

  1. Shave off sides of the 2d matrix(array)
  2. Append contents of the side to result array
var array = [
    [1,2,3],
    [4,5,6],
    [7,8,9]
];
// shave 1
var result = [1,2,3];

var array = [
    [4,5,6],
    [7,8,9]
];
// shave 2
var result = [1,2,3,6,9];

var array = [
    [4,5],
    [7,8]
];
// shave 3
var result = [1,2,3,6,9,8,7];

var array = [
    [4,5]
];
// shave 4
var result = [1,2,3,6,9,8,7,4];

var array = [
    [5]
];
// shave 5
var result = [1,2,3,6,9,8,7,4,5];

var array = [
    []
];

return result.join(',');

Hold ye horses sire… How the hell did you shave other sides off???

The answer is that we have to transpose the matrix every time before we take off one side. But not the first time!

Basically we need to turn that two-dimensional array -90°.

In any case we just identified two main functions here

  • 1
    transpose
    
    to turn the matrix
  • 1
    spiral
    
    to shave off top row
[1,2,3],
[4,5,6],
[7,8,9]
// -90 =>
[3,6,9],
[2,5,8],
[1,4,7]

Let’s write some tests for identified functions

var M = require('../lib/matrix');
describe('Matrix', function() {

  it('walks spiral', function(){
    var result = M.spiral([
      [1,2,3],
      [4,5,6],
      [7,8,9]
    ]);
    expect(result).toEqual('1,2,3,6,9,8,7,4,5');
  });

  it('transposes 3x3', function() {
    var result = M.transpose([
      [1,2,3],
      [4,5,6],
      [7,8,9]
    ]);
    expect(result).toEqual([
      [3,6,9],
      [2,5,8],
      [1,4,7]
    ]);
  });

  it('transposes 2x3', function(){
    var result = M.transpose([
      [4,5,6],
      [7,8,9]
    ]);
    expect(result).toEqual([
      [6,9],
      [5,8],
      [4,7]
    ]);
  });
});

We would have to use

1
while
loops in
1
spiral
to mimic recursion as JavaScript does not yet fully support tail call optimisation. Read more about es6.

Code below will make our tests pass.

function spiral(array) {
  var width = array[0].length;
  var height = array.length;
  var row, index=0;
  var result = new Array(width*height);
  // or
  //var result=[];

  // mutate original array
  // use it as a stack

  // recursion goes here in es6

  while(array.length) {
    row = array.shift();
    while(row.length) {
      result[index++] = row.shift();
      // or
      // result.push(row.shift());
    }
    array = transpose(array);
  }
  return result.join(',');
}

function transpose(array) {

  var column = array[0];
  if (!column) return array;

  var columns = column.length;
  var rows = array.length;
  var result = new Array(columns);
  // or
  // var result = [];
  var columnIndex,
      resultColumnIndex,
      rowIndex;
  
  // turn mattrix
  // [o,o,x]
  // [o,o,x]
  // into
  // [x,x]
  // [o,o]
  // [o,o]

  for(resultColumnIndex = 0, columnIndex = columns -1; columnIndex >= 0; columnIndex--, resultColumnIndex++) {
    result[resultColumnIndex] = new Array(rows);
    // or
    // result.push([]);
    for(rowIndex = 0; rowIndex < rows; rowIndex++) {
      result[resultColumnIndex][rowIndex] = array[rowIndex][columnIndex];
    }
  }
  return result;
}

module.exports = {
  transpose: transpose,
  spiral: spiral
};

Awesome, right? Well, NO.

How to really walk two-dimensional array spirally

Obviously solution above is over engineered and most likely you won’t get a job :(. It’s fun to play with matrixes but when it comes to real life you should never use code above in production.

Spiral problem solved by keeping track of the coordinates for the traversed two-dimensional array.

  • identify coordinates
    1
    topIndex
    
    ,
    1
    bottomIndex
    
    ,
    1
    leftIndex
    
    ,
    1
    rightIndex
    
[top ,o,o,o,  o]
[left,o,o,o,  o]
[o   ,o,o,o,rgt]
[btm ,o,o,o,  o]
  • keep reference of the current side being shaved off
function spiral(array) {
  var rows = array.length;
  var columns = array[0].length;
  var topIndex = 0;
  var bottomIndex = rows - 1;
  var leftIndex = 0;
  var rightIndex = columns - 1;

  var result = [];

  var side = 'top';
  var i;

  while(topIndex <= bottomIndex && leftIndex <= rightIndex) {
    
    if (side === 'top') {
      for(i = leftIndex; i <= rightIndex; i++) {
        result.push(array[topIndex][i]);
      }
      topIndex++;
      side = 'right';
      continue;
    }
    if (side === 'right') {
      for(i = topIndex; i <= bottomIndex; i++) {
        result.push(array[i][rightIndex]);
      }
      rightIndex--;
      side = 'bottom';
      continue;
    }
    if (side === 'bottom') {
      for(i = rightIndex; i >= leftIndex; i--) {
        result.push(array[bottomIndex][i]);
      }
      bottomIndex--;
      side = 'left';
      continue;
    }
    if (side === 'left') {
      for(i = bottomIndex; i >= topIndex; i--) {
        result.push(array[i][leftIndex]);
      }
      leftIndex++;
      side = 'top';
      continue;
    }
  }

  return result.join(',');

}

Function above runs 80% faster than overengineered solution in which you need to rotate arrays.

spiral x 882,768 ops/sec ±0.80% (84 runs sampled)