Got a crazy project you need done? Looking to build an app? Or a robot? Or start to leverage machine learning in your business? My brother and I recently started offering our services at https://madconsulting.ai. Hit us up!

One time, I was looking at a graph, and the thought occurred to me, "this graph shows where the two sides of this equation are exactly equal, but what if a graph showed also where they were almost the same?", so I wrote a little graphing web app: https://truthygraph.github.io/.

I called it "Truthy Graph", since it shows where the 2 sides of the equation are nearly true. I think some interesting shapes show up. Here's a few screenshots:

Another cool thing, is that either side of the equation can use the 'y' or 'x' variables in any way (calling functions like sin, can be squared, etc - the equations need not follow the 'y=' formula). Example:

I'm aware there are a few bugs (especially ...

I decided to try my hand at musical composition. This is my first piece. I call it "We Spin Together".

(This is just the melody)

And here's how it sounds:

Recently, I was curious if I could generate a digital radio signal from scratch just using math, and use that signal to control this radio-controlled outlet that I have.

Here's the Github source: https://github.com/calebmadrigal/radio-hacking-scripts/blob/master/radio_signal_generation.ipynb

Generating radio signals

I want to be able to generate modulated digital signals such as this one...

original radio signal

This is a binary signal coded with On-off keying (also known as Amplitude-shift keying).

I'm taking the long pulses to mean "1" and the short pulses to mean "0", so this signal is transmitting the digital code, "0110100010000000".

# Imports and boilerplate to make graphs look better
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy
import wave
from ...

The following are my notes from a talk I gave at OSCON 2013. Here is the Github source of these Jupyter/IPython notebooks: https://github.com/calebmadrigal/FourierTalkOSCON

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import ...

Hackers are the new Lawyers are the new Knights.

Back in the feudal days, Lords had Knights to protect them and their possessions from enemies (or to attack their neighbors and plunder their wealth).

In the business world today, lawyers are much like the knights of old. Companies can attack each other with the law, and lawyers are both the attacker and defenders.

In the increasingly computerized world, business are hacking each other to steal their Intellectual Property, understand their strategy, and sometimes, take them down all together (like what happened to HBGary, though that wasn't caused by another company). Today, these tactics are employed mostly by Chinese companies hacking competitors in other countries, but as multinational corporations continue to grow in power, this tactic might become ...

I've recently been getting into Software-defined Radio (SDR), mostly using a HackRF - a radio tranceiver capable of operating from 1MHz to 6GHz (which is a huge range).

One of the most simple (and most interesting attacks) which can be done with SDR is what's called a Replay Attack. It works by simply recording a signal, and then rebroadcasting it. I was able to use this attack to lock and unlock my Jeep Patriot (2006) with my computer. Here's how...

First I recorded the "unlock" and "lock" signals from my keyless entry remote using this "flowgraph" in Gnu Radio Companion:

A couple things to note about this flow graph:

• The frequency is 315 MHz (the frequency at which pretty much all keyless entry)
• It records at ...

You can capture radio signals with Software-defined Radios (SDR), such as the HackRF. Gnu Radio is the main software I use for receiving and transmitting radio signals, but I've found Audacity, a program meant primarily for editing sound files, to be a great program for viewing and editing radio signals.

So I had these remote-controlled outlets (which I bought at Home Depot):

And I wanted to see what their wireless communication looked like...

So first I recorded the signals of the different outlets with this Gnu Radio Companion flowgraph:

A few things to note about this flowgraph:

• It records at around 315 MHz, the frequency at which the remote transmits (which I found by looking up it's FCC ID code online)
• It records 2 million samples ...

These are some pretty raw notes I took while installing the software for using a HackRF on OSX. This is the software I installed:

• Gnu Radio (which comes with Gnu Radio Companion)
• Osmocom Gnu Radio Blocks
• gqrx

Gnu Radio Installation

Installed gnuradio via Homebrew

brew update

brew tap robotastic/homebrew-hackrf
brew install gnuradio --with-qt

brew install hackrf
brew install --HEAD rtlsdr
brew link --overwrite rtlsdr

This got GNU Radio and GNU Radio Companion installed, but then I needed to install the Osmocom Gnu Radio Blocks so that Gnu Radio could communicate with the HackRF.

Installing Osmocom Gnu Radio Blocks

This didn't work:

brew install --HEAD gr-osmosdr

Build with directions here: http://sdr.osmocom.org/trac/wiki/GrOsmoSDR

Had to add this to /Users/caleb/Documents/gnu_radio/repos ...

3 Years ago (2013), I built my own home security/automation system with a Raspberry Pi. Here's what it looks like:

What it does

• Allows controlling a couple outlets via a mobile web app
• Alarm system enabled/disabled via mobile web app
• If alarm enabled, magnetic door sensor would trigger siren and turn on outlets

Hardware used

• Raspberry Pi - $35
  • Edimax EW-7811Un 150Mbps 11n Wi-Fi USB Adapter - $10
• Off-the-shelf remote controlled outlet kit - $20
• Off-the-shelf wireless magnetic door sensor/outlet - $25
• Car siren - $8
• Breadboard, transistors, photoresistor, resistors, wires, soldering iron

Total cost of parts: ~$115

Software used

I wrote the controller software in Python (with ZeroMQ, Flask, Rpi.GPIO, and jQuery Mobile): https://github.com/calebmadrigal/rpi-home-automation.

It's made of up 3 components:

• Web Controller ...

DNS Tunneling can be useful for getting out of a very restrictive corporate firewall (since almost nobody blocks DNS).

DNS Tunneling works by encoding IP protocol inside of DNS packets. Iodine is a cross-platform implementation of DNS Tunneling. This mini-guide will walk through the process of setting up a DNS Tunnel with Iodine.

Prerequisite: You will need a server with public IP and domain name

Setup DNS Tunnel

Setup DNS Records

Add 2 DNS records to your domain's DNS system:

* `A record`: tunnelhost -> your ip (maps tunnelhost.yourdomain.com to your server's ip)
* `NS record`: tunnel -> tunnelhost.yourdomain.com

Example:

So now for me:

* `tunnelhost.calebmadrigal.com` now points to `104.236.122.169`
* `tunnel.calebmadrigal.com` now points to `tunnelhost.calebmadrigal.com`
    - **This is the ...

Just another painting. I did this one earlier this year.

I recently needed to call some synchronous code from asyncio. Thankfully, asyncio provides the run_in_executor function, which runs the specified function in a different thread. Here is an example of using it:

import asyncio
import time
from urllib.request import urlopen

@asyncio.coroutine
def count_to_10():
    for i in range(11):
        print("Counter: {}".format(i))
        yield from asyncio.sleep(.5)

def get_page_len(url):
    # This is the blocking sleep (not the async-friendly one)
    time.sleep(2)
    page = urlopen(url).read()
    return len(page)

@asyncio.coroutine
def run_get_page_len():
   loop = asyncio.get_event_loop()

   future1 = loop.run_in_executor(None, get_page_len, 'http://calebmadrigal.com')

   #data1 = yield from future1
   return future1

@asyncio.coroutine
def print_data_size():
   data = yield from run_get_page_len()
   print("Data size: {}".format(data))


loop = asyncio.get_event_loop()
tasks = [
    asyncio.async(count_to_10()),
    asyncio.async(print_data_size ...

I recently decided to brush up on my algorithms and data structures by writing them in Python. Though these are not optimized, they could be helpful for reference. Here is the Github repo: https://github.com/calebmadrigal/algorithms-in-python.

A some of the algorithms included:

• Mergesort in Python
• Quicksort in Python
• Depth-first search in Python
• Breadth-first search in Python
• Binary search tree algorithms in Python
• Dijkstra's in Python
• Hashtable implementation in Python
• Heap in Python
• Combinatoric algorithms in Python

Recursion is awesome, but has the downside of growing the stack, which can limit its usefulness. Some languages like Scheme, however, have Tail-call optimization, which lets programmers write Tail-recursive functions that don't grow the call stack. Python does not have Tail-call optimization (TCO), but with asyncio, we can have something like Tail-call optimization. Basically, this method uses the asyncio event loop like a trampoline function.

Example:

import asyncio

# Tail-recursive factorial using asyncio event loop as a trampoline to
# keep the stack from growing.
@asyncio.coroutine
def factorial(n, callback, acc=1):
    if n == 0:
        callback(acc)
    else:
        asyncio.async(factorial(n-1, callback, acc*n)) # async -> ensure_future in Python 3.4.4

def done_callback(result):
    print("Result: {}".format(result))
    loop = asyncio.get_event_loop()
    loop.stop ...

I had to leave -
     [or was it "flee"?]

I felt entrapped by the drywall office halls
     [unable to breathe]
The sanitized artwork hanging in the pale florescent haze 
     reminded me what art should not be - 
          [what life should not be]

I hate the straight lines in man-made buildings
     that make everything feel so...concrete,
          [so unbending]
But I love the trees with their curvy lines that say,
     "reality can bend -
          come bend with us."

So that's why I am here - 
     [to answer the trees]

• Author: Caleb Madrigal
• Date written: 3/17/2015

This picture by Melissa Vincent was part of the inspiration for this poem:

I've been playing with d3 recently, and it awesome! I'm especially amazed by the ease of animating transitions with it.

There are plenty of amazing d3 demos out there, but I wanted to write some simpler ones to help me understand the basics of d3. You can find them here:

http://calebmadrigal.github.io/d3-demos/

For example, here's how to draw a circle with d3:

Code:

  var width = 200;
  var height = 200;

  var x = width/2;
  var y = height/2;
  var ...

This is my 2nd painting (painted 2014-01-10). I like how it came out, but almost gave up while painting it because I was comparing it to the sky God painted:

I took this photo of a beautiful "vanilla sky" outside of our home in Milwaukee, WI. It served as my model for the painting (along with other memories of looking at sunsets from beneath the spruce tree).

This year (2014) has been, for me, a year of art exploration. And to end it, I'd just like to share what I believe to be the greatest prelude ever written: the Prelude to The Picture of Dorian Gray by Oscar Wilde:

"The artist is the creator of beautiful things. To reveal art and conceal the artist is art's aim. The critic is he who can translate into another manner or a new material his impression of beautiful things.

The highest as the lowest form of criticism is a mode of autobiography. Those who find ugly meanings in beautiful things are corrupt without being charming. This is a fault.

Those who find beautiful meanings in beautiful things are the cultivated. For these there is hope. They ...

I recently finished reading Phantastes by George MacDonald for the 2nd time. The first time I read it, I really liked it. This second time, it became one of my all-time favorites. I've never experience any other work of art which feels so much like being in a dream.

To give you an idea of what it's about, here is the end of the Prelude:

"In a genuine fairy-story, everything must be miraculous, mysterious, and interrelated; everything must be alive, each in its own way. The whole of Nature must be wondrously blended with the whole world of the Spirit. In fairy-story the time of anarchy, lawlessness, freedom, the natural state of Nature makes itself felt in the world... The wolrd of the fairy-story is that ...

Yesterday (12/13/2014), my wife and I went to a painting studio to try our hands at painting. I loved the experience! Here is what I came out with:

This was my first time painting since I was a kid doing watercolors in school. Definitely plan to try it again!

Intro

I first heard of Node.js about 4 years ago. I thought it was a ridiculous idea: using JavaScript as your server-side language! JavaScript always felt to me like an inferior language. Why would you use it unless you absolutely had to!? Now of course, you have to do JavaScript if you are writing code which runs in the browser, but on the server, you can use any language you choose. So why choose JavaScript?

I did choose to use JavaScript/Node.js on my current project. I have now been using it for the last several months, and I am quite impressed! Let me tell you what made me change my mind about Node.js.

Adoption by the Enterprise world

First, Node.js has been adopted ...

In the process of trying to explain to a friend how sessions work, I decided to write a barebones example to demonstrate how to do sessions in Node.js with Express.js.

Here's a live demo: http://expressjs-session-example.herokuapp.com/

And Here's what it looks like:

Before you enter your name:

After you enter your name:

And here is the code:

var express = require('express');
var cookieParser = require('cookie-parser');
var bodyParser = require('body-parser');
var session = require('cookie-session');

///////////////////////////////////////////////////////////////////////////////////////// MIDDLEWARE

var app = express();

// Needed to handle JSON posts
app.use(bodyParser.json());

// Cookie parsing needed for sessions
app.use(cookieParser('notsosecretkey'));

// Session framework
app.use(session({secret: 'notsosecretkey123'}));

// Consider all URLs under /public/ as static files, and return them raw.
app.use(express.static(__dirname + '/public'));

/////////////////////////////////////////////////////////////////////////////////////////// HANDLERS

function ...

Too much detail in art is profanity - a vulgar attempt to sum up that which is beyond finite expression.

When to use an SSH Tunnel?

An SSH Tunnel is perfect for the following scenarios:

• You want to get to a website that a corporate network is blocking.
• You want to encrypt the network traffic for a non-encrypted protocol.

What is an SSH Tunnel?

An SSH Tunnel is where you tell your local SSH Client to forward your local computer's traffic to a specified SSH Server (over the encrypted SSH protocol). The SSH Server will then act as a proxy for all requests you make on your local computer.

How to set up an SSH Tunnel

There are 2 steps to set up an SSH Tunnel, and they both happen on your local computer (no configuration is needed on the SSH server which will be acting as ...

The best art is more like a laser beam than a flood light - it doesn't hit everyone, but those it hits, it burns.

I recently was car shopping, and I had the question about gas mileage: "How much does, say, the difference between 25 mpg and 30 mpg cost?"

To answer this question, I did a bit of analysis using IPython Notebook...

price_per_gallon = 4 ...

Over the last several years, I've been writing down words (or sets of words) which I consider to be particularly beautiful.

Here is my list:

• Camp Lejeune
• Long Boat Key
• Devil May Cry
• Effervescent
• Cellar Door
• Paramore
• Nevermore
• Evermore
• Cognac
• vis-à-vis
• Commandeer

Art begets art
Hate begets hate
Love begets love
Grace begets grace.

Before we depart
We do place
A lasting mark
On others' fate.

• Author: Caleb Madrigal
• Date written: 3/18/2014
• Inspiration: I've found that taking in good art makes me want to write.
• Note: This isn't so much a poem as a truth put into a rhythmic format. It's a bit too "teachy" to be real poetry, in my opinion. Real poetry should not try to teach directly, but rather, should convey feelings (which may indirectly teach something).

Analogies are to functions as facts are to variables - facts are data which the mind can reference, but analogies change the way the mind works.

Whenever possible, I try to explain things with analogies.

Gray
I feel gray
Cannot stay
In this daze.

Even pain - 
Let it rain!
Anything… 
but this haze.

I need light
Or even night.
These gray skies
Glaze my eyes.

Stimulation!
Exhumation! 
Must awaken
To elation.

• Author: Caleb Madrigal
• Date written: 3/17/2014
• Inspiration: I was in a gray mood today, and when I looked outside and saw the gray skies, I thought, “the weather matches my mood… or does my mood match the weather?” I didn’t feel like staying in this gray mood.

Carried along 
On the words of a song,
We let ourselves go
In the waves as they roll.

This ebb and flow
Entraps our souls.
Enraptured so,
As our feelings grow.

And into this torrent,
Reason forfeit,
Our hearts implore us
To join the chorus.

• Author: Caleb Madrigal
• Date written: 3/10/2014
• Inspiration: Music is universal. We enjoy it; we cannot help ourselves. And we want to join with it.

Stay away! Stay away!
Keep away from the bay!
You know that at night,
Will in come the tide,
Sweeping out the mind
That is near the seaside.

And into this sky
You'll sink and you'll fly,
Fish floating by,
Enchanting your eyes,
No more to rise
From the deep ocean skies.

• Author: Caleb Madrigal
• Date written: 3/10/2014
• Inspiration: Sometimes, when I need to get work done, I have to stay away from things which will get me into an artistic, or I'll be swept up in it for hours.

I think Ali Jardine was going for the same idea in this beautiful picture:

Oft on cold and starry nights,
He raises arms, and the battle fights.
And storms the castle of my mind,
Removing from the throne my logical side.

And having the ship commandeered,
Into the stormy winds he steers,
And brings us into crashing waves -
Deep oceans are within his gaze.

And rejoicing on the captain's deck,
Tiring of appointments to respect,
He does not care if we should wreck
But only in beauty does he invest.

Careless and wild - this side of mine.
Roaring through life, not glancing behind
"Behold! the sails are filled with wind!
Onward towards the moon again!"

• Author: Caleb Madrigal
• Date written: 3/10/2014
• Inspiration: What it feels like when my right-brain takes control. Often, starry nights and moonlight provoke me.

The stars as they shine
Will begin to divide
And in the sky
The Morning’s alive
Which never will die,
And then we ride
To eternal life
And to find our time
Was not lived a lie,
And to meet His smile
And to see His face
To bask in His grace
And to in love, embrace.

• Author: Caleb Madrigal
• Date written: 3/9/2014
• Inspiration: Riding on a bus, looking at the beautiful sky, and thinking about meeting God.

Life is beautiful
I feel it inside
But how can I express
The inner mind?
I suppose I can try
Let’s take a ride…

Heart overflowing
No need to hide
Love is a real thing
And there’s a spiritual side.

There is magic
Hidden from our eyes
Yet felt with the soul
That romantic flow.

And to this goal
I continue to stroll
Seeking that beacon
My spirit to strengthen.

• Author: Caleb Madrigal
• Date written: 3/9/2014
• Inspiration: Looking at the sunrise in Chicago over the lake, after a night casually dressed and deep in conversation.

What's my great fear?
I'll tell you; come near.
To lay down in death
with so much left.

Passion not spent -
Oh cowardly regret!
For fear of others?
The thousand deaths.

I'm afraid to die
With no twinkle in my eye
To pass meagerly by
Yet hidden inside.     

To walk through life
Not truly alive
And to pass in the night
With an unfelt "goodbye".

• Author: Caleb Madrigal
• Date written: 3/7/2014
• Inspiration: An explanation of why living from my heart is so important to me.

He views the world through an unfiltered mind -
No defenses built up, layers over time.
He sees it in all its shining brilliance
Experiencing fully the beautiful present.

• Author: Caleb Madrigal
• Date written: 3/4/2014
• Inspiration: Rafe (my son) - when I look at his eyes, it looks like he is just driniking everything in.

In the last week or so (starting Monday, March 3, 2014) I've been in a very poetic mood. It has been so interesting - it was light a light switch flipping on. I haven't been sitting down and trying to write poetry, they've just been coming to me. Often, as soon as one ends, another begins. It has been so pleasurable. I hope people enjoy them, but even if they don't, it has been so worth it to me!

I've recently tried both Heroku and Node.js for the first time. Both are very simple, and work very nicely together. This is a quick guide to deploying a Node.js app to Heroku. To make it simple, I'll walk through deploying a Node.js sample project I created. It is a very simple message board.

Here are the steps to getting this simple app up and running on Heroku...

Get Heroku account

• Go to Heroku, and click Sign Up
• Download and install the Heroku Toolbeld for your operating system (which gives you the heroku command in your terminal)
  • Installation details
• Open your terminal and run heroku login, and enter your heroku credentials and the prompts

Deploy your app to heroku

In your terminal, do the ...

IPython Notebook provides hook methods like _repr_html_ which can be defined by Python objects to allow richer representations. Here's an example:

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of 
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in 
        IPython Notebook. """
    
    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")
            
            for col in row:
                html.append("<td>{0}</td>".format(col))
            
            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)

import random
table = ListTable()
table.append(['x', 'y', 'x-y', '(x-y)^2'])
for i in xrange(7):
    x = random.uniform(0, 10)
    y = random.uniform(0, 10)
    table.append([x, y, x-y, (x-y)**2])
table

Simple example to show how to draw lines with Matplotlib (in IPython Notebook).

ipython notebook --pylab inline

In [5]:

import matplotlib.lines as lines
fig, ax = plt.subplots()

fig.set_size_inches(6,6)          # Make graph square
scatter([-0.1],[-0.1],s=0.01)     # Move graph window a little left and down

line1 = [(0,0), (1,0)]
line2 = [(0,0), (0,1)]

# Note that the Line2D takes a list of x values and a list of y values,
# not 2 points as one might expect.  So we have to convert our points
# an x-list and a y-list.
(line1_xs, line1_ys) = zip(*line1)
(line2_xs, line2_ys) = zip(*line2)

ax.add_line(Line2D(line1_xs, line1_ys, linewidth=2, color='blue'))
ax.add_line(Line2D(line2_xs, line2_ys, linewidth=2, color='red'))
plot()
show()

Read On ↵

Is there a way to do the opposite of what the zip() function does? It turns out, there is - the zip() function with list unpacking...

x = [1,2,3,4,5]
y = [10,20,30,40,50]
points = zip(x,y)
points

[(1, 10), (2, 20), (3, 30), (4, 40), (5, 50)]

zip(*points)

[(1, 2, 3, 4, 5), (10, 20, 30, 40, 50)]

import random
rand = lambda: random.gauss(mu=1, sigma=1)
points = [(rand(), rand()) for i in xrange(1000)]

# Make the graph square
fig = plt ...

I recently migrated my blog from Wordpress to Pelican. Pelican is a static-site generating blog system which is written in Python and uses Jinja2 for templating. I'll probably do a post about the migration process later, but for now, I'll just give my reasons for moving to Pelican...

Data Longevity

I didn't want my blog data stored in a database; I vastly prefer it being stored in version-controlled Markdown format.

Markdown

• I wanted to write blog posts in Markdown (which is possible in Wordpress, but Wordpress isn't designed to use Markdown).
• I also get to use vim to write my blog posts now, which is much nicer than the Wordpress editor.

Language

• I hate PHP, but love Python. It worried me that my blog ...

Overview

This is a guide to set up IPython Notebook on a Server - specifically, on a DigitalOcean VPS. This will allow you to access your iPython Notebooks from anywhere.

Overview of Steps:

• Set up a domain name
• Get a VPS
• Install IPython Notebook (and all dependencies)
• Configure IPython Notebook to run in a server mode
• Add SSL
• Make IPython Notebook start automatically

Create a domain

Go to http://freedns.afraid.org and click "Setup an account here" Go through the signup form Click on the activation link they send to your email This will bring you back to their site; Click the link you see there called "Add a subdomain" Here is how I filled out the form:

Notes:

• Leave the Destination alone for now, and leave the ...

I've been doing a good bit of graphing in IPython Notebook recently, and I often wanted to make the graphs larger. I also often wanted to label the graph axes. So I wrote this simple function and have been using it a lot.

# Graphing helper function
def setup_graph(title='', x_label='', y_label='', fig_size=None):
    fig = plt.figure()
    if fig_size != None:
        fig.set_size_inches(fig_size[0], fig_size[1])
    ax = fig.add_subplot(111)
    ax.set_title(title)
    ax.set_xlabel(x_label)
    ax.set_ylabel(y_label)

Here's a demo of using it...

x = linspace(0, 2 * pi, 1000)
y = 5 * sin(1 * 2*pi*x) + 4 * sin(2 * 2*pi*x)

In [24]:

_ = plot(x, y)

Read On ↵

I've recently been playing with the pandas Python data analysis framework while reading the Python for Data Analysis book (which is written by the author of the pandas library). Here are 2 of my IPython Notebooks that demonstrate some of the basics:

• IPython Notebook Viewer
• US Names Examples
  • This is an example from the book which I've made some modifications to.

And here's a little screenshot of what it looks like in IPython Notebook:

I recently was exploring the Golden Ratio. I was really surprised what a simple (and recursive) relationship it comes from:

Starting with this this image, here is the derivation of the golden ratio...

IPython Notebook / Matplotlib / Pylab / Numpy is great for graphing (among other things). Below is a simple demo of how to graph with it.

To run IPython Notebook, use this command: ipython notebook --pylab inline

Here's a screenshot:

Here's an embedded IPython Notebook showing a slightly easier way:

In [5]:

x = linspace(0, 2*pi, 42)
f1 = 5 * sin(x)
f2 = 2 * sin(2*x)
f3 = 1 * sin(3*x)
plot(x, f1)
plot(x, f2, 'ro')
plot(x, f3, 'g--')
show()

Read On ↵

I'm mostly a Mac user and I also really like Linux (especially Ubuntu), but I hate Windows. However, I'm occasionally required to use Windows at work, so I've found a way to make using Windows more enjoyable: to use Ubuntu (installed on VirtualBox inside of Windows) instead of Windows for everything except for the few tasks which require Windows.

So I have made this little How to Install Ubuntu on VirtualBox in Windows setup guide:

Contents

• Install VirtualBox
• Download Ubuntu 12.04 ISO file
• Create and configure a Virtual Machine
• Tweak Virtual Machine hardware settings
• Install Ubuntu in the Virtual Machine
• Make Ubuntu nicer to use in VirtualBox
• Using your new Virtual Ubuntu system

Install VirtualBox

• Download installer from: https://www.virtualbox.org/wiki/Downloads ...

I've continued on my (hopefully) short-term fascination with music+math (it's fun, but really bad for productivity). So I found this great library for generating music in Python called Pyknon. And it can be installed using pip: pip install pyknon.

I wrote a little python script just to help me understand some concepts in music theory like intervals and chords. It is meant to be read top to bottom (which is why I intersperse functions and variables throughout). It is NOT written in good modular form (in general, I don't recommend writing python like this). This code can also be used as an intro to the pyknon library.

Behold:

from pyknon.genmidi import Midi
from pyknon.music import NoteSeq, Note, Rest

####### First we'll generate ...

For the last day, I've done some reading on music theory (trying to understand why things are the way they are in terms of math). Here are my raw unedited notes:

Important musical ratios:

• Unison: 1:1 frequency
• Octave: 2:1 frequency
  • 12 semitone increase
• Fifth: 3:2 frequency (i.e. multiply by 1.5)
  • 7 semitone increase
• Semitone: 2^(1./12) (12th root of 2) - about 1.059 - this is the "half-step" distance, so you can multiply the frequency of F by 1.059 to get the frequency for F#

Scales and keys

• Chromatic scale: 12 notes (list of all semitones in an octave)
• Diatomic scale: 7 notes
• To be in a "key" (like C-major) means to select 7 notes from the 12 notes in the ...

I was just playing around writing an annuity calculator function. Here is my first version:

def calculate_annuity(years, interest=0, addition_per_year=0, starting_amount=0):
    result = []
    running_total = starting_amount
    for year in range(years+1):
        result.append(running_total)
        running_total = (running_total + addition_per_year) * (1 + interest)
    return result

Here is a 2nd version, written using reduce instead of a loop:

def calculate_annuity2(years, interest=0, addition_per_year=0, starting_amount=0):
    return reduce(lambda result,addition: result + [(result[-1] + addition) * (1 + interest)],
                  [addition_per_year] * years, [starting_amount])

And here is a version in Clojure (basically a direct translation of the Python one:

(defn annuity [years interest addition_per_year init]
    (reduce
        #(conj %1 (* (+ (last %1) %2) (+ 1 interest)))
        [starting_amount]
        (for [i (range years)] addition_per_year)))

Here are some example use cases:

; Find investment returns by year for a $1000 investment ...

I just added a nice command to my .vimrc file to allow me to run the current file I am working on by typing :R:

command R !./%

On the VIM note, here is a link to my current vim config (.vimrc file and .vim directory) on github: Vim config.

Barcamp Milwaukee 2012 is being held October 6th-7th. I had a good time last year, so I can definitely recommend it :)

I have found the secret to not getting road rage: audiobooks. Try it. Right now, I'm listening to Dune by Frank Herbert.

I was able to get some basic facial detection working in OpenCV with Python. Here's what it looks like:

And here is the 34-line python script to draw boxes around all detected faces in live video:

import cv

HAAR_CASCADE_PATH = "/opt/local/share/OpenCV/haarcascades/haarcascade_frontalface_default.xml"
CAMERA_INDEX = 0

def detect_faces(image):
    faces = []
    detected = cv.HaarDetectObjects(image, cascade, storage, 1.2, 2, cv.CV_HAAR_DO_CANNY_PRUNING, (100,100))
    if detected:
        for (x,y,w,h),n in detected:
            faces.append((x,y,w,h))
    return faces

if __name__ == "__main__":
    cv.NamedWindow("Video", cv.CV_WINDOW_AUTOSIZE)

    capture = cv.CaptureFromCAM(CAMERA_INDEX)
    storage = cv.CreateMemStorage()
    cascade = cv.Load(HAAR_CASCADE_PATH)
    faces = []

    i = 0
    while True:
        image = cv.QueryFrame(capture)

        # Only run the Detection algorithm every 5 frames to improve performance
        if i%5==0:
            faces ...

I have started my journey into the world of OpenCV using Python on my Mac. These are my first steps with it.

Installing OpenCV for Python on the Mac

# Install numpy as a prerequisite
sudo port install numpy

# Install OpenCV with the Python API
sudo port install opencv +python27

(The port command is part of Macports)

Capturing an images from a webcam

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/usr/bin/env python2.7
import cv

cv.NamedWindow("w1", cv.CV_WINDOW_AUTOSIZE)

camera_index = 0
capture = cv.CaptureFromCAM(camera_index)

frame = cv.QueryFrame(capture)
cv.SaveImage("pic.jpg", frame)

(I tested this with the iSight camera on my Macbook Pro)

Displaying live video

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

#!/usr/bin/env python2 ...

I realized that I utilize the [NSArray head] and [NSArray tail] functions in various code snippets, but haven.t shared the actual code for that category. So here is my NSArray+FPcode.

NSArray+FP.h

#import <Foundation/Foundation.h>
typedef id(^MapBlock)(id);
@interface NSArray (FP)
- (NSArray *)map:(MapBlock)block;
- (id)head;
- (NSArray *)tail;
@end

NSArray+FP.m

#import "NSArray+FP.h"
@implementation NSArray (FP)

- (NSArray *)map:(MapBlock)block {
    NSMutableArray *resultArray = [[NSMutableArray alloc] init];
    for (id object in self) {
        [resultArray addObject:block(object)];
    }
    return resultArray;
}

- (id)head
{
    return [self objectAtIndex:0];
}

- (NSArray *)tail
{
    NSRange tailRange;
    tailRange.location = 1;
    tailRange.length = [self count] - 1;
    return [self subarrayWithRange:tailRange];
}

@end

• I've always preferred bulleted lists (and the like) over flat paragraph formats
• I believe they can be visually parsed much faster than paragraphs
• Now I have evidence: F-shaped reading pattern study :)

Here is an image which shows the eye movement of users on web pages (where red means more time was spent looking there):

In the last few weeks, I've used the [ALAssetsLibrary loadImages:callback:] method to load a list of images. For what I was doing, I wanted to wait till all the images were loaded before proceeding. I found that I could solve this problem very elegantly using a recursive solution:

- (void)loadImages:(NSArray *)imageUrls loadedImages:(NSArray *)loadedImages 
                                        callback:(void(^)(NSArray *))callback
{
    if (imageUrls == nil || [imageUrls count] == 0) {
        callback(loadedImages);
    }
    else {
        NSURL *head = [imageUrls head];
        __unsafe_unretained id unretained_self = self;        
        ALAssetsLibrary* library = [[ALAssetsLibrary alloc] init];
        [library assetForURL:head resultBlock:^(ALAsset *asset) {
            ALAssetRepresentation *assetRepresentation = asset.defaultRepresentation;

            UIImage *image = [UIImage imageWithCGImage:assetRepresentation.fullResolutionImage 
                              scale:assetRepresentation.scale
                              orientation:(UIImageOrientation)assetRepresentation.orientation];

            [unretained_self loadImages:[imageUrls tail] loadedImages:[loadedImages 
                arrayByAddingObject:image] callback:callback];
        } failureBlock:^(NSError *error) {
            [unretained_self loadImages:[imageUrls tail] loadedImages:loadedImages callback:callback];
        }];
    }
}

In ...

Since I've been getting into Machine Learning/Artificial Intelligence recently, I've been looking at various computing environments recently. Some of the contenders are:

• MATLAB - The traditional software stack for doing machine learning and statistical analysis
• GNU Octave - An open-source MATLAB clone.
• R - An open source clone of a statistical computing environment called S.
• Julia - A language for doing statistical analysis. The goals are to compete with Matlab and R.
• Matplotlib/Pylab/SciPy/NumPy - see below

Of these, I've tried Octave and Matplotlib. Matplotlib/Pylab is the software stack consisting of:

• IPython - an interactive REPL for Python with things like tab completion
• Matplotlib - a graphical plotting library
• NumPy - a matrix library
• SciPy - a collection of scientific and mathematical algorithms

I've only played with Matplotlib/Pylab ...

When using ARC (automatic reference counting) in Objective-C, you need to be careful to avoid causing retain cycles. One place where a retain cycle can occur is where self is referenced in a block. To avoid a retain cycle, you can use the __unsafe_unretained modifier as such:

__unsafe_unretained id unretained_self = self;
reachability = [Reachability reachabilityWithHostname:@"maps.google.com"];
reachability.reachableBlock = ^(Reachability *r) {
    dispatch_async(dispatch_get_main_queue(), ^{
        SurveyFormController *myself = unretained_self;
        self.mapView.reachable = YES;
        [myself configureView];
    });
};

This seems like a nice, concise book on probability and statistics for Python programmers: Think Stats.

A co-worker was just showing me a situation where another developer had written a validation framework which was only used on 2 pages. I've seen this pattern again and again, and I've put a phrase to the thought:

Premature generalization is the new premature optimization.

And it, too, is evil.

I'm taking the Stanford Machine Learning class. The first algorithm we covered is Linear Regression using Gradient Descent. I implemented this algorithm is Python. Here's what it looks like:

import subprocess

def create_hypothesis(slope, y0):
    return lambda x: slope*x + y0

def linear_regression(data_points, learning_rate=0.01, variance=0.001):
    """ Takes a set of data points in the form: [(1,1), (2,2), ...] and outputs (slope, y0). """

    slope_guess = 1.
    y0_guess = 1.

    last_slope = 1000.
    last_y0 = 1000.
    num_points = len(data_points)

    while (abs(slope_guess-last_slope) > variance or abs(y0_guess - last_y0) > variance):
        last_slope = slope_guess
        last_y0 = y0_guess

        hypothesis = create_hypothesis(slope_guess, y0_guess)

        y0_guess = y0_guess - learning_rate * (1./num_points) * sum([hypothesis(point[0]) - point[1] for point in data_points])
        slope_guess = slope_guess - learning_rate * (1./num_points) * sum([ (hypothesis(point[0]) - point[1]) * point[0] for point ...

Today, I had this XML document which I was wanting to make viewable in a browser. My plan was to run the XML document through an XSLT engine to transform it into an HTML document (which could then be opened in a browser).

Instead, I found that I could just embed the XSLT directly in the XML document. Then, when this file is opened in a browser, the XSLT executes and spits out HTML, which is then rendered into the browser window. This is great because I now have the original XML file (which we wanted to keep further analysis purposes) which can also be rendered nicely in a browser.

Here's an example:

<?xml version="1.0" encoding="ISO-8859-1"?>
<?xml-stylesheet type="text/xml" href="#stylesheet"?>
<!DOCTYPE catelog ...

I've been taking the Udacity CS373 (Robotic Car) class, and this week the topic was Particle filters. Particle filters are basically a localization algorithm that accounts for error in measurements and sensors.

Anyway, part of the Particle Filter algorithm requires the generation of a new set of these things called "particles" based on the particles' weights. So to accomplish this task, the Resample Wheel algorithm was presented in class. It is a particularly elegant method of generating a new set of particles by randomly drawing from an old set of particles (with replacement). The particle weight determines the likelihood of it being picked.

Here is the algorithm (with print statements - I used these to help me understand how the algorithm works):

import random

def generate_new_particles(old_particles, weights ...

I was recently wanting to enable a webpage to utilize some server-side functionality which was on a different server than the one serving the original page. Traditionally, this has been problematic due to browsers' Same origin policy. JSONP is a solution to this problem. Here is how to do it with jQuery...

This webpage can be saved locally to any machine, opened in a browser, and can then load data from my web server:

<html><head>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js" type="text/javascript"></script>
<script type="text/javascript">
$(document).ready(function(){
    $("button").click(function(){
        $.ajax({
            url:"http://test.calebmadrigal.com/javascript/jsonp_test.json",
            dataType:'jsonp',
            jsonpCallback: "jsonp_load",
            success:function(result){
                $("#outputdiv").html(result);
            }
        });
    });
});
</script>
</head><body>
    <button ...

Since I've been getting into functional programming more recently, the fact that Python doesn't do tail-call optimization has been bothering me. So I did a bit of searching, and found this gem: Tail Call Optimization Decorator.

Here is a snippet from it:

import sys

class TailRecurseException:
  def __init__(self, args, kwargs):
    self.args = args
    self.kwargs = kwargs

def tail_call_optimized(g):
  """
  This function decorates a function with tail call
  optimization. It does this by throwing an exception
  if it is it's own grandparent, and catching such
  exceptions to fake the tail call optimization.

  This function fails if the decorated
  function recurses in a non-tail context.
  """
  def func(*args, **kwargs):
    f = sys._getframe()
    if f.f_back and f.f_back.f_back \
        and f.f_back.f_back.f_code == f.f_code ...

I just wanted to go through the process of calculating standard deviation today, and this is how I did it in Python. Python makes such a nice calculator :)

>>> s = [2,4,4,4,5,5,7,9]
>>> def average(s): return sum(s) * 1.0 / len(s)
... 
>>> avg = average(s)
>>> avg
5.0
>>> variance = map(lambda x: (x - avg)**2, s)
>>> variance
[9.0, 1.0, 1.0, 1.0, 0.0, 0.0, 4.0, 16.0]
>>> average(variance)
4.0
>>> import math
>>> standard_deviation = math.sqrt(average(variance))
>>> standard_deviation
2.0
>>>

I just finished going through the process of installing and configuring Django (with FastCGI) on Hostmonster's Shared hosting. It was more painful than I expected, so I decided write a post about how I got it working...

First, I wanted to create a subdomain which would host my django stuff. In order to do this, I created a subdomain using the cPanel (the item is called "Subdomains"). This created the directory and a basic .htaccess file (along with a few error pages).

I then went through the basic processes described on these pages:

• http://www.infoinfoc.us/django/hostmonster
• http://www.techgeekguy.com/2011/01/how-to-install-python-svn-django-osqa-web-hosting-server-hostmonster-dreamhost-bluehost-godaddy-hostgator/

When I finished these guides, I kept getting a 500 error. I eventually found that I can access the error log ...

Today, I needed to convert an ASCII-encoded NSString to a base64 NSString. I found a number of methods that convert from an NSString to a Base64 NSData, or from NSData to a Base64 string, but couldn't find one that did exactly what I needed. So here is a method that simply converts an NSString to a Base64 NSString:

+ (NSString *)base64String:(NSString *)str
{
    NSData *theData = [str dataUsingEncoding: NSASCIIStringEncoding];
    const uint8_t* input = (const uint8_t*)[theData bytes];
    NSInteger length = [theData length];

    static char table[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";

    NSMutableData* data = [NSMutableData dataWithLength:((length + 2) / 3) * 4];
    uint8_t* output = (uint8_t*)data.mutableBytes;

    NSInteger i;
    for (i=0; i < length; i += 3) {
        NSInteger value = 0;
        NSInteger j;
        for (j = i; j < (i + 3); j++) {
            value <<= 8;

            if (j < length) {
                value |= (0xFF & input[j]);
            }
        }

        NSInteger theIndex ...

In the process of learning haskell, I've been trying to become comfortable with the function applicator ($) and function composition (.). Here is some code to show how they relate (and how they relate to using parenthesis to enforce order of operations):

let dict = [(1,'a'),(2,'b'),(3,'c')]
-- The following 3 lines of code are equivalent...
snd . head . filter (\item@(k,v) -> k == 2) $ dict -- returns 'b'
snd $ head $ filter (\item@(k,v) -> k == 2) dict -- returns 'b'
snd (head (filter (\item@(k,v) -> k == 2) dict)) -- return 'b'

Today, I was faced with an issue several times where I had an array of items of one type, and had to convert this into an array of items of a different type. This made me really want a map() function. So I added one to NSArray like so:

NSArray+FP.h

#import <Foundation/Foundation.h>
typedef id(^MapBlock)(id);
@interface NSArray (FP)
- (NSArray *)map:(MapBlock)block;
@end

NSArray+FP.m

#import "NSArray+FP.h"
@implementation NSArray (FP)
- (NSArray *)map:(MapBlock)block {
    NSMutableArray *resultArray = [[NSMutableArray alloc] init];
    for (id object in self) {
        [resultArray addObject:block(object)];
    }
    return resultArray;
}
@end

So now, I can do this!

NSArray *strArray = [NSArray arrayWithObjects:@"1",@"1",@"2",@"3",@"5",@"8",nil];
NSArray *numArray = [strArray map:^id(id object) {
    NSString *str = object;
    return [NSNumber ...

Here is how to put an image in the right portion of a UITextField:

UITextField *textField = [[UITextField alloc] initWithFrame:CGRectMake(55, 10, 660, 30)];
textField.rightViewMode = UITextFieldViewModeWhileEditing;

UIImageView *imageView = [[UIImageView alloc] initWithFrame:CGRectMake(0, 0, 15, 15)];
imageView.image = [UIImage imageNamed:@"invalidFieldCircle"];
textField.rightView = imageView;

This also assumes that an image called "invalidFieldCircle.png" is in your project's "Resources" directory, which should be located just below the main project directory (you may have to create it).

Today, I was trying to add an object to an NSDictionary and force it to be a weak reference. This can be done with the [NSValue valueWithNonretainedObject] call:

NSValue *weakRef = [NSValue valueWithNonretainedObject:loadDelegate];
NSDictionary *userInfo = [[NSDictionary alloc] initWithObjectsAndKeys:weakRef, @"loadDelegate", nil];

Just started looking into the Haskell programming language by going through the first few chapters of the awesome book: Learn You a Haskell which is free to read online. Here are some things I really like about what I have seen of Haskell so far:

• Type inference allows you to have static typing but not all the verbosity of most other statically-typed languages.
• Lazy evaluation is huge in Haskell, and it lets you do things like this: let factorial n = product (take n [1..]) So here, we have an infinite list ([1..] creates a infinite list from 1 to infinity!) which from which the first n items are taken and passed to the product function. Awesomeness...
• Nice REPL

For running most commands, powershell and CMD work the same. One big difference I've found is that, when you use periods (.) in a command, they must be surrounded by quotation marks. For instance, this won't work:

mvn clean install -U -Docns.web.dynamic.content.version=2.6.0

But this will work:

mvn clean install -U "-Docns.web.dynamic.content.version=2.6.0"