I always seem to forget how to setup a virtualenv for python 3 - with my move to a new laptop using linux I had to do this again so I kept the setup here for future reference. It's simple but easy to forget if you only ever do it, say, once every year or so.

The following will result in a virtualenv in the directory ~/venv/py-3.4 using the executable /usr/bin/python3.4:

$ sudo pip install virtualenv
[sudo] password for sean: 
Downloading/unpacking virtualenv
  Downloading virtualenv-14.0.6-py2.py3-none-any.whl (1.8MB): 1.8MB downloaded
Installing collected packages: virtualenv
Successfully installed virtualenv
Cleaning up...
$ mkdir ~/venv
$ virtualenv -p /usr/bin/python3.4 ~/venv/py-3.4
Running virtualenv with interpreter /usr/bin/python3.4
Using base prefix '/usr'
New python executable in /home/sean/venv/py-3.4/bin/python3.4
Also creating executable in /home/sean/venv/py-3.4/bin/python
Installing setuptools, pip, wheel...done.
$ python --version
Python 2.7.9
$ source ~/venv/py-3.4/bin/activate
(py-3.4) [sean@seoul ~]$ python --version
Python 3.4.2

I'm trying to stick to one-post-per-week but the last month been pretty busy for me so I've exhausted of scheduled posts that I built up. I have about 3-4 interesting things in the pipeline which will pop up over the next month, so I think I can get away with publishing something simple like this for one week :)

I picked up a couple of ESP8266 modules - an inexpensive little SoC with WiFi - from AliExpress but ran into a couple of weird issues when trying to get them up and running. A lot of documentation is a little confusing, since it usually begins with flashing an image even though when you power it most likely starts up an open WiFi access point you can connect to. 

Anyway here are the issues I ran into, in order - coincidentally they capture a pretty simple workflow for getting started on the ESP8266.

Issue 0 - uhh ... what do I connect to where?

Update: I didn't know this at the time, but you can simply install the USB to UART bridge drivers from Silicon Labs and connect to the board via USB - on my laptop it appears as /dev/SLAB_USBtoUART. Using this method you can skip this whole section. 

There's dozens of different ESP8266 boards, each with their own little configurations, layouts and labels. Since I bought unbranded boards from AliExpress I didn't really know the search terms to take me to a nice simple diagram of what needs to be connected to each pin of my TTL cable. Anyway my boards look like this:

They connect to my TTL cable a little like this

Just in case it isn't clear from the picture - since I stupidly didn't match up all the colours - here's exactly what is connected to what:

<create table showing connections>

Even though there's a micro-USB connector this set of connections is sufficient to bring up the board and send some simple commands to it.

Issue 1 - connecting via serial (screen/OS X)

There's a very key bit of info for OS X users that's usually skipped over. When you attempt to connect to the ESP8266 using a USB TTL cable and screen you can be mistaken for thinking you've messed something up or you've got a dud chip, as it fails to respond to any input:

Sending an "AT" should result in the response "OK", but in this case the cursor just jumped to the left of the screen and nothing else happened. What's happened here is that GNU Screen on OS X is handling the Return key weirdly, sending only a CR character instead of CR+LF. I played around with the man page for a while but couldn't get it to play nice - eventually I downloaded CoolTerm which did the job nicely.

Issue 2 - firmware

After you've confirmed the chip is up and running by successfully issuing an "AT" you'll probably want to do something a little more useful with it. Since you likely don't speak AT (though a reference is here if you want to try) something simpler like Lua might be suitable. This will involve downloading a binary (or building it yourself, if you feel adventurous), and setting up a couple of things in python.

To save the bother of setting up dev tools for now, we'll use a service which will build a nice little NodeMCU image for you - http://nodemcu-build.com. Here's the build config I used:

Once it's complete you'll receive an email detailing the download link - I received two and opted to use the Integer one, which worked out nicely

To let us flash the ESP8266 we need to connect up the GPIO0 pin (which is labelled "D3" on my board) and reboot it so that it's in "flash" mode - here I've done this with the white cable.

Next we'll need to open up a terminal and setup pyserial and esptool, then download our image and flash it. Both pyserial and esptool can be retrieved via git - I'm just gonna be brief with this part as I've already spent a lot of time with screenshots and pictures already.

$ git clone https://github.com/pyserial/pyserial
Cloning into 'pyserial'...
remote: Counting objects: 4305, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4305 (delta 0), reused 0 (delta 0), pack-reused 4301
Receiving objects: 100% (4305/4305), 1.06 MiB | 1.01 MiB/s, done.
Resolving deltas: 100% (3207/3207), done.
Checking connectivity... done.
$ cd pyserial
$ sudo python setup.py install
Password:
running install
... lots of lines later ...
Installed /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyserial-3.0.1-py2.7.egg
Processing dependencies for pyserial==3.0.1
Finished processing dependencies for pyserial==3.0.1
$ git clone https://github.com/themadinventor/esptool
Cloning into 'esptool'...
remote: Counting objects: 293, done.
remote: Total 293 (delta 0), reused 0 (delta 0), pack-reused 293
Receiving objects: 100% (293/293), 106.48 KiB | 0 bytes/s, done.
Resolving deltas: 100% (157/157), done.
Checking connectivity... done.
$ cd esptool

Now we can download our NodeMCU binary, double-check the path of our USB TTL device (since I always forget) and start flashing

$ curl -LO http://nodemcu-build.com/builds/nodemcu-master-7-modules-2016-01-18-22-47-15-integer.bin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  388k  100  388k    0     0   247k      0  0:00:01  0:00:01 --:--:--  633k
$ ls /dev/tty.usbserial* 
/dev/tty.usbserial-A402O00H
$ python esptool.py --port /dev/tty.usbserial-A402O00H write_flash 0x00000 ./nodemcu-master-7-modules-2016-01-18-22-47-15-integer.bin 
Connecting...
Erasing flash...
Took 1.90s to erase flash block
Wrote 398336 bytes at 0x00000000 in 43.5 seconds (73.2 kbit/s)...

Leaving...
$ 

Now the board should be ready to reboot - if we go ahead and reboot it then connect using CoolTerm (changing to use 9600 baud instead of the 115200 we previously used) there should be a nice little Lua prompt waiting.

Issue 3 - what now?

So what cool things can we do now with our tiny little WiFi chip? Well firstly let's understand and tackle that pesky cannot find init.lua error message. After the NodeMCU firmware boots it searches for a file in the onboard eeprom called "init.lua" whose purpose is to allow you to run some lua code after boot, so you can setup a simple program or define some useful functions without having to get your hands dirty and rebuild the NodeMCU firmware and reflash.

So if we want to create a simple init.lua which prints out a little message on boot, here's what we can do.

$ git clone https://github.com/4refr0nt/luatool
$ cd luatool/luatool
$ echo 'print("check out my cool new ESP8266!")' > ./init.lua
$ ./luatool.py --port /dev/tty.usbserial-A402O00H --src init.lua --dest init.lua --verbose
Upload starting
Stage 1. Deleting old file from flash memory
->file.open("init.lua", "w") -> ok
->file.close() -> ok

Stage 2. Creating file in flash memory and write first line->file.remove("init.lua") -> ok

Stage 3. Start writing data to flash memory...->file.open("init.lua", "w+") -> ok

Stage 4. Flush data and closing file->file.writeline([==[print("check out my cool new ESP8266!")]==]) -> ok
->file.flush() -> ok
->file.close() -> ok
--->>> All done <<<---
$

Now any time you reboot your ESP8266 here's what you'll see instead of that error message:

There are a few nice examples at http://nodemcu.com, which you can either key in manually or upload as part of an init.lua file so that they always run on startup - just be careful you don't leave your ESP8266 in an infinite loop or else you'll have to reflash the firmware and start again. To save you the trouble, here are a couple of simple of examples, all of which involve connecting to a wifi network and serving some data.

The first is a simple hit counter, which records the number of requests it has served, and displays these on a simple page. Send the file using luatool similar to the previous example (changing --src parameter but ensuring --dest is still "init.lua"). Here's what you'll see after visiting the page - because Chrome attempts to fetch both the page and favicon.ico each hit is registered twice :)

Code: hitCount.lua

The second example allows you to switch the onboard LED on or off using a button on the page (which sends a POST request via AJAX). Because we're just dealing with raw TCP requests, I use a very inefficient/wrong way to decide whether or not we're handling a POST request - but since it's just a demonstration it's ok.

After loading the page and toggling the LED here's what we see:

Code (program): ledToggle.lua

Code (webpage): led.html

So there we have it - a simple introduction to connecting to an ESP8266 board, flashing with NodeMCU firmware and a couple of little pointers on how to use luatool to create a lua script which runs at startup. To find out more about the libraries available on NodeMCU check out their page on Read The Docs or for a little more info about the Lua programming language check out Programming in Lua 1st Edition. 

I seem to have written a few too many "Getting Started" guides, however I'll be spending a fair bit of time in the future on the ESP8266 as it offers plenty of opportunities for interesting work.

Sometimes I've found myself with an hour or so spare and an idea best visualised through a little webapp, but hooking up a server, presentation layer, serving static files like jquery/bootstrap etc can be a tiny bit annoying if you do it more than a couple of times. To remove this little barrier I hooked all my preferred tech together in a simple bare-bones template that is pretty ugly but will suffice for 99% of these little projects I have.

Code: CherryPySkeleton

Demo: http://skeleton.mclemon.io

After my initial foray into graphics I wanted to see what was possible with the Atmega328p and the SSD1306 screen. Lots of applications which use this little microcontroller involve pretty mundane, uninspiring things like home automation and I like the idea of giving it a wee chance to shine and do something fun. The constraints of a relatively slow processor, extremely low memory, as well as a tiny and monochrome display were pretty enticing to me, especially after reading some of the early stuff on folklore.org on the hacks Bill Atkinson, Andy Hertzfeld et al employed to get most of out the Macintosh's 68000 processor.

FPS counter

Firstly we need to know the limits of the hardware. I wasn't sure whether the processor, the SPI bus or something inside the display itself would limit the frame rate to something unacceptable and put the kibosh on any sort of fast/responsive visualisations. So I whipped up a quick test which sent a bunch of empty frames to the display and every so often output the average frame rate.

Mercifully it seems that we're able to push as much as 207fps to the SSD1306, which means that we've got a fair bit of room to play with and that if anything the Atmega328p is the bottleneck.

Code: ssd1306_spi_maxfps.ino

Breakout

My first attempt was a little clone of breakout. I am apparently hopeless at it - but it's actually the first "game" I'd ever written, so that's fun.

Code: https://github.com/smcl/breakoutduino.git

3D cube

Since the Breakout clone worked smoothly, wanted to see if it was possible to produce simple 3D graphics - eventually settling on creating a spinning cube bouncing around the screen.

Initially I got a little too ambitious and attempted to roll my own general purpose library for 3D graphics - with a set of functions to manipulate a stack of transformation matrices. However I kept on blowing through my stack space  and clobbering a load of program state. This makes sense, since if we have a matrix implementation of 4x4 floats, we'd need to create one for the camera, a rotation and translation then that's already 3 matrices of 64 bytes each. Then we have to multiply the matrices and apply this to 8 vertices you can easily see how a naive implementation could at some point go a bit haywire and chew through more stack than it should.

Instead of taking the time to trim this down using the limited debug tools available to me (no stepping, watch, gdb through my simple USB cable) I just manually calculated the transformations I needed for my rotating/bouncing cube ahead of time and implemented them so they could be parameterised. So starting with the following (with Tx, Ty & Tz representing the center position of the cube, θ being angle of rotation and x, y and z being the co-ordinates of each cube)

Which means that each point can be represented by the following vector

Since I just wanted a rotating cube with wireframe lines I didn't need to do much other than use this to calculate the expected x/y positions of each point on screen and then use existing drawLine() function to connect each. Which resulted in this slightly clumsy function:

The final video involved two rotating cubes, one of which is flying around the screen.

It's quite satisfying result as it's the first time I'd produced a from-scratch 3D projection since university, and was actually surprisingly fast - the video looks a little shakey but in person it's as smooth as butter.

Code: 3D_cubes.ino

Seed Cathedral

... later that day

And after all that hard work, I saw some stuff that left me pretty deflated - two excellent implementations of graphics libraries for AVR microcontrollers. The first one is u8g - a slightly more heavy duty library than the one Adafruit provided. It also provides some nicer font handling ... including a tiny font similar to the one I did. Here's an example of someone using it to draw a rotating cube at a reasonable clip, ~40 fps no less:

The second one is even more impressive and needs to be seen to be believed. Someone managed to create a library to render a 16 bit colour scene in 3D with texturing and lighting.

Be sure to check out the video at the bottom of this link, it's a little humbling after you've just thrown together a couple of simple monochrome visualisations - http://hackaday.com/2016/01/02/better-3d-graphics-on-the-arduino

The Canon 450D supports external shutter release mechanisms using a simple 2.5mm TRS stereo plug - closing Tip or Ring to Sleeve will cause the camera to either take a picture or autofocus:

I had a few Arduino itches I wanted to scratch, bits and pieces which aren't particularly complex or interesting enough to warrant a standalone blog post, but which are nonetheless worth spending a few paragrapsh on.

Why does every Arduino build include "-R .eeprom" when producing the ELF binary?

I had previously wondered whether it was necessary to pass in "-R .eeprom" when building the ELF binary, but I didn't really explore it much at the time. Essentially all articles or guides about using the GNU toolchain to compile programs for Arduino/AVR (and indeed the Arduino IDE itself) specify this switch so it's really worth clarifying what it's all about.

The first thing is that some articles are just plain wrong in their documentation about this option, for example the Arduino Playground docs for FreeBSD state that it's required to upload our binary to the Arduino board:

This is pretty wrong, since in avr-objcopy the -R switch actually specifies sections to be removed from the binary - per the manpage below:

So let's investigate what's up - here's an avr-objdump on the binary produced by blink.c:

There's actually no .eeprom section here to be removed - so nothing in binary would end up there anyway. If we define a variable with the necessary attribute "__section__" we can force something to be placed here, like the following string:

char *foo __attribute__((__section__(".eeprom"))) = "bar";

and now if we rebuild our ELF binary and inspect using avr-objdump:

All of a sudden we've got an .eeprom section in our executable, which would be removed if we followed the default instructions.

So why is this removed by default when uploading an Arduino sketch? I can only guess, but presumably it's just a common courtesy to anyone who uses .eeprom as persistent runtime storage - so they don't accidentally hose their data when programming their board. Hopefully if you know enough to place a variable in .eeprom, then you'd have the know-how to be able to alter the build configuration to omit these flags. It could also be possible that the eeprom wears out after fewer writes than internal flash so a bit of extra care is taken to preserve it - though I really cannot say why.

How is Arduino memory laid out and what controls it?

I was curious about how the memory layout is defined in the avr-gcc toolchain used in building programs for the Arduino, and ow it can be controlled. I've played around with LDF files for Analog Devices' Blackfin and SHARC chips (which look like ADSP-BF533.ldf and ADSP-21160.ldf respectively), but have never used GCC's linker scripts before.

On OS X using CrossPack these linker scripts reside in /usr/local/CrossPack-AVR/avr/lib/ldscripts, and there are an astonishing 95 of them in total:

These linker scripts aren't defined per microprocessor (i.e. atmega328, atmega168), but are written per architecture - taking a quick look at the documentation at nongnu.org we can see that the AVR chips used in current Arduino boards are all in the avr5 family (except for the atmega2560, which is avr6) so we can narrow things down slightly, but there are still five linker scripts to choose from:

Which one of these is used depends on the arguments passed to the linker, and each of them should be different and tailored for a certain purpose - although in practice this is not strictly true:

• avr5.x: default linker script, this is used for linking straight C programs
• avr5.xn: linker script used when the "-n" flag is passed to the linker. According the the ld manpage this flag is used to turn off page alignment and mark the output as "NMAGIC" however diffing this and avr5.x shows no difference in practice.
• avr5.xbn: used when the "-N" flag is passed to ld. It's meant to mark the .text and .data sections to be readable & writeable (as well as not page-aligning the data segment apparently) however again this is script is identical to avr5.x.
• avr5.xr: used when "-r" flag passed to ld. This is to "generate relocatable output" - but essentially it's used to pre-link C++ files I think?
• avr5.xu: used when "-Ur" is passed to ld. This is used for the final link on C++. It's roughly the same as avr5.xr but includes something about ctor table

To see what the memory layout the basic linker script produces, we can take a pretty boring program like blink.c and take it apart using avr-objdump. The GCC docs have a nice wee visualisation of the SRAM usage of an atmega128, which won't be a million miles away from the 328p:

So the SRAM contains Data, BSS (a segment containing variables initialised to zero) and then some space for the heap (which starts at the address __heap_start and continues to a handful of bytes before the stack base) and the stack begins at RAMEND (address 0x8FF on Atmega328p) and grows down as necessary.

This is why you get that little "low memory" warning in the Arduino IDE - if there's only a couple hundred bytes spare in SRAM it's not too hard to accidentally write a program that uses too much stack and ends up overwriting some of heap variables. There's no memory protection (that I'm aware of) on these chips so there's no way to prevent this other than being careful.

How quickly can we access the Arduino's SRAM, Program Memory and EEPROM?

I wrote a couple of simple benchmarks to see how quickly we can access the different memory types available on the Arduino:

• read: read and sum a 256 byte array
• write: write the values 0...255 into a 256 byte array

This is not massively scientific but I just wanted ballpark figures to see how things stacked up. The results I got were:

I bought a little GL.iNet router and flashed it with a fresh OpenWRT image but I've been unable to do anything with it yet since it was unable to connect to the net via my apartment's ethernet connection. After some diagnostics I managed to find out that I needed to find the PPPoE username/password my working router used to connect, which was where my weird adventure began. 

My landlord or previous tenant had (smartly) changed the default password - admin/admin - so I wasn't able to use the web interface or SSH. I realised there was likely some Serial headers on the board I could use, so I dug out the schematics, cracked open my router and hoped to god that there was no login required, so that I could retrieve the settings from /etc/chap-secrets where I think they reside.

My router is a TP-Link WR741ND v4, a router that sadly doesn't already have serial pins which meant I needed to do a little soldering.

Two things need soldering, first is the connection between TX header and the onboard MIPS cpu as it's left open on this version of the router: 

Once this was done I just needed to connect some cables to the TX, RX and GND headers (I also connected up VCC in case I ever needed it in future):

Once these were sorted I was able to hook up my USB-TTL cable, open up terminal and crosss my fingers to see a very reassuring Linux boot in progress...

... followed swiftly by a login prompt!

I genuinely hoped that serial access would not be password protected, but it seems that I need to regroup and figure out how I can maybe retrieve the pppoe information some other way.

There is hope - typing "tpl" and hitting <Enter> during the boot process takes me to the u-boot prompt below 

So it's possible that either:

1. I could coax uboot to boot into linux in single user mode
2. Theres a way I could dump the data in the router's flash memory (I think md/mw/cp uboot commands can help here), find a way to access the filesystem this way. 
3. I could get uboot to load a minimal linux image over TFTP (using tftpboot), then mount the flash file system and browse it.

 The final alternative is to brute force the username (probably still admin) and password, which is a little inelegant and slow.

In a previous post I created a little replacement font for the Adafruit Graphics Library, but left the non-ASCII characters (values > 0x7F, 128 in total) completely blank. This is because this area is used to implement the additional symbols necessary for languages other than English, and I had initially just intended to create an English language font for my own purposes.

Obviously you can't represent all of the necessary symbols for all the world's languages in just 128 slots, and this is where ISO 8859 comes in*. There are 16 ISO/IEC 8859 character sets which implement the symbols for loosely related (and sometimes unrelated) languages in the upper 128 bytes.

The most common one is probably ISO 8859-1 which can represent most of the EU's main languages, so it should be a pretty sensible one to implement. However since I live in the Czech Republic and I love Central\Eastern Europe I decided to implement ISO 8859-2 which covers Bosnian, Polish, Croatian, Czech, Slovak, Slovene, Serbian, and Hungarian and looks like the below

Note: the above rendering I picked up someplace on the internet is actually incorrect - when the letters t and d have a caron/haček it actually looks more like an apostrophe - ď and ť.

I only implemented this for my slimmer 3x8 font, but if there's any interest I can quickly put together something for the Adafruit 5x8 one or attempt a different character set so long as it's based based on a latin script. Sadly I'm not sure if my skills are up to the task of creating something tougher like Chinese, Tamil or Thai.

I'm a novice at font design, but looking at the required diacritics it seemed to make sense to reserve the top two rows for things like Haček (e.g. č, ř and ž) and Čarka (e.g. á, ý and é) and the bottom row for anything below the letters - which leaves a 3x5 space in the middle to implement the root of each letter - which I managed to do with a couple of exceptions. Here's a quick visualisation of this:

So for example if we want to create the letter "č" it would look something like this:

With only three columns available there are a number of characters which will look a little weird - the worst of which were Đ, § and ď - and the Albanian characters with the tail (ç, ţ and the like). Here's what the font looks like:

It's a little cramped - with some characters appearing to be joined to those above/below - which is due to the fact that the top rows were previously blank and served as line spacing.

I've pushed this into my fork of the Adafruit GFX Library repo on github - check it out and copy the whole thing into your Arduino libraries folder to install it. Note that the Arduino IDE saves files using UTF-8 encoding, so you can't just throw string literals in and assume they'll work - the simplest way to output a string using this character set is to manually enter the hex values. 

So if we want to have the string representation of a nice little Czech sentence, we'd encode it as follows:

When we pass this to display.write() - not display.print(), which will just output the numeric values - we'll see this:

* = Yes I know that we have unicode to properly address this problem, but it's quite complex and is mercifully not used by the Adafruit Graphics library for simplicity's sake.

Courtesy of my very patient girlfriend, who humoured me on Christmas Day and drew a nice snowflake using pix.pe ...

... it is snowing!

The default font for the Adafruit graphics library is a 5x7 font, it looks a little something like this:

It's readable, but I thought it'd be interesting to use a more compact font like "Tom Thumb" by Robey Pointer:

The font currently bundled with the library is in a file called glcdfont.c and is defined as a big byte array:

Each line of that array is 5 bytes and represents a simple monochrome bitmap of a single ascii character, with each byte representing a column. For example example the letter "a" is represented by the following line:

0x20, 0x54, 0x54, 0x78, 0x40,

In binary each of these is:

You can almost see the shape if you twist your head 90 degrees to the right, but to make things a little clearer here's what this actually represents:

If we were to use the Tom Thumb font we could save ourselves two bytes per character, since it is a little slimmer, meaning that "a" could be represented by the following three bytes:

0x68, 0x58, 0x70

Or, in binary:

Which looks like this:

Since there's a bit of wasted space at the top (the top two bits won't get used at all) we could technically save ourselves a little more space, but for simplicity's sake I'm going to sacrifice those two bits. Currently the font can be easily retrieved and manipulated as each column fits neatly into a single byte, however we'd introduce a good deal more complexity by squeezing those additional bits - which would mean substantial changes to the driver itself to compute the correct address to read the bitmap from, and shift/mask the necessary bits.

However even with this tradeoff we can still save 2 bytes per character, which is a pretty impressive 512 bytes over the whole ascii space which is pretty substantial.

The main challenge is creating a separate .c file we can swap glcdfont.c out for. I had trouble dealing with the BDF file Robey shared, but since there's a little bitmap representation of the whole ascii space it wasn't too tough recreating the font by hand ... just extremely tedious! I hacked together a simple little app (source is here) using Processing that let me paint each character by clicking boxes and which spat out the hex values when I hit a key. I'd then copy and paste this into the glcdfont.c. As I said, extremely tedious.

Once I'd produce my file it was time to test it - the beauty of having the 3x6 font means that the entire ascii space could be displayed on a 128x64 screen:

It's actually pretty surprising how much text you can fit on screen.

And just a little reminder of how small this screen is, with a €2 coin for scale

You can see the amount of memory saved in the .text section (where program code lives) by examining binaries GCC has produced - exactly 512 bytes.

My fork of the Adafruit gfx library with the Tom Thumb font is at https://github.com/smcl/Adafruit-GFX-Library and is a drop-in replacement for the existing library (replace the entire library, though, since I made some changes in Adafruit_GFX.cpp. You can switch between the fonts by toggling the #if ... #else condition in glcdfont.h below: