ESP8266 - writing a Lua module for NodeMCU in C

Previously I wrote a little post about getting the NodeMCU firmware built and flashed to an ESP8266 board, with a couple of simple lua examples to get started. 

This is a follow-on exploring what it takes to add some new functionality to the lua libraries, so you can write slightly more complex or low-level functionality in C/assembly and expose it through a lua interface. So this could be considered a nice practical introduction to the C API section of the Programming in Lua book. 

To keep things simple in the previous post we used a prebuilt binary, but for the modifications we want to make we'll need to get the esp SDK and build NodeMCU from source.

Getting the ESP SDK

There's probably a package floating around with the ESP8266 cross-compiler and libraries (actually the processor is an xtensa lx106), but building it isn't too tough. You can either check the documentation on the project's README.md or (if you're on Debian like me) just run the following:

    $ sudo apt-get install make unrar autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python python-serial sed git unzip bash
    $ cd /opt # assuming you have write access here
    $ git clone --recursive https://github.com/pfalcon/esp-open-sdk.git
    $ make

Then add /opt/esp-open-sdk/xtensa-lx106-elf/bin to your PATH and check you can use it:

    $ xtensa-lx106-elf-gcc -dumpmachine
    xtensa-lx106-elf

Building the NodeMCU firmware

Now we've got a working xtensa-lx106 cross-compiler so we can get started on the NodeMCU tools. Change to the directory you want to do development in and checkout the code

    $ cd ~/dev
    $ git clone https://github.com/nodemcu/nodemcu-firmware
    $ cd nodemcu-firmware

We then need to decide what modules to build by editing app/include/user_modules.h and commenting/uncommenting the various macro definitions (LUA_USE_MODULES_NET and friends) to decide what modules will be present. If you want to use the same config built in previous post then this gist can be copy-pasted in.

Once this is done we can build the firmware by calling make, and flash it to the board using make flash:

    $ make && sudo make flash

Creating our module and a simple function - test.identity()

To keep things as clean as possible we'll just create a nice new module - take a look in the app/modules directory and you'll see .c files that roughly match up with the names of the NodeMCU lua modules (take a look at their ReadTheDocs page)

I'm going to very imaginatively use "test" as my module name, so I'll create app/modules/test.c as follows:

And modify my app/include/user_modules.h again so that we include the following definition

    #define LUA_USE_MODULES_TEST

Then rebuild, reflash, reconnect and play about a little to make sure things work as expected:

    > print(test.identity(1))
    1
    > print(test.identity("foo"))
    foo

Take a closer look at test.c - I'll summarise what's going on to the best of my ability. We've defined a function test_identity(lua_State *L), this is the C function we want to expose to the lua environment. Because C and Lua have a slightly different programming model (different type sizes, multiple return value support in Lua etc) the way we pass values from a function call in Lua to our C function is via this lua_State parameter which is a sort of stack:

A major component in the communication between Lua and C is an omnipresent virtual stack. Almost all API calls operate on values on this stack. All data exchange from Lua to C and from C to Lua occurs through this stack

When we want to return values from the C function back to Lua, we push them on the stack and then return an integer which says how many there are. 

We can actually see this in action if we start playing around a bit - when we call this function with more than one arguments you'll get the last one returned - as it'll be at the top of the stack:

    > print(test.identity("foo", "bar", "baz"))
    baz

The way we do this in identity() is by leaving the stack unchanged and saying we return a single value. This will be familiar to anyone who has worked with Forth or other stack machines. Most of the remainder of the file is boilerplate to define our "test" module and the functions available in it - NodeMCU has a handy set of helper macros that let us easily these.

Handling parameters and return values - test.add()

OK now we've got a super-simple example we can move on a little. Let's create a function which accepts two numbers as arguments, adds them together and returns them. In terms of the Lua stack what we want to do is take the top two values, add them together and push the result to the stack. It's better to see this in action - so add this function after test_identity() in test.c:

    // test.add(x,y) - take two values and add them
    static int test_add(lua_State *L) {
      double x = lua_tonumber(L, 1);
      double y = lua_tonumber(L, 2);
      lua_pushnumber(L, x+y);
      return 1;
    } 

It might be a little clearer to see the interaction with the stack here. We read a couple of values (x and y) using lua_tonumber() and then push the result to the stack to be returned using lua_pushnumber(). There are a handful of different functions to get differently typed values from the stack:

    int            lua_toboolean (lua_State *L, int index);
    double         lua_tonumber (lua_State *L, int index);
    const char    *lua_tostring (lua_State *L, int index);
    size_t         lua_strlen (lua_State *L, int index);

And to push different types to the stack to return them there are some more which should be self-explanatory:

    void lua_pushnil (lua_State *L);
    void lua_pushboolean (lua_State *L, int bool);
    void lua_pushnumber (lua_State *L, double n);
    void lua_pushlstring (lua_State *L, const char *s, size_t length);
    void lua_pushstring (lua_State *L, const char *s);
Since we've added a new C function we need to add a new row to the test_map initialiser so that we can call it from lua - so to make sure test_add() can be called using test.add() we'll update it like so:
    static const LUA_REG_TYPE test_map[] = {
      { LSTRKEY( "identity" ),         LFUNCVAL( test_identity ) },
      { LSTRKEY( "add" ),              LFUNCVAL( test_add ) }, // this is our new line
      { LSTRKEY( "__metatable" ),      LROVAL( test_map ) },
      { LNILKEY, LNILVAL }
    };
If we re-build, reflash and reconnect to the ESP8266 we should now be able to call test.add():
    > test.add(1, 2)
    3 
    > test.add(1, test.add(2, 3)
    6

Testing the type of values on the stack - test.add2()

If we're feeling a bit mischevous we could extend this example so we have a sort of overloaded function - we can check the parameter types and perform some different actions depending on them. We can test the types of the arguments using lua_is* functions (we need to include c_stdlib.h, since we're using c_zalloc):
    // test.add2(x,y) - take two values and add them, or take two strings and concat them
    static int test_add2(lua_State *L) {
      if (lua_isnumber(L, 1) && lua_isnumber(L, 2)) {     
        double x = lua_tonumber(L, 1);
        double y = lua_tonumber(L, 2);
        lua_pushnumber(L, x+y);
      } else if (lua_isstring(L, 1) && lua_isstring(L, 2)) {
        const char *s1 = lua_tostring(L, 1);
        const char *s2 = lua_tostring(L, 2);
        char *s = c_zalloc(strlen(s1) + strlen(s2));
        c_sprintf(s, "%s%s", s1, s2);
        lua_pushstring(L, s);
      } else {
        lua_pushnil(L);
      }
      return 1;
    }
Be careful since the lua_isstring is a wee bit cheeky - it appears that if numbers can be parsed as strings in this context. And also note that this would be a pretty nasty way to design a function, but as an illustrative example it's fine. The full complement of these functions below:
    int lua_isnumber (lua_State *L, int index);
    int lua_isstring (lua_State *L, int index);
    int lua_isboolean (lua_State *L, int index);
    int lua_isnil (lua_State *L, int index);
    int lua_isfunction (lua_State *L, int index);

Handling Lua function callbacks - test.ping()

So now we're comfortable with the Lua functions in C we can get into a slightly tougher example. What we want to do is pass a function in and use it as a callback later on. This means interacting with the lua registry - which is a key/value store for Lua objects (way better description can be found here):

As ever a little demonstration is the best way - when we want to test whether element 3 on the stack is a function and if it is store it in the registry:

    int my_ref;
    if (lua_isfunction(L, 3))
        my_ref = luaL_ref(L, LUA_REGISTRYINDEX);

When we want to retrieve the value, we use lua_rawgeti() which will retrieve it and push it to the stack:

    lua_rawgeti(L, LUA_REGISTRYINDEX, my_ref);

So if our lua function is at the top of the stack, we can call it using lua_call():

    int num_args = 1;
    int num_retvals = 0;
    lua_call(gL, num_args, num_retvals);

The second argument to lua_call is used to indicate how many arguments we've pushed to the stack to pass to the function, we're also specifying how many values we expect to be returned.

To tie all this together we can wrap the ping example which comes with lwIP (lwIP is the IP stack used by NodeMCU). It's hard to take each part piece-by-piece so I'm just going to dump the final ping functionality in-place in a GitHub fork which you can access like so:

    $ git clone https://github.com/smcl/nodemcu-firmware
    $ git checkout -b dev origin/dev
    $ make && sudo make flash

The commit in question which demonstrates the changes (there were a handful of lwip headers needing fixed) is here if you want it highlighted.

If we wanted to ping google.com, and for each response print the ping details and flash the ESP8266 board's LED we could write something like this:

    > function ping_recv(bytes, ipaddr, seqno, ttl, ping)
    >> output = string.format("%d bytes from %s: icmp_seq = %d, ttl=%d, time=%d ms", bytes, ipaddr, seqno, ttl, ping)
    >> print(output)
    >> gpio.mode(4, gpio.OUTPUT)
    >> gpio.write(4, gpio.LOW) 
    >> tmr.alarm(0, 50, tmr.ALARM_SINGLE, function() gpio.write(4, gpio.HIGH) end)
    >> end
    > test.ping("google.com", 3, ping_recv)
    > 32 bytes from 216.58.214.206: icmp_seq = 1, ttl=53, time=14 ms
    32 bytes from 216.58.214.206: icmp_seq = 2, ttl=53, time=14 ms
    32 bytes from 216.58.214.206: icmp_seq = 3, ttl=53, time=14 ms

Conclusion

There's a reason that Lua widely deployed as an extension language - it's very easy to write Lua and it only takes a medium-sized blog post to describe how to create new Lua modules in C. And since C tends to have an FFI for most languages/libraries we can quickly expose a powerful yet lightweight programming interface for almost any application.

However, even though there are a number modules (checkout the ESP8266 page on ReadTheDocs) for different bits of hardware, I'm actually surprised there aren't more given how simple it was to hook all this up.

If you'd like some more in-depth Lua documentation you really cannot go wrong with Programming in Lua. In addition there's a nice quick-reference over at Luai. The homepage might seem a little overwhelming but once you drill into a single section, like Section 4.1 Functions and Types, you see that it's pretty well laid out.

If you'd like to see more complex examples then take a look through the sources in the app/modules directory. They're not so hard to wrap your head around since lwIP and the board's built-in wifi functions do most of the heavy lifting. A lot of what I learned when writing this post was just from scanning through these files.

I'm still keen to spend more time on the ESP8266 - though in some ways for me it's building up a set of skills I'm not sure I'll ever use - but I'm enjoying myself, and if even a single person reads through this and finds it useful it'll have been worth it.

ESP8266 - getting started

I picked up a couple of ESP8266 modules - an inexpensive little SoC with WiFi - from AliExpress but ran into a couple of weird issues when trying to get them up and running. A lot of documentation is a little confusing, since it usually begins with flashing an image even though when you power it most likely starts up an open WiFi access point you can connect to. 

Anyway here are the issues I ran into, in order - coincidentally they capture a pretty simple workflow for getting started on the ESP8266.

Issue 0 - uhh ... what do I connect to where?

Update: I didn't know this at the time, but you can simply install the USB to UART bridge drivers from Silicon Labs and connect to the board via USB - on my laptop it appears as /dev/SLAB_USBtoUART. Using this method you can skip this whole section. 

There's dozens of different ESP8266 boards, each with their own little configurations, layouts and labels. Since I bought unbranded boards from AliExpress I didn't really know the search terms to take me to a nice simple diagram of what needs to be connected to each pin of my TTL cable. Anyway my boards look like this:

They connect to my TTL cable a little like this

Just in case it isn't clear from the picture - since I stupidly didn't match up all the colours - here's exactly what is connected to what:

<create table showing connections>

Even though there's a micro-USB connector this set of connections is sufficient to bring up the board and send some simple commands to it.

Issue 1 - connecting via serial (screen/OS X)

There's a very key bit of info for OS X users that's usually skipped over. When you attempt to connect to the ESP8266 using a USB TTL cable and screen you can be mistaken for thinking you've messed something up or you've got a dud chip, as it fails to respond to any input:

Sending an "AT" should result in the response "OK", but in this case the cursor just jumped to the left of the screen and nothing else happened. What's happened here is that GNU Screen on OS X is handling the Return key weirdly, sending only a CR character instead of CR+LF. I played around with the man page for a while but couldn't get it to play nice - eventually I downloaded CoolTerm which did the job nicely.

Issue 2 - firmware

After you've confirmed the chip is up and running by successfully issuing an "AT" you'll probably want to do something a little more useful with it. Since you likely don't speak AT (though a reference is here if you want to try) something simpler like Lua might be suitable. This will involve downloading a binary (or building it yourself, if you feel adventurous), and setting up a couple of things in python.

To save the bother of setting up dev tools for now, we'll use a service which will build a nice little NodeMCU image for you - http://nodemcu-build.com. Here's the build config I used:

Once it's complete you'll receive an email detailing the download link - I received two and opted to use the Integer one, which worked out nicely

To let us flash the ESP8266 we need to connect up the GPIO0 pin (which is labelled "D3" on my board) and reboot it so that it's in "flash" mode - here I've done this with the white cable.

Next we'll need to open up a terminal and setup pyserial and esptool, then download our image and flash it. Both pyserial and esptool can be retrieved via git - I'm just gonna be brief with this part as I've already spent a lot of time with screenshots and pictures already.

$ git clone https://github.com/pyserial/pyserial
Cloning into 'pyserial'...
remote: Counting objects: 4305, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4305 (delta 0), reused 0 (delta 0), pack-reused 4301
Receiving objects: 100% (4305/4305), 1.06 MiB | 1.01 MiB/s, done.
Resolving deltas: 100% (3207/3207), done.
Checking connectivity... done.
$ cd pyserial
$ sudo python setup.py install
Password:
running install
... lots of lines later ...
Installed /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyserial-3.0.1-py2.7.egg
Processing dependencies for pyserial==3.0.1
Finished processing dependencies for pyserial==3.0.1
$ git clone https://github.com/themadinventor/esptool
Cloning into 'esptool'...
remote: Counting objects: 293, done.
remote: Total 293 (delta 0), reused 0 (delta 0), pack-reused 293
Receiving objects: 100% (293/293), 106.48 KiB | 0 bytes/s, done.
Resolving deltas: 100% (157/157), done.
Checking connectivity... done.
$ cd esptool

Now we can download our NodeMCU binary, double-check the path of our USB TTL device (since I always forget) and start flashing

$ curl -LO http://nodemcu-build.com/builds/nodemcu-master-7-modules-2016-01-18-22-47-15-integer.bin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  388k  100  388k    0     0   247k      0  0:00:01  0:00:01 --:--:--  633k
$ ls /dev/tty.usbserial* 
/dev/tty.usbserial-A402O00H
$ python esptool.py --port /dev/tty.usbserial-A402O00H write_flash 0x00000 ./nodemcu-master-7-modules-2016-01-18-22-47-15-integer.bin 
Connecting...
Erasing flash...
Took 1.90s to erase flash block
Wrote 398336 bytes at 0x00000000 in 43.5 seconds (73.2 kbit/s)...

Leaving...
$ 

Now the board should be ready to reboot - if we go ahead and reboot it then connect using CoolTerm (changing to use 9600 baud instead of the 115200 we previously used) there should be a nice little Lua prompt waiting.

Issue 3 - what now?

So what cool things can we do now with our tiny little WiFi chip? Well firstly let's understand and tackle that pesky cannot find init.lua error message. After the NodeMCU firmware boots it searches for a file in the onboard eeprom called "init.lua" whose purpose is to allow you to run some lua code after boot, so you can setup a simple program or define some useful functions without having to get your hands dirty and rebuild the NodeMCU firmware and reflash.

So if we want to create a simple init.lua which prints out a little message on boot, here's what we can do.

$ git clone https://github.com/4refr0nt/luatool
$ cd luatool/luatool
$ echo 'print("check out my cool new ESP8266!")' > ./init.lua
$ ./luatool.py --port /dev/tty.usbserial-A402O00H --src init.lua --dest init.lua --verbose
Upload starting
Stage 1. Deleting old file from flash memory
->file.open("init.lua", "w") -> ok
->file.close() -> ok

Stage 2. Creating file in flash memory and write first line->file.remove("init.lua") -> ok

Stage 3. Start writing data to flash memory...->file.open("init.lua", "w+") -> ok

Stage 4. Flush data and closing file->file.writeline([==[print("check out my cool new ESP8266!")]==]) -> ok
->file.flush() -> ok
->file.close() -> ok
--->>> All done <<<---
$

Now any time you reboot your ESP8266 here's what you'll see instead of that error message:

There are a few nice examples at http://nodemcu.com, which you can either key in manually or upload as part of an init.lua file so that they always run on startup - just be careful you don't leave your ESP8266 in an infinite loop or else you'll have to reflash the firmware and start again. To save you the trouble, here are a couple of simple of examples, all of which involve connecting to a wifi network and serving some data.

The first is a simple hit counter, which records the number of requests it has served, and displays these on a simple page. Send the file using luatool similar to the previous example (changing --src parameter but ensuring --dest is still "init.lua"). Here's what you'll see after visiting the page - because Chrome attempts to fetch both the page and favicon.ico each hit is registered twice :)

Code: hitCount.lua

The second example allows you to switch the onboard LED on or off using a button on the page (which sends a POST request via AJAX). Because we're just dealing with raw TCP requests, I use a very inefficient/wrong way to decide whether or not we're handling a POST request - but since it's just a demonstration it's ok.

After loading the page and toggling the LED here's what we see:

Code (program): ledToggle.lua

Code (webpage): led.html

So there we have it - a simple introduction to connecting to an ESP8266 board, flashing with NodeMCU firmware and a couple of little pointers on how to use luatool to create a lua script which runs at startup. To find out more about the libraries available on NodeMCU check out their page on Read The Docs or for a little more info about the Lua programming language check out Programming in Lua 1st Edition

I seem to have written a few too many "Getting Started" guides, however I'll be spending a fair bit of time in the future on the ESP8266 as it offers plenty of opportunities for interesting work.