Previously I wrote a little post about getting the NodeMCU firmware built and flashed to an ESP8266 board, with a couple of simple lua examples to get started.
This is a follow-on exploring what it takes to add some new functionality to the lua libraries, so you can write slightly more complex or low-level functionality in C/assembly and expose it through a lua interface. So this could be considered a nice practical introduction to the C API section of the Programming in Lua book.
To keep things simple in the previous post we used a prebuilt binary, but for the modifications we want to make we'll need to get the esp SDK and build NodeMCU from source.
Getting the ESP SDK
There's probably a package floating around with the ESP8266 cross-compiler and libraries (actually the processor is an xtensa lx106), but building it isn't too tough. You can either check the documentation on the project's README.md or (if you're on Debian like me) just run the following:
$ sudo apt-get install make unrar autoconf automake libtool gcc g++ gperf flex bison texinfo gawk ncurses-dev libexpat-dev python python-serial sed git unzip bash
$ cd /opt # assuming you have write access here
$ git clone --recursive https://github.com/pfalcon/esp-open-sdk.git
$ make
Then add /opt/esp-open-sdk/xtensa-lx106-elf/bin to your PATH and check you can use it:
$ xtensa-lx106-elf-gcc -dumpmachine
xtensa-lx106-elf
Building the NodeMCU firmware
Now we've got a working xtensa-lx106 cross-compiler so we can get started on the NodeMCU tools. Change to the directory you want to do development in and checkout the code
$ cd ~/dev
$ git clone https://github.com/nodemcu/nodemcu-firmware
$ cd nodemcu-firmware
We then need to decide what modules to build by editing app/include/user_modules.h and commenting/uncommenting the various macro definitions (LUA_USE_MODULES_NET and friends) to decide what modules will be present. If you want to use the same config built in previous post then this gist can be copy-pasted in.
Once this is done we can build the firmware by calling make, and flash it to the board using make flash:
$ make && sudo make flash
Creating our module and a simple function - test.identity()
To keep things as clean as possible we'll just create a nice new module - take a look in the app/modules directory and you'll see .c files that roughly match up with the names of the NodeMCU lua modules (take a look at their ReadTheDocs page)
I'm going to very imaginatively use "test" as my module name, so I'll create app/modules/test.c as follows:
And modify my app/include/user_modules.h again so that we include the following definition
#define LUA_USE_MODULES_TEST
Then rebuild, reflash, reconnect and play about a little to make sure things work as expected:
> print(test.identity(1))
1
> print(test.identity("foo"))
foo
Take a closer look at test.c - I'll summarise what's going on to the best of my ability. We've defined a function test_identity(lua_State *L), this is the C function we want to expose to the lua environment. Because C and Lua have a slightly different programming model (different type sizes, multiple return value support in Lua etc) the way we pass values from a function call in Lua to our C function is via this lua_State parameter which is a sort of stack:
A major component in the communication between Lua and C is an omnipresent virtual stack. Almost all API calls operate on values on this stack. All data exchange from Lua to C and from C to Lua occurs through this stack
When we want to return values from the C function back to Lua, we push them on the stack and then return an integer which says how many there are.
We can actually see this in action if we start playing around a bit - when we call this function with more than one arguments you'll get the last one returned - as it'll be at the top of the stack:
> print(test.identity("foo", "bar", "baz"))
baz
The way we do this in identity() is by leaving the stack unchanged and saying we return a single value. This will be familiar to anyone who has worked with Forth or other stack machines. Most of the remainder of the file is boilerplate to define our "test" module and the functions available in it - NodeMCU has a handy set of helper macros that let us easily these.
Handling parameters and return values - test.add()
OK now we've got a super-simple example we can move on a little. Let's create a function which accepts two numbers as arguments, adds them together and returns them. In terms of the Lua stack what we want to do is take the top two values, add them together and push the result to the stack. It's better to see this in action - so add this function after test_identity() in test.c:
// test.add(x,y) - take two values and add them
static int test_add(lua_State *L) {
double x = lua_tonumber(L, 1);
double y = lua_tonumber(L, 2);
lua_pushnumber(L, x+y);
return 1;
}
It might be a little clearer to see the interaction with the stack here. We read a couple of values (x and y) using lua_tonumber() and then push the result to the stack to be returned using lua_pushnumber(). There are a handful of different functions to get differently typed values from the stack:
int lua_toboolean (lua_State *L, int index);
double lua_tonumber (lua_State *L, int index);
const char *lua_tostring (lua_State *L, int index);
size_t lua_strlen (lua_State *L, int index);
And to push different types to the stack to return them there are some more which should be self-explanatory:
void lua_pushnil (lua_State *L);
void lua_pushboolean (lua_State *L, int bool);
void lua_pushnumber (lua_State *L, double n);
void lua_pushlstring (lua_State *L, const char *s, size_t length);
void lua_pushstring (lua_State *L, const char *s); static const LUA_REG_TYPE test_map[] = {
{ LSTRKEY( "identity" ), LFUNCVAL( test_identity ) },
{ LSTRKEY( "add" ), LFUNCVAL( test_add ) }, // this is our new line
{ LSTRKEY( "__metatable" ), LROVAL( test_map ) },
{ LNILKEY, LNILVAL }
};
> test.add(1, 2)
3
> test.add(1, test.add(2, 3)
6
Testing the type of values on the stack - test.add2()
// test.add2(x,y) - take two values and add them, or take two strings and concat them
static int test_add2(lua_State *L) {
if (lua_isnumber(L, 1) && lua_isnumber(L, 2)) {
double x = lua_tonumber(L, 1);
double y = lua_tonumber(L, 2);
lua_pushnumber(L, x+y);
} else if (lua_isstring(L, 1) && lua_isstring(L, 2)) {
const char *s1 = lua_tostring(L, 1);
const char *s2 = lua_tostring(L, 2);
char *s = c_zalloc(strlen(s1) + strlen(s2));
c_sprintf(s, "%s%s", s1, s2);
lua_pushstring(L, s);
} else {
lua_pushnil(L);
}
return 1;
}
int lua_isnumber (lua_State *L, int index);
int lua_isstring (lua_State *L, int index);
int lua_isboolean (lua_State *L, int index);
int lua_isnil (lua_State *L, int index);
int lua_isfunction (lua_State *L, int index);
Handling Lua function callbacks - test.ping()
So now we're comfortable with the Lua functions in C we can get into a slightly tougher example. What we want to do is pass a function in and use it as a callback later on. This means interacting with the lua registry - which is a key/value store for Lua objects (way better description can be found here):
As ever a little demonstration is the best way - when we want to test whether element 3 on the stack is a function and if it is store it in the registry:
int my_ref;
if (lua_isfunction(L, 3))
my_ref = luaL_ref(L, LUA_REGISTRYINDEX);When we want to retrieve the value, we use lua_rawgeti() which will retrieve it and push it to the stack:
lua_rawgeti(L, LUA_REGISTRYINDEX, my_ref);
So if our lua function is at the top of the stack, we can call it using lua_call():
int num_args = 1;
int num_retvals = 0;
lua_call(gL, num_args, num_retvals);The second argument to lua_call is used to indicate how many arguments we've pushed to the stack to pass to the function, we're also specifying how many values we expect to be returned.
To tie all this together we can wrap the ping example which comes with lwIP (lwIP is the IP stack used by NodeMCU). It's hard to take each part piece-by-piece so I'm just going to dump the final ping functionality in-place in a GitHub fork which you can access like so:
$ git clone https://github.com/smcl/nodemcu-firmware
$ git checkout -b dev origin/dev
$ make && sudo make flash
The commit in question which demonstrates the changes (there were a handful of lwip headers needing fixed) is here if you want it highlighted.
If we wanted to ping google.com, and for each response print the ping details and flash the ESP8266 board's LED we could write something like this:
> function ping_recv(bytes, ipaddr, seqno, ttl, ping)
>> output = string.format("%d bytes from %s: icmp_seq = %d, ttl=%d, time=%d ms", bytes, ipaddr, seqno, ttl, ping)
>> print(output)
>> gpio.mode(4, gpio.OUTPUT)
>> gpio.write(4, gpio.LOW)
>> tmr.alarm(0, 50, tmr.ALARM_SINGLE, function() gpio.write(4, gpio.HIGH) end)
>> end
> test.ping("google.com", 3, ping_recv)
> 32 bytes from 216.58.214.206: icmp_seq = 1, ttl=53, time=14 ms
32 bytes from 216.58.214.206: icmp_seq = 2, ttl=53, time=14 ms
32 bytes from 216.58.214.206: icmp_seq = 3, ttl=53, time=14 ms
Conclusion
There's a reason that Lua widely deployed as an extension language - it's very easy to write Lua and it only takes a medium-sized blog post to describe how to create new Lua modules in C. And since C tends to have an FFI for most languages/libraries we can quickly expose a powerful yet lightweight programming interface for almost any application.
However, even though there are a number modules (checkout the ESP8266 page on ReadTheDocs) for different bits of hardware, I'm actually surprised there aren't more given how simple it was to hook all this up.
If you'd like some more in-depth Lua documentation you really cannot go wrong with Programming in Lua. In addition there's a nice quick-reference over at Luai. The homepage might seem a little overwhelming but once you drill into a single section, like Section 4.1 Functions and Types, you see that it's pretty well laid out.
If you'd like to see more complex examples then take a look through the sources in the app/modules directory. They're not so hard to wrap your head around since lwIP and the board's built-in wifi functions do most of the heavy lifting. A lot of what I learned when writing this post was just from scanning through these files.
I'm still keen to spend more time on the ESP8266 - though in some ways for me it's building up a set of skills I'm not sure I'll ever use - but I'm enjoying myself, and if even a single person reads through this and finds it useful it'll have been worth it.