Friday 15 April 2016

How to send emails via gmail from an ESP8266 running NodeMCU

How to send smtp emails via gmail from an ESP8266 running NodeMCU

SSL Support

When I found out about NodeMCU's SSL support (yes: I am very late to the party), one of the first things to try was sending mails. There are web services that will do that for you, but I don't like to have yet another party involved. So I needed SMTP through an SSL connection,
There is an implementation in C here in the forums, but I couldn not find anything ready-made for NodeMCU.
What I did find, was a very nicely written LUA script from "Miguel" in the NodeMCU LUA examples. This only needed a few minor modifications to run on the current DEV-version of NodeMCU:

NodeMCU custom build by frightanic.com
branch: dev
commit: 3f418f995cfccbaf7a745e65c81251c4c50759e6
SSL: true
modules: adc,crypto,file,gpio,http,i2c,net,node,tmr,u8g,uart,wifi
 build built on: 2016-04-11 20:31
 powered by Lua 5.1.4 on SDK 1.5.1(e67da894)

Not all of the modules are really used in this script, of course. So you can trim that down a bit.

Example mail on iPhone


Send an e-mail

With all of that in place, it only took a few minutes to have the first mail sent from my ESP8266-DEV board.
So here is the code for you to try:


 -- Modifications for GMAIL by Andreas "Andy" Reischle: www.AReResearch.net  
 -- See https://support.google.com/a/answer/176600?hl=de for details on smtp with gmail  
 -- Now that NodeMCU has working SSL support, we can also talk to email services that  
 -- require encryption.   
 -- Caveat: I have not looked into the SSL implementation, but I suspect it is vulnerable  
 -- to man-in-the-middle attacks as the client doesn't check the server's certificate.  
 -- 20160415 ARe  
 --------Original Credits:  
 --------  
 ------- Working Example: https://www.youtube.com/watch?v=CcRbFIJ8aeU  
 ------- @description a basic SMTP email example. You must use an account which can provide unencrypted authenticated access.  
 ------- This example was tested with an AOL and Time Warner email accounts. GMail does not offer unecrypted authenticated access.  
 ------- To obtain your email's SMTP server and port simply Google it e.g. [my email domain] SMTP settings  
 ------- For example for timewarner you'll get to this page http://www.timewarnercable.com/en/support/faqs/faqs-internet/e-mailacco/incoming-outgoing-server-addresses.html  
 ------- To Learn more about SMTP email visit:  
 ------- SMTP Commands Reference - http://www.samlogic.net/articles/smtp-commands-reference.htm  
 ------- See "SMTP transport example" in this page http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol  
 ------- @author Miguel  
 --no longer required because it is part of the crypto module: require("base64")  
 -- The email and password from the account you want to send emails from  
 local MY_EMAIL = "YOURMAILADDRESS@gmail.com"  
 local EMAIL_PASSWORD = "YOURPASSWORD"  
 -- The SMTP server and port of your email provider.  
 -- If you don't know it google [my email provider] SMTP settings  
 local SMTP_SERVER = "smtp.gmail.com"  
 local SMTP_PORT = "465"  
 -- The account you want to send email to  
 local mail_to = "RECIPIENT@HISDOMAIN.COM"  
 -- Your access point's SSID and password  
 local SSID = "YOURWIFISSID"  
 local SSID_PASSWORD = "SECRET-I-WONT-TELL"  
 -- configure ESP as a station  
 wifi.setmode(wifi.STATION)  
 wifi.sta.config(SSID,SSID_PASSWORD)  
 wifi.sta.autoconnect(1)  
 -- These are global variables. Don't change their values  
 -- they will be changed in the functions below  
 local email_subject = ""  
 local email_body = ""  
 local count = 0  
 local smtp_socket = nil -- will be used as socket to email server  
 -- The display() function will be used to print the SMTP server's response  
 function display(sck,response)  
    print("Got a response: ")  
    print(response)  
 end  
 -- The do_next() function is used to send the SMTP commands to the SMTP server in the required sequence.  
 -- I was going to use socket callbacks but the code would not run callbacks after the first 3.  
 function do_next()  
       if(count == 0)then  
         count = count+1  
         local IP_ADDRESS = wifi.sta.getip()  
         print ("Send my IP: " .. IP_ADDRESS)  
         smtp_socket:send("HELO "..IP_ADDRESS.."\r\n")  
       elseif(count==1) then  
         count = count+1  
         smtp_socket:send("AUTH LOGIN\r\n")  
       elseif(count == 2) then  
         count = count + 1  
         smtp_socket:send(crypto.toBase64(MY_EMAIL).."\r\n")  
       elseif(count == 3) then  
         count = count + 1  
         smtp_socket:send(crypto.toBase64(EMAIL_PASSWORD).."\r\n")  
       elseif(count==4) then  
         count = count+1  
         smtp_socket:send("MAIL FROM:<" .. MY_EMAIL .. ">\r\n")  
       elseif(count==5) then  
         count = count+1  
         smtp_socket:send("RCPT TO:<" .. mail_to ..">\r\n")  
       elseif(count==6) then  
         count = count+1  
         smtp_socket:send("DATA\r\n")  
       elseif(count==7) then  
         count = count+1  
         local message = string.gsub(  
         "From: \"".. MY_EMAIL .."\"<"..MY_EMAIL..">\r\n" ..  
         "To: \"".. mail_to .. "\"<".. mail_to..">\r\n"..  
         "Subject: ".. email_subject .. "\r\n\r\n" ..  
         email_body,"\r\n.\r\n","")  
         smtp_socket:send(message.."\r\n.\r\n")  
       elseif(count==8) then  
         count = count+1  
          tmr.stop(0)  
          smtp_socket:send("QUIT\r\n")  
       else  
         smtp_socket:close()  
       end  
 end  
 -- The connectted() function is executed when the SMTP socket is connected to the SMTP server.  
 -- This function will create a timer to call the do_next function which will send the SMTP commands  
 -- in sequence, one by one, every 5000 seconds.   
 -- You can change the time to be smaller if that works for you, I used 5000ms just because.  
 function connected(sck)  
   print("Connected - Starting Timer")  
   tmr.alarm(0,5000,1,do_next)  
 end  
 -- @name send_email  
 -- @description Will initiated a socket connection to the SMTP server and trigger the connected() function  
 -- @param subject The email's subject  
 -- @param body The email's body  
 function send_email(subject,body)  
    count = 0  
    email_subject = subject  
    email_body = body  
    print ("Open Connection")  
    smtp_socket = net.createConnection(net.TCP,1)  
    smtp_socket:on("connection",connected)  
    smtp_socket:on("receive",display)  
    smtp_socket:connect(SMTP_PORT,SMTP_SERVER)  
 end  
 -- Send an email  
 print ("Sending started...")  
 send_email("ESP8266-GMailSender","Hi there!")  

This will need a little tidying, but will certainly make it into one of my projects.

Caveat:

NodeMCU's SSL implementation does currently not check the server's certificate. So I suspect man-in-the-middle attacks are easy.

26 comments:

  1. hi,

    thanks for the post.
    i tried this code with the following settings

    NodeMCU custom build by frightanic.com
    branch: master
    commit: c8037568571edb5c568c2f8231e4f8ce0683b883
    SSL: true
    modules: adc,bit,crypto,file,gpio,i2c,net,node,tmr,u8g,uart,wifi
    build built on: 2016-05-17 18:16
    powered by Lua 5.1.4 on SDK 1.4.0

    there is no email in my mail box, but there is only the following message

    Sending started...
    Open Connection
    > DNS retry 1!

    can you help me with this issue?

    ReplyDelete
    Replies
    1. As the message suggests, it might be a DNS issue. Use "173.194.200.108" in the SMTP_SERVER variable. That should eliminate the need for DNS. I haven't tried that, though.

      Delete
  2. Hi, I upload this to the NodeMCU and all I get is

    Sending started...
    Open Connection

    Does not even appear to start sending...

    Any ideas?

    ReplyDelete
  3. Hi,

    I tried this and all I get is

    Sending started...
    Open Connection

    any ideas?

    ReplyDelete
    Replies
    1. That sounds like the "function send_email(subject,body)" is started. But it never gets to the point where it starts transmitting anything and "function connected(sck)" is never called. That can only be when the TCP connection to "smtp.gmail.com" on port 465 fails.
      Try if you can telnet to that port from your PC. Also: replace smtp.gmail.com with the IP 173.194.200.108 to eliminate DNS issues.

      Delete
  4. Hi,

    I used some of your code in one of my projects. You mention something about socket callbacks in the comments but just say that it didn't work past 3. I tried moving the do_next() to the display function, and it worked perfectly. It now takes 1 second to send my email rather than the 45 seconds or so that it took before. I'm not sure if this is what you tried and it didn't work on earlier firmwares or if you were doing something else. I would be interested to know.

    ReplyDelete
    Replies
    1. Thanks, that is great info. I'll try if I can replicate that.

      Delete
  5. Did you check "Miscellaneous options/ SSL support" when requesting you custom build ? I am getting looping boot info about cksum after flashing.
    Thanks,
    Curt

    ReplyDelete
  6. Success report ! I was able to use this code to successfully send email from both a esp8266-01 and from a -12e Devkit (in spite of warnings the -01 might not have enough memory) The key for me was adding,in addition to the new build at 0x0000), the esp_init_data_default.bin at 0x3fc000. Initially after a flash I was getting the esp8266 led flashing rapidly outputing error code on 74880 baud via the Lua Loader serial output. Then I did a flash with the new firmware at 0x0000 and esp_init_data_default.bin at 0x3fc000. same result. Then, after another flash with just the esp_init_data_default.bin at 0x3fc000 again, it worked. (For the custom build, I checked the SSL box in below the main selections). BTW, does anyone know how to load a library module that is referenced by "require()" ?
    Curt

    ReplyDelete
    Replies
    1. Great to hear it works for you. About including a library module: Have a look at the example here: http://www.electrodragon.com/w/ESP8266_NodeMCU_Lua
      But I never tried that myself.

      Delete
  7. Hi,

    I've been working with this code for some days. Today, out of the blue, the SMTP server didn't accept the authentication anymore. After some try and error I had to figure out that it was the fact that I had installed a float firmware in the meantime. Changing back to the integer firmware, everything was fine again. Any ideas or suggestions?

    Best regards

    Heribert

    ReplyDelete
  8. Unless the float version has no SSL-Support built in, I don't know. I hardly ever use float in my projects.

    ReplyDelete
  9. Hello Andreas,

    I am facing problems with custom builds for my ESP12E 4MB WIFI board.

    Needless to say that it works quite right with older releases like nodemcu_integer_0.9.6-dev_20150704.bin but everytime I need to flash a custom build, it desperately fails ...

    In some cases, the board is just like dead, wheras in other cases, the blue LED keeps blinking fast and the ESP keeps sending rubbish data to the ESPLORER whatever the selected speed (I tried all speeds from 2400bds up to 115200bds).

    I am now quite lost with these custom builds but I really need to use them for I want to include modules like adc,crypto,file,gpio,http,i2c,net,node,tmr,u8g,uart,wifi as suggested in this blog !

    I have carefully read the recommended upgrade notes like erasing my board before flashing (esptool.py --port com2 erase_flash) but I must confess that I did not understand this part :
    Also verify that you are using an up-to-date NodeMCU release, as some early releases of NodeMCU 1.5.4.1 did not write the SDK init data to a freshly erased chip.

    What would you suggest me to try ?

    Thanks

    ReplyDelete
  10. Hi,
    Following my previous post, I have finally solved my ESP8266 'custom build' problems (thanks to Marcel Stör here >>> http://stackoverflow.com/questions/38789491 ).
    Now that I have installed on my ESP12E the proper FW (with the right biolt-in modules), I'll be soon able to test your SMTP Lua script !
    I'll post follow-ups here ...

    ReplyDelete
    Replies
    1. Very interesting. I was not aware of that and resorted to flashing AT-Firmware before re-flashing NodeMCU. But you pointed to a much cleaner solution. Thanks!

      Delete
  11. The following python commands ***DO*** work everytime, at least on ESP12E MCU boards :
    esptool.py --port erase_flash
    esptool.py --port write_flash -fm dio -fs 32m 0x00000 .bin 0x3fc000 esp_init_data_default.bin
    ---
    I spent nights and days before I could understand that I had to flash again the esp_init_data_default.bin along with a custom-built FW !!

    Espressif changes the init data block (esp_init_data_default.bin) for their devices along the way with the SDK. So things break when a NodeMCU firmware with a certain SDK is flashed to a module which contains init data from a different SDK.

    ReplyDelete
  12. Hi Andreas,

    It definitely works quite well !!!!
    The mods you made to have the script usable on Gmail SSL SMTP works 100%.

    Now, I would like to find a ***simple*** an easy way to hide "somehow" the two following variables :
    local EMAIL_PASSWORD = "xxxxxxxx"
    local SSID_PASSWORD = "yyyyyyyyyyyyy"

    I know there is no 100%-safe way to do that in a Lua script (well, that's what I read) but maybe you would have a clue to make it ***a bit*** safer within the script as well as when it runs (big-ear proof) ...

    Thanks.

    ReplyDelete
    Replies
    1. The traditional, simple way to obfuscate passwords is to XOR them with a string. There is a NodeMCU / LUA example here:
      http://stackoverflow.com/questions/34243734/how-to-prevent-wifi-password-from-being-leaked-from-lua-code

      Delete
  13. I'll look over this, thanks alot.

    ReplyDelete
  14. Hi Andreas,
    I receive always the same error

    PANIC: unprotected error in call to Lua API (SendGMAIL.lua:75: attempt to concatenate upvalue '?' (a nil value))

    The lines are:
    local message = string.gsub(
    "From: \"".. MY_EMAIL .."\"<"..MY_EMAIL..">\r\n" ..
    "To: \"".. mail_to .. "\"<".. mail_to..">\r\n"..
    "Subject: ".. email_subject .. "\r\n\r\n" ..
    email_body,"\r\n.\r\n","")

    My LUA version is:

    NodeMCU custom build by frightanic.com
    branch: dev
    commit: 016f289f315a6caf01509d83ec31bc9dd900835b
    SSL: true
    modules: adc,crypto,file,gpio,http,i2c,net,node,tmr,uart,wifi
    build built on: 2016-11-30 18:08
    powered by Lua 5.1.4 on SDK 1.5.4.1(39cb9a32)


    So what's wrong?

    Thanks

    ReplyDelete
  15. Hi Andreas,
    I'm trying to send Gmails but nothing happens.
    This is the error:

    PANIC: unprotected error in call to Lua API (SendGMAIL.lua:75: attempt to concatenate upvalue '?' (a nil value))

    And those are the rows:

    71 local message = string.gsub(
    72 "From: \"".. MY_EMAIL .."\"<"..MY_EMAIL..">\r\n" ..
    73 "To: \"".. mail_to .. "\"<".. mail_to..">\r\n"..
    74 "Subject: ".. email_subject .. "\r\n\r\n" ..
    75 email_body,"\r\n.\r\n","")

    My Lua is:

    NodeMCU custom build by frightanic.com
    branch: dev
    commit: 016f289f315a6caf01509d83ec31bc9dd900835b
    SSL: true
    modules: adc,crypto,file,gpio,http,i2c,net,node,tmr,uart,wifi
    build built on: 2016-11-30 18:08
    powered by Lua 5.1.4 on SDK 1.5.4.1(39cb9a32)

    So , what's wrong?

    Thanks

    ReplyDelete
  16. Hi Andreas, I was wondering if there are any new advances in sending emails from POP3 or gamil yet ?

    ReplyDelete
    Replies
    1. POP3 would be for receiving mail. Miguel's original script was for sending via generic, unencrypted SMTP servers and can still be found here: https://github.com/AllAboutEE/nodemcu-firmware/blob/dev/lua_examples/email/send_email_smtp.lua

      Hope that helps.

      Delete
  17. I was wondering if anyone has any idea how much heap it takes to run this? I was able to take Andreas' code and modify to get to work in my situation. It works (I get the email message) if I estart NodeMCU and run it almost by itself starting with about 42000 heap available. But if I keep updating heap rapidly as this runs heap can get as low as 18000. It makes it hard/impossible to run much else on the module. I'm guessing it has to do with all the data coming at the connection time, but just a guess. Is this what anyone else has seen?

    ReplyDelete
    Replies
    1. My code is admittedly pretty poorly written. I also had problems with it. I think it takes a complete re-write to get more efficient, ressource-saving code.

      Delete
  18. Your code work ? I've always get error after credentials
    Got a response:
    250 smtp.gmail.com at your service

    Got a response: AUTH LOGIN
    334 VXNlcm5hbWU6

    Got a response: (LOGIN base64 encoded)
    334 UGFzc3dvcmQ6

    Got a response: (PASSWORD base64 encoded)
    535-5.7.8 Username and Password not accepted. Learn more at

    ReplyDelete