Web-enabled Arduino Temperature Logger

Sunday, February 12, 2012

Step Nine: The Front End - Flot, Bootstrap and Templates

To start, we'll use the below 'hello world' example of flot:


	<html>
	<head>
	<script language="javascript" type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
	<script language="javascript" type="text/javascript" src="http://people.iola.dk/olau/flot/jquery.flot.js"></script>
	</head>
	<body>

	<h2>Temperature</h2>
	<center><div id="temp" style="width:600px;height:300px;"></div></center>
	<h2>Humidity</h2>
	<center><div id="humidity" style="width:600px;height:300px;"></div></center>
	<script id="source">
	$(function () {
	var d = [

	[1327449645,58.01],[1327449654,58.01],[1327449941,58.51],[1327449971,58.81]

	]
	for(var i=0; i<d.length;i++){
	var obj = d[i][0];
	obj = (d[i][0]*1000)-18000000000;
	}
	$.plot($("#temp"), [d], { xaxis: { mode: "time" } });

	var h = [

	[1327449645,32.0],[1327449654,32.0],[1327449941,33.0],[1327449971,33.5],

	]
	for(var i=0; i<h.length;i++){
	var obj = h[i][0];
	obj = (h[i][0]*1000)-18000000000;
	}
	$.plot($("#humidity"), [h], { xaxis: { mode: "time" } });

	});
	</script>
	</body>
	</html>

view raw gistfile1.html hosted with ❤ by GitHub

As you can see, this example creates two flot objects for Temperature and Humidity. We can use the Google App Engine templating engine to populate the flot data - we'll do this in the example below:

	<html>
	<head>
	<script language="javascript" type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
	<script language="javascript" type="text/javascript" src="http://people.iola.dk/olau/flot/jquery.flot.js"></script>
	</head>
	<body>

	<h2>Temperature</h2>
	<center><div id="temp" style="width:600px;height:300px;"></div></center>
	<h2>Humidity</h2>
	<center><div id="humidity" style="width:600px;height:300px;"></div></center>
	<script id="source">
	$(function () {
	var d = [

	{% for entry in weather reversed %}
	{{entry.temp_entry}}
	{% endfor %}

	]
	for(var i=0; i<d.length;i++){
	var obj = d[i][0];
	obj = (d[i][0]*1000)-18000000000;
	}
	$.plot($("#temp"), [d], { xaxis: { mode: "time" } });

	var h = [

	{% for entry in weather reversed %}
	{{entry.humidity_entry}}
	{% endfor %}

	]
	for(var i=0; i<h.length;i++){
	var obj = h[i][0];
	obj = (h[i][0]*1000)-18000000000;
	}
	$.plot($("#humidity"), [h], { xaxis: { mode: "time" } });

	});
	</script>
	</body>
	</html>

view raw gistfile1.mustache hosted with ❤ by GitHub

Here, you can see that we iterate through the list of weather entries backwards to account for query selecting entries by date desc (newest first). For flot, we want to plot the points with newest last. Additionally, there is some funky-time handling present because Flot accepts time in ms.

To re-iterate, notice how the format of entries is stored in Google App Engine in a way that makes it super-simple to use with Flot. This is certainly not the most useful format for the data, but works and keeps within the free quota limits (a requirement for this project).

Saturday, February 11, 2012

Step Eight: Finalizing Google App Engine (Backend) Logic

Now that we have a crash course in how we can leverage Google App Engine and understand what kind of request the Arduino will be making, we can finalize the App Engine code.

Using the two previous posts, we can assemble the final main.py using the blocks of logic from the previous post. The complete file looks like this (see previous posts additional comments):

	#!/usr/bin/env python
	import os
	import uuid
	import cgi
	from google.appengine.ext.webapp import template
	from google.appengine.ext import webapp
	from google.appengine.ext.webapp import util
	from google.appengine.ext import db
	import datetime
	import time

	class WeatherEntry(db.Model):
	date = db.DateTimeProperty(auto_now_add=True)
	temperature = db.FloatProperty()
	humidity = db.FloatProperty()
	other = db.FloatProperty()

	class DailyWeatherEntry(db.Model):
	date = db.IntegerProperty()
	temp_entry = db.TextProperty()
	humidity_entry = db.TextProperty()

	class MainHandler(webapp.RequestHandler):
	def get(self):
	self.redirect('http://apartmentarduino.blogspot.com/')

	class Weather(webapp.RequestHandler):
	def get(self):
	temp = str(float(cgi.escape(self.request.get('t')))/100)
	humidity = str(float(cgi.escape(self.request.get('h')))/100)
	strS = str(cgi.escape(self.request.get('s')))
	if strS == 'my_secret': #secret added since I don't want just anyone to pollute my weather data!
	rightNow = int(time.time())
	dayAgo = rightNow-86400
	recent_record = DailyWeatherEntry.gql("WHERE date > :1 ORDER BY date DESC",dayAgo)
	rightNow = str(rightNow)
	if recent_record.count()!=0: #update entry
	dayObj = recent_record[0]
	dayObj.temp_entry = dayObj.temp_entry + '['+rightNow+','+temp+'],'
	dayObj.humidity_entry = dayObj.humidity_entry + '['+rightNow+','+humidity+'],'
	dayObj.put()
	else: #create entry
	newEntry = DailyWeatherEntry(
	date = int(time.time()),
	temp_entry = '['+rightNow+','+temp+'],',
	humidity_entry = '['+rightNow+','+humidity+'],'
	)
	newEntry.put()
	self.response.headers.add_header("X-Arduino-Data", temp +','+ humidity )
	else:
	self.error(500)

	class ShowWeather(webapp.RequestHandler):
	def get(self):
	the_weather = db.GqlQuery(
	"SELECT * FROM DailyWeatherEntry ORDER BY date DESC LIMIT 1")

	template_values = {
	'weather' : the_weather
	}

	path = os.path.join(os.path.dirname(__file__), 'weather.html')
	self.response.out.write(template.render(path, template_values))

	def main():
	application = webapp.WSGIApplication([('/', MainHandler),
	('/weather/.?',Weather),
	('/w',ShowWeather)],
	debug=False)

	util.run_wsgi_app(application)


	if __name__ == '__main__':
	main()

view raw main.py hosted with ❤ by GitHub

Hopefully you have some understanding of the Django-like templating engine App Engine supports. There is an excellent hello-world tutorial in Google's official docs.

Next we'll visualize the data using a javascript based graphing library Flot!

Sunday, January 29, 2012

Step Seven: Getting the Arduino Code Together

In this post, we'll finalize the code used with the Arduino.

As introduced in the previous post, we can easily use Google App Engine as a free datastore for this project.

The next step is getting the Arduino Temperature/Humidity Sensor, 7-Segment display and Ethernet Shield into one sketch, receiving a basic response from App Engine.

In order to do this we will:
1) Combine the previous sketches
2) Make app engine respond with the Temperature / Humidity included in the request.

Let's get started.

On the app-engine front, we'll simply want to make a request handler to respond with status code 200, and show the result in a header (for debugging purposes).

That request handler can be as simple as:

	class Weather(webapp.RequestHandler):
	def get(self):
	temp = str(float(cgi.escape(self.request.get('t')))/100)
	humidity = str(float(cgi.escape(self.request.get('h')))/100)
	self.response.headers.add_header("X-Arduino-Data", temp +','+ humidity )

view raw gistfile1.py hosted with ❤ by GitHub

Stitching the three sketches together yields something like the following (read through the comments for specifics!):

	//Twitter Client Libraries
	#include <SPI.h>
	#include <Ethernet.h>

	//RHT03 Temperature Humidity Sensor Libraries
	#include <DHT22.h>

	//Serial 7-Seg Libraries
	#include <SoftwareSerial.h>

	//Included for dtostrf in order to convert Float to String
	#include <stdio.h>

	//Setup Serial connection for the 7-Seg display
	SoftwareSerial mySerial(2, 3);

	//Pin 7 used for the RHT03 Sensor
	//Reminder: Connect a 4.7K resistor between VCC and the data pin (strong pullup)
	#define DHT22_PIN 7

	// Setup a DHT22 instance
	DHT22 myDHT22(DHT22_PIN);
	//Character array to hold results
	char s[4];

	// Enter a MAC address and IP address for your controller below.
	// The IP address will be dependent on your local network:
	byte mac[] = {
	0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x01 };
	IPAddress ip(192,168,1,20);

	// initialize the library instance:
	EthernetClient client;

	const int requestInterval = 20000; // delay between requests

	char serverName[] = "feb10yarf.appspot.com"; // AppEngine URL

	long lastAttemptTime = 0; // last time you connected to the server, in milliseconds

	String currentLine = ""; // string to hold the text from server

	boolean requestStatus = false; // helper for client status

	void setup() {
	// reserve space for the strings:
	currentLine.reserve(256);

	// initialize serial:
	Serial.begin(9600);
	Serial.print("Welcome! Will request temp and make API call in 20s");
	Serial.println();
	// attempt a DHCP connection:
	if (!Ethernet.begin(mac)) {
	// if DHCP fails, start with a hard-coded address:
	Ethernet.begin(mac, ip);
	}
	mySerial.begin(2400); // Different Baud rate for 7-seg display - default is 9600
	delay(2000); // Wait two seconds on initial run before printing to the 7-seg display
	mySerial.print("v"); // Print a clear character to the 7-Seg display
	mySerial.write(0x77); //Special character for display brightness
	mySerial.write((uint8_t)0x00); //Turn off all Decimal points
	mySerial.write(0x7A); //Special Character for brightness
	mySerial.write((uint8_t)0x00); //Set to max brightness
	}

	/*
	Start Loop
	1) read and format data from temperature sensor
	2) make app engine request with data in query string
	*/
	void loop()
	{
	// If we made a request, print the response (must finish up before client.stop() in the conditional below)
	if (client.connected()) {
	if (client.available()) {
	// read incoming bytes:
	char inChar = client.read();
	Serial.print(inChar); //this will spit out the response headers and body for debugging
	}
	}

	// Check to see if time has passed to make the next request
	if (millis() - lastAttemptTime > requestInterval) {
	DHT22_ERROR_t errorCode;
	client.stop(); //kill any existing client since we re-establish below
	delay(1000);
	// The sensor can only be read from every 1-2s, and requires a minimum
	// 2s warm-up after power-on.
	Serial.println();
	Serial.print("Requesting temp and humidity data...");
	Serial.println();
	errorCode = myDHT22.readData();
	float temp = myDHT22.getTemperatureC();
	temp = (temp*9/5) + 32 - 1; //calibrate here
	float humidity = myDHT22.getHumidity();
	// Convert each value to a string for our request to App Engine
	String temp_string = dtostrf(temp*100, 4, 0, s);
	String humidity_string = dtostrf(humidity*100, 4, 0, s);
	switch(errorCode)
	{
	case DHT_ERROR_NONE:
	Serial.print("Got Data From Temp Sensor");
	mySerial.write(0x77); //Special character to change decimal points
	mySerial.write(0x02); //Light up middle decimal
	Serial.println(" ");
	Serial.print(temp_string);
	Serial.print("F ");
	Serial.print(humidity_string);
	Serial.println("%");
	break;
	case DHT_ERROR_CHECKSUM:
	Serial.print("check sum error ");
	Serial.print(myDHT22.getTemperatureC());
	Serial.print("C ");
	Serial.print(myDHT22.getHumidity());
	Serial.println("%");
	break;
	case DHT_BUS_HUNG:
	Serial.println("BUS Hung ");
	break;
	case DHT_ERROR_NOT_PRESENT:
	Serial.println("Not Present ");
	break;
	case DHT_ERROR_ACK_TOO_LONG:
	Serial.println("ACK time out ");
	break;
	case DHT_ERROR_SYNC_TIMEOUT:
	Serial.println("Sync Timeout ");
	break;
	case DHT_ERROR_DATA_TIMEOUT:
	Serial.println("Data Timeout ");
	break;
	case DHT_ERROR_TOOQUICK:
	Serial.println("Polled to quick ");
	break;
	}
	// if you're not connected, and two minutes have passed since
	// your last connection, then attempt to connect again:
	// attempt to connect, and wait a millisecond:
	Serial.println("connecting to server...");
	if (client.connect(serverName, 80)) {
	requestStatus = true;
	Serial.println("making HTTP request...");
	//construct HTTP GET request to app engine:
	String toGet = "GET /weather/?t="+temp_string+"&h="+humidity_string+" HTTP/1.1";
	//Make the request
	client.println(toGet);
	//Pass a host header as well
	client.println("HOST: feb10yarf.appspot.com");
	client.println();
	}
	// note the time of this connect attempt:
	lastAttemptTime = millis();

	}
	}

view raw gistfile1.ino hosted with ❤ by GitHub

Running this sketch yields the following output:

	Welcome! Will request temp and make API call in 20s

	Requesting temp and humidity data...
	Got Data From Temp Sensor
	7078F 2700%
	connecting to server...
	making HTTP request...
	HTTP/1.1 200 OK
	Content-Type: text/html; charset=utf-8
	Cache-Control: no-cache
	X-Arduino-Data: 70.78,27.0
	Expires: Fri, 01 Jan 1990 00:00:00 GMT
	Date: Sun, 29 Jan 2012 21:22:15 GMT
	Server: Google Frontend
	Content-Length: 0


	Requesting temp and humidity data...
	Got Data From Temp Sensor
	7078F 2820%
	connecting to server...
	making HTTP request...
	HTTP/1.1 200 OK
	Content-Type: text/html; charset=utf-8
	Cache-Control: no-cache
	X-Arduino-Data: 70.78,28.2
	Expires: Fri, 01 Jan 1990 00:00:00 GMT
	Date: Sun, 29 Jan 2012 21:22:35 GMT
	Server: Google Frontend
	Content-Length: 0

view raw gistfile1.txt hosted with ❤ by GitHub

A few Notes:
-Most of the above sketch is comments / case handling for 3rd party libraries.
-If your sketch seems to hang on "connecting to server..." try restarting the sketch using the Arduino's on-board reset button.

In the next post, I will included the complete code used for Google App Engine. As of this moment, the Arduino / Hardware components are complete.

Step Six: Getting Started with Google App Engine

In order to store all this useful data with minimal thinking about databases etc. Google App Engine is an easy choice.

Warning:

I hope that this post illustrates how using only 2-3 dozen lines of code, we have a 'free' web-based datastore for this project. What follows is a primer for future posts where we will tie everything together.

Google App Engine will allow us to easily handle the Arduino's requests, store data, and let us retrieve data to display in a web browser.

To get started with Google App Engine you can sign up at: appengine.google.com.

For the purposes of this blog/tutorial, I will assume that you have some working knowledge of App Engine, that you can gain by working through the python tutorial.

If you are very technical, hopefully you can likely stich together a working app using this blog and source code provided alone (without completing the tutorial).

Once you are up and running with the App Engine Launcher and have created your first application, it's quite simple to log data.

The Model:

	class DailyWeatherEntry(db.Model):
	date = db.IntegerProperty()
	temp_entry = db.TextProperty()
	humidity_entry = db.TextProperty()

view raw gistfile1.py hosted with ❤ by GitHub

Since database reads / writes are limited (and we want to keep within App Engine's free-quota limits), storing a day of entries in a single string will be a hacky-effective solution.

I am also storing the date as a unix timestamp integer to remove some complexity of working with timestamps when it's time to graph them.

Defining an API:
Quite simply, the Arduino will pass a temperature and humidity value. Our Google App Engine application will receive these values, and record them to the database. An example request might look like:
myproject.appspot.com/t=7205&h=4302 which represents an entry of 72.05 degrees F and 43.02% humidity. Timestamp isn't required because that is handled server side.

Handling Requests from the Arduino:
The following code is heavily commented, and all records are formatted to be super-friendly to use with Flot (javascript based graphing library) in future posts.

	class Weather(webapp.RequestHandler):
	def get(self):
	temp = str(float(cgi.escape(self.request.get('t')))/100) #Convert from Arduino's 4 digit format
	humidity = str(float(cgi.escape(self.request.get('h')))/100)
	# Determine timestamp of now and one day ago to micro-manage database records
	rightNow = int(time.time())
	dayAgo = rightNow-86400
	# Get a record that is within the last day
	recent_record = DailyWeatherEntry.gql("WHERE date > :1 ORDER BY date DESC",dayAgo)
	# Convert timestamp to a string for appending to other strings
	rightNow = str(rightNow)
	# If there is an existing record for the past day, append the well-formed entry (for easily graphing later)
	if recent_record.count()!=0:
	dayObj = recent_record[0] # First record if any exist
	dayObj.temp_entry = dayObj.temp_entry + '['+rightNow+','+temp+'],' # Formatting here is for use with Flot (js graphing library)
	dayObj.humidity_entry = dayObj.humidity_entry + '['+rightNow+','+humidity+'],'
	dayObj.put() # This writes the new entry to the datastore
	# Otherwise there isn't an entry within the past day, create a new one
	else:
	newEntry = DailyWeatherEntry(
	date = int(time.time()),
	temp_entry = '['+rightNow+','+temp+'],',
	humidity_entry = '['+rightNow+','+humidity+'],'
	)
	newEntry.put()
	self.response.headers.add_header("X-Arduino-Data", temp +','+ humidity )

view raw gistfile1.py hosted with ❤ by GitHub

The above code will be used to handle requests made from the Arduino. Note the use of the X-Arduino-Data header. This will be used for debugging the Arduino sketch.

Displaying Posts:
This code will be used to handle requests from the browser in order to display our temperature/humidity records from the datastore:

	class ShowWeather(webapp.RequestHandler):
	def get(self):
	the_weather = db.GqlQuery(
	"SELECT * FROM DailyWeatherEntry ORDER BY date ASC")
	template_values = {
	'weather' : the_weather,
	#'debug' : debug
	}
	path = os.path.join(os.path.dirname(__file__), 'weather.html')
	self.response.out.write(template.render(path, template_values))

view raw gistfile1.py hosted with ❤ by GitHub

Confused? Probably. We'll be pulling everything together in the next post(s).

TLDR;

In order to proceed you need a Google App Engine account and basic understanding of App Engine. We can make use of Google App Engine using only 2-3 dozen lines of code. The next few posts will bring everything together.

Step Five: Interfacing with the Ethernet Shield

In order to get up and running with the ethernet shield, we can use an example sketch that ships with Arduino 1.0.

The TwitterClient example is most useful for a number of reasons:
1) The sketch behaves as a client (rather than a server which would be more difficult to configure)
2) A regular polling period of 5 minutes is written out of the box.

You can find this source code and documentation on the official arduino.cc website. A special thanks to Tom Igoe for writing the original!

With a hardcoded mac address and IP, my linksys router picked up the arduino and I could see recent tweets right away. Sometimes, I had to reset the arduino (using the on-board reset button) in order to have the router recognize the device. If you are having trouble, I would experiment with resetting the Arduino, and maybe even integrating DHCP.

In future posts, we will make some light modifications to this sketch to better suit our purposes:
1) Swap the api.twitter.com hostnames with our app engine hostname.
2) Pass a payload of temperature and humidity data in the query string.

TLDR;

Use the example TwitterClient example that ships with Arduino 1.0 - it's really great and only requires minor adjustments for this project.

Wednesday, January 25, 2012

Step Four: Interfacing With the 7-Segment Serial Display

In order to test my Serial 7-Segment display, I whipped together a minimal hello world sketch.

This logic can then be plugged into the temperature sketch discussed in the previous post.

Turns out that using SoftwareSerial, you can chat with the sparkfun Serial 7-Segment display quite easily:

	#include <SoftwareSerial.h>

	SoftwareSerial mySerial(2, 3); // Pin 3 on arduino to RX on 7-seg

	void setup(){
	mySerial.begin(2400); // Setup connection at baud rate 2400 (some may be default 9600)
	delay(50);
	mySerial.write(0x77); // Clear all decimal points
	mySerial.write((uint8_t)0x00); //write a 0x00 (without uint8_t an error is thrown)
	mySerial.write(0x7A); // Brightness 00 is brightest
	mySerial.write((uint8_t)0x00);
	mySerial.print("v"); // Reset current display
	}
	void loop(void)
	{
	mySerial.print("1234"); // Show 1234
	}

view raw gistfile1.ino hosted with ❤ by GitHub

Please note - as mentioned the display comes with a default 9600 baud rate. Mine got re-programmed due to a lack of pull down resistors (as discussed on the sparkfun product pages)!

Also note, many of the strange 'hex commands' send to the 7-Seg display were drawn from the data sheet. Reference it before doing any additional programming here (sparkfun user's manual).

TLDR;

Use the sketch above to get the your serial 7-Segment display up and running.

Step Three: Interfacing RHT03 Temperature & Humidity Sensor

Since this project revolves around the temperature and humidity, this was the first component I wanted to test.

Browsing sparkfun's comments on the product, I was able to find a good starting point for this component on github:

nethoncho's Arduino DHT22 Library via GitHub

This library however is not Arduino 1.0 compatible. Thankfully there is a pull request with the necessary adjustments by user ringerc on January 15th (great timing!):
https://github.com/nethoncho/Arduino-DHT22/pull/2

You can download the complete commit that is 1.0 compatible from the following page:
https://github.com/ringerc/Arduino-DHT22/tree/f1c5b3ab1e6f274ec66ecee09df729224af1cf7d

Or using this direct link:
https://github.com/ringerc/Arduino-DHT22/zipball/f1c5b3ab1e6f274ec66ecee09df729224af1cf7d

After un-zipping, you want to drag the entire folder into your 'libraries' directory within the Arduino application.

Once in that directory, you can open the example sketch within the Arduino application.

Wiring the temperature sensor based on comments in the example sketch and sparkfun customer comments yields a steady stream of temperature / humidity data as shown:

From left to right pins 1,2,3,4:
1: Connect to 5v VCC
2: Connect to Pin 7 on the Arduino with a 4.7K resistor to 5v VCC (pull up)
3: Ground
4: Ground

Running the sketch referenced above we'll the temp/humidity readings:

Next, we want to modify this sketch to leverage the 7-segment display. That will come in the next post!

TLDR;

You want to interface with the RHT03 Sensor? Just use the library here to get a hello-world sketch, and wire up the sensor's pins as described above.