Aquaponics: Real-Time Graphing Online

While this project was designed with aquaponics in mind, it does not require an aquaponics system, making it useful in any application that wants graph data online in real-time.  The included application, therefore, is bare-bones to make it easier to integrate into your own project.

You can find a background tutorial on the light-dependent-resistor here.

Intro
I can think of a multitude of reasons you would want to graph data from an Arduino sensor, but the idea of retrieving the SD card from whatever environment your Arduino is in, hooking it up to a computer, downloading the most recent data and finally loading the data into your favourite visualizer, well that gets old fast.  Instead, lets hook the Arduino up to the internet and graph the data in real-time.

This project focuses on collecting the light level from an aquaponics greenhouse using a light dependent resistor (LDR) and sending it to Google App Engine to be graphed using Google Chart Tools.  The cycle is as follows: the Arduino attempts to connect to the webapp and waits ten seconds before parsing the response.  If the response is 'Ok', a good connection was made and a bicolor LED is set to green.  If it could not connect, the LED is set to red.  If a good connection was made, the Arduino will read from analog pin zero (A0) and create a GET request to App Engine with the LDR reading.  App Engine will save the LDR reading and update the current environment property.

On startup, the client will request the server to query the datastore for the light data and receives a JSON array of the current day's data.  The array is parsed and an AJAX request is made to create a table using Chart Tools.  Finally, the client sets an interval for a function that requests the last data entry as another JSON array, parses the response, compares it to the last data in the table and adds the data if they are not the same reading, creating a dynamically updated display.

Software Versions
1. Arduino IDE: Arduino-1.0.3
2. App Engine SDK: Python, Linux, 1.7.4
3. Python: Python2.7
4. Ubuntu 12.04
 Parts List
1 x Arduino Uno R3
1 x Arduino Ethernet Shield R3
1 x Arduino Power Cord (optional, but gets better power than just the USB)
1 x Bicolor LED (Red/Green)
1 x 1/4W 470 ohm resistor
1 x 1/4W 10k ohm resistor
1 x Light Dependent Resistor
Breadboard jumper wires
Small breadboard

Step 1 - Create a new App on App Engine
Since all of our internet based projects use App Engine, we've created an independent tutorial to explain how to create a new app.

• Create a new app.

Step 2 - GAE Project Code

• Download the project code here.
• Extract the tar file in your home directory.  It will be labelled IAquaponics_LDR.  Inside you will find the App Engine directory called myapsystem and the Arduino folder arduino.
• The name of the GAE code is irrelevant, but you can rename it if you wish.  The rest of the instructions in this tutorial will use the original name.  If you choose to amend the directions accordingly.
• Inside myapsystem you will find app.yaml.  Open that in your favourite text editor and edit the first line to the project name you created in Step 1; see Figure 1.

Fig 1.  Edit the highlighted text.

 

• Take note of the bottom.  The main page of our webapp is listed last because it will catch all url requests not listed above it.  In fact, this webapp only has two pages: adacs and main.  The former is used by the Arduino and the latter is the visible interface you will see with the web browser (client).
• Finally, under main.app, you will see login: admin.  This app uses Google Accounts as you saw in Step 1, but we are restricting application to the webapp to just you.

Step 3 - Launch the AppEngine SDK with Your Project

• Open a terminal and launch the AppEngine SDK and your project.

python2.7 AppEngine/dev_appserver.py IAquaponics_LDR/myapsystem

• If you were successful, the terminal will tell you the project is located at

http://localhost:8080/

• So fire up a web browser and point it to that url.  Because we have restricted access to admin, you will need to login as admin.  The login name is irrelevant, but you will need to stay consistent in the login name you choose.  Beneath the email input box you need to check the box that says "Sign in as Administrator" - see Figure 2.  Then login. 

Fig 2.  Login as Administrator.

• At this point you should see Figure 3.

Fig 3.  Main screen.

• There are two important things to note.  First, the chart was built with a piece of data.  If you hover around the point (Now, 750), you will highlight the data.  If you wait 20 seconds, that will disappear.
• More importantly you will see a text box to input your timezone offset from UTC.  To get accurate timestamps of your readings from the Arduino, you'll need your timezone before you hook up the Arduino.  In Iowa, we are six (6) hours behind, so we would submit

-6

• Australia is eight hours ahead of UTC, so they input

8

• Next, we are going to replicate a call the Arduino will make to the server.  In the browser's url bar, type

http://localhost:8080/adacs/lightLevel?Level=800

• You'll be presented with a page that says 'Ok'.  Now go back using the Back button and you'll see a seemingly blank chart, like Figure 4.

Fig 4.  First data.

• It's actually not blank.  You can see the timestamp on the bottom of the chart and you can hover above (timestamp, 800) and see your data input.
• If you look in the terminal from which you launch the SDK and your project, you will see a recurring set of requests.  This shows the JavaScript making the AJAX requests to the datastore to get the non-existent data.

Step 4 - Go Live:  Upload Your Project to App Engine

• If everything has worked for you so far, it's time to upload the webapp to the live server.  In a terminal, from your home directory, type

cd AppEngine/

./appcfg.py update ~/IAquaponics_LDR/myapsystem

• You will be prompted to input your Google email and password you used to create the project from Step 1.
• When the update is complete, go to your webapp.

http://myapsystem.appengine.com 

• If you are not already logged into your Google account, you'll be prompted to.  Note the login is consistent for all of Google's products.
• Once you are logged in, you should see the same screen from the SDK.  Again, input your time zone offset.
• To test the live webapp, replicate the Arduino call just as you did in Step 3. 

Step 5 - Arduino

• Wire up your breadboard using the Fritzing diagram in Figure 5.  The Arduino is not shown, but obviously it resides under the Ethernet Shield.  The 470 ohm resistor is used for the bicolor LED and the 10k resistor is used for the LDR.

Fig 5.  Fritzing Diagram.

• You can find the Arduino file, LDR.ino inside the arduino folder.  Upload that to your IDE.
• There are three places the Arduino code needs to be altered, Figs 6, 7 and 8.  Replace the highlighted code with the project name you developed in Step 1.

Fig 6.  First location.

Fig 7.  Second location.

Fig 8.  Third and final location.

• Select the correct Arduino Board from the menu list and make sure to select the proper Serial Port.  Save and upload your code to your Uno, then open a serial monitor.
• When the Arduino connects, the bicolor LED will turn green.  If there is a connection issue, the Serial output will tell you and the LED will turn red (see Note 1).
• Once the Arduino connects and sends data, go back to your web browser look at your app home page.  You should see the graph update itself as it receives data from the datastore.  Figure 9 shows the our graph over a very short period of time.  The large dip at 19:23:34 was due to our hand blocking the LDR from the light source.

Fig 9.  Real-time data graph.

• And that's it.

Notes

1. Since working with AppEngine and Arduino Ethernet we have encountered one consistent error.  The Arduino will report that it failed to make the third request.  The first two are fine.  Every request after the third connects and works, but for some reason the third one will fail, every time and we don't know if it is due to App Engine or the Arduino.
2. For troubleshooting and development mode, Serial output is on.  Specifically, there are two counters provided, which need to be commented out (using double slashes at the beginning of the line: //).  If you don't, the counter can grow very, very large.

You are free to modify the code as you see fit, but I have one suggestion.  Google App Engine allows for daily free quotas, if you exceed the free quotas and don't pay a monthly fee ($9), your app will be shut down.  Two free quotas you should be aware of are instance hours and read/write/small operations on the datastore.  You get 28 free instance hours, reset daily.  The Arduino should not be set to send requests to AppEngine too often, or new instances of your app may spawn.  Similarly, the browser is set to refresh every 20 seconds.  If you combine the Arduino requests with the client requests, and make them too frequently,  you may again spawn new instances.  In fact, your app is hosted on a server with other applications.  Depending on the load on the other applications, your app may spawn new instances elsewhere.  Better safe than sorry.

Read/write operations are set to 50K each and AppEngine is a little funny about how they count these, but needless to say, making a request every ten seconds, all day from both the browser and Arduino will run up a lot of operations.  Try not to go below twenty seconds on both the client and Arduino and enjoy the free quota.

If you have trouble, hit up the comments below.  And if you include this in one of your own projects, send us a screenshot, we'd love to see it.

Related Projects:
EnvDAQ with Water Temperature Sensor
Online Temperature and Humidity
Online Relay Control
Arduino Email and Text Message Alerts

Related Parts:
pH
Dissolved Oxygen
Water Temperature

Creating a New Application on Google App Engine

The first step in following our web application tutorials is to create a new application on Google App Engine.  So we don't have to repeat the steps each time, this tutorial will be included at the beginning of each project.

Step 1 - Sign Up for Google App Engine

• If you don't already have a free Google Account,  sign-up for one.
• Using your Google account, sign up for Google App Engine.

Step 2 - Download the AppEngine Python SDK

• Our projects are built on Ubuntu Linux and use the Python SDK.  Projects are developed on the local machine in a sandbox using the Software Dev Kit, which you can download here.

Step 3 - Create a New Application

• At the AppEngine homepage, you'll be presented with a list of your applications.  At the bottom, click "Create Application" and input a name for your webapp.

Create a new application.

• All of our webapps use the default security.  Click "Create Application".

Step 4 - Update App.yaml

• The first step in either creating a new app, or using our source code, is to modify app.yaml to use your new project name.  Open your project directory (not the SDK directory) and open or create app.yaml in your text editor of choice.
• The first line is the application name, which you chose in Step 3.  In Fig 2, our project is called myapsystem, so replace this with your own project name.

Fig 2.  Generic app.yaml file

• Lastly, when you chose your project name and set the security to the default setting, you still have options.  For individual projects, each page is set to login: admin that you can see in Fig 2.  If you want to host an app to anybody with a Google login, change this to login: required.  If you don't want a login, delete this line.

 Step 5 - Upload the Web Application

New Aquaponics Tracker Screenshots

The first round of testing is about to begin for the Aquaponics Tracker webapp so we're posting the first screenshots.  Figure 1 highlights the main screen and the new jQuery Date Picker, allowing you to create inputs on any day you like.

Fig 1.  Main screen with jQuery calendar date picker.

The charts are dynamically generated and updated using HTML 5, AJAX and SVG, so the app is compatible with modern browsers and is cross-platform (will work on Android, Windows and Apple devices).  All data is created/updated/deleted in the same page, without the need to refresh the screen, making for a seamless user experience.

After the initial feedback from beta testing we'll include Responsive Web Design, to optimize the screen layout based on the device you are using.  As an example, buttons and text will be sized to accommodate small phones and touch screens, but change for larger screen sizes.

Fig 2.  Interactive charts and inputs analysis.

Finally, the charts are interactive.  Select a nitrate reading in the nitrate graph and the charts and form will automatically show the corresponding date's data, making it a breeze to access data.

If you'd like to see more screenshots, a deeper description and to sign-up for notification when it's ready, head over to the Aquaponics Tracker page.

Aquaponics: Online Temperature and Humidity

This tutorial is based on the Environment DAQ and while it is designed with aquaponics in mind, it does not require an aquaponics system, making it useful for other projects such as hydroponics or home automation. The included application, therefore, is bare-bones, making it easier to integrate into any other application.

How it works
Here we focus on capturing temperature and relative humidity and sending it to App Engine to be visualized using Google Chart Tools. Every sixty seconds, the Arduino will test its connection to App Engine. The return should be “Ok”, which is parsed by the Arduino. If the connection is ok, the bicolor LED is set to green, otherwise it is set to red. If the connection is good, the Arduino will take a reading from digital pin six (the DHT pin) and create a GET request to App Engine. App Engine will query the datastore (its database) for the Environment entity, update the temperature and relative humidity values and put the entity back in the datastore.

On startup, the web browser (client) will create a temperature and humidity gauge with values at zero. It will then make an AJAX request to the datastore which will return a JSON array. The client parses the array and updates the gauges. Finally, it sets an interval to repeat the process, giving you a dynamically updated display.

Final Result.

This project doesn't go into detail on how the webapp is put together or how and where to customize the webapp code, as that takes many, many more pages.  So we wrote a book that delves deeper into this project and others to create a single data acquisition and control system using the Arduino and App Engine.

Software Versions
1. Arduino IDE: Arduino-1.0.3
2. App Engine SDK: Python, Linux, 1.7.4
3. Python: Python2.7
4. Ubuntu 12.04
5. Adafruit's DHT Arduino library

Parts List
1 x Arduino Uno R3
1 x Arduino Ethernet Shield R3
1 x Arduino Power Cord (optional, but gets better power than just the USB)
1 x Bicolor LED (Red/Green)
1 x 1/4W 470 ohm resistor
1 x 1/4W 4.7k ohm resistor
1 x DHT22
Breadboard jumper wires
Small breadboard

Step 1 -  Install Adafruit's DHT Library

Fig 1.  Adafruit Library

• You can find the Arduino library for the DHT here, see Fig 1.
• Download and extract.  Rename the extracted folder "DHT".
• Copy the folder into: ~/Arduino/arduino-1.0.3/libraries 

Step 2 - Google App Engine

Fig 2.  New GAE Project

• If you don't already have a free Google account, sign up for one.
• Use your Google account to sign up for Google App Engine
• Download the App Engine Python SDK.  The version used here is 1.7.4, on Linux and extract it to your home directory.  In this case, I renamed the folder AppEngine.  If you depart, simply make note of where you extracted the folder.
• In the App Engine Admin Console, create a new project, Figure 2. 
• Take note of your application identifier.  You will need that for future steps.
• Keep the default security settings; we will restrict access to Admin (you) later.

Step 3 - GAE Project Code

• Download the project source code here.
• Extract the tar file in your home directory, it should be called IAquaponics_DHT.  Inside you will find the GAE directory labelled myapsystem and the Arduino folder.  The name of the folder containing the GAE code is irrelevant, so there isn't a need to rename it.  The rest of the instructions will keep the original name.  If you choose to rename the folder, amend the directions accordingly.
• Inside the GAE directory, open app.yaml.  The first line is the application name, so change it to the application identifier you made in Step 2.

Fig 3.  app.yaml.  Replace the highlighted name with your project name.

• Take note of the bottom.  The main page of our webapp is listed last because it will catch all url requests not listed above it.  In fact, this webapp only has two pages: adacs and main.  The former is used by the Arduino and the latter is the visible interface you will see with the web browser (client).
• Finally, under main.app, you will see login: admin.  This app uses Google Accounts as you saw in Step 2, but we are restricting application to the webapp to just you.

Step 4 - Launch the AppEngine SDK with Your Project

• Open a terminal and launch the AppEngine SDK and your project.
  

python2.7 AppEngine/dev_appserver.py IAquaponics_DHT/myapsystem

• If all went well, the terminal will tell you the project is located at

http://localhost:8080/

• So fire up a web browser and point it to that url.  Because we have restricted access to admin, you will need to login as admin.  The login name is irrelevant, but beneath the email input box you need to check the box that says "Sign in as Administrator".  Then login.

Fig 4.  Login as Administrator.

• At this point you should see two gauges that read zero, see Fig 5.

Fig 5.  Initial Gauges.

• Next, we are going to replicate the request the Arduino will make to the live webapp.  In the browser, type in:

http://localhost:8080/adacs/dht?Temp=69.1&RH=24.8

• You will be presented with a page that says "Ok".  Go Back to the main screen and you should be presented with the two updated gauges, like Fig 6.

Fig 6.  Update gauge display.

• That's how this works.  The Arduino will make requests to the webapp, sending data.  The webapp will save that data in the datastore (its database).  The JavaScript will make AJAX requests (asynchronous JavaScript requests) to the datastore, get the values, and assign them to the gauges giving you a constantly updated display.

Step 5 - It's Alive: Upload the Webapp to App Engine

• If everything has worked for you so far, then it's time to upload the webapp to a live server.  Open a terminal and type

cd AppEngine/

./appcfg.py update ~/IAquaponics_DHT/myapsystem

• You will likely be prompted to input your login email and password.
• When the update is done, go to your webapp.

http://myapsystem.appspot.com

• If you are not logged into your Google account, you'll be prompted to.  Note the login is consistent for all of Google's products.  At the home page, you should see two zero gauges, just like the SDK.
• To test the live webapp, repeat the replicated call made by the Arduino and return back to the home screen.  You should be presented with two updated gauges.

Step 6 - Arduino

• Wire up your breadboard following the Fritzing diagram in Fig 7.  The Ethernet shield is not shown,  but it resides on top of the Arduino and the jumper wires plug into the Ethernet shield.  The 470 ohm resistor is for the bicolor LED and the 4.7k ohm resistor is for the DHT.

Fig 7.  Fritzing layout of breadboard.

• With that done, load the Arduino file DHT.ino from the project folder.
• You will need to amend the Arduino code to send requests to your webapp.  There are three locations where the webapp link is listed.  Change ONLY the highlighted project name.

Fig 8.  First location.

Fig 9.  Second location.

Fig 10.  Third location.

• Save and upload to your Arduino.
• Open a Serial monitor
• When the Arduino connects, the bicolor LED will turn green.  If there is a connection issue, the Serial output will tell you and the LED will turn red (See Note 1).
• If it's not still open, launch a web browser and point it to your live webapp.  You should see the gauges change.  After that, depending on how much your temperature fluctuates, you may not see a noticeable change.
• And that's it.

Notes

1. Since working with AppEngine and Arduino Ethernet we have encountered one consistent error.  The Arduino will fail to make the third request.  The first two are fine.  Every request after the third connects and works, but for some reason the third one will fail, every time.
2. For troubleshooting and development mode, Serial output is on.  Specifically, there are two counters provided, which need to be commented out (using double slashes at the beginning of the line: //).  If you don't, the counter can grow very, very large.
3. There is a flaw in this code.  If the Arduino fails to update AppEngine, your display will not be changed and you won't know it.  Because this is a bare-bones application, our solution wasn't included.  The solution we developed was to save each timestamp of the Arduino request in AppEngine and create a deferred task set to delay for five minutes.  At the end of the delay, the current timestamp was compared to the last timestamp of the Arduino.  If the delay exceeded a certain time interval (such as three minutes), it meant the Arduino hadn't communicated in five minutes (obviously).  At that point, an email or text alert was sent to the owner notifying you of a connection issue and the gauge values were reset to zero.  

You are free to modify the code as you see fit, but I have one suggestion.  Google App Engine allows for daily free quotas, if you exceed the free quotas and don't pay a monthly fee ($9), your app will be shut down.  Two free quotas you should be aware of are instance hours and read/write/small operations on the datastore.  You get 28 free instance hours, reset daily.  The Arduino should not be set to send requests to AppEngine too often, or new instances of your app may spawn.  Similarly, the browser is set to refresh every 30 seconds.  If you combine the Arduino requests with the client requests, and make them too frequently,  you may again spawn new instances.

Read/write operations are set to 50K each and AppEngine is a little funny about how they count these, but needless to say, making a request every ten seconds, all day from both the browser and Arduino will run up a lot of operations.  Try not to go below twenty seconds on both the client and Arduino and enjoy the free quota.

If you have trouble, hit up the comments below.  And if you include this in one of your own projects, send us a pic, we'd love to see it.

More projects like this can be found on the Automating Aquaponics with Arduino page.

Related:   
EnvDAQ with Water Temperature Sensor 
Real-Time Graphing Online
Online Relay Control

Testing Proto Shields for Automating Aquaponics with Arduino

For Maker's who already have a proto shield such as the official shield, Adafruit's, or Make's, we thought we'd test these boards with the Arduino Data Acquisition and Control System design.

The tiny breadboard seen here is from Adafruit and was used to layout the parts specifically for Adafruit's shield, which is more of a printed breadboard design - if it could fit on the breadboard, it could fit on the shield.  The empty side of the breadboard is where the other sensors and indicators belong. 

With Adafruit's shield we ran into a clearance issue between the proto shield and the Arduino ethernet shield.  The ethernet jack is 0.5" tall and Adafruit's shield kit does not come with stackable header pins - the pins it does comes with are not long enough to make the connection when stacked on the ethernet shield.  Fortunately, we roughed the proto shield parts into place with the Uno and ethernet shield before soldering the kit together, saving us from desoldering the proto shield to add stackable pins. 

The official Arduino proto shield does not have this clearance issue and we'll test the Make shield later (since it comes with stackable header pins, we can't immediately see it having any issues).  All three shields are capable of being placed between the Uno and the Ethernet shield making the clearance seemingly irrelevant.  However, if you chose to solder the DHT sensor directly into the proto shield (flat, unlike the picture), and then sandwich the shield between the Uno and Ethernet, your temperature and humidity readings could become skewed by the ambient heat generated from the other shields.  This means you would have to use extension wires to move the sensor to the enclosure and if you don't hard solder the extension cables in, instead choosing to use removable connectors, you might be presented with clearance issues all over again.  In the end, simply adding stackable header pins was the easier option.

The Adafruit's shield is very simple to put together and barring the (easy to resolve) header issue, worked well.  Because the Adafruit's proto shield is similar to a breadboard it is more intuitive to setup a circuit design from the breadboard vs. the Make or Arduino proto shields, which we'll discuss in more detail later.

------------
Using Arduino with Aquaponics is covered in our upcoming book Automating Aquaponics with Arduino and companion kit.