in After hello world and using Wi-Fi to send data, we should take a look at where we actually store the data. An overwhelming number of online tutorials suggest using some already existing services. But where is the fun in doing that?
Database
There are so many databases you could choose for your project. You could choose any of the classical relational ones, but that is quite clumsy for a project like this. Or maybe something more modern like NoSQL, mainly if you plan to collect an unreasonable high load of data!
For example, measuring temperature several times every second. It is possible, of course, but pointless. At least with cheap modules, the precision is not super great, and who needs micro changes every second? I would argue once a minute is already quite more than enough.
Remember: there are so many databases because there are so many needs. No database can be optimized for every use case. Therefore it is crucial to ask yourself the question, what is the primary requirement? For the weather station, it’s collecting the same data changing over time and plots them in a nice graph. The main point here is the time. And for that, time-series databases exist!
My favorite, I guess, is InfluxDB. It is open source, fast (even though that is not our concern here), it has tons of options to connect to it (including several pre-made libraries for many languages!), and, mostly, it has a configurable web interface!
Technically, you can set up the InfluxDB instance, redirect your weather station to it with a library for Arduino, and prepare friendly dashboards. That's it.
Server
Unless you want a bit more. For example, last time, I said it is best to move as much work from the clients to the server. That is one of the reasons why I created my own server communicating with InfluxDB instead. The library is pleasant but still a bit more complex than what I can achieve with a simple request.
The main reason is compatibility, though. With one weather station, it is okay, kind of. But I already have five in many locations and may have even more of them in the future. If I needed to upgrade all of them simultaneously, it would be a trip for the whole day. Like this, I can keep it simple for my stations to make the API stable, and all the software kung-fu is done on the server side only.
Another minor reason is that I don't trust anything by default. Because few stations are outside my home network already, I don't want to expose InfluxDB to the world. I uncover only a minimal custom API that can do almost nothing because there is nearly nothing. If there is any severe security bug in InfluxDB, it should almost not concern me. It's actually quite more important than the practical reason above, but I also run it in its container that even if someone gets to InfluxDB, there is only that. So only weather data would be leaked (and maybe lost until I find the latest backup).
What did I choose for my web server? I love Python and Haskell, but I went with Go. Even though functional programming and typing are great, it would slow me down here. With Python, I could have it done in no time, but packing it is a bit tricky. In the end, writing such a simple server in Go is also simple, and I can build one executable containing everything to execute it on the server.
func main() {
http.HandleFunc("/write", errorMiddleware(handleWrite))
log.Fatal(http.ListenAndServe(":8000", nil))
}
func errorMiddleware(handler func(http.ResponseWriter, *http.Request) error) func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
err := handler(w, r)
if err != nil {
log.Warn(err)
http.Error(w, fmt.Sprint(err), http.StatusBadRequest)
}
}
}
func handleWrite(w http.ResponseWriter, r *http.Request) error {
location, err := getParamString(w, r, "loc")
if err != nil {
return err
}
temperature, err := getParamFloat(w, r, "temp")
if err != nil {
log.Error(err)
} else {
logTemperature(location, temperature)
}
// ...
log.WithFields(log.Fields{
"location": location,
"temperature": temperature,
// ...
}).Info("Got data")
return nil
}
func getParamFloat(w http.ResponseWriter, r *http.Request, param string) (float64, error) {
value, err := getParamString(w, r, param)
if err != nil {
return 0, err
}
number, err := strconv.ParseFloat(value, 8)
if err != nil {
return 0, fmt.Errorf("%s not a number: %s", param, err)
}
return number, nil
}
func getParamString(w http.ResponseWriter, r *http.Request, param string) (string, error) {
values, ok := r.URL.Query()[param]
if !ok {
return "", fmt.Errorf("%s not set", param)
}
return values[0], nil
}
func logTemperature(location string, value float64) {
logData(location, "temperature", "htu", value)
}
// ...
func logData(location string, sensor string, key string, value float64) {
logger := log.WithFields(log.Fields{
"location": location,
"sensor": sensor,
"value": value,
})
url := "https://INFLUX_HOST/api/v2/write?org=home&bucket=home&precision=s"
payload := fmt.Sprintf("%s,host=%s %s=%f", sensor, location, key, value)
req, err := http.NewRequest("POST", url, strings.NewReader(payload))
if err != nil {
logger.Error("Failed to make a request")
return
}
req.Header.Add("Authorization", "TOKEN")
client := &http.Client{}
_, err = client.Do(req)
if err != nil {
logger.Error("Failed to log data")
}
}
Heat index
Now you have all the building blocks to make your cool weather station a reality. Except, it can be better. For example, the temperature is one thing, but the real feel depends on many more variables, notably humidity. Outdoor also on wind speed, but I care primarily about indoor and during warm temperatures. For that, the heat index is the best to calculate the real temperature and also show warnings when it goes above a comfortable level.
Computing heat index is quite complex. You can check out the beast formula on Wikipedia. Converting it into code is not super straightforward, and the good thing is, someone already did it for InfluxDB! There were mistakes, but still, this helped a lot. If you want to use also Go server for presenting data as I did, you could use my function (I have defined a few structs representing my measurements and periods, which you should not need and it should be clear with what values to replace my variables):
func buildQueryHeatIndex(period period) (string, error) {
// https://www.influxdata.com/blog/hiding-complexity-with-custom-functions-calculating-heat-index/
query := fmt.Sprintf(`
import "math"
temperature = from(bucket: "%s")
|> range(start: %s, stop: %s)
|> filter(fn: (r) => r._measurement == "%s" and r["_field"] == "%s")
|> aggregateWindow(every: %s, fn: mean)
|> keep(columns: ["_value", "_time", "host"])
humidity = from(bucket: "%s")
|> range(start: %s, stop: %s)
|> filter(fn: (r) => r._measurement == "%s" and r["_field"] == "%s")
|> aggregateWindow(every: %s, fn: mean)
|> keep(columns: ["_value", "_time", "host"])
first_join = join(tables: {temperature: temperature, humidity: humidity}, on: ["_time", "host"])
|> map(fn: (r) => ({temperature: r._value_temperature, humidity:r._value_humidity, _time: r._time, host: r.host}))
|> keep(columns: ["_time", "humidity", "temperature", "host"])
|> map(fn: (r) => ({t: ((r.temperature*(9.0/5.0))+32.0), h: r.humidity, _time: r._time, host: r.host}))
|> map(fn: (r) => ({
r with heatIndex:
if ((0.5 * (r.t + 61.0 + ((r.t-68.0)*1.2) + (r.h*0.094)))/2.0) < 80.0 then (0.5 * (r.t + 61.0 + ((r.t - 68.0)*1.2) + (r.h*0.094)))
else if ( r.h < 13.0 and r.t > 80.0) then ((-42.379 + 2.04901523*r.t + 10.14333127*r.h - .22475541*r.t*r.h - .00683783*r.t*r.h - .05481717*r.t*r.h + .00122874*r.t*r.t*r.h + .00085282*r.t*r.h*r.h - .00000199*r.t*r.t*r.h*r.h - (((13.0-r.h)/4.0)*math.sqrt(x: ((17.0-math.abs(x: (r.t-95.0))/17.0))))))
else if r.h > 85.0 and r.t >= 80.0 and r.t <= 87.0 then ((-42.379 + 2.04901523*r.t + 10.14333127*r.h - .22475541*r.t*r.h - .00683783*r.t*r.h - .05481717*r.t*r.h + .00122874*r.t*r.t*r.h + .00085282*r.t*r.h*r.h - .00000199*r.t*r.t*r.h*r.h) + (( r.h-85.0 )/10.0) *((87.0-r.t)/5.0))
else (-42.379 + 2.04901523*r.t + 10.14333127*r.h - .22475541*r.t*r.h - .00683783*r.t*r.t - .05481717*r.h*r.h + .00122874*r.t*r.t*r.h + .00085282*r.t*r.h*r.h - .00000199*r.t*r.t*r.h*r.h)
})
)
|> map(fn: (r) => ({_value: ((r.heatIndex-32.0)/1.8), _time: r._time, host: r.host}))
|> yield(name: "HeatIndex")
`,
influxBucket,
period.start,
period.end,
temperature.name,
temperature.field,
period.aggregation,
influxBucket,
period.start,
period.end,
humidity.name,
humidity.field,
period.aggregation,
)
return query, nil
}
Wow! The problem is that this will not be visible in the UI of InfluxDB. You can either set it up there instead or prepare a custom presentation as I did. Let's take a look into that next time.
feed