Tuesday 21 May 2019

Installing MySQL on the Raspberry Pi - part 2

So, having worked through part one, you should have a working installation of MySQL.

If not, go back through and check that all of the steps have been completed.

Stored Procedures

This is a simple stored procedure to list the users in the datavalues table (remember you will need to have selected the required database using the use datastorage command).

MySQL needs to differentiate between line delimiters within the stored procedure and within the command that creates the stored procedure. The delimiter needs to be changed to differentiate between the end of line within the procedure and the end of the definition. Remember to change the delimiter back.
DELIMITER //
CREATE PROCEDURE listusers()
BEGIN
SELECT DISTINCT user from datavalues;
END//
DELIMITER ;

Use SHOW PROCEDURE STATUS; to view stored procedure names.
Use SHOW CREATE PROCEDURE <procedure name> to view the procedures.

Stored Procedure with input parameter

This is a slightly more complicated procedure which takes a username as a parameter.

DELIMITER //
CREATE PROCEDURE list_values_for_user(IN username VARCHAR(50))
BEGIN
SELECT id,value,title,created FROM datavalues WHERE user=username;
END //
DELIMITER ;

Test with:
CALL list_values_for_user('root@localhost')
MariaDB [datastorage]> call list_values_for_user('root@localhost');
+----+-------+-------+---------------------+
| id | value | title | created             |
+----+-------+-------+---------------------+
|  1 |     1 | test  | 2019-02-27 11:06:09 |
|  2 |     2 | test  | 2019-02-27 11:06:16 |
|  3 |     3 | test  | 2019-02-27 11:06:23 |
+----+-------+-------+---------------------+
3 rows in set (0.00 sec)





Wednesday 15 May 2019

Using the built-in camera with Raspbian and OpenCV

I have an elderly Samsung Netbook which is running Raspbian Stretch as a LiveUSB.

The netbook has a built in camera, and I thought I would follow up the post on using a Raspberry Pi and a Webcam with one on the Netbook.

Installation

Follow the instructions from the previous post.

Taking a photograph

Open up a Python editor (I used IDLE 3).
Create a new file.
Copy and paste the code from the previous post.
Save the program to your usual directory).
Run the program.

Footnote

It also was a convenient check that the previous post worked on a fresh install. The post required a slight edit as the PIP3 command needed to be run under SUDO.

USB Webcam using a Raspberry Pi and Python - test

Introduction

USB webcams are very cheap, and in many cases you may already own one. Though the Official Raspberry Pi Camera is often far superior (many "High Resolution" webcams are 'Nineties era VGA resolution - 640x480), the fact that webcams are cheap and cased can make them very useful for some projects.

This posting covers the simplest of Python programs to take a photograph using the OpenCV library.

Installation

Install OpenCV

sudo apt-get install python-opencv libopencv-dev

You might find it is already installed.

Install Python bindings

Remember that pip probably installs for Python 2.x, so you will need to use pip3.

sudo pip3 install opencv-python

Please note that if you forget the sudo, you do not get an error, but it also does not work (you get an error "ImportError: No module named 'cv2'").

Taking a photograph

The following Python code takes a photograph and saves it to a file in the (pi) user's home directory.

The program takes a number of frames to allow the camera and its software to determine the best settings for exposure etc.

# Test USB web cam with OpenCV

# Imports
import cv2

# Set camera to be the default webcam
camera_port=0

# Set warm up number of frames
skip_frames=7

# Destination file
file="/home/pi/image.jpg"

# Set up camera properties
camera=cv2.VideoCapture(camera_port)
camera.set(cv2.CAP_PROP_FRAME_WIDTH,640) 
camera.set(cv2.CAP_PROP_FRAME_HEIGHT,480)

# Function to get image
def get_image():
    retval,im=camera.read()
    return im

# Test USB web cam
print("Starting photograph")

# Skip frames to allow camera to settle down
for skipping in range(skip_frames):
    temp=get_image()
print("Taking photograph")

camera_capture=get_image()

# Write image to file
cv2.imwrite(file,camera_capture)

# Remove camera object - ensures release of resources
del(camera)

print("Done")



Installing MySQL on the Raspberry Pi - part 1

Introduction

MySQL is a Free and Open Source relational database owned by Oracle Corporation. Due to concerns about the future of MySQL, MySQL was forked as a community developed version called MariaDB.

MariaDB is intended to be a drop in replacement for MySQL and is increasingly being chosen as the default "MySQL" installation on Linux distributions. Debian, the upstream Linux version for Raspbian changed to use MariaDB recently, hence the references to MariaDB in the installation text.

There are many excellent examples of MySQL installation tutorials on the web, this is just one (how useful you find it is up to you). See the references section for more information and some examples.

Set Up

Update Raspbian installation

This is the standard update/upgrade process to ensure that the Raspberry Pi has the most up to date versions of all installed software before adding new software.

sudo apt-get update
sudo apt-get upgrade

Install MySQL from repository

MySQL is available from the Raspbian repository.

sudo apt-get install mysql-server

Setup and securing the database

Run the secure installation script:
sudo mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!
Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

That is the database set up.


First connection

From the command line.
sudo mysql -uroot -p
Enter the root password.

Once you are logged in, enter the following to create a database.
create database datastorage;
The ending semi-colon is the effective end of line for a command.
grant all privileges on datastorage.* TO 'root'@'localhost' IDENTIFIED BY '<root password>';

Create a local host data reader. This user can only access the database on the server itself.
create user 'localreader'@'localhost' identified by '<user password>'
grant select on datastorage.* to'localreader'@'localhost';

This user is allowed to access the database from anywhere, but only has select access.
create user 'datareader'@'%' identified by '<user password>;
grant select on datastorage.* to'datareader'@'%';

Subsequent Connections

Logging in follows the same process a s before, from the command line.
sudo mysql -uroot -p
Enter the root password

The console does not default to a specific database, so it is best to remember to set the current database (in this case datastorage created earlier).
MariaDB [(none)]> use datastorage
Database changed
MariaDB [datastorage]>

Creating tables

This example table is to record data sent from other machines, by other users.
The table has a primary key (id) which is an integer and auto increments. The created field is a DATETIME and defaults to the current time.

CREATE TABLE datavalues(
id INT NOT NULL AUTO_INCREMENT,
user VARCHAR(50) NOT NULL ,
title VARCHAR(20) NOT NULL,
value FLOAT NOT NULL,
created DATETIME NOT NULL DEFAULT NOW(),
PRIMARY KEY(id)
);

Populating the table

From the MySQL command line:
insert into datavalues(user,title,value)
values(CURRENT_USER(),'test',1);

Test the insert:
SELECT * FROM datavalues;

References




Jupyter Notebook Example - Historical currency exchange rates



Exchange Rates

In [6]:
# Import simple warning to ignore
import warnings
warnings.simplefilter('ignore', FutureWarning)

# import pandas
from pandas import *

# Import the plotting modules and set default format
import matplotlib
matplotlib.rcParams['axes.grid'] = True # show gridlines by default

# Show plots in the notepad
%matplotlib inline

import matplotlib.pyplot as plt
from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
import matplotlib.dates as mdates

# Import LXML to read HTML and extract data
from lxml import html
import requests

# Import collections (to obtain the OrderedDict)
import collections
#import csv

# Import date time to handle the dates in the data
from datetime import datetime
In [7]:
# Constants
URLUSSTERLING= "https://www.federalreserve.gov/releases/h10/hist/dat00_uk.htm"
URLUSEURO="https://www.federalreserve.gov/releases/h10/hist/dat00_eu.htm"
DATE="Date"
RATE="Rate"
In [8]:
# Function definitions

# Obtain exchange rate from US Federal Reserve website (via supplied URL)
# Note: the direction of the exchange differs for each currency - Euro and GBP are the same direction
def get_US_exchange_rates(aURL):
    exchangerates=collections.OrderedDict()
# Get the page from the URL
    # print("Get page")
    page = requests.get(aURL)
# Make an HTML tree from the text
    # print("Make HTML page")
    tree = html.fromstring(page.content)
# The data is in a table of class="pubtables"
    r=tree.xpath("//table[@class='pubtables']/tr")
# Iterate over the selected XPATH elements
    # print("Iterate over xpath")
    for x in r:
        if x[1].text_content().strip()!="ND":
            exchangerates[datetime.strptime(x[0].text_content().strip(),"%d-%b-%y")]=float(x[1].text_content().strip())   
            # Note: the values need to be "floated" so the scaling works
    # print("Done")
    return exchangerates

# Functions to obtain scaling for Y axes, rounding up to the next 0.1, down to the next 0.1
def round_up_to_tenth(value):
    return float(int((value*10)+1))/10
def round_down_to_tenth(value):
    return float(int((value*10)-1))/10

# Get the max of the two values rounded up to the next 0.1
def round_up_to_tenth_max_of_two(value1,value2):
    result=round_up_to_tenth(value1)
    roundup2=round_up_to_tenth(value2)
    if result<roundup2:
        result=roundup2
    return result

# Get the max of the two values rounded down to the next 0.1
def round_down_to_tenth_max_of_two(value1,value2):
    result=round_down_to_tenth(value1)
    roundup2=round_down_to_tenth(value2)
    if result>roundup2:
        result=roundup2
    return result
    
In [9]:
# Get US dollar to Sterling rate
c=get_US_exchange_rates(URLUSSTERLING)
print("US rates found")
#c
US rates found
In [10]:
# Make a dataframe from the exchange rates (renaming the columns)
dfus=pandas.DataFrame([x,c[x]] for x in c)
dfus.rename(columns={0: DATE, 1: RATE}, inplace=True)
# Display the last five rows from the data frame
dfus.tail()
Out[10]:
Date Rate
4853 2019-05-06 1.3093
4854 2019-05-07 1.3052
4855 2019-05-08 1.3014
4856 2019-05-09 1.3013
4857 2019-05-10 1.3036
In [11]:
# Get the US dollar to Euro exchange rates
c=get_US_exchange_rates(URLUSEURO)
print("EU rates found")
#c
EU rates found
In [12]:
# Make a dataframe from the exchange rates (renaming the columns)
dfeu=pandas.DataFrame([x,c[x]] for x in c)
dfeu.rename(columns={0: DATE, 1: RATE}, inplace=True)
# Display the last five rows from the data frame (just to sanity check the values)
dfeu.tail()
Out[12]:
Date Rate
4854 2019-05-06 1.1197
4855 2019-05-07 1.1178
4856 2019-05-08 1.1206
4857 2019-05-09 1.1228
4858 2019-05-10 1.1241
In [13]:
# Merge the two dataframes (inner merge so there will be both exchange rates for each date)
df=merge(dfus,dfeu,on=DATE)
df.rename(columns={RATE+"_x": RATE+"_US",RATE+"_y": RATE+"_EU"}, inplace=True)
# Convert US to Euro to Euro to Sterling 
df[RATE+"_EU"]=df[RATE+"_US"]/df[RATE+"_EU"]

# Only include data from the beginning of 2016
df=df[df.Date>"01-Jan-2016"]
df.tail()
Out[13]:
Date Rate_US Rate_EU
4853 2019-05-06 1.3093 1.169331
4854 2019-05-07 1.3052 1.167651
4855 2019-05-08 1.3014 1.161342
4856 2019-05-09 1.3013 1.158978
4857 2019-05-10 1.3036 1.159683
In [14]:
# Plot exchange rates

# Set plot size
fig = plt.figure(figsize=(18, 10))

# Set the X axis format
majorLocator = MultipleLocator(100)
majorFormatter = FormatStrFormatter('%d')
years = mdates.YearLocator()   # every year
months = mdates.MonthLocator()  # every month
yearsFmt = mdates.DateFormatter('%Y')

ax = fig.add_subplot(111)
ax.format_xdata = matplotlib.dates.DateFormatter('%Y-%m-%d')

ax.xaxis.set_major_locator(years)
ax.xaxis.set_major_formatter(yearsFmt)
ax.xaxis.set_minor_locator(months)

# Get maximum and minimum values for the two two plots 
yminimum=round_down_to_tenth_max_of_two(min(df[RATE+"_US"]),min(df[RATE+"_EU"]))
ymaximum=round_up_to_tenth_max_of_two(max(df[RATE+"_US"]),max(df[RATE+"_EU"]))

# Set Y limits and ticks
plt.ylim(ymin=yminimum,ymax=ymaximum)
plt.yticks(np.arange(yminimum, ymaximum,0.1))

# Plot the two sets of exchange rates
ax.plot(df[DATE],df[RATE+"_US"],color='b')
ax.plot(df[DATE],df[RATE+"_EU"],color='r')

# Add the legend and titles
plt.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3,
           ncol=2, mode="expand", borderaxespad=0.)
plt.title('Exchange Rates')
plt.xlabel('Date')
plt.ylabel('Rate', multialignment='center')
fig.autofmt_xdate()