Jupyter Notebooks

One of the first courses I studied on FutureLearn was "Learn to Code for Data Analysis". It used the Jupyter Notebook to allow the manipulate large amounts of data.

It uses a paradigm similar to spreadsheets with macros/scripting languages but inverts the emphasis. Normally with a spreadsheet, you start with the sheet of data and that is what you see. You then run some code against the data and return to the sheet.

With Jupyter Notebooks you do not look at the (raw) data but manipulate it in the background, you do not even need to look at the raw data (except during the development process).

The language used to manipulate the data is Python. As it is Python you can use the same code that you would use in a stand alone program in your notebook. If you have ever had to think which language you are using (or the version of the language you are using - such as Visual Basic, VB.Net or VBA), this is a great help.

I am not going to cover the installation, there is an installation guide here: 

https://docs.anaconda.com/anaconda/install/

You can export the results into various forms, including static HTML pages.

A simple demonstration of the power of Jupyter Notebooks

In an earlier posting, I showed a simple Python webscraping function to obtain historical currency exhange rates from the US Federal Reserve. This used a number of standard Python libraries.

Now, if you wanted to undertake analysis of exchange rates over time, you could build upon that Python code, but Jupyter Notebooks offers  a simpler way of manipulating the data. You can just use the Python code and the associated libraries.

I exported the results of the Jupyter Notebook as HTML and copied it into a post on this blog.

If you want to try it yourself, just copy and paste the cell contents into cells on your own Jupyter Notebook.

References

https://en.wikipedia.org/wiki/Project_Jupyter#Jupyter_Notebook

Historical Currency Exchange Rates from the US Federal Reserve (Python)

Sometimes you want historical exchange rates, either to track the values over time, or the value at a particular time.

The US Federal Reserve has information on exchange rates, for example Sterling to US Dollars: https://www.federalreserve.gov/releases/h10/hist/dat00_uk.htm

Note that the direction of the exchange is not constant for all currencies.

Examining the structure of the web page showed the table  containing the data is identified by it having a class of "pubtables".

LXML is used to take the table data and allow it to be searched using XPATH.

The program outputs the data as a CSV file.

# Function to extract exchange rates from US Federal Reserve

from lxml import html

import requests

import collections

import csv

from datetime import datetime

url= "https://www.federalreserve.gov/releases/h10/hist/dat00_uk.htm"

def get_US_exchange_rates(aURL):

    exchangerates=collections.OrderedDict()

# Get the page from the URL

    print("Get page")

    page = requests.get(aURL)

# Make an HTML tree from the text

    print("Make HTML page")

    tree = html.fromstring(page.content)

# The data is in a table of class="pubtables"

    r=tree.xpath("//table[@class='pubtables']/tr")

    print("Iterate over xpath")

    for x in r:

        if x[1].text_content().strip()!="ND":

            exchangerates[datetime.strptime(x[0].text_content().strip(),"%d-%b-%y")]=float(x[1].text_content().strip() )  

    print("Done")

    return exchangerates

# Testing code 

if __name__ == "__main__":

    print ('Exchange Rates:')

    if True:

        c=get_US_exchange_rates(url)

        with open('exchange.csv', 'w') as csvfile:

            exchwriter=csv.writer(csvfile,delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL)

            for x in c:

                print(x,c[x])

                exchwriter.writerow([x,c[x]])

Kitronik :MOVE mini for the BBC mIcro:bit

I ordered a Kitronik MOVE mini robot from Pimoroni last year. This provides a battery powered chassis that can be controlled using an on-board BBC micro:bit.

The kit comes in a robust cardboard box, unfortunately not big enough to take the completed robot (but see later).

The components are neatly bagged up, and include the required AA batteries.

The body is made up of laser cut acrylic pieces. There are two continuous rotation servo motors to provide the motive power.

The controller board is designed to use countersunk screws to provide the connection between the micro:bit  and the board. This does mean that it limits the control to the two motors and the light bar (there is an option to isolate the light bar, giving access to an additional - optional - servo).

The back of the board. Note at bottom right the area to  cut to access the third servo).

The kit does not include a BBC micro:bit. As one of the options is to control the robot's own micro:bit using a second micro:bit, I ordered a second one.

I covered the BBC micro:bit in an earlier post.

The instructions to build the robot can be found here. They are generally straightforward (so much so I forgot to pause to photograph the stages).

The one thing to be aware of is that the controller board only operates on the batteries. I started testing the board assuming that the USB supply would power the micro:bit and the controller board and was testing it with the battery switched off. The micro:bit was fine, but the ZIP LED light bar was not lighting up. Switching the battery pack on solved the problem.

As you can see, the micro:bit is screwed to the controlled board.  The ZIP LEDs are above the micro:bit and the 5x5 matrix is visible.

Side view. The robot has two wheels and uses the front and rear of the side walls as stabilising rails.

The ZIP LEDs are very bright.

Python Web Scraper

Search Engine Optimisation is a mysterious skill, the companies behind search engines do not want to make it easy to play their engines to force certain pages to the top of the results list.

However, one of the things that is listed as being useful is the presence of the key search words in the body of the HTML page.

The following is a simple Python script that retrieves the text from a supplied URL creates a searchable tree using LXML, extracts the text and then counts the occurrences of non numeric words longer than three characters.

# Functions to scrape the body text from a web page
# and return the top 10 occurring words
# Currently does not play well with HTML comments

# Also note that the order of the printed results is subject to change,
# this can be important if there are more than one element with the
# same occurrence as the last element displayed 

from lxml import html
import requests
import re
import collections

noisewords=['at','and','an','the','we','to','is','of','by','not','in','as','be','or','for']
def testgoodword(astring):
    if astring.strip() in noisewords:
        return False
    else:
        return True

def webscrape(aURL):

# Get the page from the URL
    page = requests.get(aURL)
# Make an HTML tree from the text
    tree = html.fromstring(page.content)

# Extract non script and non style text from the HTML tree
    bodytext=""
    for  elt in tree.getiterator():
        if elt.text is not None:
            if elt.tag!="script" and elt.tag!="style":
                if elt.text.strip()!='':
                    bodytext=bodytext+' '+elt.text
                    
# Define a regular expression to extract words
# (one or more alphanumerics followed by white space character)
    p = re.compile(r'\w\w+\s')

# Use a Counter collection to record the occurrences (the word is the key)
# Counter collections return zero if there is no element with a supplied key
    c = collections.Counter()

# Iterate through the "words2 found by the regular expression
    iterator=p.finditer(bodytext)
    for match in iterator:
        testword=match.group().strip().lower()
        if not testword.isnumeric():        # Ignore numbers
            if testgoodword(testword):      # Only use non noise words
                c[testword]+=1

    return c

# Print the most common words
def print_webscrape(c):
    print ('Most common:')
    for word, count in c.most_common(10):
        print ('\'%s\': %7d' % (word, count))

# Testing code 
if __name__ == "__main__":
    # execute only if run as a script
    c=webscrape("https://en.wikipedia.org/wiki/Python_(programming_language)")
    print_webscrape(c)

Most common:
'python':     189
'retrieved':     127
'programming':      49
'software':      35
'language':      29
'edit':      29
'with':      29
'pep':      26
'languages':      24
'march':      23
>>> 
Most common:
'python':     189
'retrieved':     127
'programming':      49
'software':      35
'edit':      29
'language':      29
'with':      29
'pep':      26
'languages':      24
'march':      23
>>> 

Note the order of "edit", "language" and "with" alter. If you really want the top ten, you need to expand the range until the you have a different count.
Most common:
'python':     189
'retrieved':     127
'programming':      49
'software':      35
'with':      29
'language':      29
'edit':      29
'pep':      26
'languages':      24
'march':      23
'february':      23
'org':      23
'from':      21
'van':      20
'december':      18
A you can see, there are three results with a count of 23. The Counter.most_common() function will return them in a random order

References

https://lxml.de/
https://docs.python.org/3/library/collections.html#collections.Counter

Simple backup of a Raspberry Pi

Back up SD card data to USB drive

The Raspberry Pi generally uses an SD (or microSD) card as its main storage. This is a cheap and generally reliable storage medium, but it is vulnerable to problems if the power is interrupted while data is being written.

If the power is interrupted (by the cat pulling the power lead out or you thinking it is the mobile 'phone's charger etc.) there is always the possibility that the SD card may become corrupted and all the data is lost.

Of course, you should be running a backup process on any system.

Back up home directory

To back up your home directory to a mounted USB drive, use the following command lines. You will need to know the name of the USB drive (have a look in /media.pi for the USB drive), replace the "name_of_usbdrive" with the actual drive name. You might want to replace "mypi" with the network name of the machine if you have a number of different systems.

First change directory to the /home directory (off of root).

cd /home/
Note space between change directory and directory path.

sudo tar czf /media/pi/name_of_usbdrive/backups/mypi_$(date +%Y%m%dT%H%M)home.tar.gz pi

Once that is running, go off and make a beverage of choice, it will take a while, and using the machine will upset the backup (as any use may cause a file change).

Note

One thing to be aware of, this will back up the contents of the trash bin. 

This might cause issues as the zip process falls over at about 4GB, and deleting files just puts them in the bin.

It might be worth emptying the bin occasionally.

You might also not want some deleted items to be backed up ^___^.

Raspbian Pixel desktop VNC Viewer

Since September 2016, all Raspberry Pi Raspbian distributions have included the RealVNC Virtual Network Computer server and viewer.

This allows the remote connection to the desktop on a Raspberry Pi machine. This is great if you want to to run your Raspberry Pi "headless" and do not want to use the Secure SHell (SSH).

You can connect to your Raspberry Pi from another (non Raspberry PI) machine by downloading and installing the RealVNC viewer (make your selection from the operating systems shown here https://www.realvnc.com/en/connect/download/viewer/. The RealVNC viewer is available from the Google Play Store (and probably from the Apple store as well). For Raspbian on the Raspberry Pi, wou can just get it from the Raspbian repository using the normal apt-get install (instructions are available from the above link)

Now I have an elderly Samsung Windows Netbook which is having a second life as a Raspbian Pixel netbook. Now this would be really useful to see what is happening on any live Raspberry Pis. Unfortunately, the non Raspberry PI distribution of Raspbian does not contain the RealVNC viewer.

So going back to the RealVNC downloads page, select the Linux option:
https://www.realvnc.com/en/connect/download/viewer/linux/

You need to select DEB x86 from the drop down menu, then click on Download VNC Viewer.

The file will (by default) appear in your Downloads directory. The one I received was called VNC-Viewer-6.18.907-Linux-x86.deb.

In the File Manager, right click and select Package Install. This will request your password to give the installer permission to make changes to the operating system (in the same way that normally you use SUDO to give heightened permissions).

Once it has been installed, the RealVNC viewer can be found in the Internet options on the start menu.

You will need to know either the ip address of the Raspberry Pi, or the network name. A default Raspberry Pi is normally called "raspberrypi", so should appear on the local network under the name "raspberrypi.home".

References:

https://www.realvnc.com/en/connect/download/viewer/

https://www.realvnc.com/en/connect/download/viewer/linux/

https://www.raspberrypi.org/blog/get-back-to-my-pi-from-anywhere-with-vnc-connect/

https://www.raspberrypi.org/search/vnc

PIMORONI Breakout Garden

The Pimoroni Breakout Garden HAT is a clever way of allowing the development of sophisticated sensor and display systems without having to do lots of soldering.

 As a HAT, it is fitted with a female 2x20 pin socket so it can be attached to any Raspberry Pi with 2x20 way expansion bus (Raspberry Pi Plus versions to date). Raspberry Pi Zero boards require the headers to be added (the Raspberry Pi Zero WH is supplied with the header already soldered/

This is going to be used with a Raspberry Pi Zero WH fitted into a Pibow Zero W case. To improve stability, a Pibow Breadboard Base was added to the bottom of the stack of laser cut acrylic slices that make up the case. The supplied bolts are long enough to take the additional slice.

Assembly was reasonably straightforward, I did need to ease a couple of the corners with a curved needle file to make the Zero board fit correctly.

To use a Zero with a keyboard and display, you do need to have an On The Go cable for the micro USB (top) and a Micro to HDMI adapter (bottom). I got mine as part of the Pimoroni OctoCam package.

The key feature of the Breakout Gatden HAT is the six large sockets on the top.

These are designed to take the special format breakout board from Pimoroni.

The first breakout board is the BME680 Air Quality Sensor.

Back

The front shows the five connectors. The power and ground connections are diode protected, so if you do plug them in the wrong way round, nothing bad (or indeed nothing at all) happens.

The breakout has temperature, pressure, humidity and air quality sensors built in.

The LSM303D is a combined accelerometer and magnetometer

It can provide X, Y, Z values for acceleration and magnetic field strength (making it suitable for use as a digital compass).

The third sensor is the BMP280 temperature, pressure and altitude sensor.

The last breakout board I have is the 1.12" 128x128 pixel OLED display board.

Next, putting them all to work.