[{"content":"The last post here was in September 2017. Time for a refresh.\nWhat changed The site moved from Pelican to Hugo. Posts are plain Markdown in a Git repo, and every push to main builds and deploys automatically.\nA code block, to check highlighting from datetime import date last_post = date(2017, 9, 16) print(f\u0026#34;{(date(2026, 11, 4) - last_post).days} days since the last post\u0026#34;) A table, to check styling Then (2017) Now (2026) Pelican Hugo reStructuredText Markdown Disqus comments none (yet) ","permalink":"https://chrisramsay.co.uk/posts/2026/11/hello-again/","summary":"Nine years after the last post, the site is back, rebuilt with Hugo.","title":"Hello Again"},{"content":"Introduction I never thought that I would be starting an article containing the statement \u0026ldquo;moving from Linux to Windows\u0026rdquo;, but this is exactly what will be happening with my main imaging scope set up in a few days. For those who don\u0026rsquo;t know anything about me, I have spent decades\u0026hellip; (hmm, yes it is now measurable in decades!) using Linux based operating systems - since the early Red Hat days of the late 1990\u0026rsquo;s, and being comfortable with that environment, it has always been the first port of call for any home project, such as a telescope mounted observation control platform.\nProblems Windows software more widely used, SGP Pro, N.I.N.A.?? Try :) Multiple issues with EKOS - corrupt DB, crashes, terrible UI, polar align tool Solution Bye bye Pi, hello mini PC 64 bit machine What am I Installing? ASCOM 6.x 64 bit drivers for stuff Pegasus UPB software SGP Pro Cartes du Ciel Stellarium (because I love it) ","permalink":"https://chrisramsay.co.uk/posts/2020/06/moving-from-linux-to-windows/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eI never thought that I would be starting an article containing the statement \u0026ldquo;moving from Linux to Windows\u0026rdquo;, but this is exactly what will be happening with my main imaging scope set up in a few days. For those who don\u0026rsquo;t know anything about me, I have spent decades\u0026hellip; (hmm, yes it is now measurable in decades!) using Linux based operating systems - since the early \u003ca href=\"https://en.wikipedia.org/wiki/Red_Hat_Linux\"\u003eRed Hat\u003c/a\u003e days of the late 1990\u0026rsquo;s, and being comfortable with that environment, it has always been the first port of call for any home project, such as a telescope mounted observation control platform.\u003c/p\u003e","title":"Moving from Linux to Windows"},{"content":"History Regarding computer hardware I have had to hand in the last two years or so, things have been in flux, you might say. I have not had ready access to a proper non-work Linux desktop environment where I could happily run Docker and Python. This has meant that keeping my Pelican blog up to date has been rather difficult, and hence the content has somewhat languished.\nI\u0026rsquo;m back to where I want to be now, having procured a new desktop machine quite capable of hosting a Docker environment effortlessly, so thought it was time to look once again at breathing life into this site.\nHere follows a description of the new publishing process!\npi@astropi-dslr:~ $ lsusb Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 003: ID 04a9:3145 Canon, Inc. EOS 450D Bus 001 Device 004: ID 1546:01a8 U-Blox AG [u-blox 8] Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub pi@astropi-dslr:~ $ dmesg | grep -i usb [ 0.106545] usbcore: registered new interface driver usbfs [ 0.106597] usbcore: registered new interface driver hub [ 0.106692] usbcore: registered new device driver usb [ 0.469467] usbcore: registered new interface driver r8152 [ 0.469538] usbcore: registered new interface driver lan78xx [ 0.469593] usbcore: registered new interface driver smsc95xx [ 0.469937] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1 [ 0.473610] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19 [ 0.473625] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.473637] usb usb1: Product: xHCI Host Controller [ 0.473649] usb usb1: Manufacturer: Linux 4.19.97-v7l+ xhci-hcd [ 0.473661] usb usb1: SerialNumber: 0000:01:00.0 [ 0.474203] hub 1-0:1.0: USB hub found [ 0.474786] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2 [ 0.474805] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed [ 0.475167] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 4.19 [ 0.475181] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.475193] usb usb2: Product: xHCI Host Controller [ 0.475204] usb usb2: Manufacturer: Linux 4.19.97-v7l+ xhci-hcd [ 0.475215] usb usb2: SerialNumber: 0000:01:00.0 [ 0.475725] hub 2-0:1.0: USB hub found [ 0.477402] usbcore: registered new interface driver uas [ 0.477496] usbcore: registered new interface driver usb-storage [ 0.484356] usbcore: registered new interface driver usbhid [ 0.484365] usbhid: USB HID core driver [ 0.838035] usb 1-1: new high-speed USB device number 2 using xhci_hcd [ 1.020651] usb 1-1: New USB device found, idVendor=2109, idProduct=3431, bcdDevice= 4.21 [ 1.020666] usb 1-1: New USB device strings: Mfr=0, Product=1, SerialNumber=0 [ 1.020679] usb 1-1: Product: USB2.0 Hub [ 1.022945] hub 1-1:1.0: USB hub found [ 1.348081] usb 1-1.2: new high-speed USB device number 3 using xhci_hcd [ 1.482114] usb 1-1.2: New USB device found, idVendor=04a9, idProduct=3145, bcdDevice= 0.02 [ 1.482132] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 1.482145] usb 1-1.2: Product: Canon Digital Camera [ 1.482157] usb 1-1.2: Manufacturer: Canon Inc. [ 1.798029] usb 1-1.1: new full-speed USB device number 4 using xhci_hcd [ 1.932515] usb 1-1.1: New USB device found, idVendor=1546, idProduct=01a8, bcdDevice= 2.01 [ 1.932531] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0 [ 1.932543] usb 1-1.1: Product: u-blox GNSS receiver [ 1.932556] usb 1-1.1: Manufacturer: u-blox AG - www.u-blox.com [ 3.948791] cdc_acm 1-1.1:1.0: ttyACM0: USB ACM device [ 3.950554] usbcore: registered new interface driver cdc_acm [ 3.950565] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters [ 4.109277] usbcore: registered new interface driver brcmfmac Aha - usb 1-1.1 -\u0026gt; ttyACM0\npi@astropi-dslr:~ $ cat /dev/ttyACM0 $GNTXT,01,01,02,u-blox AG - www.u-blox.com*4E $GNTXT,01,01,02,HW UBX-M80xx 00080000 *43 $GNTXT,01,01,02,ROM CORE 2.01 (75331) Oct 29 2013 13:28:17*4A $GNTXT,01,01,02,PROTVER 15.00*01 $GNTXT,01,01,02,GNSS OTP: GPS GLO, SEL: GPS GLO*67 $GNTXT,01,01,02,ANTSUPERV=AC SD PDoS SR*3E $GNTXT,01,01,02,ANTSTATUS=OK*25 $GNTXT,01,01,02,LLC FFFFFFFF-FFFFFFED-FFFFFFFF-FFFFFFFF-FFFFFF65*32 $GNRMC,113625.00,A,5332.26563,N,00022.97547,W,0.077,,160520,,,A*75 $GNVTG,,T,,M,0.077,N,0.142,K,A*3A $GNGGA,113625.00,5332.26563,N,00022.97547,W,1,12,0.76,43.3,M,46.5,M,,*68 $GNGSA,A,3,29,31,25,26,18,21,14,12,02,,,,1.24,0.76,0.98*1F $GNGSA,A,3,81,79,82,88,78,80,,,,,,,1.24,0.76,0.98*11 $GPGSV,4,1,13,02,14,037,12,04,14,326,18,05,09,070,15,09,00,356,*77 $GPGSV,4,2,13,12,08,103,20,14,10,230,23,16,09,283,,18,26,162,36*78 $GPGSV,4,3,13,21,25,171,36,25,36,106,35,26,41,286,36,29,73,077,37*7A $GPGSV,4,4,13,31,62,242,37*4D pi@astropi-dslr:~ $ sudo apt-get install gpsd gpsd-clients python-gps Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: gpsd gpsd-clients python-gps 0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded. Need to get 7,552 kB of archives. After this operation, 9,465 kB of additional disk space will be used. Get:1 http://raspbian.mirror.uk.sargasso.net/raspbian buster/main armhf gpsd armhf 3.17-7 [248 kB] Get:2 http://raspbian.mirror.uk.sargasso.net/raspbian buster/main armhf python-gps armhf 3.17-7 [98.6 kB] Get:3 http://raspbian.mirror.uk.sargasso.net/raspbian buster/main armhf gpsd-clients armhf 3.17-7 [7,204 kB] Fetched 7,552 kB in 8s (957 kB/s) Selecting previously unselected package gpsd. (Reading database ... 152934 files and directories currently installed.) Preparing to unpack .../archives/gpsd_3.17-7_armhf.deb ... Unpacking gpsd (3.17-7) ... Selecting previously unselected package python-gps. Preparing to unpack .../python-gps_3.17-7_armhf.deb ... Unpacking python-gps (3.17-7) ... Selecting previously unselected package gpsd-clients. Preparing to unpack .../gpsd-clients_3.17-7_armhf.deb ... Unpacking gpsd-clients (3.17-7) ... Setting up python-gps (3.17-7) ... Setting up gpsd (3.17-7) ... Creating/updating gpsd user account... Created symlink /etc/systemd/system/sockets.target.wants/gpsd.socket → /lib/systemd/system/gpsd.socket. Setting up gpsd-clients (3.17-7) ... Processing triggers for man-db (2.8.5-2) ... Processing triggers for systemd (241-7~deb10u3+rpi1) ... pi@astropi-dslr:~ $ sudo systemctl enable gpsd.socket pi@astropi-dslr:~ $ sudo systemctl start gpsd.socket pi@astropi-dslr:~ $ sudo systemctl status gpsd.socket ● gpsd.socket - GPS (Global Positioning System) Daemon Sockets Loaded: loaded (/lib/systemd/system/gpsd.socket; enabled; vendor preset: enabled) Active: active (running) since Sat 2020-05-16 12:44:26 BST; 46min ago Listen: /var/run/gpsd.sock (Stream) [::1]:2947 (Stream) 127.0.0.1:2947 (Stream) Tasks: 0 (limit: 4915) Memory: 28.0K CGroup: /system.slice/gpsd.socket May 16 12:44:26 astropi-dslr systemd[1]: Listening on GPS (Global Positioning System) Daemon Sockets. pi@astropi-dslr:~ $ gpsmon /dev/ttyACM0 NMEA0183\u0026gt; ┌──────────────────────────────────────────────────────────────────────────────┐ │Time: 2020-05-16T12:45:49.000Z Lat: 53 32\u0026#39; 16.02882\u0026#34; Non: 0 22\u0026#39; 58.06277\u0026#34; W│ └───────────────────────────────── Cooked TPV ─────────────────────────────────┘ ┌──────────────────────────────────────────────────────────────────────────────┐ │ GNRMC GNVTG GNGGA GNGSA GPGSV GLGSV GNGLL │ └───────────────────────────────── Sentences ──────────────────────────────────┘ ┌──────────────────┐┌────────────────────────────┐┌────────────────────────────┐ │Ch PRN Az El S/N ││Time: 124549.00 ││Time: 124549.00 │ │ 0 4 298 11 0 ││Latitude: 5332.27147 N ││Latitude: 5332.27147 │ │ 1 5 42 16 29 ││Longitude: 00022.97713 W ││Longitude: 00022.97713 │ │ 2 9 331 9 0 ││Speed: 0.031 ││Altitude: 38.1 │ │ 3 16 294 38 26 ││Course: ││Quality: 1 Sats: 12 │ │ 4 18 141 56 36 ││Status: A FAA: A ││HDOP: 0.73 │ │ 5 20 148 1 0 ││MagVar: ││Geoid: 46.5 │ │ 6 21 156 57 36 │└─────────── RMC ────────────┘└─────────── GGA ────────────┘ │ 7 25 122 8 26 │┌────────────────────────────┐┌────────────────────────────┐ │ 8 26 277 72 30 ││Mode: A3 ...s: 5 16 18 21 2 ││UTC: RMS: │ │ 9 27 252 11 30 ││DOP: H=0.73 V=1.06 P=1.29 ││MAJ: MIN: │ │10 29 68 42 33 ││TOFF: 0.042827049 ││ORI: LAT: │ │11 31 204 38 36 ││PPS: ││LON: ALT: │ └────── GSV ───────┘└──────── GSA + PPS ─────────┘└─────────── GST ────────────┘ (52) $GNGLL,5332.27163,N,00022.97762,W,124548.00,A,A*60 (68) $GNRMC,124549.00,A,5332.27147,N,00022.97713,W,0.031,,160520,,,A*7A (35) $GNVTG,,T,,M,0.031,N,0.057,K,A*3D (74) $GNGGA,124549.00,5332.27147,N,00022.97713,W,1,12,0.73,38.1,M,46.5,M,,*6E (60) $GNGSA,A,3,26,21,18,29,16,31,05,27,25,,,,1.29,0.73,1.06*12 (50) $GNGSA,A,3,80,81,73,79,,,,,,,,,1.29,0.73,1.06*1E (66) $GPGSV,3,1,12,04,11,298,,05,16,042,29,09,09,331,,16,38,294,26*76 (68) $GPGSV,3,2,12,18,56,141,36,20,01,148,,21,57,156,36,25,08,122,26*72 (70) $GPGSV,3,3,12,26,72,277,30,27,11,252,30,29,42,068,33,31,38,204,36*70 (64) $GLGSV,3,1,09,65,14,015,,66,02,072,,72,08,334,,73,30,319,24*68 (68) $GLGSV,3,2,09,79,23,153,32,80,75,201,31,81,36,044,22,82,89,206,*6C (29) $GLGSV,3,3,09,83,36,222,*50 (52) $GNGLL,5332.27147,N,00022.97713,W,124549.00,A,A*61 Done\n","permalink":"https://chrisramsay.co.uk/posts/2020/05/up-and-running-with-gpsd-on-raspberry-pi/","summary":"\u003ch2 id=\"history\"\u003eHistory\u003c/h2\u003e\n\u003cp\u003eRegarding computer hardware I have had to hand in the last two years or so, things have been in flux, you might say. I have not had ready access to a proper non-work Linux desktop environment where I could happily run Docker and Python. This has meant that keeping my Pelican blog up to date has been rather difficult, and hence the content has somewhat languished.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m back to where I want to be now, having procured a new desktop machine quite capable of hosting a Docker environment effortlessly, so thought it was time to look once again at breathing life into this site.\u003c/p\u003e\n\u003cp\u003eHere follows a description of the new publishing process!\u003c/p\u003e","title":"Up and Running with GPSD on Raspberry Pi"},{"content":"So here begins the June 2020 roundup. This month is a bit of a random collection of the moon, satellite passes, star trails and an inner Solar system planetary conjunction!\nMoon Almost Full Moon 5th May 2020 Moon with Bird, taken 2020-05-05 at 19:28:56 UT\nHere we have a static tripod shot of the moon with the addition of a bird flitting past at the same time. Taken with the (new to me) Canon EOS 550D with a Sigma 70-300mm lens at ƒ/14.0, 263.0 mm. Exposure 1/1600 of a second, ISO 6400. Moon phase is 96.2%, Moon age is 12.9 days.\nFlower Moon 6th May 2020 The full Moon in the month of May is known, amongst other names, as the \u0026ldquo;Flower Moon\u0026rdquo;.\nFlower Moon and telescope, taken 2020-05-06 20:48:00 UT\nCanon EOS 550D with a 19-35mm lens at ƒ/11.0, 32.0 mm. Exposure 30 seconds at ISO 200. Moon phase is 99.4%, Moon age is 14.1 days.\nStar Trails Here is the first light output from the Raspberry Pi powered DSLR controller I began building back at the beginning of May 2020.\nTesting the new astropi-DSLR - Star Trails 2020-05-13\nThe star trails above were captured in a single session starting at 22:35:32 UT with the last frame shot at 00:57:47 UT. This is a total of 70 exposures, 120 secs each at ISO 800. Total integration time of 140 minutes (2 hours 20 minutes .)\nVenus and Mercury Conjunctions Night 1, 21st May Venus and Mercury will be in close conjunction this evening, approx 1.5 degrees apart and 15 degrees altitude (for Venus) as the Sun set\nVenus and Mercury Conjunction, taken 2020-05-21 at 20:55:05 UT\nMeasure degree separation in Stellarium\nNight 2, 22nd May Venus and Mercury will be in close conjunction this evening, approx 1.5 degrees apart and 15 degrees altitude (for Venus) as the Sun sets\nVenus and Mercury Conjunction, taken 2020-05-22 at 20:45:22 UT\nISS and Associated Passes The International Space Station has made a number of bright passes in the evening skies of May.\nStatic Exposure of ISS Pass #1 22nd May 2020 ISS Pass #1 2020-05-22\nISS Pass in 24 frames, 10 seconds each, taken 2020-05-22 from 21:07:23 UT to 21:12:48 UT. Canon EOS 450D with 8mm Fisheye lens at ISO 100\nStatic Exposure of HTV-9 Pass on 22nd May 2020 Kounotori 9 (こうのとり9号機), also known as HTV-9 is the 9th flight of the H-II Transfer Vehicle, a robotic cargo spacecraft to resupply the International Space Station.\n3344 It was launched on 20 May 2020, at 17:31:00 UTC.\nHTV-9 Supply Vehicle Pass 2020-05-22\nHTV-9 Supply Vehicle Pass in 7 frames, 20 seconds each, taken 2020-05-22 from 22:01:43 UT to 22:04:01 UT. Canon EOS 450D with 8mm Fisheye lens at ISO 400\nStatic Exposure of ISS Pass #2 22nd May 2020 ISS Pass #2 2020-05-22\nISS Pass in 14 frames, 30 seconds each, taken 2020-05-22 from 22:41:23 UT to 22:49:10 UT. Canon EOS 450D with 8mm Fisheye lens at ISO 400\n","permalink":"https://chrisramsay.co.uk/posts/2020/05/june-2020-roundup/","summary":"\u003cp\u003eSo here begins the June 2020 roundup. This month is a bit of a random collection of the moon, satellite passes, star trails and an inner Solar system planetary conjunction!\u003c/p\u003e","title":"June 2020 Roundup"},{"content":"Introduction As well as the telescope pier I like to have a DSLR (an astro-modified Canon EOS 450D) on a Skywatcher Star Adventurer mount around in the garden. It makes for a nice little mobile arrangement that can be set up at a moment\u0026rsquo;s notice. Usually the set up consists of the camera, a lens warmer, power for the mount and an intervalometer for controlling the timing of pictures being captured.\nHaving really enjoyed setting up a Raspberry Pi 4 running KStars, EKOS and INDI for the pier based telescope I thought that putting a \u0026lsquo;one stop\u0026rsquo; solution together, able to do the same as the above plus a few extras, for the DSLR would be a great little side project.\nThere are a few little limitations that I like to be able to overcome with the current DSLR set up.\nBattery life on the DSLR is pretty short usually, and getting shorter as the battery ages. I am currently getting about three to three and a half hours. To do longer star trails and so on, ideally I would like that limitation to go altogether.\nThe mount and heater are powered with a portable battery, and again diminishing battery life proves to be a bit of an issue.\nLastly, a minor point; messing about with SD cards or plugging the camera into a laptop to get the images. It would be nice to be able to do that remotely without wires or carrying cards about.\nRequirements I decided that the control box would have to be able to fulfil the following requirements:\nProvide power for the DSLR, lens warmer and Skywatcher mount Download image data from the DSLR to local storage Be able to send and receive data over my local network (WiFi or ethernet) for remote desktop control and downloading images Be powered from a 12 V power supply Perform the above tasks reliably and ideally in a single, and reasonably weather-proof package Plan The overall idea for the box is that it is supplied power via a 12 V DC and has the ability to be connected to my local network via Ethernet. Why not WiFi, you might ask? I have never had really trouble-free operation with discrete devices running on WiFi; in other words, once they become unresponsive for some reason, one ends up having to walk to where they are and plug them in anyway. Having a wired connection just eliminates a frequent source of problems as far as I am concerned.\nSurprisingly, perhaps, an ethernet connection and a 12 V supply can be achieved over a single CAT5 cable; read on at Power and Network on a Single Cable for seeing how that\u0026rsquo;s possible. The peripherals have a couple of different power requirements, thankfully all of which can be supplied using USB 5 V power distribution:\nCanon EOS 450D requires 7.1 V, achievable via a USB powered battery insert Lens warmer required 5 V but has a USB connector Raspberry Pi, 5 V powered via a USB C input Skywatcher mount, 5 V powered via a mini USB input I had a search for something that would be capable of taking a 12 V input and stepping it down to 5 V with USB ports (not data, just power). Turns out, such a thing exists in the form of a 4 port USB power supply, I think for the automotive market - 12V To 5V 5A Step Down 4 Port USB DC Power Supply Module Mobile Tablets Charger as it was titled on eBay. Just make sure that if you do get one of these, always be sceptical regarding the current rating. It may well not be capable of having 5 A drawn from it!\nWith the above in mind I built up a list of things that would have to be purchased to start the project, namely:\nRaspberry Pi Model 4B 32 Gig SD Card IP66 Project Box 1 x short USB C lead (Pi power supply) 2 x Mini USB leads 1 x USB to 7.1 V Canon power pack 1 x 5 V / 5 A 4 port USB power supply, stepping down from 12 V + (sold in pairs) 5 V 40mm cooling fan Next job was then to hit a well known auction site and do some shopping with the exception of the Raspberry Pi which would come from my usual supplier RS online.\nConstruction The build took only a few hours once all the bits that I needed had arrived. Post being what it is at the moment (COVID-19 times) I had to wait a few weeks rather than a few days, but never mind\u0026hellip; keeps one in suspense! The Raspberry Pi and USB power supply\nastrpi-dslr Parts\nPower and Network on a Single Cable Yes, it\u0026rsquo;s possible! As long as you are not after gigabit transfer speeds from the control box to somewhere else on your network; not a problem for me!\nUse of sending power down 4 redundant lines in Ethernet.\nPin Function Wire Colour (T568B ) 1 TX+ white/orange stripe 2 TX- orange solid 3 RX+ white/green stripe 4 12 V DC #1 blue solid 5 Gnd #1 white/blue stripe 6 RX- green solid 7 12 V DC #2 white/brown stripe 8 Gnd #2 brown solid You can probably see that there are a couple of spare pairs of wires\u0026hellip;\nPower and network cable for astropi-dslr\nLED for power and activity - no, go for a simple acrylic light pipe arrangement.\nastrpi-dslr \u0026ldquo;on\u0026rdquo; light\nStuff\nThe astropi-dslr Internals\nThings\nTesting Network issues where the Pi won\u0026rsquo;t connect. Aha, /etc/rc.local needs editing with an ethtool command.\n# Limit net adapter speed to less than gigabit ethtool -s eth0 speed 100 duplex full autoneg off So it goes awry here when I start to connect all the bits. Low voltage causing the Pi to reset.\nResult Here it is\nUp and running in the garden\nOvernight Problem - Random Shutdown So I decided to run an overnight test by setting up a sequence of 120 frames, 120 seconds each. Set the session running before bed and left it going by itself until the early hours. Waking up and checking the drive contents the following morning it becomes apparent that the Pi has rebooted itself at some point; only 93 frames had been taken and the uptime on the Pi was less than three hours.\nThe first step was to take a look at some system logs on the Pi. As it turns out I was lucky with my first strike - the kernel log at /var/log/kern.log:\npi@astropi-dslr:/var/log $ tail -f /var/log/kern.log May 14 01:41:09 astropi-dslr kernel: [12360.802989] brcmfmac: power management disabled May 14 01:43:35 astropi-dslr kernel: [12506.779127] Under-voltage detected! (0x00050005) May 14 01:43:39 astropi-dslr kernel: [12510.939093] Voltage normalised (0x00000000) May 14 01:45:37 astropi-dslr kernel: [12629.501028] Under-voltage detected! (0x00050005) May 14 01:45:42 astropi-dslr kernel: [12633.660920] Voltage normalised (0x00000000) May 14 01:46:25 astropi-dslr kernel: [12676.809172] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready May 14 01:46:25 astropi-dslr kernel: [12676.809181] brcmfmac: power management disabled May 14 01:47:38 astropi-dslr kernel: [12750.151926] Under-voltage detected! (0x00050005) May 14 01:47:44 astropi-dslr kernel: [12756.381870] Voltage normalised (0x00000000) May 14 01:49:41 astropi-dslr kernel: [12872.863049] Under-voltage detected! (0x00050005) May 14 01:49:45 astropi-dslr kernel: [12877.023085] Voltage normalised (0x00000000) May 14 01:51:41 astropi-dslr kernel: [12992.812574] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready May 14 01:51:41 astropi-dslr kernel: [12992.812589] brcmfmac: power management disabled May 14 01:51:43 astropi-dslr kernel: [12995.584406] Under-voltage detected! (0x00050005) May 14 01:51:48 astropi-dslr kernel: [12999.744423] Voltage normalised (0x00000000) May 14 01:53:44 astropi-dslr kernel: [13116.225814] Under-voltage detected! (0x00050005) May 14 01:53:50 astropi-dslr kernel: [13122.465796] Voltage normalised (0x00000000) May 14 01:56:57 astropi-dslr kernel: [13308.816476] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready Test\n","permalink":"https://chrisramsay.co.uk/posts/2020/05/building-a-raspberry-pi-flavoured-dslr-control-unit/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eAs well as the telescope pier I like to have a DSLR (an astro-modified Canon EOS 450D) on a Skywatcher Star Adventurer mount around in the garden. It makes for a nice little mobile arrangement that can be set up at a moment\u0026rsquo;s notice. Usually the set up consists of the camera, a lens warmer, power for the mount and an intervalometer for controlling the timing of pictures being captured.\u003c/p\u003e\n\u003cp\u003eHaving really enjoyed setting up a \u003ca href=\"https://www.raspberrypi.org/products/raspberry-pi-4-model-b/\"\u003eRaspberry Pi 4\u003c/a\u003e running \u003ca href=\"https://edu.kde.org/kstars/\"\u003eKStars\u003c/a\u003e, \u003ca href=\"https://www.indilib.org/about/ekos.html\"\u003eEKOS\u003c/a\u003e and \u003ca href=\"https://indilib.org/\"\u003eINDI\u003c/a\u003e for the pier based telescope I thought that putting a \u0026lsquo;one stop\u0026rsquo; solution together, able to do the same as the above plus a few extras, for the DSLR would be a great little side project.\u003c/p\u003e","title":"Building a Raspberry Pi Flavoured DSLR Control Unit"},{"content":"It has been a strange couple of months, what with the current Coronavirus epidemic raging around us, but I have managed to capture a few astronomical events from the garden in any case. So here begins the April 2020 roundup.\nISS Passes The International Space Station has made a number of bright passes in the evening skies for the last few weeks.\nTracking Exposure 26th March 2020 ISS passing through Taurus with the Pleiades (M45), taken 2020-03-26 This is a 90 second exposure at ƒ/3.5 with an astro-modified EOS 450D and a EF50mm f1.8 II lens, at ISO 800 with a polar aligned mount tracking at Sidereal rate. In the top right corner of the frame is the Pleiades open star cluster (M45), also known as the Seven Sisters. The path of the ISS itself goes just to the left of the red giant α Tauri, Aldebaran which is the brightest star in the constellation Taurus. The stars scattered in and around Aldebaran are known as the Hyades, one of our closest open clusters, at approximately 153 ly distant.\nStatic Exposure 1st April 2020 ISS pass with clouds and trees, taken 2020-04-01 Low ISS pass taken 2020-04-01 at 20:36:38 UT on a (static) tracking mount. It is an 82 second exposure at ƒ/3.5 with an astro-modified Canon EOS 450D and an EF50mm f1.8 II lens, at ISO 400. As can be seen the stars have trailed due the static mount, however a constellation can still be seen; Orion appears in the right hand half of the frame, complete with the very bright Orion Nebula (M42) making a fuzzy patch just above the electricity cables.\nThe Moon With quite a few clear evenings in March and April this year, the Moon has been a very prominent target.\nEarthlight 27th March 2020 In March this year I purchased a well loved Samyang 135mm T2.2 lens which gives me a 9°24.00\u0026rsquo; by 6°16.50\u0026rsquo; view of the sky with the Canon EOS 450D. I decided that some Moon shots would be a great way of giving it some first light.\nEarthlight Moon taken on 2020-03-27 at 18:18:10 UT\nI quickly set up in the garden for the above picture of the Moon with the express hope of capturing some earthlight. Picture taken with the Canon EOS 450D on a tripod; a 4 second exposure at ISO 400. Cropped and played about with a bit in Affinity Photo.\nPink Moon 8th April 2020 The full Moon in April was known as the \u0026ldquo;Pink Moon\u0026rdquo; and was also a \u0026ldquo;supermoon\u0026rdquo; due to its ever so slightly larger appearance; a result of the Moon reaching perigee in its orbit of Earth.\nPink Moon Rising with Moody Clouds on 2020-04-08 at 20:23:46 UT\nThe picture above was taken with the Canon EOS 450D and the Samyang 135mm T2.2 lens; a 1/6 second exposure at ISO 1600. Moon phase is 99%, Moon age is 15.7 days.\nYoung Moon 25th April 2020 Nipped outside to take a chance picture of a rather young Moon on the evening of the 25th April.\nMoon, taken 2020-04-25 at 19:57:46 UT\nTaken using the Canon EOS 450D and the EF 50mm ƒ/1.8 II lens, 1/15 sec exposure at ISO 800. Moon phase is 3%, Moon age is 1.61 days. Cropped and played about with a bit in Affinity Photo.\nWaxing Moon 29th April 2020 As luck would have it I inherited an old Canon EOS 550D with a Sigma 70-300 mm lens. Below is a first light attempt with the Moon.\nMoon, taken 2020-04-29 at 21:02:51 UT\nAbove we can see the Moon waxing towards half with a 1/125 second exposure at ISO 800. Moon phase is 39%, Moon age is 6.3 days. Cropped and played about with a bit in Affinity Photo.\nLyrid Meteor Shower This year I decided to have an attempt at getting a picture or two from the annual Lyrid meteor shower; capturing a meteor is something I have never tried before. The obvious solution seemed to be to aim a camera at or around the radiant area in the sky around the constellation of Lyra.\nThe Lyrids are visible from around 16th April to the 26th April, peaking typically on the night of the 22nd April.\nIn order to have the best chance of getting a meteor I figured that seeing as much of the sky as possible would be sensible. So it was time to bring out the F3.5/8 mm UMC Fish Eye lens and have a few hours under the stars. I decided to put the camera (Canon 450D astro-modified) onto the Skywatcher Star Adventurer mount in order to track at Sidereal rate. Some test pictures were taken initially, mainly to test various exposures, times and angles on the sky.\nFor all three nights the camera was controlled using a laptop running a piece of software called Astrophotography Tool.\nNight 1 - 20th April 2020 I set the camera running at 23:05 UT with a planned sequence of 100 exposures, 120 seconds each at ISO 800. Unfortunately I had accidentally set the aperture to ƒ/22 whilst setting up in the dark. Not much was captured this night as a result.\nNight 2 - 21st April 2020 This night, I made sure that the lens aperture was set correctly at ƒ/3.5!\nBelow is a single exposure, taken from a series of test exposures, 120 seconds at ISO 800. Stacked using Deep Sky Stacker, finished off using Affinity Photo in black and white.\nLyrid Meteor bursts in Leo, taken 2020-04-21 at 21:21:06 UT\nThe wide angle lens takes in the long track of the meteor rather well; the track stretches from the constellation of Boötes, right across Coma Berenices and into Leo where it finishes in a flash.\nHappy with the tests, I set up for a series of exposures aimed at capturing the post midnight peak - starting at 23:04 UT and running until 03:20 UT. The sequence consisted of 102 files, 150 seconds each at ISO 800 when complete. Not a single frame had captured a meteor.\nNight 3 - 22nd April 2020 My luck was in; a third clear night in a row! Once again the camera was set up with the fish eye lens, however this time I started a little earlier due to some incoming cloud later on in the night. This nights\u0026rsquo; sequence planned in a series of 101 exposures, 152 seconds each at ISO 800 from 21:26 UT to 01:41 UT the following morning.\nThis night proved to be moderately more successful with two meteors captured. neither were awe-inspiring, but I was happy with having at least something at the end of the session.\nMeteor in Coma Berenices during the Lyrids shower, taken 2020-04-22 at 22:07:19 UT\nSo you have to look quite hard to see this first one (picured above), just to the right of centre of the frame, making its way through the constellation of Coma Berenices.\nLyrid meteor in Ursa Major and Minor 2020-04-23 at 00:22:33 UT\nThe second capture took place the other side of midnight, streaking a 26° long line between the stars of Alkaid (η UMa) and Kochab (β UMi). The image above has been cropped quite significantly and changed to black and white from a colour exposure using Affinity Photo.\nThat was it for the Lyrids this year; quiet and nowhere near as exciting as it could have been, but it does seem that most meteor hunters had a similar experience. Roll on the next meteor shower.\nStar Trails Having recently got hold of a fish eye lens I thought that it would be interesting to leave the camera pointing at the sky, take a series of reasonably long exposures and stack them together to generate some star trails. Being that the stars move approximately 15° per hour, it made sense that even a short amount of exposure time would produce some reasonable star trails.\nBack Garden Star Trails 2020-04-16\nThis first image comprises 30 exposures of 60 seconds each, at ISO 400, using the Canon EOS 405D with the F3.5/8 mm UMC Fish Eye lens.\nLooking for Lyrids 2020-04-20\nThis second image is a composite from a sequence of 102 exposures at 150 seconds each, ISO 800 from one of my attempts to capture meteors during the Lyrid meteor shower. Again these were taken with the Canon EOS 405D and F3.5/8 mm UMC Fish Eye lens No meteors imaged, but a good set of star trails nevertheless.\n","permalink":"https://chrisramsay.co.uk/posts/2020/04/april-2020-roundup/","summary":"\u003cp\u003eIt has been a strange couple of months, what with the current Coronavirus epidemic raging around us, but I have managed to capture a few astronomical events from the garden in any case. So here begins the April 2020 roundup.\u003c/p\u003e","title":"April 2020 Roundup"},{"content":"Introduction Setting up a new telescope is not an easy task. It takes some planning and thought to get the set up right first time. I have made an attempt to plan out everything that I will need before getting started. Using a handy copy of Visio I had lying about I have put together a handful of block level diagrams to help me out. This process gives me an overview of what is needed to make sure that the system is complete and that everything has been thought of, as far as possible. Following is a run through of my plans and my thoughts regarding what issues they raised.\nThe basic idea has always been that the telescope should be automated as much as possible. The extent of the automation will that I will be able to put together an imaging session or two and set it running, all from the comfort of a warm room, via a remote desktop connection to an on board computer. In this case I consider an imaging session to be: target (re)acquisition, focus, capture a set of lights via a combination of filters, occasionally re-focus depending on temperature variation, take care of meridian flips if necessary, and finally shut down when the session is complete.\nFor the (computing) brains behind the operation, a Raspberry Pi 4 has always been the goal, being that they are small, have reasonably low power requirements and run on Linux, an operating system I am truly comfortable with.\nBlock Level Diagrams As can be seen, I started by preparing some block level diagrams to assist in mapping out the entire telescope set up. The set up has been divided into multiple blocks based on what the modules are transmitting from block to block. The imaging train transmits photons between blocks, so that seemed like a pretty obvious place to start:\nImaging Train So this is how the image capture part of the set up currently looks. The photo below was taken whilst I was checking the back focus from the Skywatcher field flattener; it has to be 75 mm from a particular point on the flattener to the CCD chip surface, give or take a small amount to account for the refractive index of the filters.\nImaging train in real life\nAs is apparent in the photo above, I have not yet added the off axis guide camera (OAG) to the set up. That will happen once I have the main imaging camera properly focused. From right to left, the image shows the Skywatcher field flattener, a custom made 15 mm spacer, a Starlight Xpress Loadstar 2 camera mounted onto an off axis guider assembly, 7 position filter wheel and finally the Starlight Xpress CCD imaging camera. The interfaces between the rear face of the field flattener and the front face of the OAG / filter wheel are a male M62 thread and a male T-thread (42 mm) respectively, necessitating a custom made 15 mm spacer beautifully engineered by John at J-Tech Design.\nOnce everything was together it was time to get drawing. Below is a plan of just the imaging train as above with the light path shown with blue arrows. Not much in itself, but it makes sense once it is part of the rest of the plan.\nBlock level imaging train plan\nOverall Block Plan Next was to try and understand just how many cables will be needed, what type and what sort of lengths. Also, each piece of the puzzle has different types of USB connections. Thankfully everything that requires 12 V has the same, centre positive 5.5 mm by 2.1 mm connectors. That definitely makes life easier!\nAdditionally, the plan below has helped me to make sense of what was needed for both power distribution and USB connections. Once everything is laid out, it starts to become obvious where things are missing, such as: I need to think about getting a powered USB hub, or, do I have enough USB A to USB B leads?\nBlock level physical connections\nThe block plan has also made sure that I have not forgotten about getting power to the Raspberry Pi 4; these latest models are now powered via a USB C connector. The initial plan for this was a step down converter from 12 V to 5 V and then somehow butcher a USB-C lead to provide the power to the Pi. Other options I have considered are, for example, sending a 5 volt supply via the header pins on the Raspberry Pi circuit board; unfortunately this bypasses some current protection devices in the process, so probably not the best idea.\nUSB Connections From the overall plan I subsequently extracted the devices with a USB connection and any associated USB cable. This leads me to my first thought with the USB layout\u0026hellip; I need a powered USB hub, and therefore power to run the hub itself. I will ideally need to find a hub that will run from a 12 V supply (and in a really ideal world, having a 5.5 x 2.1 centre positive power feed!) to try and keep the amount of adaptations down to a minimum.\nBlock level USB connections\nRegarding cables, I am going to need a mix of different types. A number of mini to mini and mini to USB B cables to connect to the in-built hub on the SX 814 to provide power and control for the off axis guide camera, the filter wheel and the focus cube. The in-built hub on the imaging camera will then be connected to a theoretical standalone powered USB hub, and from there communicate with the Raspberry Pi. Control of the EQ6R Pro mount will be achieved via a Lynx Astro USB to EQ direct lead from this standalone hub also.\nThankfully I have a habit of hoarding old cables and connectors that come with electrical items purchased over the years. This has come in useful as I have been able to find all the leads as required for the cable layout on the telescope.\nPower and Network Once I was happy with the USB layout, it was time to turn to looking at the power and network layer. Again, the plan only has items that require power from 12 V cables and the cable runs themselves.\nFrom the plan, it appears that I am going to need to find a neat way of distributing 12 volts to a number of different bits of equipment on the telescope. I have spent quite some time on the astronomy forums researching how people achieve this. There does not really seem to be a single standard - no surprises there. Some systems seem to be based on automotive type connectors with blade fuses and so on, others utilise \u0026ldquo;Anderson Power Pole\u0026rdquo; connectors as used by the Ham Radio community. This is something that definitely needs more thought.\nBlock level power and network connections\nSome thought needs to be put towards handling dew that will no doubt build up on the objective lens on the telescope. Currently, I have a home built dew prevention system that requires a 12 volt supply (not a problem then) but it is very bulky; finding some space to mount it might be quite tricky.\nNext? The gold standard (perfect) solution for all this would be a combined USB hub and 12 V power supply complete with a dew heater controller system. I wonder if such a thing exists?\n","permalink":"https://chrisramsay.co.uk/posts/2020/04/equipment-setup-thoughts-part-one/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eSetting up a new telescope is not an easy task. It takes some planning and thought to get the set up right first time. I have made an attempt to plan out everything that I will need before getting started. Using a handy copy of Visio I had lying about I have put together a handful of block level diagrams to help me out. This process gives me an overview of what is needed to make sure that the system is complete and that everything has been thought of, as far as possible. Following is a run through of my plans and my thoughts regarding what issues they raised.\u003c/p\u003e","title":"Equipment Setup Thoughts - Part One"},{"content":"Introduction This series of articles will chronicle the process of building a permanent astronomy telescope pier in my garden. A permanent pier makes the setting up process for an night of astrophotography or visual observing far less laborious. This is because the pier is already aligned with the celestial pole, via the North peg on top of the mount adapter. Power and network adapters are immediately to hand, removing the work of laying cables out from the shed and putting them away later after each observing or astrophotography session.\nCutting the Turf Well, I had to start somewhere, so I decided to cut a trench to the site of the pier as my first step.\nTurf cut back where the trench will be dug The trench is approximately 50 cm deep, keeping the cables far enough underground to be out of the way of anything that might happen in the garden. Thankfully the ground here is light soil and rock-free, therefore quite easy to cut into.\nInstalling the Cabling The next step was to get some cables into the ground. I acquired some red polypropylene cable conduit, with an inside diameter of approximately 27 mm to act as a waterproof path for the cable from the shed to the pier.\nRed polypropylene conduit laid out ready to be placed in the trench The conduit carries three separate cables: a 3.0 mm2 33 Amp twin core 12 volt automotive cable, a 2.5 mm2 twin and earth 240 VAC mains cable and a Cat 5e network cable. These will all break out into waterproof enclosures mounted on the side of the pier.\nTrench cut into the path at the side of the shed As can be seen in the picture above I had to cut a trench into the path at the side of the shed to get the cables into the building where the services are. Being that the concrete was old (25 years +), it was very hard and took a long time to get through.\nCables entering the shed The cables enter the shed via a waterproof electrical box. This will make access to the conduit easier should I need to add another cable one day; highly unlikely, I think. Having the box there also avoids having to make a difficult 90° turn from under the path into the brickwork.\nThe pier site with the conduit in place At the other end of the conduit lies the pier site itself (of course!). At the moment there is just a large hole ready for the concrete pour and a lintel where the base of the pier will fit. The conduit emerges near the base of the pier site and it will then turn 90° in an upwards direction, going up inside the pier in order to break out near the top.\nWhat\u0026rsquo;s Next? Next comes the building of the pier itself; the plastic pipe for the pier itself, pier head mount adapter with associated metalwork and the breakout box for the electrical and network connections. At this stage the weather has intervened, so everything is sitting waiting for the rain to pass.\nAs usual, all outside projects are subject to the vagaries of the rather changeable British weather.\n","permalink":"https://chrisramsay.co.uk/posts/2020/01/astro-pier-part-one/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eThis series of articles will chronicle the process of building a permanent astronomy telescope pier in my garden. A permanent pier makes the setting up process for an night of astrophotography or visual observing far less laborious. This is because the pier is already aligned with the celestial pole, via the North peg on top of the mount adapter. Power and network adapters are immediately to hand, removing the work of laying cables out from the shed and putting them away later after each observing or astrophotography session.\u003c/p\u003e","title":"Astro Pier - Part One"},{"content":"History Regarding computer hardware I have had to hand in the last two years or so, things have been in flux, you might say. I have not had ready access to a proper non-work Linux desktop environment where I could happily run Docker and Python. This has meant that keeping my Pelican blog up to date has been rather difficult, and hence the content has somewhat languished.\nI\u0026rsquo;m back to where I want to be now, having procured a new desktop machine quite capable of hosting a Docker environment effortlessly, so thought it was time to look once again at breathing life into this site.\nHere follows a description of the new publishing process!\nDocker Revisited In the previous post about this I wrote about how Docker became part of the publishing process, replacing Vagrant which had become a pain to because of such heavy maintenance overheads and the lack of task focus that a Vagrant machine suffered from.\nHowever, at the time, I approached Docker very much in a Vagrant mindset; that is, I prepared Docker images with a view to logging into them once the containers were running. Once logged in I used the pelican bash script to do the usual drafting, publishing and deployment operations.\nNew Docker Image With a fresh Linux installation on a new machine, an opportunity arose to do things differently. I created a new Dockerfile with a minimal installation of Python, based on Alpine Linux\nFROM alpine:latest COPY requirements.pip /tmp/ RUN apk add --no-cache python3 RUN pip3 install --upgrade pip \u0026amp;\u0026amp; \\ pip3 install -r /tmp/requirements.pip CMD [\u0026#34;pelican\u0026#34;, \u0026#34;-h\u0026#34;] The requirements.pip is very simple, specifying a pretty minimal set of libraries:\npelican pelican-minify markdown beautifulsoup4 Thus was born a 35.29 MB image, small but perfectly formed and an ideal fit for the task.\nFile System Layout Next job was to lay out the file system so that the Pelican plugins, themes and site build requirements were all together in one area.\n. ├── pelican-plugins ├── pelican-themes └── sites ├── chrisramsay.co.uk │ ├── build │ │ ├── configs │ │ │ ├── preview_settings.py │ │ │ └── publish_settings.py │ │ ├── configs │ │ ├── content │ │ ├── preview.sh │ │ └── publish.sh │ ├── local │ └── publish └── someothersite.com The pelican-plugins and pelican-themes directories are just git clones of Pelican\u0026rsquo;s Plugins and Themes repos respectively.\nThe Preview and Publish Scripts The preview.sh and publish.sh scripts are what are used to render the HTML into either a local development directory or a publish pre-deployment directory. Essentially, these scripts start a container, mounting several volumes and then running the pelican command with the necessary child commands. Let\u0026rsquo;s have a look at preview.sh:\n#! /bin/bash BASE=/home/user sudo docker run \\ --rm \\ --volume ${BASE}/sites/chrisramsay.co.uk/build:/site \\ --volume ${BASE}/sites/chrisramsay.co.uk/build/content:/content \\ --volume ${BASE}/sites/chrisramsay.co.uk/local:/output \\ --volume ${BASE}/pelican-themes/pelican-bootstrap3:/theme \\ --volume ${BASE}/pelican-plugins:/plugins \\ chrisramsay/alpine-pelican \\ pelican /content -o /output -t /theme -s /site/configs/preview_settings.py Once the HTML site has been rendered it\u0026rsquo;s time to take a look at it using a browser. This is where NGINX comes in.\nUsing NGINX for Previewing the Site Here I am using an official Docker image from NGINX; pretty straightforward:\n$ sudo docker run --name nginx-cr -p 8080:80 -v /sites/chrisramsay.co.uk/local:/usr/share/nginx/html:ro -d nginx In the above code I am firing up an instance of and NGINX web server to serve the new static HTML files from the local directory. There is an order to doing things here\n","permalink":"https://chrisramsay.co.uk/posts/2020/01/pelican-blog-publishing-process-revisited/","summary":"\u003ch2 id=\"history\"\u003eHistory\u003c/h2\u003e\n\u003cp\u003eRegarding computer hardware I have had to hand in the last two years or so, things have been in flux, you might say. I have not had ready access to a proper non-work Linux desktop environment where I could happily run Docker and Python. This has meant that keeping my Pelican blog up to date has been rather difficult, and hence the content has somewhat languished.\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m back to where I want to be now, having procured a new desktop machine quite capable of hosting a Docker environment effortlessly, so thought it was time to look once again at breathing life into this site.\u003c/p\u003e\n\u003cp\u003eHere follows a description of the new publishing process!\u003c/p\u003e","title":"Pelican Blog Publishing Process Revisited"},{"content":"Progress This post follows on at quite a distance from my last post on the subject, and until recently I had not realised quite how long this project has been languishing on the bench. Like many of the electronics projects I undertake they need to be able to withstand being outside and exposed to weather conditions, dealing with packaging them to handle harsher environments is always the most difficult part.\nFor the same reason as above, I have also changed one of the key measurement components, the humidity sensor.\nI am also dropping the Arduino Uno in favour of the Arduino Yún. I wrote a brief article on the Yún some time ago.\nPackaging for the Outdoors Progress is now being made, finally, on the assembly of the containing hardware of the device. The constituent parts of the detector, namely the box housing for the electronics including the IR detector and the thermostatically controlled heated aluminium mirror have been completed. Now I am in the process of mounting the parts together into a detector assembly. In addition I have made progress on a weather resistant external housing for the entire system to safely be exposed to the elements outside.\nDetector Assembly After thinking about things for a little while, I decided to mount the whole assembly on an acrylic base that will slide in and out of the back of the housing to make maintenance easier. This also keeps the IR detector and mirror at a calibrated distance and angle that should not change between hardware maintenance schedules.\nThe following photo shows the whole setup:\nDetector and mirror assembly\nThe photo below shows the use of a external cable exit gland to hold the IR detector in place. The gland tightens around the plastic housing of the IR detector leaving enough flexibility for fine adjustment if required. The cables have been tidied up using some old heatshrink I had lying around from another project.\nAnother view of the detector and mirror assembly\nBelow you can see the wires exiting from underneath the aluminium mirror. The wires leading from the centre hole at the base go to a DS18B20, a temperature detector, there to measure the surface temperature of the mirror. The black and red wires on the outside edges are connected to heater wires underneath the mirror.\nAnother view of the detector and mirror assembly\nExterior Housing Progress is also being made with the exterior housing for the whole setup. The exterior housing (white part) is made from spare 114 mm square section acrylic gutter downpipe. The box (grey) for containing the Arduino and power supplies is an external IP65 rated electrical junction box.\nThe housing, detector assembly and electronics box\nMirror Heater The aluminium mirror is the one part of the detector that is exposed to the sky and all that comes from it, be it precipitation, dew and so on. Any form of moisture on the mirror surface will negatively affect the reflectivity. There is not much that can be done about protecting the mirror from the rain, and generally rain means there are clouds. When conditions are clear, meaning no or little cloud, the risk of dew forming on exposed parts is increased. Heating the aluminium gently fixes this problem. Basically, the purpose of the mirror heater is to make sure that the surface of the mirror is maintained a few degrees above dewpoint temperature to ensure that dew does not form on the reflecting surface.\nThe heater element is to be built from some nichrome wire, commonly found as a heating element in toasters, hairdryers and so on. It is commonly sold in many thicknesses (SWG) and I only need to make sure I calculate the right gauge for the size of heater required.\nCalculations Below is a brief rundown of the calculations required to find what sort of wire was required to build the heater.\nPower output I would like the heater wire to output approximately 0.3 W per cm; being that the wire needs to be approximately 37.5 cm long to cover the mirror back, the total power output PoutP_{out} will be the following:\nPout=0.3×37.5=13.5 WP_{out} = 0.3 \\times 37.5 = \\text{13.5 }WSupply Current Calculating the supply current is quite straightforward. The supply voltage VsuppV_{supp} will be 9 V, therefore:\nIsupp=PoutVsupp∴I_{supp} = \\frac{P_{out}}{V_{supp}} \\thereforeIsupp=13.59=1.5 AI_{supp} = \\frac{13.5}{9} = \\text{1.5 }ATotal Heater Wire Resistance Now we need to know what the resistance of the entire length of the heater wire will be in order to get the correct gauge nichrome resistance wire:\nRwire=VsuppIsupp∴R_{wire} = \\frac{V_{supp}}{I_{supp}} \\thereforeRwire=91.5=6 ΩR_{wire} = \\frac{9}{1.5} = \\text{6 }\\OmegaSo the wire ideally needs to have an overall resistance of 6 ohms once built. This includes an extra few centimetres at the end on which to solder on a supply and ground wire.\nLet\u0026rsquo;s say that the total length ℓtotal\\ell_{total} of the wire will be 45.5 cm ; at that length:\nℓtotal=10045.5×6≈13.2 Ω⋅m\\ell_{total} = \\frac{100}{45.5} \\times 6 \\approx \\text{13.2 }\\Omega \\cdot mConclusion Ideally the nichrome resistance wire will need to be rated at 13.2 Ω⋅m\\text{13.2 }\\Omega \\cdot m. The nearest I could find was 30 SWG (0.315 mm) at 14.27 Ω⋅m\\text{14.27 }\\Omega \\cdot m, which is what I bought in the end.\nConstruction Having scoured eBay for some wire, the first thing I was did a basic test of what I had bought. Placed one test lead of the multimeter at one end of the nichrome resistance wire and ran the other lead along the wire until I got a reading of 6 ohms. That gave a length of about 42 cm. I then dug out some thin heatshrink tubing and cut a length about 8 cm shorter than the wire. Putting the wire through the heatshrink left 4 cm exposed at either end; these ends for attaching to the supply wires.\nProbably the most difficult task in putting this heater together was soldering some stranded wire onto each end of the nichrome resistance wire; the metals don\u0026rsquo;t take to each other all that easily. The easiest way turned out to be knotting the wire and supply wire together and then dropping solder onto that.\nRear of mirror exposed with heater element and DS18B20 temperature sensor\nOnce the supply wires were attached I covered the joints with a little more heatshrink to finish the job off. The picture above shows the heater almost in place with the temperature control sensor alongside.\nOperation There is not much to write about this; the solution is not a novel one!\nThermostatic Control The basic cycle of thermostatic control for the mirror is set out below:\nDefine the hysteresis variable thystt_{hyst}. Derive current dew point temperature from a Humidity Sensor tdewt_{dew}. Measure the current temperature of the mirror surface tmirrort_{mirror}. Check the state of condition tdew\u0026gt;tmirror−thystt_{dew} \\gt t_{mirror} - t_{hyst}. If condition 4 is true then apply heat until condition 4 is false. Once condition 4 is true, switch the heat off. thystt_{hyst} is there to ensure that there is a buffer between the dew point tdewt_{dew} and the temperature of the mirror tmirrort_{mirror} in order to prevent rapid switching of the heater circuit transistor.\nElectronics Control of the heater as above is achieved by using the Arduino to run the thermostatic control cycle, having the DS18B20 to measure tmirrort_{mirror} and an externally situated HTU21D to measure tdewt_{dew}. Using these inputs the Arduino controls the base voltage of a TIP120 Darlington NPN transistor. This transistor in turn controls the voltage to the heater wire.\nBelow is a mockup of how to wire up an Arduino to control the base voltage of a transistor thus controlling the higher power supply to the (in this case) coil. Of course, the coil symbol in the diagram represents the heater.\nArduino with TIP120 High Power Control\nHumidity Sensor Sensor Update I\u0026rsquo;ve had some long time concerns about the humidity sensor I had lined up to undertake the task of determining dew point.\nUp until recently I had always considered using a DHT22 (also known as an Aosong AM2302) to do the job, but had always been a little unsure of how well adapted this sensor would be to living outside all the time. My uncertainty about their robustness comes mostly from their build quality which, to be polite about it, is not really the best.\nDHT22 Humidity Sensor\nI suppose that my main concern is how a DHT22 would fare in a cold and wet environment and potentially have its performance degrade leading to short to medium term reliability issues; longer term, I cannot imagine it lasting at all! I could find no satisfactory manufacturer\u0026rsquo;s documentary evidence as to how waterproof the packaging was. There is a good deal of in depth research here comparing the different humidity sensors (or Hygrometers as he calls them) available on the market, for those that are interested in further reading. The DHT22 has an accuracy of around 5% between 0% and 90% humidity. Above 90% humidity the device performance is a little uncertain; no real world data exists, just what is presented in the datasheet. Additionally, Robert from www.kandrsmith.org indicates a 50% device failure rate over a period of 2 years in his research.\nSo, research into DHT22 alternatives lead me to the HTU21D module.\nHTU21D Humidity Sensor on a Breakout Board\nThe stats on the HTU21D look much better; as soon as you take a peek at the datasheet, it becomes obvious that this is a much higher quality product than the DHT series sensor. For a start, there is a proper datasheet rather than the token effort offered by the developers of the DHT22. The build looks much more robust too; both the exterior housing and connector pins are very flimsy on the DHT22, whereas the HTU21D is of a clearly better quality and more robust build.\nThe HTU21D has a version that comes with a PTFE protector on the sensor window which provides the sensor with a degree of protection from the moisture ingress of an outside environment. Rather than the serial interface of the DHT22, the HTU21D provides I2C communications which I prefer.\nCalculating the Dew Point The dew point temperature measurement is derived from air temperature and humidity both of which are supplied by a humidity sensor. The code I use to derive the dewpoint is written by Rob Tillaart, I cannot even remember when or where I found it, but I have found it useful many times. Below is the code that does the business:\ndouble dewPoint(double celsius, double humidity) { // (1) Saturation Vapour Pressure = ESGG(T) double RATIO = 373.15 / (273.15 + celsius); double RHS = -7.90298 * (RATIO - 1); RHS += 5.02808 * log10(RATIO); RHS += -1.3816e-7 * (pow(10, (11.344 * (1 - 1 / RATIO ))) - 1) ; RHS += 8.1328e-3 * (pow(10, (-3.49149 * (RATIO - 1))) - 1) ; RHS += log10(1013.246); // factor -3 is to adjust units - Vapour Pressure SVP * humidity double VP = pow(10, RHS - 3) * humidity; // (2) DEWPOINT = F(Vapour Pressure) double T = log(VP / 0.61078); // temp var return (241.88 * T) / (17.558 - T); } Moving from the Uno to the Yún Over the last year or two I have been experimenting with the Arduino Yún as a replacement for the, ah, mostly reliable Arduino Uno1. In the early days I had some interesting times in getting Arduino Unos (and their counterparts) onto my local network, having to resort to some workarounds to ensure reliability.\nArduino Uno R3 and Arduino Yún in Comparison\nThere are a few reasons why I want to move this project over to using a Yún, all of them to do with better reliability and an increased range of functionality. A non-exhaustive list would go something like:\nMuch improved networking\nThe Yún has built in Ethernet and Wi-Fi support which obviates the need for custom networking hardware and restart handling electronics.\nBetter stability\nOver a few months I had the Yún running in a remote situation with real world events taking place, such as swapping out a gateway on the network, power outages (frequent because of where I live!) and so on. I never had any problems getting data from the Yún when I needed to.\nMore that just an Arduino\nOf course, the thing that makes the Yún really special is the onboard Linux bridge capabilities that it has. The Atheros AR9331 is basically a Wi-Fi system-on-chip, frequently seen in wireless access points and routers.\nThe downsides of choosing the Yún are higher initial cost, a more complex architecture and slight incompatibility with some general Arduino libraries.\nHowever, the gain of being able to put an Arduino on the network without breaking any sweat really has me hooked on the Yún. Being that networking comms is taken care of by the addition of the Bridge.h, YunServer.h and YunClient.h libraries, I have an endpoint capable of providing simple and readable RESTful URI resources; something I don\u0026rsquo;t have to even think about programming myself. Because of that, I have the great advantage of more program memory to play with.\nConclusion So, as can be seen from the collection of things mentioned above, I have moved a long way in getting this project towards completion. It has been a stop / start affair; many other things have intervened and I have had to do some time consuming research. Given the limited free time I have it I do sometimes question whether I will get this finished.\nI am hopeful that I will.\nOnce this build is out of the way and the instrument is outside gathering data the fun really starts.\nThe Arduino Uno has indeed been really trustworthy - a fabulous piece of kit that has given me years of good service.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://chrisramsay.co.uk/posts/2017/09/cloud-detection-part-five/","summary":"\u003ch2 id=\"progress\"\u003eProgress\u003c/h2\u003e\n\u003cp\u003eThis post follows on at quite a distance from \u003ca href=\"/posts/2014/09/cloud-detection-part-four/\"\u003emy last post on the subject\u003c/a\u003e, and until recently I had not realised quite how long this project has been languishing on the bench. Like many of the electronics projects I undertake they need to be able to withstand being outside and exposed to weather conditions, dealing with packaging them to handle harsher environments is always the most difficult part.\u003c/p\u003e\n\u003cp\u003eFor the same reason as above, I have also changed one of the key measurement components, the humidity sensor.\u003c/p\u003e\n\u003cp\u003eI am also dropping the Arduino Uno in favour of the \u003ca href=\"https://store.arduino.cc/arduino-yun\"\u003eArduino Yún\u003c/a\u003e. I wrote a \u003ca href=\"/posts/2016/04/arduino-yun-reliability-problems/\"\u003ebrief article on the Yún\u003c/a\u003e some time ago.\u003c/p\u003e","title":"Cloud Detection - Part Five"},{"content":"I spotted some unusual clouds over the house the other day!\nPanorama shot of mammatus over the house\nI have always enjoyed identifying clouds and if I can take pictures of any unusual formations. This type, Mammamtus, is pretty rare in these parts; I can count on one hand the number of times I have seen Mammatus clouds.\nAccording to the World Meteorological Organization\u0026rsquo;s (WMO) International Cloud Atlas, Mammatus is not a type of cloud, but a supplementary feature\nMammatus II\nThe Wikipedia article states that \u0026ldquo;they often extend from the base of a cumulonimbus, but may also be found under altocumulus, altostratus, stratocumulus, and cirrus clouds\u0026rdquo;.\nMammatus III\nThese clouds formations were seen around eight hours before a severe drop in pressure brought the first named storm of the season, Aileen.\nMammatus IV\nLooking at my weather data, the cloud base was around 625 metres above sea level and air pressure was falling steadily. There was almost no wind at ground level and the freezing altitude was around 2,300 metres, that is the estimated altitude at which the air temperature is at 0°.\nMammatus V\nThe reasons for this type of cloud formation occurring are ill understood. There seem to be a few theories but nothing yet conclusive.\nMammatus VI\nCertainly not the most uniform of Mammatus formations ever; there are some lovely examples out there that are far better than these. Nevertheless, the formation was only visible for around 25 minutes, so I was quite pleased to have looked out of the kitchen window just at the right time to have seen and photographed them.\n","permalink":"https://chrisramsay.co.uk/posts/2017/09/mammatus-clouds-spotted/","summary":"\u003cp\u003eI spotted some unusual clouds over the house the other day!\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"Panorama shot of mammatus over the house\" loading=\"lazy\" src=\"/posts/2017/09/mammatus-clouds-spotted/37050734616_94c8dc5e03_c.jpg\"\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003ePanorama shot of mammatus over the house\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eI have always enjoyed identifying clouds and if I can take pictures of any unusual formations. This type, \u003ca href=\"https://en.wikipedia.org/wiki/Mammatus_cloud\"\u003eMammamtus\u003c/a\u003e, is pretty rare in these parts; I can count on one hand the number of times I have seen Mammatus clouds.\u003c/p\u003e","title":"Mammatus Clouds Spotted"},{"content":"Multiple MLX90614 Sensors Take Two Over two years ago I wrote this article about giving an MLX90614 a new slave address, different from the default they are all set with from the factory (0x5A), in order to enable having more than one individually addressable device on the bus.\nAs before I am still using two of these MLX90614 sensors in my cloud detection project and have changed slave addresses on these devices many times before with very few problems.\nWhy am I writing a new post about this? Well I have had quite a few questions on my original article and I decided to hopefully make a few things a little clearer with some updated code examples thrown into the mix.\nWhat are we going to do in this article? Focus on the following:\n[Take a quick peek at the MLX90614 again](#Take a quick peek at the MLX90614 again) [Look at some updated code for changing slave addresses](#Look at some updated code for changing slave addresses) [See the range of supported addresses](#See the range of supported addresses) [Look at actually running the code](#Look at actually running the code) The MLX90614 Let\u0026rsquo;s take a quick look at the sensor itself; once again, note the reference image of a pinout diagram of the device from the Melexis MLX90614 Datasheet:\nMLX90614 Pinout - courtesy of Melexis NV\nRequired Code for Changing Slave Address Let\u0026rsquo;s take a look at the improved code.\nCode layout I\u0026rsquo;ve gone for a slightly different layout with the Arduino code in the hope of making the main functionality simpler. The ReadTemp function has been moved into a separate file (read_temp.ino) alongside the main remap_mlx.ino file.\n. ├──README.md └──remap_mlx ├──README.md ├──remap_mlx.ino └──read_temp.ino remap_mlx.ino What have I done to \u0026ldquo;improve\u0026rdquo; the code? Not much really: I\u0026rsquo;ve mostly concentrated on making the serial console output a little clearer so that users should have more of an idea what is going on.\nLet\u0026rsquo;s break the first 30 lines down a little - those are the most interesting part; they run the show procedure-wise.\nI start with including the i2cmaster library and setting a slave address, that is, the address we want the MLX90614 to finish up with.\nThe setup() function runs only once, as always happens in Arduino land; Serial communications are set up along with a message. i2c is initiated along with a bit of manipulation of the port registers and then a five second sleep. Next, I read the current slave address of the MLX90614 using the universal address (0) - this returns three pieces of data, the first of which is the slave address:\n\u0026gt; [ReadAddr] Reading address Using MLX univeral address, Data: 2B, 0, FF The above example is with a slave address of 2B.\nOnce the current address is read out, I start the process of setting a new slave address (NewMLXAddr).\nContents of `remap_mlx.ino`:\n// remap_mlx90614.ino #include \u0026#34;i2cmaster.h\u0026#34; // New slave address (default is 0x5A) byte NewMLXAddr = 0x2B; void setup(){ Serial.begin(9600); Serial.println(\u0026#34;Setup...\u0026#34;); // Initialise some stuff i2c_init(); PORTC = (1 \u0026lt;\u0026lt; PORTC4) | (1 \u0026lt;\u0026lt; PORTC5); delay(5000); // Read on universal address (0) ReadAddr(0); // Change to new address NewMLXAddr ChangeAddr(NewMLXAddr, 0x00); // Read on universal address (0) ReadAddr(0); Serial.println(\u0026#34;**---READY---**\u0026#34;); // Signal user to cycle power Serial.println(\u0026#34;\u0026gt; [setup] Cycle power NOW to store new address - you have 10 seconds\u0026#34;); delay(10000); // Read on universal address (0) ReadAddr(0); Serial.println(\u0026#34; Warning, next ReadTemp() may fail if address has not been set.\u0026#34;); ReadTemp(NewMLXAddr); Serial.println(\u0026#34;**---DONE---**\u0026#34;); } /** * Read temperature from MLX at new address once setup() is done. * */ void loop(){ delay(5000); ReadTemp(NewMLXAddr); } /** * Changes the address of the MLX to NewAddr1. * */ word ChangeAddr(byte NewAddr1, byte NewAddr2) { Serial.print(\u0026#34;\u0026gt; [ChangeAddr] Will change address to: \u0026#34;); Serial.println(NewAddr1, HEX); i2c_start_wait(0 + I2C_WRITE); i2c_write(0x2E); i2c_write(0x00); i2c_write(0x00); if (i2c_write(0x6F) == 0) { i2c_stop(); Serial.println(\u0026#34;\u0026gt; [ChangeAddr] Data erased.\u0026#34;); } else { i2c_stop(); Serial.println(\u0026#34;\u0026gt; [ChangeAddr] Failed to erase data\u0026#34;); return -1; } Serial.print(\u0026#34;\u0026gt; [ChangeAddr] Writing data: \u0026#34;); Serial.print(NewAddr1, HEX); Serial.print(\u0026#34;, \u0026#34;); Serial.println(NewAddr2, HEX); for (int a = 0; a != 256; a++) { i2c_start_wait(0 + I2C_WRITE); i2c_write(0x2E); i2c_write(NewAddr1); i2c_write(NewAddr2); if (i2c_write(a) == 0) { i2c_stop(); delay(100); Serial.print(\u0026#34;\u0026gt; [ChangeAddr] Found correct CRC: 0x\u0026#34;); Serial.println(a, HEX); return a; } } i2c_stop(); Serial.println(\u0026#34;\u0026gt; [ChangeAddr] Correct CRC not found\u0026#34;); return -1; } /** * Reads the MLX address. * */ byte ReadAddr(byte MLXAddress) { byte NewMLXAddress; Serial.println(\u0026#34;\u0026gt; [ReadAddr] Reading address\u0026#34;); if (MLXAddress == 0) { Serial.print(\u0026#34; Using MLX univeral address\u0026#34;); } else { Serial.print(\u0026#34; Using MLX address: \u0026#34;); Serial.print(MLXAddress, HEX); } Serial.print(\u0026#34;, Data: \u0026#34;); i2c_start_wait(MLXAddress + I2C_WRITE); i2c_write(0x2E); i2c_rep_start(MLXAddress + I2C_READ); NewMLXAddress = i2c_readAck(); Serial.print(NewMLXAddress, HEX); Serial.print(\u0026#34;, \u0026#34;); Serial.print(i2c_readAck(), HEX); Serial.print(\u0026#34;, \u0026#34;); Serial.println(i2c_readNak(), HEX); i2c_stop(); return NewMLXAddress; } I\u0026rsquo;ve moved the often called ReadTemp() function into a separate file in order to try and simplify the content of remap_mlx - see below:\nread_temp.ino Contents of `read_temp.ino`:\n// read_temp.ino /** * Utility function to read temperature from MLX at address * */ float ReadTemp(byte address) { int data_low = 0; int data_high = 0; int pec = 0; byte MLXAddress = address\u0026lt;\u0026lt;1; Serial.print(\u0026#34;\u0026gt; [ReadTemp] Read temperature \u0026#34;); if (MLXAddress == 0) { Serial.print(\u0026#34;using MLX univeral address\u0026#34;); } else { Serial.print(\u0026#34;using MLX address \u0026#34;); Serial.print(address, HEX); }; Serial.print(\u0026#34;: \u0026#34;); i2c_start_wait(MLXAddress + I2C_WRITE); i2c_write(0x07); i2c_rep_start(MLXAddress + I2C_READ); data_low = i2c_readAck(); data_high = i2c_readAck(); pec = i2c_readNak(); i2c_stop(); float temp = 0x0000; temp = (float)(((data_high \u0026amp; 0x007F) \u0026lt;\u0026lt; 8) + data_low); temp = (temp * 0.02) - 273.16; Serial.print(temp); Serial.println(\u0026#34; C\u0026#34;); return temp; } As long as remap_mlx.ino and read_temp.ino are side by side in a project directory as described above, your Arduino development environment won\u0026rsquo;t have any issues compiling the code.\nAddress Range Supported by the MLX90614 Once again, the Melexis MLX90614 Datasheet states that \u0026ldquo;the SA feature allows connecting up to 127 devices (SA=0x00\u0026hellip;0x07F) with only 2 wires\u0026rdquo;. I have experimented with many different addresses between 0x00 and 0x07F. I cannot really emphasise that enough - I\u0026rsquo;ve yet to find an address in that range that would not be accepted as a slave address.\nPurely for the avoidance of doubt, I\u0026rsquo;ve run a little Python one liner below to show the range of acceptable slave addresses. These will all work!\n\u0026gt;\u0026gt;\u0026gt; [format(x, \u0026#39;#04x\u0026#39;) for x in range(1,128)] [ \u0026#39;0x00\u0026#39;, \u0026#39;0x01\u0026#39;, \u0026#39;0x02\u0026#39;, \u0026#39;0x03\u0026#39;, \u0026#39;0x04\u0026#39;, \u0026#39;0x05\u0026#39;, \u0026#39;0x06\u0026#39;, \u0026#39;0x07\u0026#39;, \u0026#39;0x08\u0026#39;, \u0026#39;0x09\u0026#39;, \u0026#39;0x0a\u0026#39;, \u0026#39;0x0b\u0026#39;, \u0026#39;0x0c\u0026#39;, \u0026#39;0x0d\u0026#39;, \u0026#39;0x0e\u0026#39;, \u0026#39;0x0f\u0026#39;, \u0026#39;0x10\u0026#39;, \u0026#39;0x11\u0026#39;, \u0026#39;0x12\u0026#39;, \u0026#39;0x13\u0026#39;, \u0026#39;0x14\u0026#39;, \u0026#39;0x15\u0026#39;, \u0026#39;0x16\u0026#39;, \u0026#39;0x17\u0026#39;, \u0026#39;0x18\u0026#39;, \u0026#39;0x19\u0026#39;, \u0026#39;0x1a\u0026#39;, \u0026#39;0x1b\u0026#39;, \u0026#39;0x1c\u0026#39;, \u0026#39;0x1d\u0026#39;, \u0026#39;0x1e\u0026#39;, \u0026#39;0x1f\u0026#39;, \u0026#39;0x20\u0026#39;, \u0026#39;0x21\u0026#39;, \u0026#39;0x22\u0026#39;, \u0026#39;0x23\u0026#39;, \u0026#39;0x24\u0026#39;, \u0026#39;0x25\u0026#39;, \u0026#39;0x26\u0026#39;, \u0026#39;0x27\u0026#39;, \u0026#39;0x28\u0026#39;, \u0026#39;0x29\u0026#39;, \u0026#39;0x2a\u0026#39;, \u0026#39;0x2b\u0026#39;, \u0026#39;0x2c\u0026#39;, \u0026#39;0x2d\u0026#39;, \u0026#39;0x2e\u0026#39;, \u0026#39;0x2f\u0026#39;, \u0026#39;0x30\u0026#39;, \u0026#39;0x31\u0026#39;, \u0026#39;0x32\u0026#39;, \u0026#39;0x33\u0026#39;, \u0026#39;0x34\u0026#39;, \u0026#39;0x35\u0026#39;, \u0026#39;0x36\u0026#39;, \u0026#39;0x37\u0026#39;, \u0026#39;0x38\u0026#39;, \u0026#39;0x39\u0026#39;, \u0026#39;0x3a\u0026#39;, \u0026#39;0x3b\u0026#39;, \u0026#39;0x3c\u0026#39;, \u0026#39;0x3d\u0026#39;, \u0026#39;0x3e\u0026#39;, \u0026#39;0x3f\u0026#39;, \u0026#39;0x40\u0026#39;, \u0026#39;0x41\u0026#39;, \u0026#39;0x42\u0026#39;, \u0026#39;0x43\u0026#39;, \u0026#39;0x44\u0026#39;, \u0026#39;0x45\u0026#39;, \u0026#39;0x46\u0026#39;, \u0026#39;0x47\u0026#39;, \u0026#39;0x48\u0026#39;, \u0026#39;0x49\u0026#39;, \u0026#39;0x4a\u0026#39;, \u0026#39;0x4b\u0026#39;, \u0026#39;0x4c\u0026#39;, \u0026#39;0x4d\u0026#39;, \u0026#39;0x4e\u0026#39;, \u0026#39;0x4f\u0026#39;, \u0026#39;0x50\u0026#39;, \u0026#39;0x51\u0026#39;, \u0026#39;0x52\u0026#39;, \u0026#39;0x53\u0026#39;, \u0026#39;0x54\u0026#39;, \u0026#39;0x55\u0026#39;, \u0026#39;0x56\u0026#39;, \u0026#39;0x57\u0026#39;, \u0026#39;0x58\u0026#39;, \u0026#39;0x59\u0026#39;, \u0026#39;0x5a\u0026#39;, \u0026#39;0x5b\u0026#39;, \u0026#39;0x5c\u0026#39;, \u0026#39;0x5d\u0026#39;, \u0026#39;0x5e\u0026#39;, \u0026#39;0x5f\u0026#39;, \u0026#39;0x60\u0026#39;, \u0026#39;0x61\u0026#39;, \u0026#39;0x62\u0026#39;, \u0026#39;0x63\u0026#39;, \u0026#39;0x64\u0026#39;, \u0026#39;0x65\u0026#39;, \u0026#39;0x66\u0026#39;, \u0026#39;0x67\u0026#39;, \u0026#39;0x68\u0026#39;, \u0026#39;0x69\u0026#39;, \u0026#39;0x6a\u0026#39;, \u0026#39;0x6b\u0026#39;, \u0026#39;0x6c\u0026#39;, \u0026#39;0x6d\u0026#39;, \u0026#39;0x6e\u0026#39;, \u0026#39;0x6f\u0026#39;, \u0026#39;0x70\u0026#39;, \u0026#39;0x71\u0026#39;, \u0026#39;0x72\u0026#39;, \u0026#39;0x73\u0026#39;, \u0026#39;0x74\u0026#39;, \u0026#39;0x75\u0026#39;, \u0026#39;0x76\u0026#39;, \u0026#39;0x77\u0026#39;, \u0026#39;0x78\u0026#39;, \u0026#39;0x79\u0026#39;, \u0026#39;0x7a\u0026#39;, \u0026#39;0x7b\u0026#39;, \u0026#39;0x7c\u0026#39;, \u0026#39;0x7d\u0026#39;, \u0026#39;0x7e\u0026#39;, \u0026#39;0x7f\u0026#39; ] Running the Code As in the previous post, the Arduino, MLX and a power cycle switch, below:\nAn Arduino and MLX90614 with NC switch on the supply\nAgain, these instructions remain the same as the previous post, but note the warning!\nWarning\nOne note of caution - when attempting to change addresses do not have more than one MLX90614 on the circuit at the same time.\nUpload this program above to your Arduino and run it, being ready to pounce with that wire or push switch during the 10 second pause.\nHere\u0026rsquo;s what my output looked like when changing the address of a device from 0x2A to 0x2B, using the new progam from this post:\nSetup... \u0026gt; [ReadAddr] Reading address Using MLX univeral address, Data: 2A, 0, FF \u0026gt; [ChangeAddr] Will change address to: 2B \u0026gt; [ChangeAddr] Data erased. \u0026gt; [ChangeAddr] Writing data: 2B, 0 \u0026gt; [ChangeAddr] Found correct CRC: 0xE1 \u0026gt; [ReadAddr] Reading address Using MLX univeral address, Data: 2B, 0, FF **---READY---** \u0026gt; [setup] Cycle power NOW to store new address - you have 10 seconds \u0026gt; [ReadAddr] Reading address Using MLX univeral address, Data: 2B, 0, FF Warning, next ReadTemp() may fail if address has not been set. \u0026gt; [ReadTemp] Read temperature using MLX address: 2B, 18.64 C **---DONE---** \u0026gt; [ReadTemp] Read temperature using MLX address: 2B, 18.62 C Finally Hopefully you\u0026rsquo;ll find some success in remapping MLX addresses - try it out and give me some feedback on how things go.\nThe code is available here at my Github repository.\n","permalink":"https://chrisramsay.co.uk/posts/2017/09/arduino-and-multiple-mlx90614-sensors-take-two/","summary":"\u003ch2 id=\"multiple-mlx90614-sensors-take-two\"\u003eMultiple MLX90614 Sensors Take Two\u003c/h2\u003e\n\u003cp\u003eOver two years ago I wrote \u003ca href=\"/posts/2014/09/arduino-and-multiple-mlx90614-sensors/\"\u003ethis article\u003c/a\u003e about giving an MLX90614 a new slave address, different from the default they are all set with from the factory (\u003cspan class=\"title-ref\"\u003e0x5A\u003c/span\u003e), in order to enable having more than one individually addressable device on the bus.\u003c/p\u003e\n\u003cp\u003eAs before I am still using two of these MLX90614 sensors in my \u003ca href=\"/posts/2014/03/cloud-detection-part-one/\"\u003ecloud detection\u003c/a\u003e project and have changed slave addresses on these devices many times before with very few problems.\u003c/p\u003e\n\u003cp\u003eWhy am I writing a new post about this? Well I have had quite a few questions on my original article and I decided to hopefully make a few things a little clearer with some updated code examples thrown into the mix.\u003c/p\u003e\n\u003cp\u003eWhat are we going to do in this article? Focus on the following:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e[Take a quick peek at the MLX90614 again](#Take a quick peek at the MLX90614 again)\u003c/li\u003e\n\u003cli\u003e[Look at some updated code for changing slave addresses](#Look at some updated code for changing slave addresses)\u003c/li\u003e\n\u003cli\u003e[See the range of supported addresses](#See the range of supported addresses)\u003c/li\u003e\n\u003cli\u003e[Look at actually running the code](#Look at actually running the code)\u003c/li\u003e\n\u003c/ol\u003e","title":"Arduino and Multiple MLX90614 Sensors Take Two"},{"content":"In this article I have a play with PyEphem and our local star, both as an opportunity to draw some pretty graphs and answer a few basic astronomy related questions, such as: what does the equation of time look like and is there a difference in the time it takes for the sun to set during the year.\nSetting up the observer Import the libraries I am going to need and then set up the observer; I require a latitude, longitude and date as a minimum.\n# Import some bits import ephem, math, datetime # Get retina display quality for plots %config InlineBackend.figure_format = \u0026#39;retina\u0026#39; home = ephem.Observer() # Set up home.date = \u0026#39;2017-01-01 09:00:00\u0026#39; home.lat = \u0026#39;53.4975\u0026#39; home.lon = \u0026#39;-0.3154\u0026#39; Note that the standard and unmodified object has an elevation of 0.0m, temperature of 15.0 degrees Celsius and a pressure of 1010.0 millibars. I did wonder whether or not changing these values would make any difference to the various calculations. Maybe later\u0026hellip;\nhome.elev 0.0 home.temp 15.0 home.pressure 1010.0 Let’s set up the sun next and then compute it from the observer’s position.\nFun with the Sun Setting up the sun is easy, just:\nsun = ephem.Sun() sun.compute(home) Rising, Transit \u0026amp; Setting From there we can get information regarding (from the observer’s point of view) when the last sunrise was, when local noon will occur and when the next sunset is:\nrising = home.previous_rising(sun).datetime() print(\u0026#39;Sunrise is at {}:{}:{}\u0026#39;.format(rising.hour, rising.minute, rising.second)) transit = home.next_transit(sun).datetime() print(\u0026#39;Local noon is at {}:{}:{}\u0026#39;.format(transit.hour, transit.minute, transit.second)) setting = home.next_setting(sun).datetime() print(\u0026#39;Sunset is at {}:{}:{}\u0026#39;.format(setting.hour, setting.minute, setting.second)) Sunrise is at 8:16:47 Local noon is at 12:4:56 Sunset is at 15:53:17 Apparent Solar Time As our Earth does not have a perfectly circular orbit around the sun, the sun is not exactly due south (otherwise known as a transit) every day at 12:00. Depending on the time of year it can be as much as 16 minutes early or late, equating to almost 4° west or east from due south. Let’s draw a graph to illustrate what’s known as the equation of time.\nimport matplotlib.pyplot as plt import pandas as pd import matplotlib matplotlib.style.use(\u0026#39;ggplot\u0026#39;) # Prepare home.date = \u0026#39;2017/1/1\u0026#39; sun = ephem.Sun() times = [] def get_diff(tm): \u0026#34;\u0026#34;\u0026#34;Return a difference in seconds between tm and 12:00:00 on tm\u0026#39;s date\u0026#34;\u0026#34;\u0026#34; a = datetime.datetime.combine(tm, datetime.time(12, 0)) return (a-tm).total_seconds()/60 # Prepare the data for i in range(1, 368): home.date += ephem.Date(1) trans = home.next_transit(sun).datetime() times.append(get_diff(trans)) # Set up ts = pd.Series(times, index=pd.date_range(\u0026#39;2017/1/1\u0026#39;, periods=len(times))) What are we doing above? Well we are graphing the difference between the time of the transit of the Sun at the home location and the local time. Let’s have a look at a slice of ts:\nts.loc[\u0026#39;2017-04-14\u0026#39;:\u0026#39;2017-04-26\u0026#39;] 2017-04-14 -1.234778 2017-04-15 -0.997353 2017-04-16 -0.766400 2017-04-17 -0.542189 2017-04-18 -0.324979 2017-04-19 -0.115009 2017-04-20 0.087500 2017-04-21 0.282349 2017-04-22 0.469363 2017-04-23 0.648394 2017-04-24 0.819315 2017-04-25 0.982023 2017-04-26 1.136433 Freq: D, dtype: float64 Go ahead and run the plot:\nax = ts.plot() plt.rcParams[\u0026#34;figure.figsize\u0026#34;] = [9, 6] ax.set_xlabel(u\u0026#39;Date\u0026#39;, fontsize=11) ax.set_ylabel(u\u0026#39;Variation (minutes)\u0026#39;, fontsize=11) # Fire plt.show() So you can see that there are only 4 points in the year where local noon and the sun actually intersect!\nDrawing the Analemma An analemma is the shape the sun will trace out if you were to note its position in the sky at the same time of day once a week over the passage of a year. The shape is a combination of the equation of time and the Earth’s passage around the sun.\nLocal Noon Let’s have a go at drawing the analemma occurring at home at local noon (12:00:00):\n# Prepare home.date = \u0026#39;2017/1/1 12:00:00\u0026#39; sun = ephem.Sun() posx = [] posy = [] # Solstice altitude phi = 90 - math.degrees(home.lat) # Earth axial tilt epsilon = 23.439 def get_sun_az(tm): \u0026#34;\u0026#34;\u0026#34;Get the azimuth based on a date\u0026#34;\u0026#34;\u0026#34; sun.compute(tm) return math.degrees(sun.az) def get_sun_alt(tm): \u0026#34;\u0026#34;\u0026#34;Get the altitude based on a date\u0026#34;\u0026#34;\u0026#34; sun.compute(tm) return math.degrees(sun.alt) # Prepare the data for i in range(1, 368): home.date += ephem.Date(1) trans = home.next_transit(sun).datetime() posx.append(get_sun_az(home)) posy.append(get_sun_alt(home)) # Set up fig, ax = plt.subplots() ax.plot(posx, posy) ax.grid(True) ax.set_xlabel(u\u0026#39;Azimuth (°)\u0026#39;, fontsize=11) ax.set_ylabel(u\u0026#39;Altitude (°)\u0026#39;, fontsize=11) # Add some labels, lines \u0026amp; resize ax.annotate(\u0026#39;Vernal equinox\u0026#39;, xy=(min(posx), phi + 1), xytext=(min(posx), phi + 1)) ax.annotate(\u0026#39;Autumnal equinox\u0026#39;, xy=(max(posx) -2, phi + 1), xytext=(max(posx) -2, phi + 1)) ax.annotate(\u0026#39;Nothern solstice\u0026#39;, xy=(180.1, phi + epsilon + 1), xytext=(180.1, phi + epsilon + 1)) ax.annotate(\u0026#39;Southern solstice\u0026#39;, xy=(180.1, phi - epsilon - 2), xytext=(180.1, phi - epsilon - 2)) plt.plot((min(posx), max(posx)), (phi + epsilon, phi + epsilon), \u0026#39;blue\u0026#39;) plt.plot((min(posx), max(posx)), (phi, phi), \u0026#39;pink\u0026#39;) plt.plot((min(posx), max(posx)), (phi - epsilon, phi - epsilon), \u0026#39;green\u0026#39;) plt.axvline(180, color=\u0026#39;yellow\u0026#39;) plt.rcParams[\u0026#34;figure.figsize\u0026#34;] = [9, 6] plot_margin = 4 x0, x1, y0, y1 = plt.axis() plt.axis((x0, x1, y0 - plot_margin, y1 + plot_margin)) # Fire plt.show() Changing the time of day we view the analemma If we change the time of day the analemma is generated at (say 08:30:00) a very different picture emerges:\n# Prepare home.date = \u0026#39;2017/1/1 08:30:00\u0026#39; home.horizon = \u0026#39;0\u0026#39; sun = ephem.Sun() posy = [] posx = [] def get_sun_az(tm): \u0026#34;\u0026#34;\u0026#34;Get the azimuth based on a date\u0026#34;\u0026#34;\u0026#34; sun.compute(tm) return math.degrees(sun.az) def get_sun_alt(tm): \u0026#34;\u0026#34;\u0026#34;Get the altitude based on a date\u0026#34;\u0026#34;\u0026#34; sun.compute(tm) return math.degrees(sun.alt) # Prepare the data for i in range(1, 368): home.date += ephem.Date(1) posy.append(get_sun_alt(home)) posx.append(get_sun_az(home)) # Set up fig, ax = plt.subplots() ax.plot(posx, posy) # Add some labels \u0026amp; resize ax.set_xlabel(u\u0026#39;Azimuth (°)\u0026#39;, fontsize=11) ax.set_ylabel(u\u0026#39;Altitude (°)\u0026#39;, fontsize=11) plt.rcParams[\u0026#34;figure.figsize\u0026#34;] = [9, 6] # Fire plt.show() As can be seen above, at mid December southern solstice the Sun is only just above the horizon (bottom right on the graph) and almost due south-east (135°) in direction. By contrast at northern solstice in June the Sun is more or less at 40° and not all that far off due east in direction (top left on the graph).\nCalculating Twilights Which twilight, you might ask. Quite rightly so as there are many definitions:\nCivil twilight Nautical twilight Astronomical twilight Civil twilight is defined by the centre of the sun being 6° below the horizon. Under clear conditions bright planets like Venus are easily seen in the sky.\nNautical twilight is defined by the centre of the sun being 12° below the horizon. If the sun is lower it becomes impossible to navigate at sea using the horizon.\nAstronomical twilight is defined by the centre of the sun being 18° below the horizon. At this point it becomes quite easy to see stars and other objects under clear sky conditions.\nTwilight illustration By TWCarlson (Own work) CCBY−SA3.0(\u0026lt;http://creativecommons.org/licenses/by−sa/3.0\u0026gt;)orGFDL(\u0026lt;http://www.gnu.org/copyleft/fdl.html\u0026gt;)CC BY-SA 3.0 ( \u0026lt;http://creativecommons.org/licenses/by-sa/3.0\u0026gt;) or GFDL ( \u0026lt;http://www.gnu.org/copyleft/fdl.html\u0026gt;), via Wikimedia Commons\nAside: Where on the Disc? Let’s have a look a twilight calculations using ephem. By default, ephem uses the edge of the sun’s disc for sunset / sunrise calculations; standard definitions use the centre of the sun’s disc. What is the difference between using the edge of the sun and the centre of the sun’s disc to calculate when ordinary (zero degrees horizon) twilight occurs?\ninitial_set = home.next_setting(sun).datetime() # zero edge next_set = home.next_setting(sun, use_center=True).datetime() # zero centre print(\u0026#39;Centre sunset is at {}:{}:{}\u0026#39;.format(next_set.hour, next_set.minute, next_set.second)) print(\u0026#39;Edge sunset is at {}:{}:{}\u0026#39;.format(initial_set.hour, initial_set.minute, initial_set.second)) delta = initial_set - next_set print(\u0026#39;Time difference is {} mins, {} secs\u0026#39;.format(delta.seconds/60, delta.seconds%60)) Centre sunset is at 15:52:26 Edge sunset is at 15:55:20 Time difference is 2.9 mins, 54 secs The Calculations Okay, so let’s write up a basic method to print the different twilight times and how long after normal twilight they occur. The method below yields the amount of time in seconds it takes the Sun to move from sunset on the horizon to positions below the horizon of -6°, -12° and -18° respectively:\ndef get_setting_twilights(obs, body): \u0026#34;\u0026#34;\u0026#34;Returns a list of twilight datetimes in epoch format\u0026#34;\u0026#34;\u0026#34; results = [] # Twilights, their horizons and whether to use the centre of the Sun or not twilights = [(\u0026#39;0\u0026#39;, False), (\u0026#39;-6\u0026#39;, True), (\u0026#39;-12\u0026#39;, True), (\u0026#39;-18\u0026#39;, True)] for twi in twilights: # Zero the horizon obs.horizon = twi[0] try: # Get the setting time and date now = obs.next_setting(body, use_center=twi[1]).datetime() # Get seconds elapsed since midnight results.append( (now - now.replace(hour=0, minute=0, second=0, microsecond=0)).total_seconds() ) except ephem.AlwaysUpError: # There will be occasions where the sun stays up, make that max seconds results.append(86400.0) return results home.horizon = \u0026#39;0\u0026#39; twilights = get_setting_twilights(home, sun) twilights [57320.284733, 59906.438312, 62546.839518, 65098.990754] Now we can get started on calculating some twilights at the home location. First reset the date to the first day of 2017, set the horizon to zero degrees, set up a sun body and then off we go:\n# Prepare home.date = \u0026#39;2017/01/01 12:00:00\u0026#39; home.horizon = \u0026#39;0\u0026#39; sun = ephem.Sun() twidataset = [] # Calculate just over a year of data for i in range(1, 368): home.date += ephem.Date(1) twidataset.append(get_setting_twilights(home, sun)) What does twidataset contain? Well, it is just a list of lists for now as can be seen from the slice below:\ntwidataset[150:160] [[73229.081533, 76304.927372, 81102.660945, 86400.0], [73298.278985, 76390.550644, 81255.72959, 86400.0], [73365.046584, 76473.20831, 81405.712504, 86400.0], [73429.309671, 76552.779425, 81552.226662, 86400.0], [73490.995712, 76629.145042, 81694.856699, 86400.0], [73550.034189, 76702.188545, 81833.158322, 86400.0], [73606.356557, 76771.796561, 81966.651319, 86400.0], [73659.896242, 76837.858304, 82094.829296, 86400.0], [73710.588832, 76900.266566, 82217.151337, 86400.0], [73758.372248, 76958.918635, 82333.05185, 86400.0]] I’m now going to change the list into a pandas DataFrame object:\ndf = pd.DataFrame(twidataset, columns=[\u0026#39;Sunset\u0026#39;, \u0026#39;Civil\u0026#39;, \u0026#39;Nautical\u0026#39;, \u0026#39;Astronomical\u0026#39;]) Let’s have a peek at a slice of the data frame:\ndf[150:160] Sunset Civil Nautical Astronomical 150 73229.081533 76304.927372 81102.660945 86400.0 151 73298.278985 76390.550644 81255.729590 86400.0 152 73365.046584 76473.208310 81405.712504 86400.0 153 73429.309671 76552.779425 81552.226662 86400.0 154 73490.995712 76629.145042 81694.856699 86400.0 155 73550.034189 76702.188545 81833.158322 86400.0 156 73606.356557 76771.796561 81966.651319 86400.0 157 73659.896242 76837.858304 82094.829296 86400.0 158 73710.588832 76900.266566 82217.151337 86400.0 159 73758.372248 76958.918635 82333.051850 86400.0 The data is ready, so it’s time for some charting. This chart needs a couple of formatters to clean up the tick labels as well as some limit setting to focus in on the interesting bits.\nfrom matplotlib.ticker import FuncFormatter import numpy as np def timeformatter(x, pos): \u0026#34;\u0026#34;\u0026#34;The two args are the value and tick position\u0026#34;\u0026#34;\u0026#34; return \u0026#39;{}:{}:{:02d}\u0026#39;.format(int(x/3600), int(x/24/60), int(x%60)) def dateformatter(x, pos): \u0026#34;\u0026#34;\u0026#34;The two args are the value and tick position\u0026#34;\u0026#34;\u0026#34; dto = datetime.date.fromordinal(datetime.date(2017, 1, 1).toordinal() + int(x) - 1) return \u0026#39;{}-{:02d}\u0026#39;.format(dto.year, dto.month) plt.rcParams[\u0026#34;figure.figsize\u0026#34;] = [9, 6] ax = df.plot.area(stacked=False, color=[\u0026#39;#e60000\u0026#39;, \u0026#39;#80ccff\u0026#39;, \u0026#39;#33adff\u0026#39;, \u0026#39;#008ae6\u0026#39;], alpha=0.2) # Sort out x-axis # Demarcate months dim = [0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31] ax.xaxis.set_ticks(np.cumsum(dim)) ax.xaxis.set_major_formatter(FuncFormatter(dateformatter)) ax.set_xlabel(u\u0026#39;Date\u0026#39;, fontsize=11) # Sort out y-axis ax.yaxis.set_major_formatter(FuncFormatter(timeformatter)) ax.set_ylim([55000, 86400]) ax.set_ylabel(u\u0026#39;Time\u0026#39;, fontsize=11) labels = ax.get_xticklabels() plt.setp(labels, rotation=30, fontsize=9) # Done plt.show() As can be seen in the graph above, there are 78 days (day 131 to day 208 inclusive) where there is no Astronomical twilight because the sun does not reach -18° below the horizon at my home latitude. This is demonstrated below by searching a subset of the data frame accordingly:\ndf.loc[df[\u0026#39;Astronomical\u0026#39;] == 86400.0].head(1) Sunset Civil Nautical Astronomical 131 71562.188604 74280.738322 77955.756763 86400.0 df.loc[df[\u0026#39;Astronomical\u0026#39;] == 86400.0].tail(1) Sunset Civil Nautical Astronomical 208 72158.162401 74867.786881 78520.29891 86400.0 Sunset length throughout the year Sometimes I’ve wondered if there is much of a difference in the amount of time it takes the sun to set (that is the time from the full disc being visible and just touching the horizon, to none of it being visible and all below the horizon ). The sun appears to be approximately half a degree in angular diameter on average when viewed from the earth’s surface. The easy way to have a go at graphing this is to therefore make two calculations based on two sunsets, one at 0 degrees horizon, the other at -0.53 degrees horizon, and then compare.\n# Prepare home.date = \u0026#39;2017/04/01 12:00:00\u0026#39; home.horizon = \u0026#39;0\u0026#39; sun = ephem.Sun() Starting with the 0 degrees:\ns_start = home.next_setting(sun, use_center=False).datetime() s_start datetime.datetime(2017, 4, 1, 18, 37, 13, 370468) Now the -0.53 degrees:\nhome.horizon = \u0026#39;-0.53\u0026#39; s_end = home.next_setting(sun, use_center=False).datetime() s_end datetime.datetime(2017, 4, 1, 18, 41, 53, 696375) The difference is…\ndelta = s_end - s_start delta.total_seconds() 280.325907 Let’s go for a little run and finish off with a pandas Series containing some data:\nhome.date = \u0026#39;2017/01/01 12:00:00\u0026#39; settings = [] sun = ephem.Sun() for i in range(1, 368): home.date += ephem.Date(1) home.horizon = \u0026#39;0\u0026#39; start = home.next_setting(sun, use_center=False).datetime() home.horizon = \u0026#39;-0.53\u0026#39; end = home.next_setting(sun, use_center=False).datetime() settings.append((end - start).total_seconds()) ts = pd.Series(settings, index=pd.date_range(\u0026#39;2017/1/1\u0026#39;, periods=len(settings))) Examining a slice gives us:\nts[0:12] 2017-01-01 353.504381 2017-01-02 352.557403 2017-01-03 351.549113 2017-01-04 350.482556 2017-01-05 349.360751 2017-01-06 348.186956 2017-01-07 346.964359 2017-01-08 345.696319 2017-01-09 344.386193 2017-01-10 343.037395 2017-01-11 341.653190 2017-01-12 340.236993 Freq: D, dtype: float64 Interestingly, the gap between slowest and fastest sunsets is really not that much at all. I may repeat this later by adding 6 degrees for civil twilight.\nts.max(), ts.min() (384.862166, 275.37453799999997) The gap:\nts.max() - ts.min() 109.48762800000003 Let’s make a chart and have a look at the results:\nax = ts.plot.area(alpha=0.2) plt.rcParams[\u0026#34;figure.figsize\u0026#34;] = [9, 6] ax.set_xlabel(u\u0026#39;Date\u0026#39;, fontsize=11) ax.set_ylabel(u\u0026#39;Sunset length (seconds)\u0026#39;, fontsize=11) ax.set_ylim([math.floor(ts.min()) - 15, math.floor(ts.max()) + 15]) # Fire plt.show() So from the graph above, it can be seen that there are two minima in the year where the sun sets the fastest - the middle of March and towards the end of September. The third week in June gives us the longest sunset, with the third week of December the second but smaller maximum of the year. These all correspond with the equinoxes and solstices as you would expect.\nConclusion So there it is, fun times spent with PyEphem and our local star, and I’ve learned a thing or two along the way. I’ve got a few ideas for another article on this subject at some point, so keep your eyes peeled!\n","permalink":"https://chrisramsay.co.uk/posts/2017/03/fun-with-the-sun-and-pyephem/","summary":"\u003cp\u003eIn this article I have a play with \u003ca href=\"http://rhodesmill.org/pyephem/\"\u003ePyEphem\u003c/a\u003e and our local star, both as an opportunity to draw some pretty graphs and answer a few basic astronomy related questions, such as: what does the equation of time look like and is there a difference in the time it takes for the sun to set during the year.\u003c/p\u003e","title":"Fun with the Sun and PyEphem"},{"content":"del.icio.us Link Fetching Below is an outline for building a rough \u0026amp; ready script to get del.icio.us bookmark data - a script very much driven by necessity.\nRationale Why am I doing this? Well, I\u0026rsquo;ve been bookmarking links using del.icio.us since March 2009 and have, very slowly, built up a steady collection of over 900 bookmarks, most of which are probably still quite useful.\nThe ups and downs of social bookmarking service del.icio.us are very well documented here, so I won\u0026rsquo;t go into all the details, but suffice to say that reading the aforementioned article gave me all the impetus I needed to go ahead and homestead my list of bookmarks. Simple, I thought.\nThen came the snag\u0026hellip;\nWe\u0026rsquo;re sorry, but due to heavy load on our database we are no longer able to offer an export function. Our engineers are working on this and we will restore it as soon as possible.\nGreat\u0026hellip; what now? Time to write a script of course. I\u0026rsquo;ve initially created this article using a Jupyter Notebook which may, I hope, explain the recipe feel to the writing.\nThe Plan Well, initially the plan was just to grab the bookmarks by scraping my del.icio.us pages from beginning to end and turn that data straight into some form of bookmark HTML. Thinking about it more, I decided to opt for a JSON data format; that way I could easily turn the data to any format I like at my leisure.\nWhat Data? From a few investigations I found that for a bookmark to be successfully imported into most browsers, one ideally needs a few key pieces of data:\nTitle URL Date added (Human readable and epoch) Private or public Tags list Comment (or title if no description) Icon data (Optional, but provides a nice icon by each link) Annoyingly, there isn\u0026rsquo;t really a standard that browsers use for importing or exporting bookmarks; it all seems a little ad-hoc, so I\u0026rsquo;ve generalised somewhat.\nI would like to finish with some JSON that describes the bookmark like so:\n{ \u0026#34;comment\u0026#34;:\u0026#34;Docker Compose - Docker Documentation\u0026#34;, \u0026#34;add_epoch\u0026#34;:\u0026#34;1431953037\u0026#34;, \u0026#34;title\u0026#34;:\u0026#34;Docker Compose - Docker Documentation\u0026#34;, \u0026#34;url\u0026#34;:\u0026#34;http://docs.docker.com/compose/\u0026#34;, \u0026#34;tags\u0026#34;:[ \u0026#34;docker\u0026#34;, \u0026#34;development\u0026#34;, \u0026#34;python\u0026#34; ], \u0026#34;icon_uri\u0026#34;:\u0026#34;http://docs.docker.com/favicons/favicon-32x32.png\u0026#34;, \u0026#34;private\u0026#34;:0, \u0026#34;add_date\u0026#34;:\u0026#34;2015-05-18 12:43:57\u0026#34;, \u0026#34;icon_data\u0026#34;:\u0026#34;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAFA0lEQVR4Ae3SA3wjXwLA8d/DzERNUiRre3u2bdu2bRsfnG3btm2st7x/ze2myejhdtvz5Wx8B89+/N///c8T/JUud5u7iPpXPqOimRlMrcbKpS7nAMdfSFzpmtflr+GCaEu6a/BZttrfK5O2CydOvxD4Kn8hbeoN/hqy06qY3g3Xs/XGRpHGTs9Pvpe/gnZRiT8lXJru1VlSBrAIvDPO66gpnJXgEd4B3kN35e9+OpDxqu12RLrnK+/jT1m8+m0fcXbjzjt7MOUsQVX7ECYPEmv6k7hNEK8S5RndRDOjxdUr3uxJMmn/EPi9wfTqlW7On5IXyv1Jub4D7wgLZUSxDN6RxR1MnuGEwOrA04WtNS7cufQNHiDSdqvni++YAL7Fb9CyfYauhNxiwsI+4ZxXebZJeocHpMnwogfhHdIaCAtredKaw3h/ZWlzp/L0x8AqgCtVt9rqQMXWB+vR6JF79X3tvVOmOiAqc2PzQEcPDH2fbqYvfLXrL+44/CIbhBSzJKwVSiAELmnT6rQQ1lBWiqhYRgghl3YefmweFh9VaC0tNIZ/fCPgpwAiT86KLM18VNYrOy5868nmrqu5sCDr00PHGqe+81S9tOtidOOkUjaMyudCbKFEEITgPecn5KxBeI8PIhRiLTRRKbI6wISFEiBgXTRy5FQ0+qPhVnPHhWd2XaxqvK9GcYvK0tSMyuIVfe7HL3kpg6TauJEt9myUcKUgz7EB6CxFxm0QAp0laBUgnUOlMSiNzBLC83U8hCYvmFrztj6IruiKlYH4ote4bGvrwZ2r85O+f3p4Wptktjo7+o3+kR+9zOlwSFzpejfGBpEEUWwMfUccucGDPrVykatfcaBYYp0A79cDBMI7cBZUAHh+SZgMr0NAAB4Q+LBArEPmckPzB5/+ycHPvu4uKksmzuUnQBtA2yCqAJcAf8fFnRf9vsrTiuqcJZg6icxTsu2DuEIZrEV2WoQTx5Gry6T7Lo2tDiCcRc+OEo4fIdt2CNPYjg9L65d0cZLi6E9pFHtoDH//2+2+zWWv9L2AHuBNwLi4/I1v1Q/sAMrKZHa5sftxkzsveWOKEUiJVwEoDXgwBpxBAEiF1wF4jzDZWgis5XmpIMkg6aDzlC0X/OjrtfmxJ9og6gAR4IFFYPT8BPhNcbn3/qcPX+vl7f7NQhYC8KzxQrBHpPRrz/fzAs57wLNO/HrnrcXFCRiDQLBh5uTEjqFv3An4Ml1opwJ+U+/C2Cf6FsaOxFHPhZ0rILRCWEtkUppByoBy6GAzuXf8mscbi8/ytQ/vkUBj7vTcttHvPNop/WXoTlz6lnfmdxnnbt+759ArN+/aXZNK0ChF9NeqvH6pyAVOYZ1jjXPrA+cGbwx4D0IS5jHN2dOnN4//8LFX+/DbPww46E48RvJ7Xiu1vvY973ffq1z72k89NHi4US6VUErxvqElXvSTGVLnEd7hnQMHCNYEJqN6drY1MHv6I82p488HfsqfIK58/ZvQzVc/+RF1lVve4eqXuuxlHn5ocPBKzQ3NalQs8tmftXjHqUVmE4P0FmVzorSdldtLk9UzU1+pL46/Y9PEsa8CMX8GcbNLXYQ/5ivLpjJ40YteYuv27Veq9/UdqteqtZNLsfrRXCsJ0vZKkCdDxc7ykcrK7E+3ThybAHL+AuLOB7fy53rriQkJqN0C7iDwOx0OcPDXE6/jX0vy/wn8fwL/Yj8H6qJUvBfQ9+kAAAAASUVORK5CYII=\u0026#34; } Getting the data I\u0026rsquo;ll now go ahead and start getting the pieces of data I need. Firstly I am importing all the libraries I\u0026rsquo;ll need for the entire exercise, as well as a bit of logging:\nimport datetime import urllib import ssl import json import base64 import codecs import logging from urlparse import urlparse from bs4 import BeautifulSoup logging.basicConfig(format=\u0026#39;%(levelname)s:%(message)s\u0026#39;, level=logging.DEBUG) So I am starting this by fetching one links page from del.icio.us, breaking it down and getting each required data item piece by piece.\nTo be clear, before I started the next bit, I used a browser dev tool to identify the page structure. Firstly then, fetch the URL, parse the returned data with the fantastic BeautifulSoup library to get all the bookmark links outer blocks.\nurl = \u0026#39;https://del.icio.us/chrisramsay?\u0026amp;page=1\u0026#39; try: r = urllib.urlopen(url) except IOError: logging.warning(\u0026#39;Could not open %s\u0026#39;, url) else: soup = BeautifulSoup(r, \u0026#39;html.parser\u0026#39;) bookmarks = soup.find_all(\u0026#34;div\u0026#34;, class_=\u0026#34;articleThumbBlockOuter\u0026#34;) print \u0026#39;We have {} bookmarks\u0026#39;.format(len(bookmarks)) We have 10 bookmarks Result! I have ten, seeing as there are that many per page. From here on in I am working on a single bookmark - this whole next section would be in a loop over the list of bookmarks. I am going to use item zero of the list from this point.\nFirstly I get the bookmark title:\nentry = {} bookmark_zero = bookmarks[0] title = bookmark_zero.find_all(\u0026#39;div\u0026#39;, class_=\u0026#39;articleTitlePan\u0026#39;)[0] entry[\u0026#39;title\u0026#39;] = title.a.attrs[\u0026#39;title\u0026#39;] print entry { \u0026#39;title\u0026#39;: u\u0026#39;DIY Science: Measuring Light with a Photodiode II | Outside Science\u0026#39; } Next, the bookmark URL:\nhref = bookmark_zero.find_all(\u0026#39;div\u0026#39;, class_=\u0026#39;articleInfoPan\u0026#39;)[0].find_all(\u0026#39;p\u0026#39;)[0] entry[\u0026#39;url\u0026#39;] = href.a.attrs[\u0026#39;href\u0026#39;] print entry { \u0026#39;url\u0026#39;: u\u0026#39;https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/\u0026#39;, \u0026#39;title\u0026#39;: u\u0026#39;DIY Science: Measuring Light with a Photodiode II | Outside Science\u0026#39; } Get the bookmark favicon (ICON_URI):\nparsed = urlparse(entry[\u0026#39;url\u0026#39;]) entry[\u0026#39;icon_uri\u0026#39;] = u\u0026#39;{}://{}/favicon.ico\u0026#39;.format(parsed.scheme, parsed.netloc) Get the bookmark save date:\nentry[\u0026#39;add_date\u0026#39;] = str(datetime.datetime.fromtimestamp(int(bookmark_zero.attrs[\u0026#39;date\u0026#39;]))) entry[\u0026#39;add_epoch\u0026#39;] = bookmark_zero.attrs[\u0026#39;date\u0026#39;] print entry { \u0026#39;add_epoch\u0026#39;: u\u0026#39;1431293424\u0026#39;, \u0026#39;url\u0026#39;: u\u0026#39;https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/\u0026#39;, \u0026#39;title\u0026#39;: u\u0026#39;DIY Science: Measuring Light with a Photodiode II | Outside Science\u0026#39;, \u0026#39;icon_uri\u0026#39;: u\u0026#39;https://outsidescience.wordpress.com/favicon.ico\u0026#39;, \u0026#39;add_date\u0026#39;: \u0026#39;2015-05-10 21:30:24\u0026#39; } Get the bookmark tags (if there are any):\ntry: tags = bookmark_zero.find_all(\u0026#39;ul\u0026#39;, class_=\u0026#39;tagName\u0026#39;)[0].find_all(\u0026#39;li\u0026#39;) except IndexError: tags = [] entry[\u0026#39;tags\u0026#39;] = [f.a.text for f in tags] print entry { \u0026#39;add_epoch\u0026#39;: u\u0026#39;1431293424\u0026#39;, \u0026#39;tags\u0026#39;: [u\u0026#39;arduino\u0026#39;, u\u0026#39;circuits\u0026#39;, u\u0026#39;photodiodes\u0026#39;, u\u0026#39;photodiode\u0026#39;], \u0026#39;url\u0026#39;: u\u0026#39;https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/\u0026#39;, \u0026#39;title\u0026#39;: u\u0026#39;DIY Science: Measuring Light with a Photodiode II | Outside Science\u0026#39;, \u0026#39;icon_uri\u0026#39;: u\u0026#39;https://outsidescience.wordpress.com/favicon.ico\u0026#39;, \u0026#39;add_date\u0026#39;: \u0026#39;2015-05-10 21:30:24\u0026#39; } Get any comments:\ncomment = None for a in [l for l in bookmark_zero.find(\u0026#39;div\u0026#39;, class_=\u0026#39;thumbTBriefTxt\u0026#39;).children]: try: comment = a.p.contents[0] except AttributeError: continue if comment is not None: entry[\u0026#39;comment\u0026#39;] = comment else: entry[\u0026#39;comment\u0026#39;] = entry[\u0026#39;title\u0026#39;] print entry { \u0026#39;comment\u0026#39;: u\u0026#39;DIY Science: Measuring Light with a Photodiode II | Outside Science\u0026#39;, \u0026#39;add_epoch\u0026#39;: u\u0026#39;1431293424\u0026#39;, \u0026#39;tags\u0026#39;: [u\u0026#39;arduino\u0026#39;, u\u0026#39;circuits\u0026#39;, u\u0026#39;photodiodes\u0026#39;, u\u0026#39;photodiode\u0026#39;], \u0026#39;url\u0026#39;: u\u0026#39;https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/\u0026#39;, \u0026#39;title\u0026#39;: u\u0026#39;DIY Science: Measuring Light with a Photodiode II | Outside Science\u0026#39;, \u0026#39;icon_uri\u0026#39;: u\u0026#39;https://outsidescience.wordpress.com/favicon.ico\u0026#39;, \u0026#39;add_date\u0026#39;: \u0026#39;2015-05-10 21:30:24\u0026#39; } Taking a look at the whole bookmark as a JSON string dump:\njson.dumps(entry) \u0026#39;{ \u0026#34;comment\u0026#34;:\u0026#34;DIY Science: Measuring Light with a Photodiode II | Outside Science\u0026#34;, \u0026#34;add_epoch\u0026#34;:\u0026#34;1431293424\u0026#34;, \u0026#34;tags\u0026#34;:[ \u0026#34;arduino\u0026#34;, \u0026#34;circuits\u0026#34;, \u0026#34;photodiodes\u0026#34;, \u0026#34;photodiode\u0026#34; ], \u0026#34;url\u0026#34;:\u0026#34;https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/\u0026#34;, \u0026#34;title\u0026#34;:\u0026#34;DIY Science: Measuring Light with a Photodiode II | Outside Science\u0026#34;, \u0026#34;icon_uri\u0026#34;:\u0026#34;https://outsidescience.wordpress.com/favicon.ico\u0026#34;, \u0026#34;add_date\u0026#34;:\u0026#34;2015-05-10 21:30:24\u0026#34; }\u0026#39; Well, that looks about right to me. Obviously a collection of bookmarks in this JSON form (dictionary) will have to be expressed as a list of dictionaries.\nTime to think about getting the icon data.\nAside - Icons Take a look at existing bookmark HTML files from any browser and it will become apparent that they store icons locally in the form of a base 64 encoded string. This seems obvious once thought about; better to locally cache a copy than have to fetch a copy each time the browser is started up. As part of the bookmark fetching process I therefore decided to see if it would be possible to visit the bookmarked page itself and extract an icon.\nThe code is below; an explanation follows:\ndef get_icon(fetch): \u0026#34;\u0026#34;\u0026#34;Fetch an icon, base64 encode and return it.\u0026#34;\u0026#34;\u0026#34; icon_path = None icon_data = {\u0026#39;icon_uri\u0026#39;: None, \u0026#39;icon_data\u0026#39;: None} # Fetch the root page, parse the HTML and get the LINK elements within HEAD. try: r_icon = urllib.urlopen(fetch) except (IOError, ssl.CertificateError): # Total fail, just return empty data return icon_data else: # Carry on \u0026amp; get the icon rel link icon_soup = BeautifulSoup(r_icon, \u0026#39;html.parser\u0026#39;) head = icon_soup.find(\u0026#39;head\u0026#39;) if head is None: return icon_data link = head.find(\u0026#39;link\u0026#39;, rel=\u0026#39;icon\u0026#39;) if link is not None: logging.debug(\u0026#39;Icon URL: %s\u0026#39;, link.attrs[\u0026#39;href\u0026#39;]) # Check for relative or absolute paths if \u0026#39;:\u0026#39; in link.attrs[\u0026#39;href\u0026#39;]: icon_path = link.attrs[\u0026#39;href\u0026#39;] else: icon_path = u\u0026#39;{}{}\u0026#39;.format(fetch, link.attrs[\u0026#39;href\u0026#39;]) # If we have icon_path, get the icon data if icon_path is not None: icon_data[\u0026#39;icon_uri\u0026#39;] = icon_path try: icon_ulib = urllib.urlopen(icon_path) except (IOError, ssl.CertificateError): return icon_data # Get the content type to avoid things like 404 HTML content_type = icon_ulib.headers.getheader(\u0026#39;Content-Type\u0026#39;) if \u0026#39;image\u0026#39; in icon_ulib.headers.getheader(\u0026#39;Content-Type\u0026#39;): icon_data[\u0026#39;icon_data\u0026#39;] = \u0026#39;data:{};base64,{}\u0026#39;.format( content_type, base64.b64encode(icon_ulib.read())) # We are done return icon_data What I show in the code block above is neither pretty nor particularly discriminating; it grabs the first link element it finds containing a property rel with the value of icon from the head of the page, then extracting the value of the href property.\nNext, examining the href property it decides whether the link is relative or absolute and modifies the link appropriately. Whichever the case an icon URL is formed from which the icon itself is fetched. Headers are checked, content type decided and if we get to the end we have a base 64 encoded string complete with content type information.\nBelow are a couple of test run results using absolute and relative icon `href`:\nTesting: Absolute links to icons For example: https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico\nfetch = \u0026#39;https://www.gov.uk\u0026#39; sixfour_img = get_icon(fetch) DEBUG:Icon URL: https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico print sixfour_img { \u0026#39;icon_data\u0026#39;: \u0026#39;data:image/x-icon;base64,AAABAAIAEBAQAAEABAAoAQAAJgAAACAgAAABACAAqBAAAE4BAAAoAAAAEAAAACAAAAABAAQAAAAAAIAAAAAAAAAAAAAAABAAAAAAAAAALi0xACAfIwD///8AdHN2APHx8QBmZWgAWFdaAKurrQBKSUwAnZ2fAJCPkQDj4+QA1dXWAMfHyAC5uboAPDs/ABEREREREREREREREREREREREREREREREREC3d3d2yERETzUdEdOxRERu82e19u9ERazt3vK3DuBHeFGCqA0HtETkWEZkRgZMRMji6JKmDIxEG3enNeb1gERH4GkKhjxERERER1xERERERERERERERERERERERERERERERERERERAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgAAAAgAAAAQAAAAAEAIAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAADcAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA3AAAAPMAAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAADzAQEB7QUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wEBAe0CAgLtDAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/AgIC7QICAu0LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/CwoK/wwLC/8CAQH/AAAA/wEAAP8EAwP/BgUF/wgHB/8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/BwYG/wUEBP8EAwP/AQAA/wAAAP8DAgL/DAsL/wsKCv8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/BgYG/zY1Nf9GRUX/Ly4u/yEgIP8SERH/CQgI/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wEAAP8NDAz/FxYW/yEgIP8vLi7/TExM/zY1Nf8FBAT/CwoK/wsKCv8LCgr/AgIC7QICAu0LCgr/CwoK/wkICP8FBAT/yMjI///////5+fn/5ubm/9XV1f/Ly8v/v7+//729vf+qqqr/pqam/6empv+npqb/pqam/6qqqv+9vb3/wcHB/8/Pz//b2tr/5ubm//n5+f//////r66u/wAAAP8KCQn/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/AwIC/ygnJ//09PT////////////////////////////////////////////////////////////////////////////////////////////////////////////k5OT/FRQU/wYFBf8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8AAAD/amlp///////BwMD/cnJy/8bGxv///////////9PT0/+xsbH/x8fH//7+/v////////////j4+P+8vLz/srKy/9/f3////////////7u7u/93d3f/1tbW//////9JSEj/AAAA/wsKCv8LCgr/AgIC7QICAu0LCgr/CAcH/wUEBP/My8v/srKy/wkICP8AAAD/JCMj//v7+/+Lior/BQQE/wAAAP8AAAD/YWBg//z8/P/y8vL/RUVF/wAAAP8AAAD/EA8P/6Kiov/n5+f/Dg0N/wAAAP8ZGBj/zMvL/6enp/8AAAD/CwoK/wsKCv8CAgLtAgIC7QsKCv8AAAD/PTw8//f39/+Xlpb/aGho/1BPT/9PTk7/8fHx/xwbG/8AAAD/NDMz/2dmZv8AAAD/zc3N/6moqP8AAAD/enl5/ycmJv8AAAD/Ly4u/+jo6P9DQkL/WFdX/2RjY/+mpaX/8fDw/yUkJP8CAQH/CwoK/wICAu0CAgLtCQgI/wAAAP+srKz//f39//n5+f9YV1f/7u7u////////////x8bG/46Njf86Ojr//////6Ghof/Gxsb/wcDA/7i3t//39/f/Ly4u/5mZmf/GxcX////////////V1NT/ZmVl///////8/Pz/k5KS/wAAAP8KCQn/AgIC7QICAu0AAAD/XVxc//////+lpKT/rays/wkHB/9OTk7/2dnZ//f39//p6en/c3Nz/0dGRv//////v7+///Py8v/j4+P/xsbG//////80MzP/l5aW/+np6f/6+vr/wsLC/zw8PP8XFhb/tLOz/6ampv//////QUBA/wEBAf8CAgLtAQEB7QAAAP9ERET/p6en/21sbP8oJyf/AQEB/yQjI//v7+//5+fn/ygnJ/8AAAD/RkVF/0A/P/80NDT/9/f3/+Pj4/8dHBz/VVRU/zU0NP8AAAD/ODg4//n5+f/e3d3/ExIS/wkICP8wLy//enl5/6ampv8yMTH/AAAA/wICAu0AAADtQkFB/7Ozs/+dnZ3/IyEh/wAAAP8AAAD/c3Jy///////v7+//IB8f/wEAAP8DAgL/FhUV/9vb2////////////8XExP8KCQn/BgUF/wAAAP82NTX///////////9YV1f/AAAA/wAAAP8vLi7/q6ur/7S0tP82NTX/AQAA7QAAAO3BwcH///////////9vbm7/AAAA/wAAAP9gX1///////87Nzf8MCwv/CAcH/wcGBv8eHR3/SkpK/z8+Pv9APz//S0tL/xoZGf8HBwf/BQQE/xkYGP/i4uL//////0VERP8AAAD/AAAA/5KSkv///////////6empv8AAADtAAAA7ZiYmP///////////09OTv8BAQH/BQQE/yEgIP+8vLz/Q0JC/wEBAf8LCgr/DAsL/wgHB/8AAAD/AAAA/wAAAP8AAAD/CQgI/wwLC/8MCgr/AAAA/1taWv+xsLD/Dg0N/wcGBv8AAAD/a2tr////////////e3p6/wAAAO0BAQHtFxYW/2JhYf9SUVH/AQAA/wQDA/8MCwv/CAcH/wkICP8DAgL/CAcH/wQDA/8JCAj/AwMD/1VUVP/Kysr/xcXF/0dGRv8DAwP/CAcH/wQDA/8IBwf/BQQE/wkICP8JCAj/DAsL/wEAAP8GBAT/YF9f/11dXf8NDAz/AQEB7QICAu0AAAD/MzEx/8LCwv+1tbX/IB8f/wUFBf8IBwf/CAcH/wYFBf8NDAz/IB8f/wEAAP8LCgr/39/f////////////x8bG/wMDA/8FBAT/IyIi/wkICP8HBgb/CAcH/wkICP8EAwP/NjU1/8XFxf+3t7f/IyIi/wIBAf8CAgLtAAAA7QMCAv/BwcH///////////94d3f/AAAA/wYFBf8AAAD/FxYW/7y8vP/s7Oz/goKC/wUFBf/Z2Nj////////////ExMT/CAcH/52dnf/u7u7/paSk/woJCf8AAAD/AwIC/wAAAP+joqL///////////+enp7/AAAA/wEBAe0BAQHtAQEB/42Njf///////////1ZVVf97enr/wMDA/1dWVv9EQ0P////////////w8PD/CAcH/42MjP///////////3d3d/8fHh7/////////////////Kikp/3Z1df+8vLz/X19f/29ubv///////////3Jxcf8AAAD/AgEB7QICAu0JBwf/Dw4O/11cXP8/Pj7/NzY2////////////7u/v/y4tLf/Pz8///Pz8/6moqP8AAAD/NjU1///////8+/v/JCMj/wUEBP+/vr7/+/v7/7Gxsf80MzP////////////8+/v/HBsb/0tLS/9QTk7/CgkJ/wkICP8CAgLtAgIC7QwLC/8IBwf/AAAA/wAAAP8lJCT/6+vr///////V1dX/CwoK/xkYGP80MzP/GhkZ/5CQkP9VVFT/4uLi/9HQ0P9mZmb/kJCQ/xAPD/81NDT/DAwM/xkYGP/o6Oj//////9zb2/8TEhL/AAAA/wAAAP8JCAj/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/CwoK/wMCAv85OTn/bWxs/yopKf8GBQX/BQQE/wAAAP8MCwv/6ejo///////5+Pj/9/f3///////W1tb/AAAA/wAAAP8GBQX/BQQE/zg4OP9ubW3/Li0t/wYFBf8LCgr/CwoK/wsKCv8LCgr/AgIC7QICAu0LCgr/CwoK/wsKCv8LCgr/CwoK/wEAAP8AAAD/AwIC/wwLC/8LCgr/BwYG/xMSEv+8u7v/sbGx/+7u7v/e3t7/tbW1/7e2tv8DAgL/CgkJ/wsKCv8LCgr/AQAA/wAAAP8CAQH/DAoK/wsKCv8LCgr/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8KCQn/Dw4O/w8ODv8dHBz/9vb2/9vb2/8ODQ3/FhUV/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/AAAA/2hnZ////////////0tKSv8AAAD/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/AgIC7QICAu0KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8ICAj/JCMj/zc2Nv83Nzf/Hh0d/wgICP8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8CAgLtAAAA7QEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8AAAD/AAAA/wAAAP8AAAD/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wAAAO0AAADzAAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA8wAAANwAAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\u0026#39;, \u0026#39;icon_uri\u0026#39;: u\u0026#39;https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico\u0026#39; } Testing: Relative links to icons For example: /images/favicon.png?1486118657\nfetch = \u0026#39;http://scrapy.org\u0026#39; sixfour_img = get_icon(fetch) DEBUG:Icon URL: /images/favicon.png?1486118657 print sixfour_img { \u0026#39;icon_data\u0026#39;: \u0026#39;data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMAAAADACAYAAABS3GwHAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAABmJLR0QA/wD/AP+gvaeTAAA4iklEQVR42u29aXRc15WY+50agEIVClNhIkFwAOeZFGkNHDQPlGxLlizLsiTb3Z2snjJ0nE6vl+QlnbeSvJe1ko7dU2J3J+6kB7fdtiy3OrZstayRlEjJnGcSIOYZhZrnuvee9+NUASCJGgAUUABY31ItFYFbt+692PucvffZex8oUaJEiRIlSpQoUaJEiRIl7hBEsS9gufFL37sP1HMtB8oAS+q9AMxT3k+HAcRT/5ep9xqQSL2X//vF48W+xWVFSQFmSUrQARxAM1CfetWl/r0SaASqgSaU8DtT703TnFKghHwYiAE6MAL4gVFgMPU7D+BOvYaBMEBJMWZHSQFyMEXQTSghrgG2pl4bgTUoYW9ACbyT6QW8EBhAEKUQYyil6AHagSuplw+lPAaUFCMXJQWYhilmjANwAa3A3cC9wA6gFqgCKij+M5RAFAgAXuAicAL4BOgDxlGzRMl8moZi//EWFSnBtwNbgN3AAZTgb0TZ8+ZiX2Oe6Ci/oR2lCB8B54CrQKSkCJPc0QowxbwpA9YCDwMPALuADamfLwcSQAdwHngfeAfoTv38jjaT7kgFSAm+GWW77we+CNyDsuHtxb6+eSaC8iE+Bv4GOInyJfQ7URHuKAVICb4V2Ac8CjyBUgBbsa+tSMRQCvAm8HPgFJC8kxRh2SvAFIfWDtwHvAQcBtooaLRGIiVI5MR7kxAIYcYkTJiE+qqp729GYGBgGDoAhjQwpIGUOoaUCKGOEYiJ9wXEADqBo8B3gOMox3rZO87LVgGm2PcNwP3Al1GCX1uY+5ZIKTGkEniruRxHmROb1YHd6qDcYqfaVkdleQ3O8hqc5dUIBM7yGhzlNdNcgCCWDOONjiKRBON+gnEfobgPf8xDXIsQ06JEEkHCiSBJPQ6IlJKJwtySiiiNo3yE7wMfoMKty9ZPWHYKMEXw7SgT56soc8cxl/OqkV2NwQA2i516xwqanK3UO1bgsjdR72imyuaitqIBZ3kNVnNhfOikniAY9xGMe/FERhiPjOCLuhkJ9jEc7MMdHiKmRaa9zjkQRplFf44ykSKw/BRhWSlASvhtKFPnH6AUoH6251MmiIHVXI7NUkGVrY61tZtpc21nVc16amwunLZa7FZnBrNm/jCkQSQZJBjz4ouNMxToocN9gS7PFQIxDzEtSlKPIzKaXHnjRinAt1GmUWw5KcGyUIApUZ2dwMsoc6dptuczpI5ZWHA5mmmsbGFj/S62NN7Fiqq1VFgdWE1WxAILfC6klCSNBNFkmKFAN1dHT9PuPs9oaIDx8DC61DCJOS1jjAB/ifIRLrBMokZLWgGmmDv1KKH/ZWA7M3Rup5oNdmslGxt2sbVxH+vqtrG6diM2y9KMjMa0CL3edro8l7kyeor2sfNEkqG5mEkGcAn4XyhlcMPSNouWrAJMCWkeAv4F8BAqNSFvpJQYGFRYHDRWrmTXigPsXnmQZmcrzvLalHO59JFSEox7GQ72cW7wQ84PfcRoaJCoFsaEaTb3GQXeBX4POMYSDp0uub/wlFG/BeXg/hNU9uUMUNGbalstba7t3NXyANub76a2on6uZsKix5A63qibS8OfcHrgfTrHL+GPeTHNLpI0DPwRylEegKU3GywpBZhi6z+AGvUfRuXX54WUEiEEzvJa9rYcYteKA2xr2keFtbLYt1YUoskQl0dOcX7oI84MHCMY9048oxkQR4VNfw+VZrGkfIMloQBTRv064BXga6jcnbyQUoKA2ooGdjbfy+G2z7C6ZiPllhlZTMuWuBal19fO0c4fc2H4BN7oGEhmqgjdwDeAv0LVLCyJ2WDRK8AU4d8O/A7wPDOI6Uspcdpq2Nl8Lw+sf4YNrh2YTZZi39aiRDc0OsYv8v6N17kwfIJgzDdTJQgDrwL/BeUsL3olWNQKMCWN4UHg36PSk/OK8EgkFpOVrY37eGjDs2xp2Iu9zFnsW1oSRBJBro6d4d2OH3Fl9BSakZxJxMhApV//LvAeizydYtEqwJRFra8A/wpVeZXzeiUSAayq2cBD659l36oHqLa5in07SxJ/bJxT/e/z7o0f0e/rQJJ36FSiKtX+E/AXLOLFs0UZ8kgJfz3wz4F/g4ry5BZ+Kaksr+bQuk/z/M5fZ8/Kg0s2hr8YsFnsrKvbysb6XSAEY6FBElosH7NIoEpHD6NC1Rf3PN8aOftqf7FvadoLXTTcEuL818CvkEeqshr1BWtrt3Bky5fY23KYMvOdmuE8PyT0GGcGjvKzq9+l23t14pnnQQz4M+D/YxGGShfVDLDn+VZQ5YjfQBWp5AxxSmlgL3NyaN2neemuf8bG+l1YTNZi38qyw2yy0FK1ju0r7iapJxgN9adyjXIqgQVVXroJOAu4F9NMsGgUIDX6bwO+Dnwa9eBysrJ6LZ/d/ssc2fwlqm11y2b1djEihMBR5mRLw16qKlyMhQcIxf35fNQMbEaVmZ7Z83zr2GJRgqJLyxSzZyvwX1EZnFkjPVJKTCYT2xr38/T2X2F9/Q7My3wFd7GhS50b7ov83aU/4/LoSQzDyGfwMVCZpb+NauFSdHOo6FKTMnu2An8APEYewl9usXF/22f5wu7fpLVmw4KnIpdQlW0uRzObGvaQNBIMBrrQDC2XEghgPWo2OMkiMIeKNgNMGfm3MTnyZ70eiaSqvJYjm1/i4Y3PlSI8i4SYFuGd9tf42bW/JhD35uMcSyZngstQvJmgKDPALWbP14HHyWPkb6hcyed3/hoPrP8cZea8U4BKzDMWk5V1rm1U2+ro83UQTgTzmQnaUD7BacC95/lWijEbFEUBUmbPeuAPycPmN6TBquo2Xrrra+xf9RBmU9EttxK3YBImWms20ly1ml7fdfwxT77m0EZUpZn3jlCA1Oi/AviPwOfINfIj2dywlxd2/yY7mu8pRXkWMUIImpytrKhajTs8zHhkOJc5lJ4J6oATe55vDS20EiyYNE0xe6qA/wf4R+TovGZIg61N+/jSnn/KmtrNC/pgSsyNHu81vnv2D7kyciqfIEUC+G8ouQjAwvkECx0+sQK/AfwaOYRfSsm2pv28vPdrJeFfgqyp3czLe7/Gtqb9Kh09O2UomfgNlIwsGAtiAk0Z/V8A/h1qysuIlJLNjXv50t5/yuraTQv5PEoUkGpbHa016xkK9uAOD+UyX62olPc+VO7QgjjF864AU4T/PuA/oxyfjBjSYL1rOy/d9c9YV7d13h9AifmlpqKelup19Pk68ERGcymBE7VGcBboXwglmFcFmCL8q1Gx/gPZjpdIVlW38cLuf8Tmhr3zeuMlFo7aigZcjmZ6vdfzWSdoAlahutL551sJFsIHKEOlNT+a7SCJpNrm4rmdv8qO5nsW4LJKLCQ7mu/huZ2/SrXNNdGWJQuPomRm3tvTz9sMMKWa60vAvyRL2/F0Hv/nd/4q96x5rJTasExpcrZit1bS4b6Yq67AhDKFeoEL8zkLzIsCTDF99qHqQ9dkO95qtvLE5i/x0IbSCu9yxiRMNFetRpc6neMXMaSR7fAK1CLZJ8DQfCnBfA61VahY/85cB97V8gCPbfxCKbfnDsBmsfPYxi9wV8sD+Ry+EyVDVfN1PQWfAaaYPq+gujhkjOtKKdlYv5MX9/4WLscMe1uVWLKUWWy0VLfR673GeGQkV2QobQqdnY9ZYL5mgN2o3j0Zh3RDGtTZG/ns9l9mZdXaebqMEouVlVWqkKnO3pjLFLKjZGn3fFxHQWeA1Ohfieri8CRZUi2s5jKObPkSB9c9VXJ671DqK1eiGQk63BdyKUFD6v/v73m+NVHIWaBgkjfF8X0EeI4ceUZ7Wg7x8IbnSpVcdzBmYebhDc+xp+VQrkMFSqYegZtkbc4UeuhdAfwWarfFaZHSoNnZyhObXsRRNm++TYklgqOsiic2vUizsxWZfRZoRMnWikJ+f0EUYIrj+xxwMNNxEkmZpYKHNjxLm2t7Ie+jxBKmzbWdhzY8S5mlItci2UFS1kWhZoFCzgCbgF8nx+rdtqZ9HFxbsvuXC3oyQcQ3RtA9SNA9SMgzTDIendE5TMLEwbVPsa1pX65Dy1AyVrAMyTl3iZ0y+r+A6umTAUm1rY4jm18qmT5LHF1L4BvqZqz7Ev7RXqJ+N1oytWulyYS9ugFX62Zatt6NvaYhr3M6yqo4svmliT3OsriQW1Cy9h9/6Xv3zbnv6JwKYqZMQ3tQW+bsyHb8U1u/zLM7/mGpcdUSREqDeCSIu/syXad+jn+0l1jQi5aM37JNq9oj2Vpuo7JpHRsPPMu6HXcjTLlnfM1I8qOL/5M3rvxlrkMvorbEOgtzK54pRJ/w8tTFZBR+Q+qsrd3CgTVPlIR/CRIP+xnpOEfHJz/DO3gjNdpLtQOl+XYREqhZwtt7hfd7ezBe/BobduVOcLSYrBxY8wQXh07Q7b2abbeeHagNzy+jqslmTSEUoBXlmEyLRFJmtnHvmsdpqW4rwNeVWCgMXcPdc4Ubv3iToeun0eIRyHtjboHJbEaGfZx44zs0tKyl2pV7486W6jbuXfM4g4FukkYiW+r0s8CfADfmco+z9kRvyfZsyXighDbXNg6sfXIu11ligYmFfFz54FVO/OAb9F08NiH8aaRhIIQJaeiQpeTRVl7GcPtZOs6dyPu7D6x9kjbXNnJkTa9BzQJzigjNNRSzAfg8WfJ9yizl3N/2NNW2uvzPWqJoSGkw2nWRT177Qy6/9yrRYMohnSL8JrOVlVv2s+ORL7F61/2U2Z0ZlcBiMWMWBtdOHSMeDed1DdW2Ou5ve5oyS9bMYCtK9jbM5X5npQBTNO4IWSI/UkrWu7azo/nuuVxjiQVCS8ToOvU2p/7uWwy3n0Ea+m2JalJKXK0b2ff0b7D1gef51LP/mDW774cMYW0hBGVWKz3XzhELB/O+lh3Nd7PetT1XQf0WlAzOenV4LjPAClS+z7Rqqha9bBxYc6S0Q8sSIOof58LPv8O5N/+cwFhfxuOEEDhqGrFXqb+ppcxGpWslJrOZbDZLyDdOLBLK+3qqbS4OrDlCmcWWbXGsHCWDs14dnrECTNG0/ajtSqfFkAZrazezJffiRoki4x/p4cxPv03Hxz8lGQ0hsixSSinxDHQwfOMs0cA47t6rDFz5BD2ZIJtjbBg63pGBGV3XlqZ9rK3dnCtR7gGULM5qFphtFKgc+AwZ050l5WYbe1YepMGxcpZfUWK+kYbOSOd5Lr/7fcZ6rihzJ0fnPSEEAfcAJ1//JhXOOuKRAGHPSMY4v2EYJJKaUo0ZdvVrcKxkz8qDdHmuoBkZFcyOksW/R+1ZPCNmawKtBp7K+GAl1Nmb2NNyeJanLzHfaIkYnSff4uTr35wU/nyRkrB3BHfPZYLugaxJbElNJxAMY7GWUdMwc0tlT8th6uxN5Oit9RRKJmfMbBXgKSZztG9DCMGO5ntocrbO8vQl5pNo0Mvl937AuTf/gog3Z6+eaRHChDCZs5pLAD5/iEg0RsOqNiqrZx4JbHK25tMTtoEsA3I2ZqQAUwpeHiPL/l0VVgf7Vz1YyvVfhPhHejj7xre5/tHrJG+J7ReacCTKyJgHKWHHfY9gr6qd8TnMwsz+VQ9SYc26N3o5SiYrZ+oHzGYG2I3q6z8thjTYWL+LlprSqu9iwtB1hjvO8slrf0TfxQ8xtNRuLlIiDV29cvfwzJtYPMHA0DjRaJwNu+9lz+GnMOWRDzQdLTVtbKzflcsZ3sosyibzvqIpmnUPsDbTcWaTme3Nd1NVPnNtLzE/6Mk4Xad/zqnXv4lnoEP9MCX8Nmcta/c8xLp9j+KsL0zAIhZP0Dcwij8QxNW0iodf+DVqm1pmfb6q8lq2N9+dy6JYi5LNGUWDZhoFqgPuJYPiGNKgpWqd2li5xKIgFvJy/aP/w41f/D2JaHDSZpcSS3kFmw8+w/q7jwDg6W/nF3/7x4Q8wzlt+0yEQhH6h9wEQxGq65o48pXfYsPOuS+Erndtp7GyhaFgb6ZaEhNKNusAT77nzesup2jUKlST2wxIWqs3sKp6fR5nLTHf+Ia7Of3j/8H1j/6OxC3xfYnEanNQs2IdljIbljIbrtVbKLdXkSvkMh26YeAe99PVO0wgGGbFuk08+5u/y+7DTxbEz1hdsynVKTzrtd2HktG8Z4GZzgA7yLjqJim3VLChfidW87y3dCyRBSklIzfOcfHtv8bT3w5wWxRFCEE8EmC4/Qy1LRuwWMsZvPoJEb87Y1pDJmLxBKNjXsY9fnQJm/Ye5LGXfpN12wq3CGo1l7GxfhdnB4+R1FXxzTSsQMno+XzPOxMFKAMeJkMrFYmq6tnSeFfBbrrEzNESMXrOfcDVo68RGh/KUogiMJIJbvziTcZ6rlBud+IZaCceDuQVFhVCoOs6/kCY4VEPoVCECmc19z78DIef+Qp1TasKfm9bGu/CUVaFNzqWac3ZjJLRV8mzTmAmClAJfCrbASuq1tBYWfgbL5EfsaCXqx++Ttept0hEw7mrsIRAi0cZ77uGQA1i+a4JhMNRxsb9jHv9GIakZf027n/2l9l16Aks1vmxABorV7Giag3e6Fi2wz6FktW8/ICZKMBmIGP/QoFgc8MerOZSxVcx8I/2cvnd79N/+XgqVz9Pu1tMlpzk8wldNxj3+Bl1+4hGY9idNex96LPc8/jnaV67eV43MbSarWxu2MOVkVPZDmtGyWpedZIzUYB9qB08pj+RycoG185sZWwl5gFD1xm5cZbL7/2A8d6rKSkurBCmzZ1AMMKo20sgEMJsLadt56c4/MxX2bT3AGW2+W9sbBJmNrh2YjFZ0YxkpsOcKFktjAJM8aa3o1pW34aUksbKFlyO3CVvJQpHMham68y7XPvwb4n63UzaMZJCKYGUklA4invcj9cfRNN0mlo3cNfDT7Pv4WfyKnMsJC5HE42VLQz4uzLNNhUoWeWXvndfzoL5fGeAOtR+rtM/JAxaqtfjLC1+LRjRoIdrR39E5+m3ScYiE3n6NSvaiAbc+Ia7MXRtTt+RSCQZdfvweAPEEwnszho+dehJ7jnyBVas24ypCBuWO8traaleT7//BiJza9v03sM5/YB8FaANVfyekVXVbaV+PwuAlBL/SDcXf/5dhtpPTwi5s76FfZ/9NepaNpCIhbj48+/Sfe69Gcf0hYB4XMPnDzI27icciVBRWc22PQc58JmXadvxKaxlxdvExFFWxarqNj7OflgrSmYLpgDNZOj3KZFUWB3UOwrasrHENBi6xtD1U1x+7wd4B1UzBCEE0tBxtW5S8fzyCizlFTRt2E3vhaMzmgV0XcfnD+H2BAiFIwiThY2772PfI8+w495HsTmceZ9rPql3rKDC6iCmRTN1jWgkS8BmKlkVYIr9vwKYtq5RSomzvLZk/88zWiJG1+m3uXr0R0T87pvtXyHwj/QS9o5Q1bAKPZlgvPcqhqHnPK9AreIGghHcHj9+fwgpBE2r1rP/0WfZffhJahqaKbRjPRdcjiac5bVEk5FMfoCLlALk8gPymQHKUHs1TUt6d8emylLu/3wR8Y1x5YMf0nPufZLxyG15OkKY8A118vEPvkH92m2EPcOM9VzOav4IAYaRcnA9ATy+ALqmU9+yhp33Pc6nHn+O+hVr8urottA0VbZSbXMxEsq6T8A6lHxnnQLzUYAKshUdS3CW11BVansyL4z3Xefye99nuCPdpWF6gZRS4h/txT/am/OcUkI4EsPjVYKfSGhU1TWw7e6H2P/Yc6zetGte4/lzpcpWh7O8JhXxynjYasAB+LOdKx8FsJPFAbaYLNQ7VpS6PRcQKSW6rjN07SSX3/0e/uHuVELZ7IUylf1MPJHEPe7H4/UTiyewOarYc+/93PfUF2lZv43yCsesv2OhMAkT9Y4VWEyWbB0jWlGyO2cFqCCLQ2E2WUr2/ywxDANd11MvA03XSCaTJDUN9+gIF3/+IxjrxGyee7gxHk/i9anITjQao9zuYNs9h7j3iRdo2/mpJSH4U3E5mjCbLNkWxJrJsG41lXwUwEZGBZBYTNZSBCgPDCmVcCeTJBIJEonkhLBrmoamJdE0fcrxJipW7cYz3k+lDM86pVjTdDy+IB5vgHA4itlaxqa7DrHn/qfYds/DOKpqiv1oZoWaAazZukU0o2Q3KxkVYEoEqIZMK8CAyWQuNb6agpQSwzAwDIN4IkE8HicWixONxdB1fWLUNwx9wkedzt42mQSrN24n7Bsj0nUUuzl3RCeNEAJN0wgEI4yN+wgEwpgsVprXbubuxz/PzoNP4KytX9R2fi6qbS5MJnM2N6ACJbtZI0H5zACryGJ8moWFGlt9sZ9H0TAMg2RSI5lMkEhqxOIx4rE4sXgcTVMBCCnlRL3tpNCJnIO61Wpl/a77uOgfI+65hM2cq1+sSlZTkR0/Pn8IKSVNazay5/6n2P/Is1S5Gpe04KepsdVjFlnFV5AqjslGLgUQqF26Mz4xkzDlqthfVkgpSSY1YrEY0ViUeCJBIp4gkUig6ZlH6dkInZQSR2Ula3bfz42PPFgSg5hNIsOxEImoFGWfP0hS06lpWMG+h55m16EjNK/dNOui9MVIhdWRK/AyVXYzjhv5KMBKspRO1lY0FCUnZKEwDANN04gnEkQjUcLRCIm4EnZd16cZ2QuLlJLG5hYC2x5g7MyPqZXBie9Kx/Jj8UQqshMgkUzirGlg/4FHufvxz9O8eiOWIqYuzBcmk5naigZ8MXfGQ1CyO2cFsGT7ZYXVgWkRrRLOFQnomkYsbbtHo0QiERLJ5LRtQxbCnBAC2jZtI+AZI9D9ATWWJBKIxhJ4vMrBjcbi2J3V7Dh0P/sf+Rzrtu+ft8KUxYAJQYXVkUvyLOSIHefjA+RqFpnzkMWOlBJN14lEIoTDEWXHxxMkk8m5n7xAWCxmtuy9h3PBcfwjp4kF/bg9fsKRGGXlFWzed5h7nniejbsPYHNUFvtyF4DcfUzJQzBzKYAJlVWXwQQS1FbUL0kTyJASLZkkFo8TCASJRCIkNW3CrFlsjqKUEmdlJSs27ObUlU+IjwxjLq9gzZY9HPj0l9iy/34qKqsX3XXPF8oEqiebe8qk7GZ0zvLNBcqIzWJfMg9dAolEYsKsCYXDxOOJ20ybxXo/Qggi8SQD8UpWt2zl4ac+x+7DT1JV1zj3ky8xhBDYLDmr0HLagIXYJG/Ro+s60ViMQCBAOBIlHo+j6/qiFfRMJJNJRkaGicoydj72PAc/+9ySnH0XE8tWAaSUxBMJwuEw/kCAaDR2k3mz1IRfCEEoFKK/rx+r1cq2HbtKwl8Alp0CGFISDocJhUIEgiHi8Zv3TFhqgj8Vr9fL6OgojY0NrFxZSj8pBDmX0nKtPAohFkUMSNd1QuEwPn+ASCRMMjm3etjFhmEYdHV2kUgk2LF9BxUVOfO8ljWC/AazXDKcUQE0HQwDzGYwZ1lwiyUjGAVsqz1TNE0jHIng9foIhcMYhjH3ky5CIpEInZ2dyvzZvhWr9c7uv2RISSwZyXqMboCuQ7YF8IwK4A6CPwINVVBbmSnYJPFERjEMDcwLu9qo6zqhUAif308oFEZfpoIPaqTr7OzE7XbT1tbGmtWz2g1oWWEYGp7IKJmqYiRKfscCUJ0lWJRRAWJJCMZSH85WeTMPjZiy3riURKNR3OPjhEJhtPRGD8sUIQTxeJz29nY0TWPnju00NZXqL9RCWJZfS0hoSobLs0yWWdMcTDkW2ySQ0GJIFmb0TSQSeLxePF7fRKblchb+9P319PTQ09NLY0MDe/fuKfYlLQokRkr2sozNIiXDWc6TUQHGQ5JeNzhtgsYs7X680TH0PLoPzAXDMAgEgrg940SjsYJu5bOYEUIQjUa5cuUK0UiEw4cOsm7dumJf1qJAN/RcTXIJRKDXnV1WLNk+7PNKoo0kJJkjPZqhZStLmzOJRJJxzzher4/kMjd3pmNwcJD26+3U1dby4IMPFKQ8cjmgGUk0I3OkTwLRhEz4vJIae2aZyegfRxNAO4Yh6RRktnF0I4k7PDwvNymlxOvzMeYeR1uCK7dzJRqNcvr0aeLxOPfddy+tq0qt59O4w8PoWQbeVKZ4J+0Y0Sw7BWRUgBE/0AH+SGaLQzVV0vHFsk9Fs0UIgbOykvLy8jvG7Jl67zdu3KCrs4sVzc3ce++9d3zocyq+mDK9Mw2JUirZpSMlyxnIXiJUgYxraOGMG9ALdKnhj+W9J9mMsdsrcNXVYjYvn2qmXAgh8Hg8nDl9Bk3TOHToIBs2lPZdm4o/5kGXGplc3HAc4hoaFdnXcrNLVWdSIhmMJTEyaVpSTzIc6JnXm62traXKeec03tV1nUsXL9HX18eGDet58MEH7jjzLxfDgR6S+vQmkABiSQwkg3QmZ68AUlbIeJJAJI7MNNfoUsMTnR8TKI3ZZKK+3kVZ2fKtcEojhGCgf4Bz585hs9n49FNP4nKVum7ciic6lpoBpkFAJI6MJwlIWTF7BVj7b3UiCfpDsewGuC/qxhuZXyWwV1RQ73Itq8LuW0lnfH700UcEAgEOHz7E7t17in1Ziw5vZAxf1J31mFBMykiC/rX/NnuIPrM0fesEAx6IBvGF40TJ6AgLIokA3nlyhKdSW1NNlXNxtOieD3Rd59y5c3R3d7NhwwaOHHkCm235FbTPFW9sjEgikDk4LyEcJxoN4hvwAN86kfFcWYdTzQB0YgmN4XjG2UYQjPvwhEfm/cbNZvOyNYWEEPT29HLm9BkqKir4zKefYuWKUsrzdHjCIwTjvowKENcgoTGMTkzLkaSQ254QRKMJhqOJ6dMihBAE4j6GQ30LcvP2igoa6l2YTMvHKRRC4PV6OXbsGOFwmCNHnmDfvtJ+y5kYDvURiPumDQwIodawogmGEURznSu3ApiIBKP0ReLZcyqGg73EtEjO0xWC6urqZRUVisfjHD9+nL6+Pu66ay+PP/YYFsuyq1UqCDEtwnAwcwt4AUTiEIzSh4mcAplbAaxEA1GGQrHMhQUmBP2+GwRi3gV5CJaUKVRevvTtYyklFy5c4ML5C6xZs5pnP/cMVVXL18+ZK4GYl37fjYy9qCQQikEgSi8WwrnOl88MkMCg3RdWRTLTIYSJ4UBPzuSkQmKvqMBVV7ek4+NCCLq6ujhx/AQ1NTW88IUvlJLdcuCNjjEc6Mm4UYimgy8MGHRhJmdZYHYFSHvPgiF/RI4ns0SUEnqcHu+1BU1ZWOpRoeHhYT54/wOSySRPP/1Z9uzZXexLWtRIKenxXiOhZ0xNIKmDPyLHEagEtSwRIMhnBgAQDI+HGI0nM/sBEsnV0dMkjURepywEZrOZetfSiwoJIQgGgxw9epSxsTGefPIIDz5w/7Je4ygESSPB1dHTGXeFEUA8CeMhRicUIAf5PvHOhEbfqD+zHyClpNtzFU9k/sOhU0nnCi0VUyid43/s2DG6Ort45OGHefLIE0tOiYuBJzJCt+dqRitDAqN+SGj0AZ35nDMvBWiowqMbdI4FZMaNB4WAcCLI1dEzC/pQhBDU1tTgrFwa/TA1TePEiRNcvHCJffv28eyzz1C5RK692FwdPUM4EcxYpSgljAUkukFnQ1XuTbIhHwX41gnGAoDBJU+IaCxjCrYgpoW5PPKLBTWDACwWCw0N9Ys6XVgIga7rnDp1ilOnTrFnzy5e/OIXqK6uLvalLQmSRoLLI78gpoXJZIjHkuAJEcXg0liAnPY/5G8CAZwKxwmOB7PXCQ/4Oxn0dy34A0pHhUyL1BQyDIMLFy7w0YcfsWvnLl55+SWam/PazLwEMOjvYsCf2aoRAsaDEI4TBE7le96ZKMC1aILhsYDM6AiYhImRUD/d3qsL/oCEENTW1lDpXHzmhJSS9uvtHP/oOK2trbz44gulzg4zpNt7lZFQf+ZdYVLmTzTBMHAt3/PmrwAmQhj8wh2AcCLTLCBI6nGujJwmkgwt+EOyWiwqKrSITKG08L/11lvUu1y88vJLtKxcWezLWlJEkiGujJwmqceZzvxR/ie4lan+C0zkLXwzmQESCN4ZD6H7s6yvmYSZyyMnsy5XzyeVDgcu1+JYIJNScu3aNd5++23a2tbxK7/yS2zcuKHYl7XkGA72cnnkJCaRuSGAPwzjIXQE7wB5O6EzCzwLLsYSDA37VNu5TITiPs4OHMOQxenWVltTU/TISnrkf/edd2lrW8dXv/Ll0irvLDCkwdmBY4TivozH6AYM+yCWYAjBxZmcPz8FmPSm+xEc73VLYll0zJAGZweP4Y+NF+WhWSwWGoqYNi2l5Pq167z11lts2LCeL7/ySsnhnSX+2DhnB7MPprFkqv+P4DjQD+QVAYKZzgDgAU4EoxijgSxHCRgJ9nN24FjRHpzDbqeuCAtkabPn/fffZ8+e3bzyyss0N5cc3tlyduAYI8H+rKnIw14IRDGAE5Bf/D9N/gowqVEfawbd3aMyoxkkUmsCpwc+wJujdG2+EEJQt4CmkBACJFy/fp2jHxxl7549vPLySzQ13nnbFxUKb9TN6YEPiGnhjMUvugE9bolu0A18DOQ9+sPMZwCAc0iuuIMq7pppgBXCxPWxs1wdzTskW3AsFguNDfXzHhUSQpBMJrl48SLHPzrOgw88wIsvfhHnEk7UWwxcHT3F9bGzGTM/07H/8SAguQKcm+l3zEwBlGaFELwVihHvH5cYWWaBuBbldP8HRBLBoj1Ee0UFdfO8QKZpGqdOneLMmTM89ugjPP30Z3A4cm7gViILkUSQ0/0fENeiGUd/w4D+cUkoRhzBW0BoJqM/zG4GAHgDyVi/BwLR7LPApZFPuD42Y8UsGPO5QJZuXX78+HG6u7p57rnP8fjjjy3qlIylwvWxc1wa+STr6B+IQr8HkIwBb8zme2arAL0I3vCGYNAL2UoAIokQb3e8Sijuz//sBcZqsdBQ4AWydAuTD499SMDv55WXX+Lee+4pNa8tAKG4n7c7XiWSyLyeJaWSPW8IELwBzGrhabYKEAd+LCHSPiSJxLMcKaDdfYHTAx8s1PObFkeBF8h8Ph8ffPABDrudr37ly2zdumVRLL4tB04PfEC7+0LWyE8kDu1DEgkR4McomZwxM1eASRvrJPC+PwzdY5mnAIEgmgxzrPsnjIb6F+whTkch0qaFEAwODnL0g6Osb2vjxRdfYEWpfUnBGA31c6z7J0STYbJtv9g9JkllJLyPksUZRX/SzKUEaQj4qW4Q7xxRhciZBkCTMHHDfYmTfe9iyPndTCMbc10gk1Jy48YNLl+6zMGDB3jm6c+WIj0FxJA6J/ve5Yb7UsakNyGUrHWOgG4QB36KksVZMTsFmNS0nyG46glDrzu7L2BIjfc6X6fHe31hnmYG7HY7dbUzWyBLhzkvnL/A0OAgn37qSQ4fOlhydgtMj/c673W+jiGzbHwhlax5VFnAVeBnwKxGf5jbDADQAfzQ0El2jcqsESEQjIYGeLv9VeJazn5F84YQgrra/BfI0s7uubPnMJsEz3/+OTZv3lSq3y0wcS3K2+2vMhoaIJPxn478dI1KDJ0k8EOUDM6a2YcsTvbD/lUAAwg+G41TV2YVNFVnUQIJ45FhXI5mVtdsXIjnOi0mkwmr1UIwFMq5r/DY2BgdHR2sb1vHo48+Uqrgmic+7n2Lt65/n6SeyDg7SwlXBqBrBBB0Af8X4Jnt6A9znwEA+oAfSaBzRDKeJRNbCEE4EeDdjr8tWrp0GofdnrGYXgiBYRj09fUxOjLCoYMHuP/+w9hstqJe83JlONjLux1/SzgRyCj8AhgPKRlLWdo/QsnenCiEAiSAvwYuBqNwbVBmbKAFanGs03OJN699b8FaKU5/HYK62locdvttP49EInR3d2MrK+fIE0+waePGUohznohpEd689j06PZcyLnoBJHQlW0FlPV9Eydyci8/ntmozaQaNAC7gcCSOyVkhqHVk+6BkMNCNy65MoWIJl9lsxmq1EAqFJ0yhQCDAuNvNujVruPvu/djtpZSG+UJKg+M9b/LGlb9MbXiXWQ56xuByP+gGGvDfgR8As3Z+08x92fJmJXhY02mMa7CiVmTZoVugGRojoX5W12zE5SherrzVakU3DIKhEB6PB6vFwp7du1i7dm3J0Z1n2t3nefXCt/BF3ZlNn1TY81SnJBABBJeBfwe45yr8UAgFgLQCeFDrXo+G45jNpuwOsRCCYMxDIO5lY/1u7GXFqeASQlBeXkY4HKaupprNmzdRVVVVMnnmGXd4mFfPf5MO9/mspo8h4VI/dKm2swng/yUd+jw594XVwijA5CzQD3wKWBeMQm2loDqLBSGEwB0eQiLZ1LAHi6k4LcHNZjMuVx21tbWltuQLQFyL8ZMrf87HvW9l3cJRCBj0wLluSaov7QcoBQgWYvSHwjjBUxkC/gAYjSTgUp9yWrINprrUONb1Ez7sfqOoq8QlFgZD6nzY/QbHun6SeZM7lMz4I3ChVxJRru4oSrZmveo7HYVLXZy6LgCtwL5gTHk1TdWCzOa0IKHH6PN14HI0s7J6bdYckBJLF4nkVP97vHbhTwjEPFlNH81Qwt81mvoo/BXwJ0CiUKM/FFIBIK0ECZQSHAYaAxFw2AR1WUx8IQTRZIihYA+razbhspdqaJcjHe6L/M25P2Yk2JdV+EHl+lzsneg+chn4v4HeQgo/FFoBYGpEKA48qutYA1FwOQWVWTZ0EULgj40zEuxjnWsbVbbagl9aieLR7+/ku2d+n07P5czd3VCB0LEAnLohCccAQQQV9SmY4zuVwivApCl0A2hFsDeWhFgCGquzhUaVEoyHhxkO9tJWvx1neU3BL6/EwjMU7OE7p77O1dEz2YVfQDAV8hwNkF4W+Cvg60C80KM/zIcCQFoB4qio0H1AUziuQlrZ/YGUEkSGCcQ8bKjfSYXVkddXlliceKNj/ODcf+f88PGcx2o6nOuZsPsBLqBMn26g4KM/zJcCTM4Cw0AQeEhKKnxhsJihvkqQK8w+HFR7jq2v31FSgiWKNzrG35z9I072v5tz6yxDwpUByZX+CbvfC/wb4O+BOa/4ZmL+ClgnleAq0Ah8yjAwecLgKBfUOLKHRw0MBgM9+KJjtLm2UWFdfF2fS2TGExnh++f+mE/63sGQetbInpTQPQpnuiGhMiJ04H8Avw/o8yX8MJ8KAGkF0FFe/HYEGzQNvGGosQucFZk/qh6YZDjYy1Cwl9bajVSVlxzjpcBAoIvvnvkDzg5+mFP4AYa8cLJzwukFeAv416S7vM2D6ZNmfhVgchbwo5ziuxE0xTXVzChXZAhUwtRQoIdBfzerazdRVb44Oj+XuB0pJX3+Dr5z6utcGD4BZDd70hGfE9cl/ihp4T8H/A5wCZg30yfN/PfwuDlNwgscBJyxJAQiUOcU2HOU6CrHeIge7zUaKlfS4FhZUoJFhpSSy6Mn+e6Z36dj/HzOOH86v/8XHTfVkAwC/xI1A8y78MNCKABMVYJrqPSLA4A1HFfL3S6nwJ5z03eBNzJKt+cq9jIHK6vWZQ2plVg4dEPjRO/f88Pz36LXdz238AtV0/tJh2Rksl1UBPjPwJ8DxkIIPyyUAkBaAQxUaMsJ7APMoZja2duV50wQiHvpcF9AIllVsx6rubS9aDGJJEO81f59Xr/0Z4yFh3IOSgIl/Ceu3yT8CeCbwO8BquRlHu3+qSycApzsT88EceA8sALYCYj0TFBXmZ8SxPUoN9wX8UZHaXauLi2YFYnBQDc/uvinvNvxGpFkKC/hHw/dNvLrqMWu/wCM860TCyb8sJAKkEYpQQilBFuA9YAIxSbNoYpcSoBAlzo93uv0+TqornDRULky59RbojAYUufC8Md8/9x/48zA0bwiPaCif590SIZ9Ez+SKHv/d4D+hTJ7plKcRpbKHPKitrPcALSRUgJPCJy27CHSNOnUiatjp9GlTrNzNeWWPD5YYtYEYl5+3vEqr134Uwb8nXkHI4a8SvhHJ0d+A7XI9c9JtzZZwJE/TXEUYNIpdgNnUEqwHhCRuJom7eWCqgpyrhgLIYhqYa6PnWM8MkytvYGaivpSlKjAGNKg03OJ1y99m3c6XiOSDOb1jKWEvnEV5/dMRnskSvh/G7gCLEjEZzqK18p40icYA04Dm4F1gIglYdQP5VZVUWbOYdkIBFIaDPhvcHX0DEk9QZOzlXKLrVRbMEckkkDMyzsdP+S1C3/K9bGzSClzPlchVEpD5yicvDHRzQHUyP828DXgykLb/LdS/F7ekzPBSdQssB4QSQ2G/eoh1jkE1rwqFQXBuJ9293l6fNepsDpwOZowF6nUcqmT0GOcHzrODy98iw+73sCfo4gljRAq+/dCn+Rcj3qf0hcDeJO08ENRhR9YRMPjr98LsA34r8AT6Wszm2B1Pdy1Lj+/II1E4iyrYX/rQzzQ9jSrazeV1g3yxJAGvd7rvN/5d5zse5dgwjejmTQYhdNdkl73TdvpSpTw/zZwuVgmz60UfwZIM2kOnUKVVLYBZinVOsF4UFWWVdrAlMffQm3RFKPbc5WrY2fwRd3UOZqwl1WWFCEDutQZCfXz5tW/5v9c+d9cHPqYhB7Ly9YXKHt/yAeftEsGPDc1S04AP0FFe64sFuGHxaQAMNUcOgo4UOsEFoBwjInwWZVDYM1DhoUQqrlt3McNzyWujp5CNzQqy6twlDlLjnIKKQ1GQn0c736TV89/kzODRwnGvAhhylv4Y5rq3Ha6Uw1YUyaMGPBt4F8BXUDRzZ5br33xocyheuAfA78F1KR/ZTHDmnrY3ipwVeZKt7oZicQsLKyu3cjuFQfY3/pgKqVicY0DC4UhdQYDXZzse49zQx/R621Hl9qMzJ304talPkmPm1vbYvpQnRz+mAI1sio0i1MBIK0ENuArqNFjTfp6BVBbCdtXCVY3QJkl+94Et2JIHbPJQmPlKrY17efg2idpdq7Gbq1c9otpUhpEkiGGg7182P1TLo+cZDTUj25oMxoIhICEBr1jcKlf4g3dNBhJoAf4T8BfALHFKPywmBUA0koggAeBf49KopuQUIsJ1jUKNq2EhqqZn15KAyFMOMqq2Fi/i70th9lQv5PGylVFa9I1X2iGxmionw73Bc4MHKXdfZ5wIjDxDGbKWACuD6pe/drNHeYN4CPgd4H3ALlYhR8WuwJAWgkAtqOcqOdR/sEENXZY3wwbmlUahSqlmRlSGljMZayqXs/a2s1sb76bzY17cVidSzaMqhsagbiXdvc5Lg+fpNt7jX7/DTQ9MWOhTz/TaAI6hiU3hsF3e3PvMPAq8F9YoHz+ubL4FQCmKkEd8Aoqjrx24vcSzGZorIKtqwQraqDMOjOzKI3qTieoLKui1t7A2tqt7G05REt1G87y2pSZtDgfm5SSSDJEMO5lwN/J2YFjdHou442OEU6o7dRn4+8IoUoVh3xwpV91bNDVY5pKN/ANVGKbquRa5MIPS0UB0ihFMAMPAP8CeBi4qZLAaoFWF2xaIWioUmbSLPQAiQQpEcKE2WShrqKBtXVbWV2zkRVVa2lyrsJlby56wX40GWY8MsxIsJ+hQDe9vna6PVfwRMfQDU0VowtmtSIuUB3axgJwfUjSNw7J27sZxoF3UKnM7zPPNbyFZmkpAEydDVqArwL/BLipv7oAHDZoqVNmkatSzRDI2SkDKIVIdzaoLK/GZW+i2uaisbKFluo2XI5mmp2rqSqvxWyyYBKm1GtuESZD6hjSwJDGhEkzHOxlPDzMgL+T0dAA/tg445GRic3IhRCzTgFJRxl0XUV3OoZVTD8cm/bZDQN/hCpiGQCWxKh/2/0uSZQiWIFDqNngIWByrViqqbuiXM0I65tUe8Yyi2rBMReUMhhIKTGZzJSbbVjMZZSZy3GUOam3r0gl5anEPLu1kmqba0Ihyi0Vt5lRUkriWnRC4P2xcSLJEL6oG190DG9kDHdkiHAiSEKPo+kJ4noMw9BT6x2mOec9mVKRHU8IboyoET8aT5mSN586CryLGvWPAcmlJvhplq4CwNTZoB74MvDLKGf5Ng/PXq5mhNX1at+CsoL6tal9q1KCkhZEof7BVAPEYiqjuqIuNTOkNVFgSB1/1INmJGDiNzL1n5z8lonvUJ8rFAkNRvzQ61YjfmT6fdcNlHP7v4C/RC1aLrlRfypLWwHSTPoGO4GXUcpwe4ddCbYyqHfCmgZoqRPYyiazTWfjNM8GOZ0tNks7fTakJx/dUIlqAx5Jzxi4gzclrt3KCErov4Mqa11Stn7GZ1HsCygok4tn9wH/AJVUV3/bcamokbNCKcGqOqi2q1lCiIVThIUmfW+RVAlqv0cJfzA6bVQnjRuVxPZt4DiLeFFrVs+k2BdQcCbNIjtKAb4KPMotawdTKbeCqxJW1EJjlfIV8ku/XjokU7b9aEAy5FUObjyZ9SNh4OcoB/dNVNeGJW3uTMfyU4A0k4rQANwPvIAKm7qmvW8JwqRmAacNmqrVRn/VdqUMFpNyEiWLd4YQKd/AkCp8mdTUSD/kVUXowZga/aVBpr+8BMZRYc3vo7YkUrtzLTPBn3hmxb6AeWcynaICZRq9jNq8o43ptohKCbcwKaF32KChStDghCo7OMqVkljNk4cXSyGEmPwDJnUl3OG4ajg2FoSxgGo3qBkpoU8/idsxgE5UFu53UKZOlEWexlCQZ1jsC1hQJkOn+1Bm0RPAfpTfkJNyq/IVqiqg0gbOCoHTppSiolw9zKnRzYnMvVsUJJO+iOl+cIu/LFP/jidV4UkwBsGoJBSDQFSN+DlMm6nEUJV4b6LMnVMs4ZDmbLizFCDNZNRoJUoBvgjcg+pinX1n7MnIJVYLlJnV/21WpQjOCtXbyD5lpjCZbi7iMZtuL/aX8qbqKQwJhjE5skfiEEkoYQ/HlZAnNLWDelK7+bpyEEFtOHcUeA0l9IMsk6jOTLkzFSDNpJ9QhsotehiVZrEL1akid9u5qcOzmDRLxJT35VYmdsYRKMWw3FLWphlS2eepf8eT6pU2sdIjv5S3f2ceJFCtR86j0hXeQRWnqLniDhT8NHe2AtyKUgg7qmHXblT69d3ARpQyLJXKGR0l9O3AJ6j05HOovRoid7LA30pJAaZj0nF2oKJGrShFuBfYAdQCVSjHutjPUKIc1gCq2dhF4ARK8PtQUZ0wd4BDOxuK/cdb/EyaSSbUDFADbE29NqIq1Vaiwq2NqMa/81VWZqC2nBpFhScHUZVX7ag2I1dQZYh66tg72rzJh5ICzJZJxXCgslHrU686VOPfepRiVKZ+X4aKNjWjUrinSYYgjsqwjKFMmGEghBJ0N2qXdE/qvTv1+zBQEvRZUlKAQjNpPllRAm9GCbwp9Uq/nw4DpQTGlPc6SiGSlMyYEiVKlChRokSJEiVKlChRosRs+f8BXRU2DsMsRGkAAAAldEVYdGRhdGU6Y3JlYXRlADIwMTQtMTAtMTFUMDA6Mjc6MjQrMDI6MDDMouQGAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDE0LTEwLTExVDAwOjI3OjI0KzAyOjAwvf9cugAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAAASUVORK5CYII=\u0026#39;, \u0026#39;icon_uri\u0026#39;: u\u0026#39;http://scrapy.org/favicons/favicon-192x192.png\u0026#39; } Now that I have a fairly reliable method for fetching icon data, in terms of getting all the data I need for the definitive bookmarks JSON, I am about ready to put everything together into a dirty fire-and-forget script!\nJoining things up So, here is a function that basically [does the bookmark fetching](#does the bookmark fetching) and icon fetching. Simple and not too resilient but should just about get the job done. It makes use of the get_icon() [method I defined earlier](#method I defined earlier).\ndef get_entry(bookmark): \u0026#34;\u0026#34;\u0026#34;Parses an individual del.icio.us entry\u0026#34;\u0026#34;\u0026#34; entry = {} # Locate and set Title title = bookmark.find_all(\u0026#39;div\u0026#39;, class_=\u0026#39;articleTitlePan\u0026#39;)[0] entry[\u0026#39;title\u0026#39;] = title.a.attrs[\u0026#39;title\u0026#39;] # Locate and set bookmark URL href = bookmark.find_all(\u0026#39;div\u0026#39;, class_=\u0026#39;articleInfoPan\u0026#39;)[0].find_all(\u0026#39;p\u0026#39;)[0] entry[\u0026#39;url\u0026#39;] = href.a.attrs[\u0026#39;href\u0026#39;] logging.debug(\u0026#39;Getting URL: %s\u0026#39;, entry[\u0026#39;url\u0026#39;]) # Parse the URL - we need this to get the Icon parsed = urlparse(entry[\u0026#39;url\u0026#39;]) # Get the icon URL and base64 encoded icon image icon = get_icon(u\u0026#39;{}://{}\u0026#39;.format(parsed.scheme, parsed.netloc)) entry[\u0026#39;icon_uri\u0026#39;] = icon[\u0026#39;icon_uri\u0026#39;] entry[\u0026#39;icon_data\u0026#39;] = icon[\u0026#39;icon_data\u0026#39;] # Get the date the bookmark was added, both in readable and epoch format entry[\u0026#39;add_date\u0026#39;] = str(datetime.datetime.fromtimestamp(int(bookmark.attrs[\u0026#39;date\u0026#39;]))) entry[\u0026#39;add_epoch\u0026#39;] = bookmark.attrs[\u0026#39;date\u0026#39;] # Just set the bookmark to public entry[\u0026#39;private\u0026#39;] = 0 # Get all the tags (if there are any) try: tags = bookmark.find_all(\u0026#39;ul\u0026#39;, class_=\u0026#39;tagName\u0026#39;)[0].find_all(\u0026#39;li\u0026#39;) entry[\u0026#39;tags\u0026#39;] = [tag.a.text for tag in tags] except IndexError: entry[\u0026#39;tags\u0026#39;] = [\u0026#39;none\u0026#39;] # Get the comment (if there is one, else use the title) comment = None for brief_text in [l for l in bookmark.find(\u0026#39;div\u0026#39;, class_=\u0026#39;thumbTBriefTxt\u0026#39;).children]: try: comment = brief_text.p.contents[0] except AttributeError: continue if comment is not None: entry[\u0026#39;comment\u0026#39;] = comment else: entry[\u0026#39;comment\u0026#39;] = entry[\u0026#39;title\u0026#39;] # We are done return entry The above code is fairly well commented so there is not much to be said about it, so without any further ado, here is some test output:\nlink = json.dumps(get_entry(bookmarks[2])) print link DEBUG:Getting URL: http://www.martyncurrey.com/arduino-nano-as-an-isp-programmer/ DEBUG:Icon URL: http://www.martyncurrey.com/wp-content/uploads/2016/10/favicon2.ico {\u0026#34;comment\u0026#34;: \u0026#34;Arduino Nano as an ISP Programmer | Martyn Currey\u0026#34;, \u0026#34;add_epoch\u0026#34;: \u0026#34;1430895155\u0026#34;, \u0026#34;title\u0026#34;: \u0026#34;Arduino Nano as an ISP Programmer | Martyn Currey\u0026#34;, \u0026#34;url\u0026#34;: \u0026#34;http://www.martyncurrey.com/arduino-nano-as-an-isp-programmer/\u0026#34;, \u0026#34;tags\u0026#34;: [\u0026#34;arduino\u0026#34;, \u0026#34;programming\u0026#34;], \u0026#34;icon_uri\u0026#34;: \u0026#34;http://www.martyncurrey.com/wp-content/uploads/2016/10/favicon2.ico\u0026#34;, \u0026#34;private\u0026#34;: 0, \u0026#34;add_date\u0026#34;: \u0026#34;2015-05-06 06:52:35\u0026#34;, \u0026#34;icon_data\u0026#34;: \u0026#34;data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAJwOAACcDgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAO5YAECFMQB/AAAAAAAAAAAAAAAA8FgAQIozAH9kJQAgAAAAAAAAAACuQAB/YCMAv0AYAJ9AGAAgAAAAAAAAAAD/dyd/jDQA/wAAAAAAAAAAAAAAAP91JH+fOwD/ViAAQAAAAAD/YgZ/6VYA/7xFAP96LQD/QBgA7wAAAAAAAAAA/5ZYf65AAP8AAAAAAAAAAAAAAAD/hT1/xkkA/3QrAED/XwEQ/4Q87/91Jf//dSXf/3Ym36E7AP8AAAAAAAAAAP+eZn/ISgD/AAAAAJw6AHDnVQAw/3Mif89MAP+fOwBA/3QjYP+DO//eUgC/AAAAAP+UVhD/YgbPAAAAAAAAAAD/nmV/5FQA/wAAAACzQgD/w0gAf/dbAH/LSwD/xUkAQP93J3/vWAD/gzAAUAAAAAAAAAAA/3cnMAAAAAAAAAAA/55mf/9hBP/XTwBA4FMA/8JIAL/YUAB/xUkA/+NUAED/cyK/pz4A/1EeABAAAAAAAAAAAAAAAAAAAAAAAAAAAP+han//ciD/9lsAcP9hBP/aUADvykoAf7hEAP/yWQBA/3Miv4UxAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/o25//34z//lcAJ//bhnf/2cO/9dPAJ+jPAD/7lgAQP92Jr+ALwD/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/6Zyf/+FPf/fUgDP/20Yf/+NSu//XwLPkzYA/+FTAED/fjK/kTUA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP+pdn//ikX/tkMA//9gA0D/p3Ov/4E3/481AP/OTABA/4lEj79GAP9AGABAAAAAAAAAAACzQgAQAAAAAAAAAAD/rHt//4tI/5Y3AP/rVwAQ/7OGf/+bYP+SNgD/sUEAQP+WWGD/Zw//QBgAnwAAAAAAAAAAYCMArwAAAAAAAAAA/6+Af/+KRf+GMQDPAAAAAP+4jkD/qnn/oTsA/5g4AED/nmYg/6Fq/7lEAP9gIwCPbSgAr0AYAP8AAAAAAAAAAP+4jn//j03/njoAnwAAAAAAAAAA/7iP/9dPAP+lPQBAAAAAAP+/mp//omz//2wW/9FNAP92KwD/AAAAAAAAAAD/uI5g/5JTv9pQAGAAAAAAAAAAAP+3jZ//bBa/108AMAAAAAD/vZYQ/7uUn/+gaP//bhrfyEoAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP/zAADe4QAA3sEAAN7NAADW3wAA0p8AANKfAADAnwAAyJ8AAMifAADMzQA+zMEBEszBDQjc4/kE///yBA==\u0026#34;} And there it is. I ran the code against my own bookmarks in del.icio.us and, after a few false starts where their site was unreachable (no reason given by the guys at del.icio.us), I managed to safely retrieve all my data.\nAnd now for a couple of other things which are connected with what I have been doing whilst working on this bookmark business.\nAside - Outputting the Results At some point I wanted to incrementally write results to a file in JSON format. Occasionally a data fetch would fail and the entire process would end resulting in a somewhat truncated file. So, I decided to make a three part approach:\nWrite an opening list delimiter. In a loop, write out fetched JSON dictionary objects, skipping failures. Write a closing list delimiter Here\u0026rsquo;s what I came up with; rough and ready, but worked quite happily.\nfilename = \u0026#39;/testing/links.json\u0026#39; # Write starting character with open(filename, \u0026#39;w\u0026#39;) as start: start.write(\u0026#39;[\u0026#39;) # Here you might have something that fetched a page at a time and created # a list of bookmarks per page - then interate through that list next. # For each, blah, append links - add a comma after each bookmark except for the last bookmark # Here\u0026#39;s hoping that the last list item is not a failure. for idx, bookmark in enumerate(bookmarks): with codecs.open(filename, \u0026#39;a\u0026#39;, encoding=\u0026#39;utf-8\u0026#39;) as incr: incr.write(json.dumps(get_entry(bookmark))) if idx \u0026lt; len(bookmarks) - 1: incr.write(\u0026#39;,\u0026#39;) # Done, we write the ending character with open(filename, \u0026#39;a\u0026#39;) as end: end.write(\u0026#39;]\u0026#39;) Aside - Making an importable bookmarks HTML file Here are some brief notes regarding turning JSON entries into bookmark HTML which I can import into a browser.\nBelow is a line of desired link HTML:\n\u0026lt;DT\u0026gt;\u0026lt;A HREF=\u0026#34;http://www.something.com/some-article/\u0026#34; ADD_DATE=\u0026#34;1395356195\u0026#34; PRIVATE=\u0026#34;0\u0026#34; TAGS=\u0026#34;stuff,things\u0026#34; ICON_URI=\u0026#34;http://www.something.com/favicon2.ico\u0026#34; ICON=\u0026#34;data:image/x-icon;base64,[...]\u0026gt;Article Title\u0026lt;/A\u0026gt; In the code below, link is a JSON dict in the [same format as I generated earlier](#same format as I generated earlier).\nbm_load = json.loads(link) outlink = u\u0026#39;\u0026lt;DT\u0026gt;\u0026lt;A HREF=\u0026#34;{}\u0026#34; ADD_DATE=\u0026#34;{}\u0026#34; PRIVATE=\u0026#34;0\u0026#34; TAGS=\u0026#34;{}\u0026#34; ICON_URI=\u0026#34;{}\u0026#34; ICON=\u0026#34;{}\u0026gt;{}\u0026lt;/A\u0026gt;\u0026#39;.format( bm_load[\u0026#39;url\u0026#39;], bm_load[\u0026#39;add_epoch\u0026#39;], \u0026#39;,\u0026#39;.join(bm_load[\u0026#39;tags\u0026#39;]), bm_load[\u0026#39;icon_uri\u0026#39;], bm_load[\u0026#39;icon_data\u0026#39;], bm_load[\u0026#39;comment\u0026#39;] ) print outlink And here we are:\nu\u0026#39;\u0026lt;DT\u0026gt;\u0026lt;A HREF=\u0026#34;http://www.martyncurrey.com/arduino-nano-as-an-isp-programmer/\u0026#34; ADD_DATE=\u0026#34;1430895155\u0026#34; PRIVATE=\u0026#34;0\u0026#34; TAGS=\u0026#34;arduino,programming\u0026#34; ICON_URI=\u0026#34;http://www.martyncurrey.com/wp-content/uploads/2016/10/favicon2.ico\u0026#34; ICON=\u0026#34;data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAJwOAACcDgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAO5YAECFMQB/AAAAAAAAAAAAAAAA8FgAQIozAH9kJQAgAAAAAAAAAACuQAB/YCMAv0AYAJ9AGAAgAAAAAAAAAAD/dyd/jDQA/wAAAAAAAAAAAAAAAP91JH+fOwD/ViAAQAAAAAD/YgZ/6VYA/7xFAP96LQD/QBgA7wAAAAAAAAAA/5ZYf65AAP8AAAAAAAAAAAAAAAD/hT1/xkkA/3QrAED/XwEQ/4Q87/91Jf//dSXf/3Ym36E7AP8AAAAAAAAAAP+eZn/ISgD/AAAAAJw6AHDnVQAw/3Mif89MAP+fOwBA/3QjYP+DO//eUgC/AAAAAP+UVhD/YgbPAAAAAAAAAAD/nmV/5FQA/wAAAACzQgD/w0gAf/dbAH/LSwD/xUkAQP93J3/vWAD/gzAAUAAAAAAAAAAA/3cnMAAAAAAAAAAA/55mf/9hBP/XTwBA4FMA/8JIAL/YUAB/xUkA/+NUAED/cyK/pz4A/1EeABAAAAAAAAAAAAAAAAAAAAAAAAAAAP+han//ciD/9lsAcP9hBP/aUADvykoAf7hEAP/yWQBA/3Miv4UxAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/o25//34z//lcAJ//bhnf/2cO/9dPAJ+jPAD/7lgAQP92Jr+ALwD/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/6Zyf/+FPf/fUgDP/20Yf/+NSu//XwLPkzYA/+FTAED/fjK/kTUA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP+pdn//ikX/tkMA//9gA0D/p3Ov/4E3/481AP/OTABA/4lEj79GAP9AGABAAAAAAAAAAACzQgAQAAAAAAAAAAD/rHt//4tI/5Y3AP/rVwAQ/7OGf/+bYP+SNgD/sUEAQP+WWGD/Zw//QBgAnwAAAAAAAAAAYCMArwAAAAAAAAAA/6+Af/+KRf+GMQDPAAAAAP+4jkD/qnn/oTsA/5g4AED/nmYg/6Fq/7lEAP9gIwCPbSgAr0AYAP8AAAAAAAAAAP+4jn//j03/njoAnwAAAAAAAAAA/7iP/9dPAP+lPQBAAAAAAP+/mp//omz//2wW/9FNAP92KwD/AAAAAAAAAAD/uI5g/5JTv9pQAGAAAAAAAAAAAP+3jZ//bBa/108AMAAAAAD/vZYQ/7uUn/+gaP//bhrfyEoAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP/zAADe4QAA3sEAAN7NAADW3wAA0p8AANKfAADAnwAAyJ8AAMifAADMzQA+zMEBEszBDQjc4/kE///yBA==\u0026gt;Arduino Nano as an ISP Programmer | Martyn Currey\u0026lt;/A\u0026gt;\u0026#39; Finally, writing all the links out to a file as HTML. Open and read the bookmarks data JSON file; for each entry, append a formatted piece of HTML:\nimport codecs bookmarks = [] with codecs.open(\u0026#39;/testing/links.json\u0026#39;, \u0026#39;r\u0026#39;, encoding=\u0026#39;utf8\u0026#39;) as rfh: for bookmark in json.loads(rfh.read()): bookmarks.append(u\u0026#39;\u0026lt;DT\u0026gt;\u0026lt;A HREF=\u0026#34;{}\u0026#34; ADD_DATE=\u0026#34;{}\u0026#34; PRIVATE=\u0026#34;0\u0026#34; TAGS=\u0026#34;{}\u0026#34; ICON_URI=\u0026#34;{}\u0026#34; ICON=\u0026#34;{}\u0026#34;\u0026gt;{}\u0026lt;/A\u0026gt;\\n\u0026#39;.format( bookmark[\u0026#39;url\u0026#39;], bookmark[\u0026#39;add_epoch\u0026#39;], \u0026#39;,\u0026#39;.join(bookmark[\u0026#39;tags\u0026#39;]), bookmark[\u0026#39;icon_uri\u0026#39;], bookmark[\u0026#39;icon_data\u0026#39;], bookmark[\u0026#39;comment\u0026#39;] )) Create a suitable header:\nheader = u\u0026#34;\u0026#34;\u0026#34; \u0026lt;!DOCTYPE NETSCAPE-Bookmark-file-1\u0026gt; \u0026lt;META HTTP-EQUIV=\u0026#34;Content-Type\u0026#34; CONTENT=\u0026#34;text/html; charset=UTF-8\u0026#34;\u0026gt; \u0026lt;TITLE\u0026gt;Bookmarks\u0026lt;/TITLE\u0026gt; \u0026lt;H1\u0026gt;Bookmarks\u0026lt;/H1\u0026gt; \u0026lt;DL\u0026gt;\u0026lt;p\u0026gt; \u0026#34;\u0026#34;\u0026#34; Now write the whole lot out to a file ready for consumption by a browser:\nwith codecs.open(\u0026#39;/testing/links.html\u0026#39;, \u0026#39;w\u0026#39;, encoding=\u0026#39;utf8\u0026#39;) as wfh: wfh.write(header) for line in bookmarks: wfh.write(line) With that, we are done. Until next time.\n","permalink":"https://chrisramsay.co.uk/posts/2017/02/getting-bookmark-data-from-delicious/","summary":"\u003ch2 id=\"delicious-link-fetching\"\u003edel.icio.us Link Fetching\u003c/h2\u003e\n\u003cp\u003eBelow is an outline for building a rough \u0026amp; ready script to get \u003ca href=\"http://del.icio.us\"\u003edel.icio.us\u003c/a\u003e bookmark data - a script very much driven by necessity.\u003c/p\u003e\n\u003ch3 id=\"rationale\"\u003eRationale\u003c/h3\u003e\n\u003cp\u003eWhy am I doing this? Well, I\u0026rsquo;ve been bookmarking links using del.icio.us since March 2009 and have, very slowly, built up a steady collection of over 900 bookmarks, most of which are probably still quite useful.\u003c/p\u003e\n\u003cp\u003eThe ups and downs of social bookmarking service del.icio.us are \u003ca href=\"https://sixtwothree.org/posts/homesteading-a-decades-worth-of-shared-links\"\u003every well documented here\u003c/a\u003e, so I won\u0026rsquo;t go into all the details, but suffice to say that reading the aforementioned article gave me all the impetus I needed to go ahead and \u003cem\u003ehomestead\u003c/em\u003e my list of bookmarks. Simple, I thought.\u003c/p\u003e\n\u003cp\u003eThen came the snag\u0026hellip;\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eWe\u0026rsquo;re sorry, but due to heavy load on our database we are no longer able to offer an export function. Our engineers are working on this and we will restore it as soon as possible.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eGreat\u0026hellip; what now? Time to write a script of course. I\u0026rsquo;ve initially created this article using a \u003ca href=\"http://jupyter.org/\"\u003eJupyter Notebook\u003c/a\u003e which may, I hope, explain the recipe feel to the writing.\u003c/p\u003e\n\u003ch3 id=\"the-plan\"\u003eThe Plan\u003c/h3\u003e\n\u003cp\u003eWell, initially the plan was just to grab the bookmarks by scraping my del.icio.us pages from beginning to end and turn that data straight into some form of bookmark HTML. Thinking about it more, I decided to opt for a JSON data format; that way I could easily turn the data to any format I like at my leisure.\u003c/p\u003e","title":"Getting Bookmark Data from del.icio.us"},{"content":"Latest News I received some good news this week: my Show Source plugin, the one inspired by Sphinx, has had its pull request accepted into the getpelican/pelican-plugins repo. This means that the plugin becomes a part of the standard Pelican plugin canon.\nMy next task will be to make a small addition, and hence pull request, to the developers of the theme I use with this site (pelican-bootstrap3) to accomodate a couple of small template changes to support Show Source automatically.\nThe officially accepted version of show source is available now right here.\nEnjoy!\n","permalink":"https://chrisramsay.co.uk/posts/2016/12/show-source-plugin-update/","summary":"\u003ch2 id=\"latest-news\"\u003eLatest News\u003c/h2\u003e\n\u003cp\u003eI received some good news this week: my \u003ca href=\"https://github.com/chrisramsay/pelican-plugins/tree/feature/show_source/show_source\"\u003eShow Source\u003c/a\u003e plugin, the one inspired by \u003ca href=\"http://www.sphinx-doc.org/\"\u003eSphinx\u003c/a\u003e, has had its pull request accepted into the \u003ca href=\"https://github.com/getpelican/pelican-plugins\"\u003egetpelican/pelican-plugins repo\u003c/a\u003e. This means that the plugin becomes a part of the standard \u003ca href=\"https://github.com/getpelican/pelican\"\u003ePelican\u003c/a\u003e plugin canon.\u003c/p\u003e\n\u003cp\u003eMy next task will be to make a small addition, and hence pull request, to the developers of the theme I use with this site (\u003ca href=\"https://github.com/getpelican/pelican-themes/tree/master/pelican-bootstrap3\"\u003epelican-bootstrap3\u003c/a\u003e) to accomodate a couple of small template changes to support \u003ca href=\"https://github.com/chrisramsay/pelican-plugins/tree/feature/show_source/show_source\"\u003eShow Source\u003c/a\u003e automatically.\u003c/p\u003e\n\u003cp\u003eThe officially accepted version of show source \u003ca href=\"https://github.com/getpelican/pelican-plugins/tree/master/show_source\"\u003eis available now right here\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eEnjoy!\u003c/p\u003e","title":"Show Source Plugin Update"},{"content":"Rationale I have been doing a little work with AWS data pipeline recently for undertaking ETL tasks at work. AWS data pipeline handles data driven workflows called pipelines. The data pipelines take care of scheduling, data depenedencies, data sources and destinations in a nicely managed workflow. My tasks take batch datasets from SQL databases, processing and loading those datasets into S3 buckets, then import into a Redshift reporting database.\nSeeing that production database structures are frequently updated, those changes need to be reflected in the reporting backend service. Now for a couple of years I have struggled on with Amazon\u0026rsquo;s web based data pipeline architect to manage those changes to the pipeline. This has been an onerous task, as the architect does not really lend itself very well to the task of managing a large set of pipelines. Here begins a little tale of delving into the AWS data pipeline API to find another way.\nThe original problem I was presented with when I started this project was that many parallel processes were written into several large pipelines which carried out an entire set of nightly ETL tasks. My job was thus to split these monsters into many smaller, and more manageable pieces. I decided to start by exporting the whole pipeline definition as a set of JSON definition files and make some sense of it, but no luck. The whole thing was so intertwined, cluttered and hard to make sense of that I decided to abandon ship and try another way.\nNew Beginning I started by creating a brand new base pipeline definition, using the architect, that would be common to all processes and exported that as a new baseline definition JSON for all the other pipelines.\nI then wrote a Python tool to bring together a set of config parameters in JSON format, containing details such as database passwords, S3 bucket locations and so on, and a series of JSON pipeline definition files. This was so that my pipeline template files could be stored in a Git repo without the accompanying sensitive data.\nThe steps for creating a new pipeline then become quite simple:\nCreate a new set of parameters for the pipeline in the configuration file. Make a copy of the template file and amend as required for the process. Run the tool to output the JSON definition file as required by the architect. Build new pipeline within the architect tool importing the definition file created above. Moving on though, I really wanted to dump the clunky web-based architect tool altogether by making use of the data pipeline API.\nUsing the Data Pipeline API So the next step for me was to move from using the architect tool in the browser to using the data pipeline API. I initially thought that the pipeline definitions from the architect would be usable in the API, but no, the API needs definitions to be in a different format. [More on this later](#More on this later).\nBefore moving on, let us look at a typical JSON formatted definition as used by the architect.\nJSON Sample Just for this post, I have created a simple data pipeline sample. This is how it looks in the web based architect tool:\nBelow is the JSON export of the pipeline sample above:\n{ \u0026#34;objects\u0026#34;: [ { \u0026#34;*password\u0026#34;: \u0026#34;#{*myRDSPassword}\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;rds_mysql\u0026#34;, \u0026#34;jdbcProperties\u0026#34;: \u0026#34;allowMultiQueries=true\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;rds_mysql\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;RdsDatabase\u0026#34;, \u0026#34;rdsInstanceId\u0026#34;: \u0026#34;#{myRDSInstanceId}\u0026#34;, \u0026#34;username\u0026#34;: \u0026#34;#{myRDSUsername}\u0026#34; }, { \u0026#34;database\u0026#34;: { \u0026#34;ref\u0026#34;: \u0026#34;rds_mysql\u0026#34; }, \u0026#34;name\u0026#34;: \u0026#34;SourceRDSTable\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;SourceRDSTable\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;SqlDataNode\u0026#34;, \u0026#34;table\u0026#34;: \u0026#34;#{myRDSTableName}\u0026#34;, \u0026#34;selectQuery\u0026#34;: \u0026#34;select * from #{table}\u0026#34; }, { \u0026#34;instanceType\u0026#34;: \u0026#34;#{myEC2InstanceType}\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Ec2Instance\u0026#34;, \u0026#34;actionOnTaskFailure\u0026#34;: \u0026#34;terminate\u0026#34;, \u0026#34;securityGroups\u0026#34;: \u0026#34;#{myEc2RdsSecurityGrps}\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;Ec2Instance\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;Ec2Resource\u0026#34;, \u0026#34;terminateAfter\u0026#34;: \u0026#34;2 Hours\u0026#34; }, { \u0026#34;output\u0026#34;: { \u0026#34;ref\u0026#34;: \u0026#34;S3OutputLocation\u0026#34; }, \u0026#34;input\u0026#34;: { \u0026#34;ref\u0026#34;: \u0026#34;SourceRDSTable\u0026#34; }, \u0026#34;name\u0026#34;: \u0026#34;RDStoS3CopyActivity\u0026#34;, \u0026#34;runsOn\u0026#34;: { \u0026#34;ref\u0026#34;: \u0026#34;Ec2Instance\u0026#34; }, \u0026#34;id\u0026#34;: \u0026#34;RDStoS3CopyActivity\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;CopyActivity\u0026#34; }, { \u0026#34;failureAndRerunMode\u0026#34;: \u0026#34;CASCADE\u0026#34;, \u0026#34;resourceRole\u0026#34;: \u0026#34;DataPipelineDefaultResourceRole\u0026#34;, \u0026#34;role\u0026#34;: \u0026#34;DataPipelineDefaultRole\u0026#34;, \u0026#34;scheduleType\u0026#34;: \u0026#34;ONDEMAND\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Default\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;Default\u0026#34; }, { \u0026#34;directoryPath\u0026#34;: \u0026#34;#{myOutputS3Loc}/#{format(@scheduledStartTime, \u0026#39;YYYY-MM-dd-HH-mm-ss\u0026#39;)}\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;S3OutputLocation\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;S3OutputLocation\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;S3DataNode\u0026#34; } ], \u0026#34;parameters\u0026#34;: [ { \u0026#34;description\u0026#34;: \u0026#34;RDS MySQL password\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;*myRDSPassword\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;String\u0026#34; }, { \u0026#34;description\u0026#34;: \u0026#34;Output S3 folder\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;myOutputS3Loc\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;AWS::S3::ObjectKey\u0026#34; }, { \u0026#34;description\u0026#34;: \u0026#34;RDS MySQL username\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;myRDSUsername\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;String\u0026#34; }, { \u0026#34;description\u0026#34;: \u0026#34;RDS MySQL table name\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;myRDSTableName\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;String\u0026#34; }, { \u0026#34;description\u0026#34;: \u0026#34;Ec2 Security group(s)\u0026#34;, \u0026#34;isArray\u0026#34;: \u0026#34;true\u0026#34;, \u0026#34;optional\u0026#34;: \u0026#34;true\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;myEc2RdsSecurityGrps\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;String\u0026#34; }, { \u0026#34;default\u0026#34;: \u0026#34;r3.xlarge\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;EC2 instance type\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;myEC2InstanceType\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;String\u0026#34; }, { \u0026#34;description\u0026#34;: \u0026#34;RDS Instance ID\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;myRDSInstanceId\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;String\u0026#34; } ], \u0026#34;values\u0026#34;: { \u0026#34;myRDSInstanceId\u0026#34;: \u0026#34;database.abcde123456.eu-west-1.rds.amazonaws.com\u0026#34;, \u0026#34;myRDSUsername\u0026#34;: \u0026#34;rdsuser\u0026#34;, \u0026#34;myEC2InstanceType\u0026#34;: \u0026#34;r3.xlarge\u0026#34;, \u0026#34;myOutputS3Loc\u0026#34;: \u0026#34;s3://my-output-logs/\u0026#34;, \u0026#34;*myRDSPassword\u0026#34;: \u0026#34;rdspassword\u0026#34;, \u0026#34;myRDSTableName\u0026#34;: \u0026#34;rdstable\u0026#34; } } Now for even a slightly larger and more complex pipeline it does not take much imagination to see that this definition can start to look quite complex and unmanageable, both in the architect and the JSON file itself. This complexity was the driving force behind me creating many smaller pipelines.\nIf you were to import the above file (say it was saved as definition.json) into a Python console using the json library, the root level dict keys would show as objects, values and parameters:\nimport json out = json.load(\u0026#39;definition.json\u0026#39;) out.keys() #[u\u0026#39;objects\u0026#39;, u\u0026#39;values\u0026#39;, u\u0026#39;parameters\u0026#39;] The lists and dict these keys refer to will be required later, so please read on\u0026hellip;\nGetting to Work with the API I\u0026rsquo;ll basically step through what I did to get a pipeline into the AWS infrastructure.\nCreating a Connection Creating a new Pipeline Check for Success Insert JSON Here\u0026hellip; Putting the Definition Creating a Connection Firstly, create a client (the Docker container I use here keeps Amazon credentials as environment variables).\nimport boto3, os, pprint, uuid client = boto3.client( \u0026#39;datapipeline\u0026#39;, aws_access_key_id=os.environ.get(\u0026#39;AWS_ACCESS_KEY_ID\u0026#39;), aws_secret_access_key=os.environ.get(\u0026#39;AWS_SECRET_ACCESS_KEY\u0026#39;), region_name=os.environ.get(\u0026#39;AWS_DEFAULT_REGION\u0026#39;), ) Just a quick test; can I list the pipelines I already have?\npprint.pprint(client.list_pipelines()) {\u0026#39;ResponseMetadata\u0026#39;: {\u0026#39;HTTPHeaders\u0026#39;: {\u0026#39;content-length\u0026#39;: \u0026#39;297\u0026#39;, \u0026#39;content-type\u0026#39;: \u0026#39;application/x-amz-json-1.1\u0026#39;, \u0026#39;date\u0026#39;: \u0026#39;Wed, 16 Nov 2016 18:39:00 GMT\u0026#39;, \u0026#39;x-amzn-requestid\u0026#39;: \u0026#39;f0fad3de-ac2b-11e6-8797-6578b9a77e01\u0026#39;}, \u0026#39;HTTPStatusCode\u0026#39;: 200, \u0026#39;RequestId\u0026#39;: \u0026#39;f0fad3de-ac2b-11e6-8797-6578b9a77e01\u0026#39;, \u0026#39;RetryAttempts\u0026#39;: 0}, u\u0026#39;hasMoreResults\u0026#39;: False, u\u0026#39;pipelineIdList\u0026#39;: [{u\u0026#39;id\u0026#39;: u\u0026#39;df-10264073JXS47Y8NS12O\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;architect-01-s3-to-redshift\u0026#39;}, {u\u0026#39;id\u0026#39;: u\u0026#39;df-07270122JR3GK3NC7O8Z\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;architect-02-s3-to-redshift\u0026#39;}, {u\u0026#39;id\u0026#39;: u\u0026#39;df-06847673DGG6UGYJSEG7\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;architect-03-s3-to-rds\u0026#39;}, {u\u0026#39;id\u0026#39;: u\u0026#39;df-0315779437OOQ2TYL69\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;architect-04-s3-to-rds\u0026#39;}, {...}]} Creating a new Pipeline Create a new pipeline.\ncreate = client.create_pipeline(name=\u0026#39;api-test-1\u0026#39;, uniqueId=str(uuid.uuid1())) pprint.pprint(create) {\u0026#39;ResponseMetadata\u0026#39;: {\u0026#39;HTTPHeaders\u0026#39;: {\u0026#39;content-length\u0026#39;: \u0026#39;40\u0026#39;, \u0026#39;content-type\u0026#39;: \u0026#39;application/x-amz-json-1.1\u0026#39;, \u0026#39;date\u0026#39;: \u0026#39;Wed, 16 Nov 2016 18:40:51 GMT\u0026#39;, \u0026#39;x-amzn-requestid\u0026#39;: \u0026#39;33581d9a-ac2c-11e6-8797-6578b9a77e01\u0026#39;}, \u0026#39;HTTPStatusCode\u0026#39;: 200, \u0026#39;RequestId\u0026#39;: \u0026#39;33581d9a-ac2c-11e6-8797-6578b9a77e01\u0026#39;, \u0026#39;RetryAttempts\u0026#39;: 0}, u\u0026#39;pipelineId\u0026#39;: u\u0026#39;df-086679538TXJJEBBT0JI\u0026#39;} The pipelineId value from the create dictionary will be the reference required in the next step when putting the pipeline definition.\nCheck for Success Let us list the pipelines again to see that our new pipeline was created successfully:\npprint.pprint(client.list_pipelines()) {\u0026#39;ResponseMetadata\u0026#39;: {\u0026#39;HTTPHeaders\u0026#39;: {\u0026#39;content-length\u0026#39;: \u0026#39;367\u0026#39;, \u0026#39;content-type\u0026#39;: \u0026#39;application/x-amz-json-1.1\u0026#39;, \u0026#39;date\u0026#39;: \u0026#39;Wed, 16 Nov 2016 18:49:25 GMT\u0026#39;, \u0026#39;x-amzn-requestid\u0026#39;: \u0026#39;65a3ad62-ac2d-11e6-8ebf-e74a8b25a45f\u0026#39;}, \u0026#39;HTTPStatusCode\u0026#39;: 200, \u0026#39;RequestId\u0026#39;: \u0026#39;65a3ad62-ac2d-11e6-8ebf-e74a8b25a45f\u0026#39;, \u0026#39;RetryAttempts\u0026#39;: 0}, u\u0026#39;hasMoreResults\u0026#39;: False, u\u0026#39;pipelineIdList\u0026#39;: [{u\u0026#39;id\u0026#39;: u\u0026#39;df-10264073JXS47Y8NS12O\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;architect-01-s3-to-redshift\u0026#39;}, {u\u0026#39;id\u0026#39;: u\u0026#39;df-07270122JR3GK3NC7O8Z\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;architect-02-s3-to-redshift\u0026#39;}, {u\u0026#39;id\u0026#39;: u\u0026#39;df-06847673DGG6UGYJSEG7\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;architect-03-s3-to-rds\u0026#39;}, {u\u0026#39;id\u0026#39;: u\u0026#39;df-0315779437OOQ2TYL69\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;architect-04-s3-to-rds\u0026#39;}, {u\u0026#39;id\u0026#39;: u\u0026#39;df-086679538TXJJEBBT0JI\u0026#39;, u\u0026#39;name\u0026#39;: u\u0026#39;api-test-1\u0026#39;}, {...}]} And there it is, named api-test-1 as requested. We can go ahead and get more information about our new pipeline now by using the describe_pipelines call:\ndescription = client.describe_pipelines(pipelineIds=[\u0026#39;df-086679538TXJJEBBT0JI\u0026#39;]) pprint.pprint(description) {\u0026#39;ResponseMetadata\u0026#39;: {\u0026#39;HTTPHeaders\u0026#39;: {\u0026#39;content-length\u0026#39;: \u0026#39;599\u0026#39;, \u0026#39;content-type\u0026#39;: \u0026#39;application/x-amz-json-1.1\u0026#39;, \u0026#39;date\u0026#39;: \u0026#39;Wed, 16 Nov 2016 18:55:19 GMT\u0026#39;, \u0026#39;x-amzn-requestid\u0026#39;: \u0026#39;66e79486-b1ad-11e6-8249-23409048a58c\u0026#39;}, \u0026#39;HTTPStatusCode\u0026#39;: 200, \u0026#39;RequestId\u0026#39;: \u0026#39;66e79486-b1ad-11e6-8249-23409048a58c\u0026#39;, \u0026#39;RetryAttempts\u0026#39;: 0}, u\u0026#39;pipelineDescriptionList\u0026#39;: [{u\u0026#39;fields\u0026#39;: [{u\u0026#39;key\u0026#39;: u\u0026#39;pipelineCreator\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;AIDAJV2YZC22FODOTLB54\u0026#39;}, {u\u0026#39;key\u0026#39;: u\u0026#39;@creationTime\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;2016-11-16T18:49:25\u0026#39;}, {u\u0026#39;key\u0026#39;: u\u0026#39;name\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;api-test-1\u0026#39;}, {u\u0026#39;key\u0026#39;: u\u0026#39;@sphere\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;PIPELINE\u0026#39;}, {u\u0026#39;key\u0026#39;: u\u0026#39;@id\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;df-086679538TXJJEBBT0JI\u0026#39;}, {u\u0026#39;key\u0026#39;: u\u0026#39;@pipelineState\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;PENDING\u0026#39;}, {u\u0026#39;key\u0026#39;: u\u0026#39;@accountId\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;816026221619\u0026#39;}, {u\u0026#39;key\u0026#39;: u\u0026#39;uniqueId\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;70c30b52-b1ac-11e6-85e2-0242ac110003\u0026#39;}, {u\u0026#39;key\u0026#39;: u\u0026#39;@userId\u0026#39;, u\u0026#39;stringValue\u0026#39;: u\u0026#39;AIDAJV2YZC22FODOTLB54\u0026#39;}], u\u0026#39;name\u0026#39;: u\u0026#39;api_test_2\u0026#39;, u\u0026#39;pipelineId\u0026#39;: u\u0026#39;df-086679538TXJJEBBT0JI\u0026#39;, u\u0026#39;tags\u0026#39;: []}]} You may notice from the above return data that the pipeline seems okay; it is in a pending state but not yet ready to go. That is because it does not yet have a definition attached. That\u0026rsquo;s next\u0026hellip;\nInsert JSON Here\u0026hellip; Brilliant, now we have a pipeline ready to put a definition into. This is the moment where I grab the JSON definition and push that into the API request, or so I thought before diving deeper into PutPipelineDefinition documentation.\nEssentially, the data pipeline architect uses a different format of JSON to from the data pipeline API for handling the import and export of definition files.\nFirst, I load an example definition file:\nimport json definition = json.load(open(\u0026#39;definition.json\u0026#39;, \u0026#39;r\u0026#39;)) If we extract the values dict from the JSON. we get something like the below, which is quite acceptable for the architect:\npprint.pprint(definition.get(\u0026#39;values\u0026#39;)) {u\u0026#39;*myRDSPassword\u0026#39;: u\u0026#39;rdspassword\u0026#39;, u\u0026#39;myEC2InstanceType\u0026#39;: u\u0026#39;r3.xlarge\u0026#39;, u\u0026#39;myOutputS3Loc\u0026#39;: u\u0026#39;s3://my-output-logs/\u0026#39;, u\u0026#39;myRDSInstanceId\u0026#39;: u\u0026#39;database.abcde123456.eu-west-1.rds.amazonaws.com\u0026#39;, u\u0026#39;myRDSTableName\u0026#39;: u\u0026#39;rdstable\u0026#39;, u\u0026#39;myRDSUsername\u0026#39;: u\u0026#39;rdsuser\u0026#39;} What we need for the API is formatted like the below; quite different from what is suitable for the architect. Here the key value pairs of the above dict have been split into separate dictionary objects of id and stringValue pairings within a list:\npprint.pprint(parameterValues) [{\u0026#39;id\u0026#39;: u\u0026#39;myEC2InstanceType\u0026#39;, \u0026#39;stringValue\u0026#39;: u\u0026#39;r3.xlarge\u0026#39;}, {\u0026#39;id\u0026#39;: u\u0026#39;myRDSTableName\u0026#39;, \u0026#39;stringValue\u0026#39;: u\u0026#39;rdstable\u0026#39;}, {\u0026#39;id\u0026#39;: u\u0026#39;myOutputS3Loc\u0026#39;, \u0026#39;stringValue\u0026#39;: u\u0026#39;s3://my-output-logs/\u0026#39;}, {\u0026#39;id\u0026#39;: u\u0026#39;myRDSInstanceId\u0026#39;, \u0026#39;stringValue\u0026#39;: u\u0026#39;database.abcde123456.eu-west-1.rds.amazonaws.com\u0026#39;}, {\u0026#39;id\u0026#39;: u\u0026#39;*myRDSPassword\u0026#39;, \u0026#39;stringValue\u0026#39;: u\u0026#39;rdspassword\u0026#39;}, {\u0026#39;id\u0026#39;: u\u0026#39;myRDSUsername\u0026#39;, \u0026#39;stringValue\u0026#39;: u\u0026#39;rdsuser\u0026#39;}] Upon this discovery I momentarily considered writing some code to convert a definition into API objects, but then I thought, there must be another way. After all, most of what AWS provides as functionality on their console web site is built upon interactions with their API, so I reasoned that there must already be code out there that does this conversion from the web based architect front end into the data pipeline API.\nIt turns out that the awscli library presents a handy data pipeline customisations library enabling definition to API objects translation, and much more! See the code here in github for more information.\nSo, remember the lists and dict the objects, values and parameters keys referred to in the definition JSON [from earlier?](#from earlier?) Well, what needs to be done is to convert each of those into a format parseable by the API; this is where the customizations library comes into its own:\nimport awscli.customizations.datapipeline.translator as trans pipelineObjects = trans.definition_to_api_objects(definition) parameterObjects = trans.definition_to_api_parameters(definition) parameterValues = trans.definition_to_parameter_values(definition) client.put_pipeline_definition will need the three lists from the code above, and the pipelineId from the create response earlier in order to succeed.\nPutting the Definition Once we have the objects, values and parameters in the right shape, and a pipeline ID, we can make a put pipeline definition request to the API. This will finish off the pipeline creation process, as the pipeline will be installed complete with the definition that tells it what to do:\nresponse = client.put_pipeline_definition( pipelineId=create[\u0026#39;pipelineId\u0026#39;], pipelineObjects=pipelineObjects, parameterObjects=parameterObjects, parameterValues=parameterValues ) If the pipeline definition request is successful, then we should see something like this in the response object:\npprint.pprint(response) {\u0026#39;ResponseMetadata\u0026#39;: {\u0026#39;HTTPHeaders\u0026#39;: {\u0026#39;content-length\u0026#39;: \u0026#39;204\u0026#39;, \u0026#39;content-type\u0026#39;: \u0026#39;application/x-amz-json-1.1\u0026#39;, \u0026#39;date\u0026#39;: \u0026#39;Thu, 17 Nov 2016 11:55:27 GMT\u0026#39;, \u0026#39;x-amzn-requestid\u0026#39;: \u0026#39;bb12cfac-acbc-11e6-8442-d1754b82fda0\u0026#39;}, \u0026#39;HTTPStatusCode\u0026#39;: 200, \u0026#39;RequestId\u0026#39;: \u0026#39;bb12cfac-acbc-11e6-8442-d1754b82fda0\u0026#39;, \u0026#39;RetryAttempts\u0026#39;: 0}, u\u0026#39;errored\u0026#39;: False, u\u0026#39;validationErrors\u0026#39;: [], u\u0026#39;validationWarnings\u0026#39;: [{u\u0026#39;id\u0026#39;: u\u0026#39;Default\u0026#39;, u\u0026#39;warnings\u0026#39;: [u\u0026#34;\u0026#39;pipelineLogUri\u0026#39;is missing. It is recommended to set this value on Default object for better troubleshooting.\u0026#34;]}]} You may have spotted the validationWarnings key: the definition is validated by the API during creation. Clearly any fatal errors would result in a rejection, but anything that is a warning is merely notified in the response. It is worth noting that some warnings do not seem to be shown, such as incorrectly specifying a terminateAfter parameter.\nConclusion So there it is. Not that difficult to go from managing an unwieldy set of pipelines in a web browser front end to a small set of JSON files and a handful of API calls. There is also the nice bonus of having all the data pipelines under version control with the sensitive data kept separately.\n","permalink":"https://chrisramsay.co.uk/posts/2016/11/creating-aws-data-pipelines-with-boto3-and-json/","summary":"\u003ch2 id=\"rationale\"\u003eRationale\u003c/h2\u003e\n\u003cp\u003eI have been doing a little work with \u003ca href=\"https://aws.amazon.com/documentation/data-pipeline/\"\u003eAWS data pipeline\u003c/a\u003e recently for undertaking \u003ca href=\"https://en.wikipedia.org/wiki/Extract,_transform,_load\"\u003eETL\u003c/a\u003e tasks at work. AWS data pipeline handles data driven workflows called pipelines. The data pipelines take care of scheduling, data depenedencies, data sources and destinations in a nicely managed workflow. My tasks take batch datasets from SQL databases, processing and loading those datasets into S3 buckets, then import into a Redshift reporting database.\u003c/p\u003e\n\u003cp\u003eSeeing that production database structures are frequently updated, those changes need to be reflected in the reporting backend service. Now for a couple of years I have struggled on with Amazon\u0026rsquo;s web based data pipeline architect to manage those changes to the pipeline. This has been an onerous task, as the architect does not really lend itself very well to the task of managing a large set of pipelines. Here begins a little tale of delving into the AWS data pipeline API to find another way.\u003c/p\u003e","title":"Creating AWS Data Pipelines with Boto3 and JSON"},{"content":"Rationale I have always been a fan of the Sphinx Python documentation generator and it has, I think, a nice feature where you can check out the raw source of a generated piece of documentation - a Show Source link. I decided that developing a Pelican plugin to imitate that feature would be a great way of getting a little deeper into the Pelican code itself.\nThis second post of the series explains the use of the Show Source plugin that I developed from the learning in the previous post. The following article has been, in some part, reproduced in the ReadMe.rst file included as part of the plugin.\nThe Show Source plugin The plugin allows you to place a link to the source text of your posts in the same way that Sphinx does. It works for both pages and articles, reading the source file and then placing a copy next to your published HTML version ready to be served when requested.\nI have designed the plugin to show source links on two different parts of a Pelican site: a link in the section below a post, and a link in the sidebar to the left or right of a post depending on your theme.\nPlugin Activation There are four things you have to do to get the plugin working:\nAdd to the Plugins List Add a Config Setting or Two Prepare the Theme [Make Source Available for Posts](#Make Source Available for Posts) Add to the Plugins List Add an entry to the PLUGINS list of show_source.\nPLUGINS = [\u0026#39;show_source\u0026#39;, ... ] Add a Config Setting or Two Depending on where you would like the links to appear, add a config setting (or both if you want):\nSHOW_SOURCE_ON_SIDEBAR = True SHOW_SOURCE_IN_SECTION = True Prepare the Theme To get the show source links on the article or page you will have to modify your theme, either as a sidebar display or in a section underneath an article.\nSidebar Links Add SHOW_SOURCE_ON_SIDEBAR = True to your settings file and modify your theme HTML. The example here is using the pelican-bootstrap3 theme:\n{% if SHOW_SOURCE_ON_SIDEBAR %} {% if (article and article.show_source_url) or (page and page.show_source_url) %} \u0026lt;li class=\u0026#34;list-group-item\u0026#34;\u0026gt;\u0026lt;h4\u0026gt;\u0026lt;i class=\u0026#34;fa fa-tags fa-file-text\u0026#34;\u0026gt;\u0026lt;/i\u0026gt;\u0026lt;span class=\u0026#34;icon-label\u0026#34;\u0026gt;This Page\u0026lt;/span\u0026gt;\u0026lt;/h4\u0026gt; \u0026lt;ul class=\u0026#34;list-group\u0026#34;\u0026gt; \u0026lt;li class=\u0026#34;list-group-item\u0026#34;\u0026gt; {% if article %} \u0026lt;a href=\u0026#34;{{ SITEURL }}/{{ article.show_source_url }}\u0026#34;\u0026gt;Show source\u0026lt;/a\u0026gt; {% elif page %} \u0026lt;a href=\u0026#34;{{ SITEURL }}/{{ page.show_source_url }}\u0026#34;\u0026gt;Show source\u0026lt;/a\u0026gt; {% endif %} \u0026lt;/li\u0026gt; \u0026lt;/ul\u0026gt; \u0026lt;/li\u0026gt; {% endif %} {% endif %} With the theme and settings ready, any enabled article or post will have a sidebar link like this:\nArticle Section Links Add SHOW_SOURCE_IN_SECTION = True to your settings file, and modify your theme\u0026rsquo;s article template, here is an example of HTML (yes, pelican-bootstrap3 again) to enable a source link at the bottom of an article:\n{% if SHOW_SOURCE_IN_SECTION %} {% if article and article.show_source_url %} \u0026lt;section class=\u0026#34;well\u0026#34; id=\u0026#34;show-source\u0026#34;\u0026gt; \u0026lt;h4\u0026gt;This Page\u0026lt;/h4\u0026gt; \u0026lt;ul\u0026gt; \u0026lt;a href=\u0026#34;{{ SITEURL }}/{{ article.show_source_url }}\u0026#34;\u0026gt;Show source\u0026lt;/a\u0026gt; \u0026lt;/ul\u0026gt; \u0026lt;/section\u0026gt; {% endif %} {% endif %} With the theme and settings ready, any enabled article or post will have a section link looking like the following:\nFor the theme I use I created a separate include with the code in and imported it into the article HTML with Jinja2\u0026rsquo;s include directive.\nMark Source Available for Posts In order to mark posts so that their source may be seen use the following metadata values (unless overridden)- for reStructuredText documents:\n:show_source: True or, in Markdown syntax\nShow_source: True The plugin will render your source document URL to a corresponding article.show_source_url (or page.show_source_url) attribute which is then accessible in the site templates.\nOverriding Default Plugin Behaviour There are, currently, two override options to the default behaviour of the plugin.\nShow Source for All Posts The default behaviour of the plugin is that revealing source is enabled on a case by case basis. This can be changed by the use of the following in the settings file:\nSHOW_SOURCE_ALL_POSTS = True This does mean that the plugin will publish all source documents no matter whether show_source is set in the metadata or not.\nSource File Output Name Unless overridden, each document is saved as the article or page slug attribute with a .txt extension.\nSo for example, if your settings file had ARTICLE_SAVE_AS configured like so:\nARTICLE_SAVE_AS = \u0026#39;posts/{date:%Y}/{date:%m}/{slug}/index.html\u0026#39; Your static HTML post and source text document will be like the following:\nposts/2016/10/welcome-to-my-article/index.html posts/2016/10/welcome-to-my-article/welcome-to-my-article.txt You can add the SHOW_SOURCE_FILENAME variable in your settings file to override the source file name, so you could set the following:\nSHOW_SOURCE_FILENAME = \u0026#39;my_source_file.txt\u0026#39; So with the ARTICLE_SAVE_AS configured as above, the files would be saved thus:\nposts/2016/10/welcome-to-my-article/index.html posts/2016/10/welcome-to-my-article/my_source_file.txt This is the same behaviour for pages also.\nPlugin Availability Currently this plugin is awaiting a pull request with the developers at getpelican/pelican-plugins, so for now it\u0026rsquo;s available from my repo here on a feature/show_source branch, until further notice.\n","permalink":"https://chrisramsay.co.uk/posts/2016/11/the-show-source-pelican-plugin/","summary":"\u003ch2 id=\"rationale\"\u003eRationale\u003c/h2\u003e\n\u003cp\u003eI have always been a fan of the \u003ca href=\"http://www.sphinx-doc.org/\"\u003eSphinx\u003c/a\u003e Python documentation generator and it has, I think, a nice feature where you can check out the raw source of a generated piece of documentation - a \u003cem\u003eShow Source\u003c/em\u003e link. I decided that developing a \u003ca href=\"https://github.com/getpelican/pelican-plugins\"\u003ePelican plugin\u003c/a\u003e to imitate that feature would be a great way of getting a little deeper into the Pelican code itself.\u003c/p\u003e\n\u003cp\u003eThis second post of the series explains the use of the Show Source plugin that I developed from the learning in the \u003ca href=\"/posts/2016/11/basic-notes-on-pelican-plugin-architecture/\"\u003eprevious post\u003c/a\u003e. The following article has been, in some part, reproduced in the \u003ccode\u003eReadMe.rst\u003c/code\u003e file included as part of the plugin.\u003c/p\u003e","title":"The Show Source Pelican Plugin"},{"content":"Rationale I have always been a fan of the Sphinx Python documentation generator and it has, I think, a nice feature where you can check out the raw source of a generated piece of documentation - a Show Source link. I decided that developing a Pelican plugin to imitate that feature would be a great way of getting a little deeper into the Pelican code itself.\nThis first post explores the Pelican plugin architecture and the basics of building a plugin.\nThe Pelican Plugin Architecture Firstly, and mostly for my own benefit, I would like to make a few notes about the Pelican plugin architecture. Now that I have delved into it a little I have found that it is quite simple and gives you great ease of access to the entire build process. Pretty much all the important objects in the process can be accessed and modified.\nThe Pelican blog system uses the blinker Python library to emit a series of signals during build time, that a series of interested parties can subscribe to. Any plugin developed for Pelican is designed to hook into this series of signals, and there are a lot! Here is a list of signals and the objects that are available (as from pelican/pelican/signals.py) at commit 8993c55:\nSignals |-Run-level signals | |-initialized (pelican object) | |-get_generators (pelican object) | |-all_generators_finalized (generators) | |-get_writer (pelican object) | `-finalized (pelican object) |-Reader-level signals | `-readers_init (readers) |-Generator-level signals | |-generator_init (generator) | |-article_generator_init (article_generator) | |-article_generator_pretaxonomy (article_generator) | |-article_generator_finalized (article_generator) | |-article_generator_write_article (article_generator, content) | |-article_writer_finalized (article_generator, writer) | |-page_generator_init (page_generator) | |-page_generator_finalized (page_generator) | |-page_writer_finalized (page_generator, writer) | |-static_generator_init (static_generator) | `-static_generator_finalized (static_generator) |-Page-level signals | |-article_generator_preread (article_generator) | |-article_generator_context (article_generator, metadata) | |-page_generator_preread (page_generator) | |-page_generator_context (page_generator, metadata) | |-static_generator_preread (static_generator) | |-static_generator_context (static_generator, metadata) | `-content_object_init (content_object) `-Writers signals |-content_written (path, context) `-feed_written (path, context, feed) Plugin File Layout In order to build a plugin, you need at least a couple of files in your plugin directory: an __init__.py and a plugin file, say, myplugin.py, looking something like:\nmyplugin |-__init__.py `-myplugin.py The contents of __init__.py should be the following:\nfrom from .myplugin import * The contents of myplugin.py could be something like:\nfrom pelican import signals def plugin_function(article_generator, content): \u0026#34;\u0026#34;\u0026#34;Your plugin function for article_generator_write_article\u0026#34;\u0026#34;\u0026#34; # You now have access to the article_generator \u0026amp; content objects pass def plugin_function_2(path, context): \u0026#34;\u0026#34;\u0026#34;Your plugin function for content_written\u0026#34;\u0026#34;\u0026#34; # You now have access to the path \u0026amp; context objects pass def register(): \u0026#34;\u0026#34;\u0026#34; The register function is a requirement for the plugin to work. You have to call your plugin functions by `signals.x.connect(function)` with x being the required signal(s) and main_function being the function for that signal. You can have as many signal.x.connect() as you want here. \u0026#34;\u0026#34;\u0026#34; # When the signals below are emitted, functions above will be called signals.article_generator_write_article.connect(plugin_function) signals.content_written.connect(plugin_function_2) That\u0026rsquo;s It! So you can see that developing a simple plugin is really not a lot of work if you understand the basics. In order to gain a deeper knowledge of what you can do with plugins, it is definitely worth spending some time having a look through the Pelican plugin repo to see what others have done.\nWhilst you are there, have a look at how the objects that are made available in the functions are used.\nThere are more helpful hints regarding plugin development here in the Pelican docs, as well as a handy page on contributing a plugin once again in the Pelican plugin repo\nHave fun!\n","permalink":"https://chrisramsay.co.uk/posts/2016/11/basic-notes-on-pelican-plugin-architecture/","summary":"\u003ch2 id=\"rationale\"\u003eRationale\u003c/h2\u003e\n\u003cp\u003eI have always been a fan of the \u003ca href=\"http://www.sphinx-doc.org/\"\u003eSphinx\u003c/a\u003e Python documentation generator and it has, I think, a nice feature where you can check out the raw source of a generated piece of documentation - a \u003cem\u003eShow Source\u003c/em\u003e link. I decided that developing a \u003ca href=\"https://github.com/getpelican/pelican-plugins\"\u003ePelican plugin\u003c/a\u003e to imitate that feature would be a great way of getting a little deeper into the Pelican code itself.\u003c/p\u003e\n\u003cp\u003eThis first post explores the Pelican plugin architecture and the basics of building a plugin.\u003c/p\u003e","title":"Basic Notes on Pelican Plugin Architecture"},{"content":"It has been a while since I made any progress on a project, so I thought that I would put something down here as a way of summarising to myself where things are, what is left to do and so on, as a form of to-do list.\nCloud Detection This is the ongoing project to measure cloud cover, as discussed in this series of articles. So, what\u0026rsquo;s left to do?\nPackaging it up As discussed I am still looking to package the system up using a combination of external mains junction box, various bits of plastic plumbing pipe and drainage pipe, expoxy resin and silicone sealant.\nReflecting surface The IR sensor must be kept out of the way of inclement weather. This means housing it in such a way so that it takes its measurements indirectly via a reflecting surface, such as a home made polished aluminium mirror. Regarding how to deal with dew, the mirror must come equipped with a temperature sensor and heating element bonded to the rear non-reflecting surface.\nThings to do Research into the following:\nBuilding a heating element (probably Nichrome wire) Heater control and feedback (see below) Mirror thermal properties, i.e. mass and dimensions Dealing with Dew The reflecting surface will at least partially be exposed to the sky thus necessitating a method or two of keeping dew at bay. I am currently thinking of constantly sampling the mirror temperature and using that to feed back to a light duty heating system, comprising a nichrome wire heating element, to keep the mirror a couple of degrees above dew point. Additionally I am considering using a small fan to keep air moving in the space between the IR sensor and the mirror.\nThings to do Build a prototype circuit with an Arduino as a controller containing the following:\nA DHT22 humidity sensor and required code to calculate local dew point temperature. A DS18B20 temperature sensor for measuring the reflector surface temperature. A relay or transistor to control the reflector heating element and fan. A bicolour LED to indicate heating on / off. Write a brief blog entry about it in the projects section. Soil Temperature Monitoring This system has spent well over a year working \u0026ldquo;in production\u0026rdquo;. I\u0026rsquo;ve only had to address some reliability problems with the networking, as discussed here, by giving the Arduino a heartbeat and here regarding the W5100 ethernet shield.\nCamera Bird Box This is a project I have thought about doing for years. Probably these days it may be cheaper to go an buy an internet-ready bird nesting box with a HD quality camera inside, but where is the fun in that? I have spent a few nights putting a few hours into the cunstruction of a bird box with space to fit a modified cheap USB web cam from my local Maplin store. I\u0026rsquo;ve retrofitted some bright IR LEDs to give a little light so that the chicks can be seen (and hopefully only gently warmed in the IR glow!). I have all the materials required to complete this project - the only other thing I need to think about is to write an entry or two on here at some point.\nUV Light Box A friend at work and I have shared many an hour discussing Arduino projects, and after many circuits built using veroboard we decided to have a look at prototyping some custom circuit boards. Not being up for the old fashioned approach of drawing out circuits on board by hand, we decided to have a go at laser printing them instead. There is no need go through the whole process of how one can home-make PCBs, so here is a handy guide. It just suffices to say that we wanted to have a go at a UV exposure method for masking the circuit board prior to etching.\nA little bit of research will tell you that these UV light boxes are quite expensive, so why not make a DIY version instead?\nLaser Visibility Detector This is based on another thought that kept me awake one night (yes, I must learn to think about more interesting things!). Listening to the shipping forecast on Radio 4 I always wondered how the visibility was measured. Traditionally, of course, this was done simply with a manual observation of objects at known distances. These days measurements are taken using instruments using xenon beams and sensors separated by a short distance.\nI got to thinking about whether I could do something similar using a cheap laser diode and a detector to measure the loss of light by scattering over a small distance in the garden.\nAt this stage the whole thing is a proof of concept and another excuse to spend a few hours in the workshop playing with electronics. I have a few ideas about how to build the thing and have started prototyping some small circuits. Again, it is probably time to start writing an article on here about it.\n","permalink":"https://chrisramsay.co.uk/posts/2016/05/projects-update/","summary":"\u003cp\u003eIt has been a while since I made any progress on a project, so I thought that I would put something down here as a way of summarising to myself where things are, what is left to do and so on, as a form of to-do list.\u003c/p\u003e","title":"Projects Update"},{"content":"This is a little project to build a cheap UV exposure box for developing UV sensitive printed circuit boards.\nCircuit Construction I needed an even spread of light from a source close to the board being exposed so I decided to opt for some reasonably wide angle LEDs arranged in a matrix. After some calculations regarding spacing on veroboard, I settled on a figure of about 90 - (3 LEDs in series plus a current limiting resistor between single supply and common rails) times 30.\nI headed to eBay to my trusted electronics source and ordered 100 UV LEDs with the following specification:\nLens Type : Crystal Clear Case Style : 5mm Round Flat Top Viewing Angle : 120 degrees Brightness : 500mcd (Ultra Bright) Forward Voltage : 3.2v - 3.8v Forward Current : 20mA (typical), 30mA (max) Wavelength : 395nm (Ultraviolet A) Assuming a supply voltage of 12V, each LED operating current being 20mA and a mid-range voltage drop of 3.4V, three LEDs in series should require a current limiting resistor of 91Ω (nearest high value in the 5% resistor range).\nCircuit layout - LED series 1 -\u0026gt; 2 -\u0026gt; n\nConstruction of the circuit didn\u0026rsquo;t take as long as I thought it would. It was one of those jobs that just rolled along once I got into the rhythm, helped along nicely by using a new (to me, 2nd hand) solder fume extractor. Not sitting in a cloud of burning flux makes the experience so much more pleasant. Anyway, below is a close up of the finished board.\nLEDs and resistors in the board\nOf course, the worst part is powering up your creation for the first time. Imagine the anticipation, followed by a plume of black smoke. I\u0026rsquo;ve been there and it is terrible. So that didn\u0026rsquo;t happen this time; it all lit up beautifully, all 90 LEDs. It was interesting to note that the current draw on my lab bench power supply was around 0.8A - pretty much what I expected. Thankfully no fires, just an intense UV glare from all those LEDs; definitely not good for the eyes.\nFirst smoke test - it works!\nBox Construction So the next job was to find a way of boxing the whole thing up and controlling the light by way of opening and closing the lid of the box. At some point one of these cheap fake metal flight cases (by fake metal I actually mean thin wood with some sort of patterned plastic \u0026ldquo;metal effect\u0026rdquo; foil) had come into my possession. This seemed to provide the perfect sacrificial offering to my project1. Below is a picture of the completed light box, housed in its \u0026ldquo;quality\u0026rdquo; housing.\nAssembly complete\nBelow is a close up of the microswitch arrangement. What can I say more than it functions perfectly? Eyes will now be protected when lifting the lid.\nMicroswitch in place with actuator\nProject Complete Final in place testing of the on/off switch. Project complete!\nTesting the switch\nAbove you can see me just giving the microswitch arrangement a quick test. Those UV LEDs are really bright! I am very much looking forward to producing my first circuit board.\nBend to the will of my project or be detroyed in the attempt.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://chrisramsay.co.uk/posts/2016/05/uv-light-box/","summary":"\u003cp\u003eThis is a little project to build a cheap UV exposure box for developing UV sensitive printed circuit boards.\u003c/p\u003e","title":"UV Light Box"},{"content":"Arduino Yún Reliability Problems I\u0026rsquo;ve been testing out the Arduino Yún recently with a view to making use of one in a project. I\u0026rsquo;ve come across a problem or two and solutions that were probably worth making a note of, so I thought that I\u0026rsquo;d write them here as somewhere to come back to in future.\nOne little problem I\u0026rsquo;ve faced is the reliability of running the Yún: doing a job whilst maintaining a working network interface. I\u0026rsquo;ve found that even a simple task, such as using the Bridge library and Yún server and client libraries together to create a REST query-able thermometer have proved problematic. Yúns with the early software version, (2014-04-10 11:08:41 CEST) I believe, seem prone to stopping responding after a few hours. I\u0026rsquo;ve made a note here of how to hopefully resolve that.\nOther smaller, not exactly problems, just things that make it a nicer beast to work with are things such as setting the Yún up just the way you want it on a network. I\u0026rsquo;ve noted a couple of things here.\nUpdating the Yún Software Likely you\u0026rsquo;ll want to know which version of the software your Yún has installed. This is easily done by SSHing to the Yún and looking at a particular file:\nfoobox:~$ ssh root@arduino.local BusyBox v1.19.4 (2014-04-10 11:08:41 CEST) built-in shell (ash) Enter \u0026#39;help\u0026#39; for a list of built-in commands. _______ ________ __ | |.-----.-----.-----.| | | |.----.| |_ | - || _ | -__| || | | || _|| _| |_______|| __|_____|__|__||________||__| |____| |__| W I R E L E S S F R E E D O M ----------------------------------------------------- root@Arduino:~# cat /etc/arduino/openwrt-yun-release built=Tue Jul 15 21:14:21 CET 2014 The date you see after built= can be compared against the change log present in the arduino/openwrt project, right here in raw form. My version number above means the software was built as release 1.4, on July 15th, 2014.\nSo how about updating? This is actually quite easy. First, download the latest OpenWRT - Yún (which was version 1.5.3) upgrade image from here. Unzip the package and extract the bin file on to a micro SD card. Power down the Yún and plug the card into the card slot on the underside.\nNow power the Yún back up, making sure that you have a network cable plugged in. SSH to the Yún and execute run-sysupgrade with a path to the bin file on /mnt/sda1.\nroot@Arduino:~# run-sysupgrade /mnt/sda1/openwrt-ar71xx-generic-yun-16M-squashfs-sysupgrade.bin Sending TERM to remaining processes ... uhttpd dbus-daemon dnsmasq avahi-daemon thd ntpd uSDaemon blkid syslogd klogd hotplug2 ubusd netifd Sending KILL to remaining processes ... uhttpd Switching to ramdisk... Performing system upgrade... Unlocking firmware ... Writing from \u0026lt;stdin\u0026gt; to firmware ... [w] Upgrade completed Rebooting system... After a few moments you should be able to log in again and hopefully note that the build date has changed - in my case:\nroot@Arduino:~# cat /etc/arduino/openwrt-yun-release built=Fri Nov 14 03:53:51 CET 2014 According to the previously mentioned change log, that means I\u0026rsquo;m fully up to date.\nChanging eth1 to be Static I always prefer bits of kit that will be permanently installed on the network to have a fixed IP address. In my usual manner I like to keep systems undertaking certain types of job grouped in specific number ranges.\nFinding the current IP address:\nfoobox:~$ ping arduino.local PING arduino.local (192.168.100.134): 56 data bytes It just so happens that 192.168.100.134 is in a dynamic portion of my local IP range. Let\u0026rsquo;s log into the Yún and sort things out.\nroot@Arduino:~# cat /etc/config/network config interface \u0026#39;loopback\u0026#39; option ifname \u0026#39;lo\u0026#39; option proto \u0026#39;static\u0026#39; option ipaddr \u0026#39;127.0.0.1\u0026#39; option netmask \u0026#39;255.0.0.0\u0026#39; config interface \u0026#39;lan\u0026#39; option proto \u0026#39;static\u0026#39; option ipaddr \u0026#39;192.168.240.1\u0026#39; option netmask \u0026#39;255.255.255.0\u0026#39; config interface \u0026#39;wan\u0026#39; option ifname \u0026#39;eth1\u0026#39; option proto \u0026#39;dhcp\u0026#39; option metric \u0026#39;10\u0026#39; So just change the wan part (because I want a static IP at 192.168.100.11) to:\nconfig interface \u0026#39;wan\u0026#39; option ifname \u0026#39;eth1\u0026#39; option proto \u0026#39;static\u0026#39; option \u0026#39;ipaddr\u0026#39; \u0026#39;192.168.100.11\u0026#39; option \u0026#39;netmask\u0026#39; \u0026#39;255.255.255.0\u0026#39; option \u0026#39;gateway\u0026#39; \u0026#39;192.168.100.1\u0026#39; option \u0026#39;dns\u0026#39; \u0026#39;192.168.100.1\u0026#39; option metric \u0026#39;10\u0026#39; Once that is saved, just issue a /etc/init.d/network reload command and eventually your Yún will appear at the desired address.\nFinal check, back on your local machine:\nfoobox:~$ ping arduino.local PING arduino.local (192.168.100.11): 56 data bytes Changing the Yún Host Name I know at some point that I\u0026rsquo;ll end up hosting another Yún on the network, so changing the host name would be a good thing to do. This is quite easy, as it turns out.\nAscertaining that arduino.local is indeed what the Yún responds to at the moment:\nfoobox:~$ ping arduino.local PING arduino.local (192.168.100.11): 56 data bytes 64 bytes from 192.168.100.11: icmp_seq=0 ttl=64 time=4.399 ms 64 bytes from 192.168.100.11: icmp_seq=1 ttl=64 time=1.544 ms Once again, SSH to arduino.local and edit /etc/config/system, changing option hostname from Arduino to whatever you desire. How about Unicorn?\nroot@Arduino:~# vi /etc/config/system config system option hostname Unicorn option timezone UTC config timeserver ntp list server 0.openwrt.pool.ntp.org list server 1.openwrt.pool.ntp.org list server 2.openwrt.pool.ntp.org list server 3.openwrt.pool.ntp.org option enable_server 0 Next, also change the host name using the hostname on the command line and restart the avahi-daemon:\nroot@Arduino:~# hostname Unicorn root@Unicorn:~# /etc/init.d/avahi-daemon restart Now log out, and back on your local machine, ping unicorn.local to see whether it works or not:\nfoobox:~$ ping unicorn.local PING unicorn.local (192.168.100.11): 56 data bytes 64 bytes from 192.168.100.11: icmp_seq=0 ttl=64 time=3.955 ms 64 bytes from 192.168.100.11: icmp_seq=1 ttl=64 time=4.428 ms And\u0026hellip; success! You may want a slightly more meaningful name than unicorn, but that is, of course, up to you.\nJust note that some patience may be required, or perhaps the flushing of DNS caches - the way your network is set up no doubt differs to mine, so it may take some time for the new host name to be picked up by other machines on your network. It was pretty instant on mine, and I had no troubleshooting to undertake, that\u0026rsquo;s all I\u0026rsquo;ll say.\nRestoring the Bridge after a Yún Reboot via SSH When faced with a Yún based web service that has stopped responding, my natural fallback position is to SSH in and reboot the Yún. This is not a successful strategy, as I am then faced with a message that says the following:\nCould not connect to YunServer 146 Connection refused The only action I found that did fix the situation was a button press on YÚN RST or a power cycle. In a nutshell, once up and running, the web facing REST interface works well. Then after a few hours I start getting zero byte responses from the interface . Whilst the Yún is still up and running and responding to pings and able to be SSHed into, the Bridge software interface seems to stop passing data back and forth from the Arduino side of things. Disclaimer here\u0026hellip; that is only my guess at what may happening under the hood and I could easily be wrong.\nIn order to understand what the Bridge library does it would be helpful to briefly explain the Yún architecture. The Yún has two processors on board: the ATmega32U4, which is the 8 bit MCU beating heart of many an Arduino, and an Atheros AR9331 processor, which runs Linino (based on the OpenWRT Linux distribution).\nThe Bridge library itself enables communication between the two on board processors, giving Arduino sketches the ability to run shell scripts, communicate with network interfaces, and receive information from the AR9331 processor. The USB host, network interfaces and SD card are not connected to the ATmega32U4, but the AR9331, and the Bridge library also enables the Arduino to interface with those peripherals.\nAnyway, after a bit of rooting about on the Yún\u0026rsquo;s file system, I discovered a reset-mcu command. This command sits in the file: /etc/rc.local. The rc stands for run control, and this file is normally executed once all the system services are up and running at the end of a process of switching to a multiuser run level (levels 2-5 in most Linuxes). It is an ideal location to start custom services or run special commands. Here is the output of rc.local on my Yún:\nroot@Unicorn:~# cat /etc/rc.local # Put your custom commands here that should be executed once # the system init finished. By default this file does nothing. wifi-live-or-reset boot-complete-notify # Uncomment the following line in order to reset the microntroller # right after linux becomes ready #reset-mcu # Uncomment the following line in order to disable kernel console # debug messages, thus having a silent and clean serial communication # with the microcontroller #echo 0 \u0026gt; /proc/sys/kernel/printk exit 0 And there is the magic -\n# Uncomment the following line in order to reset the microntroller\n# right after linux becomes ready\nSo after uncommenting the reset-mcu line I rebooted the Yún from the command line and as I hoped, it came back up with a working bridge.\n","permalink":"https://chrisramsay.co.uk/posts/2016/04/arduino-yun-reliability-problems/","summary":"\u003ch2 id=\"arduino-yún-reliability-problems\"\u003eArduino Yún Reliability Problems\u003c/h2\u003e\n\u003cp\u003eI\u0026rsquo;ve been testing out the \u003ca href=\"https://www.arduino.cc/en/Main/ArduinoBoardYun\"\u003eArduino Yún\u003c/a\u003e recently with a view to making use of one in a project. I\u0026rsquo;ve come across a problem or two and solutions that were probably worth making a note of, so I thought that I\u0026rsquo;d write them here as somewhere to come back to in future.\u003c/p\u003e","title":"Arduino Yún Reliability Problems"},{"content":"Current Virtual Server It has been about a year since I started using a virtual server on the laptop to locally host my blogging efforts with Pelican. Up to now I have used a combination of Vagrant (with Virtualbox in the background) and a Puppet built CentOS VM. At work we are increasingly introducing the concept of containerised services in the form of Docker containers, and I felt that revisiting my blogging setup with a view to making use of a container would be a great way of getting to know the technology a little better.\nFrom Vagrant to Docker I should begin by stating that, although you will frequently see the question \u0026ldquo;What is better, Docker or Vagrant?\u0026rdquo;, it is not really a fair comparison. The two technologies fulfil different use cases and overlap only in a minimal way.\nVagrant is generally more focused on providing a complete development environment, therefore your Vagrant box might provide a web server and a database within the same box.\nDocker is an application oriented virtual environment with the Docker container only providing what is absolutely necessary for the application to run, for example just a web server. Should you require a database then that would be provided by a separate Docker container.\nSo essentially, I am moving from a virtual machine focused on providing a whole system to one that provides only bare essentials for getting the job done.\nGetting Started with Docker As there are hundreds of guides out there on how to install and get going with Docker, I am not going to repeat my experience for you. Probably the most helpful thing I could do would be to go ahead and deconstruct my Dockerfile, so here goes.\nThis container is based on a clean Debian linux distribution. The latest tag indicates that this will be latest stable release (debian:jessie at the time of writing this). The MAINTAINER field is just metadata indicating who the author is; quite self explanatory really!\nFROM debian:latest MAINTAINER Chris Ramsay \u0026lt;chris@ramsay-family.net\u0026gt; This next block below gets the machine to run a general software update and in addition install some basic requirements: Python, pip (which is a Python package manager) and some development tools; those tools are purely to assist in building some of the packages later.\n# Update \u0026amp; add prerequisites RUN apt-get -y update \u0026amp;\u0026amp; apt-get install -y \\ python \\ python-dev \\ python-pip \\ git Now that all the Python prerequisites are in place it is time to go ahead and add the packages that will be required for Pelican, some of the Pelican plugins and the build and deployment process, which I orchestrate using the Fabric Python library. Note that I also include a custom .bashrc file so that I can give the terminal a nice command prompt and some handy aliases.\nWORKDIR /srv ADD requirements.txt /srv/requirements.txt ADD files/bashrc /root/.bashrc Separately, the requirements.txt file that is brought in by the ADD command looks like this:\nFabric Jinja2 Markdown MarkupSafe Pygments Unidecode argparse beautifulsoup4 blinker docutils ecdsa feedgenerator paramiko pelican pycrypto python-dateutil pytz six smartypants typogrify wsgiref Below, the final block uses Python\u0026rsquo;s pip package manager to install the packages from the requirements file above:\nRUN pip install -r requirements.txt So if we have got here successfully it means that it is time to move a bunch of files to the right place in the container OS in order to allow us to be able to securely commit code changes to remote git repositories and digitally sign commits using GNUPG. All these important files are never stored in the Docker image itself as they would become available to anyone with the ability to download my images and run Docker. In order to make them accessible to the Docker instance they are mounted in at run time from the host machine:\n-v $wd/git/.gitconfig:/root/.gitconfig \\ -v $wd/gnupg:/root/.gnupg \\ -v $wd/ssh:/root/.ssh \\ The three volume mount commands take place in a handy little script as described in the next section. That is pretty all there is to it; just the most basic things that I need to spend happy hours playing with the blog.\nUp and Running I have created a small shell script for when it comes to actually running the container; it looks something like this:\n#!/bin/bash ### # Add more mounts for your projects after the git/gnupg/ssh ones # e.g. -v ~/stuff/myproject:/project \\ wd=$(pwd) docker rm -f pelicanbox docker run \\ --name pelicanbox \\ -p 80:8000 \\ -v $wd/git/.gitconfig:/root/.gitconfig \\ -v $wd/gnupg:/root/.gnupg \\ -v $wd/ssh:/root/.ssh \\ -v ~/work/blog:/project \\ -v ~/extras/pelican-plugins:/project/pelican-plugins \\ -v ~/extras/pelican-themes:/project/pelican-themes \\ -ti \\ chrisramsay/docker-pelicanbox:latest \\ /bin/bash The script above just saves me the hassle of having to remember what to type on the command line each time. Once I am logged into the container, I just navigate to the build directory and run the command to serve the site over HTTP, then point the browser to (in my case) 192.168.99.100.\nThis is the End\u0026hellip; In summation, my early days with Docker have gone well and it is beginning to prove itself to be a lightweight replacement for the Vagrant boxes I used to run. As a bonus I have found that my laptop battery is lasting longer too!\n","permalink":"https://chrisramsay.co.uk/posts/2015/12/pelican-blogging-with-a-docker-container/","summary":"\u003ch2 id=\"current-virtual-server\"\u003eCurrent Virtual Server\u003c/h2\u003e\n\u003cp\u003eIt has been about a year since I started using a virtual server on the laptop to locally host my blogging efforts with \u003ca href=\"http://blog.getpelican.com/\"\u003ePelican\u003c/a\u003e. Up to now I have used a combination of \u003ca href=\"https://www.vagrantup.com/\"\u003eVagrant\u003c/a\u003e (with Virtualbox in the background) and a \u003ca href=\"https://puppetlabs.com/\"\u003ePuppet\u003c/a\u003e built CentOS VM. At work we are increasingly introducing the concept of containerised services in the form of \u003ca href=\"https://www.docker.com/\"\u003eDocker\u003c/a\u003e containers, and I felt that revisiting my blogging setup with a view to making use of a container would be a great way of getting to know the technology a little better.\u003c/p\u003e","title":"Pelican Blogging with a Docker Container"},{"content":"This is a race report for the 2015 Gruesome Twosome Half Marathon\nDetails Useful information about the race.\nName\nThe Gruesome Twosome, Swallow, Lincolnshire, UK.\nCategory\nHilly (mostly) off road half marathon in teams of two.\nDate\n17th October 2015\nTime\nMid morning, 11:00 start.\nWeather Conditions\nDry with broken cloud and a nagging North Easterly wind.\nCourse/Road Conditions\nThe course was a mixture of wooded sections, open fields, small country roads and technical sections with logs to clear. The course profile provided very few flat sections; it was mostly either up a hill or down one! Conditions were largely dry; a choice of road shoes rather than trail shoes would not have significantly disadvantaged anyone.\nTaking Part Details on taking part in the race.\nThings I liked A very well established local event with a really good team of race organisers in the background (Tape2Tape). Set in the beautiful rolling hills of the Lincolnshire Wolds, the race provides great views and access to some lovely tracks, trails, woods and fields throughout the course.\nObjectives The main objective was to keep up with my speedy team mate, Steve Ellis, and post a reasonable time.\nGoals Being that this was my second ever competitive cross country half marathon, to beat my time from the Caythorpe Dash, earlier this year.\nExperience The race began with a wide mass start in a field then quickly funneled through a gate and subsequently took in three sides of a large grassy field on a hillside, making up the first mile of the course. This, somewhat beneficially, gradually thinned out the runners prior to us entering the first wooded section of the run, which introduced itself with a hop over some logs into a lovely set of trails under the trees. These were often lined on both sides with nettles, of which I am personally a great fan! Much of this section was only wide enough for a single line of traffic, providing very few opportunities for overtaking.\nAs we briefly climbed out of the woods, a panorama of fields opened up to the left before we disappeared once again into trees down a steep dip with a style to climb over at the bottom. Still moving in a westerly direction, we began the climb up to the highest point on the course just beyond mile three at the top of the hill between the villages of Caybourne and Rothwell. It was at this point, having just turned south that things literally went south; I suddenly felt my wedding ring drop away from me and onto the muddy track upon which several hundred pairs of feet were going to trample.\nI notified my team mate, Steve, and thus began a frantic search for a few minutes but to no avail. I decided to cut the loss and get on with the race, making a mental note of a bunch of yellow flowers growing at the side of the farm track as a rough reminder of the spot for returning later.\nSo we began again, albeit a few dozen places further back, and dropped down a rugged, part-surfaced track then onto tarmac and into the village of Rothwell where, rounding the Blacksmiths\u0026rsquo; Arms pub, we headed on to the longest leg of the race, heading in a north-westerly direction. This stretch was bound to be challenging, what with its length of over five miles heading directly into the strengthening wind. After a water stop and, bizarrely, encountering a lady taking a stroll with an owl perched on her arm, we continued out of Rothwell along the valley bottom through the Limagrain site and over some muddy fields.\nCrossing a small wooden bridge, we ascended the side of the valley and over the hilltop into another drop to the yard of a small farm at Cuxwold. Exiting the yard on the right, we climbed a steep concrete road slowly turning to the west and up into the wind once again. Now began a three mile long slog across wind exposed fields with little in the way of cover except for hedges and the odd tree or two. There was a welcome water stop along the way and some great views towards Barton Street, Grimsby, the mouth of the Humber river and the coast.\nMile nine, which was the lowest elevation we had been at since the start, saw us jump through a gap in a hedge and finally turn almost in the opposite direction, much to our relief, with the wind now at our backs. We moved along the edge of a field and into shelter along a secluded track overlooked by a coppice of mixed woodland. With the shelter of the trees and sunlight streaming into our faces, this was one of the most pleasant parts of the course. Climbing to the third and final water stop, we hit the first of a handful of road sections that characterized the last three miles. These sections of tarmac afforded us the option of speeding up a little with the added advantage of the wind being still at our backs.\nWe were now almost continually losing altitude for the next two miles, but with the knowledge that there was one more hill to come. We cut through a few fields, one with a particularly short but steep climb and steadily made progress. Once again, the views, this time to the west, were really rather good, more than making up for the gruesomeness of the course so far! Turning briefly northward we reached a deserted farmyard with a sharp right turn at the end. This took us to the foot of the final steep climb at mile twelve and it seriously did not disappoint. It was an ascent of approximately one third of a mile and quite unrelenting. I think that congratulations are definitely owed to the course creators for this feature, as it really added something to the experience that had everyone talking afterwards.\nOnce the hill was crested, it was a hop and a skip before heading down again to the very final leg along the old A46 to the finish line in a playing field at the side of the Swallow crossroads. It was a most welcome sight. The processing of the race finishers was of the usual high standard provided by the organisers at Tape2Tape and we were afforded with a choice of black or luminous green for this years\u0026rsquo; coveted finisher beanie hat.\n\u0026hellip; and the Wedding Ring? Well as things turn out I found that one of the marshals at the finish line was none other than Jimmy Dutch, veteran Caistor and Cleethorpes running club member with well over a hundred marathons to his name. Not content with being an accomplished runner in his retirement, he is also chairman of the local metal detecting society, and as luck would have it, had his metal detector with him in the car. We returned to the spot of the unfortunate loss. It only took him three minutes to find the practically invisible ring in the grass and mud; I can definitely say that without Jimmy and his trusty metal detector, the search would have been fruitless.\nPros Where to start? This is probably my favourite event so far since I started taking part in races. The event itself has the highest standard of organisation that I\u0026rsquo;ve seen considering its size. The support from all the marshals (especially the Caistor Running Club ones (I have to say that as I am biased!) was brilliant. It was not the warmest of days, and standing on the same spot for hours in such conditions is not easy - I know because I\u0026rsquo;ve been there. The course itself was new this year in order to avoid the dangerous section on Caistor High Street - not having run the old course, I can\u0026rsquo;t comment about whether this was an improvement. What I can say is that the course was excellent from my point of view; nice and undulating with a good mix of surfaces made more interesting my the weather conditions.\nCons Really there is nothing I can criticise, other than the one thing out of human control - the weather. Even that would be a little churlish, as the conditions played their part in adding to the dynamics of the event.\nRace/Event Rating In my opinion everything is great about this event: course, marshals, organising, even the post race beanie hat! The maximum 5/5 is well earned.\nLessons Learned I took part in a recce run of the course the week before organised by Nicola of Tape2Tape, and I found that having the prior knowledge of the course really helped on the day, especially with my team mate not knowing the course.\n","permalink":"https://chrisramsay.co.uk/posts/2015/11/2015-gruesome-twosome/","summary":"\u003cp\u003eThis is a race report for the 2015 Gruesome Twosome Half Marathon\u003c/p\u003e\n\u003ch2 id=\"details\"\u003eDetails\u003c/h2\u003e\n\u003cp\u003eUseful information about the race.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eName\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe Gruesome Twosome, Swallow, Lincolnshire, UK.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCategory\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eHilly (mostly) off road half marathon in teams of two.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e17th October 2015\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTime\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMid morning, 11:00 start.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeather Conditions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDry with broken cloud and a nagging North Easterly wind.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCourse/Road Conditions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe course was a mixture of wooded sections, open fields, small country roads and technical sections with logs to clear. The course profile provided very few flat sections; it was mostly either up a hill or down one! Conditions were largely dry; a choice of road shoes rather than trail shoes would not have significantly disadvantaged anyone.\u003c/p\u003e","title":"Race Report: 2015 Gruesome Twosome"},{"content":"This is a race report for the 2015 Great North Run Half Marathon\nDetails Useful information about the race.\nName\nThe Great North Run, Tyneside, UK.\nCategory\nAll road half marathon. Probably one of the most famous half marathons in the world.\nDate\n13th September 2015\nTime\nMid morning, 10:45 start.\nWeather Conditions\nWarming up as the morning went on, broken cloud.\nCourse/Road Conditions\nRoad was mostly dry; course profile was largely flat with gentle undulations where there were bridges, underpasses and so on. Only a couple of miles before the end presented any kind of real slope.\nTaking Part Details on taking part in the race.\nThings I liked A massive race from any aspect! The number of participants (greater than 50 k), the super organisation, huge crowds watching and so on.\nObjectives Having heard about the crowded conditions and given the start position I was handed (well down the field), it was a case of trying to see if under 2 hours would be possible. Therefore the strategy was to put up with the congestion and run slowly at the start. Increase pace as the field thins out after a few miles.\nGoals Fight the crowds of runners and get round in less than 2 hours. That was it.\nExperience I got a lift up north with another club runner who had taken part in the event a couple of years before. His advice was to park up at the end, in South Shields, and get a metro or bus into Newcastle where the start was. We parked the car on a common right near where the end of the race would be and walked down towards the town centre. In the end we caught a specially laid on bus and that only took about half an hour to get into the city centre. Although the queue to get on the bus had been quite long, say 10 or 15 minutes, that paled into comparison when you heard stories of people waiting one hour or more just to get to the metro platform.\nGetting into the start area was an interesting experience; by the time I got to my designated pen it was already overflowing and getting into it was quite a challenge. The gun duly went off at 10:45 after plenty of stading around trying to warm up with the masses; we didn\u0026rsquo;t actually cross the start line until 22 minutes later.\nBy the time I was underway the weather was getting increasingly brighter and as a result the temperature was climbing. The early couple of miles through the city centre passed in a blur of underpasses and bridges and the crowds, both runners on the road and the superb support at the sidelines, did not lessen in intensity.\nOne thing that did surprise me was the number of people walking; those walking on the flat after a couple of miles, and the general number of people who seemed unwilling to run up even the gentlest of slopes. It only really hit me then that there was a huge number of new runners here. It simply never occurred to me the number of people taking part in this sort of event for the first time, or taking part on the back of very little training.\nIt was great fun reading people\u0026rsquo;s T-shirts as I passed them; every one of them had a fascinating story to tell about why they were running, or who they were running for. Having never done this race before, I had a crazy notion that as the race progressed the density of the runners along the route would diminish as we went along. I am happy to report that this notion could not have been further from the truth. I spent the entire race trying to push on at least a little time wise and this meant pretty much constantly dodging left and right through gaps between people whenever possible.\nSpecial mention has to go to fellow runner, club member and fireman Steve Critten, who was running the race with a friend; both of them in full breathing apparatus and fire service gear. I passed them at about mile three just as the weather was warming up and they were looking pretty toasted.\nAs I progressed the number of walkers increased. There were so many people running along that these walkers became a real hazard. You just wouldn\u0026rsquo;t see them until the last second as two or three people in front of you parted to reveal a slow moving human being\u0026hellip; brakes on and dodge!\nAs is very well known about this race, the support from the sidelines was incredible; the route was lined with spectators on almost its entire length. People offered sweets, encouragement and all the kids wanted high fives. There was even a chap serving half pints of bitter to runners from a beer barrel.\nSo the miles progressed, and the crowds on the course and on the sidelines were endless and the atmosphere was brilliant.\nWith three miles to go I had half an hour to get under 2 hours. Plenty of time I thought, but how wrong I was! We arrived in South Shields where the race ended. The last two miles or so consist of a very long gentle slope up followed be a steep drop to the sea and then along the sea front to the finish. The support was probably the best here with the folk of South Shields being rightly proud to bring the race to an end. These last two miles, especially the final slope up saw many runners struggle and fight on with the supporting crowds encouraging them all the way.\nI pushed my way along the sea front to the finish and squeaked under the two hour mark by less than a minute. As I walked through the finish are to meet my club mates the Red Arrows made an appearance from over the sea; a fitting finale for such a great race experience.\nPros Sideline support probably second to none in the world.\nCons The success of the race can be measured in the size of the event. It is a truly enormous undertaking for the organisers, and a very interesting experience to have taken part in. It\u0026rsquo;s not a race to try and get a good half marathon time in, unless you are at the front where the field is considerably thinner.\nRace/Event Rating To organise such a large event and have it run like clockwork is a huge achievement. The sheer volume of participants and supporters make the numbers on the day mind boggling. A strong 4/5.\nLessons Learned As I said earlier, this was not a race I was trying to go for a personal best time in as the numbers taking part, and my position in the field made things difficult. I just took part and enjoyed the experience.\n","permalink":"https://chrisramsay.co.uk/posts/2015/09/2015-great-north-run/","summary":"\u003cp\u003eThis is a race report for the 2015 Great North Run Half Marathon\u003c/p\u003e\n\u003ch2 id=\"details\"\u003eDetails\u003c/h2\u003e\n\u003cp\u003eUseful information about the race.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eName\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe Great North Run, Tyneside, UK.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCategory\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll road half marathon. Probably one of the most famous half marathons in the world.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e13th September 2015\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTime\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMid morning, 10:45 start.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeather Conditions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWarming up as the morning went on, broken cloud.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCourse/Road Conditions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eRoad was mostly dry; course profile was largely flat with gentle undulations where there were bridges, underpasses and so on. Only a couple of miles before the end presented any kind of real slope.\u003c/p\u003e","title":"Race Report: 2015 Great North Run"},{"content":"This article mainly concerns the many trials of reliably bringing a remotely installed cloned Arduino and Wiznet W5100 Ethernet shield back to life after a power cut. The W5100 board is being explored as an alternative to the cheaper 28J60 board which itself ended up with its own reliability issues as described here in another article.\nPlease note that this particular article references DCcduinos only. The problems described within do not affect a genuine Arduino in my experience.\nThe Issue - Unreliable Reset I had experienced a great number of issues with keeping a reliable connection between my network enabled Arduino, using the ENC28J60 module, and the rest of my home network once systems were installed out in the field. After a seemingly random amount of time the system would drop off the network.\nThe only way to get round the issue was to cycle the power to the system to bring it back to life. Knowing that manual intervention would not be a viable option in the medium term, I went for another approach -giving the Arduino a heartbeat. The trigger for the heartbeat signal to be sent was whether or not the Arduino could ping the gateway IP address on my local network.\nThis worked flawlessly for nearly four months, and then system stability issues started to manifest themselves once again. Eventually the 28J60 was so frequently in a state of failure that the system spent a high percentage of its up time resetting itself.\nInitially, I thought that the cause might be connections expanding and contracting due to the fairly large temperature fluctuations in the box when the system was outside. I took a fairly conservative approach and soldered previously header pinned connections bit by bit, in an attempt to narrow down where the problem might lie but unfortunately all to no avail. I would always get a few hours, or even days of service before another random failure would occur.\nAfter a few weeks of research I finally managed to narrow it down to a possible current supply problem, where the 28J60\u0026rsquo;s is unable to draw enough current from the 3.3V regulator I had on the board, or that the voltage supplied to the system fluctuated a little; something to which the 28J60 was rather sensitive to. Either way, I read quite a few articles where people suggested a fix where supplying the 28J60 with 5 volts was a solution to these power problems.\nAccording to the Microchip data sheet it is does have 5 volt tolerant data inputs, but will run rather hot, but the data sheet specifies a 3.1V to 3.6V as a supply voltage. I suspected that some people may be confusing 5 volt logic level compatibility with the chip supply voltage.\nAn Alternative - Cloned Arduinos and Wiznet Shields I decided to go back and look at the reason why I had originally opted to use the 28J60; it was mainly a cost based decision, as the original Arduino Ethernet shields were rather expensive. Fast forward a couple of years and I decided to reconsider the Wiznet W5100 based shield.\nEarly experiments with the new Ethernet shield went very well indeed. My code only needed a very small modification to work with the new shield, and everything else was unaffected by the change.\nOr so I thought\u0026hellip;\nIt was about an hour or so of experimentation on the bench before I noticed that the DCcduino would not always appear on the network after a power cycle until I had pressed the reset button on the board. A quick observation of what was going on with the lights on the W5100 board confirmed this. Most of the time the on board green power LED and orange RJ45 speed LED would come up and stay lit with no further activity, almost like the whole thing had crashed. On the occasions that a proper start up happened, various on board LEDs would come on and the RJ45 LEDs would light up as per a normally functioning Ethernet connector.\nA little more investigation showed that the occasional successful start up only happened when the DCcduino was plugged into the computer with the USB cable. The same could not be said if I powered the DCcduino either via a plug in the wall USB charger or via a feed into the DC power input.\nIt seemed that the W5100 board was unable to start up without the reset signal sent from the computer USB connection; the DCcduino resetting it on its own was not enough.\nFinding Some (Non-Working) Solutions I spent a not inconsiderable amount of time looking for possible solutions to this problem. It was not long before I started to see that quite a lot of people experienced the same issue with a wide variety of combinations of Arduinos (and clones thereof) and WIZnet W5100 based Ethernet shields. Once again I must say that I never encountered any problems when using a genuine Arduino.\nBelow I have noted a selection of solutions that I came across in my research. Most require some type of hardware modification or addition, but not all. As can be seen from this section title, none of them actually worked for me when tested on the bench.\nTrigger Reset Pin Bend out or cut the reset pin on the Ethernet shield from the reset connector on the DCcduino, thus isolating the W5100 chip reset signal when the boards are connected together. Alternatively, and more drastically, cut the reset trace on the Ethernet shield. Then connect a wire between an output pin on the DCcduino (in the below case pin 3) and the reset pin on the W5100 shield.\n#include \u0026lt;SPI.h\u0026gt; #include \u0026lt;Ethernet.h\u0026gt; // The pin below must be connected to the reset on the shield int W5100_RESET_PIN = 3; byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; IPAddress ip(192,168,100,11); void setup() { Serial.begin(9600); Serial.println(); Serial.println(F(\u0026#34;node starting...\u0026#34;)); pinMode(W5100_RESET_PIN, OUTPUT); digitalWrite(W5100_RESET_PIN, LOW); delay(100); digitalWrite(W5100_RESET_PIN, HIGH); // give the Ethernet shield a second to initialize: delay(1000); Serial.print(F(\u0026#34;static IP...\u0026#34;)); Ethernet.begin(mac, ip); Serial.print(F(\u0026#34;local IP:\u0026#34;)); Serial.println(Ethernet.localIP()); Serial.println(F(\u0026#34;node started.\u0026#34;)); } Unfortunately, despite trying a variety of delay times, I had no success with this solution.\nCapacitor Delay The following solutions require either a modification on one of the boards or a very minimal amount of additional external circuitry to be constructed.\nCapacitor Between Reset and Ground Solder a capacitor between the reset pin and ground on the Ethernet shield. Capacitor values make a difference - 100nF means the shield resets every time but that renders the DCcduino unresponsive when uploading a program because the software reset no longer works. Reports are that 50nF or 470nF seems to be the ideal value.\nCapacitor Between Reset and Ground and a Resistor This variation is slightly more work than the above proposal; solder a 10 nF capacitor between the reset pin and ground on the Ethernet shield. As with the solution presented in Trigger Reset Pin, the reset pin on the W5100 shield requires isolation. Next, do one of the following:\n+5V Variation\nSolder a 10K resistor between the W5100 reset pin and the +5V supply. Tried all three powering options, i.e. via USB from the computer, USB from an AC adapter and raw power in. Only the USB from the computer worked occasionally; the other two options failed consistently.\n+3.3V Variation\nSolder a 10K resistor between the W5100 reset pin and the +3.3V supply. Again, tried all three powering options, i.e. via USB from the computer, USB from an AC adapter and raw power in. Only the USB from the computer worked occasionally; again, the other two options failed consistently.\nA Working Solution Thanks to some sterling work blogged about right here by a good friend of mine and fellow Arduino experimenter, John Burrin, I finally have a reliable solution. He also goes on to explain why he thinks that the W5100 board has a hard time getting out of bed. His idea is that the DCcduino powers up too quickly for the W5100 to get itself in a ready state. This therefore leads to a solution where the start up on the Arduino needs to be slowed down enough for the W5100 to be ready. He goes on to show this with running some experimental code before and after the application of an RC network to bridge the ground and network pins.\nTest circuit layout\nJohn\u0026rsquo;s test code relies on the fact that if the W5100 is able to start properly then the call server.begin() will succeed, and the code will not hang in the setup() loop. A successful completion of the setup means that the main loop() will run and the LED at pin 2 will flash at 1Hz. I have reproduced the test code below with some small modifications.\n#include \u0026lt;SPI.h\u0026gt; #include \u0026lt;Ethernet.h\u0026gt; int LED_PIN = 2 byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; IPAddress ip(192,168,100,21); // Initialise the Ethernet server library EthernetServer server(80); void setup() { // Initialise digital pin 2 as an output. pinMode(LED_PIN, OUTPUT); Ethernet.begin(mac, ip); server.begin(); } void loop() { // Turn the LED on (HIGH is the voltage level) digitalWrite(LED_PIN, HIGH); // Wait for a second delay(1000); // Turn the LED off by making the voltage LOW digitalWrite(LED_PIN, LOW); // Wait for a second delay(1000); } My findings have been a little different from John\u0026rsquo;s, but interesting nevertheless. I have found that even if server.begin() does fail the LED will still blink, meaning of course that loop() is being executed.\nThe experiments below have the network adapter configured with the ip address 192.168.100.21. I also disconnected the pin 2 LED in an attempt to only measure the current draw of the DCcduino (and shields).\nW5100 Without RC Network This is just the DCcduino with the W5100 network shield attached.\nLoop LED Behaviour\nFlashed at 1Hz once out of setup().\nNetwork Adapter Behaviour\nNetwork adapter LEDs flicker in unison occasionally, frequency undetermined. The only light that stays on constantly is the PWR LED.\nCommand Line Ping\nFirst 6 pings:\n$ ping 192.168.100.21 PING 192.168.100.21 (192.168.100.21): 56 data bytes Request timeout for icmp_seq 0 Request timeout for icmp_seq 1 Request timeout for icmp_seq 2 Request timeout for icmp_seq 3 ping: sendto: No route to host Request timeout for icmp_seq 4 ping: sendto: Host is down Request timeout for icmp_seq 5 ping: sendto: Host is down Power Supply\n9 volts via the 5.5mm DC input socket on the DCcduino; current usage on the bench supply oscillates between 170mA and 250mA with pin 2 LED disconnected.\nW5100 With RC Network This is the DCcduino, complete with the W5100 network shield and the prototype shield with an RC network built on it.\nLoop LED Behaviour\nFlashed at 1Hz once out of setup().\nNetwork Adapter Behaviour\nNetwork adapter LEDs steadily on (PWR, LNK, 100M, FULLD) and the TX RX LEDs light on ping response.\nCommand Line Ping\nFirst 6 pings:\n$ ping 192.168.100.21 PING 192.168.100.21 (192.168.100.21): 56 data bytes 64 bytes from 192.168.100.21: icmp_seq=0 ttl=128 time=7.868 ms 64 bytes from 192.168.100.21: icmp_seq=1 ttl=128 time=4.131 ms 64 bytes from 192.168.100.21: icmp_seq=2 ttl=128 time=1.891 ms 64 bytes from 192.168.100.21: icmp_seq=3 ttl=128 time=2.721 ms 64 bytes from 192.168.100.21: icmp_seq=4 ttl=128 time=0.809 ms 64 bytes from 192.168.100.21: icmp_seq=5 ttl=128 time=0.855 ms Power Supply\n9 volts via the 5.5mm DC input socket on the DCcduino; current usage on the bench supply steady at 190mA with pin 2 LED disconnected.\nOnly Time will Tell I am hoping that finally, once again thanks to John for putting me on the right track, I have a viable solution to reliably leaving remotely network connected Arduinos to get on with their business. Early indications with the RC network fix are promising, however as the saying goes\u0026hellip; only time will tell!\n","permalink":"https://chrisramsay.co.uk/posts/2015/08/some-fun-with-a-cloned-arduino-and-w5100-ethernet-shield/","summary":"\u003cp\u003eThis article mainly concerns the many trials of reliably bringing a remotely installed cloned Arduino and Wiznet W5100 Ethernet shield back to life after a power cut. The W5100 board is being explored as an alternative to the cheaper 28J60 board which itself ended up with its own reliability issues as described \u003ca href=\"/posts/2015/04/giving-an-arduino-a-heartbeat/\"\u003ehere in another article\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003ePlease note that this particular article references \u003cem\u003eDCcduinos\u003c/em\u003e only. The problems described within do not affect a genuine Arduino in my experience.\u003c/p\u003e","title":"Some Fun with a Cloned Arduino and W5100 Ethernet Shield"},{"content":"The Problem How would one be able to restart a system that has been deployed in a remote area which gives no immediate access for human intervention? The answer is to give the system an ability to reset itself in case of a system crash.\nThis is where a watchdog timer (WDT) comes into use. It is a hardware based timer capable of automatically resetting a system should the main program fail to periodically send out a heartbeat. Now, the Arduino does indeed have a WDT on board, as do probably all microcontrollers, but a small amount of reading will inform you that once it has started running a sketch, there is no software function that has the same effect as pressing the reset button.\nI\u0026rsquo;ll admit that arranging for the possibility that you may want to have external WDT capability in your program is not a run of the mill requirement. This changes, however, when you start to introduce things like supporting external boards, such as Ethernet modules like the ENC28J60. There are some situations where a small power fluctuation can cause a module such as that to crash, and in that case a reset is the only option for recovery.\nThere are many examples where people have tried to address resetting the Arduino using just software and, for example, connecting one of the I/O pins to the reset pin to trigger the reset. However, ATMEL themselves, in their ATmega328 datasheet, describe the reset function as requiring a pulse width of at least 2.5µs, and that may not be possible with self shunting the Arduino. This is because as soon as the reset is activated, all the Arduino\u0026rsquo;s I/O pins switch to input mode, meaning that making a long enough reset pulse is not guaranteed.\nSo, how about an external circuit as a solution? As it turns out, it was easier than I thought it would be. Read on\u0026hellip;\nAn External Solution One solution for this problem is to have an external circuit act as a watchdog with the capability of activating the Arduino\u0026rsquo;s reset port when the conditions dictate. After a fair amount of reading on the subject it seems that most sensible suggestions are based around an Astable Oscillator implementation using the popular, and almost infinitely versatile 555 timer IC1.\nThe Circuit Below is a circuit diagram containing an Arduino Pro Mini, a 555 based astable oscillator, and a heartbeat LED indicator. This solution only really has to tie up one I/O pin on the Arduino which will function as the pulse output; the other I/O pin just lights up a pretty heartbeat LED - entirely dispensable if you are pushed for I/O real estate.\nArduino 555 Watchdog Timer Circuit Diagram\nRead on for an outline on how the circuit works.\nHow it Works Basically, I am using the above circuit as a countdown timer that restarts itself once it gets to zero, with an interrupting reset facility added in. So let\u0026rsquo;s see what the circuit does:\nWhen the circuit is powered up, pin 3 (Q) goes high and the capacitor C2, begins to charge, taking current from VCC on the Arduino via the resistors R1 and R2. Note that the values of resistors R1 and R2 dictate the time taken for C2 to charge.\nIf the voltage in the capacitor is ever allowed to reach 2/3 of the VCC voltage, the 555 pulls down pin 7 (DIS), in turn discharging C2 through R2 until it drops to 1/3 of VCC\u0026rsquo;s voltage. Note that the value of R2 dictates how long it takes for C2 to discharge.\nAt this point the 555 stops discharging and the whole cycle begins again - this is the countdown timer having reached zero, and starting the countdown all over again.\nOnce the discharge of C2, is complete, the connection between pin 3 on the 555 and RESET on the Arduino is pulled low, thus resetting the Arduino through the diode D1.\nIn order to prevent the 555 from constantly charging and discharging it needs a periodical reset, and this is where pin D2 on the Arduino is used. During normal operation, pin D2 is held high, but in order to reinitialise the timer, pin D2 is made to go low, thus discharging C2 through the resistor R3. This, of course, interrupts the charge cycle of the 555 and takes it back to the start point.\nSo the trick to keep the Arduino from resetting and thus in a steady state is to make sure that the capacitor C2 never reaches 2/3 VCC voltage by dropping pin D2 in good time. The moment that pin D2 stops dropping, perhaps because of a crash in the software running on the Arduino, C2 reaches the critical 2/3 VCC voltage and the discharge process as described above begins.\nRegarding Timing Some notes regarding charge/discharge time for the watchdog circuit. I was looking for a timeout of somewhere between 10 and 20 seconds so my component values are thus reflected below.\nC2 Charge Time The formula for calculating charge time in the watchdog circuit, using standard astable 555 maths, is as follows:\nTcharge=0.693(R1+R2)CT_{charge}=0.693(R_{1}+R_{2})CSo for the values of R1, R2 and C2 I have chosen, the maximum time the circuit will wait for a heartbeat, before resetting the Arduino, will be:\nTcharge=0.693(1000000+3300)0.000022=15.3secsT_{charge}=0.693(1000000+3300)0.000022 = 15.3 secsC2 Discharge Time The formula for calculating discharge time in the watchdog circuit, again using standard astable 555 maths, is below:\nTdischarge=0.693R2CT_{discharge}=0.693 R_{2} CWith the values of R2 and C2, we have:\nTdischarge=0.693×3300×0.000022=0.05secsT_{discharge}=0.693 \\times 3300 \\times 0.000022 = 0.05 secsThe above is the minimum time the Arduino will have spend with the pulse pin D2 pulled low in order to sink sufficient current from the capacitor C2 to ensure that it is at 1/3 charge.\nTesting it Out I decided to prototype the watchdog circuit on a breadboard using the Arduino Pro Mini 5v - a favourite of mine at the moment.\nBreadboard layout Below is the breadboard layout - nothing fancy apart from the heartbeat LED!\nArduino 555 Watchdog Timer Circuit Breadboard Assembly\nA Test Program The code below is fairly straightforward; just loop and count 10 seconds before issuing another heartbeat to the watchdog. This is just a basic script to test the functionality of the watchdog; there are myriad reasons you might enable a heartbeat to be sent, such as: a resource being available on a network, or a signal on one of the I/O pins. The choice of application is yours.\nint pulsePin = 2; int led = 4; unsigned long lastHeartbeat = 0; unsigned long lastUptimeReport = 0; void setup() { Serial.begin(9600); Serial.println(\u0026#34;*** Arduino reset ***\u0026#34;); pinMode(led, OUTPUT); // Send an initial heartbeat. heartbeat(\u0026#34; \u0026lt;3 Initial heartbeat\u0026#34;); } void loop() { unsigned long uptime = millis(); // Every 10 seconds issue a heartbeat... // ...unless you have a reason not to! if ((uptime - lastUptimeReport) \u0026gt;= 10000) { lastUptimeReport = (uptime - (uptime % 10000)); heartbeat(\u0026#34; \u0026lt;3 Still alive!\u0026#34;); } // delay in between loops delay(500); } /** * Utility function to handle heartbeat pulse generation * LED and a serial message */ void heartbeat(String message) { // Sink current to drain charge from C2 pinMode(pulsePin, OUTPUT); digitalWrite(pulsePin, LOW); // LED on digitalWrite(led, HIGH); // Give enough time for C2 to discharge (should discharge in 50 ms) delay(300); // Return to high impedance pinMode(pulsePin, INPUT); // LED off digitalWrite(led, LOW); lastHeartbeat = millis(); Serial.println(message); } Test Plan So the idea of the test was to see what happened when I simulated a failure of some sort by stopping the heartbeat pulse from the Arduino to the watchdog. I was hoping that once the watchdog lost the pulse, it would wait a maximum of about 15 seconds before resetting the Arduino. I would expect the watchdog to keep resetting the Arduino until it got a heartbeat response back. I would also expect that a simple reset of the Arduino, by pressing the on board reset button would not adversely affect things.\nThus the test plan was as follows:\nStart the Arduino See it run normally with no reset After 60 seconds, disconnect D2 (the pulse pin) See it reset every 15 seconds or so After 120 seconds, reconnect D2 See it run normally with no reset After 180 seconds, press reset on the Arduino See it run normally with no reset End test Below is an annotated extract of a test run from the serial console:\n\u0026lt;\u0026lt; Start \u0026gt;\u0026gt; )--- (1) *** Arduino reset *** | \u0026lt;3 Initial heartbeat | \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;3 Still alive! |--- (2) \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;\u0026lt; Disconnected D2 after 60 secs \u0026gt;\u0026gt; )--- (3) \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;3 Still�*** Arduino reset *** | \u0026lt;3 Initial heartbeat | \u0026lt;3 Still alive! | *** Arduino reset *** |--- (4) \u0026lt;3 Initial heartbeat | \u0026lt;3 Still alive! | *** Arduino reset *** | \u0026lt;3 Initial heartbeat | \u0026lt;\u0026lt; Reconnected D2 after 120 secs \u0026gt;\u0026gt; )--- (5) \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;3 Still alive! |--- (6) \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;\u0026lt; Pressed reset after 180 secs \u0026gt;\u0026gt; )--- (7) *** Arduino reset *** | \u0026lt;3 Initial heartbeat | \u0026lt;3 Still alive! | \u0026lt;3 Still alive! |--- (8) \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;3 Still alive! | \u0026lt;\u0026lt; End \u0026gt;\u0026gt; )--- (9) All in all, after a few days of fiddling about, testing and more fiddling about I felt that I reached a point where I was happy enough with the circuit to go ahead and make a prototype on veroboard.\nThe Final Build The intention of this watchdog timer project was to get the circuit retrofitted to an already installed Arduino, so I had to allow for two sets of male breakaway header pins to be added. This would enable me to be present the watchdog as a daughter board to the installation.\nBelow is an image of the completed watchdog timer. Note that the use of header pins means that the business part of the circuit is off to one side. The grey and white flying lead is for the heartbeat LED.\nWatchdog Module for the Arduino Pro Mini\nLuckily, I had already planned for the possibility of retrofitting a circuit to my Arduino Pro Mini, so I had soldered in some long tailed female header connectors. This meant that fitting the watchdog was a simple case of \u0026lsquo;plug it in\u0026rsquo;.\nWatchdog Module Retrofitted to the Arduino Pro Mini\nAbove you can see the Arduino up and running with the watchdog circuit retrofitted.\nIn the Field After a few weeks being plagued by crashes from a remote installation (my soil temperature measuring project, in case you are interested) using an Arduino with an ENC28J60 Ethernet module, I became fed up with the drag of having to recycle the power supply and the gaps in the data I was gathering. This drove me to researching watchdog circuits and hence the work in this article. I am happy to report that since the installation of the watchdog daughter board I have had some four months of uninterrupted work from the installation.\nOkay, so I love these ICs - I spent many an hour abusing these in my youth.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://chrisramsay.co.uk/posts/2015/04/giving-an-arduino-a-heartbeat/","summary":"\u003ch2 id=\"the-problem\"\u003eThe Problem\u003c/h2\u003e\n\u003cp\u003eHow would one be able to restart a system that has been deployed in a remote area which gives no immediate access for human intervention? The answer is to give the system an ability to reset itself in case of a system crash.\u003c/p\u003e\n\u003cp\u003eThis is where a \u003ca href=\"http://en.wikipedia.org/wiki/Watchdog_timer\"\u003ewatchdog timer\u003c/a\u003e (WDT) comes into use. It is a hardware based timer capable of automatically resetting a system should the main program fail to periodically send out a heartbeat. Now, the Arduino does indeed have a WDT on board, as do probably all microcontrollers, but a small amount of reading will inform you that once it has started running a sketch, there is no software function that has the same effect as pressing the reset button.\u003c/p\u003e","title":"Giving an Arduino a Heartbeat"},{"content":"This is a race report for the 2015 Caythorpe Dash.\nDetails Useful information about the race.\nName\nThe Caythorpe Dash, Caythorpe, Lincolnshire, UK.\nCategory\nMostly off-road/trail/track half marathon. 4 person (2 man, 2 woman) teams with aggregated times or solo.\nDate\n1st February 2015\nTime\nMid morning, 11:00 start.\nWeather Conditions\nDry, cold air temperature (only just above freezing) with a fresh northerly breeze bringing in quite a wind chill. Broken cloud.\nCourse/Road Conditions\nRoad was mostly dry; trail and track were occasionally muddy with puddles. Course profile was largely flat for the first 16 km after an initial drop, followed by a hillier last 5 km or so.\nTaking Part Details on taking part in the race.\nThings I liked The race took place in quite a pleasant part of the county that I was not too familiar with. The roads on which the race occasionally happened upon were very quiet. The food provided at the end was amazing.\nObjectives Start slowly and not worry too much about position in the race. Move up the ranks later if I still had the energy.\nGoals Being that this was my first half marathon, it was a case making it to the end. My previous longest distance had been 17 kilometers.\nExperience Right from the start I really enjoyed this race; it was to be a really positive experience from beginning to end. It was the first event where I was taking part as a member of a four person team, my club having entered a total of four teams. In terms of strategy, the team I was in had one simple aim - every team member to make their own way to the end. That was fine by me as I did not want to be slowing anyone else down!\nThe first leg of the race headed out across fields in a vaguely easterly direction. For pretty much the entire duration of this leg we were accompanied by a biting northerly wind, which after a while, started to make my face feel like it was freezing. Even though I had bare legs (and got the occasional incredulous comment regarding this), they remained warm as they were doing all the work. There was the occasional bit of relief when, sheltered from the wind by a hedge or a stand of trees, one could feel the vague warmth of a weak and watery winter sun. We continued along an assortment of mud tracks, field edges and minor roads until it came time to turn left towards the south.\nThe next leg went from track to minor road and rose into the small village of Stubton where water was being dispensed by a couple of particularly hardy looking marshals, given the conditions. I managed to both drink and breathe the water, which gave them something to chuckle about. From there, road sloped gently away down the rise and from the village and I decided to step up the pace a little to try and make a little time up while the going was good. This leg did not last too long before turning vaguely eastwards, where the cold wind was once more cutting at right angles across my onward progress.\nIt was on this longer leg that the race took on the feel of a cross country again, with mud tracks and the edges of ploughed fields forming the majority of the terrain underfoot. It was in this direction too, that the Lincoln Edge began looming into view, and I was left thinking that at some point a certain amount of vertical ascent was going to be required. Approaching the hill, I caught sight of a pair of Caistor Running Club vests in the distance and decided to put my foot down and catch them up. It really mentally lifted me to see some familiar faces after quite a long time without. We trudged up the hill together, comparing notes on the run so far. I think that the consensus was \u0026lsquo;good fun so far, and looking forward to the sausages, crumble and bread \u0026amp; butter pudding\u0026rsquo;. It was at this point that I realised that I was now in uncharted territory as far as distance was concerned: I had run as far as I had ever run in my life, and there were still kilometres to go.\nFrom the top of the hill, I followed a small and winding lane flanked by trees up to the small hamlet of Gelston. This, then, formed the start of the final north-easterly leg of the race and elevation-wise was the highest point on the course at a staggering 85 metres according to the GPS. The route cut across a small depression with a river and a footbridge at the bottom before climbing out into more open gently undulating fields. At this point the cold and penetrating wind had started to become a defining feature once again. Crossing a couple of stiles brought me to the aptly named hill top village of Hough-on-the-Hill. Here was another water stop, near the village church, manned by some more hardy volunteers, and water was once again gratefully received.\nThe final leg continued through the church yard, down a cobbled alley and on to a road out of the village for the final 2.5 km across the fields towards the finish line in Caythorpe. I think that this last stretch was probably the hardest of the lot, with quite a head wind and a number of stiles to cross; one of the stiles collapsed under my weight and I caught myself on some barbed wire, but no damage done! Finally, Caythorpe came into view and with it a small burst of energy. A couple of streets and an alleyway later and it was all over. I\u0026rsquo;d crossed the line in 2 hours and 6 minutes with some great support from a happy bunch of CRC runners at the finish, feeling very pleased that I\u0026rsquo;d managed to make the distance.\nPros It was my first HM and the support from my CRC friends was superb - I couldn\u0026rsquo;t have asked for more. The sausages and bread \u0026amp; butter pudding were just the right thing after a two hour run in the cold. The ladies serving the food and drink in the Caythorpe cricket pavilion did a wonderful job, and the event would be a far poorer affair without their support.\nCons I would struggle to think of any cons at this point, so I\u0026rsquo;ll pass.\nRace/Event Rating The excellent post race food, marshalling and course organisation gives this event a healthy 5/5.\nLessons Learned I feel that my strategy to make up some time on the road sections of this race worked quite well. Being that this was my first HM, I was a little tentative and conscious not to burn out too early. Knowing what I do now, I hope that I can take on a fully road based HM in well under two hours.\n","permalink":"https://chrisramsay.co.uk/posts/2015/02/2015-caythorpe-dash/","summary":"\u003cp\u003eThis is a race report for the 2015 Caythorpe Dash.\u003c/p\u003e\n\u003ch2 id=\"details\"\u003eDetails\u003c/h2\u003e\n\u003cp\u003eUseful information about the race.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eName\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe Caythorpe Dash, Caythorpe, Lincolnshire, UK.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCategory\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMostly off-road/trail/track half marathon. 4 person (2 man, 2 woman) teams with aggregated times or solo.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e1st February 2015\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTime\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eMid morning, 11:00 start.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWeather Conditions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDry, cold air temperature (only just above freezing) with a fresh northerly breeze bringing in quite a wind chill. Broken cloud.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCourse/Road Conditions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eRoad was mostly dry; trail and track were occasionally muddy with puddles. Course profile was largely flat for the first 16 km after an initial drop, followed by a hillier last 5 km or so.\u003c/p\u003e","title":"Race Report: 2015 Caythorpe Dash"},{"content":"The Problem Since getting a Garmin Forerunner 610 I have been largely satisfied with the various aspects of getting the data from the said watch on to the Internet. It is a multi stage process, each with risks and issues very much dependent on non functional qualities, such as network and USB resilience, usability and so on. To give an example, I get the data from my watch using a USB ANT+ stick, Mac computer, Garmin\u0026rsquo;s Express software, Garmin Connect online service and finally, Strava via Strava\u0026rsquo;s linked account capability. That\u0026rsquo;s a lot of steps, and a lot of potential failures lying in wait. I am not going to get into my rather low estimation of both Garmin\u0026rsquo;s Connect service and Express software; needless to say, I think they are both rather substandard at the moment.\nNow the usual method of getting data from the watch is a trivial affair when everything just works: wake up the Mac, plug in the ANT stick, fire up Garmin Express and put the watch within the vicinity. Just recently I had noticed that the occasional run session would not upload to the computer. The transfer progress bar would appear on the watch and then disappear almost immediately with the Garmin Express software announcing that the latest workout was uploaded to Garmin Connect.\nA small amount of research will tell you that, whilst Garmin gear is great when it works, there is frustratingly little in the way of support or information when things go wrong.\nA Solution Bear in mind (in case it\u0026rsquo;s not obvious) that this article is written in the context of someone who is using a Unix / Linux based operating system. If you are running Windows then I am sorry - you have my sympathy.\nIn the somewhat unlikely event that things are going wrong, and all you want to do is get the data off that damn watch, there is a solution to your woes. Bear with me, however, there\u0026rsquo;s going to be some command line work involved\u0026hellip;\nSo, first things first, fire up a terminal session.\nGo ahead and make yourself a directory to do all the work in:\n$ mkdir ~/Documents/garmin $ cd ~/Documents/garmin Now get the software you are going to need to get the job done - this involves downloading (cloning) some code from Github:\n$ git clone https://github.com/Tigge/openant.git $ cd openant $ sudo python setup.py install $ cd .. openant is a library that is a dependency for the Garmin Forerunner Extractor software. Below, you can see how you clone the code, install it and make it executable.\n$ git clone https://github.com/Tigge/Garmin-Forerunner-610-Extractor.git GFR610 $ cd GFR610 $ sudo python setup.py install $ chmod u+x antfs-cli.py Now just run it as a test it to make sure that it works:\n$ ./antfs-cli.py --help usage: antfs-cli.py [-h] [--upload] [--debug] [--pair] [-a] Extracts FIT files from ANT-FS based sport watches. optional arguments: -h, --help show this help message and exit --upload enable uploading --debug enable debug --pair force pairing even if already paired -a, --skip-archived don\u0026#39;t download files marked as \u0026#39;archived\u0026#39; on the watch Now get your ANT usb stick and plug it in to any port on your machine and then run the program again. Now start up your watch and put it somewhere nearby. Next, just watch the output in the terminal:\n$ ./antfs-cli.py Driver available: [\u0026lt;class ant.base.driver.SerialDriver at 0x100c69ae0\u0026gt;, \u0026lt;class ant.base.driver.USB2Driver at 0x100d692c0\u0026gt;, \u0026lt;class ant.base.driver.USB3Driver at 0x100d69328\u0026gt;] - Using: ant.base.driver.USB3Driver Request basic information... Capabilities: array(\u0026#39;B\u0026#39;, [8, 8, 0, 186, 54, 0, 223]) Starting system... Key done... Searching... Authenticating with Forerunner 610 (3890343564) - Pairing: OK Downloading 7 file(s) Downloading 1989-12-31_00-00-00_1_65535.fit: [..............................] ETA: 0:00:00 Downloading 1989-12-31_00-00-00_2_65535.fit: [..............................] ETA: 0:00:00 Downloading 1989-12-31_00-00-00_3_3.fit: [..............................] ETA: 0:00:00 Downloading 1989-12-31_00-00-00_3_1.fit: [..............................] ETA: 0:00:00 Downloading 2015-01-06_20-40-04_4_28.fit: [..............................] ETA: 0:00:00 Downloading 1989-12-31_00-00-00_9_65535.fit: [..............................] ETA: 0:00:00 Downloading 1989-12-31_00-00-00_10_65535.fit: [..............................] ETA: 0:00:00 You\u0026rsquo;ll notice that there is a point where the terminal output mentions \u0026lsquo;Searching\u0026rsquo; and then \u0026lsquo;Authenticating ......\u0026rsquo;. At this point the watch will ask you if you wish to pair the watch, at which point you select \u0026lsquo;yes\u0026rsquo; on the watch screen.\nOnce that is successful, you will see the files start to download. You will notice that there is a series of files with names beginning 1989-12-31_00-00-00 - these are not workouts you did many years ago; they are system files which will appear each time you run this process and can be safely ignored.\nFinally, all the files that have downloaded successfully are placed in a directory in your home directory, named .config/antfs-cli/nnn, where nnn is, I presume, the device identifier of your watch. The files you will be interested in are those in the activities directory.\n$ ls -al ~/.config/antfs-cli/3890343564/*/* -rw-r--r-- 1 chrisramsay staff 65363 7 Jan 11:33 activities/2015-01-06_20-40-04_4_28.fit -rw-r--r-- 1 chrisramsay staff 46457 13 Jan 22:11 activities/2015-01-09_07-15-36_4_30.fit -rw-r--r-- 1 chrisramsay staff 10255 13 Jan 22:12 activities/2015-01-09_08-07-18_4_31.fit -rw-r--r-- 1 chrisramsay staff 9624 13 Jan 22:12 activities/2015-01-09_17-00-52_4_32.fit -rw-r--r-- 1 chrisramsay staff 48803 13 Jan 22:13 activities/2015-01-09_18-34-28_4_33.fit -rw-r--r-- 1 chrisramsay staff 61218 13 Jan 22:14 activities/2015-01-13_19-57-02_4_34.fit -rw-r--r-- 1 chrisramsay staff 3586 13 Jan 22:14 activities/2015-01-13_20-09-52_4_35.fit -rw-r--r-- 1 chrisramsay staff 573 7 Jan 11:28 settings/1989-12-31_00-00-00_2_65535.fit -rw-r--r-- 1 chrisramsay staff 1213 7 Jan 11:28 sports/1989-12-31_00-00-00_3_1.fit -rw-r--r-- 1 chrisramsay staff 1213 7 Jan 11:28 sports/1989-12-31_00-00-00_3_3.fit -rw-r--r-- 1 chrisramsay staff 125 7 Jan 11:33 totals/1989-12-31_00-00-00_10_65535.fit -rw-r--r-- 1 chrisramsay staff 72 7 Jan 11:33 weight/1989-12-31_00-00-00_9_65535.fit So now you can go ahead and grab those .fit files and upload them directly to Strava via their \u0026lsquo;add manual entry\u0026rsquo; page. Note that these files will contain heart rate data if you used your heart rate monitor strap during your run.\n","permalink":"https://chrisramsay.co.uk/posts/2015/02/getting-data-from-a-garmin-fr610/","summary":"\u003ch2 id=\"the-problem\"\u003eThe Problem\u003c/h2\u003e\n\u003cp\u003eSince getting a Garmin Forerunner 610 I have been largely satisfied with the various aspects of getting the data from the said watch on to the Internet. It is a multi stage process, each with risks and issues very much dependent on non functional qualities, such as network and USB resilience, usability and so on. To give an example, I get the data from my watch using a \u003ca href=\"http://www.dcrainmaker.com/2012/02/much-better-usb-ant-stick-for-your.html\"\u003eUSB ANT+ stick\u003c/a\u003e, Mac computer, Garmin\u0026rsquo;s Express software, \u003ca href=\"http://connect.garmin.com/\"\u003eGarmin Connect online\u003c/a\u003e service and finally, \u003ca href=\"http://www.strava.com\"\u003eStrava\u003c/a\u003e via Strava\u0026rsquo;s linked account capability. That\u0026rsquo;s a lot of steps, and a lot of potential failures lying in wait. I am not going to get into my rather low estimation of both Garmin\u0026rsquo;s Connect service and Express software; needless to say, I think they are both rather substandard at the moment.\u003c/p\u003e","title":"Getting Data from a Garmin FR610"},{"content":"Final Assembly Some would say that at this stage I was almost there with the soil sensor project. All there was left to do was get the thing into the ground. Being that I am a flawed perfectionist, I had some other ideas . I was not going to simply chuck it in the ground without some more prevarication.\nAnyway, onwards\u0026hellip;\nThe next step in the assembly was to attach all the sensor cables to the main printed circuit board in the box. Apart from spending much time getting rubber grommets into place in the holes through the box, this was fairly straightforward. As can be seen below, this is how things looked once all the sensors were cabled up.\nSensors cabled up - ready for attachment to wooden post\nOnce this step was complete, it was time to move on to the burial (site Installation) phase.\nSite Installation I decided early on that I would attach the box with the electronics inside to a post in the ground in close proximity to the sensors themselves, rather than have the sensors a long way from the electronics on long cables. This was to minimise the amount of cable trailing around the garden and have the advantage of other on-site measurement possibilities, such as minimum concrete and grass temperatures.\nThe wood work involved in making a post and small cross bar to attach the sensors to was very trivial indeed. Probably the most time consuming exercise was the dousing of all the wooden parts in wood preserver and the consequent wait whilst the parts dried.\nThere was a small moment of guilt when I returned to the site where I had originally dug the one meter deep hole for the sensors. Being that the excavation had taken place six months ago, and I had taken care to cover the hole with a concrete slab, I was not expecting to find any visitors within. Much to my sadness, a small number of frogs had managed to get down the hole and starve to death. Oh well, I thought, at least they will receive a decent burial.\nSystem at the final burial stage\nRegarding power supply and network connectivity I went down the same route as discussed here in part four of the cloud detection project: a CAT 5 cable delivering network, with two of the four unused cores delivery a supply voltage and a ground connection.\nYou may have noticed that there is a slant cut into the top of the post which is driven into the ground. This will eventually be the fixing place for a small roof to keep the worst of the rain from directly falling on the box of electronics.\nApart from the roof, the final step for system installation was the addition of the minimum grass and concrete temperature sensors.\nConcrete and grass temperature sensors installed\nAs part of the site preparation for the sensors I created a wooden frame from some scrap wood. I thought that the frame would keep the area looking a little tidier, would be easier to manage with respect to keeping weeds at bay and would provide a nice edge for the concrete slab and pieces of fake grass to butt up against.\nInitial Start Up Well, I am glad to report that due to the hours of prototyping and bench testing the system started up first time, with no hitches whatsoever. This was, more or less, what I was expecting and anything else would have been a disappointment. In addition to seeing that all sensors were connected and functioning, the system communicated with the rest of my network immediately and without error. So far, so good, you might say.\nFast forward two and a half days later and we have a slightly different story\u0026hellip;\nI set up a cron job on a local server to grab the data from the system every 15 minutes, and up to the time the system stopped responding, it had been flawless. Once the system stopped responding, that was it - no more data until a hard restart was made on the system (for that, read power cycle!). After a little research it looks as though the ENC28J60 module locks up for an indeterminate reason, becoming completely unresponsive until a reset is performed.\nIn short, there\u0026rsquo;s some investigation to be done. Until next time!\n","permalink":"https://chrisramsay.co.uk/posts/2014/11/soil-temperature-monitoring-part-four/","summary":"\u003ch2 id=\"final-assembly\"\u003eFinal Assembly\u003c/h2\u003e\n\u003cp\u003eSome would say that at this stage I was almost there with the soil sensor project. All there was left to do was get the thing into the ground. Being that I am a flawed perfectionist, I had some other ideas . I was not going to simply chuck it in the ground without some more prevarication.\u003c/p\u003e\n\u003cp\u003eAnyway, onwards\u0026hellip;\u003c/p\u003e","title":"Soil Temperature Monitoring - Part Four"},{"content":"Reviving the Project A very brief part three of the, somewhat delayed, soil temperature monitoring project. This project has been left a little on the side gathering dust whilst I spent some time concentrating on the cloud detection project, but I am now determined to get this system up and running and its sensors in the ground before the winter weather sets in. Being in the ground soon will also mean that I get a good run before the cold weather hits.\nAt this point I have progressed with getting the electronics from breadboard to circuit board, well, Veroboard anyway. Also, the mission has crept slightly: after some more research, I have decided that the project is to become more of a surface/ground temperature station with the addition of minimum grass and concrete temperature measurements.\nExtra measurements Seeing as I was going to go to all the trouble of putting a box of electronics in the garden, I thought that it would be a good opportunity to make the most of it and add a couple of extras into the mix.\nGrass Minimum Temperature Grass minimum temperature is a useful property to keep an eye on, especially if you have a garden; a grass temperature of 0°C or less is an excellent indicator of ground frost. As from the Met Office\u0026rsquo;s surface observations fact sheet the method of measuring grass minimum temperature is thus described:\nFor many years the grass minimum temperature has been defined as the lowest overnight temperature measured by a thermometer, fully exposed to the open sky, suspended horizontally over an area covered with short cropped turf and in contact with the tips of grass blades\nNow many automated weather stations these days do not go quite that far; to attain a grass measurement, thermometers are now suspended 35 mm over an artificial grass surface to just about the same effect. I am going to aim to do something similar with a packaged DS18B20 suspended over a strip of cheap fake grass and see what happens.\nConcrete Minimum Temperature As with grass minimum temperature, concrete minimum temperature is useful if you want to know when ice has started to form, particularly if you happen to be needing to drive anywhere in the winter. It is certainly not a difficult measurement to take, as it will consist of one packaged DS18B20 and a small slab of home made concrete. Basically, I will lay a slab of concrete down on the ground so that it is almost level with the surrounding grass, making sure that the surface is completely exposed to the open sky (no overshadowing trees or whatever). I\u0026rsquo;ll then need a couple of small strips of metal to use as bindings to ensure that the DS18B20 is fully in contact with the concrete.\nConstruction The Circuit As an outline idea, I had the circuit design in my head for a few months before wires were laid onto breadboard. I wanted a bi-state LED to indicate sensor status: green for OK and red for a problem; the reason being that once the soil sensors were underground it would be very useful to have an at-a-glance checking system. As above, I also decided that the circuit should be able to support up to five DS18B20 sensors (three for ground temperature, one for concrete temperature and one for grass temperature)1.\nOnce committed to breadboard, it was but a short hop to getting the design on to some Veroboard. A few evenings of fun with solder later, and it lives!\nThe circuit on a burn in cycle, sort of.\nAs can be seen from the above photograph, the left hand part is the built circuit (not as yet cut out from the rest of the Veroboard); in the center is a small breadboard with a few of the warning lights (all green because all the sensors are working fine); on the right hand side is the ENC28J60 module which will handle the communication over Ethernet.\nIn terms of managing on board power things are slightly complicated. The Arduino itself is a 5 volt model: an Arduino Pro Mini 328 - 5V/16MHz, and all the DS18B20 sensors are fine with 5 v also. However the ENC28J60 module is 3.3 v only, so it became a case of having to have multiple voltage regulators on the board. Not overly tricky, just a case of being careful as to where I would be routing power.\nAs an initial test I decided to \u0026lsquo;burn in\u0026rsquo; the system on a 12 volt supply, and thankfully that was a success. I did not feel like I was over-supplying the circuit too much; just enough to get things nice and warm for a while and ensure that there was to be no smoke or fireworks. Once out in the field, I think that the supply voltage will be nearer 6 volts, depending on the length of cable, and hence voltage drop, between system and supply.\nI chose to use sets of flying leads made from ribbon cable to route power, data and other things between the three separate circuit boards. The leads are all connected using 2.54 mm Dupont connectors rather than being soldered straight into the boards. I felt that this would enable me to dive in and repair or replace bits fairly easily in the future.\nBoxing it Up Choice of project box for this system was fairly limited; I wanted something reasonably hard wearing, weather proof and easy to work with. To my utter gratitude I found that my local DIY store stocked a range of ABS weather proof outdoor junction boxes - decision made. As for power and communications: the decision was made to go with a home made version of power over Ethernet, something I\u0026rsquo;ve since started to build into my cloud detection project - see here in part four for a further description.\nAnother overnight test of the system in its new home\nSo that\u0026rsquo;s it for now; the next step is to get the whole setup outside. I have a feeling that there will be a whole load of challenges to get over before this system is in the ground and doing its thing, unencumbered by me making fixes to the hardware or software.\nI have since added a sixth sensor to give me the internal box temperature.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://chrisramsay.co.uk/posts/2014/11/soil-temperature-monitoring-part-three/","summary":"\u003ch2 id=\"reviving-the-project\"\u003eReviving the Project\u003c/h2\u003e\n\u003cp\u003eA very brief part three of the, somewhat delayed, soil temperature monitoring project. This project has been left a little on the side gathering dust whilst I spent some time concentrating on the \u003ca href=\"/tag/cloud-detection/\"\u003ecloud detection project\u003c/a\u003e, but I am now determined to get this system up and running and its sensors in the ground before the winter weather sets in. Being in the ground soon will also mean that I get a good run before the cold weather hits.\u003c/p\u003e\n\u003cp\u003eAt this point I have progressed with getting the electronics from breadboard to circuit board, well, \u003ca href=\"http://en.wikipedia.org/wiki/Veroboard\"\u003eVeroboard\u003c/a\u003e anyway. Also, the mission has crept slightly: after some more research, I have decided that the project is to become more of a surface/ground temperature station with the addition of minimum grass and concrete temperature measurements.\u003c/p\u003e","title":"Soil Temperature Monitoring - Part Three"},{"content":"Moving from Map My Run to Strava I have been running a little while now and am always in the habit of taking my phone with me to get some monitoring in on my progress with the aid of GPS and so on. I have been using a really nice app called mapmyrun. It has given me months of good service and what with the moderate amount of running I have done, I have committed a fair few hours worth of data online.\nMeanwhile I have decided that it would be great to get out running with a group of people rather than on my own all the time, so I have joined up with a very friendly local running group in Caistor, Lincolnshire. Quite a number of them use a different app called Strava, and I thought that I would give it a go based on some positive recommendations from members of the group.\nYou might wonder what this article is doing in the software section of this site\u0026hellip; well, read on.\nThe Problem All the fun started when I decided to investigate how to export my running data from mapmyrun into a format I could potentially import to another site, such as Strava. Simple, I thought, just go to each workout1 in turn and export the session as a GPX file. To clarify, the GPX file standard is intended to be a commonly understood way of sharing GPS data between software applications in XML format.\nWithout getting too technical, GPX files are capable of describing two types of track point sets, where a track point set is a collection of one or more latitude and longitude coordinates:\nA route is a set of track points purely describing a route from start point to end point. A track is a set of track points describing a route that has been travelled, complete with time and date at each point. So, after some investigation I discovered that mapmyrun does allow GPX file export, but only of route data, not track data. One might say that this left me slightly perplexed and not a little disappointed. I did a little research and found Mike Palumbo\u0026rsquo;s MapMyRide GPX Converter; now I wanted something similar but for their running site.\nIt was time to roll up my sleeves hack some quick and dirty code.\nThe Gory Details If you are interested in using the code I have put together then it is available here from my GitHub account. It is very rough and ready, but should run out of the box, providing you have the pre-requisites sorted, i.e. Linux box etcetera.\nHere follows a brief description of what the code does:\nMakes a cURL request to the mapmyrun server - specifically the end point that serves background data for the maps rendered on the workout page. Loads the data recovered from the request above and creates a JSON object. Creates a datetime object from the start time contained in the data. Creates a XML GPX skeleton to which the processed data will be added. For each item in the data series the time and date is calculated and appended to the GPX skeleton, along with lat, lon and altitude data. When the series has been fully processed the completely formed XML document is dumped into a GPX file locally on the file system. Drawbacks and Improvements Each workout needs to be publically accessible in order for the script to be able to access the data; there is no form of authentication. Actually, the mapmyrun site does have an API which supports OAuth and it may an interesting future exercise to have another attempt at getting GPX data out with it.\nRunning the code is a pretty basic affair also: editing the script before every run is far from ideal. The script would be much improved by being able to accept command line instruction instead.\nIn Conclusion It is fairly likely that I will stick with using Strava, for several reasons: in my testing so far it seems like a more robust app and in general a little easier to use. Strava does seem to use a little less battery than mapmyrun and in addition the interaction with the GPS on my Android phone is much improved.\nAlso, as previously mentioned, quite a few folks from the running club are already using Strava and therefore there is a wealth of comparison data, routes, personal bests and so on.\nAt some point I shall get round to improving the code; it is there mostly for my benefit, however I am leaving it as an open source project for others to use, abuse or possibly even improve.\nGood luck, and free your data.\nWorkouts are what the mapmyrun site calls individual training sessions.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://chrisramsay.co.uk/posts/2014/10/moving-from-mapmyrun-to-strava/","summary":"\u003ch2 id=\"moving-from-map-my-run-to-strava\"\u003eMoving from Map My Run to Strava\u003c/h2\u003e\n\u003cp\u003eI have been running a little while now and am always in the habit of taking my phone with me to get some monitoring in on my progress with the aid of GPS and so on. I have been using a really nice app called \u003ca href=\"http://www.mapmyrun.com/\"\u003emapmyrun\u003c/a\u003e. It has given me months of good service and what with the moderate amount of running I have done, I have committed a fair few hours worth of data online.\u003c/p\u003e\n\u003cp\u003eMeanwhile I have decided that it would be great to get out running with a group of people rather than on my own all the time, so I have joined up with a very friendly local \u003ca href=\"http://www.caistorrunningclub.com/\"\u003erunning group in Caistor, Lincolnshire\u003c/a\u003e. Quite a number of them use a different app called \u003ca href=\"http://www.strava.com/\"\u003eStrava\u003c/a\u003e, and I thought that I would give it a go based on some positive recommendations from members of the group.\u003c/p\u003e\n\u003cp\u003eYou might wonder what this article is doing in the \u003ca href=\"/category/software/\"\u003esoftware section\u003c/a\u003e of this site\u0026hellip; well, read on.\u003c/p\u003e","title":"Moving from Map My Run to Strava"},{"content":"Purpose This is a brief memory aid for me with some small, and perhaps useful, tips when writing articles using reStructuredText_1. I am aiming to keep on top of this document and maintain it as a living thing; when I find something new and useful, it will be appended.\nHints and Tips Subscripts and Superscripts The difficulty with sub and superscripts is that whitespace is not usually needed with them as they usually accompany one or more letters closely. So when using them in reStructuredText whitespace can be backslash escaped - this means that the whitespace added for readability here will be removed during processing. Here is an example for a superscript.\nInter Intergrated Circuit, or I\\ :sup:`2`\\ C, was invented by Phillips Semiconductor in 1982. This renders like the following:\nInter Intergrated Circuit, or I2C, was invented by Phillips Semiconductor in 1982.\nSubscripts work in pretty much the same way, so without further ado, here is an example. Note the mix of backslash escapes and genuinely desired whitespace.\nThe difference between t\\ :sub:`ground` and t\\ :sub:`sky` gives us a reading known as t\\ :sub:`∆`\\ . The above looks like this when rendered:\nThe difference between tground and tsky gives us a reading known as t∆.\nMaths Otherwise known as math by our American friends, mathematical formulae can fairly easily be rendered rather nicely using a combination of reStructuredText and standard LaTeX{\\LaTeX} syntax.\nMaths can be written up as a standard reStructuredText role such as:\nThe well known quadratic formula: .. math:: x = -b \\pm \\frac{\\sqrt{b^{2}-4ac}}{2a} Which renders like this:\nThe well known quadratic formula:\nx=−b±b2−4ac2ax = -b \\pm \\frac{\\sqrt{b^{2}-4ac}}{2a} One can also include maths inline, such as the following:\nThe power series, :math:`\\frac{1}{0!}+\\frac{2}{1!}x+\\frac{3}{2!}x^2+\\frac{4}{3!}x^3+...` states that... Which looks like this when rendered:\nThe power series, 10!+21!x+32!x2+43!x3+...\\frac{1}{0!}+\\frac{2}{1!}x+\\frac{3}{2!}x^2+\\frac{4}{3!}x^3+... states that\u0026hellip;\nFor further information on what LaTeX{\\LaTeX} can do with mathematical formulae visit this very informative site.\nCustom Roles Roles are a custom interpreted text role that, once registered with the parser by first declaring them, can be used throughout the document. Custom roles can be used to extend existing roles by specifying them as a second argument.\nGeneral Role Usage Let us take a look at applying in line styles using custom roles.\n.. role:: banana(strong) We all love :banana:`yellow fruit` flavoured milk shakes. Renders as:\nWe all love yellow fruit flavoured milk shakes.\nSome roles accept an extra parameter, depending on the base role which they are extending. For example, one very useful role to define is an inline syntax highlight:\n.. role:: highlight(code) :language: python Here is a suggestion :highlight:`[\u0026#39;%s=%s\u0026#39; % (n, v) for n, v in zip(self.all_names, self)]`. I think that will do what you need. Renders as:\nHere is a suggestion ['%s=%s' % (n, v) for n, v in zip(self.all_names, self)] . I think that will do what you need.\nOne must bear in mind that there are a couple of considerations to be made when using roles. The first consideration is that we cannot define this role more than once in a given document. Well, that\u0026rsquo;s not entirely true, because you might do the following:\n.. role:: highlight(code) :language: haskell .. role:: highlight(code) :language: python This will not cause an error in the reStructuredText parser; instead, the last role definition will simply override any previous roles with the same name. This leads to another important point: as we now know, because a highlight role is only usable once, it is only worth using if you have more than a small amount of syntax highlighting to do within a document, and the language you are referring to remains the same throughout.\nNext, and rather importantly, your new role name is inexorably linked to the class name defined in your CSS. Let us define a role named porcupine:\n.. role:: porcupine(code) :language: c This is important :porcupine:`float t_ref = read_dev(dev, TEMP_OBJ);` As can be seen below, the rendered code is formatted correctly as C code by the work of the Pygments syntax highlighting engine, so no problem there then.\n\u0026lt;code class=\u0026#34;porcupine c\u0026#34;\u0026gt; \u0026lt;span class=\u0026#34;kt\u0026#34;\u0026gt;float\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;t_ref\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;o\u0026#34;\u0026gt;=\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;read_dev\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;(\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;dev\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;,\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;TEMP_OBJ\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;);\u0026lt;/span\u0026gt; \u0026lt;/code\u0026gt; However, unless you have a porcupine class in your CSS defining your syntax colour rules, the syntax highlighting will not be coloured correctly by the browser. This leads us nicely to problems particular to code syntax highlighting.\nUsage of Roles for Syntax highlighting Let us have a look at how the highlighted part above is rendered. By default the Pygments theme CSS that comes with many Pelican themes will not syntax highlight anything other than code in pre tags. Take a look at the HTML below; this is what the rendered Python from the previous section looks like:\n\u0026lt;code class=\u0026#34;highlight python\u0026#34;\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;[\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;s\u0026#34;\u0026gt;\u0026#39;\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;si\u0026#34;\u0026gt;%s\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;s\u0026#34;\u0026gt;=\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;si\u0026#34;\u0026gt;%s\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;s\u0026#34;\u0026gt;\u0026#39;\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;o\u0026#34;\u0026gt;%\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;(\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;n\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;,\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;v\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;)\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;k\u0026#34;\u0026gt;for\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;n\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;,\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;v\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;ow\u0026#34;\u0026gt;in\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;nb\u0026#34;\u0026gt;zip\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;(\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;bp\u0026#34;\u0026gt;self\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;o\u0026#34;\u0026gt;.\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;n\u0026#34;\u0026gt;all_names\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;,\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;bp\u0026#34;\u0026gt;self\u0026lt;/span\u0026gt; \u0026lt;span class=\u0026#34;p\u0026#34;\u0026gt;)]\u0026lt;/span\u0026gt; \u0026lt;/code\u0026gt; As mentioned, the whichever Pygments theme CSS you are using, it may need to have the following change made:\n[...] .highlight pre .g { color: #657b83 } /* Generic */ .highlight pre .k { color: #859900 } /* Keyword */ .highlight pre .l { color: #657b83 } /* Literal */ [...] to:\n[...] .highlight .g, .highlight pre .g { color: #657b83 } /* Generic */ .highlight .k, .highlight pre .k { color: #859900 } /* Keyword */ .highlight .l, .highlight pre .l { color: #657b83 } /* Literal */ [...] This takes into account that we also want to highlight items that are direct descendants of the .highlight class, as well as those grandchildren with pre as a parent tag2.\nComments I do not know about anyone else, but I find adding comments to articles that I am writing incredibly useful3; they help me by serving as small reminders to complete a section or just give me a reason why something has been put together in a certain way.\nComments are processed into a comment element by the reStructuredText parser but, depending on the output formatter, are removed prior to being processed output on the final document.\nBelow are some comment mark up examples:\n.. This is a comment .. [this is also] a comment .. |this is| also a comment .. _this: is also a comment .. .. finally:: this is also a comment The only restriction on comments is that they may not contain any other markup syntax constructs, unless the comment begins with a .. on a line by itself.\nIn Conclusion In writing this document I have read the excellent reStructuredText documentation so much more thoroughly; more so than I would have done otherwise. In itself, this has been a genuine learning experience and I have been able to put some key lessons into practice in the production of this document. So there we are, a few useful bits of information to keep referring back to as and when.\nThis may also be useful for other people.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nNot being a CSS expert, this explanation might be simplistic, or even plain wrong but that is pretty much how I understand it at this stage.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nIn fact, I find them so useful that they are left behind in completed articles, in perpetuity.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://chrisramsay.co.uk/posts/2014/10/useful-restructuredtext/","summary":"\u003ch2 id=\"purpose\"\u003ePurpose\u003c/h2\u003e\n\u003cp\u003eThis is a brief memory aid for me with some small, and perhaps useful, tips when writing articles using reStructuredText_\u003csup id=\"fnref:1\"\u003e\u003ca href=\"#fn:1\" class=\"footnote-ref\" role=\"doc-noteref\"\u003e1\u003c/a\u003e\u003c/sup\u003e. I am aiming to keep on top of this document and maintain it as a living thing; when I find something new and useful, it will be appended.\u003c/p\u003e","title":"Useful reStructuredText"},{"content":"The Publishing Process This brief article focuses on only one part of using Pelican as a static site generation tool - the day to day usage once the set up is established. There are loads of really good articles on how to get started with Pelican and I am not about to add to that growing pile; there is just no need.\nI don\u0026rsquo;t use the default make tool that comes with Pelican, because having a Python programming background made me feel much more comfortable with utilising the ever excellent Fabric tool.\nRepositories Some may call this overkill, but I actually maintain three repositories for this site; two private ones and a public one where this site is hosted. One private repository is a store for all the site in the build phase. This contains all the .rst files, config files, draft articles and build script; it is basically the site in a raw form. The public repo is where the site is published, in HTML form. The second private repo functions purely as a complete backup of the public repo. As a final note all commits are signed, using git commit -S, with my keybase.io identity.\nThe Process I have boiled down my article creation process to four basic, but not necessarily linear, steps: write, read, back up and publish.\nWrite Articles are created in the reStructuredText format mostly using the superb Sublime Text 3 editor; if you have not tried it you should definitely give it a go, and it works on Linux, Mac and even Windows. The articles are stored in category specific sub-directories as required.\nRead For local viewing prior to publishing I use Python\u0026rsquo;s SimpleHTTPServer. All configuration values for the local Pelican build are handled by a local_settings.py file in the build/configs directory.\nBack Up Whilst writing I frequently make back up commits to the private build repo. Any changes I make to the build process are also committed to that repo. As nothing I ever write gets finished first time, these first three steps of write, read and back up are performed in an almost infinite loop.\nPublish This is the point at which the article to be published has passed my somewhat meagre level of quality control and it is ready to be launched onto the unsuspecting masses, such as yourself. In a nutshell, the fabric file below takes care of clearing out the deploy directory and adding the newly built HTML and theme sundries. All draft articles are removed, because I do not want them on the live site, and the site changes are committed to the public repository. In addition, as described above, a private back up is also made.\nFile and Directory Structure Here\u0026rsquo;s a slightly truncated view of the directory layout for the complete site creation and maintenance process.\n. ├── build │ ├── .git (Back up repo) │ ├── cache │ ├── configs │ │ ├── __init__.py │ │ ├── deploy_settings.py │ │ └── local_settings.py │ │ │ └── content │ ├── extras │ ├── pages │ └── [...] ├── deploy │ ├── .git (user.io repo) │ ├── [...] │ └── theme │ ├── css │ ├── fonts │ └── js └── local ├── [...] └── theme ├── css ├── fonts └── js build This directory is where any editing and configuration takes place. This is backed up by the fab backup target to this repo. Following are some important points about the sub-directories:\nconfigs\nHerein is contained a __init__.py file - this is here to enable fab to import a local_settings.py file for setting a few useful environmental variables. A deploy_settings.py file also lives in here for when the fab publish target is run.\ncontent\nThis is where the site articles are stored and edited - primarily as reStructuredText documents.\nlocal The local directory is where the site is published, via the fab serve target, and evaluated locally.\ndeploy The deploy directory is itself a git repo and is where the site is published, using the fab publish fabric command, prior to committing and pushing into the github.io repo etc. As mentioned before, all drafts are deleted prior to the git commit.\nThe Fabric File Below is a copy of my current fabric script. It has turned into a bit of a monster but I am quite happy that there is nothing in there that does not get at least some regular use. The targets (apologies, but I am in the habit of calling the defs below targets from my days spent using Apache Ant) I mostly use are serve, backup and publish. You may notice that both the targets concerned with committing code to git repos support optional commit messages, something I thought was quite a nice touch.\nimport time from fabric.api import local, settings, env, cd import fabric.contrib.project as project import os # There must be at least a local_settings.py file in the configs directory. import configs.local_settings as conf # Local path configuration (can be absolute or relative to fabfile) - they are # all added to env for convenience \u0026amp; abuse. # Repo settings - from local_settings.py env.git_url = conf.URL1 env.git_backup_url = conf.URL2 env.host_repo = conf.PUBLISH_REPO env.build_repo = conf.BUILD_REPO env.host_backup_repo = conf.PUBLISH_BACKUP_REPO # Local file paths env.config_path = \u0026#39;./configs\u0026#39; env.local_settings_file = \u0026#39;local_settings.py\u0026#39; env.deploy_settings_file = \u0026#39;deploy_settings.py\u0026#39; env.project_path = \u0026#39;../build\u0026#39; env.deploy_path = \u0026#39;../deploy\u0026#39; env.local_path = \u0026#39;../local\u0026#39; env.content_path = \u0026#39;./content\u0026#39; def clean(): \u0026#34;\u0026#34;\u0026#34; Cleans and recreates local copy directory \u0026#34;\u0026#34;\u0026#34; if os.path.isdir(DEPLOY_PATH): local(\u0026#39;rm -rf {local_path}\u0026#39;.format(**env)) local(\u0026#39;mkdir {local_path}\u0026#39;.format(**env)) def kill(): \u0026#34;\u0026#34;\u0026#34; Kills errant SimpleHTTPServer processes - always outputs an error but works nevertheless \u0026#34;\u0026#34;\u0026#34; with settings(warn_only=True): local(\u0026#39;ps aux | grep SimpleHTTPServer | awk \\\u0026#39;{print $2}\\\u0026#39; | xargs kill -9\u0026#39;) def build(): \u0026#34;\u0026#34;\u0026#34; Runs local build with debug output - pushes out content to local copy directory, using local settings \u0026#34;\u0026#34;\u0026#34; local(\u0026#39;pelican -D {content_path} -o {local_path} -s {config_path}/{local_settings_file}\u0026#39;.format(**env)) def backup(commit_msg=False): \u0026#34;\u0026#34;\u0026#34; Backs up this local copy to bitbucket with an optional \u0026#34; delimited commit message. \u0026#34;\u0026#34;\u0026#34; with lcd(env.project_path): if commit_msg is False: command = \u0026#39;git commit -S -m \u0026#34;Back up {0}\u0026#34;\u0026#39; else: command = \u0026#39;git commit -S -m \u0026#34;Back up {0} - {1}\u0026#34;\u0026#39; local(\u0026#39;git add -A .\u0026#39;.format(**env)) local(command.format( time.strftime(\u0026#34;%d %b %Y %H:%M:%S\u0026#34;, time.localtime()), commit_msg, **env ) ) local(\u0026#39;git push {git_backup_url}/{build_repo} master\u0026#39;.format(**env)) def rebuild(): \u0026#34;\u0026#34;\u0026#34; Cleans and pushes content to local copy directory, running clean() and build() \u0026#34;\u0026#34;\u0026#34; clean() build() def remove_drafts(): \u0026#34;\u0026#34;\u0026#34; Removes drafts from deploy_path/drafts as we don\u0026#39;t want publicly accessible \u0026#34;\u0026#34;\u0026#34; with lcd(env.deploy_path): local(\u0026#39;rm -fr drafts\u0026#39;.format(**env)) def regenerate(): \u0026#34;\u0026#34;\u0026#34; Regenerates site content \u0026#34;\u0026#34;\u0026#34; local(\u0026#39;pelican -r -s {local_settings_file}\u0026#39;.format(**env)) def serve(): \u0026#34;\u0026#34;\u0026#34; Serves content from local copy directory, calling clean() and build() beforehand \u0026#34;\u0026#34;\u0026#34; rebuild() with lcd(env.local_path): local(\u0026#39;python -m SimpleHTTPServer\u0026#39;.format(**env)) def pub_deploy(): \u0026#34;\u0026#34;\u0026#34; Prepares for publishing - runs non debug build to deployment path output \u0026#34;\u0026#34;\u0026#34; with lcd(env.deploy_path): local(\u0026#39;rm -fr *\u0026#39;.format(**env)); local(\u0026#39;pelican {content_path} -o {deploy_path} -s {config_path}/{deploy_settings_file}\u0026#39;.format(**env)) def publish(commit_msg=False): \u0026#34;\u0026#34;\u0026#34; Makes a build over to the deploy path, using deployment settings, cleaning up beforehand with pub_deploy() with an optional \u0026#34; delimited commit message. \u0026#34;\u0026#34;\u0026#34; with lcd(env.deploy_path): if commit_msg is False: command = \u0026#39;git commit -S -m \u0026#34;Publication {0}\u0026#34;\u0026#39; else: command = \u0026#39;git commit -S -m \u0026#34;Publication {0} - {1}\u0026#34;\u0026#39; # Publish to deployment directory pub_deploy() # Remove draft htmls - don\u0026#39;t want them in a public repo remove_drafts() # Git add and commit with lcd(env.deploy_path): local(\u0026#39;git add -A .\u0026#39;.format(**env)) local(command.format( time.strftime(\u0026#34;%d %b %Y %H:%M:%S\u0026#34;, time.localtime()), commit_msg, **env ) ) local(\u0026#39;git push {git_url}/{host_repo} master\u0026#39;.format(**env)) local(\u0026#39;git push {git_backup_url}/{host_backup_repo} master\u0026#39;.format(**env)) So there you have it, not quite as brief as I intended\u0026hellip; feel free to pass comment, suggest improvements etc.\n","permalink":"https://chrisramsay.co.uk/posts/2014/10/my-pelican-publishing-process/","summary":"\u003ch2 id=\"the-publishing-process\"\u003eThe Publishing Process\u003c/h2\u003e\n\u003cp\u003eThis brief article focuses on only one part of using \u003ca href=\"http://blog.getpelican.com/\"\u003ePelican\u003c/a\u003e as a static site generation tool - the day to day usage once the set up is established. There are loads of really good articles on how to get started with Pelican and I am not about to add to that growing pile; there is just no need.\u003c/p\u003e\n\u003cp\u003eI don\u0026rsquo;t use the default \u003ca href=\"http://www.gnu.org/software/make/\"\u003emake tool\u003c/a\u003e that comes with Pelican, because having a \u003ca href=\"https://www.python.org/\"\u003ePython\u003c/a\u003e programming background made me feel much more comfortable with utilising the ever excellent \u003ca href=\"http://www.fabfile.org/\"\u003eFabric\u003c/a\u003e tool.\u003c/p\u003e","title":"My Pelican Publishing Process"},{"content":"Further Thoughts This post follows on quite quickly (for me at least!) from my last one regarding my latest tests and reflections on environmental issues with some thoughts about how to measure ground temperatures in a reliable way. I am also going to take a look at the difficulties I might face in trying to automatically interpret cloud cover the whole year through. Finally I am going to share some thoughts on how I might approach packaging the system for deployment in the open air, and what to do about supplying power and communications.\nRegarding Temperatures If there is one thing this project is providing, it is many hours of thought about temperature; both in terms of obtaining measurements that are as accurate as I can make them, and understanding the data that this system will eventually provide.\nMeasuring Ground Temperature Initially I started taking ground temperature from the ambient reading given by the MLX90614 sensors. This was just as a matter of quick prototyping to get circuitry outside and gathering test data. It worked well until I packaged the circuit into a plastic box exposed to solar radiation; albeit the lid of the box had two generous squares cut out so that the sensors had access to the sky. Over a few days of data gathering I noticed that the ambient readings being provided by the two MLX90614 variants were consistently higher than the local ground temperature from my weather station and, more importantly, differed from each other by a few degrees Celsius.\nIt quickly became clear that it was time to find an external way of providing a baseline ground temperature.\nA graph comparing ambient measurements taken by the MLX90614 sensors versus a DS18B20\nI decided to get a DS18B20 1-wire digital temperature sensor on the scene to in order to provide a baseline from which to see how the two MLX90614 were performing in comparison. The results can be seen in the graph above. It is apparent that indeed sitting a sensor in an small box will not give a very reliable ground temperature reading. In any amount of sunshine the heat inside the box builds very quickly indeed. It is noticeable that the BCF variant gives a reading that is slightly closer to the DS18B20 than the BCC variant. I think that this is probably due to the larger size of the BCF and that it is therefore poking through the cut out in the box lid.\nThe DS18B20 may not be the only answer: there may be an option to try out pointing a MLX90614 down at the ground; this is, however, something I may have to try at a later date when I get closer to assembling a mark 2 prototype.\nLong term I think that measuring ground temperature using an external means such as this will be the only answer to getting accurate measurements down the line. I will be casting my eye over the Met Office surface observations fact sheet in the near future to get more of an idea where I should concentrating my ground temperature measurement efforts.\nInterpreting Sky and Ground Temperature Part of the enjoyment of this project so far has been trying to understand the seasonal variations in temperature of both the sky and the ground. Over years of observing the weather I like to think that, as far as ground temperature goes, I have a pretty good understanding, as would anyone given enough time. However, sky temperature is not something most people take an interest in so it is not surprising that there is little data out there on the subject.\nAny excuse to get a cloud photograph in\u0026hellip;\nIn the months so far since I started sampling the sky temperature (which is not quite a year yet), I have seen that a lovely clear blue sky is pretty much at the same temperature on a winters\u0026rsquo; day as it is at the height of summer; both round about -38°C when measured with my IR thermometer.\nOvercast skies tend to differ more between the winter and summer months with those in winter tending to be quite a few degrees colder. This is usually because the cloud base is at or near the same temperature as the ground temperature in the respective season. So a typical summer cloud base temperature might find itself at 12°C; a winter cloud base temperature at -2°C.\nIn essence the temperature gap between clear skies and cloudy skies is not a straightforward relationship. It depends very much on season and local variation.\nLet us look at a simple example:\nSeason Cloud Sky °C Ground °C Gap °C winter Clear\n-38\n-4\n34\nwinter Overcast\n-2\n-2\n0\nsummer Clear\n-38\n20\n58\nsummer Overcast\n12\n14\n2\nAs of yet I have not firmly planned how I am going to graph the data; I am just aware that having data and making sense of it are two totally different things. It is likely that, at least initially, I will look to just graphing the ground and sky temperatures side by side and leave the interpretation to some manual comparison. Below is an example of such a graph.\nShowing both sky and ground temp together; both sensor variants\nUltimately I would love to end up with some sort of heat map with a quarter hour resolution over a 24 hour period. Perhaps this is just mission creep though; the initial idea was just to observe clouds in the night sky. Be that as it may, the ability to do something like this will depend on understanding the temperature limits in the system. All food for thought in the future of this project.\nPutting the System Outside Running a prototype on the bench is nice and simple. There are no concerns with the environmental conditions, other than that your shed is warm, and that there is a cup of tea to hand. Also, there is no concern to the issues with power or communicating with the Arduino; it is all at the end of a USB lead. Of course, this all changes the minute we shift the whole kit outside.\nPackaging When thinking about what type of materials to consider packaging this system in there is only one really sensible answer: plastic. This is because plastic is readily available, easy to work with (in terms of shaping, cutting and bonding), comes in many easy to use shapes and it is relatively cheap. Falling back on experience gained from the soil temperature project I will again look to plastic plumbing supplies for the raw packaging materials. I am also planning to investigate some exterior electrical enclosures, such as weather resistant junction boxes, as a ready made alternative.\nCommunications I had originally considered using serial communications but this presented a number of problems, such as what happens as soon as I have more than one Arduino based device collecting data; I am then into the realms of serial networks and so on.\nBriefly, I considered using radio of some sort such as Sparkfun\u0026rsquo;s 915MHz wireless transceiver, but then power supply would still be an issue. So my thinking went back to some sort of wired communications. I then came across the marvellous ENC28J60 module which opened up the possibility of using communications over IP - something I am already quite happy with using. The ENC28J60 itself is a fairly self contained 10BASE-T Ethernet controller with a whole host of features including an SPI interface. Here is a data sheet if you are interested.\nI won\u0026rsquo;t go into describing how to get an Arduino and ENC28J60 module as there are plenty of articles on the net on how to get started. I will, however, recommend using the UIPEthernet library written by Norbert Truchsess. In my opinion it is by far the best library of the small selection available. It has proven to be very reliable when left running for extended periods, unlike other libraries which have crashed after a few hours.\nPower Supply Once the decision was taken to use wired communications in the form of Ethernet it was but a small light bulb moment to realise that Ethernet does not use all the twisted pairs in a cable; commonly only two of the four available. therefore it seemed to me a no-brainer to use a couple of the spares as a supply and a ground. Being that I am passing DC at a relatively low voltage, I do not anticipate any issues with interference with the network signal.\nIn Summation I think that I am closer to coming to a conclusion with the physical build of the project than I am with the interpretation of the data once that starts to flow forth. For now I am just going to concentrate on getting the project built; once I am gathering data the fun is really going to start!\n","permalink":"https://chrisramsay.co.uk/posts/2014/09/cloud-detection-part-four/","summary":"\u003ch2 id=\"further-thoughts\"\u003eFurther Thoughts\u003c/h2\u003e\n\u003cp\u003eThis post follows on quite quickly (for me at least!) from my last one regarding my \u003ca href=\"/posts/2014/09/cloud-detection-part-three/\"\u003elatest tests and reflections on environmental issues\u003c/a\u003e with some thoughts about how to measure ground temperatures in a reliable way. I am also going to take a look at the difficulties I might face in trying to automatically interpret cloud cover the whole year through. Finally I am going to share some thoughts on how I might approach packaging the system for deployment in the open air, and what to do about supplying power and communications.\u003c/p\u003e","title":"Cloud Detection - Part Four"},{"content":"Title Case As a rather inexperienced writer, it is the little things I struggle with, as well as the big things you might add. I find that one of the most annoying is getting title case correct. There are a lot of style guides about which may or may not help; I find some of them cryptic at times, and rather than being strict rules, they seem to be more matters of style. Until recently I had been taking style advice from yourdictionary.com\u0026rsquo;s site, but I always found having to undertake this task manually a bit of a bore.\nHowever, as with most programmers (or even ex-programmers like myself), I find that as soon as I am faced with a repetetive task I look to find a way of automating it; and thus it happened with this. Five minutes later and I had turned up Pat Pannuto\u0026rsquo;s Python based titlecase package.\nUp and Running Getting titlecase up and running couldn\u0026rsquo;t be simpler:\n$ pip install titlecase Downloading/unpacking titlecase Downloading titlecase-0.7.1.tar.gz Running setup.py (path:/home/vagrant/Envs/blog/build/titlecase/setup.py) egg_info for package titlecase no previously-included directories found matching \u0026#39;doc/.build\u0026#39; Installed /home/vagrant/Envs/blog/build/titlecase/nose-1.3.4-py2.7.egg Installing collected packages: titlecase Running setup.py install for titlecase Successfully installed titlecase Cleaning up... Using it couldn\u0026rsquo;t be much simpler either:\n$ python Python 2.7.3 (default, Jan 2 2013, 13:56:14) [GCC 4.7.2] on linux2 Type \u0026#34;help\u0026#34;, \u0026#34;copyright\u0026#34;, \u0026#34;credits\u0026#34; or \u0026#34;license\u0026#34; for more information. \u0026gt;\u0026gt;\u0026gt; from titlecase import titlecase \u0026gt;\u0026gt;\u0026gt; titlecase(\u0026#39;the lion, the witch and the wardrobe\u0026#39;) \u0026#39;The Lion, the Witch and the Wardrobe\u0026#39; \u0026gt;\u0026gt;\u0026gt; titlecase(\u0026#39;Titlecase: a Great find\u0026#39;) \u0026#39;Titlecase: A Great Find\u0026#39; \u0026gt;\u0026gt;\u0026gt; titlecase(\u0026#39;Why it is never too late to learn grammar\u0026#39;) \u0026#39;Why It Is Never Too Late to Learn Grammar\u0026#39; So, in summation, thumbs up to Mr Pannuto and thanks for such a useful library. I plan to make much use of it in the future.\n","permalink":"https://chrisramsay.co.uk/posts/2014/09/titlecase-a-great-find/","summary":"\u003ch2 id=\"title-case\"\u003eTitle Case\u003c/h2\u003e\n\u003cp\u003eAs a rather inexperienced writer, it is the little things I struggle with, as well as the big things you might add. I find that one of the most annoying is getting title case correct. There are a lot of style guides about which may or may not help; I find some of them cryptic at times, and rather than being strict rules, they seem to be more matters of style. Until recently I had been taking style advice from \u003ca href=\"http://grammar.yourdictionary.com/capitalization/rules-for-capitalization-in-titles.html\"\u003eyourdictionary.com\u0026rsquo;s\u003c/a\u003e site, but I always found having to undertake this task manually a bit of a bore.\u003c/p\u003e\n\u003cp\u003eHowever, as with most programmers (or even ex-programmers like myself), I find that as soon as I am faced with a repetetive task I look to find a way of automating it; and thus it happened with this. Five minutes later and I had turned up \u003ca href=\"http://patpannuto.com\"\u003ePat Pannuto\u0026rsquo;s\u003c/a\u003e Python based \u003ca href=\"https://github.com/ppannuto/python-titlecase\"\u003etitlecase package\u003c/a\u003e.\u003c/p\u003e","title":"Titlecase: A Great Find"},{"content":"Latests Tests Since the last part in this series I have continued to conduct experiments comparing and contrasting the two variants of MLX90614, the wide field of view BAA and the considerably more focused BCF. Whilst undertaking these \u0026lsquo;in the field\u0026rsquo; tests I have also come up against a few, what you might call, environmental problems which need addressing before I can move from the prototype phase to the implementation phase.\nAn Arduino has a day out in a box\nEnvironmental Problems So prototyping electronics within the pleasant warm and dry atmosphere, or \u0026lsquo;in the lab\u0026rsquo; if you prefer, is one thing. However, placing your precious prototype out into the elements is rather another challenge altogether.\nDealing with Dew One major issue you will come across in these wonderful isles of the United Kingdom if you spend any time in the outdoors at night, particularly in the Autumn months, is high humidity and therefore plenty of dew settling on to surfaces, horizontal ones in particular.\nDew tends to form on surfaces that are at or below a temperature called the dew point. Simply put, there is always water vapour in the air, and if that air is cooled sufficiently the saturation point is reached, which is also called the dew point of the air. Hence the dew point is the temperature at which water vapour in the air will condense on a solid surface. The dew point and air temperature are linked by a property known as relative humidity: the amount of water vapour held in the air.\nRelative humidity (RH) from saturation to dry air, is not an absolute scale - at high air temperatures the scale can encompass quite a difference between air and dew point temperatures, say 14°C. At lower air temperatures, near freezing point for example, the scale may only stretch 4°C.\nThe graph below produced by my weather station illustrates what happens to the relationship between dew point, air temperature and relative humidity over a couple of days.\nA graph comparing air temperature, dew point and relative humidity\nThankfully, there are a couple lessons which can be learned from the field of amateur observational astronomy; it can be a real battle to keep dew from forming on the mirrors and lenses of telescopes. Essentially there are two countermeasures available against dew in the outdoors: the application of heat or the application of airflow.\nHeat\nOne method of preventing dew from forming is to ensure that the object in question is warmed to slightly above the dew point temperature. This is something that is often done with telescope mirrors to prevent them from misting up during the night. You may have seen this sort of thing in hotel rooms where a portion of a bathroom mirror is heated from behind to prevent condensation from forming while you are running the hot taps.\nAirflow\nIntroducing a flow of air across an object can also be a successful method of preventing dew from forming, in conditions where the relative humidity is not too high. Bryan Greer mentions, in an article relating to fans on astronomical telescope optics, that\n\u0026ldquo;The kinetic energy of the forced airflow makes it harder for seeds of condensation to form\u0026rdquo;\n\u0026ndash; Bryan Greer\nPerhaps this may be possible with the inclusion of a small 5 volt fan somewhere near the sensor.\nNow these above lessons may be applicable to keeping the sensors clear of moisture. I\u0026rsquo;m going to have to experiment!\nRain, Rain and more Rain Another joy of living in these British Isles is the rain. There is rather a lot which for one makes this country fairly green and pleasant, but it does also mean that whatever you do with your precious electronics, you need to make sure that you package them in a decent water resistant package.\nMLX90614 are, according to the data sheet, hermetically sealed but what does that actually mean in practice when it comes to dealing with potentially wet environments? The definition of something being hermetically sealed means that it is airtight; this does not mean that the seal is an effective barrier against water.\nThe data sheet makes no mention of any IP regulation (IP: Ingress Protection) weather proofing so I think that it would be prudent to keep it from the elements in some way. I do not want to have to be replacing these beasts too often as they are not the cheapest of components.\nAdditionally, whilst the BCF variant of the sensor is fairly flat on the top, the BCF variant has the sensor window in a 20mm well which could potentially fill up with water in a heavy downpour.\nIn the end I think that this all points to putting the sensor in some sort of housing that will shield it from the weather.\nFinal Reflection on Rain and Dew After some thought I have decided that I do not want to leave things to chance: for me there is no question of having the sensor out in the open and pointing to the sky. This leaves only one option which is indirect observation of the sky via a reflective surface, keeping the sensor in the dry and only exposing a mirror of some sort to the sky.\n","permalink":"https://chrisramsay.co.uk/posts/2014/09/cloud-detection-part-three/","summary":"\u003ch2 id=\"latests-tests\"\u003eLatests Tests\u003c/h2\u003e\n\u003cp\u003eSince the \u003ca href=\"/posts/2014/03/cloud-detection-part-two/\"\u003elast part in this series\u003c/a\u003e I have continued to conduct experiments comparing and contrasting the two variants of MLX90614, the wide field of view BAA and the considerably more focused BCF. Whilst undertaking these \u0026lsquo;in the field\u0026rsquo; tests I have also come up against a few, what you might call, environmental problems which need addressing before I can move from the prototype phase to the implementation phase.\u003c/p\u003e","title":"Cloud Detection - Part Three"},{"content":"I finally have some energy back and am getting on with some projects again. It is about time really. I have spent most of the summer evenings not doing a great deal; just feeling like I needed the time to recover from the rigours of a very busy working day. I am normally one of these people who is not comfortable with just sitting down and meditating on things. I think on the whole it has probably been a positive step to sit back and de-stress in front of a book or two.\nMy reading material has mostly consisted of indulging in the stories of those crazy and committed souls who undertake adventure racing or just incredibly long travels on foot. Maybe this is my mind and body telling me that it is about time that I took my running to the next level.\nMeanwhile the list of things that need doing has grown. Boring jobs such as exterior painting on the garage, laying some concrete for a hard standing; I\u0026rsquo;m sure that you get the idea. In tandem another, altogether more interesting, list of \u0026ldquo;things I would like to do\u0026rdquo; has also grown. In another universe, life would be spent only undertaking the items on the more exciting list, and outsourcing the boring rubbish to someone else.\nAt least, if nothing else happens, this site is the beneficiary of some new found enthusiasm and I am pouring out more words for the unsuspecting public to read. Pity them.\n","permalink":"https://chrisramsay.co.uk/posts/2014/09/energised-finally/","summary":"\u003cp\u003eI finally have some energy back and am getting on with some projects again. It is about time really. I have spent most of the summer evenings not doing a great deal; just feeling like I needed the time to recover from the rigours of a very busy working day. I am normally one of these people who is not comfortable with just sitting down and meditating on things. I think on the whole it has probably been a positive step to sit back and de-stress in front of a book or two.\u003c/p\u003e","title":"Energised, finally!"},{"content":"Multiple MLX90614 Sensors Nearly all articles you might look for regarding trying out an MLX90614 with an Arduino contain examples of using one sensor at a time. This makes sense as probably 99% of people are going to prototype a system with only one sensor in it. Quite a few articles make the incorrect assumption that having more than one sensor in a circuit is not even possible because the device slave addresses are hard coded. They do all come out of the factory with a default slave address of 0x5A, but this can be changed with a small amount of effort.\nNow I plan to use two of these MLX90614 sensors in my cloud detection project so there was no two ways about it - I need 2 sensors, hence two slave addresses. Below I am going to take a look at the code to change the slave address on one of these devices, and then a way of actually implementing the change.\nFor reference purposes, I have lifted a pinout diagram of the device from the Melexis MLX90614 Datasheet, and reproduced it below:\nMLX90614 Pinout - courtesy of Melexis NV\nGiving an MLX90614 a New Address After some research on the net I came across this forum post which proved to be very helpful indeed. It was posted by a user called paulrd about whom I\u0026rsquo;ve not been able to find any other information. Owing paulrd a debt of gratitude I am reproducing the code here in a slightly modified form.\n// remap_mlx90614.ino #include \u0026#34;i2cmaster.h\u0026#34; // New slave address, purposefully left shifted byte NewMLXAddr = 0x1B; // 0x5A is the default address - uncomment this to set it back // byte NewMLXAddr = 0x5A; void setup(){ Serial.begin(9600); Serial.println(\u0026#34;Setup...\u0026#34;); // Initialise the i2c bus, enable pullups and then wait i2c_init(); PORTC = (1 \u0026lt;\u0026lt; PORTC4) | (1 \u0026lt;\u0026lt; PORTC5); delay(5000); // Read current address bytes ReadAddr(0); // Change address to new value (NewMLXAddr); ChangeAddr(NewMLXAddr, 0x00); // Read address bytes ReadAddr(0); Serial.print(\u0026#34;\u0026gt; Cycle power NOW to set address to: \u0026#34;); Serial.print(NewMLXAddr, HEX); Serial.println(\u0026#34; - you have 10 seconds\u0026#34;); // Cycle power to MLX during this 10 second pause delay(10000); // Read temperature using default address ReadTemp(0); // Read temperature using new address (note left bit shift for reading) ReadTemp(NewMLXAddr\u0026lt;\u0026lt;1); Serial.println(\u0026#34;**---DONE---**\u0026#34;); } void loop(){ delay(5000); ReadTemp(NewMLXAddr\u0026lt;\u0026lt;1); } word ChangeAddr(byte NewAddr1, byte NewAddr2) { Serial.println(\u0026#34;\u0026gt; Change address\u0026#34;); // Send start condition and write bit i2c_start_wait(0 + I2C_WRITE); // Send command for device to return address i2c_write(0x2E); // Send low byte zero to erase i2c_write(0x00); // Send high byte zero to erase i2c_write(0x00); if (i2c_write(0x6F) == 0) { // Release bus, end transaction i2c_stop(); Serial.println(\u0026#34;\u0026gt; Data erased.\u0026#34;); } else { // Release bus, end transaction i2c_stop(); Serial.println(\u0026#34;\u0026gt; Failed to erase data\u0026#34;); return -1; } Serial.print(\u0026#34;\u0026gt; Writing data: \u0026#34;); Serial.print(NewAddr1, HEX); Serial.print(\u0026#34;, \u0026#34;); Serial.println(NewAddr2, HEX); for (int a = 0; a != 256; a++) { // Send start condition and write bit i2c_start_wait(0 + I2C_WRITE); // Send command for device to return address i2c_write(0x2E); // Send low byte zero to erase i2c_write(NewAddr1); // Send high byte zero to erase i2c_write(NewAddr2); if (i2c_write(a) == 0) { // Release bus, end transaction then wait 10ms i2c_stop(); delay(100); Serial.print(\u0026#34;\u0026gt; Found correct CRC: 0x\u0026#34;); Serial.println(a, HEX); return a; } } // Release bus, end transaction i2c_stop(); Serial.println(\u0026#34;\u0026gt; Correct CRC not found\u0026#34;); return -1; } void ReadAddr(byte Address) { Serial.println(\u0026#34;\u0026gt; Read address\u0026#34;); // Inform the user Serial.print(\u0026#34; MLX address: \u0026#34;); Serial.print(Address, HEX); Serial.print(\u0026#34;, Data: \u0026#34;); // Send start condition and write bit i2c_start_wait(Address + I2C_WRITE); // Send command for device to return address i2c_write(0x2E); i2c_rep_start(Address + I2C_READ); // Read 1 byte and then send ack (x2) Serial.print(i2c_readAck(), HEX); Serial.print(\u0026#34;, \u0026#34;); Serial.print(i2c_readAck(), HEX); Serial.print(\u0026#34;, \u0026#34;); Serial.println(i2c_readNak(), HEX); i2c_stop(); } float ReadTemp(byte Address) { int data_low = 0; int data_high = 0; int pec = 0; Serial.println(\u0026#34;\u0026gt; Read temperature\u0026#34;); // Inform the user Serial.print(\u0026#34; MLX address: \u0026#34;); Serial.print(Address, HEX); Serial.print(\u0026#34;, \u0026#34;); i2c_start_wait(Address + I2C_WRITE); // Address of temp bytes i2c_write(0x07); // Read - the famous repeat start i2c_rep_start(Address + I2C_READ); // Read 1 byte and then send ack (x2) data_low = i2c_readAck(); data_high = i2c_readAck(); pec = i2c_readNak(); i2c_stop(); // This converts high and low bytes together and processes the temperature // MSB is a error bit and is ignored for temperatures // Zero out the data float temp = 0x0000; // This masks off the error bit of the high byte, then moves it left // 8 bits and adds the low byte. temp = (float)(((data_high \u0026amp; 0x007F) \u0026lt;\u0026lt; 8) + data_low); temp = (temp * 0.02) - 273.16; Serial.print(temp); Serial.println(\u0026#34; C\u0026#34;); return temp; } When it comes to choosing a new address for your sensors, you will find that you have quite a wide selection of possibilities. The Melexis MLX90614 Datasheet states that \u0026ldquo;the SA feature allows connecting up to 127 devices (SA=0x00\u0026hellip;0x07F) with only 2 wires\u0026rdquo;. I have experimented with a number of different addresses between 0x00 and 0x07F without a single problem. Just be aware that most libraries for Arduino refer to the factory default address of 0x5A.\nRunning the Code The first thing to do is to connect the Arduino and MLX90614 in a circuit, such as in the picture below. I won\u0026rsquo;t go to any lengths to explain it1. Please note that pin 1 on the device (SCL) leads to A5 on the Arduino; pin 2 (SDA) leads to A4 on the Arduino - I have made the mistake of mixing these two up more than once. Other than the preceding point, the layout is standard for such devices and there are many other places on the net which will extemporise on these matters.\nAn Arduino and MLX90614 with NC switch on the supply\nThe only thing of note is that during the program run you will have to recycle the supply voltage to the MLX90614. Some people (like me) just pull out and then push back in a wire in the breadboard; others with more time to spend put an NC push switch on the supply to the sensor - as pictured above. Do whatever you feel is necessary to get the job done.\nWarning\nOne note of caution - when attempting to change addresses do not have more than one MLX90614 on the circuit at the same time.\nNext, upload the program above to your Arduino and run it, being ready to pounce with that wire or push switch during the 10 second pause.\nHere\u0026rsquo;s what my output looked like when changing the address of a device from 0x2A to 0x2B:\nSetup... \u0026gt; Read address MLX address: 0, Data: 2A, 0, 4D \u0026gt; Change address \u0026gt; Data erased. \u0026gt; Writing data: 2B, 0 \u0026gt; Found correct CRC: 0x56 \u0026gt; Read address MLX address: 0, Data: 2B, 0, 0 \u0026gt; Cycle power NOW - you have 10 seconds \u0026gt; Read temperature MLX address: 0, 21.36 C \u0026gt; Read temperature MLX address: 56, 21.36 C **---DONE---** \u0026gt; Read temperature MLX address: 56, 21.26 C \u0026gt; Read temperature MLX address: 56, 28.90 C \u0026gt; Read temperature MLX address: 56, 21.04 C One last item to note is that if at some point you end up with a pile of these sensors whose addresses you have forgotten because you changed them and did not note it anywhere, do not panic. The universal address (0) will allow you to access the individual device in order to change the slave address again or whatever.\nExcept for the following addendum!\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://chrisramsay.co.uk/posts/2014/09/arduino-and-multiple-mlx90614-sensors/","summary":"\u003ch2 id=\"multiple-mlx90614-sensors\"\u003eMultiple MLX90614 Sensors\u003c/h2\u003e\n\u003cp\u003eNearly all articles you might look for regarding trying out an MLX90614 with an Arduino contain examples of using one sensor at a time. This makes sense as probably 99% of people are going to prototype a system with only one sensor in it. Quite a few articles make the incorrect assumption that having more than one sensor in a circuit is not even possible because the device slave addresses are hard coded. They do all come out of the factory with a default slave address of \u003cspan class=\"title-ref\"\u003e0x5A\u003c/span\u003e, but this can be changed with a small amount of effort.\u003c/p\u003e\n\u003cp\u003eNow I plan to use two of these MLX90614 sensors in my \u003ca href=\"/posts/2014/03/cloud-detection-part-one/\"\u003ecloud detection\u003c/a\u003e project so there was no two ways about it - I need 2 sensors, hence two slave addresses. Below I am going to take a look at the code to change the slave address on one of these devices, and then a way of actually implementing the change.\u003c/p\u003e","title":"Arduino and Multiple MLX90614 Sensors"},{"content":"I\u0026rsquo;ve been making a slow but steady stream of improvements to the blog this year which began with a migration from an ancient installation of Wordpress to the static Pelican HTML site generator. Many other blogs explain how this is achieved so I won\u0026rsquo;t bore you with it here.\nPost migration I decided to pretty much ditch all my previous posts because many of them were related to programming and were rather out of date. Oh, and in my experience, re-reading stuff you wrote years ago is never good for the soul.\nWhen I first started using Pelican, I converted my old Wordpress HTML to Markdown text formatting, and I loved the simplicity of Markdown very much. But then I started to hit some limitations with it, such as getting a real fight on my hands making syntax highlighting work properly without resorting to external JS libraries and so on.\nLuckily, having something of a Python background, I rekindled my passion for reStructuredText (documentation available here) and set about a morning of converting the Markdown entries to reStructuredText. This would not have been achieved without the aid of a brilliant piece of software, written by Prof. John MacFarlane, Pandoc\n","permalink":"https://chrisramsay.co.uk/posts/2014/09/latest-on-this-blog/","summary":"\u003cp\u003eI\u0026rsquo;ve been making a slow but steady stream of improvements to the blog this year which began with a migration from an ancient installation of Wordpress to the static \u003ca href=\"http://blog.getpelican.com/\"\u003ePelican\u003c/a\u003e HTML site generator. Many other blogs explain how this is achieved so I won\u0026rsquo;t bore you with it here.\u003c/p\u003e\n\u003cp\u003ePost migration I decided to pretty much ditch all my previous posts because many of them were related to programming and were rather out of date. Oh, and in my experience, re-reading stuff you wrote years ago is never good for the soul.\u003c/p\u003e","title":"Latest on this Blog"},{"content":"A little bit of work stemming from from the soil temperature monitoring project; this is an aside on DS18B20 temperature sensing device address discovery with the Arduino. Being that I am going to have a handful of sensors in the ground and therefore not easily accessible for identification I need to find a way of easily finding out all the device addresses in one fell swoop.\nArduino - Device Address Discovery As already discussed in Soil Temperature Monitoring - Part One connecting the DS18B20 to an Arduino is trivial; ground, positive and data are all that is required, with data to the Arduino digital pin 3. Pull up is achieved with a 4K7.\nSimple circuit employed for address discovery\nBelow is a sample of the code I used to discover the sensor addresses.\n#include \u0026lt;OneWire.h\u0026gt; #define ONE_WIRE_BUS 3 OneWire oneWire(ONE_WIRE_BUS); void setup() { Serial.begin(9600); discover(); } void loop() {} void discover(void) { byte i; byte data[12]; byte addr[8]; Serial.print(\u0026#34;Starting...\\n\\r\u0026#34;); while(oneWire.search(addr)) { Serial.print(\u0026#34;\\n\\r\\n\\rDevice has address:\\n\\r\u0026#34;); for( i = 0; i \u0026lt; 8; i++) { Serial.print(\u0026#34;0x\u0026#34;); if (addr[i] \u0026lt; 16) { Serial.print(\u0026#39;0\u0026#39;); } Serial.print(addr[i], HEX); if (i \u0026lt; 7) { Serial.print(\u0026#34;, \u0026#34;); } } if ( OneWire::crc8( addr, 7) != addr[7]) { Serial.print(\u0026#34;CRC is not valid!\\n\\r\u0026#34;); return; } } Serial.println(); Serial.print(\u0026#34;Done\u0026#34;); oneWire.reset_search(); return; } This is what it looked like with five sensors attached:\nStarting... Device has address: 0x28, 0xC8, 0xCC, 0xBF, 0x04, 0x00, 0x00, 0xB6 Device has address: 0x28, 0xBA, 0x17, 0xD0, 0x04, 0x00, 0x00, 0x64 Device has address: 0x28, 0xF5, 0xE4, 0x83, 0x05, 0x00, 0x00, 0x26 Device has address: 0x28, 0xE3, 0x67, 0xCE, 0x04, 0x00, 0x00, 0x9C Device has address: 0x28, 0xE7, 0x17, 0x11, 0x05, 0x00, 0x00, 0x1D Done Now it will just be a process of elimination to find out which sensor matches which address.\n","permalink":"https://chrisramsay.co.uk/posts/2014/08/ds18b20-device-discovery/","summary":"\u003cp\u003eA little bit of work stemming from from the soil temperature monitoring project; this is an aside on DS18B20 temperature sensing device address discovery with the Arduino. Being that I am going to have a handful of sensors in the ground and therefore not easily accessible for identification I need to find a way of easily finding out all the device addresses in one fell swoop.\u003c/p\u003e","title":"DS18B20 Device Discovery"},{"content":"That Pesky Error in PyLint\u0026rsquo;s Results How to avoid PyLint\u0026rsquo;s Module \u0026lsquo;hashlib\u0026rsquo; has no \u0026lsquo;md5\u0026rsquo; member error when using hashlib\u0026rsquo;s md5() function in your code, and the consequences of doing so.\nI do not use md5 much at all these days, as it is known to be fairly unsafe; having said that, md5 does still have its uses when it comes to comparing file likeness in non-critical applications. It just so happens that PyLint does not seem to like invoking hashlib with specific constructors and will therefore mark down code ratings as a result. See the example below:\n\u0026gt;\u0026gt;\u0026gt; import hashlib \u0026gt;\u0026gt;\u0026gt; HASH = hashlib.md5() \u0026gt;\u0026gt;\u0026gt; HASH.update(\u0026#39;Test my hash\u0026#39;) \u0026gt;\u0026gt;\u0026gt; HASH.hexdigest() \u0026#39;f94b71b31c1d09d352db8b59d4f98892\u0026#39; Checking the source code for a particular file using this method of invocating md5, PyLint invariably gives me an error message similar to the following:\nraz@foobox:~/Python/soap$ pylint soap_client.py No config file found, using default configuration ************* Module somemodule E:231:SomeClass._some_method: Module \u0026#39;hashlib\u0026#39; has no \u0026#39;md5\u0026#39; member [...] Messages -------- +-----------+------------+ |message id |occurrences | +===========+============+ |E1101 |1 | +-----------+------------+ Global evaluation ----------------- Your code has been rated at 9.68/10 Hmm, that is not good at all. I want a better PyLint score, something like 10/10 would do just fine, thanks.\nWhat to do Slightly Differently Seeing as I would like my code to score as highly as possible in PyLint tests, what to do? The answer is to use md5 in a subtly different way to get rid of that error\u0026hellip;\nInstead of invocating the named constructor, such as hashlib.md5(), use the generic new(), constructor `hashlib.new(\u0026lsquo;md5\u0026rsquo;)`:\n\u0026gt;\u0026gt;\u0026gt; import hashlib \u0026gt;\u0026gt;\u0026gt; HASH = hashlib.new(\u0026#39;md5\u0026#39;) \u0026gt;\u0026gt;\u0026gt; HASH.update(\u0026#39;Test my hash\u0026#39;) \u0026gt;\u0026gt;\u0026gt; HASH.hexdigest() \u0026#39;f94b71b31c1d09d352db8b59d4fq98892\u0026#39; The interesting bit of test output now shows:\nGlobal evaluation ----------------- Your code has been rated at 10.00/10 (previous run: 9.68/10) Hooray, full marks at last! But, read on\u0026hellip;!\nA Minor Cautionary Note This does come with a minor health warning: as briefly mentioned in the hashlib documentation, the new() constructor runs slower than the named constructors. Let me put this to a very basic test, using Python\u0026rsquo;s timeit module.\nFirstly with the generic constructor:\n\u0026gt;\u0026gt;\u0026gt; import timeit \u0026gt;\u0026gt;\u0026gt; s = \u0026#34;\u0026#34;\u0026#34;\\ ... import hashlib ... HASH = hashlib.new(\u0026#39;md5\u0026#39;) ... HASH.update(\u0026#39;Test my hash\u0026#39;) ... HASH.hexdigest() ... \u0026#34;\u0026#34;\u0026#34; \u0026gt;\u0026gt;\u0026gt; t = timeit.Timer(stmt=s) \u0026gt;\u0026gt;\u0026gt; print \u0026#34;%.2f usec/pass\u0026#34; % (1000000 * t.timeit(number=100000)/100000) 3.02 usec/pass Now, with the named constructor:\n\u0026gt;\u0026gt;\u0026gt; import timeit \u0026gt;\u0026gt;\u0026gt; s = \u0026#34;\u0026#34;\u0026#34;\\ ... import hashlib ... HASH = hashlib.md5() ... HASH.update(\u0026#39;Test my hash\u0026#39;) ... HASH.hexdigest() ... \u0026#34;\u0026#34;\u0026#34; \u0026gt;\u0026gt;\u0026gt; t = timeit.Timer(stmt=s) \u0026gt;\u0026gt;\u0026gt; print \u0026#34;%.2f usec/pass\u0026#34; % (1000000 * t.timeit(number=100000)/100000) 2.41 usec/pass So we have knocked off approximately 20% of execution time in this code by using the named constructor at the expense of a drop in the overall PyLint score.\nFinally As with most things, this is not a black and white issue. If performance is an quality of important consideration in your code, you may just have to put up with lower PyLint code ratings!\n","permalink":"https://chrisramsay.co.uk/posts/2014/04/pylint-and-the-module-hashlib-has-no-md5-member-error/","summary":"\u003ch2 id=\"that-pesky-error-in-pylints-results\"\u003eThat Pesky Error in PyLint\u0026rsquo;s Results\u003c/h2\u003e\n\u003cp\u003eHow to avoid \u003ca href=\"http://www.logilab.org/857\"\u003ePyLint\u0026rsquo;s\u003c/a\u003e \u003cspan class=\"title-ref\"\u003eModule \u0026lsquo;hashlib\u0026rsquo; has no \u0026lsquo;md5\u0026rsquo; member\u003c/span\u003e error when using \u003ca href=\"http://docs.python.org/library/hashlib.html\"\u003ehashlib\u0026rsquo;s md5()\u003c/a\u003e function in your code, and the consequences of doing so.\u003c/p\u003e\n\u003cp\u003eI do not use \u003cspan class=\"title-ref\"\u003emd5\u003c/span\u003e much at all these days, as it is \u003ca href=\"http://en.wikipedia.org/wiki/MD5#Security\"\u003eknown to be fairly unsafe\u003c/a\u003e; having said that, \u003cspan class=\"title-ref\"\u003emd5\u003c/span\u003e does still have its uses when it comes to comparing file likeness in non-critical applications. It just so happens that PyLint does not seem to like invoking hashlib with specific constructors and will therefore mark down code ratings as a result. See the example below:\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" style=\"color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;\"\u003e\u003ccode class=\"language-python\" data-lang=\"python\"\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e \u003cspan style=\"color:#f92672\"\u003eimport\u003c/span\u003e hashlib\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e HASH \u003cspan style=\"color:#f92672\"\u003e=\u003c/span\u003e hashlib\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003emd5()\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e HASH\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003eupdate(\u003cspan style=\"color:#e6db74\"\u003e\u0026#39;Test my hash\u0026#39;\u003c/span\u003e)\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#f92672\"\u003e\u0026gt;\u0026gt;\u0026gt;\u003c/span\u003e HASH\u003cspan style=\"color:#f92672\"\u003e.\u003c/span\u003ehexdigest()\n\u003c/span\u003e\u003c/span\u003e\u003cspan style=\"display:flex;\"\u003e\u003cspan\u003e\u003cspan style=\"color:#e6db74\"\u003e\u0026#39;f94b71b31c1d09d352db8b59d4f98892\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e","title":"Pylint and the Module hashlib has no md5 member Error"},{"content":"Part two on the soil temperature monitoring project. Detailing some thoughts on how to get the sensors in to the ground in a way that will keep them protected from the underground environment. Following that, some subequent details on the construction of the sensor housings.\nHousing the Sensors It is time to turn to thinking about packaging the sensors into a probe housing so that they can be buried at their required depths in the soil. Ideally I do not just want to plant the sensors straight into the soil; I require a something that is much more robust and weatherproof. The solution needs to be in place for several years, and be largely maintenance free. I would not want to have to be digging the sensors up every few weeks to sort out problems.\nBuilding the Probe Packaging After some research on the net, I was inspired by a product, the Delta-T PR2, designed by a UK company (page here if you are interested) which is basically a lance with aluminium spacers set at intervals down the probe shaft.\nMy own design is a very simplified version of the PR2, but with just one sensor per probe making the design and manufacture easier. I decided to make the probe shaft from some 10mm polypropylene central heating pipe as it comes in handy 2m lengths, is cheap and is easy to work with.\nI liked the idea of the probe having an aluminium casing acting as a large heatsink with the sensor inside so I got hold of some 20mm stock aluminium and turned some probe heads on a lathe.\nFinished probe head being assembled\nAbove, you can see how the sensor fits inside the probe head and how in turn the probe head fits into the probe shaft. Both sensor and shaft are interference fits - there\u0026rsquo;s no need to complicate the build further than that.\nPrior to installation I will coat the sensors with a layer of lithium grease to prevent the stainless steel of the sensors and aluminium of the probe heads bonding over time in a damp environment.\nThe three probes along with a metre rule\nWiring Up the Sensors I decided to use 4 core telephone cable as a an option for hooking up the sensors, basically because I had a large reel of it hanging around the workshop. The cable is connected to the sensor cable inside the probe shaft and then exits via a grommet, to provide a small amount of water resistance, at the above ground end of the shaft.\nThe final step to completing the probe packaging is to discover the address of each sensor and have that printed on the part of the probe shaft which will be above ground once installed. This will obviously make it easier to identify which one is which later.\nAbove ground end of probe package complete with a hat on the top!\nFor details on how I identified each sensor take a look here.\n","permalink":"https://chrisramsay.co.uk/posts/2014/04/soil-temperature-monitoring-part-two/","summary":"\u003cp\u003ePart two on the soil temperature monitoring project. Detailing some thoughts on how to get the sensors in to the ground in a way that will keep them protected from the underground environment. Following that, some subequent details on the construction of the sensor housings.\u003c/p\u003e","title":"Soil Temperature Monitoring - Part Two"},{"content":"The Whys and Wherefores of Soil Temperature This project has its roots in a chance sighting of some, what I thought were, interesting looking API parameters on the Met Office weather data submission service. I\u0026rsquo;ve been running a small weather station for about a year with its data output being harvested by a RaspberryPi and packed off to the Met Office observation service.\nMany months ago a friend and I were idly browsing to see what other interesting data people were collecting and sending there when we spotted, to our surprise, soil temperature measurements using sensors buried at 10, 30 and 100 centimetres. These depths are certainly not arbitrary; they may represent three of the many soil horizons: A, B and C (See the Wikipedia page regarding soil horizons). This sparked off a thoughts on the possibilities of home made sensing, gathering and aggregating soil temperature readings to a service on the internet.\nYou may ask why people would be interested in measuring soil temperature in the first place. The temperature of the soil affects many processes within it, such as chemical processes, the breakdown speed of organic materials, how fast micro organisms work and the time taken for seeds to germinate. All these are of course affected in turn by the soil type and profile i.e. sandy versus clay soils, wet versus dry, plant cover, sunny or shaded aspect and so on. Being that my interest is purely meteorological, I will attempt to set up soil temperature measurement based approximately on Met Office guidelines.\nNow clearly, measuring soil temperature means needing to bury a sensor in the soil and everything that entails being exposed to the elements i.e. rain, extremes of heat and cold and possible intrusion from soil dwelling creatures and so on. I am therefore going to have to make sure that the sensor I find is suitable for the environment.\nYou may at this point be wondering what would be used to get the data from these sensors. Well, there are two simple options: RaspberryPi or Arduino - I am going to check out the Arduino because that is the ideal fit for this particular project; small versions of the Arduino are relatively inexpensive and I think that a RaspberryPi would be overkill for such a simple task.\nThe Trusty DS18B20 After some further research into sensors it seems that there is one ideal candidate sensor for the job, both in terms of dealing with the environment and also not too expensive.\nThe DS18B20 is a 1 wire digital programmable resolution contact thermometer neatly built in, amongst other options, a TO92 package. It offers an impressive set of characteristics; just take a look:\nSupports 1-wire protocol User selectable resolution from 9 to 12 bit Individually addressable (feature of 1-wire) 750ms response time Can be directly or parasitically powered via the data line Measures Temperatures from -55°C to +125°C It is available from a wide range of suppliers on line, so go google for it.\nThe DS18B20 as-is\nAs well as a naked TO-92 package, lots of people offer the DS18B20 as a packaged and wired version, whereby the TO-92 version of the sensor is potted into a small stainless steel tube complete with a heat shrunk 1 metre cable. This version seems the most useful as it provides adequate (and waterproof) protection for the sensor. Just Google or take a look on Ebay for these - they are not usually particularly expensive, especially if you are happy to wait 4 weeks for postage from Hong Kong!\nA packaged version of the DS18B20\nAs you can see from the above photo the package is pretty well made and ideal for outdoor experimentation.\nWith a decision made, it is time to play with a sensor or two.\nConnecting Some DS18B20 Sensors to an Arduino Connecting the DS18B20 to an Arduino could not be easier; in order to use it in powered mode (not on parasite power) you just need ground, positive supply and a data line, in this case, to digital pin 3. You will also need a pull up resistor, in this case 4K7, between the positive and data lines.\nDS18B20 with an Arduino\nIn the circuit above there are two sensors connected and I am using three of the four busses on the breadboard for power, ground and data respectively, from top to bottom. Basically this enables one to connect as many sensors as required.\nWith regards to finding some code to actually read data from these probes you\u0026rsquo;ll probably find many examples, such as the Arduino Playground example, some reading from the devices directly and others using Miles Burton\u0026rsquo;s excellent Dallas Temperature Control Library.\nWith a view to minimising the number of libraries I might require down the road, below is a slightly more complex method for directly addressing a number of DS18B20 probes, only using the standard OneWirelibrary.\n#include \u0026lt;OneWire.h\u0026gt; // Data wire is plugged into pin 2 on the Arduino #define ONE_WIRE_BUS 3 // Scratchpad locations #define TEMP_LSB 0 #define TEMP_MSB 1 OneWire sensor_bus(ONE_WIRE_BUS); // Probe addresses - obviously, you need to provide addresses for your own probes uint8_t dev1[8] = { 0x28, 0xC8, 0xCC, 0xBF, 0x04, 0x00, 0x00, 0xB6 }; uint8_t dev2[8] = { 0x28, 0xBA, 0x17, 0xD0, 0x04, 0x00, 0x00, 0x64 }; uint8_t dev3[8] = { 0x28, 0xE3, 0x67, 0xCE, 0x04, 0x00, 0x00, 0x9C }; void setup(void) { // start serial port Serial.begin(9600); } void loop(void) { Serial.println(get_temperature(dev1)); Serial.println(get_temperature(dev2)); Serial.println(get_temperature(dev3)); } float get_temperature(uint8_t *address) { byte data[12]; int t_lsb, t_msb t_read, sign_bit, x; sensor_bus.reset(); sensor_bus.select(address); sensor_bus.write(0x44,1); sensor_bus.reset(); sensor_bus.select(address); sensor_bus.write(0xBE,1); // Read in 9 bytes for(x=0; x\u0026lt;9; x++){ data[x] = sensor_bus.read(); } t_msb = data[TEMP_LSB]; t_lsb = data[TEMP_MSB]; t_read = (t_lsb \u0026lt;\u0026lt; 8) + t_msb; sign_bit = t_read \u0026amp; 0x8000; // test most sig bit // sign_bit == -32768 when negative, 0 otherwise. if (sign_bit) { t_read = 0 - ((t_read ^ 0xffff) + 1); // 2\u0026#39;s comp } return (float) t_read * (float) 0.0625; } ","permalink":"https://chrisramsay.co.uk/posts/2014/04/soil-temperature-monitoring-part-one/","summary":"\u003ch2 id=\"the-whys-and-wherefores-of-soil-temperature\"\u003eThe Whys and Wherefores of Soil Temperature\u003c/h2\u003e\n\u003cp\u003eThis project has its roots in a chance sighting of some, what I thought were, interesting looking API parameters on the \u003ca href=\"http://wow.metoffice.gov.uk/\"\u003eMet Office weather data submission service\u003c/a\u003e. I\u0026rsquo;ve been running a small weather station for about a year with its data output being harvested by a RaspberryPi and packed off to the Met Office observation service.\u003c/p\u003e\n\u003cp\u003eMany months ago a friend and I were idly browsing to see what other interesting data people were collecting and sending there when we spotted, to our surprise, soil temperature measurements using sensors buried at 10, 30 and 100 centimetres. These depths are certainly not arbitrary; they may represent three of the many soil horizons: A, B and C (See the \u003ca href=\"http://en.wikipedia.org/wiki/Soil_horizon\"\u003eWikipedia page regarding soil horizons\u003c/a\u003e). This sparked off a thoughts on the possibilities of home made sensing, gathering and aggregating soil temperature readings to a service on the internet.\u003c/p\u003e","title":"Soil Temperature Monitoring - Part One"},{"content":"Onwards This is the next installment on the cloud detection project. This time I am starting to test out hooking up an MLX90614 or two to an Arduino and try out pointing it to the sky in the aim of get some interesting results.\nJust as a friendly warning: calculations done here are rough and ready - don\u0026rsquo;t write to me to say that the cat died because of some experiment based on anything shown here. It\u0026rsquo;s all at your own risk, however, just remember the great Mr Churchill\u0026rsquo;s wise words:\nThe greatest lesson in life is to know that even fools are right sometimes.\nWith that in mind, onwards with the experiments\u0026hellip;\nField of View My early experiments, as shown in the first installment were interesting because they showed that IR radiation from even a moderately clear sky would measure well below 0°C; clear skies were regularly below -40°C - out of range on the thermometer. However when testing with the first MLX90614 I bought, I found that the readings were much higher averaging only around -9°C for a clear sky.\nI believe that this is mainly due to the field of view of which ever thermometer I was using. I therefore set about making some very simplistic calculations.\nDigital Hand Held Thermometer I calculated the field of view (FOV) of this thermometer to be approximately 7.16°. Compare that with the Sun\u0026rsquo;s angular radius of ~0.5°, then you have a rough idea of how much sky is being covered.\nFollowing that, working out the ratio of viewable sky (r) is not too hard. Being that the field of view is 7.16°, the angular radius (θ) is 3.58°, hence:\nr=(1−cos(θ))/2r = (1 - cos(θ))/2which gives us 0.000976 - equivalent to about 1/1025th of the whole sky.\nMelexis MLX90614 Family The MLX90614 family of sensors come in many flavours for a massive range of industrial and medical applications- you can find out absolutely everything you\u0026rsquo;ll ever need to know about these sensors in the datasheet right here. For the rest of this post I am going to look at just two variants.\nSide by side comparison: BCF on the left, BAA on the right\nThese sensors sample their ambient temperature as part of the process of measuring infra red object temperature, and as a bonus they make both of these available to the user.\nMLX90614 (BAA) Variant If you were to purchase an MLX90614 from most popular electronics web sites you will more than likely end up with a MLX90614ESF-BAA-000-TU-ND variant in a plain TO-39 package minus fancy lens attachment. Whilst this is probably the cheapest of the MLX90614 family it is also the model with the widest FOV, at a whopping 90° according to the datasheet.\nThe BAA in the name gives us the option code as described by the datasheet:\nB - 3V supply voltage A - Single zone of detection A - Standard package (giving the 90° FOV) The derived θ for this model is 45°, therefore r is 0.146447 giving an equivalent (and rather approximate) view of a massive 1/7th of the sky.\nMLX90614 (BCF) Variant The MLX90614ESF-BCF-000-TU-ND variant of this sensor is a considerably more expensive option; still packaged in a TO-39 case but with a long lens housing. With the price, however, comes a smaller 10° FOV and is built to be more resistant to environmental temperature fluctuations.\nHere\u0026rsquo;s what the BCF options give us:\nB - 3V supply voltage C - Internal temperature gradient compensation F - 10° FOV The θ for this beast is 5°, with an r value of 0.001903 giving an equivalent view of approximately 1/526th of the sky - very roughly half the value of the digital hand held thermometer.\nHooking up the MLX90614 to an Arduino Enough of the theory, let\u0026rsquo;s talk about using one.\nConnecting to the Arduino Connecting an MLX90614 to an Arduino is a fairly trivial affair. Looking from the top of the package with the tab at 12 o\u0026rsquo;clock, there are four legs; clockwise from the tab:\nGround - connects to any ground on the Arduino. Supply - providing you\u0026rsquo;ve got the Z3V model, 3.3V on the Arduino is fine. SDA - the data line which connects to A4 on the Arduino. SCL - the clock line which connects to A5 on the Arduino. In addition to the above connections a couple of 4k7 pull up resistors are required; one between the SDA line and supply, one between the SCL line and positive. I also like to add a small ceramic capacitor between ground and supply just to stabilise supply voltages a little. You may note that there are two sensors present in the circuit. This presents challenges all of its own, such as how to address more than one sensor in your software. Basically you will need to be able to change the slave address of one of the sensors, as they all come from the manufacturing line with the same address.\nI have written up this article here on how to solve that particular problem this article here on how to change the slave address on an MLX90614.\nAn example of 2 MLX90614 devices and Arduino Uno connected\nSoftware on the Arduino There aren\u0026rsquo;t a huge number of options for software to get you started on the Arduino; here is just a couple that I found:\nAdafruit have a library here on Github which utilises the Wire library in the background. NanoSatisfy host their code on Github which also utilises the Wire library. I\u0026rsquo;ve put together my own library for now which is a slimmed down and lightly modified version of NanoSatisfy\u0026rsquo;s code. Using that, I have written up a quick .ino file which polls both sensors and then outputs a comma separated set of values to the serial port.\nTesting the Two Together After amending the slave address in the BAA variant sensor I assembled the circuit as pictured. I then dug out my 4.8 metre USB lead and placed the Arduino and breadboard in the garden with both sensors side by side, pointing towards an unobstructed view of the night sky. I then uploaded the program and tailed the serial port waiting for data to come in.\nSome preliminary test data for both MLX90614 variants\nAfter a couple of hours I turned my log of the serial output into a file, uploaded it to a Google spread sheet and tidied up the presentation a little. Above is a set of 30 sampled results from the test, the samples being taken from the sensors every few minutes.\nConclusion Below I have graphed a comparison in sky/ground temperature difference between the BAA (blue) and BCF (red) variants, sourced from the test CSV data above. Essentially, the greater the temperature difference between the ground and the sky, the less cloud cover there is.\nA graph of both MLX90614 variants compared\nIt becomes apparent quite quickly that the sensor with the smaller field of view is more responsive to changes in cloud cover, giving a greater dynamic range. This makes sense and was indeed expected, as a sensor\u0026rsquo;s field of view will have a direct relationship with the amount of infra red light it is able to collect from what is, in essence, an infinite distance.\nI think that the BAA\u0026rsquo;s ability to see 1/7th of the sky is probably more than what I required for this project. I am only really interested in the sky directly overhead and I think that the BCF will achieve this with the added bonus of better dynamic range.\nMore testing will be in order, that is for sure, but the indications above seem to me to show that the BCF variant will be the one I will end up using.\n","permalink":"https://chrisramsay.co.uk/posts/2014/03/cloud-detection-part-two/","summary":"\u003ch2 id=\"onwards\"\u003eOnwards\u003c/h2\u003e\n\u003cp\u003eThis is the next installment on the cloud detection project. This time I am starting to test out hooking up an MLX90614 or two to an Arduino and try out pointing it to the sky in the aim of get some interesting results.\u003c/p\u003e\n\u003cp\u003eJust as a friendly warning: calculations done here are rough and ready - don\u0026rsquo;t write to me to say that the cat died because of some experiment based on anything shown here. It\u0026rsquo;s all at your own risk, however, just remember the great Mr Churchill\u0026rsquo;s wise words:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cem\u003eThe greatest lesson in life is to know that even fools are right sometimes\u003c/em\u003e.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eWith that in mind, onwards with the experiments\u0026hellip;\u003c/p\u003e","title":"Cloud Detection - Part Two"},{"content":"The Problem This project has its roots in a discussion I had a while back with a friend who is a keen amateur astronomer. He had been complaining to me about how he thought that there was a distinct lack of cloud free nights in our corner of Northern Lincolnshire. In fact, he suggested, the weather was so poor that it was hardly worth having a telescope at all.\nOkay, so maybe that last sentence was a little exaggeration, but his argument certainly got me thinking - how many clear nights are there in the year? Being a bit of a skeptic, I am sure that the night sky is clear on many more nights than my friend suggests.\nSome clouds, not far from where I live\nSo this got me thinking - how to measure? Being not only a skeptic, but a lazy skeptic, I wondered how I would be able to count clear nights without having to get up out of bed to go and physically look. There was no way I was going to set alarms at two hourly intervals to find that out. Naturally, the mind turns to remote detection, automation and logging. This sparked off a few days of research into how to measure cloud cover, and took me in many directions; some of which proved to be very interesting. In the end, pretty much all methods for measuring cloud cover come down to one thing - measuring radiated infra red light.\nIt seems that the easiest way to see how cloudy the sky might be (at night, don\u0026rsquo;t forget), is to look at comparative amounts of infra red light; from the ground and from the sky and then generalise a measurement from there. Essentially, a cloudless sky (whether at night or during the day) radiates less heat than a cloudy one. Compare that to the temperature of the ground under your detector set up and with a lot of calibration and a little bit of luck, you can make a good stab at the level cloud cover.\nCloudy versus clear nights and resulting infra red radiation\nWhat about data collection? The measurement method is only half the story, as once measurements are made they will need to be recorded somewhere. It seems to me that the ideal candidate will be an Arduino combined with a way of data retrieval, via serial, ethernet or some other route.\nMeasurement Methods During my research I only found two possible methods for comparing ground and sky temperature. All other possibilities were discarded on grounds of cost, my technical (and brain!) limitations, and feasibility.\nPeltier cooler on the left, MLX90614 sensor on the right\nBoth methods discussed are based on measuring the difference between sky and ground temperature.\nOption One: The Peltier Cooler According to the Kitt Peak National Observatory web site it is possible to build a thermoelectric sensor based on a device known as a peltier cooler. This detection method utilises the thermoelectric effect whereby a differential of temperature between two surfaces generates a voltage output. This is a reversal of a Peltier cooler\u0026rsquo;s normal use, where a voltage input generates a temperature difference between the two surfaces.\nIt is therefore supposed a Peltier cooler sandwiched between two isolated aluminium plates, with one plate facing the sky and the other facing the ground, should able to measure the temperature difference between the sky and the ground.\nI am assuming that there would have to be some kind of masking set up to ensure that the plates had a view of a measurable area of sky.\nBelow are some of the advantages and disadvantages of such a system.\nAdvantages: Build costs are relatively cheap; Peltier coolers are readily available at low cost. System is reasonably weatherproof; won\u0026rsquo;t require specific protection from the elements. The electronics should be fairly simple to put together; just a case of reading analogue output from the sensor. Disadvantages: System will not work well in dewy conditions and does not work at all in the rain. Temperature measurement will have to be inferred, as it does not yield an actual temperature reading. There exists uncertainty regarding how sensitive the system will be. Because of the sensitivity issue, calibration may also be difficult. Option Two: The Infrared Thermometer - MLX90614 Series I originally found this MLX series thermometer on the U.S. based Sparkfun site site and was immediately intrigued. The 90614 is a hermetically sealed factory calibrated digital infra red thermometer, packaged in a small TO-39 metal can. They calculate the temperature of the object in view by taking into account the difference between the object and the ambient temperature within the package, measured separately. The cheapest (and most widely available) model has an FoV of 90°. I do have minor concerns that the FoV might be a little too much; testing will tell me all I need to know.\nMy plan for these thermometers would be to have one unit facing the sky, another unit facing the ground and taking the difference in readings as an approximate measure of cloud cover.\nAgain, here are some advantages and disadvantages of such a system.\nAdvantages: With higher sensitivity the system should be able to detect more nuanced changes in cloud cover. Whilst rain will still stop this system from working, it should not be so badly affected by dew. Will yield an actual temperature and won\u0026rsquo;t need much calibration Disadvantages: Build costs are higher than those of the Peltier cooler - the thermometers are more costly and not as easy to get hold of. The electronics may be more complex, such as having to deal with I²C busses and so on. Early Experiments Before starting on any prototype electronics I thought that I would take some representative measurements of the sky and the ground in different conditions with a digital non-contact thermometer, just to get an idea of the temperature range I might be dealing with here. I got hold of a cheap point and shoot laser type model from my local electronics shop. The thermometer has a digital readout with a range of between -40°C to 365°C, and a (very roughly calculated) field of view (FoV) around 7° when pointed at the sky.\nTest Results Some data output from early tests\nIt is my belief that neither the MLX or the Peltier (discussed above) will be as sensitive as the thermometer I used in this experiment, for reasons that shall become clear later, therefore the extremes of -40°C should not be experienced in the future.\nFinally Over the next few weeks and months I am going to try out both solutions hooked up to an Arduino and let you know how I get on.\n","permalink":"https://chrisramsay.co.uk/posts/2014/03/cloud-detection-part-one/","summary":"\u003ch2 id=\"the-problem\"\u003eThe Problem\u003c/h2\u003e\n\u003cp\u003eThis project has its roots in a discussion I had a while back with a friend who is a keen amateur astronomer. He had been complaining to me about how he thought that there was a distinct lack of cloud free nights in our corner of Northern Lincolnshire. In fact, he suggested, the weather was so poor that it was hardly worth having a telescope at all.\u003c/p\u003e\n\u003cp\u003eOkay, so maybe that last sentence was a little exaggeration, but his argument certainly got me thinking - how many clear nights are there in the year? Being a bit of a skeptic, I am sure that the night sky is clear on many more nights than my friend suggests.\u003c/p\u003e","title":"Cloud Detection - Part One"},{"content":"Abstract So, you want to be able to get QFTest to talk to your MySQL database server during test execution; just a small matter of getting the Jython client to play ball - easy when you know how.\nThis shows how to add MySQL support to the QFTest Jython client. It requires the installation of MySQL Connector/J on to the host system and a small amount of hacking of the QFTest code.\nMySQL Connector/J From the package information for libmysql-java:\nMySQL Connector/J is a JDBC-3.0 Type 4 driver, which means that it is pure Java, implements version 3.0 of the JDBC specification, and communicates directly with the MySQL server using the MySQL protocol. Sounds ideal, so let\u0026rsquo;s go ahead and install it.\nInstallation Firstly, download and install the latest MySQL Connector/J or install it through your usual package management system. Next, link to the installed jar files (mysql-connector-java-5.1.10.jar and mysql-5.1.10.jar) in /usr/share/java to your QFTest installation library directory; in my case /usr/share/qfest/qftest-3.4.7/lib/.\n$ sudo ln -s /usr/share/java/mysql-5.1.10.jar mysql.jar $ sudo ln -s /usr/share/java/mysql-connector-java-5.1.10.jar \\ \u0026gt; mysql-connector-java.jar Now to check that the links are correctly in place:\n$ ls -al /usr/share/qftest/qftest-3.4.7/lib/ \\ \u0026gt; | awk \u0026#39;/mysql/ {print $8$9$10}\u0026#39; mysql-connector-java.jar -\u0026gt; /usr/share/java/mysql-connector-java-5.1.10.jar mysql.jar -\u0026gt; /usr/share/java/mysql-5.1.10.jar Next you need to edit the fixcp file at /usr/share/qftest/qftest-n.n.n/bin/ as this file is in charge of specifying the libraries to be used by QFTest at load time.\nChange the lines:\n15 # libjars for server and client 16 libjars=\u0026#34;bsf gnu groovy-all jansi jline jniwrapper jython png\u0026#34; To:\n15 # EXTRA JARS FOR MYSQL 16 mysqljars=\u0026#34;mysql-connector-java mysql\u0026#34; 17 # libjars for server and client 18 libjars=\u0026#34;$mysqljars bsf gnu groovy-all jansi jline jniwrapper jython png\u0026#34; Checking Run the qftest shell file with debug arguments to see if the new libs are loading:\n$ sh qftest -dbg You should see something such as the following (some elements truncated with ......):\n------------------------------------------- This is qftest Java executable: java Java arguments: qftest arguments: [...] libjars: mysql-connector-java mysql bsf gnu groovy-all jansi jline jniwrapper jython png commons jide-oss junit jxl looks poi truezip libdir: /usr/share/qftest/qftest-3.4.7/lib qflibdir: /usr/share/qftest/qftest-3.4.7/qflib plugindir: /usr/share/qftest/plugin Logging internal output to /usr/share/qftest/log/qftest.log executing \u0026#34;java\u0026#34; -Xbootclasspath/p [...] /usr/share/qftest/qftest-3.4.7/lib/jide-oss.jar:/usr/share/qftest/ qftest-3.4.7/lib/mysql.jar:/usr/share/qftest/qftest-3.4.7/lib/ mysql-connector-java.jar: [...] de.qfs.apps.qftest.start.QFTest -shellarg=-c -shell=/bin/sh \u0026#34;-options=/usr/share/qftest/qftest-3.4.7/bin/qftest.options\u0026#34; \u0026#34;-logfile=/usr/share/qftest/log/qftest.log\u0026#34; ------------------------------------------- Essentially, you are looking to see the two mysql jars are amongst all the jars listed in the output above.\nTesting Fire up QFTest, open a Jython terminal window and run the following:\nJython 2.5.1+ (Release_2_5_1:13409M, Mrz 9 2011, 15:05:52) [OpenJDK Server VM (Sun Microsystems Inc.)] on java1.6.0_22 \u0026gt;\u0026gt;\u0026gt; from com.ziclix.python.sql import zxJDBC \u0026gt;\u0026gt;\u0026gt; conn = zxJDBC.connect(\u0026#39;jdbc:mysql://testing.local\u0026#39;, \u0026#39;user\u0026#39;, \u0026#39;passwd\u0026#39;, \\ ... \u0026#39;com.mysql.jdbc.Driver\u0026#39;) \u0026gt;\u0026gt;\u0026gt; print conn \u0026lt;PyConnection object at 0x1 user=\u0026#39;user@10.10.10.88\u0026#39;, url=\u0026#39;jdbc:mysql://testing.local\u0026#39;\u0026gt; \u0026gt;\u0026gt;\u0026gt; ","permalink":"https://chrisramsay.co.uk/posts/2014/02/adding-mysql-support-to-the-qftest-jython-client/","summary":"\u003ch2 id=\"abstract\"\u003eAbstract\u003c/h2\u003e\n\u003cp\u003eSo, you want to be able to get QFTest to talk to your MySQL database server during test execution; just a small matter of getting the Jython client to play ball - easy when you know how.\u003c/p\u003e\n\u003cp\u003eThis shows how to add MySQL support to the QFTest Jython client. It requires the installation of MySQL Connector/J on to the host system and a small amount of hacking of the QFTest code.\u003c/p\u003e","title":"Adding MySQL Support to the QFTest Jython Client"},{"content":"This site is a collection of attempted writings about my interests, some of which intersect with my day job and others are just pure learning pursuits.\nBlog I am maintaining a rather sparse blog right here which has my non-project related items and general ramblings.\nProjects Most of my are projects currently electronics based, with the Open Source Arduino prototyping platform at the heart of them. If you like, take a look at the projects I play with in my, somewhat limited, spare time.\nAn Arduino Pro Mini 5v - currently the beast I base many a project on.\nCloud detection I have always taken an interest in clouds, whether observing, photographing or just reading about them. This project looks at measuring the amount of cloud cover over a hypothetical astronomical observation site. Please go here for my series on cloud detection.\nSoil Temperature This series has been prompted by some previous work to send general weather data from a personal weather station to the UK\u0026rsquo;s Met Office. It follows a project to measure soil temperature at different depths in the ground. Click here for further information on this work.\nWeather Report Both the above projects\u0026rsquo; data output are summarised on a separate Github Pages hosted site called Weather Report.\nSoftware I still work in software, and enjoy writing the occasional piece about it.\nAbout Me First and foremost and inhabitant of a quiet corner but a stone\u0026rsquo;s throw from the largely undiscovered Lincolnshire Wolds, an area of outstanding natural beauty.\nAfter years spent as a software dev, I\u0026rsquo;m now fully recovered, and have turned my hand to being a certified test/QA engineer.\nWhen there is free time from the family, the Open University and work, I like to indulge in running, getting my fingers burnt with electronics, taking part in amateur astronomy or observing the weather.\nI also love getting high, so to speak. Walking on Goat Fell in Arran, if you wondered.\n","permalink":"https://chrisramsay.co.uk/pages/about/","summary":"The obligatory about me page","title":"About"},{"content":"Built with Hugo and the PaperMod theme, deployed by GitHub Actions to Cloudflare.\nContent is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where indicated otherwise.\n","permalink":"https://chrisramsay.co.uk/pages/colophon/","summary":"\u003cp\u003eBuilt with \u003ca href=\"https://gohugo.io/\"\u003eHugo\u003c/a\u003e and the \u003ca href=\"https://github.com/adityatelange/hugo-PaperMod\"\u003ePaperMod\u003c/a\u003e\ntheme, deployed by GitHub Actions to Cloudflare.\u003c/p\u003e\n\u003cp\u003eContent is licensed under a\n\u003ca href=\"https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en\"\u003eCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License\u003c/a\u003e,\nexcept where indicated otherwise.\u003c/p\u003e","title":"Colophon"}]