Getting Bookmark Data from del.icio.us
######################################
:title: Getting Bookmark Data from del.icio.us
:date: 2017-02-19
:category: Software
:tags: programming, python
:slug: getting-bookmark-data-from-delicious
:author: Chris Ramsay
:status: published
:language: en
:show_source: True
.. role:: py(code)
:language: python
.. contents::
del.icio.us Link Fetching
=========================
Below is an outline for building a rough & ready script to get
`del.icio.us`_ bookmark data - a script very much driven by necessity.
Rationale
---------
.. PELICAN_BEGIN_SUMMARY
Why am I doing this? Well, I've been bookmarking links using del.icio.us since
March 2009 and have, very slowly, built up a steady collection of over 900
bookmarks, most of which are probably still quite useful.
The ups and downs of social bookmarking service del.icio.us are
`very well documented here`_, so I won't go into all the details, but suffice to
say that reading the aforementioned article gave me all the impetus I needed to
go ahead and *homestead* my list of bookmarks. Simple, I thought.
Then came the snag...
..
We're sorry, but due to heavy load on our database we are no longer able to
offer an export function. Our engineers are working on this and we will
restore it as soon as possible.
Great... what now? Time to write a script of course. I've initially created this
article using a `Jupyter Notebook`_ which may, I hope, explain the recipe feel
to the writing.
The Plan
--------
Well, initially the plan was just to grab the bookmarks by scraping my
del.icio.us pages from beginning to end and turn that data straight into some
form of bookmark HTML. Thinking about it more, I decided to opt for a JSON data
format; that way I could easily turn the data to any format I like at my
leisure.
.. PELICAN_END_SUMMARY
What Data?
----------
From a few investigations I found that for a bookmark to be successfully
imported into most browsers, one ideally needs a few key pieces of data:
- Title
- URL
- Date added (Human readable and epoch)
- Private or public
- Tags list
- Comment (or title if no description)
- Icon data (Optional, but provides a nice icon by each link)
Annoyingly, there isn't really a standard that browsers use for importing or
exporting bookmarks; it all seems a little ad-hoc, so I've generalised somewhat.
I would like to finish with some JSON that describes the bookmark like so:
.. code-block:: javascript
{
"comment":"Docker Compose - Docker Documentation",
"add_epoch":"1431953037",
"title":"Docker Compose - Docker Documentation",
"url":"http://docs.docker.com/compose/",
"tags":[
"docker",
"development",
"python"
],
"icon_uri":"http://docs.docker.com/favicons/favicon-32x32.png",
"private":0,
"add_date":"2015-05-18 12:43:57",
"icon_data":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAFA0lEQVR4Ae3SA3wjXwLA8d/DzERNUiRre3u2bdu2bRsfnG3btm2st7x/ze2myejhdtvz5Wx8B89+/N///c8T/JUud5u7iPpXPqOimRlMrcbKpS7nAMdfSFzpmtflr+GCaEu6a/BZttrfK5O2CydOvxD4Kn8hbeoN/hqy06qY3g3Xs/XGRpHGTs9Pvpe/gnZRiT8lXJru1VlSBrAIvDPO66gpnJXgEd4B3kN35e9+OpDxqu12RLrnK+/jT1m8+m0fcXbjzjt7MOUsQVX7ECYPEmv6k7hNEK8S5RndRDOjxdUr3uxJMmn/EPi9wfTqlW7On5IXyv1Jub4D7wgLZUSxDN6RxR1MnuGEwOrA04WtNS7cufQNHiDSdqvni++YAL7Fb9CyfYauhNxiwsI+4ZxXebZJeocHpMnwogfhHdIaCAtredKaw3h/ZWlzp/L0x8AqgCtVt9rqQMXWB+vR6JF79X3tvVOmOiAqc2PzQEcPDH2fbqYvfLXrL+44/CIbhBSzJKwVSiAELmnT6rQQ1lBWiqhYRgghl3YefmweFh9VaC0tNIZ/fCPgpwAiT86KLM18VNYrOy5868nmrqu5sCDr00PHGqe+81S9tOtidOOkUjaMyudCbKFEEITgPecn5KxBeI8PIhRiLTRRKbI6wISFEiBgXTRy5FQ0+qPhVnPHhWd2XaxqvK9GcYvK0tSMyuIVfe7HL3kpg6TauJEt9myUcKUgz7EB6CxFxm0QAp0laBUgnUOlMSiNzBLC83U8hCYvmFrztj6IruiKlYH4ote4bGvrwZ2r85O+f3p4Wptktjo7+o3+kR+9zOlwSFzpejfGBpEEUWwMfUccucGDPrVykatfcaBYYp0A79cDBMI7cBZUAHh+SZgMr0NAAB4Q+LBArEPmckPzB5/+ycHPvu4uKksmzuUnQBtA2yCqAJcAf8fFnRf9vsrTiuqcJZg6icxTsu2DuEIZrEV2WoQTx5Gry6T7Lo2tDiCcRc+OEo4fIdt2CNPYjg9L65d0cZLi6E9pFHtoDH//2+2+zWWv9L2AHuBNwLi4/I1v1Q/sAMrKZHa5sftxkzsveWOKEUiJVwEoDXgwBpxBAEiF1wF4jzDZWgis5XmpIMkg6aDzlC0X/OjrtfmxJ9og6gAR4IFFYPT8BPhNcbn3/qcPX+vl7f7NQhYC8KzxQrBHpPRrz/fzAs57wLNO/HrnrcXFCRiDQLBh5uTEjqFv3An4Ml1opwJ+U+/C2Cf6FsaOxFHPhZ0rILRCWEtkUppByoBy6GAzuXf8mscbi8/ytQ/vkUBj7vTcttHvPNop/WXoTlz6lnfmdxnnbt+759ArN+/aXZNK0ChF9NeqvH6pyAVOYZ1jjXPrA+cGbwx4D0IS5jHN2dOnN4//8LFX+/DbPww46E48RvJ7Xiu1vvY973ffq1z72k89NHi4US6VUErxvqElXvSTGVLnEd7hnQMHCNYEJqN6drY1MHv6I82p488HfsqfIK58/ZvQzVc/+RF1lVve4eqXuuxlHn5ocPBKzQ3NalQs8tmftXjHqUVmE4P0FmVzorSdldtLk9UzU1+pL46/Y9PEsa8CMX8GcbNLXYQ/5ivLpjJ40YteYuv27Veq9/UdqteqtZNLsfrRXCsJ0vZKkCdDxc7ykcrK7E+3ThybAHL+AuLOB7fy53rriQkJqN0C7iDwOx0OcPDXE6/jX0vy/wn8fwL/Yj8H6qJUvBfQ9+kAAAAASUVORK5CYII="
}
.. _`does the bookmark fetching`:
Getting the data
----------------
I'll now go ahead and start getting the pieces of data I need. Firstly I am
importing all the libraries I'll need for the entire exercise, as well as a bit
of logging:
.. code-block:: python
import datetime
import urllib
import ssl
import json
import base64
import codecs
import logging
from urlparse import urlparse
from bs4 import BeautifulSoup
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
So I am starting this by fetching one links page from del.icio.us, breaking it
down and getting each required data item piece by piece.
To be clear, before I started the next bit, I used a browser dev tool to
identify the page structure. Firstly then, fetch the URL, parse the returned
data with the fantastic `BeautifulSoup library`_ to get all the bookmark links
outer blocks.
.. code-block:: python
url = 'https://del.icio.us/chrisramsay?&page=1'
try:
r = urllib.urlopen(url)
except IOError:
logging.warning('Could not open %s', url)
else:
soup = BeautifulSoup(r, 'html.parser')
bookmarks = soup.find_all("div", class_="articleThumbBlockOuter")
print 'We have {} bookmarks'.format(len(bookmarks))
.. parsed-literal::
We have 10 bookmarks
Result! I have ten, seeing as there are that many per page. From here on in I am
working on a single bookmark - this whole next section would be in a loop over
the list of bookmarks. I am going to use item zero of the list from this point.
Firstly I get the bookmark title:
.. code-block:: python
entry = {}
bookmark_zero = bookmarks[0]
title = bookmark_zero.find_all('div', class_='articleTitlePan')[0]
entry['title'] = title.a.attrs['title']
print entry
.. parsed-literal::
{
'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science'
}
Next, the bookmark URL:
.. code-block:: python
href = bookmark_zero.find_all('div', class_='articleInfoPan')[0].find_all('p')[0]
entry['url'] = href.a.attrs['href']
print entry
.. parsed-literal::
{
'url': u'https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/',
'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science'
}
Get the bookmark favicon (ICON\_URI):
.. code-block:: python
parsed = urlparse(entry['url'])
entry['icon_uri'] = u'{}://{}/favicon.ico'.format(parsed.scheme, parsed.netloc)
Get the bookmark save date:
.. code-block:: python
entry['add_date'] = str(datetime.datetime.fromtimestamp(int(bookmark_zero.attrs['date'])))
entry['add_epoch'] = bookmark_zero.attrs['date']
print entry
.. parsed-literal::
{
'add_epoch': u'1431293424',
'url': u'https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/',
'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science',
'icon_uri': u'https://outsidescience.wordpress.com/favicon.ico',
'add_date': '2015-05-10 21:30:24'
}
Get the bookmark tags (if there are any):
.. code-block:: python
try:
tags = bookmark_zero.find_all('ul', class_='tagName')[0].find_all('li')
except IndexError:
tags = []
entry['tags'] = [f.a.text for f in tags]
print entry
.. parsed-literal::
{
'add_epoch': u'1431293424',
'tags': [u'arduino', u'circuits', u'photodiodes', u'photodiode'],
'url': u'https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/',
'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science',
'icon_uri': u'https://outsidescience.wordpress.com/favicon.ico',
'add_date': '2015-05-10 21:30:24'
}
Get any comments:
.. code-block:: python
comment = None
for a in [l for l in bookmark_zero.find('div', class_='thumbTBriefTxt').children]:
try:
comment = a.p.contents[0]
except AttributeError:
continue
if comment is not None:
entry['comment'] = comment
else:
entry['comment'] = entry['title']
print entry
.. parsed-literal::
{
'comment': u'DIY Science: Measuring Light with a Photodiode II | Outside Science',
'add_epoch': u'1431293424',
'tags': [u'arduino', u'circuits', u'photodiodes', u'photodiode'],
'url': u'https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/',
'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science',
'icon_uri': u'https://outsidescience.wordpress.com/favicon.ico',
'add_date': '2015-05-10 21:30:24'
}
.. _`same format as I generated earlier`:
Taking a look at the whole bookmark as a JSON string dump:
.. code-block:: python
json.dumps(entry)
.. parsed-literal::
'{
"comment":"DIY Science: Measuring Light with a Photodiode II | Outside Science",
"add_epoch":"1431293424",
"tags":[
"arduino",
"circuits",
"photodiodes",
"photodiode"
],
"url":"https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/",
"title":"DIY Science: Measuring Light with a Photodiode II | Outside Science",
"icon_uri":"https://outsidescience.wordpress.com/favicon.ico",
"add_date":"2015-05-10 21:30:24"
}'
Well, that looks about right to me. Obviously a collection of bookmarks in this
JSON form (dictionary) will have to be expressed as a list of dictionaries.
Time to think about getting the icon data.
.. _`method I defined earlier`:
Aside - Icons
-------------
Take a look at existing bookmark HTML files from any browser and it will become
apparent that they store icons locally in the form of a base 64 encoded string.
This seems obvious once thought about; better to locally cache a copy than have
to fetch a copy each time the browser is started up. As part of the bookmark
fetching process I therefore decided to see if it would be possible to visit the
bookmarked page itself and extract an icon.
The code is below; an explanation follows:
.. code-block:: python
def get_icon(fetch):
"""Fetch an icon, base64 encode and return it."""
icon_path = None
icon_data = {'icon_uri': None, 'icon_data': None}
# Fetch the root page, parse the HTML and get the LINK elements within HEAD.
try:
r_icon = urllib.urlopen(fetch)
except (IOError, ssl.CertificateError):
# Total fail, just return empty data
return icon_data
else:
# Carry on & get the icon rel link
icon_soup = BeautifulSoup(r_icon, 'html.parser')
head = icon_soup.find('head')
if head is None:
return icon_data
link = head.find('link', rel='icon')
if link is not None:
logging.debug('Icon URL: %s', link.attrs['href'])
# Check for relative or absolute paths
if ':' in link.attrs['href']:
icon_path = link.attrs['href']
else:
icon_path = u'{}{}'.format(fetch, link.attrs['href'])
# If we have icon_path, get the icon data
if icon_path is not None:
icon_data['icon_uri'] = icon_path
try:
icon_ulib = urllib.urlopen(icon_path)
except (IOError, ssl.CertificateError):
return icon_data
# Get the content type to avoid things like 404 HTML
content_type = icon_ulib.headers.getheader('Content-Type')
if 'image' in icon_ulib.headers.getheader('Content-Type'):
icon_data['icon_data'] = 'data:{};base64,{}'.format(
content_type, base64.b64encode(icon_ulib.read()))
# We are done
return icon_data
What I show in the code block above is neither pretty nor particularly
discriminating; it grabs the first `link` element it finds containing a property
`rel` with the value of `icon` from the `head` of the page, then extracting the
value of the `href` property.
Next, examining the `href` property it decides whether the link is relative or
absolute and modifies the link appropriately. Whichever the case an icon URL is
formed from which the icon itself is fetched. Headers are checked, content type
decided and if we get to the end we have a base 64 encoded string complete with
content type information.
Below are a couple of test run results using absolute and relative icon `href`:
Testing: Absolute links to icons
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For example: https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico
.. code-block:: python
fetch = 'https://www.gov.uk'
sixfour_img = get_icon(fetch)
.. parsed-literal::
DEBUG:Icon URL: https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico
.. code-block:: python
print sixfour_img
.. parsed-literal::
{
'icon_data': 'data:image/x-icon;base64,AAABAAIAEBAQAAEABAAoAQAAJgAAACAgAAABACAAqBAAAE4BAAAoAAAAEAAAACAAAAABAAQAAAAAAIAAAAAAAAAAAAAAABAAAAAAAAAALi0xACAfIwD///8AdHN2APHx8QBmZWgAWFdaAKurrQBKSUwAnZ2fAJCPkQDj4+QA1dXWAMfHyAC5uboAPDs/ABEREREREREREREREREREREREREREREREREC3d3d2yERETzUdEdOxRERu82e19u9ERazt3vK3DuBHeFGCqA0HtETkWEZkRgZMRMji6JKmDIxEG3enNeb1gERH4GkKhjxERERER1xERERERERERERERERERERERERERERERERERERAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgAAAAgAAAAQAAAAAEAIAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAADcAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA3AAAAPMAAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAADzAQEB7QUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wEBAe0CAgLtDAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/AgIC7QICAu0LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/CwoK/wwLC/8CAQH/AAAA/wEAAP8EAwP/BgUF/wgHB/8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/BwYG/wUEBP8EAwP/AQAA/wAAAP8DAgL/DAsL/wsKCv8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/BgYG/zY1Nf9GRUX/Ly4u/yEgIP8SERH/CQgI/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wEAAP8NDAz/FxYW/yEgIP8vLi7/TExM/zY1Nf8FBAT/CwoK/wsKCv8LCgr/AgIC7QICAu0LCgr/CwoK/wkICP8FBAT/yMjI///////5+fn/5ubm/9XV1f/Ly8v/v7+//729vf+qqqr/pqam/6empv+npqb/pqam/6qqqv+9vb3/wcHB/8/Pz//b2tr/5ubm//n5+f//////r66u/wAAAP8KCQn/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/AwIC/ygnJ//09PT////////////////////////////////////////////////////////////////////////////////////////////////////////////k5OT/FRQU/wYFBf8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8AAAD/amlp///////BwMD/cnJy/8bGxv///////////9PT0/+xsbH/x8fH//7+/v////////////j4+P+8vLz/srKy/9/f3////////////7u7u/93d3f/1tbW//////9JSEj/AAAA/wsKCv8LCgr/AgIC7QICAu0LCgr/CAcH/wUEBP/My8v/srKy/wkICP8AAAD/JCMj//v7+/+Lior/BQQE/wAAAP8AAAD/YWBg//z8/P/y8vL/RUVF/wAAAP8AAAD/EA8P/6Kiov/n5+f/Dg0N/wAAAP8ZGBj/zMvL/6enp/8AAAD/CwoK/wsKCv8CAgLtAgIC7QsKCv8AAAD/PTw8//f39/+Xlpb/aGho/1BPT/9PTk7/8fHx/xwbG/8AAAD/NDMz/2dmZv8AAAD/zc3N/6moqP8AAAD/enl5/ycmJv8AAAD/Ly4u/+jo6P9DQkL/WFdX/2RjY/+mpaX/8fDw/yUkJP8CAQH/CwoK/wICAu0CAgLtCQgI/wAAAP+srKz//f39//n5+f9YV1f/7u7u////////////x8bG/46Njf86Ojr//////6Ghof/Gxsb/wcDA/7i3t//39/f/Ly4u/5mZmf/GxcX////////////V1NT/ZmVl///////8/Pz/k5KS/wAAAP8KCQn/AgIC7QICAu0AAAD/XVxc//////+lpKT/rays/wkHB/9OTk7/2dnZ//f39//p6en/c3Nz/0dGRv//////v7+///Py8v/j4+P/xsbG//////80MzP/l5aW/+np6f/6+vr/wsLC/zw8PP8XFhb/tLOz/6ampv//////QUBA/wEBAf8CAgLtAQEB7QAAAP9ERET/p6en/21sbP8oJyf/AQEB/yQjI//v7+//5+fn/ygnJ/8AAAD/RkVF/0A/P/80NDT/9/f3/+Pj4/8dHBz/VVRU/zU0NP8AAAD/ODg4//n5+f/e3d3/ExIS/wkICP8wLy//enl5/6ampv8yMTH/AAAA/wICAu0AAADtQkFB/7Ozs/+dnZ3/IyEh/wAAAP8AAAD/c3Jy///////v7+//IB8f/wEAAP8DAgL/FhUV/9vb2////////////8XExP8KCQn/BgUF/wAAAP82NTX///////////9YV1f/AAAA/wAAAP8vLi7/q6ur/7S0tP82NTX/AQAA7QAAAO3BwcH///////////9vbm7/AAAA/wAAAP9gX1///////87Nzf8MCwv/CAcH/wcGBv8eHR3/SkpK/z8+Pv9APz//S0tL/xoZGf8HBwf/BQQE/xkYGP/i4uL//////0VERP8AAAD/AAAA/5KSkv///////////6empv8AAADtAAAA7ZiYmP///////////09OTv8BAQH/BQQE/yEgIP+8vLz/Q0JC/wEBAf8LCgr/DAsL/wgHB/8AAAD/AAAA/wAAAP8AAAD/CQgI/wwLC/8MCgr/AAAA/1taWv+xsLD/Dg0N/wcGBv8AAAD/a2tr////////////e3p6/wAAAO0BAQHtFxYW/2JhYf9SUVH/AQAA/wQDA/8MCwv/CAcH/wkICP8DAgL/CAcH/wQDA/8JCAj/AwMD/1VUVP/Kysr/xcXF/0dGRv8DAwP/CAcH/wQDA/8IBwf/BQQE/wkICP8JCAj/DAsL/wEAAP8GBAT/YF9f/11dXf8NDAz/AQEB7QICAu0AAAD/MzEx/8LCwv+1tbX/IB8f/wUFBf8IBwf/CAcH/wYFBf8NDAz/IB8f/wEAAP8LCgr/39/f////////////x8bG/wMDA/8FBAT/IyIi/wkICP8HBgb/CAcH/wkICP8EAwP/NjU1/8XFxf+3t7f/IyIi/wIBAf8CAgLtAAAA7QMCAv/BwcH///////////94d3f/AAAA/wYFBf8AAAD/FxYW/7y8vP/s7Oz/goKC/wUFBf/Z2Nj////////////ExMT/CAcH/52dnf/u7u7/paSk/woJCf8AAAD/AwIC/wAAAP+joqL///////////+enp7/AAAA/wEBAe0BAQHtAQEB/42Njf///////////1ZVVf97enr/wMDA/1dWVv9EQ0P////////////w8PD/CAcH/42MjP///////////3d3d/8fHh7/////////////////Kikp/3Z1df+8vLz/X19f/29ubv///////////3Jxcf8AAAD/AgEB7QICAu0JBwf/Dw4O/11cXP8/Pj7/NzY2////////////7u/v/y4tLf/Pz8///Pz8/6moqP8AAAD/NjU1///////8+/v/JCMj/wUEBP+/vr7/+/v7/7Gxsf80MzP////////////8+/v/HBsb/0tLS/9QTk7/CgkJ/wkICP8CAgLtAgIC7QwLC/8IBwf/AAAA/wAAAP8lJCT/6+vr///////V1dX/CwoK/xkYGP80MzP/GhkZ/5CQkP9VVFT/4uLi/9HQ0P9mZmb/kJCQ/xAPD/81NDT/DAwM/xkYGP/o6Oj//////9zb2/8TEhL/AAAA/wAAAP8JCAj/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/CwoK/wMCAv85OTn/bWxs/yopKf8GBQX/BQQE/wAAAP8MCwv/6ejo///////5+Pj/9/f3///////W1tb/AAAA/wAAAP8GBQX/BQQE/zg4OP9ubW3/Li0t/wYFBf8LCgr/CwoK/wsKCv8LCgr/AgIC7QICAu0LCgr/CwoK/wsKCv8LCgr/CwoK/wEAAP8AAAD/AwIC/wwLC/8LCgr/BwYG/xMSEv+8u7v/sbGx/+7u7v/e3t7/tbW1/7e2tv8DAgL/CgkJ/wsKCv8LCgr/AQAA/wAAAP8CAQH/DAoK/wsKCv8LCgr/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8KCQn/Dw4O/w8ODv8dHBz/9vb2/9vb2/8ODQ3/FhUV/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/AAAA/2hnZ////////////0tKSv8AAAD/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/AgIC7QICAu0KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8ICAj/JCMj/zc2Nv83Nzf/Hh0d/wgICP8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8CAgLtAAAA7QEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8AAAD/AAAA/wAAAP8AAAD/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wAAAO0AAADzAAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA8wAAANwAAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=',
'icon_uri': u'https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico'
}
Testing: Relative links to icons
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For example: /images/favicon.png?1486118657
.. code-block:: python
fetch = 'http://scrapy.org'
sixfour_img = get_icon(fetch)
.. parsed-literal::
DEBUG:Icon URL: /images/favicon.png?1486118657
.. code-block:: python
print sixfour_img
.. parsed-literal::
{
'icon_data': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMAAAADACAYAAABS3GwHAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAABmJLR0QA/wD/AP+gvaeTAAA4iklEQVR42u29aXRc15WY+50agEIVClNhIkFwAOeZFGkNHDQPlGxLlizLsiTb3Z2snjJ0nE6vl+QlnbeSvJe1ko7dU2J3J+6kB7fdtiy3OrZstayRlEjJnGcSIOYZhZrnuvee9+NUASCJGgAUUABY31ItFYFbt+692PucvffZex8oUaJEiRIlSpQoUaJEiRIl7hBEsS9gufFL37sP1HMtB8oAS+q9AMxT3k+HAcRT/5ep9xqQSL2X//vF48W+xWVFSQFmSUrQARxAM1CfetWl/r0SaASqgSaU8DtT703TnFKghHwYiAE6MAL4gVFgMPU7D+BOvYaBMEBJMWZHSQFyMEXQTSghrgG2pl4bgTUoYW9ACbyT6QW8EBhAEKUQYyil6AHagSuplw+lPAaUFCMXJQWYhilmjANwAa3A3cC9wA6gFqgCKij+M5RAFAgAXuAicAL4BOgDxlGzRMl8moZi//EWFSnBtwNbgN3AAZTgb0TZ8+ZiX2Oe6Ci/oR2lCB8B54CrQKSkCJPc0QowxbwpA9YCDwMPALuADamfLwcSQAdwHngfeAfoTv38jjaT7kgFSAm+GWW77we+CNyDsuHtxb6+eSaC8iE+Bv4GOInyJfQ7URHuKAVICb4V2Ac8CjyBUgBbsa+tSMRQCvAm8HPgFJC8kxRh2SvAFIfWDtwHvAQcBtooaLRGIiVI5MR7kxAIYcYkTJiE+qqp729GYGBgGDoAhjQwpIGUOoaUCKGOEYiJ9wXEADqBo8B3gOMox3rZO87LVgGm2PcNwP3Al1GCX1uY+5ZIKTGkEniruRxHmROb1YHd6qDcYqfaVkdleQ3O8hqc5dUIBM7yGhzlNdNcgCCWDOONjiKRBON+gnEfobgPf8xDXIsQ06JEEkHCiSBJPQ6IlJKJwtySiiiNo3yE7wMfoMKty9ZPWHYKMEXw7SgT56soc8cxl/OqkV2NwQA2i516xwqanK3UO1bgsjdR72imyuaitqIBZ3kNVnNhfOikniAY9xGMe/FERhiPjOCLuhkJ9jEc7MMdHiKmRaa9zjkQRplFf44ykSKw/BRhWSlASvhtKFPnH6AUoH6251MmiIHVXI7NUkGVrY61tZtpc21nVc16amwunLZa7FZnBrNm/jCkQSQZJBjz4ouNMxToocN9gS7PFQIxDzEtSlKPIzKaXHnjRinAt1GmUWw5KcGyUIApUZ2dwMsoc6dptuczpI5ZWHA5mmmsbGFj/S62NN7Fiqq1VFgdWE1WxAILfC6klCSNBNFkmKFAN1dHT9PuPs9oaIDx8DC61DCJOS1jjAB/ifIRLrBMokZLWgGmmDv1KKH/ZWA7M3Rup5oNdmslGxt2sbVxH+vqtrG6diM2y9KMjMa0CL3edro8l7kyeor2sfNEkqG5mEkGcAn4XyhlcMPSNouWrAJMCWkeAv4F8BAqNSFvpJQYGFRYHDRWrmTXigPsXnmQZmcrzvLalHO59JFSEox7GQ72cW7wQ84PfcRoaJCoFsaEaTb3GQXeBX4POMYSDp0uub/wlFG/BeXg/hNU9uUMUNGbalstba7t3NXyANub76a2on6uZsKix5A63qibS8OfcHrgfTrHL+GPeTHNLpI0DPwRylEegKU3GywpBZhi6z+AGvUfRuXX54WUEiEEzvJa9rYcYteKA2xr2keFtbLYt1YUoskQl0dOcX7oI84MHCMY9048oxkQR4VNfw+VZrGkfIMloQBTRv064BXga6jcnbyQUoKA2ooGdjbfy+G2z7C6ZiPllhlZTMuWuBal19fO0c4fc2H4BN7oGEhmqgjdwDeAv0LVLCyJ2WDRK8AU4d8O/A7wPDOI6Uspcdpq2Nl8Lw+sf4YNrh2YTZZi39aiRDc0OsYv8v6N17kwfIJgzDdTJQgDrwL/BeUsL3olWNQKMCWN4UHg36PSk/OK8EgkFpOVrY37eGjDs2xp2Iu9zFnsW1oSRBJBro6d4d2OH3Fl9BSakZxJxMhApV//LvAeizydYtEqwJRFra8A/wpVeZXzeiUSAayq2cBD659l36oHqLa5in07SxJ/bJxT/e/z7o0f0e/rQJJ36FSiKtX+E/AXLOLFs0UZ8kgJfz3wz4F/g4ry5BZ+Kaksr+bQuk/z/M5fZ8/Kg0s2hr8YsFnsrKvbysb6XSAEY6FBElosH7NIoEpHD6NC1Rf3PN8aOftqf7FvadoLXTTcEuL818CvkEeqshr1BWtrt3Bky5fY23KYMvOdmuE8PyT0GGcGjvKzq9+l23t14pnnQQz4M+D/YxGGShfVDLDn+VZQ5YjfQBWp5AxxSmlgL3NyaN2neemuf8bG+l1YTNZi38qyw2yy0FK1ju0r7iapJxgN9adyjXIqgQVVXroJOAu4F9NMsGgUIDX6bwO+Dnwa9eBysrJ6LZ/d/ssc2fwlqm11y2b1djEihMBR5mRLw16qKlyMhQcIxf35fNQMbEaVmZ7Z83zr2GJRgqJLyxSzZyvwX1EZnFkjPVJKTCYT2xr38/T2X2F9/Q7My3wFd7GhS50b7ov83aU/4/LoSQzDyGfwMVCZpb+NauFSdHOo6FKTMnu2An8APEYewl9usXF/22f5wu7fpLVmw4KnIpdQlW0uRzObGvaQNBIMBrrQDC2XEghgPWo2OMkiMIeKNgNMGfm3MTnyZ70eiaSqvJYjm1/i4Y3PlSI8i4SYFuGd9tf42bW/JhD35uMcSyZngstQvJmgKDPALWbP14HHyWPkb6hcyed3/hoPrP8cZea8U4BKzDMWk5V1rm1U2+ro83UQTgTzmQnaUD7BacC95/lWijEbFEUBUmbPeuAPycPmN6TBquo2Xrrra+xf9RBmU9EttxK3YBImWms20ly1ml7fdfwxT77m0EZUpZn3jlCA1Oi/AviPwOfINfIj2dywlxd2/yY7mu8pRXkWMUIImpytrKhajTs8zHhkOJc5lJ4J6oATe55vDS20EiyYNE0xe6qA/wf4R+TovGZIg61N+/jSnn/KmtrNC/pgSsyNHu81vnv2D7kyciqfIEUC+G8ouQjAwvkECx0+sQK/AfwaOYRfSsm2pv28vPdrJeFfgqyp3czLe7/Gtqb9Kh09O2UomfgNlIwsGAtiAk0Z/V8A/h1qysuIlJLNjXv50t5/yuraTQv5PEoUkGpbHa016xkK9uAOD+UyX62olPc+VO7QgjjF864AU4T/PuA/oxyfjBjSYL1rOy/d9c9YV7d13h9AifmlpqKelup19Pk68ERGcymBE7VGcBboXwglmFcFmCL8q1Gx/gPZjpdIVlW38cLuf8Tmhr3zeuMlFo7aigZcjmZ6vdfzWSdoAlahutL551sJFsIHKEOlNT+a7SCJpNrm4rmdv8qO5nsW4LJKLCQ7mu/huZ2/SrXNNdGWJQuPomRm3tvTz9sMMKWa60vAvyRL2/F0Hv/nd/4q96x5rJTasExpcrZit1bS4b6Yq67AhDKFeoEL8zkLzIsCTDF99qHqQ9dkO95qtvLE5i/x0IbSCu9yxiRMNFetRpc6neMXMaSR7fAK1CLZJ8DQfCnBfA61VahY/85cB97V8gCPbfxCKbfnDsBmsfPYxi9wV8sD+Ry+EyVDVfN1PQWfAaaYPq+gujhkjOtKKdlYv5MX9/4WLscMe1uVWLKUWWy0VLfR673GeGQkV2QobQqdnY9ZYL5mgN2o3j0Zh3RDGtTZG/ns9l9mZdXaebqMEouVlVWqkKnO3pjLFLKjZGn3fFxHQWeA1Ohfieri8CRZUi2s5jKObPkSB9c9VXJ671DqK1eiGQk63BdyKUFD6v/v73m+NVHIWaBgkjfF8X0EeI4ceUZ7Wg7x8IbnSpVcdzBmYebhDc+xp+VQrkMFSqYegZtkbc4UeuhdAfwWarfFaZHSoNnZyhObXsRRNm++TYklgqOsiic2vUizsxWZfRZoRMnWikJ+f0EUYIrj+xxwMNNxEkmZpYKHNjxLm2t7Ie+jxBKmzbWdhzY8S5mlItci2UFS1kWhZoFCzgCbgF8nx+rdtqZ9HFxbsvuXC3oyQcQ3RtA9SNA9SMgzTDIendE5TMLEwbVPsa1pX65Dy1AyVrAMyTl3iZ0y+r+A6umTAUm1rY4jm18qmT5LHF1L4BvqZqz7Ev7RXqJ+N1oytWulyYS9ugFX62Zatt6NvaYhr3M6yqo4svmliT3OsriQW1Cy9h9/6Xv3zbnv6JwKYqZMQ3tQW+bsyHb8U1u/zLM7/mGpcdUSREqDeCSIu/syXad+jn+0l1jQi5aM37JNq9oj2Vpuo7JpHRsPPMu6HXcjTLlnfM1I8qOL/5M3rvxlrkMvorbEOgtzK54pRJ/w8tTFZBR+Q+qsrd3CgTVPlIR/CRIP+xnpOEfHJz/DO3gjNdpLtQOl+XYREqhZwtt7hfd7ezBe/BobduVOcLSYrBxY8wQXh07Q7b2abbeeHagNzy+jqslmTSEUoBXlmEyLRFJmtnHvmsdpqW4rwNeVWCgMXcPdc4Ubv3iToeun0eIRyHtjboHJbEaGfZx44zs0tKyl2pV7486W6jbuXfM4g4FukkYiW+r0s8CfADfmco+z9kRvyfZsyXighDbXNg6sfXIu11ligYmFfFz54FVO/OAb9F08NiH8aaRhIIQJaeiQpeTRVl7GcPtZOs6dyPu7D6x9kjbXNnJkTa9BzQJzigjNNRSzAfg8WfJ9yizl3N/2NNW2uvzPWqJoSGkw2nWRT177Qy6/9yrRYMohnSL8JrOVlVv2s+ORL7F61/2U2Z0ZlcBiMWMWBtdOHSMeDed1DdW2Ou5ve5oyS9bMYCtK9jbM5X5npQBTNO4IWSI/UkrWu7azo/nuuVxjiQVCS8ToOvU2p/7uWwy3n0Ea+m2JalJKXK0b2ff0b7D1gef51LP/mDW774cMYW0hBGVWKz3XzhELB/O+lh3Nd7PetT1XQf0WlAzOenV4LjPAClS+z7Rqqha9bBxYc6S0Q8sSIOof58LPv8O5N/+cwFhfxuOEEDhqGrFXqb+ppcxGpWslJrOZbDZLyDdOLBLK+3qqbS4OrDlCmcWWbXGsHCWDs14dnrECTNG0/ajtSqfFkAZrazezJffiRoki4x/p4cxPv03Hxz8lGQ0hsixSSinxDHQwfOMs0cA47t6rDFz5BD2ZIJtjbBg63pGBGV3XlqZ9rK3dnCtR7gGULM5qFphtFKgc+AwZ050l5WYbe1YepMGxcpZfUWK+kYbOSOd5Lr/7fcZ6rihzJ0fnPSEEAfcAJ1//JhXOOuKRAGHPSMY4v2EYJJKaUo0ZdvVrcKxkz8qDdHmuoBkZFcyOksW/R+1ZPCNmawKtBp7K+GAl1Nmb2NNyeJanLzHfaIkYnSff4uTr35wU/nyRkrB3BHfPZYLugaxJbElNJxAMY7GWUdMwc0tlT8th6uxN5Oit9RRKJmfMbBXgKSZztG9DCMGO5ntocrbO8vQl5pNo0Mvl937AuTf/gog3Z6+eaRHChDCZs5pLAD5/iEg0RsOqNiqrZx4JbHK25tMTtoEsA3I2ZqQAUwpeHiPL/l0VVgf7Vz1YyvVfhPhHejj7xre5/tHrJG+J7ReacCTKyJgHKWHHfY9gr6qd8TnMwsz+VQ9SYc26N3o5SiYrZ+oHzGYG2I3q6z8thjTYWL+LlprSqu9iwtB1hjvO8slrf0TfxQ8xtNRuLlIiDV29cvfwzJtYPMHA0DjRaJwNu+9lz+GnMOWRDzQdLTVtbKzflcsZ3sosyibzvqIpmnUPsDbTcWaTme3Nd1NVPnNtLzE/6Mk4Xad/zqnXv4lnoEP9MCX8Nmcta/c8xLp9j+KsL0zAIhZP0Dcwij8QxNW0iodf+DVqm1pmfb6q8lq2N9+dy6JYi5LNGUWDZhoFqgPuJYPiGNKgpWqd2li5xKIgFvJy/aP/w41f/D2JaHDSZpcSS3kFmw8+w/q7jwDg6W/nF3/7x4Q8wzlt+0yEQhH6h9wEQxGq65o48pXfYsPOuS+Erndtp7GyhaFgb6ZaEhNKNusAT77nzesup2jUKlST2wxIWqs3sKp6fR5nLTHf+Ia7Of3j/8H1j/6OxC3xfYnEanNQs2IdljIbljIbrtVbKLdXkSvkMh26YeAe99PVO0wgGGbFuk08+5u/y+7DTxbEz1hdsynVKTzrtd2HktG8Z4GZzgA7yLjqJim3VLChfidW87y3dCyRBSklIzfOcfHtv8bT3w5wWxRFCEE8EmC4/Qy1LRuwWMsZvPoJEb87Y1pDJmLxBKNjXsY9fnQJm/Ye5LGXfpN12wq3CGo1l7GxfhdnB4+R1FXxzTSsQMno+XzPOxMFKAMeJkMrFYmq6tnSeFfBbrrEzNESMXrOfcDVo68RGh/KUogiMJIJbvziTcZ6rlBud+IZaCceDuQVFhVCoOs6/kCY4VEPoVCECmc19z78DIef+Qp1TasKfm9bGu/CUVaFNzqWac3ZjJLRV8mzTmAmClAJfCrbASuq1tBYWfgbL5EfsaCXqx++Ttept0hEw7mrsIRAi0cZ77uGQA1i+a4JhMNRxsb9jHv9GIakZf027n/2l9l16Aks1vmxABorV7Giag3e6Fi2wz6FktW8/ICZKMBmIGP/QoFgc8MerOZSxVcx8I/2cvnd79N/+XgqVz9Pu1tMlpzk8wldNxj3+Bl1+4hGY9idNex96LPc8/jnaV67eV43MbSarWxu2MOVkVPZDmtGyWpedZIzUYB9qB08pj+RycoG185sZWwl5gFD1xm5cZbL7/2A8d6rKSkurBCmzZ1AMMKo20sgEMJsLadt56c4/MxX2bT3AGW2+W9sbBJmNrh2YjFZ0YxkpsOcKFktjAJM8aa3o1pW34aUksbKFlyO3CVvJQpHMham68y7XPvwb4n63UzaMZJCKYGUklA4invcj9cfRNN0mlo3cNfDT7Pv4WfyKnMsJC5HE42VLQz4uzLNNhUoWeWXvndfzoL5fGeAOtR+rtM/JAxaqtfjLC1+LRjRoIdrR39E5+m3ScYiE3n6NSvaiAbc+Ia7MXRtTt+RSCQZdfvweAPEEwnszho+dehJ7jnyBVas24ypCBuWO8traaleT7//BiJza9v03sM5/YB8FaANVfyekVXVbaV+PwuAlBL/SDcXf/5dhtpPTwi5s76FfZ/9NepaNpCIhbj48+/Sfe69Gcf0hYB4XMPnDzI27icciVBRWc22PQc58JmXadvxKaxlxdvExFFWxarqNj7OflgrSmYLpgDNZOj3KZFUWB3UOwrasrHENBi6xtD1U1x+7wd4B1UzBCEE0tBxtW5S8fzyCizlFTRt2E3vhaMzmgV0XcfnD+H2BAiFIwiThY2772PfI8+w495HsTmceZ9rPql3rKDC6iCmRTN1jWgkS8BmKlkVYIr9vwKYtq5RSomzvLZk/88zWiJG1+m3uXr0R0T87pvtXyHwj/QS9o5Q1bAKPZlgvPcqhqHnPK9AreIGghHcHj9+fwgpBE2r1rP/0WfZffhJahqaKbRjPRdcjiac5bVEk5FMfoCLlALk8gPymQHKUHs1TUt6d8emylLu/3wR8Y1x5YMf0nPufZLxyG15OkKY8A118vEPvkH92m2EPcOM9VzOav4IAYaRcnA9ATy+ALqmU9+yhp33Pc6nHn+O+hVr8urottA0VbZSbXMxEsq6T8A6lHxnnQLzUYAKshUdS3CW11BVansyL4z3Xefye99nuCPdpWF6gZRS4h/txT/am/OcUkI4EsPjVYKfSGhU1TWw7e6H2P/Yc6zetGte4/lzpcpWh7O8JhXxynjYasAB+LOdKx8FsJPFAbaYLNQ7VpS6PRcQKSW6rjN07SSX3/0e/uHuVELZ7IUylf1MPJHEPe7H4/UTiyewOarYc+/93PfUF2lZv43yCsesv2OhMAkT9Y4VWEyWbB0jWlGyO2cFqCCLQ2E2WUr2/ywxDANd11MvA03XSCaTJDUN9+gIF3/+IxjrxGyee7gxHk/i9anITjQao9zuYNs9h7j3iRdo2/mpJSH4U3E5mjCbLNkWxJrJsG41lXwUwEZGBZBYTNZSBCgPDCmVcCeTJBIJEonkhLBrmoamJdE0fcrxJipW7cYz3k+lDM86pVjTdDy+IB5vgHA4itlaxqa7DrHn/qfYds/DOKpqiv1oZoWaAazZukU0o2Q3KxkVYEoEqIZMK8CAyWQuNb6agpQSwzAwDIN4IkE8HicWixONxdB1fWLUNwx9wkedzt42mQSrN24n7Bsj0nUUuzl3RCeNEAJN0wgEI4yN+wgEwpgsVprXbubuxz/PzoNP4KytX9R2fi6qbS5MJnM2N6ACJbtZI0H5zACryGJ8moWFGlt9sZ9H0TAMg2RSI5lMkEhqxOIx4rE4sXgcTVMBCCnlRL3tpNCJnIO61Wpl/a77uOgfI+65hM2cq1+sSlZTkR0/Pn8IKSVNazay5/6n2P/Is1S5Gpe04KepsdVjFlnFV5AqjslGLgUQqF26Mz4xkzDlqthfVkgpSSY1YrEY0ViUeCJBIp4gkUig6ZlH6dkInZQSR2Ula3bfz42PPFgSg5hNIsOxEImoFGWfP0hS06lpWMG+h55m16EjNK/dNOui9MVIhdWRK/AyVXYzjhv5KMBKspRO1lY0FCUnZKEwDANN04gnEkQjUcLRCIm4EnZd16cZ2QuLlJLG5hYC2x5g7MyPqZXBie9Kx/Jj8UQqshMgkUzirGlg/4FHufvxz9O8eiOWIqYuzBcmk5naigZ8MXfGQ1CyO2cFsGT7ZYXVgWkRrRLOFQnomkYsbbtHo0QiERLJ5LRtQxbCnBAC2jZtI+AZI9D9ATWWJBKIxhJ4vMrBjcbi2J3V7Dh0P/sf+Rzrtu+ft8KUxYAJQYXVkUvyLOSIHefjA+RqFpnzkMWOlBJN14lEIoTDEWXHxxMkk8m5n7xAWCxmtuy9h3PBcfwjp4kF/bg9fsKRGGXlFWzed5h7nniejbsPYHNUFvtyF4DcfUzJQzBzKYAJlVWXwQQS1FbUL0kTyJASLZkkFo8TCASJRCIkNW3CrFlsjqKUEmdlJSs27ObUlU+IjwxjLq9gzZY9HPj0l9iy/34qKqsX3XXPF8oEqiebe8qk7GZ0zvLNBcqIzWJfMg9dAolEYsKsCYXDxOOJ20ybxXo/Qggi8SQD8UpWt2zl4ac+x+7DT1JV1zj3ky8xhBDYLDmr0HLagIXYJG/Ro+s60ViMQCBAOBIlHo+j6/qiFfRMJJNJRkaGicoydj72PAc/+9ySnH0XE8tWAaSUxBMJwuEw/kCAaDR2k3mz1IRfCEEoFKK/rx+r1cq2HbtKwl8Alp0CGFISDocJhUIEgiHi8Zv3TFhqgj8Vr9fL6OgojY0NrFxZSj8pBDmX0nKtPAohFkUMSNd1QuEwPn+ASCRMMjm3etjFhmEYdHV2kUgk2LF9BxUVOfO8ljWC/AazXDKcUQE0HQwDzGYwZ1lwiyUjGAVsqz1TNE0jHIng9foIhcMYhjH3ky5CIpEInZ2dyvzZvhWr9c7uv2RISSwZyXqMboCuQ7YF8IwK4A6CPwINVVBbmSnYJPFERjEMDcwLu9qo6zqhUAif308oFEZfpoIPaqTr7OzE7XbT1tbGmtWz2g1oWWEYGp7IKJmqYiRKfscCUJ0lWJRRAWJJCMZSH85WeTMPjZiy3riURKNR3OPjhEJhtPRGD8sUIQTxeJz29nY0TWPnju00NZXqL9RCWJZfS0hoSobLs0yWWdMcTDkW2ySQ0GJIFmb0TSQSeLxePF7fRKblchb+9P319PTQ09NLY0MDe/fuKfYlLQokRkr2sozNIiXDWc6TUQHGQ5JeNzhtgsYs7X680TH0PLoPzAXDMAgEgrg940SjsYJu5bOYEUIQjUa5cuUK0UiEw4cOsm7dumJf1qJAN/RcTXIJRKDXnV1WLNk+7PNKoo0kJJkjPZqhZStLmzOJRJJxzzher4/kMjd3pmNwcJD26+3U1dby4IMPFKQ8cjmgGUk0I3OkTwLRhEz4vJIae2aZyegfRxNAO4Yh6RRktnF0I4k7PDwvNymlxOvzMeYeR1uCK7dzJRqNcvr0aeLxOPfddy+tq0qt59O4w8PoWQbeVKZ4J+0Y0Sw7BWRUgBE/0AH+SGaLQzVV0vHFsk9Fs0UIgbOykvLy8jvG7Jl67zdu3KCrs4sVzc3ce++9d3zocyq+mDK9Mw2JUirZpSMlyxnIXiJUgYxraOGMG9ALdKnhj+W9J9mMsdsrcNXVYjYvn2qmXAgh8Hg8nDl9Bk3TOHToIBs2lPZdm4o/5kGXGplc3HAc4hoaFdnXcrNLVWdSIhmMJTEyaVpSTzIc6JnXm62traXKeec03tV1nUsXL9HX18eGDet58MEH7jjzLxfDgR6S+vQmkABiSQwkg3QmZ68AUlbIeJJAJI7MNNfoUsMTnR8TKI3ZZKK+3kVZ2fKtcEojhGCgf4Bz585hs9n49FNP4nKVum7ciic6lpoBpkFAJI6MJwlIWTF7BVj7b3UiCfpDsewGuC/qxhuZXyWwV1RQ73Itq8LuW0lnfH700UcEAgEOHz7E7t17in1Ziw5vZAxf1J31mFBMykiC/rX/NnuIPrM0fesEAx6IBvGF40TJ6AgLIokA3nlyhKdSW1NNlXNxtOieD3Rd59y5c3R3d7NhwwaOHHkCm235FbTPFW9sjEgikDk4LyEcJxoN4hvwAN86kfFcWYdTzQB0YgmN4XjG2UYQjPvwhEfm/cbNZvOyNYWEEPT29HLm9BkqKir4zKefYuWKUsrzdHjCIwTjvowKENcgoTGMTkzLkaSQ254QRKMJhqOJ6dMihBAE4j6GQ30LcvP2igoa6l2YTMvHKRRC4PV6OXbsGOFwmCNHnmDfvtJ+y5kYDvURiPumDQwIodawogmGEURznSu3ApiIBKP0ReLZcyqGg73EtEjO0xWC6urqZRUVisfjHD9+nL6+Pu66ay+PP/YYFsuyq1UqCDEtwnAwcwt4AUTiEIzSh4mcAplbAaxEA1GGQrHMhQUmBP2+GwRi3gV5CJaUKVRevvTtYyklFy5c4ML5C6xZs5pnP/cMVVXL18+ZK4GYl37fjYy9qCQQikEgSi8WwrnOl88MkMCg3RdWRTLTIYSJ4UBPzuSkQmKvqMBVV7ek4+NCCLq6ujhx/AQ1NTW88IUvlJLdcuCNjjEc6Mm4UYimgy8MGHRhJmdZYHYFSHvPgiF/RI4ns0SUEnqcHu+1BU1ZWOpRoeHhYT54/wOSySRPP/1Z9uzZXexLWtRIKenxXiOhZ0xNIKmDPyLHEagEtSwRIMhnBgAQDI+HGI0nM/sBEsnV0dMkjURepywEZrOZetfSiwoJIQgGgxw9epSxsTGefPIIDz5w/7Je4ygESSPB1dHTGXeFEUA8CeMhRicUIAf5PvHOhEbfqD+zHyClpNtzFU9k/sOhU0nnCi0VUyid43/s2DG6Ort45OGHefLIE0tOiYuBJzJCt+dqRitDAqN+SGj0AZ35nDMvBWiowqMbdI4FZMaNB4WAcCLI1dEzC/pQhBDU1tTgrFwa/TA1TePEiRNcvHCJffv28eyzz1C5RK692FwdPUM4EcxYpSgljAUkukFnQ1XuTbIhHwX41gnGAoDBJU+IaCxjCrYgpoW5PPKLBTWDACwWCw0N9Ys6XVgIga7rnDp1ilOnTrFnzy5e/OIXqK6uLvalLQmSRoLLI78gpoXJZIjHkuAJEcXg0liAnPY/5G8CAZwKxwmOB7PXCQ/4Oxn0dy34A0pHhUyL1BQyDIMLFy7w0YcfsWvnLl55+SWam/PazLwEMOjvYsCf2aoRAsaDEI4TBE7le96ZKMC1aILhsYDM6AiYhImRUD/d3qsL/oCEENTW1lDpXHzmhJSS9uvtHP/oOK2trbz44gulzg4zpNt7lZFQf+ZdYVLmTzTBMHAt3/PmrwAmQhj8wh2AcCLTLCBI6nGujJwmkgwt+EOyWiwqKrSITKG08L/11lvUu1y88vJLtKxcWezLWlJEkiGujJwmqceZzvxR/ie4lan+C0zkLXwzmQESCN4ZD6H7s6yvmYSZyyMnsy5XzyeVDgcu1+JYIJNScu3aNd5++23a2tbxK7/yS2zcuKHYl7XkGA72cnnkJCaRuSGAPwzjIXQE7wB5O6EzCzwLLsYSDA37VNu5TITiPs4OHMOQxenWVltTU/TISnrkf/edd2lrW8dXv/Ll0irvLDCkwdmBY4TivozH6AYM+yCWYAjBxZmcPz8FmPSm+xEc73VLYll0zJAGZweP4Y+NF+WhWSwWGoqYNi2l5Pq167z11lts2LCeL7/ySsnhnSX+2DhnB7MPprFkqv+P4DjQD+QVAYKZzgDgAU4EoxijgSxHCRgJ9nN24FjRHpzDbqeuCAtkabPn/fffZ8+e3bzyyss0N5cc3tlyduAYI8H+rKnIw14IRDGAE5Bf/D9N/gowqVEfawbd3aMyoxkkUmsCpwc+wJujdG2+EEJQt4CmkBACJFy/fp2jHxxl7549vPLySzQ13nnbFxUKb9TN6YEPiGnhjMUvugE9bolu0A18DOQ9+sPMZwCAc0iuuIMq7pppgBXCxPWxs1wdzTskW3AsFguNDfXzHhUSQpBMJrl48SLHPzrOgw88wIsvfhHnEk7UWwxcHT3F9bGzGTM/07H/8SAguQKcm+l3zEwBlGaFELwVihHvH5cYWWaBuBbldP8HRBLBoj1Ee0UFdfO8QKZpGqdOneLMmTM89ugjPP30Z3A4cm7gViILkUSQ0/0fENeiGUd/w4D+cUkoRhzBW0BoJqM/zG4GAHgDyVi/BwLR7LPApZFPuD42Y8UsGPO5QJZuXX78+HG6u7p57rnP8fjjjy3qlIylwvWxc1wa+STr6B+IQr8HkIwBb8zme2arAL0I3vCGYNAL2UoAIokQb3e8Sijuz//sBcZqsdBQ4AWydAuTD499SMDv55WXX+Lee+4pNa8tAKG4n7c7XiWSyLyeJaWSPW8IELwBzGrhabYKEAd+LCHSPiSJxLMcKaDdfYHTAx8s1PObFkeBF8h8Ph8ffPABDrudr37ly2zdumVRLL4tB04PfEC7+0LWyE8kDu1DEgkR4McomZwxM1eASRvrJPC+PwzdY5mnAIEgmgxzrPsnjIb6F+whTkch0qaFEAwODnL0g6Osb2vjxRdfYEWpfUnBGA31c6z7J0STYbJtv9g9JkllJLyPksUZRX/SzKUEaQj4qW4Q7xxRhciZBkCTMHHDfYmTfe9iyPndTCMbc10gk1Jy48YNLl+6zMGDB3jm6c+WIj0FxJA6J/ve5Yb7UsakNyGUrHWOgG4QB36KksVZMTsFmNS0nyG46glDrzu7L2BIjfc6X6fHe31hnmYG7HY7dbUzWyBLhzkvnL/A0OAgn37qSQ4fOlhydgtMj/c673W+jiGzbHwhlax5VFnAVeBnwKxGf5jbDADQAfzQ0El2jcqsESEQjIYGeLv9VeJazn5F84YQgrra/BfI0s7uubPnMJsEz3/+OTZv3lSq3y0wcS3K2+2vMhoaIJPxn478dI1KDJ0k8EOUDM6a2YcsTvbD/lUAAwg+G41TV2YVNFVnUQIJ45FhXI5mVtdsXIjnOi0mkwmr1UIwFMq5r/DY2BgdHR2sb1vHo48+Uqrgmic+7n2Lt65/n6SeyDg7SwlXBqBrBBB0Af8X4Jnt6A9znwEA+oAfSaBzRDKeJRNbCEE4EeDdjr8tWrp0GofdnrGYXgiBYRj09fUxOjLCoYMHuP/+w9hstqJe83JlONjLux1/SzgRyCj8AhgPKRlLWdo/QsnenCiEAiSAvwYuBqNwbVBmbKAFanGs03OJN699b8FaKU5/HYK62locdvttP49EInR3d2MrK+fIE0+waePGUohznohpEd689j06PZcyLnoBJHQlW0FlPV9Eydyci8/ntmozaQaNAC7gcCSOyVkhqHVk+6BkMNCNy65MoWIJl9lsxmq1EAqFJ0yhQCDAuNvNujVruPvu/djtpZSG+UJKg+M9b/LGlb9MbXiXWQ56xuByP+gGGvDfgR8As3Z+08x92fJmJXhY02mMa7CiVmTZoVugGRojoX5W12zE5SherrzVakU3DIKhEB6PB6vFwp7du1i7dm3J0Z1n2t3nefXCt/BF3ZlNn1TY81SnJBABBJeBfwe45yr8UAgFgLQCeFDrXo+G45jNpuwOsRCCYMxDIO5lY/1u7GXFqeASQlBeXkY4HKaupprNmzdRVVVVMnnmGXd4mFfPf5MO9/mspo8h4VI/dKm2swng/yUd+jw594XVwijA5CzQD3wKWBeMQm2loDqLBSGEwB0eQiLZ1LAHi6k4LcHNZjMuVx21tbWltuQLQFyL8ZMrf87HvW9l3cJRCBj0wLluSaov7QcoBQgWYvSHwjjBUxkC/gAYjSTgUp9yWrINprrUONb1Ez7sfqOoq8QlFgZD6nzY/QbHun6SeZM7lMz4I3ChVxJRru4oSrZmveo7HYVLXZy6LgCtwL5gTHk1TdWCzOa0IKHH6PN14HI0s7J6bdYckBJLF4nkVP97vHbhTwjEPFlNH81Qwt81mvoo/BXwJ0CiUKM/FFIBIK0ECZQSHAYaAxFw2AR1WUx8IQTRZIihYA+razbhspdqaJcjHe6L/M25P2Yk2JdV+EHl+lzsneg+chn4v4HeQgo/FFoBYGpEKA48qutYA1FwOQWVWTZ0EULgj40zEuxjnWsbVbbagl9aieLR7+/ku2d+n07P5czd3VCB0LEAnLohCccAQQQV9SmY4zuVwivApCl0A2hFsDeWhFgCGquzhUaVEoyHhxkO9tJWvx1neU3BL6/EwjMU7OE7p77O1dEz2YVfQDAV8hwNkF4W+Cvg60C80KM/zIcCQFoB4qio0H1AUziuQlrZ/YGUEkSGCcQ8bKjfSYXVkddXlliceKNj/ODcf+f88PGcx2o6nOuZsPsBLqBMn26g4KM/zJcCTM4Cw0AQeEhKKnxhsJihvkqQK8w+HFR7jq2v31FSgiWKNzrG35z9I072v5tz6yxDwpUByZX+CbvfC/wb4O+BOa/4ZmL+ClgnleAq0Ah8yjAwecLgKBfUOLKHRw0MBgM9+KJjtLm2UWFdfF2fS2TGExnh++f+mE/63sGQetbInpTQPQpnuiGhMiJ04H8Avw/o8yX8MJ8KAGkF0FFe/HYEGzQNvGGosQucFZk/qh6YZDjYy1Cwl9bajVSVlxzjpcBAoIvvnvkDzg5+mFP4AYa8cLJzwukFeAv416S7vM2D6ZNmfhVgchbwo5ziuxE0xTXVzChXZAhUwtRQoIdBfzerazdRVb44Oj+XuB0pJX3+Dr5z6utcGD4BZDd70hGfE9cl/ihp4T8H/A5wCZg30yfN/PfwuDlNwgscBJyxJAQiUOcU2HOU6CrHeIge7zUaKlfS4FhZUoJFhpSSy6Mn+e6Z36dj/HzOOH86v/8XHTfVkAwC/xI1A8y78MNCKABMVYJrqPSLA4A1HFfL3S6nwJ5z03eBNzJKt+cq9jIHK6vWZQ2plVg4dEPjRO/f88Pz36LXdz238AtV0/tJh2Rksl1UBPjPwJ8DxkIIPyyUAkBaAQxUaMsJ7APMoZja2duV50wQiHvpcF9AIllVsx6rubS9aDGJJEO81f59Xr/0Z4yFh3IOSgIl/Ceu3yT8CeCbwO8BquRlHu3+qSycApzsT88EceA8sALYCYj0TFBXmZ8SxPUoN9wX8UZHaXauLi2YFYnBQDc/uvinvNvxGpFkKC/hHw/dNvLrqMWu/wCM860TCyb8sJAKkEYpQQilBFuA9YAIxSbNoYpcSoBAlzo93uv0+TqornDRULky59RbojAYUufC8Md8/9x/48zA0bwiPaCif590SIZ9Ez+SKHv/d4D+hTJ7plKcRpbKHPKitrPcALSRUgJPCJy27CHSNOnUiatjp9GlTrNzNeWWPD5YYtYEYl5+3vEqr134Uwb8nXkHI4a8SvhHJ0d+A7XI9c9JtzZZwJE/TXEUYNIpdgNnUEqwHhCRuJom7eWCqgpyrhgLIYhqYa6PnWM8MkytvYGaivpSlKjAGNKg03OJ1y99m3c6XiOSDOb1jKWEvnEV5/dMRnskSvh/G7gCLEjEZzqK18p40icYA04Dm4F1gIglYdQP5VZVUWbOYdkIBFIaDPhvcHX0DEk9QZOzlXKLrVRbMEckkkDMyzsdP+S1C3/K9bGzSClzPlchVEpD5yicvDHRzQHUyP828DXgykLb/LdS/F7ekzPBSdQssB4QSQ2G/eoh1jkE1rwqFQXBuJ9293l6fNepsDpwOZowF6nUcqmT0GOcHzrODy98iw+73sCfo4gljRAq+/dCn+Rcj3qf0hcDeJO08ENRhR9YRMPjr98LsA34r8AT6Wszm2B1Pdy1Lj+/II1E4iyrYX/rQzzQ9jSrazeV1g3yxJAGvd7rvN/5d5zse5dgwjejmTQYhdNdkl73TdvpSpTw/zZwuVgmz60UfwZIM2kOnUKVVLYBZinVOsF4UFWWVdrAlMffQm3RFKPbc5WrY2fwRd3UOZqwl1WWFCEDutQZCfXz5tW/5v9c+d9cHPqYhB7Ly9YXKHt/yAeftEsGPDc1S04AP0FFe64sFuGHxaQAMNUcOgo4UOsEFoBwjInwWZVDYM1DhoUQqrlt3McNzyWujp5CNzQqy6twlDlLjnIKKQ1GQn0c736TV89/kzODRwnGvAhhylv4Y5rq3Ha6Uw1YUyaMGPBt4F8BXUDRzZ5br33xocyheuAfA78F1KR/ZTHDmnrY3ipwVeZKt7oZicQsLKyu3cjuFQfY3/pgKqVicY0DC4UhdQYDXZzse49zQx/R621Hl9qMzJ304talPkmPm1vbYvpQnRz+mAI1sio0i1MBIK0ENuArqNFjTfp6BVBbCdtXCVY3QJkl+94Et2JIHbPJQmPlKrY17efg2idpdq7Gbq1c9otpUhpEkiGGg7182P1TLo+cZDTUj25oMxoIhICEBr1jcKlf4g3dNBhJoAf4T8BfALHFKPywmBUA0koggAeBf49KopuQUIsJ1jUKNq2EhqqZn15KAyFMOMqq2Fi/i70th9lQv5PGylVFa9I1X2iGxmionw73Bc4MHKXdfZ5wIjDxDGbKWACuD6pe/drNHeYN4CPgd4H3ALlYhR8WuwJAWgkAtqOcqOdR/sEENXZY3wwbmlUahSqlmRlSGljMZayqXs/a2s1sb76bzY17cVidSzaMqhsagbiXdvc5Lg+fpNt7jX7/DTQ9MWOhTz/TaAI6hiU3hsF3e3PvMPAq8F9YoHz+ubL4FQCmKkEd8Aoqjrx24vcSzGZorIKtqwQraqDMOjOzKI3qTieoLKui1t7A2tqt7G05REt1G87y2pSZtDgfm5SSSDJEMO5lwN/J2YFjdHou442OEU6o7dRn4+8IoUoVh3xwpV91bNDVY5pKN/ANVGKbquRa5MIPS0UB0ihFMAMPAP8CeBi4qZLAaoFWF2xaIWioUmbSLPQAiQQpEcKE2WShrqKBtXVbWV2zkRVVa2lyrsJlby56wX40GWY8MsxIsJ+hQDe9vna6PVfwRMfQDU0VowtmtSIuUB3axgJwfUjSNw7J27sZxoF3UKnM7zPPNbyFZmkpAEydDVqArwL/BLipv7oAHDZoqVNmkatSzRDI2SkDKIVIdzaoLK/GZW+i2uaisbKFluo2XI5mmp2rqSqvxWyyYBKm1GtuESZD6hjSwJDGhEkzHOxlPDzMgL+T0dAA/tg445GRic3IhRCzTgFJRxl0XUV3OoZVTD8cm/bZDQN/hCpiGQCWxKh/2/0uSZQiWIFDqNngIWByrViqqbuiXM0I65tUe8Yyi2rBMReUMhhIKTGZzJSbbVjMZZSZy3GUOam3r0gl5anEPLu1kmqba0Ihyi0Vt5lRUkriWnRC4P2xcSLJEL6oG190DG9kDHdkiHAiSEKPo+kJ4noMw9BT6x2mOec9mVKRHU8IboyoET8aT5mSN586CryLGvWPAcmlJvhplq4CwNTZoB74MvDLKGf5Ng/PXq5mhNX1at+CsoL6tal9q1KCkhZEof7BVAPEYiqjuqIuNTOkNVFgSB1/1INmJGDiNzL1n5z8lonvUJ8rFAkNRvzQ61YjfmT6fdcNlHP7v4C/RC1aLrlRfypLWwHSTPoGO4GXUcpwe4ddCbYyqHfCmgZoqRPYyiazTWfjNM8GOZ0tNks7fTakJx/dUIlqAx5Jzxi4gzclrt3KCErov4Mqa11Stn7GZ1HsCygok4tn9wH/AJVUV3/bcamokbNCKcGqOqi2q1lCiIVThIUmfW+RVAlqv0cJfzA6bVQnjRuVxPZt4DiLeFFrVs+k2BdQcCbNIjtKAb4KPMotawdTKbeCqxJW1EJjlfIV8ku/XjokU7b9aEAy5FUObjyZ9SNh4OcoB/dNVNeGJW3uTMfyU4A0k4rQANwPvIAKm7qmvW8JwqRmAacNmqrVRn/VdqUMFpNyEiWLd4YQKd/AkCp8mdTUSD/kVUXowZga/aVBpr+8BMZRYc3vo7YkUrtzLTPBn3hmxb6AeWcynaICZRq9jNq8o43ptohKCbcwKaF32KChStDghCo7OMqVkljNk4cXSyGEmPwDJnUl3OG4ajg2FoSxgGo3qBkpoU8/idsxgE5UFu53UKZOlEWexlCQZ1jsC1hQJkOn+1Bm0RPAfpTfkJNyq/IVqiqg0gbOCoHTppSiolw9zKnRzYnMvVsUJJO+iOl+cIu/LFP/jidV4UkwBsGoJBSDQFSN+DlMm6nEUJV4b6LMnVMs4ZDmbLizFCDNZNRoJUoBvgjcg+pinX1n7MnIJVYLlJnV/21WpQjOCtXbyD5lpjCZbi7iMZtuL/aX8qbqKQwJhjE5skfiEEkoYQ/HlZAnNLWDelK7+bpyEEFtOHcUeA0l9IMsk6jOTLkzFSDNpJ9QhsotehiVZrEL1akid9u5qcOzmDRLxJT35VYmdsYRKMWw3FLWphlS2eepf8eT6pU2sdIjv5S3f2ceJFCtR86j0hXeQRWnqLniDhT8NHe2AtyKUgg7qmHXblT69d3ARpQyLJXKGR0l9O3AJ6j05HOovRoid7LA30pJAaZj0nF2oKJGrShFuBfYAdQCVSjHutjPUKIc1gCq2dhF4ARK8PtQUZ0wd4BDOxuK/cdb/EyaSSbUDFADbE29NqIq1Vaiwq2NqMa/81VWZqC2nBpFhScHUZVX7ag2I1dQZYh66tg72rzJh5ICzJZJxXCgslHrU686VOPfepRiVKZ+X4aKNjWjUrinSYYgjsqwjKFMmGEghBJ0N2qXdE/qvTv1+zBQEvRZUlKAQjNpPllRAm9GCbwp9Uq/nw4DpQTGlPc6SiGSlMyYEiVKlChRokSJEiVKlChRosRs+f8BXRU2DsMsRGkAAAAldEVYdGRhdGU6Y3JlYXRlADIwMTQtMTAtMTFUMDA6Mjc6MjQrMDI6MDDMouQGAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDE0LTEwLTExVDAwOjI3OjI0KzAyOjAwvf9cugAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAAASUVORK5CYII=',
'icon_uri': u'http://scrapy.org/favicons/favicon-192x192.png'
}
Now that I have a fairly reliable method for fetching icon data, in terms of
getting all the data I need for the definitive bookmarks JSON, I am about ready
to put everything together into a dirty fire-and-forget script!
Joining things up
-----------------
So, here is a function that basically `does the bookmark fetching`_ **and**
icon fetching. Simple and not too resilient but should just about get the job
done. It makes use of the :py:`get_icon()` `method I defined earlier`_.
.. code-block:: python
def get_entry(bookmark):
"""Parses an individual del.icio.us entry"""
entry = {}
# Locate and set Title
title = bookmark.find_all('div', class_='articleTitlePan')[0]
entry['title'] = title.a.attrs['title']
# Locate and set bookmark URL
href = bookmark.find_all('div', class_='articleInfoPan')[0].find_all('p')[0]
entry['url'] = href.a.attrs['href']
logging.debug('Getting URL: %s', entry['url'])
# Parse the URL - we need this to get the Icon
parsed = urlparse(entry['url'])
# Get the icon URL and base64 encoded icon image
icon = get_icon(u'{}://{}'.format(parsed.scheme, parsed.netloc))
entry['icon_uri'] = icon['icon_uri']
entry['icon_data'] = icon['icon_data']
# Get the date the bookmark was added, both in readable and epoch format
entry['add_date'] = str(datetime.datetime.fromtimestamp(int(bookmark.attrs['date'])))
entry['add_epoch'] = bookmark.attrs['date']
# Just set the bookmark to public
entry['private'] = 0
# Get all the tags (if there are any)
try:
tags = bookmark.find_all('ul', class_='tagName')[0].find_all('li')
entry['tags'] = [tag.a.text for tag in tags]
except IndexError:
entry['tags'] = ['none']
# Get the comment (if there is one, else use the title)
comment = None
for brief_text in [l for l in bookmark.find('div', class_='thumbTBriefTxt').children]:
try:
comment = brief_text.p.contents[0]
except AttributeError:
continue
if comment is not None:
entry['comment'] = comment
else:
entry['comment'] = entry['title']
# We are done
return entry
The above code is fairly well commented so there is not much to be said about
it, so without any further ado, here is some test output:
.. code-block:: python
link = json.dumps(get_entry(bookmarks[2]))
print link
.. parsed-literal::
DEBUG:Getting URL: http://www.martyncurrey.com/arduino-nano-as-an-isp-programmer/
DEBUG:Icon URL: http://www.martyncurrey.com/wp-content/uploads/2016/10/favicon2.ico
.. parsed-literal::
{"comment": "Arduino Nano as an ISP Programmer | Martyn Currey", "add_epoch": "1430895155", "title": "Arduino Nano as an ISP Programmer | Martyn Currey", "url": "http://www.martyncurrey.com/arduino-nano-as-an-isp-programmer/", "tags": ["arduino", "programming"], "icon_uri": "http://www.martyncurrey.com/wp-content/uploads/2016/10/favicon2.ico", "private": 0, "add_date": "2015-05-06 06:52:35", "icon_data": "data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAJwOAACcDgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAO5YAECFMQB/AAAAAAAAAAAAAAAA8FgAQIozAH9kJQAgAAAAAAAAAACuQAB/YCMAv0AYAJ9AGAAgAAAAAAAAAAD/dyd/jDQA/wAAAAAAAAAAAAAAAP91JH+fOwD/ViAAQAAAAAD/YgZ/6VYA/7xFAP96LQD/QBgA7wAAAAAAAAAA/5ZYf65AAP8AAAAAAAAAAAAAAAD/hT1/xkkA/3QrAED/XwEQ/4Q87/91Jf//dSXf/3Ym36E7AP8AAAAAAAAAAP+eZn/ISgD/AAAAAJw6AHDnVQAw/3Mif89MAP+fOwBA/3QjYP+DO//eUgC/AAAAAP+UVhD/YgbPAAAAAAAAAAD/nmV/5FQA/wAAAACzQgD/w0gAf/dbAH/LSwD/xUkAQP93J3/vWAD/gzAAUAAAAAAAAAAA/3cnMAAAAAAAAAAA/55mf/9hBP/XTwBA4FMA/8JIAL/YUAB/xUkA/+NUAED/cyK/pz4A/1EeABAAAAAAAAAAAAAAAAAAAAAAAAAAAP+han//ciD/9lsAcP9hBP/aUADvykoAf7hEAP/yWQBA/3Miv4UxAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/o25//34z//lcAJ//bhnf/2cO/9dPAJ+jPAD/7lgAQP92Jr+ALwD/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/6Zyf/+FPf/fUgDP/20Yf/+NSu//XwLPkzYA/+FTAED/fjK/kTUA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP+pdn//ikX/tkMA//9gA0D/p3Ov/4E3/481AP/OTABA/4lEj79GAP9AGABAAAAAAAAAAACzQgAQAAAAAAAAAAD/rHt//4tI/5Y3AP/rVwAQ/7OGf/+bYP+SNgD/sUEAQP+WWGD/Zw//QBgAnwAAAAAAAAAAYCMArwAAAAAAAAAA/6+Af/+KRf+GMQDPAAAAAP+4jkD/qnn/oTsA/5g4AED/nmYg/6Fq/7lEAP9gIwCPbSgAr0AYAP8AAAAAAAAAAP+4jn//j03/njoAnwAAAAAAAAAA/7iP/9dPAP+lPQBAAAAAAP+/mp//omz//2wW/9FNAP92KwD/AAAAAAAAAAD/uI5g/5JTv9pQAGAAAAAAAAAAAP+3jZ//bBa/108AMAAAAAD/vZYQ/7uUn/+gaP//bhrfyEoAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP/zAADe4QAA3sEAAN7NAADW3wAA0p8AANKfAADAnwAAyJ8AAMifAADMzQA+zMEBEszBDQjc4/kE///yBA=="}
And there it is. I ran the code against my own bookmarks in del.icio.us and,
after a few false starts where their site was unreachable (no reason given by
the guys at del.icio.us), I managed to safely retrieve all my data.
And now for a couple of other things which are connected with what I have been
doing whilst working on this bookmark business.
Aside - Outputting the Results
------------------------------
At some point I wanted to incrementally write results to a file in JSON format.
Occasionally a data fetch would fail and the entire process would end resulting
in a somewhat truncated file. So, I decided to make a three part approach:
1. Write an opening list delimiter.
2. In a loop, write out fetched JSON dictionary objects, skipping failures.
3. Write a closing list delimiter
Here's what I came up with; rough and ready, but worked quite happily.
.. code-block:: python
filename = '/testing/links.json'
# Write starting character
with open(filename, 'w') as start:
start.write('[')
# Here you might have something that fetched a page at a time and created
# a list of bookmarks per page - then interate through that list next.
# For each, blah, append links - add a comma after each bookmark except for the last bookmark
# Here's hoping that the last list item is not a failure.
for idx, bookmark in enumerate(bookmarks):
with codecs.open(filename, 'a', encoding='utf-8') as incr:
incr.write(json.dumps(get_entry(bookmark)))
if idx < len(bookmarks) - 1:
incr.write(',')
# Done, we write the ending character
with open(filename, 'a') as end:
end.write(']')
Aside - Making an importable bookmarks HTML file
------------------------------------------------
Here are some brief notes regarding turning JSON entries into bookmark HTML
which I can import into a browser.
Below is a line of desired link HTML:
.. code-block:: html
{}\n'.format(
bookmark['url'], bookmark['add_epoch'], ','.join(bookmark['tags']), bookmark['icon_uri'],
bookmark['icon_data'], bookmark['comment']
))
Create a suitable header:
.. code-block:: python
header = u"""
Bookmarks
Bookmarks
"""
Now write the whole lot out to a file ready for consumption by a browser:
.. code-block:: python
with codecs.open('/testing/links.html', 'w', encoding='utf8') as wfh:
wfh.write(header)
for line in bookmarks:
wfh.write(line)
With that, we are done. Until next time.
.. footnotes
.. links
.. _`del.icio.us`: http://del.icio.us
.. _`very well documented here`: https://sixtwothree.org/posts/homesteading-a-decades-worth-of-shared-links
.. _`Jupyter Notebook`: http://jupyter.org/
.. _`BeautifulSoup library`: https://www.crummy.com/software/BeautifulSoup/