Getting Bookmark Data from del.icio.us

Table of Contents

del.icio.us Link Fetching
#

Below is an outline for building a rough & ready script to get del.icio.us bookmark data - a script very much driven by necessity.

Rationale
#

Why am I doing this? Well, I’ve been bookmarking links using del.icio.us since March 2009 and have, very slowly, built up a steady collection of over 900 bookmarks, most of which are probably still quite useful.

The ups and downs of social bookmarking service del.icio.us are very well documented here, so I won’t go into all the details, but suffice to say that reading the aforementioned article gave me all the impetus I needed to go ahead and homestead my list of bookmarks. Simple, I thought.

Then came the snag…

We’re sorry, but due to heavy load on our database we are no longer able to offer an export function. Our engineers are working on this and we will restore it as soon as possible.

Great… what now? Time to write a script of course. I’ve initially created this article using a Jupyter Notebook which may, I hope, explain the recipe feel to the writing.

The Plan
#

Well, initially the plan was just to grab the bookmarks by scraping my del.icio.us pages from beginning to end and turn that data straight into some form of bookmark HTML. Thinking about it more, I decided to opt for a JSON data format; that way I could easily turn the data to any format I like at my leisure.

What Data?
#

From a few investigations I found that for a bookmark to be successfully imported into most browsers, one ideally needs a few key pieces of data:

Title
URL
Date added (Human readable and epoch)
Private or public
Tags list
Comment (or title if no description)
Icon data (Optional, but provides a nice icon by each link)

Annoyingly, there isn’t really a standard that browsers use for importing or exporting bookmarks; it all seems a little ad-hoc, so I’ve generalised somewhat.

I would like to finish with some JSON that describes the bookmark like so:

{
   "comment":"Docker Compose - Docker Documentation",
   "add_epoch":"1431953037",
   "title":"Docker Compose - Docker Documentation",
   "url":"http://docs.docker.com/compose/",
   "tags":[
      "docker",
      "development",
      "python"
   ],
   "icon_uri":"http://docs.docker.com/favicons/favicon-32x32.png",
   "private":0,
   "add_date":"2015-05-18 12:43:57",
 "icon_data":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAFA0lEQVR4Ae3SA3wjXwLA8d/DzERNUiRre3u2bdu2bRsfnG3btm2st7x/ze2myejhdtvz5Wx8B89+/N///c8T/JUud5u7iPpXPqOimRlMrcbKpS7nAMdfSFzpmtflr+GCaEu6a/BZttrfK5O2CydOvxD4Kn8hbeoN/hqy06qY3g3Xs/XGRpHGTs9Pvpe/gnZRiT8lXJru1VlSBrAIvDPO66gpnJXgEd4B3kN35e9+OpDxqu12RLrnK+/jT1m8+m0fcXbjzjt7MOUsQVX7ECYPEmv6k7hNEK8S5RndRDOjxdUr3uxJMmn/EPi9wfTqlW7On5IXyv1Jub4D7wgLZUSxDN6RxR1MnuGEwOrA04WtNS7cufQNHiDSdqvni++YAL7Fb9CyfYauhNxiwsI+4ZxXebZJeocHpMnwogfhHdIaCAtredKaw3h/ZWlzp/L0x8AqgCtVt9rqQMXWB+vR6JF79X3tvVOmOiAqc2PzQEcPDH2fbqYvfLXrL+44/CIbhBSzJKwVSiAELmnT6rQQ1lBWiqhYRgghl3YefmweFh9VaC0tNIZ/fCPgpwAiT86KLM18VNYrOy5868nmrqu5sCDr00PHGqe+81S9tOtidOOkUjaMyudCbKFEEITgPecn5KxBeI8PIhRiLTRRKbI6wISFEiBgXTRy5FQ0+qPhVnPHhWd2XaxqvK9GcYvK0tSMyuIVfe7HL3kpg6TauJEt9myUcKUgz7EB6CxFxm0QAp0laBUgnUOlMSiNzBLC83U8hCYvmFrztj6IruiKlYH4ote4bGvrwZ2r85O+f3p4Wptktjo7+o3+kR+9zOlwSFzpejfGBpEEUWwMfUccucGDPrVykatfcaBYYp0A79cDBMI7cBZUAHh+SZgMr0NAAB4Q+LBArEPmckPzB5/+ycHPvu4uKksmzuUnQBtA2yCqAJcAf8fFnRf9vsrTiuqcJZg6icxTsu2DuEIZrEV2WoQTx5Gry6T7Lo2tDiCcRc+OEo4fIdt2CNPYjg9L65d0cZLi6E9pFHtoDH//2+2+zWWv9L2AHuBNwLi4/I1v1Q/sAMrKZHa5sftxkzsveWOKEUiJVwEoDXgwBpxBAEiF1wF4jzDZWgis5XmpIMkg6aDzlC0X/OjrtfmxJ9og6gAR4IFFYPT8BPhNcbn3/qcPX+vl7f7NQhYC8KzxQrBHpPRrz/fzAs57wLNO/HrnrcXFCRiDQLBh5uTEjqFv3An4Ml1opwJ+U+/C2Cf6FsaOxFHPhZ0rILRCWEtkUppByoBy6GAzuXf8mscbi8/ytQ/vkUBj7vTcttHvPNop/WXoTlz6lnfmdxnnbt+759ArN+/aXZNK0ChF9NeqvH6pyAVOYZ1jjXPrA+cGbwx4D0IS5jHN2dOnN4//8LFX+/DbPww46E48RvJ7Xiu1vvY973ffq1z72k89NHi4US6VUErxvqElXvSTGVLnEd7hnQMHCNYEJqN6drY1MHv6I82p488HfsqfIK58/ZvQzVc/+RF1lVve4eqXuuxlHn5ocPBKzQ3NalQs8tmftXjHqUVmE4P0FmVzorSdldtLk9UzU1+pL46/Y9PEsa8CMX8GcbNLXYQ/5ivLpjJ40YteYuv27Veq9/UdqteqtZNLsfrRXCsJ0vZKkCdDxc7ykcrK7E+3ThybAHL+AuLOB7fy53rriQkJqN0C7iDwOx0OcPDXE6/jX0vy/wn8fwL/Yj8H6qJUvBfQ9+kAAAAASUVORK5CYII="
}

Getting the data
#

I’ll now go ahead and start getting the pieces of data I need. Firstly I am importing all the libraries I’ll need for the entire exercise, as well as a bit of logging:

import datetime
import urllib
import ssl
import json
import base64
import codecs
import logging
from urlparse import urlparse
from bs4 import BeautifulSoup
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

So I am starting this by fetching one links page from del.icio.us, breaking it down and getting each required data item piece by piece.

To be clear, before I started the next bit, I used a browser dev tool to identify the page structure. Firstly then, fetch the URL, parse the returned data with the fantastic BeautifulSoup library to get all the bookmark links outer blocks.

url = 'https://del.icio.us/chrisramsay?&page=1'
try:
    r = urllib.urlopen(url)
except IOError:
    logging.warning('Could not open %s', url)
else:
    soup = BeautifulSoup(r, 'html.parser')
    bookmarks = soup.find_all("div", class_="articleThumbBlockOuter")
    print 'We have {} bookmarks'.format(len(bookmarks))

We have 10 bookmarks

Result! I have ten, seeing as there are that many per page. From here on in I am working on a single bookmark - this whole next section would be in a loop over the list of bookmarks. I am going to use item zero of the list from this point.

Firstly I get the bookmark title:

entry = {}
bookmark_zero = bookmarks[0]
title = bookmark_zero.find_all('div', class_='articleTitlePan')[0]
entry['title'] = title.a.attrs['title']
print entry

{
    'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science'
}

Next, the bookmark URL:

href = bookmark_zero.find_all('div', class_='articleInfoPan')[0].find_all('p')[0]
entry['url'] = href.a.attrs['href']
print entry

{
    'url': u'https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/',
    'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science'
}

Get the bookmark favicon (ICON_URI):

parsed = urlparse(entry['url'])
entry['icon_uri'] = u'{}://{}/favicon.ico'.format(parsed.scheme,  parsed.netloc)

Get the bookmark save date:

entry['add_date'] = str(datetime.datetime.fromtimestamp(int(bookmark_zero.attrs['date'])))
entry['add_epoch'] = bookmark_zero.attrs['date']
print entry

{
    'add_epoch': u'1431293424',
    'url': u'https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/',
    'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science',
    'icon_uri': u'https://outsidescience.wordpress.com/favicon.ico',
    'add_date': '2015-05-10 21:30:24'
}

Get the bookmark tags (if there are any):

try:
    tags = bookmark_zero.find_all('ul', class_='tagName')[0].find_all('li')
except IndexError:
    tags = []
entry['tags'] = [f.a.text for f in tags]
print entry

{
    'add_epoch': u'1431293424',
    'tags': [u'arduino', u'circuits', u'photodiodes', u'photodiode'],
    'url': u'https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/',
    'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science',
    'icon_uri': u'https://outsidescience.wordpress.com/favicon.ico',
    'add_date': '2015-05-10 21:30:24'
}

Get any comments:

comment = None
for a in [l for l in bookmark_zero.find('div', class_='thumbTBriefTxt').children]:
    try:
        comment = a.p.contents[0]
    except AttributeError:
        continue
if comment is not None:
    entry['comment'] = comment
else:
    entry['comment'] = entry['title']
print entry

{
    'comment': u'DIY Science: Measuring Light with a Photodiode II | Outside Science',
    'add_epoch': u'1431293424',
    'tags': [u'arduino', u'circuits', u'photodiodes', u'photodiode'],
    'url': u'https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/',
    'title': u'DIY Science: Measuring Light with a Photodiode II | Outside Science',
    'icon_uri': u'https://outsidescience.wordpress.com/favicon.ico',
    'add_date': '2015-05-10 21:30:24'
}

Taking a look at the whole bookmark as a JSON string dump:

json.dumps(entry)

'{
   "comment":"DIY Science: Measuring Light with a Photodiode II | Outside Science",
   "add_epoch":"1431293424",
   "tags":[
      "arduino",
      "circuits",
      "photodiodes",
      "photodiode"
   ],
   "url":"https://outsidescience.wordpress.com/2012/11/03/diy-science-measuring-light-with-a-photodiode-ii/",
   "title":"DIY Science: Measuring Light with a Photodiode II | Outside Science",
   "icon_uri":"https://outsidescience.wordpress.com/favicon.ico",
   "add_date":"2015-05-10 21:30:24"
}'

Well, that looks about right to me. Obviously a collection of bookmarks in this JSON form (dictionary) will have to be expressed as a list of dictionaries.

Time to think about getting the icon data.

Aside - Icons
#

Take a look at existing bookmark HTML files from any browser and it will become apparent that they store icons locally in the form of a base 64 encoded string. This seems obvious once thought about; better to locally cache a copy than have to fetch a copy each time the browser is started up. As part of the bookmark fetching process I therefore decided to see if it would be possible to visit the bookmarked page itself and extract an icon.

The code is below; an explanation follows:

def get_icon(fetch):
    """Fetch an icon, base64 encode and return it."""
    icon_path = None
    icon_data = {'icon_uri': None, 'icon_data': None}
    # Fetch the root page, parse the HTML and get the LINK elements within HEAD.
    try:
        r_icon = urllib.urlopen(fetch)
    except (IOError, ssl.CertificateError):
        # Total fail, just return empty data
        return icon_data
    else:
        # Carry on & get the icon rel link
        icon_soup = BeautifulSoup(r_icon, 'html.parser')
        head = icon_soup.find('head')
        if head is None:
            return icon_data
        link = head.find('link', rel='icon')
        if link is not None:
            logging.debug('Icon URL: %s', link.attrs['href'])
            # Check for relative or absolute paths
            if ':' in link.attrs['href']:
                icon_path = link.attrs['href']
            else:
                icon_path = u'{}{}'.format(fetch, link.attrs['href'])
    # If we have icon_path, get the icon data
    if icon_path is not None:
        icon_data['icon_uri'] = icon_path
        try:
            icon_ulib = urllib.urlopen(icon_path)
        except (IOError, ssl.CertificateError):
            return icon_data
        # Get the content type to avoid things like 404 HTML
        content_type = icon_ulib.headers.getheader('Content-Type')
        if 'image' in icon_ulib.headers.getheader('Content-Type'):
            icon_data['icon_data'] = 'data:{};base64,{}'.format(
                content_type, base64.b64encode(icon_ulib.read()))
    # We are done
    return icon_data

What I show in the code block above is neither pretty nor particularly discriminating; it grabs the first link element it finds containing a property rel with the value of icon from the head of the page, then extracting the value of the href property.

Next, examining the href property it decides whether the link is relative or absolute and modifies the link appropriately. Whichever the case an icon URL is formed from which the icon itself is fetched. Headers are checked, content type decided and if we get to the end we have a base 64 encoded string complete with content type information.

Below are a couple of test run results using absolute and relative icon `href`:

Testing: Absolute links to icons
#

For example: https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico

fetch = 'https://www.gov.uk'
sixfour_img = get_icon(fetch)

DEBUG:Icon URL: https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico

print sixfour_img

{
    'icon_data': 'data:image/x-icon;base64,AAABAAIAEBAQAAEABAAoAQAAJgAAACAgAAABACAAqBAAAE4BAAAoAAAAEAAAACAAAAABAAQAAAAAAIAAAAAAAAAAAAAAABAAAAAAAAAALi0xACAfIwD///8AdHN2APHx8QBmZWgAWFdaAKurrQBKSUwAnZ2fAJCPkQDj4+QA1dXWAMfHyAC5uboAPDs/ABEREREREREREREREREREREREREREREREREC3d3d2yERETzUdEdOxRERu82e19u9ERazt3vK3DuBHeFGCqA0HtETkWEZkRgZMRMji6JKmDIxEG3enNeb1gERH4GkKhjxERERER1xERERERERERERERERERERERERERERERERERERAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgAAAAgAAAAQAAAAAEAIAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAADcAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA3AAAAPMAAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAADzAQEB7QUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wUEBP8FBAT/BQQE/wEBAe0CAgLtDAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/DAsL/wwLC/8MCwv/AgIC7QICAu0LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/CwoK/wwLC/8CAQH/AAAA/wEAAP8EAwP/BgUF/wgHB/8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/BwYG/wUEBP8EAwP/AQAA/wAAAP8DAgL/DAsL/wsKCv8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/BgYG/zY1Nf9GRUX/Ly4u/yEgIP8SERH/CQgI/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wEAAP8NDAz/FxYW/yEgIP8vLi7/TExM/zY1Nf8FBAT/CwoK/wsKCv8LCgr/AgIC7QICAu0LCgr/CwoK/wkICP8FBAT/yMjI///////5+fn/5ubm/9XV1f/Ly8v/v7+//729vf+qqqr/pqam/6empv+npqb/pqam/6qqqv+9vb3/wcHB/8/Pz//b2tr/5ubm//n5+f//////r66u/wAAAP8KCQn/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/AwIC/ygnJ//09PT////////////////////////////////////////////////////////////////////////////////////////////////////////////k5OT/FRQU/wYFBf8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8AAAD/amlp///////BwMD/cnJy/8bGxv///////////9PT0/+xsbH/x8fH//7+/v////////////j4+P+8vLz/srKy/9/f3////////////7u7u/93d3f/1tbW//////9JSEj/AAAA/wsKCv8LCgr/AgIC7QICAu0LCgr/CAcH/wUEBP/My8v/srKy/wkICP8AAAD/JCMj//v7+/+Lior/BQQE/wAAAP8AAAD/YWBg//z8/P/y8vL/RUVF/wAAAP8AAAD/EA8P/6Kiov/n5+f/Dg0N/wAAAP8ZGBj/zMvL/6enp/8AAAD/CwoK/wsKCv8CAgLtAgIC7QsKCv8AAAD/PTw8//f39/+Xlpb/aGho/1BPT/9PTk7/8fHx/xwbG/8AAAD/NDMz/2dmZv8AAAD/zc3N/6moqP8AAAD/enl5/ycmJv8AAAD/Ly4u/+jo6P9DQkL/WFdX/2RjY/+mpaX/8fDw/yUkJP8CAQH/CwoK/wICAu0CAgLtCQgI/wAAAP+srKz//f39//n5+f9YV1f/7u7u////////////x8bG/46Njf86Ojr//////6Ghof/Gxsb/wcDA/7i3t//39/f/Ly4u/5mZmf/GxcX////////////V1NT/ZmVl///////8/Pz/k5KS/wAAAP8KCQn/AgIC7QICAu0AAAD/XVxc//////+lpKT/rays/wkHB/9OTk7/2dnZ//f39//p6en/c3Nz/0dGRv//////v7+///Py8v/j4+P/xsbG//////80MzP/l5aW/+np6f/6+vr/wsLC/zw8PP8XFhb/tLOz/6ampv//////QUBA/wEBAf8CAgLtAQEB7QAAAP9ERET/p6en/21sbP8oJyf/AQEB/yQjI//v7+//5+fn/ygnJ/8AAAD/RkVF/0A/P/80NDT/9/f3/+Pj4/8dHBz/VVRU/zU0NP8AAAD/ODg4//n5+f/e3d3/ExIS/wkICP8wLy//enl5/6ampv8yMTH/AAAA/wICAu0AAADtQkFB/7Ozs/+dnZ3/IyEh/wAAAP8AAAD/c3Jy///////v7+//IB8f/wEAAP8DAgL/FhUV/9vb2////////////8XExP8KCQn/BgUF/wAAAP82NTX///////////9YV1f/AAAA/wAAAP8vLi7/q6ur/7S0tP82NTX/AQAA7QAAAO3BwcH///////////9vbm7/AAAA/wAAAP9gX1///////87Nzf8MCwv/CAcH/wcGBv8eHR3/SkpK/z8+Pv9APz//S0tL/xoZGf8HBwf/BQQE/xkYGP/i4uL//////0VERP8AAAD/AAAA/5KSkv///////////6empv8AAADtAAAA7ZiYmP///////////09OTv8BAQH/BQQE/yEgIP+8vLz/Q0JC/wEBAf8LCgr/DAsL/wgHB/8AAAD/AAAA/wAAAP8AAAD/CQgI/wwLC/8MCgr/AAAA/1taWv+xsLD/Dg0N/wcGBv8AAAD/a2tr////////////e3p6/wAAAO0BAQHtFxYW/2JhYf9SUVH/AQAA/wQDA/8MCwv/CAcH/wkICP8DAgL/CAcH/wQDA/8JCAj/AwMD/1VUVP/Kysr/xcXF/0dGRv8DAwP/CAcH/wQDA/8IBwf/BQQE/wkICP8JCAj/DAsL/wEAAP8GBAT/YF9f/11dXf8NDAz/AQEB7QICAu0AAAD/MzEx/8LCwv+1tbX/IB8f/wUFBf8IBwf/CAcH/wYFBf8NDAz/IB8f/wEAAP8LCgr/39/f////////////x8bG/wMDA/8FBAT/IyIi/wkICP8HBgb/CAcH/wkICP8EAwP/NjU1/8XFxf+3t7f/IyIi/wIBAf8CAgLtAAAA7QMCAv/BwcH///////////94d3f/AAAA/wYFBf8AAAD/FxYW/7y8vP/s7Oz/goKC/wUFBf/Z2Nj////////////ExMT/CAcH/52dnf/u7u7/paSk/woJCf8AAAD/AwIC/wAAAP+joqL///////////+enp7/AAAA/wEBAe0BAQHtAQEB/42Njf///////////1ZVVf97enr/wMDA/1dWVv9EQ0P////////////w8PD/CAcH/42MjP///////////3d3d/8fHh7/////////////////Kikp/3Z1df+8vLz/X19f/29ubv///////////3Jxcf8AAAD/AgEB7QICAu0JBwf/Dw4O/11cXP8/Pj7/NzY2////////////7u/v/y4tLf/Pz8///Pz8/6moqP8AAAD/NjU1///////8+/v/JCMj/wUEBP+/vr7/+/v7/7Gxsf80MzP////////////8+/v/HBsb/0tLS/9QTk7/CgkJ/wkICP8CAgLtAgIC7QwLC/8IBwf/AAAA/wAAAP8lJCT/6+vr///////V1dX/CwoK/xkYGP80MzP/GhkZ/5CQkP9VVFT/4uLi/9HQ0P9mZmb/kJCQ/xAPD/81NDT/DAwM/xkYGP/o6Oj//////9zb2/8TEhL/AAAA/wAAAP8JCAj/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/CwoK/wMCAv85OTn/bWxs/yopKf8GBQX/BQQE/wAAAP8MCwv/6ejo///////5+Pj/9/f3///////W1tb/AAAA/wAAAP8GBQX/BQQE/zg4OP9ubW3/Li0t/wYFBf8LCgr/CwoK/wsKCv8LCgr/AgIC7QICAu0LCgr/CwoK/wsKCv8LCgr/CwoK/wEAAP8AAAD/AwIC/wwLC/8LCgr/BwYG/xMSEv+8u7v/sbGx/+7u7v/e3t7/tbW1/7e2tv8DAgL/CgkJ/wsKCv8LCgr/AQAA/wAAAP8CAQH/DAoK/wsKCv8LCgr/CwoK/wsKCv8CAgLtAgIC7QsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8KCQn/Dw4O/w8ODv8dHBz/9vb2/9vb2/8ODQ3/FhUV/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wICAu0CAgLtCwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/AAAA/2hnZ////////////0tKSv8AAAD/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/CwoK/wsKCv8LCgr/AgIC7QICAu0KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8ICAj/JCMj/zc2Nv83Nzf/Hh0d/wgICP8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8KCQn/CgkJ/woJCf8CAgLtAAAA7QEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8AAAD/AAAA/wAAAP8AAAD/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wEBAf8BAQH/AQEB/wAAAO0AAADzAAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA/wAAAP8AAAD/AAAA8wAAANwAAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADtAAAA7QAAAO0AAADcAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=',
    'icon_uri': u'https://assets.publishing.service.gov.uk/static/favicon-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.ico'
}

Testing: Relative links to icons
#

For example: /images/favicon.png?1486118657

fetch = 'http://scrapy.org'
sixfour_img = get_icon(fetch)

DEBUG:Icon URL: /images/favicon.png?1486118657

print sixfour_img

{
    'icon_data': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMAAAADACAYAAABS3GwHAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAABmJLR0QA/wD/AP+gvaeTAAA4iklEQVR42u29aXRc15WY+50agEIVClNhIkFwAOeZFGkNHDQPlGxLlizLsiTb3Z2snjJ0nE6vl+QlnbeSvJe1ko7dU2J3J+6kB7fdtiy3OrZstayRlEjJnGcSIOYZhZrnuvee9+NUASCJGgAUUABY31ItFYFbt+692PucvffZex8oUaJEiRIlSpQoUaJEiRIl7hBEsS9gufFL37sP1HMtB8oAS+q9AMxT3k+HAcRT/5ep9xqQSL2X//vF48W+xWVFSQFmSUrQARxAM1CfetWl/r0SaASqgSaU8DtT703TnFKghHwYiAE6MAL4gVFgMPU7D+BOvYaBMEBJMWZHSQFyMEXQTSghrgG2pl4bgTUoYW9ACbyT6QW8EBhAEKUQYyil6AHagSuplw+lPAaUFCMXJQWYhilmjANwAa3A3cC9wA6gFqgCKij+M5RAFAgAXuAicAL4BOgDxlGzRMl8moZi//EWFSnBtwNbgN3AAZTgb0TZ8+ZiX2Oe6Ci/oR2lCB8B54CrQKSkCJPc0QowxbwpA9YCDwMPALuADamfLwcSQAdwHngfeAfoTv38jjaT7kgFSAm+GWW77we+CNyDsuHtxb6+eSaC8iE+Bv4GOInyJfQ7URHuKAVICb4V2Ac8CjyBUgBbsa+tSMRQCvAm8HPgFJC8kxRh2SvAFIfWDtwHvAQcBtooaLRGIiVI5MR7kxAIYcYkTJiE+qqp729GYGBgGDoAhjQwpIGUOoaUCKGOEYiJ9wXEADqBo8B3gOMox3rZO87LVgGm2PcNwP3Al1GCX1uY+5ZIKTGkEniruRxHmROb1YHd6qDcYqfaVkdleQ3O8hqc5dUIBM7yGhzlNdNcgCCWDOONjiKRBON+gnEfobgPf8xDXIsQ06JEEkHCiSBJPQ6IlJKJwtySiiiNo3yE7wMfoMKty9ZPWHYKMEXw7SgT56soc8cxl/OqkV2NwQA2i516xwqanK3UO1bgsjdR72imyuaitqIBZ3kNVnNhfOikniAY9xGMe/FERhiPjOCLuhkJ9jEc7MMdHiKmRaa9zjkQRplFf44ykSKw/BRhWSlASvhtKFPnH6AUoH6251MmiIHVXI7NUkGVrY61tZtpc21nVc16amwunLZa7FZnBrNm/jCkQSQZJBjz4ouNMxToocN9gS7PFQIxDzEtSlKPIzKaXHnjRinAt1GmUWw5KcGyUIApUZ2dwMsoc6dptuczpI5ZWHA5mmmsbGFj/S62NN7Fiqq1VFgdWE1WxAILfC6klCSNBNFkmKFAN1dHT9PuPs9oaIDx8DC61DCJOS1jjAB/ifIRLrBMokZLWgGmmDv1KKH/ZWA7M3Rup5oNdmslGxt2sbVxH+vqtrG6diM2y9KMjMa0CL3edro8l7kyeor2sfNEkqG5mEkGcAn4XyhlcMPSNouWrAJMCWkeAv4F8BAqNSFvpJQYGFRYHDRWrmTXigPsXnmQZmcrzvLalHO59JFSEox7GQ72cW7wQ84PfcRoaJCoFsaEaTb3GQXeBX4POMYSDp0uub/wlFG/BeXg/hNU9uUMUNGbalstba7t3NXyANub76a2on6uZsKix5A63qibS8OfcHrgfTrHL+GPeTHNLpI0DPwRylEegKU3GywpBZhi6z+AGvUfRuXX54WUEiEEzvJa9rYcYteKA2xr2keFtbLYt1YUoskQl0dOcX7oI84MHCMY9048oxkQR4VNfw+VZrGkfIMloQBTRv064BXga6jcnbyQUoKA2ooGdjbfy+G2z7C6ZiPllhlZTMuWuBal19fO0c4fc2H4BN7oGEhmqgjdwDeAv0LVLCyJ2WDRK8AU4d8O/A7wPDOI6Uspcdpq2Nl8Lw+sf4YNrh2YTZZi39aiRDc0OsYv8v6N17kwfIJgzDdTJQgDrwL/BeUsL3olWNQKMCWN4UHg36PSk/OK8EgkFpOVrY37eGjDs2xp2Iu9zFnsW1oSRBJBro6d4d2OH3Fl9BSakZxJxMhApV//LvAeizydYtEqwJRFra8A/wpVeZXzeiUSAayq2cBD659l36oHqLa5in07SxJ/bJxT/e/z7o0f0e/rQJJ36FSiKtX+E/AXLOLFs0UZ8kgJfz3wz4F/g4ry5BZ+Kaksr+bQuk/z/M5fZ8/Kg0s2hr8YsFnsrKvbysb6XSAEY6FBElosH7NIoEpHD6NC1Rf3PN8aOftqf7FvadoLXTTcEuL818CvkEeqshr1BWtrt3Bky5fY23KYMvOdmuE8PyT0GGcGjvKzq9+l23t14pnnQQz4M+D/YxGGShfVDLDn+VZQ5YjfQBWp5AxxSmlgL3NyaN2neemuf8bG+l1YTNZi38qyw2yy0FK1ju0r7iapJxgN9adyjXIqgQVVXroJOAu4F9NMsGgUIDX6bwO+Dnwa9eBysrJ6LZ/d/ssc2fwlqm11y2b1djEihMBR5mRLw16qKlyMhQcIxf35fNQMbEaVmZ7Z83zr2GJRgqJLyxSzZyvwX1EZnFkjPVJKTCYT2xr38/T2X2F9/Q7My3wFd7GhS50b7ov83aU/4/LoSQzDyGfwMVCZpb+NauFSdHOo6FKTMnu2An8APEYewl9usXF/22f5wu7fpLVmw4KnIpdQlW0uRzObGvaQNBIMBrrQDC2XEghgPWo2OMkiMIeKNgNMGfm3MTnyZ70eiaSqvJYjm1/i4Y3PlSI8i4SYFuGd9tf42bW/JhD35uMcSyZngstQvJmgKDPALWbP14HHyWPkb6hcyed3/hoPrP8cZea8U4BKzDMWk5V1rm1U2+ro83UQTgTzmQnaUD7BacC95/lWijEbFEUBUmbPeuAPycPmN6TBquo2Xrrra+xf9RBmU9EttxK3YBImWms20ly1ml7fdfwxT77m0EZUpZn3jlCA1Oi/AviPwOfINfIj2dywlxd2/yY7mu8pRXkWMUIImpytrKhajTs8zHhkOJc5lJ4J6oATe55vDS20EiyYNE0xe6qA/wf4R+TovGZIg61N+/jSnn/KmtrNC/pgSsyNHu81vnv2D7kyciqfIEUC+G8ouQjAwvkECx0+sQK/AfwaOYRfSsm2pv28vPdrJeFfgqyp3czLe7/Gtqb9Kh09O2UomfgNlIwsGAtiAk0Z/V8A/h1qysuIlJLNjXv50t5/yuraTQv5PEoUkGpbHa016xkK9uAOD+UyX62olPc+VO7QgjjF864AU4T/PuA/oxyfjBjSYL1rOy/d9c9YV7d13h9AifmlpqKelup19Pk68ERGcymBE7VGcBboXwglmFcFmCL8q1Gx/gPZjpdIVlW38cLuf8Tmhr3zeuMlFo7aigZcjmZ6vdfzWSdoAlahutL551sJFsIHKEOlNT+a7SCJpNrm4rmdv8qO5nsW4LJKLCQ7mu/huZ2/SrXNNdGWJQuPomRm3tvTz9sMMKWa60vAvyRL2/F0Hv/nd/4q96x5rJTasExpcrZit1bS4b6Yq67AhDKFeoEL8zkLzIsCTDF99qHqQ9dkO95qtvLE5i/x0IbSCu9yxiRMNFetRpc6neMXMaSR7fAK1CLZJ8DQfCnBfA61VahY/85cB97V8gCPbfxCKbfnDsBmsfPYxi9wV8sD+Ry+EyVDVfN1PQWfAaaYPq+gujhkjOtKKdlYv5MX9/4WLscMe1uVWLKUWWy0VLfR673GeGQkV2QobQqdnY9ZYL5mgN2o3j0Zh3RDGtTZG/ns9l9mZdXaebqMEouVlVWqkKnO3pjLFLKjZGn3fFxHQWeA1Ohfieri8CRZUi2s5jKObPkSB9c9VXJ671DqK1eiGQk63BdyKUFD6v/v73m+NVHIWaBgkjfF8X0EeI4ceUZ7Wg7x8IbnSpVcdzBmYebhDc+xp+VQrkMFSqYegZtkbc4UeuhdAfwWarfFaZHSoNnZyhObXsRRNm++TYklgqOsiic2vUizsxWZfRZoRMnWikJ+f0EUYIrj+xxwMNNxEkmZpYKHNjxLm2t7Ie+jxBKmzbWdhzY8S5mlItci2UFS1kWhZoFCzgCbgF8nx+rdtqZ9HFxbsvuXC3oyQcQ3RtA9SNA9SMgzTDIendE5TMLEwbVPsa1pX65Dy1AyVrAMyTl3iZ0y+r+A6umTAUm1rY4jm18qmT5LHF1L4BvqZqz7Ev7RXqJ+N1oytWulyYS9ugFX62Zatt6NvaYhr3M6yqo4svmliT3OsriQW1Cy9h9/6Xv3zbnv6JwKYqZMQ3tQW+bsyHb8U1u/zLM7/mGpcdUSREqDeCSIu/syXad+jn+0l1jQi5aM37JNq9oj2Vpuo7JpHRsPPMu6HXcjTLlnfM1I8qOL/5M3rvxlrkMvorbEOgtzK54pRJ/w8tTFZBR+Q+qsrd3CgTVPlIR/CRIP+xnpOEfHJz/DO3gjNdpLtQOl+XYREqhZwtt7hfd7ezBe/BobduVOcLSYrBxY8wQXh07Q7b2abbeeHagNzy+jqslmTSEUoBXlmEyLRFJmtnHvmsdpqW4rwNeVWCgMXcPdc4Ubv3iToeun0eIRyHtjboHJbEaGfZx44zs0tKyl2pV7486W6jbuXfM4g4FukkYiW+r0s8CfADfmco+z9kRvyfZsyXighDbXNg6sfXIu11ligYmFfFz54FVO/OAb9F08NiH8aaRhIIQJaeiQpeTRVl7GcPtZOs6dyPu7D6x9kjbXNnJkTa9BzQJzigjNNRSzAfg8WfJ9yizl3N/2NNW2uvzPWqJoSGkw2nWRT177Qy6/9yrRYMohnSL8JrOVlVv2s+ORL7F61/2U2Z0ZlcBiMWMWBtdOHSMeDed1DdW2Ou5ve5oyS9bMYCtK9jbM5X5npQBTNO4IWSI/UkrWu7azo/nuuVxjiQVCS8ToOvU2p/7uWwy3n0Ea+m2JalJKXK0b2ff0b7D1gef51LP/mDW774cMYW0hBGVWKz3XzhELB/O+lh3Nd7PetT1XQf0WlAzOenV4LjPAClS+z7Rqqha9bBxYc6S0Q8sSIOof58LPv8O5N/+cwFhfxuOEEDhqGrFXqb+ppcxGpWslJrOZbDZLyDdOLBLK+3qqbS4OrDlCmcWWbXGsHCWDs14dnrECTNG0/ajtSqfFkAZrazezJffiRoki4x/p4cxPv03Hxz8lGQ0hsixSSinxDHQwfOMs0cA47t6rDFz5BD2ZIJtjbBg63pGBGV3XlqZ9rK3dnCtR7gGULM5qFphtFKgc+AwZ050l5WYbe1YepMGxcpZfUWK+kYbOSOd5Lr/7fcZ6rihzJ0fnPSEEAfcAJ1//JhXOOuKRAGHPSMY4v2EYJJKaUo0ZdvVrcKxkz8qDdHmuoBkZFcyOksW/R+1ZPCNmawKtBp7K+GAl1Nmb2NNyeJanLzHfaIkYnSff4uTr35wU/nyRkrB3BHfPZYLugaxJbElNJxAMY7GWUdMwc0tlT8th6uxN5Oit9RRKJmfMbBXgKSZztG9DCMGO5ntocrbO8vQl5pNo0Mvl937AuTf/gog3Z6+eaRHChDCZs5pLAD5/iEg0RsOqNiqrZx4JbHK25tMTtoEsA3I2ZqQAUwpeHiPL/l0VVgf7Vz1YyvVfhPhHejj7xre5/tHrJG+J7ReacCTKyJgHKWHHfY9gr6qd8TnMwsz+VQ9SYc26N3o5SiYrZ+oHzGYG2I3q6z8thjTYWL+LlprSqu9iwtB1hjvO8slrf0TfxQ8xtNRuLlIiDV29cvfwzJtYPMHA0DjRaJwNu+9lz+GnMOWRDzQdLTVtbKzflcsZ3sosyibzvqIpmnUPsDbTcWaTme3Nd1NVPnNtLzE/6Mk4Xad/zqnXv4lnoEP9MCX8Nmcta/c8xLp9j+KsL0zAIhZP0Dcwij8QxNW0iodf+DVqm1pmfb6q8lq2N9+dy6JYi5LNGUWDZhoFqgPuJYPiGNKgpWqd2li5xKIgFvJy/aP/w41f/D2JaHDSZpcSS3kFmw8+w/q7jwDg6W/nF3/7x4Q8wzlt+0yEQhH6h9wEQxGq65o48pXfYsPOuS+Erndtp7GyhaFgb6ZaEhNKNusAT77nzesup2jUKlST2wxIWqs3sKp6fR5nLTHf+Ia7Of3j/8H1j/6OxC3xfYnEanNQs2IdljIbljIbrtVbKLdXkSvkMh26YeAe99PVO0wgGGbFuk08+5u/y+7DTxbEz1hdsynVKTzrtd2HktG8Z4GZzgA7yLjqJim3VLChfidW87y3dCyRBSklIzfOcfHtv8bT3w5wWxRFCEE8EmC4/Qy1LRuwWMsZvPoJEb87Y1pDJmLxBKNjXsY9fnQJm/Ye5LGXfpN12wq3CGo1l7GxfhdnB4+R1FXxzTSsQMno+XzPOxMFKAMeJkMrFYmq6tnSeFfBbrrEzNESMXrOfcDVo68RGh/KUogiMJIJbvziTcZ6rlBud+IZaCceDuQVFhVCoOs6/kCY4VEPoVCECmc19z78DIef+Qp1TasKfm9bGu/CUVaFNzqWac3ZjJLRV8mzTmAmClAJfCrbASuq1tBYWfgbL5EfsaCXqx++Ttept0hEw7mrsIRAi0cZ77uGQA1i+a4JhMNRxsb9jHv9GIakZf027n/2l9l16Aks1vmxABorV7Giag3e6Fi2wz6FktW8/ICZKMBmIGP/QoFgc8MerOZSxVcx8I/2cvnd79N/+XgqVz9Pu1tMlpzk8wldNxj3+Bl1+4hGY9idNex96LPc8/jnaV67eV43MbSarWxu2MOVkVPZDmtGyWpedZIzUYB9qB08pj+RycoG185sZWwl5gFD1xm5cZbL7/2A8d6rKSkurBCmzZ1AMMKo20sgEMJsLadt56c4/MxX2bT3AGW2+W9sbBJmNrh2YjFZ0YxkpsOcKFktjAJM8aa3o1pW34aUksbKFlyO3CVvJQpHMham68y7XPvwb4n63UzaMZJCKYGUklA4invcj9cfRNN0mlo3cNfDT7Pv4WfyKnMsJC5HE42VLQz4uzLNNhUoWeWXvndfzoL5fGeAOtR+rtM/JAxaqtfjLC1+LRjRoIdrR39E5+m3ScYiE3n6NSvaiAbc+Ia7MXRtTt+RSCQZdfvweAPEEwnszho+dehJ7jnyBVas24ypCBuWO8traaleT7//BiJza9v03sM5/YB8FaANVfyekVXVbaV+PwuAlBL/SDcXf/5dhtpPTwi5s76FfZ/9NepaNpCIhbj48+/Sfe69Gcf0hYB4XMPnDzI27icciVBRWc22PQc58JmXadvxKaxlxdvExFFWxarqNj7OflgrSmYLpgDNZOj3KZFUWB3UOwrasrHENBi6xtD1U1x+7wd4B1UzBCEE0tBxtW5S8fzyCizlFTRt2E3vhaMzmgV0XcfnD+H2BAiFIwiThY2772PfI8+w495HsTmceZ9rPql3rKDC6iCmRTN1jWgkS8BmKlkVYIr9vwKYtq5RSomzvLZk/88zWiJG1+m3uXr0R0T87pvtXyHwj/QS9o5Q1bAKPZlgvPcqhqHnPK9AreIGghHcHj9+fwgpBE2r1rP/0WfZffhJahqaKbRjPRdcjiac5bVEk5FMfoCLlALk8gPymQHKUHs1TUt6d8emylLu/3wR8Y1x5YMf0nPufZLxyG15OkKY8A118vEPvkH92m2EPcOM9VzOav4IAYaRcnA9ATy+ALqmU9+yhp33Pc6nHn+O+hVr8urottA0VbZSbXMxEsq6T8A6lHxnnQLzUYAKshUdS3CW11BVansyL4z3Xefye99nuCPdpWF6gZRS4h/txT/am/OcUkI4EsPjVYKfSGhU1TWw7e6H2P/Yc6zetGte4/lzpcpWh7O8JhXxynjYasAB+LOdKx8FsJPFAbaYLNQ7VpS6PRcQKSW6rjN07SSX3/0e/uHuVELZ7IUylf1MPJHEPe7H4/UTiyewOarYc+/93PfUF2lZv43yCsesv2OhMAkT9Y4VWEyWbB0jWlGyO2cFqCCLQ2E2WUr2/ywxDANd11MvA03XSCaTJDUN9+gIF3/+IxjrxGyee7gxHk/i9anITjQao9zuYNs9h7j3iRdo2/mpJSH4U3E5mjCbLNkWxJrJsG41lXwUwEZGBZBYTNZSBCgPDCmVcCeTJBIJEonkhLBrmoamJdE0fcrxJipW7cYz3k+lDM86pVjTdDy+IB5vgHA4itlaxqa7DrHn/qfYds/DOKpqiv1oZoWaAazZukU0o2Q3KxkVYEoEqIZMK8CAyWQuNb6agpQSwzAwDIN4IkE8HicWixONxdB1fWLUNwx9wkedzt42mQSrN24n7Bsj0nUUuzl3RCeNEAJN0wgEI4yN+wgEwpgsVprXbubuxz/PzoNP4KytX9R2fi6qbS5MJnM2N6ACJbtZI0H5zACryGJ8moWFGlt9sZ9H0TAMg2RSI5lMkEhqxOIx4rE4sXgcTVMBCCnlRL3tpNCJnIO61Wpl/a77uOgfI+65hM2cq1+sSlZTkR0/Pn8IKSVNazay5/6n2P/Is1S5Gpe04KepsdVjFlnFV5AqjslGLgUQqF26Mz4xkzDlqthfVkgpSSY1YrEY0ViUeCJBIp4gkUig6ZlH6dkInZQSR2Ula3bfz42PPFgSg5hNIsOxEImoFGWfP0hS06lpWMG+h55m16EjNK/dNOui9MVIhdWRK/AyVXYzjhv5KMBKspRO1lY0FCUnZKEwDANN04gnEkQjUcLRCIm4EnZd16cZ2QuLlJLG5hYC2x5g7MyPqZXBie9Kx/Jj8UQqshMgkUzirGlg/4FHufvxz9O8eiOWIqYuzBcmk5naigZ8MXfGQ1CyO2cFsGT7ZYXVgWkRrRLOFQnomkYsbbtHo0QiERLJ5LRtQxbCnBAC2jZtI+AZI9D9ATWWJBKIxhJ4vMrBjcbi2J3V7Dh0P/sf+Rzrtu+ft8KUxYAJQYXVkUvyLOSIHefjA+RqFpnzkMWOlBJN14lEIoTDEWXHxxMkk8m5n7xAWCxmtuy9h3PBcfwjp4kF/bg9fsKRGGXlFWzed5h7nniejbsPYHNUFvtyF4DcfUzJQzBzKYAJlVWXwQQS1FbUL0kTyJASLZkkFo8TCASJRCIkNW3CrFlsjqKUEmdlJSs27ObUlU+IjwxjLq9gzZY9HPj0l9iy/34qKqsX3XXPF8oEqiebe8qk7GZ0zvLNBcqIzWJfMg9dAolEYsKsCYXDxOOJ20ybxXo/Qggi8SQD8UpWt2zl4ac+x+7DT1JV1zj3ky8xhBDYLDmr0HLagIXYJG/Ro+s60ViMQCBAOBIlHo+j6/qiFfRMJJNJRkaGicoydj72PAc/+9ySnH0XE8tWAaSUxBMJwuEw/kCAaDR2k3mz1IRfCEEoFKK/rx+r1cq2HbtKwl8Alp0CGFISDocJhUIEgiHi8Zv3TFhqgj8Vr9fL6OgojY0NrFxZSj8pBDmX0nKtPAohFkUMSNd1QuEwPn+ASCRMMjm3etjFhmEYdHV2kUgk2LF9BxUVOfO8ljWC/AazXDKcUQE0HQwDzGYwZ1lwiyUjGAVsqz1TNE0jHIng9foIhcMYhjH3ky5CIpEInZ2dyvzZvhWr9c7uv2RISSwZyXqMboCuQ7YF8IwK4A6CPwINVVBbmSnYJPFERjEMDcwLu9qo6zqhUAif308oFEZfpoIPaqTr7OzE7XbT1tbGmtWz2g1oWWEYGp7IKJmqYiRKfscCUJ0lWJRRAWJJCMZSH85WeTMPjZiy3riURKNR3OPjhEJhtPRGD8sUIQTxeJz29nY0TWPnju00NZXqL9RCWJZfS0hoSobLs0yWWdMcTDkW2ySQ0GJIFmb0TSQSeLxePF7fRKblchb+9P319PTQ09NLY0MDe/fuKfYlLQokRkr2sozNIiXDWc6TUQHGQ5JeNzhtgsYs7X680TH0PLoPzAXDMAgEgrg940SjsYJu5bOYEUIQjUa5cuUK0UiEw4cOsm7dumJf1qJAN/RcTXIJRKDXnV1WLNk+7PNKoo0kJJkjPZqhZStLmzOJRJJxzzher4/kMjd3pmNwcJD26+3U1dby4IMPFKQ8cjmgGUk0I3OkTwLRhEz4vJIae2aZyegfRxNAO4Yh6RRktnF0I4k7PDwvNymlxOvzMeYeR1uCK7dzJRqNcvr0aeLxOPfddy+tq0qt59O4w8PoWQbeVKZ4J+0Y0Sw7BWRUgBE/0AH+SGaLQzVV0vHFsk9Fs0UIgbOykvLy8jvG7Jl67zdu3KCrs4sVzc3ce++9d3zocyq+mDK9Mw2JUirZpSMlyxnIXiJUgYxraOGMG9ALdKnhj+W9J9mMsdsrcNXVYjYvn2qmXAgh8Hg8nDl9Bk3TOHToIBs2lPZdm4o/5kGXGplc3HAc4hoaFdnXcrNLVWdSIhmMJTEyaVpSTzIc6JnXm62traXKeec03tV1nUsXL9HX18eGDet58MEH7jjzLxfDgR6S+vQmkABiSQwkg3QmZ68AUlbIeJJAJI7MNNfoUsMTnR8TKI3ZZKK+3kVZ2fKtcEojhGCgf4Bz585hs9n49FNP4nKVum7ciic6lpoBpkFAJI6MJwlIWTF7BVj7b3UiCfpDsewGuC/qxhuZXyWwV1RQ73Itq8LuW0lnfH700UcEAgEOHz7E7t17in1Ziw5vZAxf1J31mFBMykiC/rX/NnuIPrM0fesEAx6IBvGF40TJ6AgLIokA3nlyhKdSW1NNlXNxtOieD3Rd59y5c3R3d7NhwwaOHHkCm235FbTPFW9sjEgikDk4LyEcJxoN4hvwAN86kfFcWYdTzQB0YgmN4XjG2UYQjPvwhEfm/cbNZvOyNYWEEPT29HLm9BkqKir4zKefYuWKUsrzdHjCIwTjvowKENcgoTGMTkzLkaSQ254QRKMJhqOJ6dMihBAE4j6GQ30LcvP2igoa6l2YTMvHKRRC4PV6OXbsGOFwmCNHnmDfvtJ+y5kYDvURiPumDQwIodawogmGEURznSu3ApiIBKP0ReLZcyqGg73EtEjO0xWC6urqZRUVisfjHD9+nL6+Pu66ay+PP/YYFsuyq1UqCDEtwnAwcwt4AUTiEIzSh4mcAplbAaxEA1GGQrHMhQUmBP2+GwRi3gV5CJaUKVRevvTtYyklFy5c4ML5C6xZs5pnP/cMVVXL18+ZK4GYl37fjYy9qCQQikEgSi8WwrnOl88MkMCg3RdWRTLTIYSJ4UBPzuSkQmKvqMBVV7ek4+NCCLq6ujhx/AQ1NTW88IUvlJLdcuCNjjEc6Mm4UYimgy8MGHRhJmdZYHYFSHvPgiF/RI4ns0SUEnqcHu+1BU1ZWOpRoeHhYT54/wOSySRPP/1Z9uzZXexLWtRIKenxXiOhZ0xNIKmDPyLHEagEtSwRIMhnBgAQDI+HGI0nM/sBEsnV0dMkjURepywEZrOZetfSiwoJIQgGgxw9epSxsTGefPIIDz5w/7Je4ygESSPB1dHTGXeFEUA8CeMhRicUIAf5PvHOhEbfqD+zHyClpNtzFU9k/sOhU0nnCi0VUyid43/s2DG6Ort45OGHefLIE0tOiYuBJzJCt+dqRitDAqN+SGj0AZ35nDMvBWiowqMbdI4FZMaNB4WAcCLI1dEzC/pQhBDU1tTgrFwa/TA1TePEiRNcvHCJffv28eyzz1C5RK692FwdPUM4EcxYpSgljAUkukFnQ1XuTbIhHwX41gnGAoDBJU+IaCxjCrYgpoW5PPKLBTWDACwWCw0N9Ys6XVgIga7rnDp1ilOnTrFnzy5e/OIXqK6uLvalLQmSRoLLI78gpoXJZIjHkuAJEcXg0liAnPY/5G8CAZwKxwmOB7PXCQ/4Oxn0dy34A0pHhUyL1BQyDIMLFy7w0YcfsWvnLl55+SWam/PazLwEMOjvYsCf2aoRAsaDEI4TBE7le96ZKMC1aILhsYDM6AiYhImRUD/d3qsL/oCEENTW1lDpXHzmhJSS9uvtHP/oOK2trbz44gulzg4zpNt7lZFQf+ZdYVLmTzTBMHAt3/PmrwAmQhj8wh2AcCLTLCBI6nGujJwmkgwt+EOyWiwqKrSITKG08L/11lvUu1y88vJLtKxcWezLWlJEkiGujJwmqceZzvxR/ie4lan+C0zkLXwzmQESCN4ZD6H7s6yvmYSZyyMnsy5XzyeVDgcu1+JYIJNScu3aNd5++23a2tbxK7/yS2zcuKHYl7XkGA72cnnkJCaRuSGAPwzjIXQE7wB5O6EzCzwLLsYSDA37VNu5TITiPs4OHMOQxenWVltTU/TISnrkf/edd2lrW8dXv/Ll0irvLDCkwdmBY4TivozH6AYM+yCWYAjBxZmcPz8FmPSm+xEc73VLYll0zJAGZweP4Y+NF+WhWSwWGoqYNi2l5Pq167z11lts2LCeL7/ySsnhnSX+2DhnB7MPprFkqv+P4DjQD+QVAYKZzgDgAU4EoxijgSxHCRgJ9nN24FjRHpzDbqeuCAtkabPn/fffZ8+e3bzyyss0N5cc3tlyduAYI8H+rKnIw14IRDGAE5Bf/D9N/gowqVEfawbd3aMyoxkkUmsCpwc+wJujdG2+EEJQt4CmkBACJFy/fp2jHxxl7549vPLySzQ13nnbFxUKb9TN6YEPiGnhjMUvugE9bolu0A18DOQ9+sPMZwCAc0iuuIMq7pppgBXCxPWxs1wdzTskW3AsFguNDfXzHhUSQpBMJrl48SLHPzrOgw88wIsvfhHnEk7UWwxcHT3F9bGzGTM/07H/8SAguQKcm+l3zEwBlGaFELwVihHvH5cYWWaBuBbldP8HRBLBoj1Ee0UFdfO8QKZpGqdOneLMmTM89ugjPP30Z3A4cm7gViILkUSQ0/0fENeiGUd/w4D+cUkoRhzBW0BoJqM/zG4GAHgDyVi/BwLR7LPApZFPuD42Y8UsGPO5QJZuXX78+HG6u7p57rnP8fjjjy3qlIylwvWxc1wa+STr6B+IQr8HkIwBb8zme2arAL0I3vCGYNAL2UoAIokQb3e8Sijuz//sBcZqsdBQ4AWydAuTD499SMDv55WXX+Lee+4pNa8tAKG4n7c7XiWSyLyeJaWSPW8IELwBzGrhabYKEAd+LCHSPiSJxLMcKaDdfYHTAx8s1PObFkeBF8h8Ph8ffPABDrudr37ly2zdumVRLL4tB04PfEC7+0LWyE8kDu1DEgkR4McomZwxM1eASRvrJPC+PwzdY5mnAIEgmgxzrPsnjIb6F+whTkch0qaFEAwODnL0g6Osb2vjxRdfYEWpfUnBGA31c6z7J0STYbJtv9g9JkllJLyPksUZRX/SzKUEaQj4qW4Q7xxRhciZBkCTMHHDfYmTfe9iyPndTCMbc10gk1Jy48YNLl+6zMGDB3jm6c+WIj0FxJA6J/ve5Yb7UsakNyGUrHWOgG4QB36KksVZMTsFmNS0nyG46glDrzu7L2BIjfc6X6fHe31hnmYG7HY7dbUzWyBLhzkvnL/A0OAgn37qSQ4fOlhydgtMj/c673W+jiGzbHwhlax5VFnAVeBnwKxGf5jbDADQAfzQ0El2jcqsESEQjIYGeLv9VeJazn5F84YQgrra/BfI0s7uubPnMJsEz3/+OTZv3lSq3y0wcS3K2+2vMhoaIJPxn478dI1KDJ0k8EOUDM6a2YcsTvbD/lUAAwg+G41TV2YVNFVnUQIJ45FhXI5mVtdsXIjnOi0mkwmr1UIwFMq5r/DY2BgdHR2sb1vHo48+Uqrgmic+7n2Lt65/n6SeyDg7SwlXBqBrBBB0Af8X4Jnt6A9znwEA+oAfSaBzRDKeJRNbCEE4EeDdjr8tWrp0GofdnrGYXgiBYRj09fUxOjLCoYMHuP/+w9hstqJe83JlONjLux1/SzgRyCj8AhgPKRlLWdo/QsnenCiEAiSAvwYuBqNwbVBmbKAFanGs03OJN699b8FaKU5/HYK62locdvttP49EInR3d2MrK+fIE0+waePGUohznohpEd689j06PZcyLnoBJHQlW0FlPV9Eydyci8/ntmozaQaNAC7gcCSOyVkhqHVk+6BkMNCNy65MoWIJl9lsxmq1EAqFJ0yhQCDAuNvNujVruPvu/djtpZSG+UJKg+M9b/LGlb9MbXiXWQ56xuByP+gGGvDfgR8As3Z+08x92fJmJXhY02mMa7CiVmTZoVugGRojoX5W12zE5SherrzVakU3DIKhEB6PB6vFwp7du1i7dm3J0Z1n2t3nefXCt/BF3ZlNn1TY81SnJBABBJeBfwe45yr8UAgFgLQCeFDrXo+G45jNpuwOsRCCYMxDIO5lY/1u7GXFqeASQlBeXkY4HKaupprNmzdRVVVVMnnmGXd4mFfPf5MO9/mspo8h4VI/dKm2swng/yUd+jw594XVwijA5CzQD3wKWBeMQm2loDqLBSGEwB0eQiLZ1LAHi6k4LcHNZjMuVx21tbWltuQLQFyL8ZMrf87HvW9l3cJRCBj0wLluSaov7QcoBQgWYvSHwjjBUxkC/gAYjSTgUp9yWrINprrUONb1Ez7sfqOoq8QlFgZD6nzY/QbHun6SeZM7lMz4I3ChVxJRru4oSrZmveo7HYVLXZy6LgCtwL5gTHk1TdWCzOa0IKHH6PN14HI0s7J6bdYckBJLF4nkVP97vHbhTwjEPFlNH81Qwt81mvoo/BXwJ0CiUKM/FFIBIK0ECZQSHAYaAxFw2AR1WUx8IQTRZIihYA+razbhspdqaJcjHe6L/M25P2Yk2JdV+EHl+lzsneg+chn4v4HeQgo/FFoBYGpEKA48qutYA1FwOQWVWTZ0EULgj40zEuxjnWsbVbbagl9aieLR7+/ku2d+n07P5czd3VCB0LEAnLohCccAQQQV9SmY4zuVwivApCl0A2hFsDeWhFgCGquzhUaVEoyHhxkO9tJWvx1neU3BL6/EwjMU7OE7p77O1dEz2YVfQDAV8hwNkF4W+Cvg60C80KM/zIcCQFoB4qio0H1AUziuQlrZ/YGUEkSGCcQ8bKjfSYXVkddXlliceKNj/ODcf+f88PGcx2o6nOuZsPsBLqBMn26g4KM/zJcCTM4Cw0AQeEhKKnxhsJihvkqQK8w+HFR7jq2v31FSgiWKNzrG35z9I072v5tz6yxDwpUByZX+CbvfC/wb4O+BOa/4ZmL+ClgnleAq0Ah8yjAwecLgKBfUOLKHRw0MBgM9+KJjtLm2UWFdfF2fS2TGExnh++f+mE/63sGQetbInpTQPQpnuiGhMiJ04H8Avw/o8yX8MJ8KAGkF0FFe/HYEGzQNvGGosQucFZk/qh6YZDjYy1Cwl9bajVSVlxzjpcBAoIvvnvkDzg5+mFP4AYa8cLJzwukFeAv416S7vM2D6ZNmfhVgchbwo5ziuxE0xTXVzChXZAhUwtRQoIdBfzerazdRVb44Oj+XuB0pJX3+Dr5z6utcGD4BZDd70hGfE9cl/ihp4T8H/A5wCZg30yfN/PfwuDlNwgscBJyxJAQiUOcU2HOU6CrHeIge7zUaKlfS4FhZUoJFhpSSy6Mn+e6Z36dj/HzOOH86v/8XHTfVkAwC/xI1A8y78MNCKABMVYJrqPSLA4A1HFfL3S6nwJ5z03eBNzJKt+cq9jIHK6vWZQ2plVg4dEPjRO/f88Pz36LXdz238AtV0/tJh2Rksl1UBPjPwJ8DxkIIPyyUAkBaAQxUaMsJ7APMoZja2duV50wQiHvpcF9AIllVsx6rubS9aDGJJEO81f59Xr/0Z4yFh3IOSgIl/Ceu3yT8CeCbwO8BquRlHu3+qSycApzsT88EceA8sALYCYj0TFBXmZ8SxPUoN9wX8UZHaXauLi2YFYnBQDc/uvinvNvxGpFkKC/hHw/dNvLrqMWu/wCM860TCyb8sJAKkEYpQQilBFuA9YAIxSbNoYpcSoBAlzo93uv0+TqornDRULky59RbojAYUufC8Md8/9x/48zA0bwiPaCif590SIZ9Ez+SKHv/d4D+hTJ7plKcRpbKHPKitrPcALSRUgJPCJy27CHSNOnUiatjp9GlTrNzNeWWPD5YYtYEYl5+3vEqr134Uwb8nXkHI4a8SvhHJ0d+A7XI9c9JtzZZwJE/TXEUYNIpdgNnUEqwHhCRuJom7eWCqgpyrhgLIYhqYa6PnWM8MkytvYGaivpSlKjAGNKg03OJ1y99m3c6XiOSDOb1jKWEvnEV5/dMRnskSvh/G7gCLEjEZzqK18p40icYA04Dm4F1gIglYdQP5VZVUWbOYdkIBFIaDPhvcHX0DEk9QZOzlXKLrVRbMEckkkDMyzsdP+S1C3/K9bGzSClzPlchVEpD5yicvDHRzQHUyP828DXgykLb/LdS/F7ekzPBSdQssB4QSQ2G/eoh1jkE1rwqFQXBuJ9293l6fNepsDpwOZowF6nUcqmT0GOcHzrODy98iw+73sCfo4gljRAq+/dCn+Rcj3qf0hcDeJO08ENRhR9YRMPjr98LsA34r8AT6Wszm2B1Pdy1Lj+/II1E4iyrYX/rQzzQ9jSrazeV1g3yxJAGvd7rvN/5d5zse5dgwjejmTQYhdNdkl73TdvpSpTw/zZwuVgmz60UfwZIM2kOnUKVVLYBZinVOsF4UFWWVdrAlMffQm3RFKPbc5WrY2fwRd3UOZqwl1WWFCEDutQZCfXz5tW/5v9c+d9cHPqYhB7Ly9YXKHt/yAeftEsGPDc1S04AP0FFe64sFuGHxaQAMNUcOgo4UOsEFoBwjInwWZVDYM1DhoUQqrlt3McNzyWujp5CNzQqy6twlDlLjnIKKQ1GQn0c736TV89/kzODRwnGvAhhylv4Y5rq3Ha6Uw1YUyaMGPBt4F8BXUDRzZ5br33xocyheuAfA78F1KR/ZTHDmnrY3ipwVeZKt7oZicQsLKyu3cjuFQfY3/pgKqVicY0DC4UhdQYDXZzse49zQx/R621Hl9qMzJ304talPkmPm1vbYvpQnRz+mAI1sio0i1MBIK0ENuArqNFjTfp6BVBbCdtXCVY3QJkl+94Et2JIHbPJQmPlKrY17efg2idpdq7Gbq1c9otpUhpEkiGGg7182P1TLo+cZDTUj25oMxoIhICEBr1jcKlf4g3dNBhJoAf4T8BfALHFKPywmBUA0koggAeBf49KopuQUIsJ1jUKNq2EhqqZn15KAyFMOMqq2Fi/i70th9lQv5PGylVFa9I1X2iGxmionw73Bc4MHKXdfZ5wIjDxDGbKWACuD6pe/drNHeYN4CPgd4H3ALlYhR8WuwJAWgkAtqOcqOdR/sEENXZY3wwbmlUahSqlmRlSGljMZayqXs/a2s1sb76bzY17cVidSzaMqhsagbiXdvc5Lg+fpNt7jX7/DTQ9MWOhTz/TaAI6hiU3hsF3e3PvMPAq8F9YoHz+ubL4FQCmKkEd8Aoqjrx24vcSzGZorIKtqwQraqDMOjOzKI3qTieoLKui1t7A2tqt7G05REt1G87y2pSZtDgfm5SSSDJEMO5lwN/J2YFjdHou442OEU6o7dRn4+8IoUoVh3xwpV91bNDVY5pKN/ANVGKbquRa5MIPS0UB0ihFMAMPAP8CeBi4qZLAaoFWF2xaIWioUmbSLPQAiQQpEcKE2WShrqKBtXVbWV2zkRVVa2lyrsJlby56wX40GWY8MsxIsJ+hQDe9vna6PVfwRMfQDU0VowtmtSIuUB3axgJwfUjSNw7J27sZxoF3UKnM7zPPNbyFZmkpAEydDVqArwL/BLipv7oAHDZoqVNmkatSzRDI2SkDKIVIdzaoLK/GZW+i2uaisbKFluo2XI5mmp2rqSqvxWyyYBKm1GtuESZD6hjSwJDGhEkzHOxlPDzMgL+T0dAA/tg445GRic3IhRCzTgFJRxl0XUV3OoZVTD8cm/bZDQN/hCpiGQCWxKh/2/0uSZQiWIFDqNngIWByrViqqbuiXM0I65tUe8Yyi2rBMReUMhhIKTGZzJSbbVjMZZSZy3GUOam3r0gl5anEPLu1kmqba0Ihyi0Vt5lRUkriWnRC4P2xcSLJEL6oG190DG9kDHdkiHAiSEKPo+kJ4noMw9BT6x2mOec9mVKRHU8IboyoET8aT5mSN586CryLGvWPAcmlJvhplq4CwNTZoB74MvDLKGf5Ng/PXq5mhNX1at+CsoL6tal9q1KCkhZEof7BVAPEYiqjuqIuNTOkNVFgSB1/1INmJGDiNzL1n5z8lonvUJ8rFAkNRvzQ61YjfmT6fdcNlHP7v4C/RC1aLrlRfypLWwHSTPoGO4GXUcpwe4ddCbYyqHfCmgZoqRPYyiazTWfjNM8GOZ0tNks7fTakJx/dUIlqAx5Jzxi4gzclrt3KCErov4Mqa11Stn7GZ1HsCygok4tn9wH/AJVUV3/bcamokbNCKcGqOqi2q1lCiIVThIUmfW+RVAlqv0cJfzA6bVQnjRuVxPZt4DiLeFFrVs+k2BdQcCbNIjtKAb4KPMotawdTKbeCqxJW1EJjlfIV8ku/XjokU7b9aEAy5FUObjyZ9SNh4OcoB/dNVNeGJW3uTMfyU4A0k4rQANwPvIAKm7qmvW8JwqRmAacNmqrVRn/VdqUMFpNyEiWLd4YQKd/AkCp8mdTUSD/kVUXowZga/aVBpr+8BMZRYc3vo7YkUrtzLTPBn3hmxb6AeWcynaICZRq9jNq8o43ptohKCbcwKaF32KChStDghCo7OMqVkljNk4cXSyGEmPwDJnUl3OG4ajg2FoSxgGo3qBkpoU8/idsxgE5UFu53UKZOlEWexlCQZ1jsC1hQJkOn+1Bm0RPAfpTfkJNyq/IVqiqg0gbOCoHTppSiolw9zKnRzYnMvVsUJJO+iOl+cIu/LFP/jidV4UkwBsGoJBSDQFSN+DlMm6nEUJV4b6LMnVMs4ZDmbLizFCDNZNRoJUoBvgjcg+pinX1n7MnIJVYLlJnV/21WpQjOCtXbyD5lpjCZbi7iMZtuL/aX8qbqKQwJhjE5skfiEEkoYQ/HlZAnNLWDelK7+bpyEEFtOHcUeA0l9IMsk6jOTLkzFSDNpJ9QhsotehiVZrEL1akid9u5qcOzmDRLxJT35VYmdsYRKMWw3FLWphlS2eepf8eT6pU2sdIjv5S3f2ceJFCtR86j0hXeQRWnqLniDhT8NHe2AtyKUgg7qmHXblT69d3ARpQyLJXKGR0l9O3AJ6j05HOovRoid7LA30pJAaZj0nF2oKJGrShFuBfYAdQCVSjHutjPUKIc1gCq2dhF4ARK8PtQUZ0wd4BDOxuK/cdb/EyaSSbUDFADbE29NqIq1Vaiwq2NqMa/81VWZqC2nBpFhScHUZVX7ag2I1dQZYh66tg72rzJh5ICzJZJxXCgslHrU686VOPfepRiVKZ+X4aKNjWjUrinSYYgjsqwjKFMmGEghBJ0N2qXdE/qvTv1+zBQEvRZUlKAQjNpPllRAm9GCbwp9Uq/nw4DpQTGlPc6SiGSlMyYEiVKlChRokSJEiVKlChRosRs+f8BXRU2DsMsRGkAAAAldEVYdGRhdGU6Y3JlYXRlADIwMTQtMTAtMTFUMDA6Mjc6MjQrMDI6MDDMouQGAAAAJXRFWHRkYXRlOm1vZGlmeQAyMDE0LTEwLTExVDAwOjI3OjI0KzAyOjAwvf9cugAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAAASUVORK5CYII=',
    'icon_uri': u'http://scrapy.org/favicons/favicon-192x192.png'
}

Now that I have a fairly reliable method for fetching icon data, in terms of getting all the data I need for the definitive bookmarks JSON, I am about ready to put everything together into a dirty fire-and-forget script!

Joining things up
#

So, here is a function that basically [does the bookmark fetching](#does the bookmark fetching) and icon fetching. Simple and not too resilient but should just about get the job done. It makes use of the get_icon() [method I defined earlier](#method I defined earlier).

def get_entry(bookmark):
    """Parses an individual del.icio.us entry"""
    entry = {}
    # Locate and set Title
    title = bookmark.find_all('div', class_='articleTitlePan')[0]
    entry['title'] = title.a.attrs['title']
    # Locate and set bookmark URL
    href = bookmark.find_all('div', class_='articleInfoPan')[0].find_all('p')[0]
    entry['url'] = href.a.attrs['href']
    logging.debug('Getting URL: %s', entry['url'])
    # Parse the URL - we need this to get the Icon
    parsed = urlparse(entry['url'])
    # Get the icon URL and base64 encoded icon image
    icon = get_icon(u'{}://{}'.format(parsed.scheme, parsed.netloc))
    entry['icon_uri'] = icon['icon_uri']
    entry['icon_data'] = icon['icon_data']
    # Get the date the bookmark was added, both in readable and epoch format
    entry['add_date'] = str(datetime.datetime.fromtimestamp(int(bookmark.attrs['date'])))
    entry['add_epoch'] = bookmark.attrs['date']
    # Just set the bookmark to public
    entry['private'] = 0
    # Get all the tags (if there are any)
    try:
        tags = bookmark.find_all('ul', class_='tagName')[0].find_all('li')
        entry['tags'] = [tag.a.text for tag in tags]
    except IndexError:
        entry['tags'] = ['none']
    # Get the comment (if there is one, else use the title)
    comment = None
    for brief_text in [l for l in bookmark.find('div', class_='thumbTBriefTxt').children]:
        try:
            comment = brief_text.p.contents[0]
        except AttributeError:
            continue
    if comment is not None:
        entry['comment'] = comment
    else:
        entry['comment'] = entry['title']
    # We are done
    return entry

The above code is fairly well commented so there is not much to be said about it, so without any further ado, here is some test output:

link = json.dumps(get_entry(bookmarks[2]))
print link

DEBUG:Getting URL: http://www.martyncurrey.com/arduino-nano-as-an-isp-programmer/
DEBUG:Icon URL: http://www.martyncurrey.com/wp-content/uploads/2016/10/favicon2.ico

{"comment": "Arduino Nano as an ISP Programmer | Martyn Currey", "add_epoch": "1430895155", "title": "Arduino Nano as an ISP Programmer | Martyn Currey", "url": "http://www.martyncurrey.com/arduino-nano-as-an-isp-programmer/", "tags": ["arduino", "programming"], "icon_uri": "http://www.martyncurrey.com/wp-content/uploads/2016/10/favicon2.ico", "private": 0, "add_date": "2015-05-06 06:52:35", "icon_data": "data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAJwOAACcDgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAO5YAECFMQB/AAAAAAAAAAAAAAAA8FgAQIozAH9kJQAgAAAAAAAAAACuQAB/YCMAv0AYAJ9AGAAgAAAAAAAAAAD/dyd/jDQA/wAAAAAAAAAAAAAAAP91JH+fOwD/ViAAQAAAAAD/YgZ/6VYA/7xFAP96LQD/QBgA7wAAAAAAAAAA/5ZYf65AAP8AAAAAAAAAAAAAAAD/hT1/xkkA/3QrAED/XwEQ/4Q87/91Jf//dSXf/3Ym36E7AP8AAAAAAAAAAP+eZn/ISgD/AAAAAJw6AHDnVQAw/3Mif89MAP+fOwBA/3QjYP+DO//eUgC/AAAAAP+UVhD/YgbPAAAAAAAAAAD/nmV/5FQA/wAAAACzQgD/w0gAf/dbAH/LSwD/xUkAQP93J3/vWAD/gzAAUAAAAAAAAAAA/3cnMAAAAAAAAAAA/55mf/9hBP/XTwBA4FMA/8JIAL/YUAB/xUkA/+NUAED/cyK/pz4A/1EeABAAAAAAAAAAAAAAAAAAAAAAAAAAAP+han//ciD/9lsAcP9hBP/aUADvykoAf7hEAP/yWQBA/3Miv4UxAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/o25//34z//lcAJ//bhnf/2cO/9dPAJ+jPAD/7lgAQP92Jr+ALwD/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/6Zyf/+FPf/fUgDP/20Yf/+NSu//XwLPkzYA/+FTAED/fjK/kTUA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP+pdn//ikX/tkMA//9gA0D/p3Ov/4E3/481AP/OTABA/4lEj79GAP9AGABAAAAAAAAAAACzQgAQAAAAAAAAAAD/rHt//4tI/5Y3AP/rVwAQ/7OGf/+bYP+SNgD/sUEAQP+WWGD/Zw//QBgAnwAAAAAAAAAAYCMArwAAAAAAAAAA/6+Af/+KRf+GMQDPAAAAAP+4jkD/qnn/oTsA/5g4AED/nmYg/6Fq/7lEAP9gIwCPbSgAr0AYAP8AAAAAAAAAAP+4jn//j03/njoAnwAAAAAAAAAA/7iP/9dPAP+lPQBAAAAAAP+/mp//omz//2wW/9FNAP92KwD/AAAAAAAAAAD/uI5g/5JTv9pQAGAAAAAAAAAAAP+3jZ//bBa/108AMAAAAAD/vZYQ/7uUn/+gaP//bhrfyEoAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP/zAADe4QAA3sEAAN7NAADW3wAA0p8AANKfAADAnwAAyJ8AAMifAADMzQA+zMEBEszBDQjc4/kE///yBA=="}

And there it is. I ran the code against my own bookmarks in del.icio.us and, after a few false starts where their site was unreachable (no reason given by the guys at del.icio.us), I managed to safely retrieve all my data.

And now for a couple of other things which are connected with what I have been doing whilst working on this bookmark business.

Aside - Outputting the Results
#

At some point I wanted to incrementally write results to a file in JSON format. Occasionally a data fetch would fail and the entire process would end resulting in a somewhat truncated file. So, I decided to make a three part approach:

Write an opening list delimiter.
In a loop, write out fetched JSON dictionary objects, skipping failures.
Write a closing list delimiter

Here’s what I came up with; rough and ready, but worked quite happily.

filename = '/testing/links.json'

# Write starting character
with open(filename, 'w') as start:
    start.write('[')

# Here you might have something that fetched a page at a time and created
# a list of bookmarks per page - then interate through that list next.

# For each, blah, append links - add a comma after each bookmark except for the last bookmark
# Here's hoping that the last list item is not a failure.
for idx, bookmark in enumerate(bookmarks):
    with codecs.open(filename, 'a', encoding='utf-8') as incr:
        incr.write(json.dumps(get_entry(bookmark)))
        if idx < len(bookmarks) - 1:
            incr.write(',')

# Done, we write the ending character
with open(filename, 'a') as end:
    end.write(']')

Aside - Making an importable bookmarks HTML file
#

Here are some brief notes regarding turning JSON entries into bookmark HTML which I can import into a browser.

Below is a line of desired link HTML:

<DT><A HREF="http://www.something.com/some-article/" ADD_DATE="1395356195" PRIVATE="0" TAGS="stuff,things" ICON_URI="http://www.something.com/favicon2.ico" ICON="data:image/x-icon;base64,[...]>Article Title</A>

In the code below, link is a JSON dict in the [same format as I generated earlier](#same format as I generated earlier).

bm_load = json.loads(link)
outlink = u'<DT><A HREF="{}" ADD_DATE="{}" PRIVATE="0" TAGS="{}" ICON_URI="{}" ICON="{}>{}</A>'.format(
    bm_load['url'], bm_load['add_epoch'], ','.join(bm_load['tags']), bm_load['icon_uri'],
    bm_load['icon_data'], bm_load['comment']
)
print outlink

And here we are:

u'<DT><A HREF="http://www.martyncurrey.com/arduino-nano-as-an-isp-programmer/" ADD_DATE="1430895155" PRIVATE="0" TAGS="arduino,programming" ICON_URI="http://www.martyncurrey.com/wp-content/uploads/2016/10/favicon2.ico" ICON="data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAJwOAACcDgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAO5YAECFMQB/AAAAAAAAAAAAAAAA8FgAQIozAH9kJQAgAAAAAAAAAACuQAB/YCMAv0AYAJ9AGAAgAAAAAAAAAAD/dyd/jDQA/wAAAAAAAAAAAAAAAP91JH+fOwD/ViAAQAAAAAD/YgZ/6VYA/7xFAP96LQD/QBgA7wAAAAAAAAAA/5ZYf65AAP8AAAAAAAAAAAAAAAD/hT1/xkkA/3QrAED/XwEQ/4Q87/91Jf//dSXf/3Ym36E7AP8AAAAAAAAAAP+eZn/ISgD/AAAAAJw6AHDnVQAw/3Mif89MAP+fOwBA/3QjYP+DO//eUgC/AAAAAP+UVhD/YgbPAAAAAAAAAAD/nmV/5FQA/wAAAACzQgD/w0gAf/dbAH/LSwD/xUkAQP93J3/vWAD/gzAAUAAAAAAAAAAA/3cnMAAAAAAAAAAA/55mf/9hBP/XTwBA4FMA/8JIAL/YUAB/xUkA/+NUAED/cyK/pz4A/1EeABAAAAAAAAAAAAAAAAAAAAAAAAAAAP+han//ciD/9lsAcP9hBP/aUADvykoAf7hEAP/yWQBA/3Miv4UxAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/o25//34z//lcAJ//bhnf/2cO/9dPAJ+jPAD/7lgAQP92Jr+ALwD/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA/6Zyf/+FPf/fUgDP/20Yf/+NSu//XwLPkzYA/+FTAED/fjK/kTUA/wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP+pdn//ikX/tkMA//9gA0D/p3Ov/4E3/481AP/OTABA/4lEj79GAP9AGABAAAAAAAAAAACzQgAQAAAAAAAAAAD/rHt//4tI/5Y3AP/rVwAQ/7OGf/+bYP+SNgD/sUEAQP+WWGD/Zw//QBgAnwAAAAAAAAAAYCMArwAAAAAAAAAA/6+Af/+KRf+GMQDPAAAAAP+4jkD/qnn/oTsA/5g4AED/nmYg/6Fq/7lEAP9gIwCPbSgAr0AYAP8AAAAAAAAAAP+4jn//j03/njoAnwAAAAAAAAAA/7iP/9dPAP+lPQBAAAAAAP+/mp//omz//2wW/9FNAP92KwD/AAAAAAAAAAD/uI5g/5JTv9pQAGAAAAAAAAAAAP+3jZ//bBa/108AMAAAAAD/vZYQ/7uUn/+gaP//bhrfyEoAUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//8AAP/zAADe4QAA3sEAAN7NAADW3wAA0p8AANKfAADAnwAAyJ8AAMifAADMzQA+zMEBEszBDQjc4/kE///yBA==>Arduino Nano as an ISP Programmer | Martyn Currey</A>'

Finally, writing all the links out to a file as HTML. Open and read the bookmarks data JSON file; for each entry, append a formatted piece of HTML:

import codecs
bookmarks = []
with codecs.open('/testing/links.json', 'r', encoding='utf8') as rfh:
    for bookmark in json.loads(rfh.read()):
        bookmarks.append(u'<DT><A HREF="{}" ADD_DATE="{}" PRIVATE="0" TAGS="{}" ICON_URI="{}" ICON="{}">{}</A>\n'.format(
            bookmark['url'], bookmark['add_epoch'], ','.join(bookmark['tags']), bookmark['icon_uri'],
            bookmark['icon_data'], bookmark['comment']
        ))

Create a suitable header:

header = u"""
<!DOCTYPE NETSCAPE-Bookmark-file-1>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
<TITLE>Bookmarks</TITLE>
<H1>Bookmarks</H1>
<DL><p>
"""

Now write the whole lot out to a file ready for consumption by a browser:

with codecs.open('/testing/links.html', 'w', encoding='utf8') as wfh:
    wfh.write(header)
    for line in bookmarks:
        wfh.write(line)

With that, we are done. Until next time.

del.icio.us Link Fetching#

Rationale#

The Plan#

What Data?#

Getting the data#

Aside - Icons#

Testing: Absolute links to icons#

Testing: Relative links to icons#

Joining things up#

Aside - Outputting the Results#

Aside - Making an importable bookmarks HTML file#

Related