LiveURLs - my best hack till date
Have you ever wanted to share a portion of a web page with others? We've made it easier for you. Toufeeq and I have been working on this Firefox extension project for sometime and now it is out for everyone to try and put to use.
The extension page at Firefox addons
The extension page at Mozdev
Screenshots
Toufeeq, mahendra and I exchange a lot of information through URLs. One day, I wanted to point out a certain paragraph on a lengthy page to Toufeeq. I realized I had very little option of doing such a thing. I could either mail the link to him or copy-paste the contents in the mail. There was practically no way of telling him to visit this URL and read this paragraph (unless it had a named anchor though). We believed there must be way out. After spending a week or so, we zeroed in on a few ways to implement it. We were pretty surprised that there has been no implementation that solves this pain yet. Then we discovered Ahoy! - Everything can be a link with Mozilla's window.getSelection(). Ahoy also aimed at generating links to portions of web pages. It is a very very beautifully described implementation . And it was under the GPL :) Thanks a ton, Brian Donovan, you set our ball rolling.
The most obvious place of putting the information about the selected content was the fragment identifier of the URL. After all the fragment identifier was meant to locate fragments. We wrote out a greasemonkey script (based on the Ahoy idea) the same evening we found about Ahoy and by midnight we had the first version of the Firefox extension ready.
The Ahoy idea uses DOM tree offsets to indicate the starting and ending nodes of the selected range. Though we had a fully functional extension we were'nt very convinced about using DOM offsets to indicate ranges. It was'nt immune to the smallest changes in the DOM structure. With literally every page flaunting dynamic ads and client side scripting usage at an all time high, one can never be sure about the same DOM tree being generated for two rendering instances of the same page.
We needed something to reflect the content that was selected and not the position alone. This was because, in most cases, people would want to share content, rather than the position of the content. Marooned without internet access on a weekend at home, we started off with a crude search algorithm which included as part of the fragment identifer the starting word, ending word, number of characters and number of words of the selection. This worked well enough. But it still did not reflect the exact content. This was nagging me for sometime. Then I recollected a discussion that we had at InfyLUG (The Infosys Linux Users Group) during the initial stages of development. Raghuveer (a Delivery Manager and an InfyLUG old hand) had suggested we should be using some sort of a checksum to 'denote' content. And what a brilliant piece of idea that was. We removed the end word and number of words parameters from the fragment identifier and put in a CRC checksum parameter in their place (thanks to the sample PNG CRC32 implementation). We now have a method that can find content (arbitrary text ranges) within a page with a very high degree of accuracy.
Now that we had a way to locate content reliably, we started working on reducing the length of the fragment identifier. We minimized the use of delimiters. Instead we used a header character indicating the length of data to follow. We converted numbers to their hexadecimal notation to save up on crucial character space. Intially we had used ":" as the delimiter and its url-encoded form (%3A) was consuming valuable space. Three characters for a delimiter was not a luxury we could afford. We changed the delimiter to "+" instead and saved valuable character space. Now the average length of the fragment identifier for the checksum/search based approach is about 18-20 characters and around 26 characters for the DOM based approach, which is'nt very bad at all, considering the things you could do with it. Please read the technology page for more information.
In between all this, we managed to get an opportunity to present this work at the 15th International World Wide Web conference. The poster we submitted to this conference has been selected. We got some company press here.
At the end of all of this, I've got this to say
The impatient ones please
The sender and receiver of the LiveURL should have the same version of the LiveURL extension installed. Though fully functional, this is in its early stage of development. So backward compatibility of LiveURLs might not be supported i.e. there is no guarantee that LiveURLs generated with one version of the extension will work with another version.
This is one hack that I will be proud of, all my life. I hope you find it useful.
The extension page at Firefox addons
The extension page at Mozdev
Screenshots
Toufeeq, mahendra and I exchange a lot of information through URLs. One day, I wanted to point out a certain paragraph on a lengthy page to Toufeeq. I realized I had very little option of doing such a thing. I could either mail the link to him or copy-paste the contents in the mail. There was practically no way of telling him to visit this URL and read this paragraph (unless it had a named anchor though). We believed there must be way out. After spending a week or so, we zeroed in on a few ways to implement it. We were pretty surprised that there has been no implementation that solves this pain yet. Then we discovered Ahoy! - Everything can be a link with Mozilla's window.getSelection(). Ahoy also aimed at generating links to portions of web pages. It is a very very beautifully described implementation . And it was under the GPL :) Thanks a ton, Brian Donovan, you set our ball rolling.
The most obvious place of putting the information about the selected content was the fragment identifier of the URL. After all the fragment identifier was meant to locate fragments. We wrote out a greasemonkey script (based on the Ahoy idea) the same evening we found about Ahoy and by midnight we had the first version of the Firefox extension ready.
The Ahoy idea uses DOM tree offsets to indicate the starting and ending nodes of the selected range. Though we had a fully functional extension we were'nt very convinced about using DOM offsets to indicate ranges. It was'nt immune to the smallest changes in the DOM structure. With literally every page flaunting dynamic ads and client side scripting usage at an all time high, one can never be sure about the same DOM tree being generated for two rendering instances of the same page.
We needed something to reflect the content that was selected and not the position alone. This was because, in most cases, people would want to share content, rather than the position of the content. Marooned without internet access on a weekend at home, we started off with a crude search algorithm which included as part of the fragment identifer the starting word, ending word, number of characters and number of words of the selection. This worked well enough. But it still did not reflect the exact content. This was nagging me for sometime. Then I recollected a discussion that we had at InfyLUG (The Infosys Linux Users Group) during the initial stages of development. Raghuveer (a Delivery Manager and an InfyLUG old hand) had suggested we should be using some sort of a checksum to 'denote' content. And what a brilliant piece of idea that was. We removed the end word and number of words parameters from the fragment identifier and put in a CRC checksum parameter in their place (thanks to the sample PNG CRC32 implementation). We now have a method that can find content (arbitrary text ranges) within a page with a very high degree of accuracy.
Now that we had a way to locate content reliably, we started working on reducing the length of the fragment identifier. We minimized the use of delimiters. Instead we used a header character indicating the length of data to follow. We converted numbers to their hexadecimal notation to save up on crucial character space. Intially we had used ":" as the delimiter and its url-encoded form (%3A) was consuming valuable space. Three characters for a delimiter was not a luxury we could afford. We changed the delimiter to "+" instead and saved valuable character space. Now the average length of the fragment identifier for the checksum/search based approach is about 18-20 characters and around 26 characters for the DOM based approach, which is'nt very bad at all, considering the things you could do with it. Please read the technology page for more information.
In between all this, we managed to get an opportunity to present this work at the 15th International World Wide Web conference. The poster we submitted to this conference has been selected. We got some company press here.
At the end of all of this, I've got this to say
- FOSS rocks. Without an awesome open development platform like Firefox, all this just might not have been possible at all.
- Thanks to all the projects that helped us do what we wanted to do. The Ahoy project which gave us a headstart. The awesome Scrapbook project was a great source of information on how to handle node ranges of a DOM tree. A special thanks to the bookmarklets at www.squarefree.com. It is a very useful Javascript debugging and development tool.
- Thanks to the wonderful team at InfyLUG - Raghuveer, Mahendra, Naveen Unni, Sanjay Eknath, Anoop Varghese, Basavesh and everyone who helped make this happen. The credits page goes out here.
- Patents in their current form suck. You start thinking about them and they suck the creative energy in you.
- Our day would be made when the phrase RTFH (Read the Fine Highlight) becomes as ubiquitous as RTFM ;
The impatient ones please
- Install the Firefox extension from here.
- Select a portion of a web page and invoke the context menu (i.e. right click menu)
- Select either "Copy link to selected content" or "Copy position of selected content". This action copies the selected content as a URL to the clipboard.
- Open the LiveURL in a different tab/window of Firefox which has the same version of the LiveURLs extension installed.
- The browser should show the previously selected content highlighted. (If it does not work, report the bug with the LiveURL that did not work)
The sender and receiver of the LiveURL should have the same version of the LiveURL extension installed. Though fully functional, this is in its early stage of development. So backward compatibility of LiveURLs might not be supported i.e. there is no guarantee that LiveURLs generated with one version of the extension will work with another version.
This is one hack that I will be proud of, all my life. I hope you find it useful.