One of my key criteria for the end-user part of this project was to provide information about site updates via an RSS service so that information is available to users when they want it by subscription to the feed, using their favourite RSS reader.
By a bit of clever XSLT and CSS, the initial RSS service was quickly extended to include a web page alternative that users can browse to at any time to see the latest updates to their favourite sites and drill-down into the history of changes.
I am now experimenting with a third notification mechanism; e-mail notifications. This starts to overlap with other providers of similar notification systems for web updates, and it was originally my intention not to provide such a facility. It has an obvious conflict with a principal ethos of the site - it’s going to require a registration because I’ll need to know users’ email addresses to send the message to.
Let me know what you think. If this is a popular feature, I can add an email address registration to the site and provide an email notifications in addition to the current RSS and Website mechanism.
The Site Update Notification project is a private project that does not have any funding. All software and the web site is designed and developed in my spare time, and the sunaweb spider runs from my home machine using a standard, low bandwidth broadband service.
I have sunaweb scheduled to run hourly, although this schedule is enabled and disabled as necessary, and I provide no guarantee as to the frequency of monitoring web pages. The update notification service has proved popular and recently the spider crawl of all web pages on all user watchlists has started to take a long time and threatened to exceed the one hour window of my schedule.
Some analysis of the site visitor records suggested to me that less than half of the users who have added web pages to be monitored are actually reading the RSS feed or viewing the change history of the web pages on the project site.
This means that sunaweb is crawling web pages unnecessarily.
The solution I have now implemented is to use the site visitor record to identify the last time that a web change notification feed for each user was read, and to ignore watchlists for users who have not read the notification for seven days. Importantly, I do not delete any records of users, watchlists or site changes, this is purely an additional filter for determining which sites to crawl with the sunaweb spider. If a user reads their change notification after nine days, for example, then this will re-instate the web pages on their watchlist to be crawled in the future.
When you create a watchlist at http://www.siteupdatenotification.com/watch/, we will monitor web pages for you and let you know when they change. Your watchlist is uniquely associated to you - without having to subscribe - and if you have Cookies enabled in your browser, then will be able to access your watchlist and see what changes have been made just by visiting your watchlist page.
You can share your watchlist with your friends and they will be able to see a read-only version, and access the change log for your pages. To help you spread the word, we have included a link to your personal watchlist on the watch homepage, along with a help link to email your friend directly from the site.
The watchlist pages at siteupdatenotification.com now use SnapShots to help users visualise the page being monitored without having to leave the watchlist page.
I have added a new page to display the work queue and status of the sunaweb crawler. The crawler status page shows a comprehensive list of web pages that are monitored for updates along with a date & time stamp of when the page was last modified and how many users have the page on the watchlist.
The RSS feed from siteupdatenotification.com to alert users of changes being made to web pages in their watchlist has had a makeover. The feeds quickly grew to large sizes where a web page changes regularly, or if a user is monitoring more than a few web pages, and large RSS feeds make for a less reponsive user experience.
The feed has now been modified to contain only the most recent set of changes to each page in the user’s watchlist. Most users want to know about new content, so this serves their purpose perfectly. In place of the full change history, the feed now includes links to images that have changed, or have newly appeared. This means that new & updated images can be seen directly in the RSS reader, or on the siteupdatenotification.com website before visiting the changed site.
For users that would like to see the change history of a page, a link is provided to a page showing the history as seen by the sunaweb crawler for a specific page.
Yesterday on this blog, I announced the launch of the web page change notification service that is now available here on siteupdatenotification.com. Over the coming weeks I’ll blog about some of the features of the service, the technologies used, and the reasons for providing this service as a part of the Site Update Notification project.
In the meanwhile, here are some highlights and benefits of the service:
- It is completely Free
- Monitor unlimited web pages - there are NO restrictions
- No user registration is needed
- No software to download
- Complete privacy - nobody knows who’s monitoring what
- Share your personal watchlist with your friends
- Read about changes to pages in a web browser
- Read about changes in an RSS reader
I am very pleased to announce the launch of the web page change notification service here on siteupdatenotification.com.
The notification service is available to anyone to use free of charge. Add any web page address to your personal watchlist and we’ll monitor the page and tell you when the page has changed. There are two convenient ways to see what pages have changed; either through a standard RSS feed or as a web page on our site.
It’s FREE and you do NOT need to registration. Click here to start using the service today.
From today, the Site Update Notification site includes a Wall powered by Google Friend Connect. Users can login to the site with their Google, Yahoo, AIM or OpenID account and post comments and interact with other users.
Please visit and leave a message about the web crawler research project or Deep Web crawling.
Next month, December 2008, will see the launch of a free web page change notification service on www.siteupdatenotification.com. This service will be provided free of charge and will not require any registration.
Users will be able to create their own watchlist - a list of pages about which they would like to be notified when changes are made - and share the list with their friends.
Final alpha testing is currently underway. Check back soon or subscribe to the blog’s RSS feed to be alerted to the feature launch, and other site news.
Subscribe to the RSS feed for updates.
