Tracking Web-site changes

2-minute read

Preface

Just wanted to share the results of my research for automated tools that can track changes on web-pages and notify you.

I look after pretty complex CI\CD infrastructure for various platforms, architectures, compilers, thirdparty tools. It took me quite some time to containerize everything, and now my Dockerfiles routinely install 30-40 various things each.

Sometimes I just install the latest version of a tool and rely on automated smoke and compile tests. But sometimes upgrade to a new version is a more formal process. And I need to know when a new version is released.

Instead of checking 100 web-pages on a daily basis, I did a small research for a software that can do this for me.

There are much more complex Data Miners of course. But here I needed just a simple tracker that would email me when some number on a web-page has changed.

Candidates

I’ve tried several trackers:

Distill Web Monitor

I really recommend a browser extension though - for Chrome, Firefox or Opera.

Configuring New Feed

You can use a visual picker to just click on a part of a web-page and it automatically generates an XPath for it, which you can edit.
What’s more - you can refine result with a RegEx.

configuring new feed

Notifications options

Email is given, but being a browser extension it can even Show Notification Popup or even Play Audio Clip.

adding notification actions

Dashboard With Extracted Data

Dashboard is awesome - clean, concise and everything in one place.
Once there is a change it shows the updated feed in bold font.

3 hours check interval is default, but you can choose it as small as you like.
There is even a “Live (beta)” option which should be useful for online auctions.

dashboard with extracted data

For each feed you can see a history of change - all previous values with timestamps, or a list of current values if your XPath returns several of them.

history with timestamps

Email Alert

And here is the end result. Nicely shown in a git diff manner.

email alert

Summary

Automation, automation, automation.
So I can sit on a beach with cocktail and my girls…