Tracking Web-site changes
Just wanted to share the results of my research for automated tools that can track changes on web-pages and notify you.
I look after pretty complex CI\CD infrastructure for various platforms, architectures, compilers, thirdparty tools. It took me quite some time to containerize everything, and now my Dockerfiles routinely install 30-40 various things each.
Sometimes I just install the latest version of a tool and rely on automated smoke and compile tests. But sometimes upgrade to a new version is a more formal process. And I need to know when a new version is released.
Instead of checking 100 web-pages on a daily basis, I did a small research for a software that can do this for me.
There are much more complex Data Miners of course. But here I needed just a simple tracker that would email me when some number on a web-page has changed.
I've tried several trackers:
- https://distill.io <- my faviourite one so far, see details below
Distill Web Monitor
- Web: https://distill.io
I really recommend a browser extension though - for Chrome, Firefox or Opera.
Configuring New Feed
You can use a visual picker to just click on a part of a web-page and it automatically generates an XPath for it, which you can edit.
What's more - you can refine result with a RegEx.
Email is given, but being a browser extension it can even Show Notification Popup or even Play Audio Clip.
Dashboard With Extracted Data
Dashboard is awesome - clean, concise and everything in one place.
Once there is a change it shows the updated feed in bold font.
3 hours check interval is default, but you can choose it as small as you like.
There is even a “Live (beta)” option which should be useful for online auctions.
For each feed you can see a history of change - all previous values with timestamps, or a list of current values if your XPath returns several of them.
And here is the end result. Nicely shown in a
git diff manner.
Automation, automation, automation.
So I can sit on a beach with cocktail and my girls…