404-finder 1.0.6

an Abbot package for Microsoft Teams by Serious Business

Check all the links on a page to make sure they all resolve

Abbot is a programmable bot that turns your team chat into a shared command center. We handle all the boilerplate of building and running these conmmands so that you can focus on making tools that help you ship faster.

We built Abbot because we saw the power of this style of work (called ChatOps), when we worked at GitHub. ChatOps made it possible for GitHub to work productively without meetings, while globally distributed. We think it's a pretty great way to work, so we made it easy to use in Slack, Discord, and Microsoft Teams.

You can read more about Abbot here, check out our blog, or take a look at some of the other cool packages available as a one-click install from Abbot's Package Directory.

# Description: Check all the links on a page to make sure they all resolve
#
# Package URL: https://ab.bot/packages/aseriousbiz/404-finder
#
# Usage:
# This skill is best used as a scheduled skill.
#
# @abbot schedule 404-finder

from bs4 import BeautifulSoup
from urllib.parse import urlparse
import requests

# Let's pretend to be Chrome in order to check Twitter 
bot_headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36'}

def check_link(url):
  if url.startswith("mailto:") or url.startswith("tel:"):
    return
  try:
    r = requests.head(url, headers=bot_headers)
    if r.status_code in [404, 500, 502, 503, 504]:
        bot.reply(":warning: {} returned status **{}**".format(url, r.status_code))
  
  except:
    bot.reply(":octagonal-sign: Could not reach `{}` while crawling `{}`".format(url, args))
    

def check_links(url):
  """Check any hyperlinks in the data and report any issues to chat."""
  try:
    r = requests.get(url, headers=bot_headers)
    r.raise_for_status()
    
    soup = BeautifulSoup(r.text, 'html.parser')
    links = soup.find_all('a', href=True)
    url_parts = urlparse(url)
    for link in links:
      target = link['href']
      if target[0] == "/" or target[0] == "#":
          target = "{}://{}{}".format(url_parts.scheme, url_parts.netloc, link['href'])
      check_link(target)
    bot.reply("Finished checking `{}` for 404s".format(url))
  except ConnectionError:
    bot.reply("Error checking for 404s. Could not reach `{}`.".format(url))

if len(bot.arguments) > 0:
  check_links(bot.arguments)
else:
  bot.reply("You didn't say anything")

Version	Created	Author	Release Notes	Installs
1.0.6	11/2/2021	Paul	Handle situations where a domain is not resolving at all.	0
1.0.5	8/4/2021	Paul	Better messages	0
1.0.4	8/4/2021	Paul	Better handle non-root website URLs	0
1.0.3	8/3/2021	Paul	Ignore `tel` links since a browser can't try to navigate to them.	1
1.0.2	6/5/2021	Paul	Only report specific errors: Not Found, 50x Server Error (was previously reporting authentication and redirection errors as warnings incorrectly)	1
1.0.1	6/4/2021	Paul	Handle relative URLs correctly	0
1.0.0	2/13/2021	Paul	First release!	3

Sign in to add this to Microsoft Teams You need an account to install this package with Abbot. Signing in with Microsoft Teams will create an account if you don't have one already.

Not using Microsoft Teams? No worries! Abbot works in Slack and Discord.

It's free to get started!

Details

Last updated: Tuesday, 02 November 2021 17:11
Installs: 5 total installs
Language: Python
License: MIT License

?	Display shortcuts
Cmd+K	Display shortcuts when a form input has focus.
t	Jump to skill
ctrl+t	Jump to skill when a form input has focus.
Esc	Dismiss modal. Also used to remove focus from a text input to enable keyboard shortcut.
g m	Go to the My Bot page
g s	Go to the skills list
g p	Go to the package directory
g a	Go to the About page
g d	Go to the Docs site
g l	Go to Activity Log

About Abbot

README

Usage

Code

Version History

Details

Keyboard Shortcuts

Jump to Skill