Challenges (#860)

* beginning commit

creating the base of the hangman, code needs to be linted in the future

* updated words list

* adding images to show the hangman person

* added images, though it is a bit laggy

* replacing images with discord attachment urls

* adding error if filters aren't found

* fixing typo in ``filter_not_found_embed``

* final lints + removing `mode` parameter as it renders useless

* linting flake8 errors

* adding newline at the end of `top_1000_used_words.txt`

* minor change to filter message

* beginning commit -- trying to add bs4 to pyproject.toml, though it is currently failing

* kata information section done, ready for issue

* fixing bugs with the query not being fully picked up, also allowing query only with no kyu

* fixing bug where user cannot leave all arguments blank

* typo - forgot unary before the level within the `language and not query` if statement

* changing to random kata chosen

* ensuring that if the user provides a query that won't work, that it won't error out

* limiting choice to smaller numbers if a query was provided, so the user gets what they want

* improving hangman docstring

* removing `bot/resources/evergreen/hangman` directory as file attachments are used

* replacing single quotes with double quotes, to adhere to the style guide.

* fixing style inconsistencies and other problems with how the code looks - as per requested by Objectivix

* fixing `IMAGES` style inconsistency

* adding trailing commas and switching to `Colours` for consistency

* adding trailing commas and switching to `Colours` for consistency

* fixing the remnants of non-trailing commas and allowing specification for single player vs mulitplayer

* removing all 2 letter words from the hangman word choosing and removing words that @Objectivix found that shouldn't be in the list of words

* removing some inappropriate words from the txt file

* Adding space for grammatical errors

Co-authored-by: ChrisJL <[email protected]>

* changing two periods to a full stop & wrapping try and except block to only the part that can raise it

* using negative replies instead along with fixing grammatical errors in the sentence

* removing words that could be considered inappropirate

* removing `TOP_WORDS_FILE_PATH` and making `ALL_WORDS` a global variable.

* error handling

* fixing the overcomplication of the bs4 portion

* adding button and dropdowns to the challenges command

* more specific docstring

* more specific docstring

* finishing dropdowns/buttons

* putting the dropdown on top of the link button

* replacing ' with a double quote for some strings

* Removing more words

The words removed shouldn't really belong here

* Update bot/exts/utilities/challenges.py

Co-authored-by: Bluenix <[email protected]>

* replacing mapping_of_images with IMAGES and other fixes

* Dedenting

Co-authored-by: Bluenix <[email protected]>

* Improving tries logic

Co-authored-by: Bluenix <[email protected]>

* Updating `positions` list to set

Co-authored-by: Bluenix <[email protected]>

* Updating setup docstring

Co-authored-by: Bluenix <[email protected]>

* Updating comment in callback function of the dropdown

Co-authored-by: Bluenix <[email protected]>

* fixing too many blank lines

* Hardcode dictionary

Co-authored-by: Bluenix <[email protected]>

* restructuring

* fixing errors

* Remove unnecessary comments

Co-authored-by: Bluenix <[email protected]>

* Remove unnecessary comments

Co-authored-by: Bluenix <[email protected]>

* Improve comment explanation

Co-authored-by: Bluenix <[email protected]>

* Remove redundant extra membership test

Co-authored-by: Bluenix <[email protected]>

* Removing verbose variable definition

Co-authored-by: Bluenix <[email protected]>

* Redundant list

Co-authored-by: Bluenix <[email protected]>

* Shorten 'social distancing' (too many separations) between related lines

Co-authored-by: Bluenix <[email protected]>

* improving docstring in `kata_id`

* sending embed if error occurs with api or bs4, also hardcoding params dictionary

* Better comments

Co-authored-by: Bluenix <[email protected]>

* better docstring

Co-authored-by: Bluenix <[email protected]>

* Removing f-string inception and replacing it with more readable code

Co-authored-by: Bluenix <[email protected]>

* More specific docstring

Co-authored-by: Bluenix <[email protected]>

* Removing redundant comments

Co-authored-by: Bluenix <[email protected]>

* Fixing linting errors

* mapping of kyu -> constant

* adding trailing comma

* specific comment regarding where colors are from for `MAPPING_OF_KYU`

* changing name to link too along with link button

* adding ellipsis to make it more clear for `Read more`

* removing redundant sentences from all docstrings of embed creator functions

* fixing unboundlocalerror due to kata_url only being defined under a certain condition

* only allowing supported languages on codewars.com

* fixing url glitch with embed

* Delete hangman.py

* Delete top_1000_used_words.txt

* hangman dependencies leaked into this PR, removing them

* add bs4 and lxml back to lock file

* Capitalize comments

Co-authored-by: Bluenix <[email protected]>

* Improving comments (capitalization)

Co-authored-by: Bluenix <[email protected]>

* polishing

* explaining that self.original_message will be set later in the callback function of the dropdown

* fixing nitpicks

* cast to integer from hex

* removing unnecessary trailing commas

* Simplifying L274-L276

Co-authored-by: Bluenix <[email protected]>

* Add ellipsis to end of description if it's too long

Co-authored-by: Bluenix <[email protected]>

* Changing to hex

Co-authored-by: Bluenix <[email protected]>

* Running blocking function (BeautifulSoup.find_all) to thread

Co-authored-by: Bluenix <[email protected]>

* logger.error errors

* Fixing error with to_thread

* Fixing errors with MAPPING_OF_KYU

Co-authored-by: Bluenix <[email protected]>

* changing `query` to `-query` if the query is a kata level

* changing embed names to add the kata name

* Mimicking mailing list's behavior

Co-authored-by: Bluenix <[email protected]>

* url attribute for all embeds & title for all embeds

* remove view after a certain amount of tikme

* disabling view after waiting instead of just editing it out

* styling

* remove view to avoid spamming errors

* changing `logger` to `log`

Co-authored-by: Xithrius <[email protected]>

* Change `logger` to `log` for logging errors

Co-authored-by: ChrisJL <[email protected]>
Co-authored-by: Bluenix <[email protected]>
Co-authored-by: Xithrius <[email protected]>

3 files changed, 434 insertions, 1 deletions

diff --git a/bot/exts/utilities/challenges.py b/bot/exts/utilities/challenges.py
new file mode 100644
index 00000000..234eb0be
--- /dev/null
+++ b/bot/exts/utilities/challenges.py
@@ -0,0 +1,335 @@
+import logging
+from asyncio import to_thread
+from random import choice
+from typing import Union
+
+from bs4 import BeautifulSoup
+from discord import Embed, Interaction, SelectOption, ui
+from discord.ext import commands
+
+from bot.bot import Bot
+from bot.constants import Colours, Emojis, NEGATIVE_REPLIES
+
+log = logging.getLogger(__name__)
+API_ROOT = "https://www.codewars.com/api/v1/code-challenges/{kata_id}"
+
+# Map difficulty for the kata to color we want to display in the embed.
+# These colors are representative of the colors that each kyu's level represents on codewars.com
+MAPPING_OF_KYU = {
+    8: 0xdddbda, 7: 0xdddbda, 6: 0xecb613, 5: 0xecb613,
+    4: 0x3c7ebb, 3: 0x3c7ebb, 2: 0x866cc7, 1: 0x866cc7
+}
+
+# Supported languages for a kata on codewars.com
+SUPPORTED_LANGUAGES = {
+    "stable": [
+        "c", "c#", "c++", "clojure", "coffeescript", "coq", "crystal", "dart", "elixir",
+        "f#", "go", "groovy", "haskell", "java", "javascript", "kotlin", "lean", "lua", "nasm",
+        "php", "python", "racket", "ruby", "rust", "scala", "shell", "sql", "swift", "typescript"
+    ],
+    "beta": [
+        "agda", "bf", "cfml", "cobol", "commonlisp", "elm", "erlang", "factor",
+        "forth", "fortran", "haxe", "idris", "julia", "nim", "objective-c", "ocaml",
+        "pascal", "perl", "powershell", "prolog", "purescript", "r", "raku", "reason", "solidity", "vb.net"
+    ]
+}
+
+
+class InformationDropdown(ui.Select):
+    """A dropdown inheriting from ui.Select that allows finding out other information about the kata."""
+
+    def __init__(self, language_embed: Embed, tags_embed: Embed, other_info_embed: Embed, main_embed: Embed):
+        options = [
+            SelectOption(
+                label="Main Information",
+                description="See the kata's difficulty, description, etc.",
+                emoji="🌎"
+            ),
+            SelectOption(
+                label="Languages",
+                description="See what languages this kata supports!",
+                emoji=Emojis.reddit_post_text
+            ),
+            SelectOption(
+                label="Tags",
+                description="See what categories this kata falls under!",
+                emoji=Emojis.stackoverflow_tag
+            ),
+            SelectOption(
+                label="Other Information",
+                description="See how other people performed on this kata and more!",
+                emoji="ℹ"
+            )
+        ]
+
+        # We map the option label to the embed instance so that it can be easily looked up later in O(1)
+        self.mapping_of_embeds = {
+            "Main Information": main_embed,
+            "Languages": language_embed,
+            "Tags": tags_embed,
+            "Other Information": other_info_embed,
+        }
+
+        super().__init__(
+            placeholder="See more information regarding this kata",
+            min_values=1,
+            max_values=1,
+            options=options
+        )
+
+    async def callback(self, interaction: Interaction) -> None:
+        """Callback for when someone clicks on a dropdown."""
+        # Edit the message to the embed selected in the option
+        # The `original_message` attribute is set just after the message is sent with the view.
+        # The attribute is not set during initialization.
+        result_embed = self.mapping_of_embeds[self.values[0]]
+        await self.original_message.edit(embed=result_embed)
+
+
+class Challenges(commands.Cog):
+    """
+    Cog for the challenge command.
+
+    The challenge command pulls a random kata from codewars.com.
+    A kata is the name for a challenge, specific to codewars.com.
+
+    The challenge command also has filters to customize the kata that is given.
+    You can specify the language the kata should be from, difficulty and topic of the kata.
+    """
+
+    def __init__(self, bot: Bot):
+        self.bot = bot
+
+    async def kata_id(self, search_link: str, params: dict) -> Union[str, Embed]:
+        """
+        Uses bs4 to get the HTML code for the page of katas, where the page is the link of the formatted `search_link`.
+
+        This will webscrape the search page with `search_link` and then get the ID of a kata for the
+        codewars.com API to use.
+        """
+        async with self.bot.http_session.get(search_link, params=params) as response:
+            if response.status != 200:
+                error_embed = Embed(
+                    title=choice(NEGATIVE_REPLIES),
+                    description="We ran into an error when getting the kata from codewars.com, try again later.",
+                    color=Colours.soft_red
+                )
+                log.error(f"Unexpected response from codewars.com, status code: {response.status}")
+                return error_embed
+
+            soup = BeautifulSoup(await response.text(), features="lxml")
+            first_kata_div = await to_thread(soup.find_all, "div", class_="item-title px-0")
+
+            if not first_kata_div:
+                raise commands.BadArgument("No katas could be found with the filters provided.")
+            elif len(first_kata_div) >= 3:
+                first_kata_div = choice(first_kata_div[:3])
+            elif "q=" not in search_link:
+                first_kata_div = choice(first_kata_div)
+            else:
+                first_kata_div = first_kata_div[0]
+
+            # There are numerous divs before arriving at the id of the kata, which can be used for the link.
+            first_kata_id = first_kata_div.a["href"].split("/")[-1]
+            return first_kata_id
+
+    async def kata_information(self, kata_id: str) -> Union[dict, Embed]:
+        """
+        Returns the information about the Kata.
+
+        Uses the codewars.com API to get information about the kata using `kata_id`.
+        """
+        async with self.bot.http_session.get(API_ROOT.format(kata_id=kata_id)) as response:
+            if response.status != 200:
+                error_embed = Embed(
+                    title=choice(NEGATIVE_REPLIES),
+                    description="We ran into an error when getting the kata information, try again later.",
+                    color=Colours.soft_red
+                )
+                log.error(f"Unexpected response from codewars.com/api/v1, status code: {response.status}")
+                return error_embed
+
+            return await response.json()
+
+    @staticmethod
+    def main_embed(kata_information: dict) -> Embed:
+        """Creates the main embed which displays the name, difficulty and description of the kata."""
+        kata_description = kata_information["description"]
+        kata_url = f"https://codewars.com/kata/{kata_information['id']}"
+
+        # Ensuring it isn't over the length 1024
+        if len(kata_description) > 1024:
+            kata_description = "\n".join(kata_description[:1000].split("\n")[:-1]) + "..."
+            kata_description += f" [continue reading]({kata_url})"
+
+        kata_embed = Embed(
+            title=kata_information["name"],
+            description=kata_description,
+            color=MAPPING_OF_KYU[int(kata_information["rank"]["name"].replace(" kyu", ""))],
+            url=kata_url
+        )
+        kata_embed.add_field(name="Difficulty", value=kata_information["rank"]["name"], inline=False)
+        return kata_embed
+
+    @staticmethod
+    def language_embed(kata_information: dict) -> Embed:
+        """Creates the 'language embed' which displays all languages the kata supports."""
+        kata_url = f"https://codewars.com/kata/{kata_information['id']}"
+
+        languages = "\n".join(map(str.title, kata_information["languages"]))
+        language_embed = Embed(
+            title=kata_information["name"],
+            description=f"```yaml\nSupported Languages:\n{languages}\n```",
+            color=Colours.python_blue,
+            url=kata_url
+        )
+        return language_embed
+
+    @staticmethod
+    def tags_embed(kata_information: dict) -> Embed:
+        """
+        Creates the 'tags embed' which displays all the tags of the Kata.
+
+        Tags explain what the kata is about, this is what codewars.com calls categories.
+        """
+        kata_url = f"https://codewars.com/kata/{kata_information['id']}"
+
+        tags = "\n".join(kata_information["tags"])
+        tags_embed = Embed(
+            title=kata_information["name"],
+            description=f"```yaml\nTags:\n{tags}\n```",
+            color=Colours.grass_green,
+            url=kata_url
+        )
+        return tags_embed
+
+    @staticmethod
+    def miscellaneous_embed(kata_information: dict) -> Embed:
+        """
+        Creates the 'other information embed' which displays miscellaneous information about the kata.
+
+        This embed shows statistics such as the total number of people who completed the kata,
+        the total number of stars of the kata, etc.
+        """
+        kata_url = f"https://codewars.com/kata/{kata_information['id']}"
+
+        embed = Embed(
+            title=kata_information["name"],
+            description="```nim\nOther Information\n```",
+            color=Colours.grass_green,
+            url=kata_url
+        )
+        embed.add_field(
+            name="`Total Score`",
+            value=f"```css\n{kata_information['voteScore']}\n```",
+            inline=False
+        )
+        embed.add_field(
+            name="`Total Stars`",
+            value=f"```css\n{kata_information['totalStars']}\n```",
+            inline=False
+        )
+        embed.add_field(
+            name="`Total Completed`",
+            value=f"```css\n{kata_information['totalCompleted']}\n```",
+            inline=False
+        )
+        embed.add_field(
+            name="`Total Attempts`",
+            value=f"```css\n{kata_information['totalAttempts']}\n```",
+            inline=False
+        )
+        return embed
+
+    @staticmethod
+    def create_view(dropdown: InformationDropdown, link: str) -> ui.View:
+        """
+        Creates the discord.py View for the Discord message components (dropdowns and buttons).
+
+        The discord UI is implemented onto the embed, where the user can choose what information about the kata they
+        want, along with a link button to the kata itself.
+        """
+        view = ui.View()
+        view.add_item(dropdown)
+        view.add_item(ui.Button(label="View the Kata", url=link))
+        return view
+
+    @commands.command(aliases=["kata"])
+    @commands.cooldown(1, 5, commands.BucketType.user)
+    async def challenge(self, ctx: commands.Context, language: str = "python", *, query: str = None) -> None:
+        """
+        The challenge command pulls a random kata (challenge) from codewars.com.
+
+        The different ways to use this command are:
+        `.challenge <language>` - Pulls a random challenge within that language's scope.
+        `.challenge <language> <difficulty>` - The difficulty can be from 1-8,
+        1 being the hardest, 8 being the easiest. This pulls a random challenge within that difficulty & language.
+        `.challenge <language> <query>` - Pulls a random challenge with the query provided under the language
+        `.challenge <language> <query>, <difficulty>` - Pulls a random challenge with the query provided,
+        under that difficulty within the language's scope.
+        """
+        if language.lower() not in SUPPORTED_LANGUAGES["stable"] + SUPPORTED_LANGUAGES["beta"]:
+            raise commands.BadArgument("This is not a recognized language on codewars.com!")
+
+        get_kata_link = f"https://codewars.com/kata/search/{language}"
+        params = {}
+
+        if language and not query:
+            level = f"-{choice([1, 2, 3, 4, 5, 6, 7, 8])}"
+            params["r[]"] = level
+        elif "," in query:
+            query_splitted = query.split("," if ", " not in query else ", ")
+
+            if len(query_splitted) > 2:
+                raise commands.BadArgument(
+                    "There can only be one comma within the query, separating the difficulty and the query itself."
+                )
+
+            query, level = query_splitted
+            params["q"] = query
+            params["r[]"] = f"-{level}"
+        elif query.isnumeric():
+            params["r[]"] = f"-{query}"
+        else:
+            params["q"] = query
+
+        params["beta"] = str(language in SUPPORTED_LANGUAGES["beta"]).lower()
+
+        first_kata_id = await self.kata_id(get_kata_link, params)
+        if isinstance(first_kata_id, Embed):
+            # We ran into an error when retrieving the website link
+            await ctx.send(embed=first_kata_id)
+            return
+
+        kata_information = await self.kata_information(first_kata_id)
+        if isinstance(kata_information, Embed):
+            # Something went wrong when trying to fetch the kata information
+            await ctx.d(embed=kata_information)
+            return
+
+        kata_embed = self.main_embed(kata_information)
+        language_embed = self.language_embed(kata_information)
+        tags_embed = self.tags_embed(kata_information)
+        miscellaneous_embed = self.miscellaneous_embed(kata_information)
+
+        dropdown = InformationDropdown(
+            main_embed=kata_embed,
+            language_embed=language_embed,
+            tags_embed=tags_embed,
+            other_info_embed=miscellaneous_embed
+        )
+        kata_view = self.create_view(dropdown, f"https://codewars.com/kata/{first_kata_id}")
+        original_message = await ctx.send(
+            embed=kata_embed,
+            view=kata_view
+        )
+        dropdown.original_message = original_message
+
+        wait_for_kata = await kata_view.wait()
+        if wait_for_kata:
+            await original_message.edit(embed=kata_embed, view=None)
+
+
+def setup(bot: Bot) -> None:
+    """Load the Challenges cog."""
+    bot.add_cog(Challenges(bot))
diff --git a/poetry.lock b/poetry.lock
index 289f2039..21373a92 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -101,6 +101,21 @@ docs = ["sphinx", "jaraco.packaging (>=8.2)", "rst.linker (>=1.9)"]
 testing = ["pytest (>=4.6)", "pytest-flake8", "pytest-cov", "pytest-black (>=0.3.7)", "pytest-mypy", "pytest-checkdocs (>=2.4)", "pytest-enabler (>=1.0.1)"]
 
 [[package]]
+name = "beautifulsoup4"
+version = "4.10.0"
+description = "Screen-scraping library"
+category = "main"
+optional = false
+python-versions = ">3.0.0"
+
+[package.dependencies]
+soupsieve = ">1.2"
+
+[package.extras]
+html5lib = ["html5lib"]
+lxml = ["lxml"]
+
+[[package]]
 name = "certifi"
 version = "2021.5.30"
 description = "Python package for providing Mozilla's CA Bundle."
@@ -173,6 +188,7 @@ voice = ["PyNaCl (>=1.3.0,<1.5)"]
 [package.source]
 type = "url"
 url = "https://github.com/Rapptz/discord.py/archive/45d498c1b76deaf3b394d17ccf56112fa691d160.zip"
+
 [[package]]
 name = "distlib"
 version = "0.3.2"
@@ -356,6 +372,20 @@ optional = false
 python-versions = ">=3.7"
 
 [[package]]
+name = "lxml"
+version = "4.6.3"
+description = "Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API."
+category = "main"
+optional = false
+python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, != 3.4.*"
+
+[package.extras]
+cssselect = ["cssselect (>=0.7)"]
+html5 = ["html5lib"]
+htmlsoup = ["beautifulsoup4"]
+source = ["Cython (>=0.29.7)"]
+
+[[package]]
 name = "matplotlib"
 version = "3.4.3"
 description = "Python plotting package"
@@ -654,6 +684,14 @@ optional = false
 python-versions = "*"
 
 [[package]]
+name = "soupsieve"
+version = "2.2.1"
+description = "A modern CSS selector implementation for Beautiful Soup."
+category = "main"
+optional = false
+python-versions = ">=3.6"
+
+[[package]]
 name = "taskipy"
 version = "1.8.1"
 description = "tasks runner for python projects"
@@ -730,7 +768,7 @@ multidict = ">=4.0"
 [metadata]
 lock-version = "1.1"
 python-versions = "^3.9"
-content-hash = "9efbf6be5298ab8ace2588e218be309e105987bfdfa8317453d584a1faac4934"
+content-hash = "6cced4e3fff83ad6ead9a18b3f585b83426fab34f6e2bcf2466c2ebbbf66dac4"
 
 [metadata.files]
 aiodns = [
@@ -800,6 +838,10 @@ attrs = [
     {file = "backports.entry_points_selectable-1.1.0-py2.py3-none-any.whl", hash = "sha256:a6d9a871cde5e15b4c4a53e3d43ba890cc6861ec1332c9c2428c92f977192acc"},
     {file = "backports.entry_points_selectable-1.1.0.tar.gz", hash = "sha256:988468260ec1c196dab6ae1149260e2f5472c9110334e5d51adcb77867361f6a"},
 ]
+beautifulsoup4 = [
+    {file = "beautifulsoup4-4.10.0-py3-none-any.whl", hash = "sha256:9a315ce70049920ea4572a4055bc4bd700c940521d36fc858205ad4fcde149bf"},
+    {file = "beautifulsoup4-4.10.0.tar.gz", hash = "sha256:c23ad23c521d818955a4151a67d81580319d4bf548d3d49f4223ae041ff98891"},
+]
 certifi = [
     {file = "certifi-2021.5.30-py2.py3-none-any.whl", hash = "sha256:50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8"},
     {file = "certifi-2021.5.30.tar.gz", hash = "sha256:2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee"},
@@ -1016,6 +1058,56 @@ kiwisolver = [
     {file = "kiwisolver-1.3.2-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:bcadb05c3d4794eb9eee1dddf1c24215c92fb7b55a80beae7a60530a91060560"},
     {file = "kiwisolver-1.3.2.tar.gz", hash = "sha256:fc4453705b81d03568d5b808ad8f09c77c47534f6ac2e72e733f9ca4714aa75c"},
 ]
+lxml = [
+    {file = "lxml-4.6.3-cp27-cp27m-macosx_10_9_x86_64.whl", hash = "sha256:df7c53783a46febb0e70f6b05df2ba104610f2fb0d27023409734a3ecbb78fb2"},
+    {file = "lxml-4.6.3-cp27-cp27m-manylinux1_i686.whl", hash = "sha256:1b7584d421d254ab86d4f0b13ec662a9014397678a7c4265a02a6d7c2b18a75f"},
+    {file = "lxml-4.6.3-cp27-cp27m-manylinux1_x86_64.whl", hash = "sha256:079f3ae844f38982d156efce585bc540c16a926d4436712cf4baee0cce487a3d"},
+    {file = "lxml-4.6.3-cp27-cp27m-win32.whl", hash = "sha256:bc4313cbeb0e7a416a488d72f9680fffffc645f8a838bd2193809881c67dd106"},
+    {file = "lxml-4.6.3-cp27-cp27m-win_amd64.whl", hash = "sha256:8157dadbb09a34a6bd95a50690595e1fa0af1a99445e2744110e3dca7831c4ee"},
+    {file = "lxml-4.6.3-cp27-cp27mu-manylinux1_i686.whl", hash = "sha256:7728e05c35412ba36d3e9795ae8995e3c86958179c9770e65558ec3fdfd3724f"},
+    {file = "lxml-4.6.3-cp27-cp27mu-manylinux1_x86_64.whl", hash = "sha256:4bff24dfeea62f2e56f5bab929b4428ae6caba2d1eea0c2d6eb618e30a71e6d4"},
+    {file = "lxml-4.6.3-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:64812391546a18896adaa86c77c59a4998f33c24788cadc35789e55b727a37f4"},
+    {file = "lxml-4.6.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:c1a40c06fd5ba37ad39caa0b3144eb3772e813b5fb5b084198a985431c2f1e8d"},
+    {file = "lxml-4.6.3-cp35-cp35m-manylinux1_i686.whl", hash = "sha256:74f7d8d439b18fa4c385f3f5dfd11144bb87c1da034a466c5b5577d23a1d9b51"},
+    {file = "lxml-4.6.3-cp35-cp35m-manylinux1_x86_64.whl", hash = "sha256:f90ba11136bfdd25cae3951af8da2e95121c9b9b93727b1b896e3fa105b2f586"},
+    {file = "lxml-4.6.3-cp35-cp35m-manylinux2010_i686.whl", hash = "sha256:4c61b3a0db43a1607d6264166b230438f85bfed02e8cff20c22e564d0faff354"},
+    {file = "lxml-4.6.3-cp35-cp35m-manylinux2014_x86_64.whl", hash = "sha256:5c8c163396cc0df3fd151b927e74f6e4acd67160d6c33304e805b84293351d16"},
+    {file = "lxml-4.6.3-cp35-cp35m-win32.whl", hash = "sha256:f2380a6376dfa090227b663f9678150ef27543483055cc327555fb592c5967e2"},
+    {file = "lxml-4.6.3-cp35-cp35m-win_amd64.whl", hash = "sha256:c4f05c5a7c49d2fb70223d0d5bcfbe474cf928310ac9fa6a7c6dddc831d0b1d4"},
+    {file = "lxml-4.6.3-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:d2e35d7bf1c1ac8c538f88d26b396e73dd81440d59c1ef8522e1ea77b345ede4"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux1_i686.whl", hash = "sha256:289e9ca1a9287f08daaf796d96e06cb2bc2958891d7911ac7cae1c5f9e1e0ee3"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:bccbfc27563652de7dc9bdc595cb25e90b59c5f8e23e806ed0fd623755b6565d"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux2010_i686.whl", hash = "sha256:d916d31fd85b2f78c76400d625076d9124de3e4bda8b016d25a050cc7d603f24"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux2014_aarch64.whl", hash = "sha256:820628b7b3135403540202e60551e741f9b6d3304371712521be939470b454ec"},
+    {file = "lxml-4.6.3-cp36-cp36m-manylinux2014_x86_64.whl", hash = "sha256:c47ff7e0a36d4efac9fd692cfa33fbd0636674c102e9e8d9b26e1b93a94e7617"},
+    {file = "lxml-4.6.3-cp36-cp36m-win32.whl", hash = "sha256:5a0a14e264069c03e46f926be0d8919f4105c1623d620e7ec0e612a2e9bf1c04"},
+    {file = "lxml-4.6.3-cp36-cp36m-win_amd64.whl", hash = "sha256:92e821e43ad382332eade6812e298dc9701c75fe289f2a2d39c7960b43d1e92a"},
+    {file = "lxml-4.6.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:efd7a09678fd8b53117f6bae4fa3825e0a22b03ef0a932e070c0bdbb3a35e654"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:efac139c3f0bf4f0939f9375af4b02c5ad83a622de52d6dfa8e438e8e01d0eb0"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:0fbcf5565ac01dff87cbfc0ff323515c823081c5777a9fc7703ff58388c258c3"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux2010_i686.whl", hash = "sha256:36108c73739985979bf302006527cf8a20515ce444ba916281d1c43938b8bb96"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux2014_aarch64.whl", hash = "sha256:122fba10466c7bd4178b07dba427aa516286b846b2cbd6f6169141917283aae2"},
+    {file = "lxml-4.6.3-cp37-cp37m-manylinux2014_x86_64.whl", hash = "sha256:cdaf11d2bd275bf391b5308f86731e5194a21af45fbaaaf1d9e8147b9160ea92"},
+    {file = "lxml-4.6.3-cp37-cp37m-win32.whl", hash = "sha256:3439c71103ef0e904ea0a1901611863e51f50b5cd5e8654a151740fde5e1cade"},
+    {file = "lxml-4.6.3-cp37-cp37m-win_amd64.whl", hash = "sha256:4289728b5e2000a4ad4ab8da6e1db2e093c63c08bdc0414799ee776a3f78da4b"},
+    {file = "lxml-4.6.3-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:b007cbb845b28db4fb8b6a5cdcbf65bacb16a8bd328b53cbc0698688a68e1caa"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux1_i686.whl", hash = "sha256:76fa7b1362d19f8fbd3e75fe2fb7c79359b0af8747e6f7141c338f0bee2f871a"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:26e761ab5b07adf5f555ee82fb4bfc35bf93750499c6c7614bd64d12aaa67927"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux2010_i686.whl", hash = "sha256:e1cbd3f19a61e27e011e02f9600837b921ac661f0c40560eefb366e4e4fb275e"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux2014_aarch64.whl", hash = "sha256:66e575c62792c3f9ca47cb8b6fab9e35bab91360c783d1606f758761810c9791"},
+    {file = "lxml-4.6.3-cp38-cp38-manylinux2014_x86_64.whl", hash = "sha256:1b38116b6e628118dea5b2186ee6820ab138dbb1e24a13e478490c7db2f326ae"},
+    {file = "lxml-4.6.3-cp38-cp38-win32.whl", hash = "sha256:89b8b22a5ff72d89d48d0e62abb14340d9e99fd637d046c27b8b257a01ffbe28"},
+    {file = "lxml-4.6.3-cp38-cp38-win_amd64.whl", hash = "sha256:2a9d50e69aac3ebee695424f7dbd7b8c6d6eb7de2a2eb6b0f6c7db6aa41e02b7"},
+    {file = "lxml-4.6.3-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:ce256aaa50f6cc9a649c51be3cd4ff142d67295bfc4f490c9134d0f9f6d58ef0"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux1_i686.whl", hash = "sha256:7610b8c31688f0b1be0ef882889817939490a36d0ee880ea562a4e1399c447a1"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:f8380c03e45cf09f8557bdaa41e1fa7c81f3ae22828e1db470ab2a6c96d8bc23"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux2010_i686.whl", hash = "sha256:3082c518be8e97324390614dacd041bb1358c882d77108ca1957ba47738d9d59"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux2014_aarch64.whl", hash = "sha256:884ab9b29feaca361f7f88d811b1eea9bfca36cf3da27768d28ad45c3ee6f969"},
+    {file = "lxml-4.6.3-cp39-cp39-manylinux2014_x86_64.whl", hash = "sha256:6f12e1427285008fd32a6025e38e977d44d6382cf28e7201ed10d6c1698d2a9a"},
+    {file = "lxml-4.6.3-cp39-cp39-win32.whl", hash = "sha256:33bb934a044cf32157c12bfcfbb6649807da20aa92c062ef51903415c704704f"},
+    {file = "lxml-4.6.3-cp39-cp39-win_amd64.whl", hash = "sha256:542d454665a3e277f76954418124d67516c5f88e51a900365ed54a9806122b83"},
+    {file = "lxml-4.6.3.tar.gz", hash = "sha256:39b78571b3b30645ac77b95f7c69d1bffc4cf8c3b157c435a34da72e78c82468"},
+]
 matplotlib = [
     {file = "matplotlib-3.4.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:5c988bb43414c7c2b0a31bd5187b4d27fd625c080371b463a6d422047df78913"},
     {file = "matplotlib-3.4.3-cp37-cp37m-manylinux1_i686.whl", hash = "sha256:f1c5efc278d996af8a251b2ce0b07bbeccb821f25c8c9846bdcb00ffc7f158aa"},
@@ -1401,6 +1493,10 @@ sortedcontainers = [
     {file = "sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0"},
     {file = "sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88"},
 ]
+soupsieve = [
+    {file = "soupsieve-2.2.1-py3-none-any.whl", hash = "sha256:c2c1c2d44f158cdbddab7824a9af8c4f83c76b1e23e049479aa432feb6c4c23b"},
+    {file = "soupsieve-2.2.1.tar.gz", hash = "sha256:052774848f448cf19c7e959adf5566904d525f33a3f8b6ba6f6f8f26ec7de0cc"},
+]
 taskipy = [
     {file = "taskipy-1.8.1-py3-none-any.whl", hash = "sha256:2b98f499966e40175d1f1306a64587f49dfa41b90d0d86c8f28b067cc58d0a56"},
     {file = "taskipy-1.8.1.tar.gz", hash = "sha256:7a2404125817e45d80e13fa663cae35da6e8ba590230094e815633653e25f98f"},
diff --git a/pyproject.toml b/pyproject.toml
index 7848f593..08287b23 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -12,12 +12,14 @@ aiodns = "~=2.0"
 aioredis = "~1.3"
 rapidfuzz = "~=1.4"
 arrow = "~=1.1.0"
+beautifulsoup4 = "~=4.9"
 pillow = "~=8.1"
 sentry-sdk = "~=0.19"
 PyYAML = "~=5.4"
 async-rediscache = {extras = ["fakeredis"], version = "~=0.1.4"}
 emojis = "~=0.6.0"
 matplotlib = "~=3.4.1"
+lxml = "~=4.4"
 
 [tool.poetry.dev-dependencies]
 flake8 = "~=3.8"