aboutsummaryrefslogtreecommitdiffstats
path: root/tests/helpers.py (unfollow)
Commit message (Collapse)AuthorLines
2021-01-10Rename CachedParser to BatchParser and move it to its own moduleGravatar Numerlor-166/+180
2021-01-10Defer import to avoid circular importsGravatar Numerlor-1/+1
2021-01-10Move functions strictly related to parsing html to the _html moduleGravatar Numerlor-120/+126
Some constants need to be shared between html and parsing, because they may also be wanted to be edited by the cog user to change the behaviour, they were moved into the package's init.
2021-01-10Expand docstringGravatar Numerlor-1/+6
2021-01-10Bump markdownify to 0.6.1-0.6.*Gravatar Numerlor-10/+15
The 0.6 release brought a new parameter that has to be included in all tag handling methods
2021-01-10Set exception on futureGravatar Numerlor-1/+2
Without the exception set, to the user the bot would fail silently if an exception was handled here
2021-01-10Refresh inventories when the redis cache is clearedGravatar Numerlor-0/+2
Because the futures are cleaned up and Markdown only exists in the cache after a short time, items that were requested previously and had the cache cleared would be missing from the CachedParser
2021-01-10Handle equal DocItems in the queueGravatar Numerlor-2/+6
This could be handled by using sets to hold the items in _page_symbols, but ultimately the check has a much smaller cost than having thousands of sets for the urls. Because we create futures for every item that ends up in the queue we can also skip the .get is None check and instead fetch the future directly from the dict
2021-01-10Stop scheduled and long running tasks on cog unloadGravatar Numerlor-0/+6
2021-01-10Periodically clear unnecessary futures from the _item_futures dictGravatar Numerlor-1/+33
The code has no way of reaching futures through new requests after their result has been set as that also includes setting its value in redis.
2021-01-09Do not add package name to the front of the symbol if it's already thereGravatar Numerlor-2/+11
2021-01-09Do not ignore symbols with slashes.Gravatar Numerlor-2/+0
In some cases these are actual symbols that we can look up
2021-01-09Ensure no symbols get overwritten while generating symbol mappingsGravatar Numerlor-19/+55
The code handling this was moved to a function to achieve this cleanly. Includes fixes for bugs where incorrect package was added to the symbol name in the second branch and an incorrect symbol being added in the third branch Co-authored-by: MarkKoz <[email protected]>
2021-01-09Do not attempt to set cache values for symbols that were not foundGravatar Numerlor-1/+2
2021-01-09Remove redundant variableGravatar Numerlor-2/+1
Co-authored-by: MarkKoz <[email protected]>
2021-01-09Add comments to truncation handling codeGravatar Numerlor-0/+6
Co-authored-by: MarkKoz <[email protected]>
2021-01-09Use update_wrapper instead of wrapsGravatar Numerlor-6/+5
We're not using it as a decorator so using wraps only complicates the call syntax
2021-01-09Return the fetched inventory in the Inventory converterGravatar Numerlor-46/+50
Instead of fetching it again in the cog, the converter now returns the inventory for later use. The set command now no longer attempts to reschedule the inventory, and a bug that caused the inventory rescheduling to do nothing in `update_single` was fixed after moving it to its own method
2021-01-09Wait for the inventory to be refreshed before attempting any fetchingGravatar Numerlor-3/+5
Previously the bot returned an error if a symbol was not found while inventories were refreshing, but we can just wait for the to finish refreshing and then the symbol may be filled in. A logging call to notify of the refresh being done was also added.
2021-01-09Create futures for all items in the queueGravatar Numerlor-5/+18
Creating futures for everything and then awaiting at the end takes care of all the potential race conditions that may pop up from items that are parsed and sent to redis while the get_markdown method is in the middle of fetching a page. In case it happens with the implementation we'll just need to move the item to the front and the future will get a result set soon afterwards.
2021-01-09Clear up docstring so it doesn't rely on private attributeGravatar Numerlor-1/+1
Co-authored-by: MarkKoz <[email protected]>
2021-01-09Revert "Clear up docstring so it doesn't rely on private attribute"Gravatar Numerlor-1/+1
This reverts commit ad90978f
2021-01-09Run html parsing in an executorGravatar Numerlor-1/+5
The parsing may take up to a few hundred ms depending on the amount of work it has to do
2021-01-09Simplify the implementation of the custom strainerGravatar Numerlor-15/+10
The strainer now forces the text attribute to be None, simplifying the check on strings and falls back to the superclass' method on non string elements
2021-01-09Strip whitespace from symbol Markdown before returning itGravatar Numerlor-1/+1
The html we parse frequently ends up with trailing and sometimes leading newlines which get stripped out by discord anyway, we have no reason to keep those around when sending the Markdown over to redis
2021-01-09Use cancel_all instead of manually calling cancel repeatedlyGravatar Numerlor-2/+1
2021-01-09Use send_denial util instead of creating embed manuallyGravatar Numerlor-7/+2
The symbol is also no longer sent back to the user, as it is not necessary and we can skip the cleanup on it
2021-01-09Return the sent messageGravatar Numerlor-2/+2
This allows the caller to work with the message further
2021-01-09Move copyright outside of license textGravatar Numerlor-4/+2
Co-authored-by: MarkKoz <[email protected]>
2021-01-09Use string addition instead of joinGravatar Numerlor-1/+1
With only two strings, the addition is a bit clearer than constructing and joining a tuple Co-authored-by: MarkKoz <[email protected]>
2021-01-09Change param styling to be consistent with the repoGravatar Numerlor-5/+6
2020-12-15Call command method directlyGravatar Numerlor-1/+1
Co-authored-by: MarkKoz <[email protected]>
2020-12-15Make reschedule delays a module constantGravatar Numerlor-3/+10
2020-12-15Reuse form body to construct log messageGravatar Numerlor-3/+1
Co-authored-by: MarkKoz <[email protected]>
2020-12-15Clear up grammarGravatar Numerlor-4/+4
Co-authored-by: MarkKoz <[email protected]>
2020-12-15Simplify flowGravatar Numerlor-2/+2
The else is a bit clearer than the early return
2020-12-15Lock inventory refreshesGravatar Numerlor-3/+14
All commands that refresh the inventories in some way are now locked to prevent various race conditions that may have occurred in the unlikely scenario that they got triggered together, the fetching part of the get command now also has to wait for the running inventory refresh to finish before proceeding to fetch and parse the html
2020-12-15Create function for merging function and decorator wrapper globalsGravatar Numerlor-6/+30
discord.py uses the globals of functions to resolve forward refs in commands, previously decorators applied before commands broke the bot with forwardrefs to names that weren't in the namespace of the module where they were defined, the new function takes care of merging the globals in a new function to mitigate this issue. closes: #1323
2020-12-15Provide feedback to user when no cache to clear was foundGravatar Numerlor-2/+4
While technically correct, always sending success could be misleading in case of a typo on the package
2020-12-15Move parse_queue cleanup into finally blockGravatar Numerlor-15/+15
The finally will make sure we reset the task and log it no matter what happens, additionally the clearing of the variable is now only done in one place as the finally also executes when the coro is cancelled
2020-12-15Ensure only one future is created for each doc_itemGravatar Numerlor-2/+3
Previously in case get_markdown for an item ran twice, the one that ran second would overwrite the future created by the first one, potentially causing the coro to wait for it infinitely as _parse_queue would only be able to set the last future
2020-12-15Remove internal CachedParser result cacheGravatar Numerlor-12/+8
We no longer need to keep the items around since everything is in redis and the costs of always going through redis is fairly small
2020-12-15Use global bot http_session instead of parameterGravatar Numerlor-14/+13
2020-12-11Restructure doc cache to handle caches of whole pagesGravatar Numerlor-61/+37
Previously we used packages as the top level keys and fields contained the url and the symbol id, however if we want to store all symbols from fetched pages instead of only the ones that were fetched by the users this comes worse off than using the page url in the field and setting EXPIREs for them instead of doing it manually in python. The new implementation uses package:url as the redis key and only the symbol id for field names, with the expire being set to a week on the key, this means we have to pattern match the keys when deleting the cache for a package but that's being done far less than the expire checking done previously.
2020-12-11Make sure PR build artifact is always uploadedGravatar Sebastiaan Zeeff-1/+1
GitHub Actions has an implicit status condition, `success()`, that is added whenever an `if` condition lacks a status function check of its own. In this case, while the upload step did check for the outcome of the previous "always" step, it did not have an actual status check and, thus, only ran on success. Since we always want to upload the artifact, even if other steps failed, I've added the "always" status function now.
2020-12-10Send a custom workflow status embed to DiscordGravatar Sebastiaan Zeeff-0/+100
This commit introduces the same custom status embed as is already being used for Sir Lancebot. The default embeds GitHub sends are disabled, as they were causing slight issues with rate limits from time to time. It works like this: - The Lint & Test workflow stores an artifact with PR information, if we are linting/testing a PR. - Whenever we reach the end of a workflow run sequence, a status embed is send with the conclusion status. Signed-off-by: Sebastiaan Zeeff <[email protected]>
2020-12-10Use Snowflake converter for snowflake commandGravatar ks129-1/+2
2020-12-09Delete verify.py, integrate `!verify` command into verification.py.Gravatar Steele-45/+16
There wasn't any reason the command needed its own cog, so the exact same functionality is now in the Verification cog.
2020-12-09Create Verify cog for new `!verify` command.Gravatar Steele-1/+46
`!verify` command allows moderators to apply the Developer role to a user. `!verify` is therefore removed as an alias for `!accept`.
2020-12-09Create a generator instead of returning a listGravatar Numerlor-6/+4
The result of _split_parameters is only iterated over, so a list is not needed. Making it lazy may also save some time in cases where we don't use all parameters
generated by cgit v1.2.3 (git 2.39.1) at 2025-10-31 18:20:54 +0000