aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeLines
...
| | * | | | | | | Wait for the inventory to be refreshed before attempting any fetchingGravatar Numerlor2021-01-09-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously the bot returned an error if a symbol was not found while inventories were refreshing, but we can just wait for the to finish refreshing and then the symbol may be filled in. A logging call to notify of the refresh being done was also added.
| | * | | | | | | Create futures for all items in the queueGravatar Numerlor2021-01-09-5/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Creating futures for everything and then awaiting at the end takes care of all the potential race conditions that may pop up from items that are parsed and sent to redis while the get_markdown method is in the middle of fetching a page. In case it happens with the implementation we'll just need to move the item to the front and the future will get a result set soon afterwards.
| | * | | | | | | Clear up docstring so it doesn't rely on private attributeGravatar Numerlor2021-01-09-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | Revert "Clear up docstring so it doesn't rely on private attribute"Gravatar Numerlor2021-01-09-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit ad90978f
| | * | | | | | | Run html parsing in an executorGravatar Numerlor2021-01-09-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The parsing may take up to a few hundred ms depending on the amount of work it has to do
| | * | | | | | | Simplify the implementation of the custom strainerGravatar Numerlor2021-01-09-15/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The strainer now forces the text attribute to be None, simplifying the check on strings and falls back to the superclass' method on non string elements
| | * | | | | | | Strip whitespace from symbol Markdown before returning itGravatar Numerlor2021-01-09-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The html we parse frequently ends up with trailing and sometimes leading newlines which get stripped out by discord anyway, we have no reason to keep those around when sending the Markdown over to redis
| | * | | | | | | Use cancel_all instead of manually calling cancel repeatedlyGravatar Numerlor2021-01-09-2/+1
| | | | | | | | |
| | * | | | | | | Use send_denial util instead of creating embed manuallyGravatar Numerlor2021-01-09-7/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The symbol is also no longer sent back to the user, as it is not necessary and we can skip the cleanup on it
| | * | | | | | | Return the sent messageGravatar Numerlor2021-01-09-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows the caller to work with the message further
| | * | | | | | | Move copyright outside of license textGravatar Numerlor2021-01-09-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | Use string addition instead of joinGravatar Numerlor2021-01-09-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With only two strings, the addition is a bit clearer than constructing and joining a tuple Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | Change param styling to be consistent with the repoGravatar Numerlor2021-01-09-5/+6
| | | | | | | | |
| | * | | | | | | Call command method directlyGravatar Numerlor2020-12-15-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | Make reschedule delays a module constantGravatar Numerlor2020-12-15-3/+10
| | | | | | | | |
| | * | | | | | | Reuse form body to construct log messageGravatar Numerlor2020-12-15-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | Clear up grammarGravatar Numerlor2020-12-15-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | Simplify flowGravatar Numerlor2020-12-15-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The else is a bit clearer than the early return
| | * | | | | | | Lock inventory refreshesGravatar Numerlor2020-12-15-3/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | All commands that refresh the inventories in some way are now locked to prevent various race conditions that may have occurred in the unlikely scenario that they got triggered together, the fetching part of the get command now also has to wait for the running inventory refresh to finish before proceeding to fetch and parse the html
| | * | | | | | | Create function for merging function and decorator wrapper globalsGravatar Numerlor2020-12-15-6/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | discord.py uses the globals of functions to resolve forward refs in commands, previously decorators applied before commands broke the bot with forwardrefs to names that weren't in the namespace of the module where they were defined, the new function takes care of merging the globals in a new function to mitigate this issue. closes: #1323
| | * | | | | | | Provide feedback to user when no cache to clear was foundGravatar Numerlor2020-12-15-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While technically correct, always sending success could be misleading in case of a typo on the package
| | * | | | | | | Move parse_queue cleanup into finally blockGravatar Numerlor2020-12-15-15/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The finally will make sure we reset the task and log it no matter what happens, additionally the clearing of the variable is now only done in one place as the finally also executes when the coro is cancelled
| | * | | | | | | Ensure only one future is created for each doc_itemGravatar Numerlor2020-12-15-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously in case get_markdown for an item ran twice, the one that ran second would overwrite the future created by the first one, potentially causing the coro to wait for it infinitely as _parse_queue would only be able to set the last future
| | * | | | | | | Remove internal CachedParser result cacheGravatar Numerlor2020-12-15-12/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We no longer need to keep the items around since everything is in redis and the costs of always going through redis is fairly small
| | * | | | | | | Use global bot http_session instead of parameterGravatar Numerlor2020-12-15-14/+13
| | | | | | | | |
| | * | | | | | | Merge remote-tracking branch 'upstream/master' into doc-impGravatar Numerlor2020-12-11-1635/+2259
| | |\ \ \ \ \ \ \
| | * | | | | | | | Restructure doc cache to handle caches of whole pagesGravatar Numerlor2020-12-11-61/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously we used packages as the top level keys and fields contained the url and the symbol id, however if we want to store all symbols from fetched pages instead of only the ones that were fetched by the users this comes worse off than using the page url in the field and setting EXPIREs for them instead of doing it manually in python. The new implementation uses package:url as the redis key and only the symbol id for field names, with the expire being set to a week on the key, this means we have to pattern match the keys when deleting the cache for a package but that's being done far less than the expire checking done previously.
| | * | | | | | | | Create a generator instead of returning a listGravatar Numerlor2020-12-09-6/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The result of _split_parameters is only iterated over, so a list is not needed. Making it lazy may also save some time in cases where we don't use all parameters
| | * | | | | | | | Merge current_search and expected_end inGravatar Numerlor2020-12-09-10/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The two variables were initialized and cleared together and contained related information
| | * | | | | | | | Improve handling of stringsGravatar Numerlor2020-12-09-7/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously the code assumed ' and " can be used interchangeably, and strings that were inside of brackets were ignored for depth but their contents weren't causing strings like "ab[cd" to increase the depth
| | * | | | | | | | Remove redundant suppressGravatar Numerlor2020-11-29-2/+2
| | | | | | | | | |
| | * | | | | | | | Various grammar and sentence structure changesGravatar Numerlor2020-11-29-13/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | | Clear up docstring so it doesn't rely on private attributeGravatar Numerlor2020-11-29-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | | Use pop instead of getitem and delGravatar Numerlor2020-11-29-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Co-authored-by: MarkKoz <[email protected]>
| | * | | | | | | | Use timedelta instead of constructing duration manuallyGravatar Numerlor2020-11-29-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A newline was also added to set to keep it consistent with set_if_exists
| | * | | | | | | | Ensure packages from PRIORITY_PACKAGES are directly accessibleGravatar Numerlor2020-11-15-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some packages (currently only python) should be prioritised to others, the previous cleanup didn't account for other packages loading before it which resulted in duplicate symbols getting the python prefix and the original symbols linking to most probably undesired pages
| | * | | | | | | | Add command for clearing the cache of packagesGravatar Numerlor2020-11-15-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We also clear the cache when removing a package
| | * | | | | | | | Update existing redis values when parsing pagesGravatar Numerlor2020-11-15-6/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If we're parsing a page for a symbol that's out of the cache and encounter a symbol that was already cached we can update that symbol to keep it up to date without additional requests
| | * | | | | | | | Rework the doc redis cache to work with hashesGravatar Numerlor2020-11-15-12/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This rework requires us to delete packages caches easily with deleting the package hash instead of having to pattern match all keys and delete those. The interface was also updated to accept DocItems instead of requiring callers to construct the keys
| | * | | | | | | | Limit newlines in doc descriptionsGravatar Numerlor2020-11-15-16/+32
| | | | | | | | | |
| | * | | | | | | | Intern relative url pathsGravatar Numerlor2020-11-15-4/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Group name interning was also moved to the DocItem creation to group the behaviour
| | * | | | | | | | Cancel scheduled inventory updates on all refreshesGravatar Numerlor2020-11-15-6/+2
| | | | | | | | | |
| | * | | | | | | | Adjust unparseable symbol behaviourGravatar Numerlor2020-11-15-3/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With redis we need to make sure we don't send the "error" string into the cache, returning None instead of the string and then setting it manually in the caller makes this nicer compared to checking against a string
| | * | | | | | | | Update outdated docstringGravatar Numerlor2020-11-15-1/+1
| | | | | | | | | |
| | * | | | | | | | Rename variables for clarityGravatar Numerlor2020-11-15-11/+11
| | | | | | | | | |
| | * | | | | | | | Generalise tag filter hint to accept all containersGravatar Numerlor2020-11-10-5/+5
| | | | | | | | | |
| | * | | | | | | | Relock Pipfile.lockGravatar Numerlor2020-11-10-298/+136
| | | | | | | | | |
| | * | | | | | | | Merge remote-tracking branch 'upstream/master' into doc-impGravatar Numerlor2020-11-10-782/+2177
| | |\ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | # Conflicts: # LICENSE-THIRD-PARTY # bot/exts/info/doc.py
| | * | | | | | | | | Move InventoryURL converer to the converters fileGravatar Numerlor2020-11-10-21/+22
| | | | | | | | | | |
| | * | | | | | | | | Simplify duplicate symbol name handling codeGravatar Numerlor2020-11-10-10/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | With the catchall else condition and symbols from FORCE_PREFIX_GROUPS getting renamed even when being overwritten, we can ignore the package handling and let it go to the else which adds the package prefix instead of a group
generated by cgit v1.2.3 (git 2.39.1) at 2025-10-26 06:30:19 +0000