aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeLines
...
| * | | | | | | | | | | | | | | | | Log exceptions from parsing taskGravatar Numerlor2020-09-20-3/+7
| | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | Merge remote-tracking branch 'upstream/master' into doc-impGravatar Numerlor2020-09-20-2369/+6238
| |\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | # Conflicts: # bot/cogs/doc/cog.py # bot/converters.py # tests/bot/test_converters.py
| * | | | | | | | | | | | | | | | | | Remove unused cacheGravatar Numerlor2020-09-20-32/+0
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Replace forward ref with future annotations importGravatar Numerlor2020-09-20-1/+3
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Replace caching of soups with new class.Gravatar Numerlor2020-09-20-39/+119
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Storing BeautifulSoup objects could lead to memory problems because of their large footprint, the new class replaces the long term storage by parsing all items on the first fetch of the page and only storing their markdown string.
| * | | | | | | | | | | | | | | | | | Don't return fragment in DocItem urlGravatar Numerlor2020-09-16-9/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The fragment is only needed for the user and required sparingly returning only the url while keeping the fragment behind symbol_id simplifies the uses of the url without it.
| * | | | | | | | | | | | | | | | | | Remove old commentGravatar Numerlor2020-09-16-1/+0
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Properly parse labels add fallback for non dt tagsGravatar Numerlor2020-07-26-5/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Labels point to tags that aren't in description lists, like modules or doc symbols which we already handle. If by chance we get a symbol that we don't have in the group for general parsing and which isn't a dt tag, log it and don't attempt to parse signature and use general description parsing instead of parsing a dd tag.
| * | | | | | | | | | | | | | | | | | Rename description functions to be more generalGravatar Numerlor2020-07-26-8/+8
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Add more loggingGravatar Numerlor2020-07-26-0/+4
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Move MarkdownConverter subclass to separate moduleGravatar Numerlor2020-07-26-57/+60
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Fix markdownify's handling of h tags.Gravatar Numerlor2020-07-26-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Discord only allows `**` for bolding while the markdown from the default MarkdownConverter tries to use # time n with h*n* tags for different font weights.
| * | | | | | | | | | | | | | | | | | Parse NavigableStrings in symbol descriptions.Gravatar Numerlor2020-07-26-14/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a symbol, such as [term.numpy](https://matplotlib.org/3.1.1/glossary/index.html#term-numpy) had NavigableStrings as direct children, they were not included as bs4's SoupStrainer won't include both strings and tags in its filters. The implementation goes around the limitation by introducing a new optional flag, bypassing the default check which skips matching tags when the `text` argument is present.
| * | | | | | | | | | | | | | | | | | Ensure all renamed symbols are keptGravatar Numerlor2020-07-25-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After the restructure behaviour change in d790c404ca3dba3843f351d6f42e766956aa73a1, the add to renamed_symbols was not readded and symbols that only passed the first check were being missed.
| * | | | | | | | | | | | | | | | | | Change ValidPythonIdentifier tests to PackageName.Gravatar Numerlor2020-07-22-11/+10
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Remove paragraph chars from descriptionsGravatar Numerlor2020-07-22-1/+1
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Hold url parts in DocItem separately.Gravatar Numerlor2020-07-22-3/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This allows us to save up some memory by not creating unique strings with the base url repeated between them.
| * | | | | | | | | | | | | | | | | | Remove caching from get_symbol_embed.Gravatar Numerlor2020-07-22-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The web request is already cached, and parsing doesn't much more time, but without moving the logic around the cache prevents the stat increase when a symbol is requested.
| * | | | | | | | | | | | | | | | | | Change DocCog to use the new parsing module fully.Gravatar Numerlor2020-07-22-62/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The parsing module provides an interface for fetching the markdown from the symbol data provided to it. Because it's now fully done in an another module we can remove the needless parts from the cog.
| * | | | | | | | | | | | | | | | | | Create the parsing interface function.Gravatar Numerlor2020-07-22-21/+71
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Other functions from the module are not intended to be used directly, with the interface of it being the added function which accepts the symbol and calls internals. All other names except imports and log had the underscore prefix added to accommodate this.
| * | | | | | | | | | | | | | | | | | Create a function for getting the result markdown.Gravatar Numerlor2020-07-22-0/+21
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Fix ordered list indices in markdown converter.Gravatar Numerlor2020-07-22-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | markdownify relies on the parent tag's index method, which goes through all of its contents, if there is anything else in the contents apart from the li tags, those indices are then shifted.
| * | | | | | | | | | | | | | | | | | Move DocMarkdownConverter to parsing.Gravatar Numerlor2020-07-22-34/+34
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Fix handling of elements when fetching signatures.Gravatar Numerlor2020-07-22-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After the change to `find_elements_until_tag`, the text contentsneed to be extracted from the tags instead of passing them directly to re.
| * | | | | | | | | | | | | | | | | | Remove conversion to str when finding elements.Gravatar Numerlor2020-07-22-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The tags need to be processed down the line, which is not viable on strings.
| * | | | | | | | | | | | | | | | | | Create a function for collecting signatures.Gravatar Numerlor2020-07-20-36/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | By getting the signatures without the description we get more flexibility of parsing different symbol groups and decouple the logic from the description which can be parsed directly with the new `find_elements_until_tag` based function.
| * | | | | | | | | | | | | | | | | | Simplify module parsing method.Gravatar Numerlor2020-07-20-9/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Instead of returning None and multiple values, the method now only returns the string of the description. Previously the parsing returned None and quit when appropriate tags for shortening the description were not found, but the new implementation simply defaults to the provided start tag if a better alternative is not found.
| * | | | | | | | | | | | | | | | | | Add function for finding tags until a matching tagGravatar Numerlor2020-07-20-0/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This will allow flexibility in the future when collecting tags for the description and signature of symbols. The base is a function which accepts a callable which is called and iterated over, but 3 names with a partial function that has the callable supplied are provided to keep the outside interface neater.
| * | | | | | | | | | | | | | | | | | Simplify cutoff text.Gravatar Numerlor2020-07-20-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | "read more" seemed out of place with no permalink over it.
| * | | | | | | | | | | | | | | | | | Remove permalink from truncated markdown.Gravatar Numerlor2020-07-20-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The permalink serves no functional purpose in the embed, as it is already included in the title. But it does add the complexity of passing in the url to the parser.
| * | | | | | | | | | | | | | | | | | Rename parser.py to parsing.py.Gravatar Numerlor2020-07-20-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Parser is a stdlib module name, a rename avoids shadowing it.
| * | | | | | | | | | | | | | | | | | Move markdown truncation into parser moduleGravatar Numerlor2020-07-19-25/+31
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Move main parsing methods into a new moduleGravatar Numerlor2020-07-18-96/+108
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Move async_cache into a separate moduleGravatar Numerlor2020-07-18-31/+34
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Create a package for the Doc cog.Gravatar Numerlor2020-07-18-6/+8
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Rename inventories to doc_symbols.Gravatar Numerlor2020-07-14-10/+10
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Change docstrings to use suffixed command names.Gravatar Numerlor2020-07-14-3/+3
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Change package name converter to only accept _a-z.Gravatar Numerlor2020-07-14-15/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Package names are now directly used for stats, where the lowercase a-z characters and _ are used.
| * | | | | | | | | | | | | | | | | | Make the symbol parameter optional.Gravatar Numerlor2020-07-14-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The commands were changed to be greedy, this however made them required arguments breaking the access to the default listing of the available inventories
| * | | | | | | | | | | | | | | | | | Add doc suffix to doc commands.Gravatar Numerlor2020-07-14-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The `set` command shadowed the `set` symbol, causing the command to seemingly not work. A suffix was added to all commands to keep them consistent and future proof; the shorthands were kept unchanged
| * | | | | | | | | | | | | | | | | | Remove codeblock from symbol embed title.Gravatar Numerlor2020-07-10-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The code block caused the url to not highlight the title text on mobile
| * | | | | | | | | | | | | | | | | | Use the group attribute instead of checking the symbol name.Gravatar Numerlor2020-07-07-1/+1
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Create method to fetch and create a BeautifulSoup object from an url.Gravatar Numerlor2020-07-07-6/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Moving this part of the logic into a separate method allows us to put a cache on it, which caches the whole HTML document from the given url, removing the need to do requests to the same URL for every symbol behind it.
| * | | | | | | | | | | | | | | | | | Intern `group_names`Gravatar Numerlor2020-07-06-1/+4
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Add option for user to delete the not found message before it's auto deleted.Gravatar Numerlor2020-06-29-4/+11
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Trigger typing in converter instead of command.Gravatar Numerlor2020-06-29-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The converter does a web request so triggering typing in the command itself left out a period where the bot seemed inactive.
| * | | | | | | | | | | | | | | | | | Add stat for packages of fetched symbols.Gravatar Numerlor2020-06-29-10/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | An additional variable is added to the DocItem named tuple to accommodate this. The `_package_name` is separated from `api_package_name` it previously overwrote and is now used for the stats and renamed symbols because the names are in a friendlier format.
| * | | | | | | | | | | | | | | | | | Only update added inventory instead of all.Gravatar Numerlor2020-06-28-1/+1
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Only include one newline for `p` tags in `li` elements.Gravatar Numerlor2020-06-27-0/+7
| | | | | | | | | | | | | | | | | | |
| * | | | | | | | | | | | | | | | | | Redesign `find_all_text_until_tag` to search through all direct children.Gravatar Numerlor2020-06-27-27/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The previous approach didn't work for arbitrary tags with text.