notes/website/2020-07-06-suggest-require.md

Suggest-require

Suggest-require is a tiny Lua library, for discovering importable modules for Lua's require function. The intended purpose is as a part of an auto-complete system for Lua written in Lua. The script can, when run as a standalone program, list all of the available modules in the current environment.

The library has been tested with LuaJIT and Lua 5.1 under Linux. Compatibility with Windows, MacOS and other Lua versions is possible, but not assured.

Source code on Gitfub. Usage and examples are available in the source file itself. License is Beerware.

Why suggest-require?

I have been writing a debugging shell in LÖVE for a while now. The shell itself, called Xenoterm, is pretty powerful at this point, with complex interactions and autocompletions, inspired by Fish shell. One of Xenoterm most powerful features is argument auto-completion, where the shell detects which function is being tab-completed on, and suggests several possible arguments. This is similar to when Fish provides suggestions for command-line options, listing files in the local directory, or password files for pass. This allows the user to offload the need to remember specific names or files, and allows them to focus on more important things, like writing code.

Argument completion raison d'être was to make life easier for me and Sketchwhale while working on our games, as we had a habit of forgetting the ids of items and enemies.

For obvious reasons, argument autocompletion makes most sense for functions with a single string argument, and a limited number of inputs. Giving autocompletions for string.lower would not make any sense. A prime candidate from Lua's standard library was require, Lua's module importing functionality, which is why I wrote the suggest-require library written back in 2017.

How is suggest-require implemented?

A curious thing about Lua is that it aims to be POSIX compatible in order to be as portable as possible. This means pretty much no built-in knowledge of filesystems. Instead, Lua stores any information about the file system configuration in the package global: Directory separators, library locations, and so on. Depending upon which platform your Lua is compiled for the proper values are fitted in, but all of this can be inspected and modified at runtime. require will update its behaviour based upon the values in the table. So if you wanted to figure out the importable modules, you have all the information you want.

Unfortunately, Lua is very filesystem agnostic, to the point of having no API for looking up files in a directory. The Lua Filesystem Module exists for filesystem lookup, but I don't like dependencies for a simple script. Lua does have the io.popen function for interacting with the system's shell, and this can be used to query the filesystem. On Unixes, we use the find program to find files, and its builtin pattern matching allowed a bit of a speedup.

So the entire process is roughly:

1. Produce glob patterns, based on information in the global package table.
2. Use find to execute those globs, retrieving a list of possible libraries.
3. The library paths in the package table may contain multiple wildcards (marked as ? in the path), which, when using require is replaced with the requested module. Thus our glob approach finds too many files, and we need to prune some of them, (so ./?/?.lua matches ./a/a.lua, but not ./a/b/.lua.)
4. Add the names of the already loaded libraries into the mix; these are easily found by looking in the global package.loaded table.

Sugggest-require will thus most likely only work on desktop computers, and other full featured environments. These are also the environments where I could see the script being used; a script deep in a game engine or on an embedded device will probably know exactly which modules to import.