Scheme index

Scheme index is a tool for indexing and searching through procedures and syntax from Scheme specifications and select SRFI libraries. The index may be used through a site (https://index.scheme.org) or through REST API.

Userguide for site visitors

Scheme index search consists of control pane on the left, and result list in the center.

Control pane

In the control pane, optionally select values from the filter list, optionally enter search query in the text field, and press either enter or button with magnifying glass to display the results. You can use control + f to quickly focus to the query text field.

Result list

Each result item is either a procedure, a macro, or a value. At the top it shows a library it is exported from. Identifiers that are exported from multiple libraries will appear in the result list multiple times. The library name can be clicked, which will result in starting new search using said library as a filter. On the right side top, associated list of tags with the result is shown. Tags are also clickable, and start a search using the clicked tag as a filter.

Procedures and value names are rendered in red. Procedures are distinguished by being surrounded with parenthesis, as if being called. Following the name until the "⇒" sign are parameters. If parameter is just a name, it means it has an unspecified type. Otherwise, parameter may be a list of type and name. Types are usually represented as predicate procedures (sometimes a type may be opaque or logical, and without a disjoint predicate function to distinguish it). It can also be a logical "or" of multiple types, separated by /. Sometimes #f literal is used as a type due to its significant role in scheme to act what would be null elsewhere.

After the "⇒" sign is the return type. The return type is same as parameter type, and additionally may contain:

a "*" symbol (meaning it returns a value of unknown type);
a list with "values" word in car position (meaning it has a multivalue return);

Types expressed as predicates in blue are links. Pressing on a type in parameter position will search using that type as return value filter; pressing on a type in return value position will search using that type as parameter filter.

If one of parameters or return value is procedure?, list?, vector?, its signature might be further elaborated below the main procedure’s signature.

Macros are rendered in green. If macro is complex, some parts of it are grouped, and the syntax of those groups shown below the syntax of whole macro. Macro literals are also rendered in green. To make parenthesis more obvious, auxiliary parenthesis coloring is used, however this coloring doesn’t signify any information.

Filtering logic

Selecting multiple libraries, will return results that are part of any of the given libraries.

Selecting multiple tags, will return results that contain all of the given tags.

Selecting multiple parameter types will return results that match all of the given types.

Some types contain logical "parent" / more general types, eg. number? is a parent of integer?. When searching by parameter type, the procedures that take parent type will be matched. However, the results with more precise "child" type are weight more heavily and should appear at the start of the search.

Some types are defined as a union of other types; for example list? is a union of pair? and null?. If loose parameter filtering is enabled under settings, the composing parts will be treated as parent of union type; ie searching for list? you’d be also presented with procedures applicable to pair?. On one hand this presents extra results you might be interested, on the other hand those results might not be applicable in general case.

Selecting multiple return types will return results that match all of the given types. As and in case filter by parameter type, when filtering by return type the type hierarchy is taken into consideration. Searching by a "parent" return type will also yield procedures returning more precise "child" types.

The text query is parsed by edismax parser (read more about it https://solr.apache.org/guide/6_6/the-dismax-query-parser.html#the-dismax-query-parser), and is used to filter by name and parameter / subsyntax names. This means it supports and interprets common searching syntax, such as using "-" in front of the word to exclude results containing said word. This has its disadvantages; eg. if you tried to search for coercion functions and typed "→string" (without quotes) into the search input field, you wouldn’t find anything interesting, because the leading minus was interpreted specially. Instead, you’d have to search using "exact phrase", by putting double quotes around the search.

Obtaining and running index yourself

Download

Prebuilt versions are available at https://github.com/arvyy/r7rs-index-site/releases

Docker image (always corresponding to latest deployed version) may be pulled from arvyy/scheme-index:master.

Building native

Build processes requires following tools on path:

ant - Apache Ant build tool
mvn - Apache maven project tool
asciidoctor - Asciidoctor documentation compiler
sass - SCSS compiler
(optional) npm - nodejs / npm tooling to build CLI client application.

You can build scheme index by running ant from the root of the source directory. Pass -DbuildClient=true to include CLI client application build.

After the build successfully completes, you should find everything scheme index needs in dist directory.

Running native

Once you have obtained a built version, you can run it using:

java -jar scheme-index.jar

Note that working directory is important; type files, configuration are resolved relative to the working dir, not relative to jar file.

See Configuration section for configuring the application behavior.

Building docker image

Build a docker image using

docker build -t scmindex-builder:latest -f docker/base/Dockerfile .
docker build -t scheme-index -f docker/scheme-index/Dockerfile .

The built image has same structure as a native build, under path /app inside the image.

Running docker image

To run with docker, execute

docker run -p 8080:8080 --init scheme-index

The application resides in /app location. Consult rest of the documentation for details, but in particular you might want to mount a volume to /app/logs to catch log files, or a volume file to /app/config/configuration.scm to overwrite index config.

You can build & run locally built images using

cd docker
mkdir logs
docker rmi docker-nginx:latest docker-app:latest scmindex-builder:latest -f
docker-compose -f docker/docker-compose.local.yml up

To run deployed images from registry use

cd docker
mkdir logs
docker-compose up

Running in development (native)

First, install dependencies under kawa-web-collection submodule (make sure the git submodule is initialized / updated)

cd kawa-web-collection
mvn install

Afterwards, run the application with

mvn kawa:run

Launch sass compiler with

sass -w src/main/scss/main.scss static/css/scmindex.css

Execute unit tests using

mvn test

Running in development (docker)

Make sure kawa-web-collection submodule is initialized

Run with

docker-compose -f docker/docker-compose.dev.yml up app sass nginx

Execute unit tests using

docker-compose -f docker/docker-compose.dev.yml up test

Configuration

configuration.scm

Core application behavior is managed using a scm configuration file, containing a list of key-value pairs, where each key is a symbol (ie. an alist). Running the application, first argument is interpreted as a path to configuration file. If not set, it defaults to ./config/configuration.scm

The following is exhaustive list of valid properties

 Property

Description

 Default

 solr-embed

Specify whether embed solr / lucene index inside the application itself (if #t), or connect to a standalone solr instance (if #f)

#t

 solr-home

Directory contain solr home / configuration. Relevant only if solr-embed is #t

 "./solrhome"

 solr-url

Solr url to connect to. Relevant only if solr-embed is #f

 "http://localhost:8983/solr"

 solr-core

Solr core to use

 "scmindex"

 port

Which port to use. Relevant only if enable-web is #t

 cache-templates

Whether templates should be cached (ie compiled once and remembered). Use #f in development, so that you don’t need to restart the app to see changes

#t

 serve-static

Whether application should serve static resources from ./static folder. Use #f if you have some other web server (eg nginx in front) serving the content.

#t

 spec-index

Index of definitions to load (see Identifiers definitions section)

 "./types/index.scm"

 filterset-index

Index of filtersets to load (see Filterset definitions section)

 "filters/index.scm"

 sqlite-data

File to use for sqlite data

 "sqlitedb"

 enable-user-settings

Allow user to save certain settings in cookies and show settings page

#t

 downloads

Location of downloads config file, listing available CLI app downloads. The file should be a list, where each item is an alist with symbol keys name, checksum, and url and string values. It’s not an error if file doesn’t exist.

 "config/downloads.scm"

Identifiers definitions

Identifiers definitions are grouped into files (mostly one for each library), and a single root index file which defines names and paths to those files. The index file is specified in configuration under spec-index property, with a default value "./types/index.scm".

Index should be a list of pairs - where car is a symbolic library name, and cdr is associated information regarding definitions to load from. cdr may be an alist, with following fields:

'file - path from where to load definitions.
'exclude - a list of definitions to ignore from the file.

Alternatively, cdr might be a string, in which case it’s treated as a file path.

(
    ((scheme base) . "types/scheme.base.scm")
    ((scheme r5rs) . ((file . "types/r5rs.scm")
                      (exclude . (transcript-on transcript-off))))
)

Each definitions file is a list of entries, where each entry is an association list, using symbols as keys. An entry may be defining a function, a syntax macro, or a plain value.

key

description

name

The name of the function / syntax / value. The name doesn’t have to be unique - in particular with procedures, definition is repeated for all possible invocation overloads with different parameter counts. Required.

signature

S-expr defining signature. The format depends on a type (function / syntax / value) being described.

Function

In case of function, the format is either (lambda (<parameter> …) <return type>) or (case-lambda ((<parameter> …) <return type>) …).

Each parameter is either of the form (<type> param-name) or just param-name, when type is "any". Each parameter may be succeeded by a … literal to indicate varargs / repetition.

Parameter type is either:

a predicate name (eg. list?);
a type union in the form of (or <type> …) (eg. (or list? number?)). When using or, #f might be used as a type inside of it due to its common appearance as a "null" value.

Return type can be same as parameter type described above, plus:

undefined, indicating a value that shouldn’t be used in portable code (eg. result of for-each);
*, indicating a useful value but of unknown type.
(values <type> …) form when the function returns more than one value.

Syntax

In case of syntax, the format is (syntax-rules (<keyword> …) (<pattern> <optional-return-type>) …)

Pattern should be: symbol, ., …, or grouping of patterns inside (). If optional return type is provided, it describes the resultant type when syntax is used as an expression. As a hatch to rendering outside of parens, use (_append ) form, which concatenates its arguments.

Value

In case of simple value, the format is (value <type>)

where type is same as described under functions.

tags

A list of symbols. Tags don’t have inherent special treatment (ie., definition can make use of any tag it wants). Currently used tags are described in Tags

subsigs

Additional elaboration of signatures for the parameters / returns / syntax fragments / datastructure content. Each entry in this list is a list, where first element is a name (corresponding to parameter / syntax fragment / etc), and second parameter is its signature. The signature can be either one of as described in signature section, as well as additionally:

(pattern <pattern> …) - describes subsyntax pattern, expanding template variables used in main signature of the syntax;
(alist <car> <cdr>) - describes an alist datastrucutre, where <car> and <cdr> has same format as function parameter definition;
(list <element>) - describes an list datastrucutre, where <element> has same format as function parameter definition;
(vector <element>) - describes an list datastrucutre, where <element> has same format as function parameter definition;

syntax-param-signatures

Applicable only to syntax type. The value is a list, where each element is itself a 2 element list. First element is a symbol, matching one of the symbols in the signature. Second argument is parameter type definition, as described under functions. This allows annotating syntax when it expects to receive values of certain types for specific places in the pattern.

parameterized-by

List of strings, denoting parameters created by make-parameter that influence behavior of this entry

spec-values

Sometimes procedures have very limited input set for specific parameter (eg. using symbols as a union). Sometimes procedures handle values with specific logic when it detects specific format in it (eg. printf format specifier). In both cases it’s sensible to list these special handlings, and this is what this field is for. The value of spec-values is an alist, where car corresponds to parameter name, and cdr is a list. Each element in said list is also a list - where first element corresponds to specific value (written as a string), and cadr corresponds to the value’s description (also a string).

Filterset definitions

Filterset is a coarse filter, and defines which libraries to include (and optionally how to rename them). Filterset definitions are structured into files one for each filterset option, and a single root index file which defines names and paths to those files.

The index file is specified in configuration under filterset-index property, with a default value "./filters/index.scm".

Index should be a list of alists. Each alist contains three fields:

code - used in links;
name - displayed text;
file - associated filterset definition file.

Filterset definition file should be a list of pairs. car is a symbolic library name (matching what is defined in index for identifiers definitions). cdr is either a symbolic library name (in which case, all the usages of library in car are renamed to what is in cdr), or a #t value (indicating to include library in car as is, without a rename).

Logging

Scheme index uses logback for logging. By default (as defined in src/main/resources/logback.xml) it only does rolling file logging into ./logs directory, and not into standard output. You can provide custom logging configuration by running

java -Dlogback.configurationFile=/path/to/config.xml -jar scheme-index.jar

Consult logback documentation for details.

REST API

All of the following endpoints accept wt query parameter. If the parameter’s value is sexpr, the results are returned as if with write, using scheme-json convention as defined in srfi 180. Otherwise, results are returned as json.

`/rest/filterset`

List of filtersets as strings. For all endpoints below, {filterset name} must correspond to one of the values returned here.

JSON schema

{
    "type": "array",
    "items": {
        "type": "string"
    }
}

`/rest/filterset/{filterset name}/libs`

Libraries found in index as strings.

JSON schema

{
    "type": "array",
    "items": {
        "type": "string"
    }
}

`/rest/filterset/{filterset name}/tags`

Tags found in index as strings.

JSON schema

{
    "type": "array",
    "items": {
        "type": "string"
    }
}

`/rest/filterset/{filterset name}/returns`

Types, which were used as a return type, found in index as strings.

JSON schema

{
    "type": "array",
    "items": {
        "type": "string"
    }
}

`/rest/filterset/{filterset name}/params`

Types, which were used as a parameter type, found in index as strings.

JSON schema

{
    "type": "array",
    "items": {
        "type": "string"
    }
}

`/rest/filterset/{filterset name}/parameterized`

Dynamic parameters (ie., ones created with make-parameter) that affect procedures in the index.

JSON schema

{
    "type": "array",
    "items": {
        "type": "string"
    }
}

`/rest/filterset/{filterset name}/search`

Returns found identifiers and faceting meta data.

Query parameters

 query

text search parameter. Query is parsed using edismax, and therefore some text tokens have special meaning

 start

result offset (pagination).

 rows

size of returned result list. Defaults to 40 if unspecified.

lib

library filter. Possible values returned in /rest/libs. The parameter can appear multiple times, and the result will include procedures from any of given libraries.

tag

tag filter. Possible values returned in /rest/tags. The parameter can appear multiple times, and the result will include procedures which contain all given tags.

 param

param type filter. Possible values returned in /rest/params. The parameter can appear multiple times, and the result will include procedures which contain all given parameter types.

 filter_loose

whether enable loose filtering as described in Filtering logic

 return

return type filter. Possible values returned in /rest/returns. The parameter can appear multiple times, and the result will include procedures which contain all given return types.

 parameterized

return parameterization filter. Possible values returned in /rest/parameterized. The parameter can appear multiple times, and the result will include procedures which are parameterized by all values.

 facet

whether to return facet information of the query. Defaults to "true"

JSON schema

{
    "type":"object",
    "properties":{
        "total":{
            "type":"integer"
        },
        "libs":{
            "$ref":"#/$defs/facet"
        },
        "params":{
            "$ref":"#/$defs/facet"
        },
        "returns":{
            "$ref":"#/$defs/facet"
        },
        "tags":{
            "$ref":"#/$defs/facet"
        },
        "parameterized":{
            "$ref":"#/$defs/facet"
        },
        "items":{
            "type":"array",
            "items":{
                "$ref":"#/$defs/search-item"
            }
        }
    },
    "$defs":{
        "facet":{
            "type":"array",
            "items":{
                "$ref":"#/$defs/facet-value"
            }
        },
        "facet-value":{
            "type":"object",
            "properties":{
                "value":{
                    "type":"string"
                },
                "count":{
                    "type":"integer"
                }
            }
        },
        "spec-value-block":{
            "type":"object",
            "properties":{
                "field":{
                    "type":"string"
                },
                "values":{
                    "type":"array",
                    "items":{
                        "type":"object",
                        "properties":{
                            "value":{
                                "type":"string"
                            },
                            "desc":{
                                "type":"string"
                            }
                        }
                    }
                }
            }
        },
        "param":{
            "type":"object",
            "properties":{
                "name":{
                    "type":"string"
                },
                "types":{
                    "type":"array",
                    "items":{
                        "type":"string"
                    }
                }
            }
        },
        "return":{
            "type":"object",
            "properties":{
                "kind":{
                    "type":"string",
                    "pattern":"^(or)|(values)|(return)$"
                },
                "items":{
                    "type":"array",
                    "items":{
                        "$ref":"#/$defs/return"
                    }
                },
                "type":{
                    "type":"string"
                }
            }
        },
        "func-signature":{
            "type":"object",
            "properties":{
                "type":{
                    "type":"string",
                    "pattern":"^function$"
                },
                "variants":{
                    "type":"array",
                    "items":{
                        "type":"object",
                        "properties":{
                            "params":{
                                "type":"array",
                                "items":{
                                    "$ref":"#/$defs/param"
                                }
                            },
                            "return":{
                                "$ref":"#/$defs/return"
                            }
                        }
                    }
                }
            }
        },
        "syntax-signature":{
            "type":"object",
            "properties":{
                "type":{
                    "type":"string",
                    "pattern":"^syntax$"
                },
                "literals":{
                    "type":"array",
                    "items":{
                        "type":"string"
                    }
                },
                "patterns":{
                    "type":"array",
                    "items":{
                        "type":"object",
                        "properties":{
                            "pattern":{
                                "type":"string"
                            },
                            "type":{
                                "types":[
                                    "string",
                                    "null"
                                ]
                            }
                        }
                    }
                }
            }
        },
        "value-signature":{
            "type":"object",
            "properties":{
                "type":{
                    "type":"string",
                    "pattern":"^value$"
                },
                "value":{
                    "$ref":"#/$defs/return"
                }
            }
        },
        "pattern-signature":{
            "type":"object",
            "properties":{
                "type":{
                    "type":"string",
                    "pattern":"^pattern$"
                },
                "patterns":{
                    "type":"array",
                    "items":{
                        "type":"string"
                    }
                }
            }
        },
        "alist-signature":{
            "type":"object",
            "properties":{
                "type":{
                    "type":"string",
                    "pattern":"^alist$"
                },
                "car":{
                    "$ref":"#/$defs/param"
                },
                "cdr":{
                    "$ref":"#/$defs/param"
                }
            }
        },
        "list-signature":{
            "type":"object",
            "properties":{
                "type":{
                    "type":"string",
                    "pattern":"^list$"
                },
                "element":{
                    "$ref":"#/$defs/param"
                }
            }
        },
        "vector-signature":{
            "type":"object",
            "properties":{
                "type":{
                    "type":"string",
                    "pattern":"^vector$"
                },
                "element":{
                    "$ref":"#/$defs/param"
                }
            }
        },
        "signature": {
            "oneOf": [{
                "$ref":"#/$defs/func-signature"
            },{
                "$ref":"#/$defs/syntax-signature"
            },{
                "$ref":"#/$defs/value-signature"
            }, {
                "$ref":"#/$defs/pattern-signature"
            }, {
                "$ref":"#/$defs/alist-signature"
            }, {
                "$ref":"#/$defs/list-signature"
            }, {
                "$ref":"#/$defs/vector-signature"
            }]
        },
        "search-item":{
            "type":"object",
            "properties":{
                "lib":{
                    "type":"string"
                },
                "name":{
                    "type":"string"
                },
                "description":{
                    "type":"string"
                },
                "type":{
                    "type":"string"
                },
                "signature": {
                    "$ref":"#/$defs/signature"
                },
                "subsignatures":{
                    "type":"array",
                    "items":{
                        "type":"object",
                        "properties":{
                            "name":{
                                "type":"string"
                            },
                            "signature":{
                                "$ref":"#/$defs/signature"
                            }
                        }
                    }
                },
                "tags":{
                    "type":"array",
                    "items":{
                        "type":"string"
                    }
                },
                "param_types":{
                    "type":"array",
                    "items":{
                        "type":"string"
                    }
                },
                "return_types":{
                    "type":"array",
                    "items":{
                        "type":"string"
                    }
                },
                "super_types":{
                    "type":"array",
                    "items":{
                        "type":"string"
                    }
                },
                "parameterized_by":{
                    "type":"array",
                    "items":{
                        "type":"string"
                    }
                },
                "spec_values":{
                    "type":"array",
                    "items":{
                        "$ref":"#/$defs/spec-value-block"
                    }
                }
            }
        }
    }
}