Development Guide

Workflow

App

Routing

Loader

Logging

Edit on GitHub

Loader

view.py's route loader works by generating a list of Route objects, and then passing them to finalize().

load_fs

Filesystem loading implementation. Similiar to NextJS's routing system. You take target_dir and search it, if a file is found and not prefixed with _, then convert the directory structure to a path. For example, target_dir/hello/world/index.py would be converted to a route for /hello/world

Parameters:

Name	Type	Description	Default
`app`	`App`	App to attach routes to.	required
`target_dir`	`Path`	Directory to search for routes.	required

Source code in src/view/_loader.py

def load_fs(app: ViewApp, target_dir: Path) -> None:
    """Filesystem loading implementation.
    Similiar to NextJS's routing system. You take `target_dir` and search it,
    if a file is found and not prefixed with _, then convert the directory structure
    to a path. For example, target_dir/hello/world/index.py would be converted to a
    route for /hello/world

    Args:
        app: App to attach routes to.
        target_dir: Directory to search for routes.
    """
    Internal.info("loading using filesystem")
    Internal.debug(f"loading {app}")

    routes: list[Route] = []

    for root, _, files in os.walk(target_dir):
        for f in files:
            if f.startswith("_"):
                continue

            path = os.path.join(root, f)
            Internal.info(f"loading: {path}")
            mod = runpy.run_path(path)
            current_routes: list[Route] = []

            for i in mod.values():
                if isinstance(i, Route):
                    if i.method in [x.method for x in current_routes]:
                        warnings.warn(
                            "same method used twice during filesystem loading",
                            LoaderWarning,
                        )
                    current_routes.append(i)

            if not current_routes:
                raise ValueError(f"{path} has no set routes")

            for x in current_routes:
                if x.path:
                    warnings.warn(
                        f"path was passed for {x} when filesystem loading is enabled"  # noqa
                    )
                else:
                    path_obj = Path(path)
                    stripped = list(
                        path_obj.parts[len(target_dir.parts) :]
                    )  # noqa
                    if stripped[-1] == "index.py":
                        stripped.pop(len(stripped) - 1)

                    stripped_obj = Path(*stripped)
                    stripped_path = str(stripped_obj).rsplit(
                        ".",
                        maxsplit=1,
                    )[0]
                    x.path = "/" + stripped_path

            for x in current_routes:
                routes.append(x)

    finalize(routes, app)

load_simple

Simple loading implementation. Simple loading is essentially searching a directory recursively for files, and then extracting Route instances from each file.

If a file is prefixed with _, it will not be loaded.

Parameters:

Name	Type	Description	Default
`app`	`App`	App to attach routes to.	required
`target_dir`	`Path`	Directory to search for routes.	required

Source code in src/view/_loader.py

def load_simple(app: ViewApp, target_dir: Path) -> None:
    """Simple loading implementation.
    Simple loading is essentially searching a directory recursively
    for files, and then extracting Route instances from each file.

    If a file is prefixed with _, it will not be loaded.

    Args:
        app: App to attach routes to.
        target_dir: Directory to search for routes.

    """
    Internal.info("loading using simple strategy")
    routes: list[Route] = []

    for root, _, files in os.walk(target_dir):
        for f in files:
            if f.startswith("_"):
                continue

            path = os.path.join(root, f)
            Internal.info(f"loading: {path}")
            mod = runpy.run_path(path)
            mini_routes: list[Route] = []

            for i in mod.values():
                if isinstance(i, Route):
                    mini_routes.append(i)

            for route in mini_routes:
                if not route.path:
                    raise ValueError(
                        "omitting path is only supported"
                        " on filesystem loading",
                    )

                routes.append(route)

    finalize(routes, app)

Finalizing

This function call method functions on the App instance. For example, a Route object generated by @get() will correspond to _get on App (technically, it originates from the ViewApp class, as it's a C function). It will also call _format_inputs, which generates dictionaries that the C loader can understand at runtime.

If a route has inputs that do not have an Any type (e.g. in @app.query("hello", str) the type is str), then it will start a complicated process called type code generation.

finalize

Attach list of routes to an app and validate all parameters.

Parameters:

Name	Type	Description	Default
`routes`	`list[Route]`	List of routes.	required
`app`	`App`	App to attach to.	required

Source code in src/view/_loader.py

def finalize(routes: list[Route], app: ViewApp):
    """Attach list of routes to an app and validate all parameters.

    Args:
        routes: List of routes.
        app: App to attach to.
    """
    virtual_routes: dict[str, list[Route]] = {}

    targets = {
        Method.GET: app._get,
        Method.PATCH: app._post,
        Method.PUT: app._put,
        Method.PATCH: app._patch,
        Method.DELETE: app._delete,
        Method.OPTIONS: app._options,
    }

    for route in routes:
        set_load(route)
        target = targets[route.method]

        if (not route.path) and (not route.parts):
            raise TypeError("route did not specify a path")
        lst = virtual_routes.get(route.path or "")

        if lst:
            if route.method in [i.method for i in lst]:
                raise ValueError(
                    f"duplicate route: {route.method.name} for {route.path}",
                )
            lst.append(route)
        else:
            virtual_routes[route.path or ""] = [route]

        app.loaded_routes.append(route)
        target(
            route.path,  # type: ignore
            route.func,
            route.cache_rate,
            _format_inputs(route.inputs),
            route.errors or {},
            route.parts,  # type: ignore
        )

_format_inputs

Convert a list of route inputs to a proper dictionary that the C loader can handle. This function also will generate the typecodes for the input.

Source code in src/view/_loader.py

def _format_inputs(inputs: list[RouteInput]) -> list[RouteInputDict]:
    """Convert a list of route inputs to a proper dictionary that the C loader can handle.
    This function also will generate the typecodes for the input."""
    result: list[RouteInputDict] = []

    for i in inputs:
        type_codes = _build_type_codes(i.tp)
        result.append(
            {
                "name": i.name,
                "type_codes": type_codes,
                "default": i.default,  # type: ignore
                "validators": i.validators,
                "is_body": i.is_body,
                "has_default": i.default is not _NoDefault,
            }
        )

    return result

Type Codes

Type codes are easily the most complex part of the loader. It's essentially the converting of types to a tuple that the C ASGI implementation can quickly validate at runtime. The type code system exists to speed up type validation. At runtime, calling lots of PyObject_IsInstance functions on the C side can be expensive. view.py solves this by creating a structure in which every type supported has its own type code and information on how to parse it at runtime.

The main entry point for type code generation is _build_type_codes:

Generate types from a list of types.

Parameters:

Name	Type	Description	Default
`inp`	`Iterable[type[ValueType]]`	Iterable containing each type.	required
`doc`	`dict[Any, LoaderDoc] \| None`	Auto-doc dictionary when a docstring is extracted.	`None`
`key_name`	`str \| None`	Name of the current key. Only needed for auto-doc purposes.	`None`
`default`	`Any \| _NoDefault`	Default value. Only needed for auto-doc purposes.	`_NotSet`

Source code in src/view/_loader.py

def _build_type_codes(
    inp: Iterable[type[ValueType]],
    doc: dict[Any, LoaderDoc] | None = None,
    *,
    key_name: str | None = None,
    default: Any | _NoDefault = _NotSet,
) -> list[TypeInfo]:
    """Generate types from a list of types.

    Args:
        inp: Iterable containing each type.
        doc: Auto-doc dictionary when a docstring is extracted.
        key_name: Name of the current key. Only needed for auto-doc purposes.
        default: Default value. Only needed for auto-doc purposes."""
    if not inp:
        return []

    codes: list[TypeInfo] = []

    for tp in inp:
        if is_annotated(tp):
            if doc is None:
                raise TypeError(f"Annotated is not valid here ({tp})")

            if not key_name:
                raise RuntimeError("internal error: key_name is None")

            if default is _NotSet:
                raise RuntimeError("internal error: default is _NotSet")

            tmp = tp.__origin__
            doc[key_name] = LoaderDoc(tp.__metadata__[0], tmp, default)
            tp = tmp
        elif doc is not None:
            if not key_name:
                raise RuntimeError("internal error: key_name is None")

            if default is _NotSet:
                raise RuntimeError("internal error: default is _NotSet")

            doc[key_name] = LoaderDoc("No description provided.", tp, default)

        type_code = _BASIC_CODES.get(tp)

        if type_code:
            codes.append((type_code, None, []))
            continue

        if (TypedDict in getattr(tp, "__orig_bases__", [])) or (
            type(tp) == _TypedDictMeta
        ):
            try:
                body = get_type_hints(tp)
            except KeyError:
                body = tp.__annotations__

            opt = getattr(tp, "__optional_keys__", None)

            class _Transport:
                @staticmethod
                def __view_construct__(**kwargs):
                    return kwargs

            doc = {}
            codes.append(
                (
                    TYPECODE_CLASS,
                    _Transport,
                    _format_body(body, doc, tp, not_required=opt),
                ),
            )
            setattr(tp, "_view_doc", doc)
            continue

        if (NamedTuple in getattr(tp, "__orig_bases__", [])) or (
            hasattr(tp, "_field_defaults")
        ):
            defaults = tp._field_defaults  # type: ignore
            tps = {}
            try:
                hints = get_type_hints(tp)
            except KeyError:
                hints = getattr(tp, "_field_types", tp.__annotations__)

            for k, v in hints.items():
                if k in defaults:
                    tps[k] = BodyParam(v, defaults[k])
                else:
                    tps[k] = v

            doc = {}
            codes.append((TYPECODE_CLASS, tp, _format_body(tps, doc, tp)))
            setattr(tp, "_view_doc", doc)
            continue

        dataclass_fields: dict[str, Field] | None = getattr(
            tp, "__dataclass_fields__", None
        )

        if dataclass_fields:
            tps = {}
            for k, v in dataclass_fields.items():
                if isinstance(v.default, _MISSING_TYPE) and (
                    isinstance(v.default_factory, _MISSING_TYPE)
                ):
                    tps[k] = v.type
                else:
                    default = (
                        v.default
                        if not isinstance(v.default, _MISSING_TYPE)
                        else v.default_factory
                    )
                    tps[k] = BodyParam(v.type, default)

            doc = {}
            codes.append((TYPECODE_CLASS, tp, _format_body(tps, doc, tp)))
            setattr(tp, "_view_doc", doc)
            continue

        pydantic_fields: dict[str, ModelField] | None = getattr(
            tp, "__fields__", None
        ) or getattr(tp, "model_fields", None)
        if pydantic_fields:
            tps = {}

            for k, v in pydantic_fields.items():
                if (not v.default) and (not v.default_factory):
                    tps[k] = v.outer_type_
                else:
                    tps[k] = BodyParam(
                        v.outer_type_,
                        v.default or v.default_factory,
                    )

            doc = {}
            codes.append((TYPECODE_CLASS, tp, _format_body(tps, doc, tp)))
            setattr(tp, "_view_doc", doc)
            continue

        vbody = getattr(tp, "__view_body__", None)
        if vbody:
            if callable(vbody):
                vbody_types = vbody()
            else:
                vbody_types = vbody

            doc = {}
            codes.append(
                (TYPECODE_CLASS, tp, _format_body(vbody_types, doc, tp))
            )
            setattr(tp, "_view_doc", doc)
            continue

        origin = getattr(tp, "__origin__", None)  # typing.GenericAlias

        if (type(tp) in {UnionType, TypingUnionType}) and (origin is not dict):
            new_codes = _build_type_codes(get_args(tp))
            codes.extend(new_codes)
            continue

        if origin is not dict:
            raise InvalidBodyError(f"{tp} is not a valid type for routes")

        key, value = get_args(tp)

        if key is not str:
            raise InvalidBodyError(
                f"dictionary keys must be strings, not {key}"
            )

        value_args = get_args(value)

        if not len(value_args):
            value_args = (value,)

        tp_codes = _build_type_codes(value_args)
        codes.append((TYPECODE_DICT, None, tp_codes))

    return codes

On the Python side, type info should contain four things: - Type Code - Type Object (only set when using a __view_body__ object) - Children (i.e. the int part of dict[str, int]) - Default (only set when typecode is TYPECODE_CLASSTYPES)

More on what these mean in a second, but for reference here are the available typecodes:

TYPECODE_ANY (0): Any type may be passed, no further validation is needed.
TYPECODE_STR (1): A string object. Validation will never fail if this is the type, since everything can be stringified.
TYPECODE_INT (2): Any number.
TYPECODE_BOOL (3): Any boolean.
TYPECODE_FLOAT (4): Any number or floating point.
TYPECODE_DICT (5): Any dictionary object. If this has children, they will be validated. Note that dictionary keys can only be strings.
TYPECODE_NONE (6): None or null is allowed.
TYPECODE_CLASS (7): This is a __view_body__ object. The second part should contain a Python class, and the children will be typecodes of TYPECODE_CLASSTYPES.
TYPECODE_CLASSTYPES (8): This is reserved for children under a TYPECODE_CLASS type code. If used otherwise, view.py will crash.

So, how does it work?

To start, a type info list represents all the types available for a certain input. So, if it has one type, only one type is supported (assuming that type is not Any, since that can be applied to any type). For example, at the top level a type info list that supports str | int | bool looks like the following:

[str_type_info, int_type_info, bool_type_info]`

But what does a type info part actually look like? Unless the typecode is TYPECODE_CLASSTYPES, it's simply a tuple containing three items:

(typecode_number, typecode_class_object, list_of_children_typecodes)

Above, typecode_number is one of the typecodes specified above. typecode_class_object is the Python class to instantiate at runtime if the typecode is TYPECODE_CLASS. If the typecode is something else, then this is None. Finally, list_of_children_typecodes is for TYPECODE_CLASS and TYPECODE_DICT. It's a list containing type code parts for use at runtime. This will be explained more in depth later.

So, as an example, the type info for str looks like this:

(TYPECODE_STR, None, [])

And then when put into the entire list from earlier, it looks like this:

[(TYPECODE_STR, None, []), (TYPECODE_INT, None, []), (TYPECODE_BOOL, None, [])]

Now, for those other parameters, let's start with dictionaries. JSON is taken in by queries or HTTP bodies, and in JSON keys can only be strings. So, we already know the first part of the type: dict[str], meaning we don't have to pass any type codes for it since it will always be str.

But how do we specify types for the second parameter? That's where the children come in. Let's use the type dict[str, int] as an example. Once again, we start with a simple typecode part for dictionaries:

(TYPECODE_DICT, None, [])

The above is actually a valid type part, as [] is just read as Any by view.py, so the above is actually the type info for dict[str, Any]. To add a type, we just add the type part to the children part of it, like so:

(TYPECODE_DICT, None, [(TYPECODE_INT, None, [])])`

Easy as that! The above is now valid type info for dict[str, int]. But what if we want to add unions (i.e. more types)? Just add more type parts to the list, like we did earlier:

(TYPECODE_DICT, None, [(TYPECODE_INT, None, []), (TYPECODE_STR, None, [])])

The above is proper for dict[str, int | str]. Easy enough so far, right? Now it starts to get really complicated.

Let's dive into how TYPECODE_CLASS works. Say we have an object called TC with a __view_body__:

class TC:
    __view_body__ = {
        "a": str,
    }

Ok, let's start small. First, set the type code and object, let's ignore children for now:

(TYPECODE_CLASS, TC, [])

The above is technically valid, but not very useful. It would assume that TC has no parameters. So how do we add those parameters from the __view_body__? This process is called body formatting, and it's main entry point is _format_body:

Generate a type info list from view body types.

Source code in src/view/_loader.py

def _format_body(
    vbody_types: dict,
    doc: dict[Any, LoaderDoc],
    origin: type[Any],
    *,
    not_required: set[str] | None = None,
) -> list[TypeInfo]:
    """Generate a type info list from view body types."""
    not_required = not_required or set()
    if not isinstance(vbody_types, dict):
        raise InvalidBodyError(
            f"__view_body__ should return a dict, not {type(vbody_types)}",  # noqa
        )

    vbody_final = {}
    vbody_defaults = {}

    for k, raw_v in vbody_types.items():
        if not isinstance(k, str):
            raise InvalidBodyError(
                f"all keys returned by __view_body__ should be strings, not {type(k)}"  # noqa
            )

        default = _NoDefault
        v = raw_v.types if isinstance(raw_v, BodyParam) else raw_v

        if isinstance(v, str):
            scope = getattr(origin, "_view_scope", globals())
            v = _eval_type(ForwardRef(v), scope, scope)

        if isinstance(raw_v, BodyParam):
            default = raw_v.default

        if (getattr(raw_v, "__origin__", None) in _NOT_REQUIRED_TYPES) or (
            k in not_required
        ):
            v = get_args(raw_v)
            default = _ViewNotRequired
        iter_v = v if isinstance(v, (tuple, list)) else (v,)
        vbody_final[k] = _build_type_codes(
            iter_v,
            doc,
            key_name=k,
            default=default,
        )
        vbody_defaults[k] = default

    return [
        (TYPECODE_CLASSTYPES, k, v, vbody_defaults[k])
        for k, v in vbody_final.items()
    ]

But, how does that actually work? Let's take a look at children again:

[]

This is simply a list that should contain other type codes, so all we have to do is add some to it. But how do we specify an attribute name?

This is where TYPECODE_CLASSTYPES comes in. TYPECODE_CLASSTYPES breaks the rules, and can only exist in the children of a TYPECODE_CLASS. It expects a tuple containing four items:

(TYPECODE_CLASSTYPES, attribute_name_as_str, allowed_typeinfo, default_value)

attribute_name_as_str is a string containing the attribute name, so for a: str it would be the string "a". allowed_typeinfo works the same as children. It's a list of type parts that say what types are allowed. default_value is the default value in case it wasn't passed with the JSON. If no default is wanted, use the _NoDefault object.

Let's build a TYPECODE_CLASSTYPES for a: str:

(TYPECODE_CLASSTYPES, "a", [(TYPECODE_STR, None, [])], _NoDefault)

(TYPECODE_CLASS, TC, [])

Easy enough, right? Now, let's bring it back to the original TYPECODE_CLASS:

(TYPECODE_CLASS, TC, [(TYPECODE_CLASSTYPES, "a", [(TYPECODE_STR, None, [])], _NoDefault)])

Once again, we go through all this work because trying to mash lots of this information together at runtime is error prone and expensive. The type code system speeds things up by a lot.

Congratulations! You now understand one of the most complicated systems of view.py

Routing

Logging

Loader

load_fs

load_simple

Finalizing

finalize

_format_inputs

Type Codes

_build_type_codes

_format_body