Packaging an External Library#

The Defining a Library guide shows how to subclass Design, StdCellLibrary, and PDK to describe a piece of IP. That is enough when the library class lives alongside the design that uses it. Once you want to share a library, across projects, across a team, or as a versioned deliverable, you package it as a standalone, pip-installable Python distribution.

This guide covers that packaging layer: the project layout, the pyproject.toml, and how bundled data files are located no matter where the package is installed. A single package can bundle libraries, PDKs, flows, and even custom tool drivers, and expose them through a target. For the details of the individual module classes, this guide links back to libraries, pdks, flows, tools, and targets. Throughout, the package is given the generic name mylib so you can substitute your own.

Why package a library#

Distributing a library as its own package buys you:

Reuse — any project can pip install mylib and import it, instead of copying files into each design.
Versioning — the package carries a version, so a design can pin the exact library revision it was signed off against.
Data locality — data files (LEF, GDS, .lib, …) travel with the package and resolve automatically, whether the package is installed from PyPI or in editable mode from a local checkout.

Package layout#

A library package is an ordinary Python distribution with an importable module at its root. A typical layout mirrors the categories of module you provide:

mylib/                    # repository root
├── pyproject.toml        # build + packaging configuration
├── MANIFEST.in           # controls which files land in the sdist
├── README.md
├── LICENSE
└── mylib/                # the importable Python package
    ├── __init__.py       # package init (optionally a shared dataroot mixin)
    ├── _version.py       # auto-generated by setuptools_scm
    ├── libs/             # standard-cell / hard IP (StdCellLibrary subclasses)
    ├── pdks/             # PDK definitions (siliconcompiler.PDK subclasses)
    ├── flows/            # custom Flowgraph definitions
    ├── tools/            # custom tool/task drivers
    └── targets/          # targets bundling any of the above

Every module subpackage is optional — include only what you ship. A package might provide just a library, just a PDK, just a flow or tool driver, or any combination, and expose it through a target.

Small data files (a hand-edited LEF, a black-box Verilog stub) can be committed directly under the relevant subpackage, e.g. mylib/libs/mymacro/mymacro.lef. Large or proprietary data (full foundry decks) is usually kept outside the package and referenced through an environment variable, see Two dataroot strategies.

The `pyproject.toml`#

The build configuration does two jobs: build the package and derive its version.

[build-system]
requires = ["setuptools >= 80", "setuptools_scm[toml] >= 8"]
build-backend = "setuptools.build_meta"

# Derive the version from git tags and write it into the package.
[tool.setuptools_scm]
version_file = "mylib/_version.py"

[project]
name = "mylib"
description = "Example external library for SiliconCompiler."
readme = "README.md"
requires-python = ">= 3.10"
# For proprietary IP, use a license reference instead of an SPDX id, e.g.
# license = "LicenseRef-MyCompany-Proprietary"
license = "Apache-2.0"
license-files = ["LICENSE"]
dynamic = ["version"]
dependencies = [
    "siliconcompiler == 0.38.10",
    # Depend on other libraries you build on, e.g.:
    # "lambdapdk",
]

[tool.setuptools]
include-package-data = true

Points worth calling out:

Versioning — setuptools_scm derives the package version from git tags and writes it into mylib/_version.py on every build, so each release is stamped with a version that designs can pin against. Because that file is generated, add it to .gitignore rather than committing it:
```
# .gitignore
mylib/_version.py
```
``include-package-data = true`` — together with MANIFEST.in, this ensures committed data files ship inside the wheel. Prune what you do not need (tests, examples) and explicitly include your package sources and data:
```
# MANIFEST.in
prune tests
prune examples
recursive-include mylib *.py
recursive-include mylib *.lef *.gds *.v
```
Pinning SiliconCompiler — pin siliconcompiler in dependencies to the exact release you develop against using an == constraint; the current release is 0.38.10.

Note

More advanced packages can also register SiliconCompiler entry points to plug into documentation source links, show/open tasks, data source resolvers, and tool-install helpers, plus [project.scripts] for CLI apps. SiliconCompiler registers its own built-ins in code before consulting these groups, so an entry point you provide takes precedence wherever the two overlap.

# pyproject.toml
[project.entry-points."siliconcompiler.showtask"]
mylib = "mylib.showtools:showtasks"

[project.entry-points."siliconcompiler.path_resolver"]
myscheme = "mylib.resolver:get_resolver"

[project.entry-points."siliconcompiler.docs"]
linkcode = "mylib.docs:get_codeurl"

[project.entry-points."siliconcompiler.install"]
toolscripts = "mylib.install:get_toolscripts_dir"
groups = "mylib.install:get_install_groups"
fingerprint = "mylib.install:compute_fingerprint"

The names in the siliconcompiler.docs and siliconcompiler.install groups are fixed, since SiliconCompiler looks them up by name. Adding tools to sc-install covers toolscripts. Names in the siliconcompiler.showtask and siliconcompiler.path_resolver groups are free-form; pick something unique to your package.

Locating data files#

For SiliconCompiler to find a module’s data files no matter where the package is installed, each module registers a dataroot. The simplest approach anchors the dataroot to the module’s own source file: passing __file__ to set_dataroot roots the data at the directory containing that module, so files are added relative to it.

self.set_dataroot(f"mylib-{self.name}", __file__)

Giving the dataroot a name unique to the module (here mylib-<name>) keeps it from clashing when several libraries are combined into one project.

Note

If you would rather share a single dataroot across the whole package — for example to reference files that live in sibling subpackages — define a small mixin that every module inherits and point it at the installed package:

# mylib/__init__.py
from siliconcompiler.schema_support.pathschema import PathSchema


class _MyLibPath(PathSchema):
    def __init__(self):
        super().__init__()
        self.set_dataroot("mylib", "python://mylib")

The python://mylib dataroot resolves to the installed location of the mylib package, and python://mylib/libs/mymacro to a directory inside it. To additionally fetch large data files that are not shipped in the wheel, give the dataroot a git fallback keyed to the package version with PythonPathResolver.set_dataroot (see python://).

Two dataroot strategies#

Real libraries commonly mix two kinds of dataroot:

Packaged data — the per-module dataroot rooted at __file__ above, for files committed to and shipped with the package. Use this for small files (edited LEFs, black-box stubs, RTL stubs).
External data — foundry decks are often too large or too restricted to commit. Point a second dataroot at an environment variable so users supply the data out-of-band:
```
self.set_dataroot("foundry", "$FOUNDRY_ROOT/tech/mylib/v1p0")
```
Files added under this dataroot resolve against $FOUNDRY_ROOT at run time. Whenever a module relies on an environment variable like this, set a default for it in the target so users are not left to configure it by hand.

Defining modules inside the package#

Libraries, PDKs, flows, and tool drivers are defined exactly as in the libraries, pdks, flows, and tools guides. The only packaging addition is that any module shipping data files sets a dataroot from __file__ (and typically mixes in CachedSchema) so those files travel with the package. A hard-IP library looks like:

# mylib/libs/mymacro/__init__.py
from siliconcompiler import StdCellLibrary
from siliconcompiler.schema import CachedSchema
from siliconcompiler.tools.openroad import OpenROADStdCellLibrary
from siliconcompiler.tools.klayout import KLayoutLibrary

from mylib.pdks import mypdk


class MyMacro(OpenROADStdCellLibrary, KLayoutLibrary,
              StdCellLibrary, CachedSchema):
    def __init__(self):
        super().__init__()
        self.set_name("mymacro")
        self.package.set_vendor("MyVendor")
        self.package.set_version("v1p0")

        # Root data files at this module's own directory.
        self.set_dataroot(f"mylib-{self.name}", __file__)

        # Associate the PDK this IP targets.
        self.add_asic_pdk(mypdk.MyPDK())

        # Files committed next to this module, added relative to its directory.
        with self.active_dataroot(f"mylib-{self.name}"):
            with self.active_fileset("models.physical"):
                self.add_file("mymacro.lef")
                self.add_file("mymacro.gds")
                self.add_asic_aprfileset()

        # Timing models come from an external dataroot. Keep each format in
        # its own fileset, tagged by corner and delay model with
        # add_asic_libcornerfileset(corner, model). Liberty (.lib) uses a
        # model like "nldm"; a compiled binary timing model (here ".bin")
        # uses the "<model>-bin" variant.
        self.set_dataroot("foundry", "$FOUNDRY_ROOT/tech/mylib/v1p0")
        with self.active_dataroot("foundry"):
            for corner in ("slow", "typical", "fast"):
                with self.active_fileset(f"models.timing.{corner}.nldm"):
                    self.add_file(f"lib/mymacro_{corner}.lib.gz")
                    self.add_asic_libcornerfileset(corner, "nldm")

                # Compiled binary timing model for the same corner.
                with self.active_fileset(f"models.timing.{corner}.nldm-bin"):
                    self.add_file(f"bin/mymacro_{corner}.bin", filetype="bin")
                    self.add_asic_libcornerfileset(corner, "nldm-bin")

The tool mixins (OpenROADStdCellLibrary, KLayoutLibrary, …) attach tool-specific parameters so the library carries everything a flow needs; see Tool Extension Classes for the full list. PDKs follow the same recipe with PDK and the tool *PDK mixins; see pdks.

Note

Base class order matters when building on a bundled base class. Some packages ship a base class that already combines several mixins — lambdapdk’s LambdaLibrary, for example, bundles the Yosys, OpenROAD, KLayout and Bambu library mixins together with StdCellLibrary. Such a base class has to be listed before anything it already inherits, so extend it like this:

from lambdapdk import LambdaLibrary
from mylib.tools.mytool import MyToolStdCellLibrary


# LambdaLibrary first, and no separate StdCellLibrary: it brings its own.
class MyMacro(LambdaLibrary, MyToolStdCellLibrary, CachedSchema):
    ...

Repeating StdCellLibrary in the base list after the bundled class asks Python for an inheritance order that cannot exist, and the class fails to define with TypeError: Cannot create a consistent method resolution order (MRO). If you hit that error, remove the mixins the bundled base class already provides and move it to the front of the list.

A tool driver is a Task subclass. It names its tool and task, declares the executable, and ships any reference scripts alongside the module:

# mylib/tools/mytool/__init__.py
from siliconcompiler import Task


class MyToolTask(Task):
    def tool(self):
        return "mytool"

    def task(self):
        return "convert"

    def parse_version(self, stdout):
        return stdout.strip()

    def setup(self):
        super().setup()
        self.set_exe("mytool", vswitch="--version", format="tcl")
        self.add_version(">=1.0.0")

        # Reference scripts committed next to this module.
        self.set_dataroot("mytool-scripts", __file__)
        with self.active_dataroot("mytool-scripts"):
            self.set_refdir("scripts")
        self.set_script("convert.tcl")

    def runtime_options(self):
        options = super().runtime_options()
        # Each token is its own list item — never "-flag value" as one string.
        options.append("-batch")
        options.extend(["-log", "convert.log"])
        return options

A task with no external executable overrides run() (returning an exit code) instead of declaring an exe; everything else about the class is the same. See tools for the full task API.

A flow is a Flowgraph subclass that wires tasks into a graph of steps with node() and edge() — and it can reference both built-in tasks and the package’s own:

# mylib/flows/myflow.py
from siliconcompiler import Flowgraph
from siliconcompiler.tools.slang.elaborate import Elaborate

from mylib.tools.mytool import MyToolTask


class MyFlow(Flowgraph):
    def __init__(self, name: str = "myflow"):
        super().__init__(name)
        self.node("elaborate", Elaborate())
        self.node("convert", MyToolTask())
        self.edge("elaborate", "convert")

See flows for composing larger graphs.

ASIC timing models and the delay model#

Standard-cell timing ships in more than one format. Liberty (.lib) is read by the open-source tools; some libraries also provide a compiled, binary timing model (here .bin) that a tool can load without parsing Liberty. A library keeps each format in its own fileset, tagged with a corner and a delay model using add_asic_libcornerfileset(corner, model) (see the library example above). By convention the compiled variant of a delay model is named <model>-bin:

Delay model	Fileset	Files
`nldm`	`models.timing.<corner>.nldm`	Liberty (`.lib`)
`nldm-bin`	`models.timing.<corner>.nldm-bin`	compiled (`.bin`)
`ccs`	`models.timing.<corner>.ccs`	CCS Liberty (`.lib`)
`ccs-bin`	`models.timing.<corner>.ccs-bin`	compiled (`.bin`)

A target chooses one with project.set_asic_delaymodel("nldm") (or "nldm-bin" for a flow that reads the compiled files). delaymodel is a free-form string, so <model>-bin works purely by matching the fileset name you registered. Tool drivers resolve the files by looking up (corner, delaymodel); for how a driver consumes these filesets in Python and TCL, see Reading ASIC standard-cell timing.

Bundling into a target#

To give users a one-call setup, provide a target function that assembles the PDK, libraries, and flow — and sets a default for any environment variable the modules depend on:

# mylib/targets/mytarget.py
import os

from siliconcompiler import ASIC

from mylib.pdks import mypdk
from mylib.libs import mymacro
from mylib.flows.myflow import MyFlow


def mytarget(project: ASIC):
    # Default the external dataroot only if the user has not already set it,
    # in the environment or on the project.
    if os.getenv("FOUNDRY_ROOT") is None and \
            project.option.get_env("FOUNDRY_ROOT") is None:
        project.option.set_env("FOUNDRY_ROOT", "/opt/foundry")

    project.set_pdk(mypdk.MyPDK())
    project.set_mainlib(mymacro.MyMacro())
    project.set_flow(MyFlow())

    # Pick the timing view the tools read (see the delay-model section above);
    # use "nldm-bin" instead for a flow that reads the compiled timing model.
    project.set_asic_delaymodel("nldm")

If a module points a dataroot at an environment variable such as $FOUNDRY_ROOT, setting it here with project.option.set_env means users get a working default without configuring anything themselves.

Adding tools to sc-install#

A package that drives a tool SiliconCompiler does not build can make it installable with sc-install. Lay out a directory the way SiliconCompiler lays out its own toolscripts, and return it from a toolscripts entry point:

mylib/toolscripts/
├── _tools.json                  <- pins, in the same fields as SiliconCompiler's
└── ubuntu24/
    └── install-mytool.sh        <- one directory per supported OS

# pyproject.toml
[project.entry-points."siliconcompiler.install"]
toolscripts = "mylib.install:get_toolscripts_dir"

# mylib/install.py
import os.path


def get_toolscripts_dir():
    return os.path.join(os.path.dirname(__file__), "toolscripts")

sc-install runs each script where it is installed, with SC_TOOLSCRIPTS naming a directory that holds the shared helpers, _prereqs.sh and _tools.py, beside a _tools.json that merges your pins into SiliconCompiler’s. An install script finds the helpers there and reads its pins through _tools.py:

#!/bin/bash

set -ex

src_path="${SC_TOOLSCRIPTS:?run this script through sc-install}"

. "${src_path}/_prereqs.sh"
install_prereqs git

git clone $(python3 ${src_path}/_tools.py --tool mytool --field git-url) mytool
cd mytool
git checkout $(python3 ${src_path}/_tools.py --tool mytool --field git-commit)

Commit the script executable (git update-index --chmod=+x), since it is run directly.

Overriding a pin. An entry for a tool SiliconCompiler already pins changes only the fields it names: {"yosys": {"git-commit": "v0.70"}} builds yosys v0.70 with SiliconCompiler’s own script. Ship ubuntu24/install-yosys.sh as well only to replace the recipe on that OS.
Rebuilds. A tool’s fingerprint covers its own pin and the pins of what it lists in docker-depends, so sc-install rebuilds it when either moves, including when the move comes from your override.
One package per tool. Two installed packages supplying the same tool, whether by pin or by script, are an error.

The older tools entry point, which maps tool names straight to scripts, still works, and its scripts are given SC_TOOLSCRIPTS too.

To keep your pins current the way SiliconCompiler keeps its own, call its tool updater from a workflow of yours. Every entry in your _tools.json that has a git-url and "auto-update": true gets a pull request moving it to the newest commit or release tag. An override is read on its own, without SiliconCompiler’s fields, so it is updated only if it carries both itself.

# .github/workflows/tool_updater.yml
name: 'Tool Updater'

on:
  workflow_dispatch:
  schedule:
    - cron: '0 0 * * *'

jobs:
  update:
    permissions:
      contents: read
      pull-requests: read
    uses: siliconcompiler/siliconcompiler/.github/workflows/update_tools.yml@main
    with:
      manifest: mylib/toolscripts/_tools.json
      app-client-id: ${{ vars.MYLIB_BOT_CLIENT_ID }}
      # Only for pins cloned from private repositories of the same owner
      repositories: mytool
      source-app-client-id: ${{ vars.MYLIB_READ_BOT_CLIENT_ID }}
    secrets:
      app-private-key: ${{ secrets.MYLIB_BOT_PRIVATE_KEY }}
      source-app-private-key: ${{ secrets.MYLIB_READ_BOT_PRIVATE_KEY }}

The pull requests are opened by a GitHub App you name with app-client-id and app-private-key, which needs contents and pull requests write on your repository. A pull request opened with GITHUB_TOKEN would run none of your checks, and that token cannot read another private repository. The repositories listed in repositories are read through source-app-client-id and source-app-private-key, an App that needs only contents read on them; without one, the App that opens the pull requests reads them too.

SiliconCompiler builds a docker image of each tool, tagged by a hash of everything that goes into it, and yours can be built the same way, on SiliconCompiler’s images:

python3 -m siliconcompiler.utils.toolimages --image_prefix myorg/mylib_ --plan
python3 -m siliconcompiler.utils.toolimages --image_prefix myorg/mylib_ \
    --generate_files --output_dir docker

The tags are computed over your pins merged into SiliconCompiler’s. An image that would come out the same as SiliconCompiler’s is taken from ghcr.io, where SiliconCompiler publishes it; the rest are built as ghcr.io/myorg/mylib_<tool>, or under another registry given with --registry: your own tools, any tool you override, and everything built against one. --plan lists both, and --json_tools --stage 1, then 2 and so on, gives the order to build them in: an image is built after every built image it is built on. Each generated directory is a build context. A token that can read a private source goes in the git_token build secret, where every install script built with it can read it:

docker build --secret id=git_token,env=GIT_TOKEN docker/mylib_mytool

The image your CI runs in is mylib_tools, and --plan says which of three it is:

sc_tools itself, when you build nothing.
sc_tools with your tools added as one layer, when you only add tools.
Assembled from every tool’s image, the way sc_tools is, when you change a pin or a recipe of one of SiliconCompiler’s: the version it replaces would otherwise survive underneath.

SiliconCompiler’s workflow builds whatever does not exist yet and returns the image’s name:

jobs:
  images:
    permissions:
      contents: read
      packages: write
    uses: siliconcompiler/siliconcompiler/.github/workflows/tool_images.yml@main
    with:
      image-prefix: myorg/mylib_
      # Only for pins cloned from private repositories of the same owner
      repositories: mytool
      source-app-client-id: ${{ vars.MYLIB_READ_BOT_CLIENT_ID }}
    secrets:
      source-app-private-key: ${{ secrets.MYLIB_READ_BOT_PRIVATE_KEY }}

  test:
    needs: images
    runs-on: ubuntu-latest
    container:
      image: ${{ needs.images.outputs.image }}
      credentials:
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

It installs your package from the calling repository, editable so that its data resolves to the checkout, and pushes to ghcr.io with GITHUB_TOKEN; another registry is named with registry, logged in to with registry-username and the registry-password secret. SiliconCompiler’s images are always pulled from ghcr.io.

Publishing to PyPI#

For an open-source library, publishing to PyPI lets anyone install it with a plain pip install mylib. Build the wheel and upload it:

python3 -m build                 # build the wheel and sdist into dist/
python3 -m twine upload dist/*   # publish to PyPI

Most projects prefer to automate this with a CI workflow — for example a GitHub Action using pypa/gh-action-pypi-publish — that builds and publishes on every tagged release. Proprietary libraries skip PyPI and are shared instead through a private package index or straight from the git remote (see the VCS install form below).

Installing and using the library#

Once published, the library is installed like any Python package:

pip install mylib
# for local development against your working tree:
pip install -e .
# or install a specific tagged revision straight from a git remote:
pip install "mylib @ git+https://company.com/git/[email protected]"

The last form uses pip’s VCS direct-reference syntax, which is handy for private libraries hosted on an internal git server.

Note

Installing next to a SiliconCompiler source checkout. If the environment runs SiliconCompiler from a git checkout (pip install -e .) rather than from a PyPI release, install the library with --no-deps:

pip install --no-deps mylib

setuptools_scm stamps a checkout with a development version such as 0.38.5.dev4+g1a2b3c4, and no == pin matches a development version. The exact pin recommended above therefore looks unsatisfied, so pip resolves it by replacing the editable checkout with a release wheel from PyPI, silently undoing the source install. --no-deps leaves the checkout alone; install the library’s remaining dependencies yourself, or relax the pin to >= while developing against a checkout.

The package is consumed either through its target or by adding its modules directly:

import siliconcompiler
from mylib.libs import mymacro
from mylib.targets.mytarget import mytarget

project = siliconcompiler.ASIC()

# Option A: one-call setup via the target.
mytarget(project)

# Option B: attach modules directly.
project.add_asiclib(mymacro.MyMacro())

# project.run()

Because each library roots its dataroot at its own __file__, all of its data files resolve automatically, whether it was installed from PyPI or checked out in editable mode.

Reference#

Defining a Library — the library classes and filesets.
Defining a PDK — packaging a process kit.
Building a Flowgraph — defining flows to ship in the package.
Integrating a New Tool — writing custom tool drivers.
Creating a Target — bundling modules into a target.

Packaging an External Library#

Why package a library#

Package layout#

The pyproject.toml#

Locating data files#

Two dataroot strategies#

Defining modules inside the package#

ASIC timing models and the delay model#

Bundling into a target#

Adding tools to sc-install#

Publishing to PyPI#

Installing and using the library#

Reference#

The `pyproject.toml`#