🚚 The Ktra Book

Your Little Cargo Registry.

ktra at crates.io

Sections

About Ktra

To know about Ktra.

Installation

To install Ktra.

Quick Start

To start Ktra quickly.

Appendices

About Ktra

To know about Ktra:

Overview

What is Ktra?

Ktra is a implementation of the Alternate Registry that is introduced for non-public crates in Rust/Cargo 1.34.

In other words, Ktra is an all-in-one package for the private cargo registry.

When I/we need Ktra?

If you write a Rust crate in your job, your codes might be proprietary and/or closed-source.
crates.io cannot host closed-source crates so you have to construct a private registry by yourself.
Ktra could help to reduce that annoying operations.

What does Ktra mean?

Ktra is named after shortened name of Kei truck which is a japanese style mini pickup truck.

You can pronounce Ktra as you like.
For your information, most native japanese speaker do it KEI-TORA or KE-TORA.

License

Ktra is icensed under either of Apache License, Version 2.0 or MIT license at your option.

Features

Ktra has now features listed below.

  • Almost pure Rust.
  • Minimum Alternate Registry implementation.
  • Secure user management with Argon2 or as an OpenID Service Provider (compatible with .well-known discoverable identity providers such as gitlab)
  • Sled as its internal database.
    • via db-sled feature turned on by default.
  • Redis support.
    • via db-redis feature.
  • MongoDB support.
    • via db-mongo feature.

Installation

To install Ktra:

Install With Cargo

You can install Ktra simply with just running a command below.
This command will install the default Ktra enabled secure-auth and db-sled features.

cargo install ktra

Change database implementation

If you want to select the backended database:

  • db-redis 
cargo install ktra --no-default-features --features=secure-auth,db-redis
  • db-mongo 
cargo install ktra --no-default-features --features=secure-auth,db-mongo

Add openid support

Openid support can be added with the extra "openid" feature, which is not compatible with secure-auth:

cargo install ktra --no-default-features --features=openid

You can add other db-* features to change the backing database implementation as above.

Install With Docker

You can also install Ktra with the Docker image.

docker pull ghcr.io/moriturus/ktra:latest

Or, if you want to pull a fixed version image, try this.

docker pull ghcr.io/moriturus/ktra:0.7.0

Note:
The latest published Ktra version is ktra at crates.io.

Tags

There are several images tagged with features and releases.
Any commit on develop branch builds images listed below:

  • latest
    • db-sled featured image.
  • openid-latest
    • db-sled featured image.
    • openid support for user authentication and authorization
  • db-redis-latest
    • db-redis featured image.
  • db-redis-openid-latest
    • db-redis featured image.
    • openid support for user authentication and authorization
  • db-mongo-latest
    • db-mongo featured image.
  • db-mongo-openid-latest
    • db-mongo featured image.
    • openid support for user authentication and authorization

Similarly, images below are built automatically when tags are pushed:

  • {VERSION} (e.g. 0.7.0)
    • db-sled featured image.
  • openid-{VERSION}
    • db-sled featured image.
    • openid support for user authentication and authorization
  • db-redis-{VERSION}
    • db-redis featured image.
  • db-redis-openid-{VERSION}
    • db-redis featured image.
    • openid support for user authentication and authorization
  • db-mongo-{VERSION}
    • db-mongo featured image.
  • db-mongo-openid-{VERSION}
    • db-mongo featured image.
    • openid support for user authentication and authorization

Note:
The openid images are only available starting with 0.7 version

Registry

All of the docker images are built with emk/rust-musl-builder image and stored at GitHub Container Registry.
These are public images so you can pull them without any authentication.

Quick Start

To start Ktra quickly:

Create Index Git Repository

The Index Repository serves data read by cargo command when you build a crate depending on another crate hosted on Ktra.
You can use any git repository hosting service such as GitHub, GitLab and Bitbucket even if its repository is private.
Additionally, of course, you can use private git repository managed by yourself.

This book introduces the simplest way to create the index repository with GitHub.

1. Go to Create new repository page. (login needed)

For more details, please see GitHub official documents.

  • The index repository's name is arbitrary.
  • Public or private either is fine.

2. Clone the repository created above.

Note:
You have to create an access token in advance to clone with HTTPS protocol.
Please see the GitHub official documents.

git clone https://github.com/YOUR_USERNAME/YOUR_INDEX.git

3. Create config.json file then commit and push it to remote repository.

Note:
Please make sure your branch name.
main is the default on GitHub but GitLab, Bitbucket and other git repository hosting services might default to master instead of main.

cd YOUR_INDEX
echo '{"dl":"http://localhost:8000/dl","api":"http://localhost:8000"}' > config.json
git add config.json
git commit -am "initial commit"
git push origin main

4. Edit your .cargo/config.toml.

Note 1:
Please see Cargo official documents for more details.

Note 2:
To make git remember your credential, please use the credential helper.

[registries]
ktra = { index = "https://github.com/YOUR_USERNAME/YOUR_INDEX.git" }

Create Ktra Configuration File

About

Ktra Configuration File is a TOML formatted document file.
To configure Ktra's database, index and crate files, you should write it and pass the path to it as a command argument.

Sample

Note 1:
All configurations are listed on Appendix: Configurations page.

Note 2:
Please make sure your branch name AGAIN.
main is the default on GitHub but GitLab, Bitbucket and other git repository hosting services might default to master instead of main.

[index_config]
remote_url = "https://github.com/YOUR_USERNAME/YOUR_INDEX.git"
https_username = "YOUR_USERNAME"
https_password = "GITHUB_ACCESS_TOKEN" 
branch = "main"

And run Ktra with passing the path to above file.

ktra -c /path/to/config/your_config.toml
  • Please remember that Ktra searches ./ktra.toml as a default configuration file if not specified.

Create User And Login

Ktra has several additional web APIs called Ktra Web APIs to manage users.
You should call these APIs to create a new user.

Note:
All APIs are listed and detailed on Appendix: Ktra Web APIs page.

1. Create a new user

Run this command. curl command required.
ALICE just means the first user. Of course you can use a username you like.

curl -X POST -H 'Content-Type: application/json' -d '{"password":"PASSWORD"}' http://localhost:8000/ktra/api/v1/new_user/ALICE

Above command responds and prints a new token like this:

{"token":"TOKEN"}

Please copy TOKEN, the printed token.

2. Login with cargo command

Run this command with the TOKEN that you copied.

Note:
ktra in below command is the registry name you specified in .cargo/config.toml file.

cargo login --registry=ktra TOKEN

It's OK if you get this message:

Login token for `ktra` saved

OpenId Service Provider registration

Warning:
The support of OpenID is experimental, and only confirmed to work well with Gitlab as the Identity Provider.

Note:
A user who can "login" to the registry only gains the right to own crates on the registry, and call the associated cargo publish, cargo yank, and cargo owner --add/--remove APIs. You do not need to be logged in, in the cargo login sense, to pull crates from the registry; just as users in general do not need to run cargo login before being able to pull crates from crates-io like serde.

Prerequisites

To be able to use OpenId for authentication you need to know which Identity Provider (IP) you want to use, and that Identity Provider needs to have the following properties:

  • you must be able to register a new application to it (obtaining a client ID and a client secret),
  • the IP must have a discoverable configuration (through GET /.well-known/openid-configuration), and
  • the discovered configuration must have a userinfo_endpoint.

For example, using a tool like jq, checking that Gitlab is valid is as easy as:

$ curl https://gitlab.com/.well-known/openid-configuration 2>/dev/null | jq ".userinfo_endpoint"
"https://gitlab.com/oauth/userinfo"

And an example of a provider that won't work:

$ curl https://github.com/.well-known/openid-configuration 2>/dev/null | jq ".userinfo_endpoint"
parse error: Invalid numeric literal at line 1, column 4

Note:
The feature has been fully implemented and tested thoroughly with Gitlab as the IP. The aim is to have OpenID support for all providers that meet the above criteria, so you might encounter issues with other IP, and contributions (issue reports, patches...) are welcome to help with the coverage.

If you want to use a provider that is not Gitlab, check the API documentation of the userinfo endpoint to see which claims you need to obtain. For example, Gitlab API documentation says here that extra profile information might require the profile and email claims, explaining why it shows later in the configuration.

From now on, all the steps will have pictures and steps specific to using Gitlab as the IP, to give a concrete, working example.

Setting up Ktra as Service Provider (SP)

Most of the time, adding Ktra as a SP will go as "Adding an application" menu for the IP. Here for gitlab, we are adding an application:

  • with the required "scopes" (matching the claims we found necessary for the userinfo endpoint), and
  • with 1 or 2 redirect URLs depending on whether you want to allow revoking tokens (2 is recommended):
    • ${BASE_URL}/ktra/api/v1/openid/me is mandatory to support cargo login
    • ${BASE_URL}/ktra/api/v1/openid/replace_token is optional, but will allow to use an endpoint to replace your saved token with a new one, effectively revoking the old one (useful when one token leaks)

When doing it with Gitlab, we end up on a screen looking like this (note that there is only 1 URL registered in this example):

Screenshot of 3rd Party application settings on gitlab

Take note of the client_id and client_secret to go to the next steps.

Note:
With Ktra, a user can only have a single active token. That means using cargo login multiple times (or going to the /me endpoint) will always return the same, existing token if the user already went through a first login.

Extra Ktra configuration

The client id and secret are used in the extra [openid_config] section in the TOML configuration of ktra (you can also use the equivalent commandline argument if you prefer).

As the implementation has been more fleshed out with Gitlab, only Gitlab allows some extra relevant confguration variables related to authorization:

  • gitlab_authorized_groups contains a list of the groups allowed to login on the registry
  • gitlab_authorized_users contains a list of logins allowed to login on the registry

A user needs to be at least in one authorized list to be allowed, and if both configuration variables are absent or empty, then everyone will be able to login.

[openid_config]
issuer_url = "https://gitlab.com"
# This needs to match the ${BASE_URL} or the IP will refuse to authenticate users because of mismatch in the
# redirect URLs
redirect_url = "http://registry.szl"
client_id = "717dc...."
client_secret = "[redacted]"
# The scopes come from checking the documentation of the userinfo_endpoint
additional_scopes = ["openid", "profile", "email"]
# Using groups allow to synchronize authorizations with an external group of users.
gitlab_authorized_groups = [ "firm/groups/crate_owners" ]
# If/When gitlab allows technical accounts, this is the best place to put a bors-like username,
# to be able to authorize crate publishing only from some CI pipeline or using webhooks.
gitlab_authorized_users = [ "gagbo" ]

Logging in

Using cargo login with the correct registry name will prompt you to go the /me endpoint, where you will start an authentication flow with the Identity provider until you arrive at a json payload containing the token (the exact name and values are different in the screenshot and in the documentation, the documentation here in the book is correct.)

Screenshot of the payload displayed after finishing an authentication flow

Pushing a crate

You can then use cargo publish to publish crates!

Screenshot of a successful crate publishing from the commandline

Appendix: adding IP-specific support

The ability to work on extra information like the group a Gitlab user belongs to comes from a deserializable struct in Ktra's openid handling code:


#![allow(unused)]
fn main() {
/// The additional claims OpenId providers may send
///
/// All fields here are options so that the extra claims are caught when presents
#[cfg(feature = "openid")]
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct Claims {
    pub(crate) sub: Option<String>,
    pub(crate) sub_legacy: Option<String>,
    // Gitlab claims return the groups a user is in.
    // This property is used when gitlab_authorized_groups is set in the configuration
    pub(crate) groups: Option<Vec<String>>,
}
}

If extra info is returned by the IP and necessary later to implement authorization logic, it should start from there, to be later used in the callback handler in such fashion:

if let Some(ref auth_groups) = openid_config.gitlab_authorized_groups {
        if let Some(ref groups) = userinfo.additional_claims().groups {
            for group in groups {
                if auth_groups.contains(group) {
                    tracing::info!("matched authorized group {}, authorizing.", group);
                    return true;
                }
            }
        }
    }

Publish And Use

🎉 Congratulations!
It's time to publish your first and private crate!

1. Create a new crate and publish

Just run:

Note 1:
ktra in below command is the registry name you specified in .cargo/config.toml file.

Note 2:
--allow-dirty option is not needed if your local repository is clean.

cargo new my_crate --lib
cd my_crate
cargo publish --allow-dirty --registry=ktra

If you get no errors your publishing has done perfectly.

2. Use your crate

To use your published crate from another one, you just specify a registry at dependencies section in Cargo.toml.

[package]
name = "my_another_crate"
version = "0.1.0"
authors = ["alice <[email protected]>"]
edition = "2018"

[dependencies]
my_crate = { version = "0.1", registry = "ktra" }

Appendix: Configurations

Note:
All configurations are able to set via command arguments. See ktra -h for more details.

About Index Configurations

index_config has these configurations:

  • remote_url
    • The essential configuration.
    • Sets a URL for the remote index git repository.
  • local_path
    • The default value is ./index
    • Sets a path for local index git repository.
  • branch
    • The default value is main.
    • Sets a branch name of the index git repository.
  • https_username
    • Required unless you use SSH protocol.
    • Sets a username to use for the authentication.
  • https_password
    • Required unless you use SSH protocol.
    • Sets a password to use for the authentication.
  • ssh_privkey_path
    • Required unless you use HTTPS protocol.
    • Sets a private key path to use for the authentication.
  • ssh_key_passphrase
    • Optional.
    • Sets a private key's passphrase to use for authentication if the remote index git repository uses SSH protocol.
  • ssh_pubkey_path
    • Optional.
    • Sets a public key path to use for authentication if the remote index git repository uses SSH protocol.
  • ssh_username
    • Optional.
    • Sets a username to use for authentication if the remote index git repository uses SSH protocol.
  • git_email
  • git_name
    • The default value is ktra-driver.
    • Sets an author and committer name.

About Crate Files Configurations

crate_files_config is optional and it has these configurations:

  • dl_dir_path
    • The default value is ./crates.
    • Sets the crate files directory.
  • cache_dir_path
    • crates-io-mirroring feature needed.
    • The default value is ./crates_io_caches.
    • Sets the cache files directory.
  • dl_path
    • The default value is ["dl"].
    • Sets crate file download paths.
      • Sample: ["path", "to", "download"] to be https://example.com/path/to/download
      • This configuration MUST correspond with the dl field's value in config.json at the index git repository.

About Database Configurations

db_config is optional and it has these configurations:

  • db_dir_path
    • The default value is db if you use db-sled feature.
    • Sets a database directory.
  • redis_url
    • The default value is redis://localhost if you use db-redis feature.
    • Sets a Redis URL. This can contain a username and password phrase to use the authentication.
  • mongodb_url
    • The default value is redis://localhost:27017 if you use db-mongo feature.
    • Sets a Redis URL. This can contain a username and password phrase to use the authentication.
  • login_prefix
    • The default value, for backwards compatible reasons, is ktra-secure-auth:
    • Sets a prefix to the usernames in the database/registry.

About Server Configurations

server_config is optional and it has these configurations:

  • address
    • The default value is [0, 0, 0, 0].
    • Sets an address HTTP server runs.
  • port
    • The default value is 8000.
    • Sets a port number HTTP server listens.

About OpenId Configurations

openid_config is optional and it has these configurations:

  • issuer_url
    • Mandatory if openid_config is specified
    • Sets the URL of the Identity Provider to use
  • redirect_url
    • Mandatory if openid_config is specified
    • Sets the base URL to use in the redirect URI for the authentication flow. This should match the URL ktra is exposed to, in order to comply with redirect URL verifications done by the Identity Provider.
  • client_id
    • Mandatory if openid_config is specified
    • Sets the ID of the ktra application as registered on the Identity Provider.
  • client_secret
    • Mandatory if openid_config is specified
    • Sets the secret of the ktra application as registered on the Identity Provider.
  • additional_scopes
    • Default is an empty list
    • Sets the additional scopes queried by the application for OpenId. Usually this value depends on the issuer.
  • gitlab_authorized_groups
    • Default is unset
    • Sets the authorized Gitlab groups whose members are allowed to create an account on the registry and be publishers/owners. Leave empty or unset not to check groups.
  • gitlab_authorized_users
    • Default is unset
    • Sets the authorized Gitlab users who are allowed to create an account on the registry and be publishers/owners. Leave empty or unset not to check users.

Appendix: Ktra Web APIs

Ktra Web APIs are extra web APIs that are not specified in the specification but required to manage users.

Login/Password users API

Since all APIs send passwords in cleartext, it is highly recommended that you connect the registry from your local network only OR use an HTTPS connection.

Create a new user

  • Specification
Endpoint /ktra/api/v1/new_user/{user_name}
Method POST
Body { "password": "PASSWORD" }
  • Response
{
    "token": "TOKEN"
}

Login

  • Specification
Endpoint /ktra/api/v1/login/{user_name}
Method POST
Body { "password": "PASSWORD" }
  • Response
{
    "token": "NEW TOKEN"
}

Change password

  • Specification
Endpoint /ktra/api/v1/change_password/{user_name}
Method POST
Body { "old_password": "OLD PASSWORD", "new_password": "NEW PASSWORD" }
  • Response
{
    "token": "NEW TOKEN"
}

Crates.io Mirroring

Note:
This API can be available when crates-io-mirroring feature is enabled.

  • Specification
Endpoint /ktra/api/v1/mirror/{crate_name}/{version}/download
Method GET
Body -
  • Response

Binary file.

OpenId API

When using the OpenId feature, you do not have any of the insecure user management endpoints active, and instead you get one endpoint to see your existing token/create a new one, and one endpoint to revoke your old token replacing it with a new one.

As the token end up being sent in cleartext still, it is still recommended to be either inside a small, trusted VPN or use HTTPS, or both.

For both endpoints, you will go through an OpenId authentication flow, so unless you already have an opened session, you must call this endpoint interactively.

See existing token, creating it for first time users

This is the same endpoint as the one being exposed for the cargo login flow

  • Specification
Endpoint /me
Method GET
  • Response

If the user already had a token:

{
    "username": "issuer:user_identity",
    "existing_token": "SOME TOKEN"
}

If the user never had a token:

{
    "username": "issuer:user_identity",
    "new_token": "SOME TOKEN",
    "revoked_token": null
}

Force token replacement

  • Specification
Endpoint /replace_token
Method GET
  • Response
{
    "username": "issuer:user_identity",
    "new_token": "SOME TOKEN",
    "revoked_token": "A NOW INVALID TOKEN"
}