Skip to content

Kubernetes environment

When using Kubernetes environments, each user session runs in a dedicated Pod inside a defined Kubernetes cluster. This is in contrast to the local environment, where each user session runs on the same host as Bifröst.

With Bifröst

As the resulting Pod runs inside of Kubernetes and the terminal session will be connected to this Pod, the user can access (depending on the cluster settings) access everything directly inside the cluster.

You can prepare images with all required tolls installed (like database client, curl, ...) and can access all resource directly with the cluster internal addresses like my-service.my-namespace (or my-service.my-namespace.svc.cluster.local). Assuming at my-service.my-namespace is a REST service running at port 80 the following command will run out-of-the box once you connected via ssh: 

1
curl http://my-service.my-namespace

The SSH-client port-forwarding, works too and even the SOCKS5 proxy, where you can call every cluster internal URL from within your desktop browser.

If there is also kubectl installed in the used image and an adequate serviceAccount is configured, you can even use kubectl without any further authorizations.

This simply means: You do not need tools like kubectl locally installed and configured, your favorite SSH-client is enough.

Without Bifröst

Usually there are the following scenarios used to access resources inside a kubernetes cluster:

  1. Directly from the local command line via kubectl using dedicated credentials, which needs direct access to the cluster API, which may not always be desirable.
  2. Using a Bastion/Jump Host where all the stuff is already installed, as in point 1.

Or to interact with resources inside the cluster itself (like connect to a database, REST service, ...), usually the following scenarios are used:

  1. Using kubectl (in any way as described before), using port forwarding.
  2. Or (also with kubectl) execute inside an existing inside the desired cluster with all required tools installed.

Configuration

type

Environment Type = "kubernetes"

Has to be set to kubernetes to enable the Kubernetes environment.

config

Kubeconfig Core

Holds a kubeconfig (in YAML format) which defines the access to the desired Kubernetes cluster.

Ensure that the used configuration points to a user/service account with sufficient permissions to get/list/watch/create/modify/delete pods, secrets (depends on the configuration of Bifröst) and namespaces (depends on the configuration of Bifröst).

If the content is explicitly set to incluster it assumes that Bifröst runs inside a Kubernetes Pod and a valid service account was configured, all required resources are present (KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variable, /var/run/secrets/kubernetes.io/serviceaccount/token, /var/run/secrets/kubernetes.io/serviceaccount/ca.crt and /var/run/secrets/kubernetes.io/serviceaccount/namespace).

Default behavior

If this property is not defined, the following steps will be evaluated:

  1. If the environment variable KUBE_CONFIG exists, its content will be evaluated.
  2. If the environment variable KUBECONFIG exists it should contain file names, these will be read and its content will be evaluated.
  3. If the file ~/.kube/config exists, its content will be read and its content will be evaluated.
  4. If all dependencies like incluster exists, it will be used.
  5. Fail.

Examples

  1. Using direct yaml: 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
    config: |
     clusters:
     - cluster:
       name: my-cluster
         server: https://k8s.example.org/k8s/clusters/c-xxyyzz
     users:
     - name: my-auth
         token: foo
     current-context: my-context
     contexts:
     - context:
       name: my-context
         user: my-auth
         cluster: my-cluster
  2. Using content from file: 
    1
    config: "{{ file `/etc/engity/bifroest/kubernetes/my-config` }}"
  3. Using content from environment variable: 
    1
    config: "{{ env `MY_GREAT_CONFIG` }}"
  4. Use incluster config: 
    1
    config: "incluster"

context

string Core

Defines which context of the config should be used.

If not defined the default one of the config will be used. If also there is nothing defined, this will result in an error.

loginAllowed

bool Authorization = true

Has to be true (after being evaluated) that the user is allowed to use this environment.

Examples

  1. Require that the existing local user has the group ssh: 
    1
2
3
4
5
    loginAllowed: |
   {{ or
       (.authorization.user.group.name | eq "ssh" )
       (.authorization.user.groups     | firstMatching `{{.name | eq "ssh"}}`)
   }}

name

string Authorization = "bifroest-{{.session.id}}"

Kubernetes name of the Pod which will be created.

namespace

string Authorization

Kubernetes namespace where the Pod will be created inside.

If this is empty, the configured namespace of the provided config will be used. If this is empty too, default will be used.

If the namespace does not exist, it will be created.

os

Os Authorization = "linux"

Defines the operating system the resulting container should use. There are only support values supported where also a distribution of Bifröst in combination with the matching value of the arch property exists.

arch

Arch Authorization = "amd64"

Defines the architecture the resulting container should use. There are only support values supported where also a distribution of Bifröst in combination with the matching value of the os property exists.

serviceAccountName

string Authorization = ""

Is the name of the Kubernetes Service Account which should be used by the Pod will be created.

It is used to grant/restrict the direct access to the Kubernetes internal APIs, by creating role-base access control (RBAC) matching the service account. For more details see Grant ServiceAccount permissions of the Kubernetes documentation.

If this is configured, a kubectl installed inside the Pod's image, can access the clusters resources (following the permissions granted via RBAC).

image

string Authorization = "alpine"

An OCI/Docker image which should be used for the container. Everything inside this image will be available to the user who is executed via SSH into this Pod's container.

There can be any public available image be used or also private images. If you're using private images, you need to provide imagePullSecretName and/or imagePullCredentials. Always ensure that the image can be accessed from the Kubernetes cluster itself; Bifröst itself will not interact with this image.

imagePullPolicy

Pull Policy Authorization = "ifAbsent"

Controls when the image of the resulting Pod should be pulled.

imagePullSecretName

string Authorization

If defined, this secret will be used to pull the defined image from the image registry. This is usually necessary if private images are used. See Kubernetes documentation how to pull images from private registries for more information and to find out how to create such a secret.

This secret has to exist, otherwise the creation of the Pod will fail.

If imagePullCredentials is defined, this secret does not have to exist, it will be created, based on this property's content. If imagePullSecretName is empty in this case, the resulting name will be pull-secret.<pod-name>.

imagePullCredentials

string Authorization

If defined, these credentials are used to pull the defined image from the image registry. It will result in a pull secret which will be created with the name of imagePullSecretName and the content of this property.

Examples

  1. Using direct json: 
    1
2
    imagePullCredentials: |
 {"username":"foo","password":"bar"}
  2. Using base64 URL encoded json: 
    1
2
    imagePullCredentials: |
 eyJ1c2VybmFtZSI6ImZvbyIsInBhc3N3b3JkIjoiYmFyIn0
  3. Using content from file: 
    1
    imagePullCredentials: "{{ file `/etc/engity/bifroest/secrets/my-great-secret` }}"
  4. Using content from environment variable: 
    1
    imagePullCredentials: "{{ env `MY_GREAT_SECRET` }}"

readyTimeout

Duration Authorization = "5m"

Defines the maximum time the Pod is allowed to be successfully started within. If this time is passed and the Pod is not ready, it will be marked as failed and will be reported back to the user who tries to connect via SSH.

removeTimeout

Duration = "1m"

Defines the maximum time the Pod is allowed to gracefully shutdown within. Longer will result in an error.

capabilities

[]string Authorization

List of Unix kernel capabilities to be added to the container. This enables a more fine-grained version in contrast to give all capabilities to the container with privileged = true.

Does only work on Unix based systems.

privileged

bool Authorization = false

If this is set to true this container will have all capabilities of the system.

Danger

Only enable this feature if you really need this, and you know what you're doing.

dnsServers

[]string Authorization

Defines a list of external DNS server the container should use.

dnsSearch

[]string Authorization

Defines custom DNS search domains for the container.

shellCommand

[]string Authorization = "<os specific>"

The shell which should be used to execute the user into.

If not defined, the following command will be used:

  • Linux: ["/bin/sh"]
  • Windows: ["C:\WINDOWS\system32\cmd.exe"]

execCommand

[]string Authorization = "<os specific>"

If execute is used, this is the command prefix which will used for the command.

If not defined, the following command will be used:

  • Linux: ["/bin/sh", "-c"]
  • Windows: ["C:\WINDOWS\system32\cmd.exe", "/C"]

sftpCommand

[]string Authorization = ["bifroest", "sftp-server"]

Defines the sftp server command which should be used. Usually you should not be required to modify this, because by default Bifröst is handling this by itself.

directory

File Path Authorization

Defines the working directory of the initial process inside the container for each execution.

If not defined the value WORKDIR will be used. If this is absent it defaults to: /.

user

string Authorization

Defines the user which will run inside the container.

If not defined the value USER will be used. If this is absent it defaults to: root.

group

string Authorization

Defines the group of the user which will run inside the container.

If this is absent it defaults to the group of the user.

banner

string Authorization = ""

Will be displayed to the user upon connection to its environment.

Examples

  1. If local user is used, show its name in a message: 
    1
    banner: "Hello, {{.authorization.user.name}}!\n"
  2. If users authorized via OIDC is used, show its name in a message: 
    1
    banner: "Hello, {{.authorization.idToken.name}}!\n"

portForwardingAllowed

bool Authorization = true

If true, users are allowed to use SSH's port forwarding mechanism.

cleanOrphan

bool Container = true

While the housekeeping iterations this environment will look for pods that can be inspected based on the provided config. Is there any container that does not belong to any flow of this Bifröst instance, it will be removed.

This is useful to clean up old pods which are leftovers after you have changed the configuration of Bifröst.

Warning

If multiple Bifröst installations are using the same config/cluster, this should be disabled. Otherwise, each instance is removing the pods of the other instance.

Preparation Processes

If events about preparation processes are emitted by this environment, they are picked up by connections (like SSH) and handled.

The kubernetes environment can emit the following processes:

create-pod

As the creation of a pod creation can take up-to some minutes, this process will be emitted. As this process does not really expose progress of downloading a required image or similar, there is currently no progress reporting supported (only a fake 0% .. 30% .. 100%).

Properties

name

string

Target name of the Pod. See property name for more details.

namespace

string

Target namespace of the Pod. See property namespace for more details.

image

string

Target image of the Pod. See property image for more details.

remove-pod

If it is required to remove an existing pod before a new will be created, this process can be made visible to the user. There is currently no progress reporting supported.

Properties

name

string

Name of the existing Pod. See property name for more details.

namespace

string

Namespace of the existing Pod. See property namespace for more details.

Examples

  1. Basic 

    1
2
3
4
5
    type: kubernetes
## This will create a Kubernetes environment based on the
## default context, defined inside `~/.kube/config`
## OR the content of the environment variable KUBE_CONFIG
## with alpine image.

  2. Explicit context 

    1
2
3
4
5
    type: kubernetes
context: "my-context"
## This will create a Kubernetes environment based on the
## context "my-context", defined inside `~/.kube/config`,
## with alpine image.

  3. Custom kube-config 

    1
2
3
4
5
    type: kubernetes
config: "/etc/kube/my-kube-config"
## This will create a Kubernetes environment based on the
## default context of the kube config stored inside
## /etc/kube/my-kube-config.

  4. With ubuntu image, custom shell and login restriction 

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    type: kubernetes
## This will create a Kubernetes environment based on the
## default context, defined inside `~/.kube/config`.

image: ubuntu
## Using /bin/bash instead of /bin/sh,
## because it does exist in the image
shellCommand: [/bin/bash]
execCommand: [/bin/bash, -c]

## Only allow login if the OIDC's groups has "my-great-group-uuid"
## ...and the tid (tenant ID) is "my-great-tenant-uuid"
loginAllowed: |
   {{ and
     (.authorization.idToken.groups | has "my-great-group-uuid")
     (.authorization.idToken.tid    | eq  "my-great-tenant-uuid")
   }}

  5. Using my own registry with secret from file: 

    1
2
3
4
5
    type: kubernetes
image: my.own.registry.com/foo/bar
## Using the pull credentials, which are stored inside:
## /etc/engity/bifroest/secrets/my.own.registry.com
imagePullCredentials: "{{ file `/etc/engity/bifroest/secrets/my.own.registry.com` }}"

  6. Using my own registry with secret from environment variable: 

    1
2
3
4
5
    type: kubernetes
image: my.own.registry.com/foo/bar
## Using the pull credentials, which are stored inside
## MY_GREAT_SECRET environment variable
imagePullCredentials: "{{ env `MY_GREAT_SECRET` }}"

Compatibility

linux windows
/ /