# onyx-dev - Container Server of nbit Informatik GmbH

onyx-dev.nbit.ch is used to run rootless Podman containers, using Traefik as a Reverse Proxy

It is the Development Server, the main server is onyx.nbit.ch

Additionally it acts as a primary nameserver ns1.nbit.ch

In this directory (/data/onyx-dev_pods), you will find all configuration files to run the containers (with Podman and Kubernetes YAML files)

Specs:
- Rocky Linux 10
- Hetzner Cloud Server CX21
  - 2 vCPUs
  - 4 GB RAM
  - 40 GB Disk
  - Location Nuremberg

Persistent data is stored in /data

## Create Server

Name:    onyx-dev.nbit.ch

Set Root-Password (by hand)

`
# dnf update
# init 6 (Reboot)
# groupadd containers
# useradd -m -g containers containers
# passwd containers

For containers User, add the following to ~/.bash_profile:

if [ -z "$XDG_RUNTIME_DIR" ] && [ -d "/run/user/$(id -u)" ]; then
    export XDG_RUNTIME_DIR=/run/user/$(id -u)
fi

# hostnamectl set-hostname onyx-dev.nbit.ch
# timedatectl set-timezone Europe/Zurich
# dnf install glibc-langpack-de
# localectl set-locale LANG=en_US.UTF-8
# localectl set-locale LC_TIME=de_CH.UTF-8

Set Journalctl-Config to persistent Storage:
cp /usr/lib/systemd/journald.conf /etc/systemd
/etc/systemd/journald.conf:
Storage=persistent

# Enable Selinux in Enforcing Mode: change /etc/selinux/config
# dnf install setroubleshoot-server

Disable kdump:
systemctl disable --now kdump
grubby --update-kernel=ALL --remove-args="crashkernel"
reboot
`

enable EPEL Repo (for instance because of fail2ban):
`
# dnf install epel-release
`

## Firewall

```bash
# dnf install firewalld
# systemctl enable --now firewalld
# firewall-cmd --add-service={http,https} --permanent
# firewall-cmd --remove-service=cockpit --permanent
# firewall-cmd --reload

List Rules:
# firewall-cmd  --list-all
```

## fail2ban on Host for ssh

```bash
# dnf install fail2ban
# cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
edit /etc/fail2ban/jail.local:
enabled = true  => below [sshd]

# systemctl enable fail2ban
# systemctl restart fail2ban

command to check who is banned:
# fail2ban-client status sshd
```

## Install Software

```bash
# dnf install passt
# dnf install git
# dnf install podman
# dnf install jq
# dnf install sysstat
# dnf install lftp
# dnf install telnet
# dnf install binutils
```

## Setup Mail

```bash
# dnf install s-nail procmail msmtp
# cp /usr/share/doc/msmtp/msmtprc-system.example /etc/msmtprc
```

## Backup Server

```bash
# dnf install restic
# mkdir /backup
# mkdir /backup-restic
# restic init --repo /backup-restic/restic-repo-$(hostname --short)      # Passwort in Keepass

SSH Keypaar fuer User root erstellen und auf Hetzner Storagebox hinterlegen:

# ssh-keygen

Restic Script:

/usr/local/bin/backup-to-disk.sh
#!/bin/bash
# Backup der wichtigsten Verzeichnisse nach einem Verzeichnis
#
# Es wird restic verwendet.
#
PATH=$PATH:/usr/local/bin
export RESTIC_PASSWORD="$(hostname --short)7355"
restic backup --quiet --repo /backup-restic/restic-repo-$(hostname --short) /home /etc /var /opt /data /usr/local/bin /backup --exclude=/var/log

if [ $? -eq 0 ]; then
  restic forget --quiet --repo /backup-restic/restic-repo-$(hostname --short) --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 20 --prune
else
  >&2 echo "Problem with restic Backup $(hostname --short)"
fi

/etc/cron.d/backup-to-disk:
#
# Backup important Files to Disk
#
55 6 * * * root /usr/local/bin/backup-to-disk.sh >/dev/null

Backup auf Storag Box:

# cat > /etc/cron.d/rsync-backup-to-other-host <<HERE
#
# Rsync /backup-restic to backup space
#
20 7 * * * root /usr/bin/rsync -avzH --delete --numeric-ids -e 'ssh -p23' /backup-restic u152662@u152662.your-storagebox.de:onyx-backup-restic-rsync >/dev/null
HERE
```


List Usage on Storage Box:

```
# ssh -p 23 u152662@u152662.your-storagebox.de du -h -d 1
261G	./arq-backup-mbp
8.3G	./moby-backup-restic-rsync
650M	./onyx-dev-backup-restic-rsync
1.6G	./mb2-backup-rsync
1.2G	./wobisch2-backup-rsync
1.6G	./mb1-backup-rsync
11G	./mail-backup-restic-rsync
20G	./onyx-backup-restic-rsync
44G	./cinnamon.nbit.ch
10K	./.ssh
348G	.
```



### Setup Env for Podman

```bash
# mkdir /data
# chown containers:containers /data

Set Defaults:

containers$ cat ~/.config/containers/containers.conf
[network]
network_backend = "netavark"


Credentials for Docker Registry should be available after Reboot

# echo 'net.ipv4.ip_unprivileged_port_start=80' >> /etc/sysctl.d/containers.conf
# loginctl enable-linger containers
[containers@onyx-dev ~]$ podman login git.nbit.ch
Username: jlehmann
Password:
Login Succeeded!

[containers@onyx-dev ~]$ mkdir ~/.config/containers
[containers@onyx-dev ~]$ cp /run/user/1000/containers/auth.json ~/.config/containers/auth.json


```

### Setup Traefik

Traefik will be started with podman play kube (with yaml file) and attach to HostNetwork (hostNetwork: true in yaml).
Backends will map there port and will be accessed on localhost:<port>

we do not use the dynamic config using docker provider, but rather the file provider (one file per backend)
```bash
Example Backend Service File:

[containers@onyx onyx_pods]$ cat traefik/configuration/nbitwebsite.yml
http:
  routers:
    nbitwebsite:
      entrypoints:
      - websecure
      tls:
        certresolver: "myresolver"
        domains:
          - main: "www.linux-freelancer.ch"
            sans: "linux-freelancer.ch"
      rule: "Host(`linux-freelancer.ch`,`www.linux-freelancer.ch`)"
      service: nbitwebsite

  services:
    nbitwebsite:
      loadBalancer:
        servers:
        - url: http://127.0.0.1:9000/
        passHostHeader: false
```

```bash
# echo 'net.ipv4.ip_unprivileged_port_start=80' >> /etc/sysctl.d/containers.conf
# loginctl enable-linger containers
# The following fixes "Failed to connect to bus: No medium found"
export XDG_RUNTIME_DIR=/run/user/$(id -u)

containers$ touch /data/traefik/acme.json
containers$ chmod 0600 /data/traefik/acme.json

Important: set SELinux Context on /data for containers:


[root@onyx-dev named]# semanage fcontext -a -s unconfined_u -t container_file_t "/data(/.*)?"
[root@onyx-dev named]# restorecon -r -v /data

Check:
find /data ! -context '*:container_file_t:*'


/etc/subgid:containers:100000:65536
/etc/subuid:containers:100000:65536

!!!!!!!!!!!!!!
IMPORTANT: we changed the startup Method to Quadlets, put into
~containers/.config/containers/systemd/

See also https://www.redhat.com/sysadmin/quadlet-podman

Example of such a quadlet config (Network=pasta is not to be used for Traefik):

[containers@onyx-dev ~]$ more ~containers/.config/containers/systemd/onyx-dev-mqtt.kube
[Unit]
Description=MQTT
Before=local-fs.target

[Kube]
Yaml=/home/containers/onyx-dev_pods/mqtt/mqtt.yaml
LogDriver=journald
Network=pasta

[Service]
Restart=always
RestartSec=15
TimeoutStartSec=300

[Install]
# Start by default on boot
WantedBy=multi-user.target default.target
!!!!!!!!!!!!!!

```

### Setup Backend Services

```bash
Create Kubernetes YAML File:

backendservice=nbitwebsite
containers$ mkdir ~/onyx_pods/${backendservice}

Create File ~/onyx_pods/${backendservice}/${backendservice}.yaml

```

## Postfix

```bash
# dnf install postfix

We do not want to accept emails from outside, so we do not change the config (we only listen on localhost)

# systemctl enable postfix
# systemctl start postfix

Creat DNS Records:
mx1.nbit.ch and mx2.nbit.ch (A/AAAA Records to the same IPs as onyx and onyx-dev)

Change SPF Records to include mx1 and mx2
```

## Primary Nameserver

```bash
# dnf install bind bind-utils
# systemctl enable --now named
# firewall-cmd --add-service=dns --permanent
# firewall-cmd --reload

Adopt /etc/named.conf:

Important => recursion no

[root@onyx-dev etc]# diff named.conf named.conf.6nov2026
11,12c11,12
< 	listen-on port 53 { any; };
< 	listen-on-v6 port 53 { any; };
---
> 	listen-on port 53 { 127.0.0.1; };
> 	listen-on-v6 port 53 { ::1; };
19,20c19
< 	allow-query     { any; };
<         version "You are too curious";
---
> 	allow-query     { localhost; };
32c31
< 	recursion no;
---
> 	recursion yes;
61,124d59
< acl slaves {
<    95.216.148.212;
<    2a01:4f9:c010:5dd::1;
<    23.88.33.113;
<    2a01:4f8:c010:90e1::1;
<    5.75.251.111;
<    2a01:4f8:c012:c21c::1;
<    127.0.0.1;
<    ::1;
< };
<
< zone "nbit.ch" IN {
<   type master;
<   file "master/nbit.ch.zone";
<   allow-update { none; };
<   allow-transfer {
<         slaves;
<   };
< };
<
< zone "mini-beieli.ch" IN {
<   type master;
<   file "master/mini-beieli.ch.zone";
<   allow-update { none; };
<   allow-transfer {
<         slaves;
<   };
< };
<
< zone "wo-bisch.ch" IN {
<   type master;
<   file "master/wo-bisch.ch.zone";
<   allow-update { none; };
<   allow-transfer {
<         slaves;
<   };
< };
<
< zone "ch-wirth.ch" IN {
<   type master;
<   file "master/ch-wirth.ch.zone";
<   allow-update { none; };
<   allow-transfer {
<         slaves;
<   };
< };
<
< zone "cmoag.com" IN {
<   type master;
<   file "master/cmoag.com.zone";
<   allow-update { none; };
<   allow-transfer {
<         slaves;
<   };
< };
<
< zone "acmoag.com" IN {
<   type master;
<   file "master/acmoag.com.zone";
<   allow-update { none; };
<   allow-transfer {
<         slaves;
<   };
< };


on master:

[root@onyx-dev named]# semanage fcontext -a -t named_cache_t "/var/named/master(/.*)?"
[root@onyx-dev named]# restorecon -r -v /var/named/master
...
```