SHA256
+221
@@ -0,0 +1,221 @@
|
||||
# SemiOS Build Server
|
||||
|
||||
A web-based package build server for SemiOS. Uses SvelteKit SSR for the UI and a Rust agent for executing builds on remote machines.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Build Server
|
||||
|
||||
- **Node.js** >= 18
|
||||
- **npm**
|
||||
- **Git**
|
||||
- A running instance of the [packie](https://gitea.semilabs.org/semios/packie.git) package manager on the build machines
|
||||
|
||||
### Build Agent (per machine)
|
||||
|
||||
- **Rust** >= 1.85 (edition 2024)
|
||||
- **Git**
|
||||
- The [ports](https://gitea.semilabs.org/semios/ports.git) repository cloned on the machine
|
||||
- `packie` installed and available in `$PATH`
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Clone and Build the Server
|
||||
|
||||
```bash
|
||||
git clone https://gitea.semilabs.org/semios/semios-build-server.git
|
||||
cd semios-build-server
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
### 2. Configure the Server
|
||||
|
||||
Copy and edit `config.toml` at the project root:
|
||||
|
||||
```toml
|
||||
[server]
|
||||
host = "0.0.0.0"
|
||||
port = 3000
|
||||
|
||||
[ports]
|
||||
repo_url = "https://gitea.semilabs.org/semios/ports.git"
|
||||
repo_path = "/path/to/ports-cache"
|
||||
sync_interval_secs = 300
|
||||
|
||||
[repo]
|
||||
publish_dir = "/path/to/repo-output"
|
||||
|
||||
[agent]
|
||||
token = "your-shared-secret-token"
|
||||
listen_port = 3001
|
||||
|
||||
[users]
|
||||
admin = "scrypt$<salt>$<hash>"
|
||||
```
|
||||
|
||||
| Key | Description |
|
||||
|-----|-------------|
|
||||
| `server.host` | Bind address (`0.0.0.0` for all interfaces) |
|
||||
| `server.port` | Web UI port |
|
||||
| `ports.repo_url` | Git URL of the ports repository |
|
||||
| `ports.repo_path` | Local path where the server clones/caches the ports repo |
|
||||
| `ports.sync_interval_secs` | How often to pull from upstream (seconds) |
|
||||
| `repo.publish_dir` | Where published packages are collected |
|
||||
| `agent.token` | Shared secret for agent authentication |
|
||||
| `agent.listen_port` | Port the server expects agents to expose (for display only) |
|
||||
| `users.*` | User accounts (username = scrypt password hash) |
|
||||
|
||||
### 3. Add Users
|
||||
|
||||
Only logged-in users can start builds, add packages to the repository, or delete artifacts.
|
||||
|
||||
Use the **registration helper** at `/register` to generate password hashes. For each user:
|
||||
|
||||
1. Open `http://localhost:3000/register` in a browser
|
||||
2. Enter username and password, click **Generate Config Snippet**
|
||||
3. Copy the `[users]` line into `config.toml`
|
||||
4. Restart the server
|
||||
|
||||
Or generate hashes from the command line:
|
||||
|
||||
```bash
|
||||
node -e "
|
||||
const crypto = require('crypto');
|
||||
const salt = crypto.randomBytes(16).toString('hex');
|
||||
const hash = crypto.scryptSync('your-password', salt, 64).toString('hex');
|
||||
console.log('scrypt\$' + salt + '\$' + hash);
|
||||
"
|
||||
```
|
||||
|
||||
### 3. Start the Server
|
||||
|
||||
```bash
|
||||
BODY_SIZE_LIMIT=52428800 node build/index.js
|
||||
```
|
||||
|
||||
Or use the npm script:
|
||||
|
||||
```bash
|
||||
npm run start
|
||||
```
|
||||
|
||||
The server listens on `http://0.0.0.0:3000` by default.
|
||||
|
||||
#### Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `BODY_SIZE_LIMIT` | - | Max request body size in bytes (set to `52428800` for artifact uploads) |
|
||||
| `PORT` | - | Override the listen port |
|
||||
|
||||
### 4. Build and Start the Agent
|
||||
|
||||
On each build machine:
|
||||
|
||||
```bash
|
||||
cd agent
|
||||
cargo build --release
|
||||
cp agent.toml /etc/semios-build-agent/agent.toml # or any location
|
||||
./target/release/semios-build-agent
|
||||
```
|
||||
|
||||
Edit `agent.toml` before running:
|
||||
|
||||
```toml
|
||||
master_url = "http://server-ip:3000"
|
||||
name = "worker-1"
|
||||
hostname = "192.168.1.100"
|
||||
port = 3001
|
||||
token = "your-shared-secret-token"
|
||||
ports_repo_path = "/path/to/ports-on-this-machine"
|
||||
```
|
||||
|
||||
| Key | Description |
|
||||
|-----|-------------|
|
||||
| `master_url` | URL of the build server |
|
||||
| `name` | Unique machine name (used for routing builds) |
|
||||
| `hostname` | Reachable hostname/IP for the server to connect back |
|
||||
| `port` | Port the agent listens on |
|
||||
| `token` | Must match the server's `agent.token` |
|
||||
| `ports_repo_path` | Path to the ports repo on this machine |
|
||||
|
||||
### 5. Register a Machine
|
||||
|
||||
Machines register automatically when they connect. You can also add them manually via the web UI at `/machines` or by calling the API:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/machines \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"name":"worker-1","hostname":"192.168.1.100","port":3001}'
|
||||
```
|
||||
|
||||
### 6. Load Packages
|
||||
|
||||
Click **Sync from Git** in the web UI (`/packages`) or call:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3000/api/sync
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────┐ ┌──────────────┐
|
||||
│ Web UI │◄───────►│ Build Server │
|
||||
│ (browser) │ HTTP │ Node.js │
|
||||
└──────────────┘ └──────┬───────┘
|
||||
│
|
||||
┌───────────┼───────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Agent A │ │ Agent B │ │ Agent C │
|
||||
│ (aarch64)│ │ (x86_64) │ │ (riscv64)│
|
||||
└──────────┘ └──────────┘ └──────────┘
|
||||
```
|
||||
|
||||
- **Build Server** — SvelteKit SSR app with SQLite database. Manages packages, builds, artifacts, and dispatches build jobs to agents.
|
||||
- **Build Agent** — Rust binary. Polls the server for pending builds, executes `./x --atombuild --target <target> build <package>`, streams logs in real-time, and uploads `.pkg` artifacts.
|
||||
|
||||
## File Layout
|
||||
|
||||
```
|
||||
~/.cache/semios-build/
|
||||
ports/ # cloned ports repo (server-side)
|
||||
repo/ # published packages output
|
||||
|
||||
project-root/
|
||||
config.toml # server configuration
|
||||
build-server.db # SQLite database (auto-created)
|
||||
uploads/ # build artifacts (build-<id>/)
|
||||
logs/ # build log files
|
||||
agent/
|
||||
agent.toml # agent configuration
|
||||
```
|
||||
|
||||
## Build Workflow
|
||||
|
||||
1. A user clicks **Build** on a package, selects architecture and atombuild option in the dialog.
|
||||
2. The server creates a build record with status `queued` and computes the target (e.g. `aarch64-semios-linux`).
|
||||
3. The dispatcher assigns the build to an idle machine with matching architecture.
|
||||
4. The agent clones/updates the ports repo, runs the build, and streams logs to the server via `POST /api/builds/<id>/log`.
|
||||
5. On completion, the agent discovers `.pkg` files and uploads them as artifacts.
|
||||
6. The server marks the build as `success` or `failed`.
|
||||
7. Artifacts can be added to the repository via **Add to Repo**, which calls `packie repo-admin --add-package`.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
npm run dev # starts vite dev server with HMR
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Fix |
|
||||
|---------|-----|
|
||||
| 500 errors on pages | Kill stale processes, rebuild with `rm -rf build .svelte-kit && npm run build`, restart |
|
||||
| Agent not receiving builds | Check `master_url`, `token`, and network connectivity |
|
||||
| Build fails immediately | Ensure `packie` is in `$PATH` and ports repo is cloned at `ports_repo_path` |
|
||||
| Artifact upload fails | Set `BODY_SIZE_LIMIT=52428800` (50 MB) on the server |
|
||||
| 401 on build/repo actions | Login required — use the sidebar login link or visit `/login` |
|
||||
| Cannot add users | Generate hash via `/register` helper or CLI snippet, add `[users]` section to `config.toml` |
|
||||
Reference in New Issue
Block a user