Requirements

1. docker
2. docker-compose

Getting started

# create env files, for instance 
# tee api/.env api/.env.production < api/.env >/dev/null

# follow getting started instructions in api/

# create external network
docker network create public

# workaround: fix dsiem file permission issue
mkdir -p dsiem/ && touch dsiem/siem_alarms.json dsiem/siem_alarm_events.json

# start services
make api
make dashboard
make fluentd

# optionally, add directives using the dashboard then send events using ../plugins

Example Commands

# run api only
# --debug=false Disable noisy sqlite logs
SLIM=1 go run . serve --debug=false

# test api in docker without Elasticsearch
SLIM=1 docker compose --profile staging up --no-deps api

# refresh fluentd config
# vim fluentd/conf/fluent.conf
./remote rsync
./remote compose restart fluentd

# update dashboard/api
./remote compose up -d --build dashboard
./remote compose up -d --build api

# show latest logs
# --timestamps -t show timestamps | --tail 50
./remote compose logs --since 1h

# clone dsiem directives to api.dsiem fixtures
rsync -r --include="***/directives_*.json" --exclude="*"  ../dsiem/conf/ ./dsiem/directive/fixtures/

# check if every .env file has a sibling .env.example
find . -not -path "*/.git/*" -not -path "*/node_modules/*" -type f -name ".env" -exec sh -c 'test ! -e "${1%/*}/.env.example"' _ {} \; -print

Deploy

# set up external reverse proxy, e.g. traefik

chmod u+x ./remote

export SERVER_USERNAME=ubuntu
export SERVER_ALIAS=pet

# ./remote rsync -d # preview files that will be rsync'ed
./remote setup
./remote rsync

./remote compose config # preview compose file
./remote compose up

# configure pocketbase - admin account, application name/url, smtp, import schema*

Flowchart

graph LR
    fb[fluentbit] --> |events|fd[fluentd] --> |normalized events|ds[dsiem] --> |alarms<br>via fluentd|es[(elastic<br>search)]
    fd--> |normalized events|es

Loading

Spec

• Ingest logs from fluentbit
• Normalize logs to dsiem spec using fluentd processing
• Send normalized events to dsiem using http
• Send normalized events to elasticsearch
• Generate alarms
• Read dsiem output (siem_alarms.json) using fluentd tail
• Send alarms to elasticsearch
• Create web interface for elasticsearch

Docs

• Fluentd Docker Setup