Skip to content
Merged

Dev #320

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,10 @@ jobs:
src/services/__tests__/csrf.spec.ts
src/stores/__tests__/auth.spec.ts
src/views/__tests__/Login.spec.ts
src/views/manage/__tests__/ContactPersons.spec.ts
src/views/manage/__tests__/ApplicationSettings.spec.ts
src/views/public/__tests__/ReportAccess.spec.ts
src/__tests__/router.spec.ts

- name: Build frontend
working-directory: ./src/frontend/app
Expand Down
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ It contains a `docker-compose.release.yml` pinned to the exact released image
digests, an environment example, checksums, image metadata, and upgrade
instructions. No source checkout or local image build is required.

Backend startup automatically widens `scan_host_technologies.location_name` to `LONGTEXT` on existing MariaDB installations, preserving complete host-technology location lists. The upgrade is idempotent, keeps existing values, and requires the application database user to have `ALTER` permission. Back up the database before upgrading; the first startup may take longer while this table is altered. Re-import reports that previously failed with a location-length error after deploying the updated backend.

The commands below start a development checkout:

```bash
Expand Down Expand Up @@ -77,6 +79,12 @@ reachable from containers at `ldap://ldap:389` and from the host at

Upon first usage or database refresh, no user is available in the backend. Using the frontend you will be requested to setup an initial admin account which you can later on remove if need be.

### Report notification generations

By default, synchronized Greenbone reports notify only for the newest imported report generation of each task. In **Application Settings > Email Automation**, turn **Only notify for the latest report per scan task** on or off and click **Save**; no restart is required. Turning it off allows older reports under the normal notification, access, and retention rules. Different task IDs remain independent even when they share a target. When enabled, older generations stay skipped even if they were never delivered or a contact was assigned later; historical resend requests follow the same rule. Per-host and institution exports belonging to the newest report remain eligible. Reports without a known Greenbone task ID are not grouped by filename, target, or IP, to avoid suppressing unrelated scans.

Application Settings includes a search field matching names, descriptions, setting keys, and section names. Search for "latest report" to find this switch directly; filtering preserves unsaved changes.

## Documentation

📚 **[Read the Full Documentation](https://vulnerabilityhub.readthedocs.io/en/latest/)**
Expand Down
1 change: 1 addition & 0 deletions deploy/release/production.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ SMTP_SENDER=VulnerabilityHub <noreply@example.com>
SMTP_USE_TLS=false
SMTP_USE_STARTTLS=true
SMTP_TIMEOUT_SECONDS=10
REPORT_NOTIFICATIONS_LATEST_ONLY=true

# Optional LDAP defaults
LDAP_ENABLED=false
Expand Down
49 changes: 13 additions & 36 deletions docs/development/development.rst
Original file line number Diff line number Diff line change
Expand Up @@ -566,42 +566,19 @@ Frontend Debugging
}


Database Migrations
-------------------


Manual Migration
~~~~~~~~~~~~~~~~


1. Update ``database/vulnerability_scanner.sql``
2. Create migration script:

.. code-block:: sql

-- migrations/001_add_my_feature.sql
ALTER TABLE my_table ADD COLUMN new_field VARCHAR(255);


3. Apply migration:

.. code-block:: bash

mariadb -u root -p vulnerabilityhub < migrations/001_add_my_feature.sql


Future: Alembic Integration
~~~~~~~~~~~~~~~~~~~~~~~~~~~


Consider adding Alembic for automated migrations:

.. code-block:: bash

pip install alembic
alembic init alembic
alembic revision --autogenerate -m "Add my feature"
alembic upgrade head
Database Schema
---------------

``src/database/vulnerability_scanner.sql`` is the source of truth for the current
schema and built-in settings. Update this script together with the ORM models
when changing the database layout. The application and demo-data generator assume
an up-to-date database; they do not create tables, alter columns, or insert missing
setting definitions at runtime.

Database initialization runs when provisioning a new database. Changing the SQL
script or restarting the application does not update an existing database volume.
Provision the required schema outside the application before starting it. Do not
delete a production database volume to apply a schema change.


Performance Optimization
Expand Down
1 change: 1 addition & 0 deletions docs/getting_started/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ These settings control when report notification emails are sent automatically.
They can also be managed from **Manage > Configuration > Application Settings**.

- ``AUTO_EMAIL_ENABLED``: Master switch for automated report emails (default: ``true``).
- ``REPORT_NOTIFICATIONS_LATEST_ONLY``: Notify only for the newest imported Greenbone report per task, including late contact assignments and historical resends (default: ``true``). Disable to allow older generations under normal delivery rules. The GUI override takes effect without a restart.
- ``AUTO_EMAIL_ON_REPORT_MATCH``: Send report emails when uploaded XML and visual reports are matched (default: ``true``).
- ``AUTO_EMAIL_ON_CONTACT_CREATE_ASSIGNMENT``: Send report emails when a newly created contact is assigned existing IPs or CIDRs (default: ``true``).
- ``AUTO_EMAIL_ON_CONTACT_UPDATE_ASSIGNMENT``: Send report emails when an existing contact receives newly assigned IPs or CIDRs (default: ``true``).
Expand Down
1 change: 1 addition & 0 deletions example.env
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ SMTP_SENDER=VulnerabilityHub <noreply@localhost>
SMTP_USE_TLS=false
SMTP_USE_STARTTLS=false
SMTP_TIMEOUT_SECONDS=10
REPORT_NOTIFICATIONS_LATEST_ONLY=true

# Development helpers
MAILHOG_SMTP_PORT=1025
Expand Down
1 change: 1 addition & 0 deletions src/backend/core/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ class Settings(BaseSettings):

# Automated report email triggers
AUTO_EMAIL_ENABLED: bool = True
REPORT_NOTIFICATIONS_LATEST_ONLY: bool = True
AUTO_EMAIL_ON_REPORT_MATCH: bool = True
AUTO_EMAIL_ON_CONTACT_CREATE_ASSIGNMENT: bool = True
AUTO_EMAIL_ON_CONTACT_UPDATE_ASSIGNMENT: bool = True
Expand Down
10 changes: 9 additions & 1 deletion src/backend/crud/contact_persons.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
from models.user import User
from schemas.contact_person import ContactPersonCreate, ContactPersonUpdate
from services.application_settings import is_auto_email_enabled
from services.latest_reports import latest_notification_scans

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -1126,8 +1127,15 @@ def notify_contact_for_unnotified_scans(
scans_map[scan.id] = {"scan": scan, "ips": set()}
scans_map[scan.id]["ips"].add(ip)

# 2. Process each unique scan
latest_ids = {
scan.id for scan in latest_notification_scans(
db, [data["scan"] for data in scans_map.values()]
)
}
# 2. Process only the latest generation of each independent task.
for scan_id, data in scans_map.items():
if scan_id not in latest_ids:
continue
scan = data["scan"]
relevant_ips = list(data["ips"])

Expand Down
1 change: 0 additions & 1 deletion src/backend/generate_demo_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -463,7 +463,6 @@ def create_demo_data():
db = SessionLocal()
try:
print("Creating demo data...")
ReportFileMatch.__table__.create(db.get_bind(), checkfirst=True)
create_demo_ldap_data(db)

# 1. Create University Users and Contact Persons
Expand Down
2 changes: 0 additions & 2 deletions src/backend/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -171,10 +171,8 @@ async def startup_event():
from crud.report_matches import repair_matched_visual_scan_dates
from services.application_settings import apply_persisted_settings
from services.greenbone_service import ensure_greenbone_credential_key
from services.notification_jobs import ensure_notification_job_table
from core.db import SessionLocal

ensure_notification_job_table()
db = SessionLocal()
try:
apply_persisted_settings(db)
Expand Down
4 changes: 3 additions & 1 deletion src/backend/models/parser_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
Text,
UniqueConstraint,
)
from sqlalchemy.dialects.mysql import LONGTEXT
from sqlalchemy.orm import relationship

from models.base import Base
Expand Down Expand Up @@ -178,7 +179,8 @@ class ScanHostTechnology(Base):
host_ip = Column(String(128), nullable=False, index=True)
technology_type = Column(String(32), nullable=False)
technology_name = Column(String(512), nullable=False)
location_name = Column(String(1024))
# A technology can be observed at thousands of paths in one scan.
location_name = Column(Text().with_variant(LONGTEXT(), "mysql", "mariadb"))
source_name = Column(String(255))
source_oid = Column(String(1024))

Expand Down
105 changes: 87 additions & 18 deletions src/backend/routes/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,63 @@ def _require_admin(current_user: Any):
raise HTTPException(status_code=403, detail="Not authorized")


def _get_public_access_network_context(
scan: Scan, db: Session, report_group: Optional[Dict[str, Any]]
) -> Tuple[List[int], List[str], List[str]]:
"""Describe the full public delivery without relying on a user's mappings."""
effective_scan, fallback_ips = _get_access_scan_and_ips(scan, db)
if report_group and report_group.get("source") == "greenbone":
group_files = [
item for item in report_group.get("files", [])
if item.get("status") == "completed"
]
scan_ids = {item["scan_id"] for item in group_files}
host_ips = {item["host_ip"] for item in group_files if item.get("host_ip")}
family_scans = db.query(Scan).filter(Scan.id.in_(scan_ids)).all()
else:
family_scans = crud_scans.get_related_report_scans(db, scan)
scan_ids = {item.id for item in family_scans}
host_ips = set()

scan_ids.add(effective_scan.id)
host_ips.update(fallback_ips)
host_ips.update(
row[0] for row in db.query(ScanResult.host_ip)
.filter(ScanResult.scan_id.in_(scan_ids)).distinct().all()
if row[0]
)
host_ips.update(item.target_ip for item in family_scans if item.target_ip)
parsed_ips = {}
for value in host_ips:
try:
parsed_ips[str(ipaddress.ip_address(value))] = ipaddress.ip_address(value)
except ValueError:
logger.warning("Skipping invalid report host IP: %s", value)

institution_ids = {item.institution_id for item in family_scans}
matching_networks = {}
for (cidr,) in db.query(InstitutionCIDR.cidr_range).filter(
InstitutionCIDR.institution_id.in_(institution_ids)
).all():
try:
network = ipaddress.ip_network(cidr, strict=False)
except ValueError:
logger.warning("Skipping invalid institution CIDR: %s", cidr)
continue
if any(address in network for address in parsed_ips.values()):
matching_networks[str(network)] = network

uncovered_ips = [
value for value, address in parsed_ips.items()
if not any(address in network for network in matching_networks.values())
]
return (
sorted(scan_ids),
sorted(parsed_ips),
[*sorted(matching_networks), *sorted(uncovered_ips)],
)


def _get_dispute_or_404(db: Session, dispute_id: int) -> Dispute:
dispute = (
db.query(Dispute)
Expand Down Expand Up @@ -828,14 +885,17 @@ def _authorized_visual_bundle_files(
)


def _public_visual_bundle_files(db: Session, anchor_scan: Scan) -> List[Dict[str, Any]]:
"""List variants covered by the exact public link without widening scope.
def _public_visual_bundle_files(
db: Session, anchor_scan: Scan, report_group: Optional[Dict[str, Any]] = None
) -> List[Dict[str, Any]]:
"""List all visual variants in the public link's concrete scan execution.

A per-host visual link may reveal sibling formats for that same host, while
a master visual link may reveal only other master formats. An XML anchor is
similarly limited to master visuals. This avoids turning one host's link
into an index of every per-host artifact in the report family.
Public access grants the report delivery, not just the initially selected
host/format. Greenbone groups identify one report execution (not a task)
and also cover exports attached to separate per-host XML slices.
"""
if report_group is None:
report_group = _report_group_for_scan(db, anchor_scan.id)
anchor_path = (getattr(anchor_scan, "file_path", "") or "").lower()
anchor_match = None
xml_scan_id = anchor_scan.id if anchor_path.endswith(".xml") else None
Expand All @@ -845,7 +905,14 @@ def _public_visual_bundle_files(db: Session, anchor_scan: Scan) -> List[Dict[str
xml_scan_id = anchor_match.xml_scan_id

candidates: List[Tuple[Scan, Optional[str]]] = []
if xml_scan_id:
if report_group and report_group.get("source") == "greenbone":
candidates.extend(
(crud_scans.get_scan(db, item["scan_id"]), item.get("host_ip"))
for item in report_group.get("files", [])
if item.get("format") in {"PDF", "HTML"}
and item.get("status") == "completed"
)
elif xml_scan_id:
matches = (
db.query(ReportFileMatch)
.filter(
Expand All @@ -854,11 +921,10 @@ def _public_visual_bundle_files(db: Session, anchor_scan: Scan) -> List[Dict[str
)
.all()
)
anchor_host_ip = getattr(anchor_match, "host_ip", None)
candidates.extend(
(match.visual_scan, match.host_ip)
for match in matches
if match.visual_scan is not None and match.host_ip == anchor_host_ip
if match.visual_scan is not None
)
else:
candidates.extend(
Expand Down Expand Up @@ -1157,21 +1223,22 @@ def get_public_report_metadata(
if not is_expired:
retention_remaining = _report_retention_delta() - time_since_access

effective_scan, ip_list = _get_access_scan_and_ips(scan, db)
ip_list = sorted(set(ip_list))
report_group = _report_group_for_scan(db, scan.id)
scan_ids, ip_list, network_scopes = _get_public_access_network_context(
scan, db, report_group
)
vuln_query = (
db.query(Vulnerability)
.join(ScanResult)
.filter(ScanResult.scan_id == effective_scan.id)
.filter(ScanResult.scan_id.in_(scan_ids))
)
if ip_list:
vuln_query = vuln_query.filter(ScanResult.host_ip.in_(ip_list))
vuln_count = vuln_query.count()
report_group = _report_group_for_scan(db, scan.id)
report_name = (
report_group.get("title") if report_group else None
) or scan.report_id
files = _public_visual_bundle_files(db, scan)
files = _public_visual_bundle_files(db, scan, report_group)

return {
"report_id": scan.report_id,
Expand All @@ -1181,7 +1248,7 @@ def get_public_report_metadata(
"access_mode": "magic_link",
"scan_date": scan.scan_date.isoformat() if scan.scan_date else None,
"ip_addresses": ip_list,
"network_scopes": ip_list,
"network_scopes": network_scopes,
"vulnerability_count": vuln_count,
"last_accessed_at": (
effective_last_accessed_at.isoformat()
Expand Down Expand Up @@ -1414,16 +1481,19 @@ def download_public_single_host_reports(
request: Request,
db: Session = Depends(get_db),
):
"""Download the same-host variants covered by an unauthenticated link."""
"""Download all single-host variants from the public link's delivery."""
if not download_request.confirm_responsibility:
raise HTTPException(status_code=400, detail="Must confirm responsibility")
_validate_public_magic_link_hash(file_hash)
scan = crud_scans.get_scan_by_hash(db, file_hash)
if not scan:
raise HTTPException(status_code=404, detail="Report not found")

report_group = _report_group_for_scan(db, scan.id)
host_files = [
file for file in _public_visual_bundle_files(db, scan) if file.get("host_ip")
file
for file in _public_visual_bundle_files(db, scan, report_group)
if file.get("host_ip")
]
host_scans = [
visual_scan
Expand All @@ -1441,7 +1511,6 @@ def download_public_single_host_reports(
detail="One or more report files are no longer available.",
)

report_group = _report_group_for_scan(db, scan.id)
archive_path = _create_single_host_reports_archive(host_scans, report_group)
try:
_validate_report_download_retention(scan, db)
Expand Down
13 changes: 13 additions & 0 deletions src/backend/services/application_settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,19 @@
from models.application_setting import ApplicationSetting

logger = logging.getLogger(__name__)
LATEST_REPORT_ONLY_KEY = "REPORT_NOTIFICATIONS_LATEST_ONLY"


def latest_report_only_enabled(db: Session) -> bool:
"""Read persisted policy on each send so workers do not use stale settings."""
definition = _get_definition(db, LATEST_REPORT_ONLY_KEY)
if not isinstance(definition, ApplicationSetting):
return settings.REPORT_NOTIFICATIONS_LATEST_ONLY
return bool(
_deserialize(definition.value, definition)
if definition.value is not None
else _default_setting_value(definition)
)


def _jsonable(value: Any) -> Any:
Expand Down
Loading
Loading