Skip to content

harden: use yaml.safe_load in generate_index_of_sub_section.py... - #398

Open
anupamme wants to merge 175 commits into
mat3ra:devfrom
anupamme:fix-repo-documentation-bandit-b506-yaml-safe-load
Open

harden: use yaml.safe_load in generate_index_of_sub_section.py...#398
anupamme wants to merge 175 commits into
mat3ra:devfrom
anupamme:fix-repo-documentation-bandit-b506-yaml-safe-load

Conversation

@anupamme

Copy link
Copy Markdown

Summary

Harden input handling in scripts/generate_index_of_sub_section.py (flagged by semgrep).

Vulnerability

Field Value
ID gitlab.bandit.B506
Severity HIGH
Scanner semgrep
Rule gitlab.bandit.B506
File scripts/generate_index_of_sub_section.py:61
Assessment Defensive hardening

Description: The application was found using an unsafe version of yaml load which is vulnerable to
deserialization attacks. Deserialization attacks exploit the process of reading serialized
data and turning it back
into an object. By constructing malicious objects and serializing them, an adversary may
attempt to:

  • Inject code that is executed upon object construction, which occurs during the
    deserialization process.
  • Exploit mass assignment by including fields that are not normally a part of the serialized
    data but are read in during deserialization.

To remediate this issue, use safe_load() or call yaml.load() with the Loader argument
set to
yaml.SafeLoader.

Example loading YAML using safe_load:

import yaml

# Use safe_load to load data into an intermediary object
intermediary_object = yaml.safe_load("""user:
    name: 'test user'"""
)
# Create our real object, copying over only the necessary fields
user_object = {'user': {
        # Assign the deserialized data from intermediary object
        'name': intermediary_object['user']['name'],
        # Add in protected data in object definition (or set it from a class constructor)
        'is_admin': False,
    }
}
# Work with user_object
# ...

For more details on deserialization attacks in general, see OWASP's guide:

Threat Model Context

This is a web service - vulnerabilities in request handlers are directly exploitable by remote attackers.

Changes

  • scripts/generate_index_of_sub_section.py

Behavior Preservation

The change is scoped to 1 file on the vulnerable path; it only tightens handling of untrusted input and leaves valid inputs unaffected.


This patch removes an exploit primitive — a code pattern that, while not independently exploitable today, could be chained with other weaknesses by automated exploit-development tooling. Proactive removal of such primitives raises the bar against increasingly capable automated attack tools.


Automated security fix by OrbisAI Security

timurbazhirov and others added 30 commits August 7, 2025 19:26
chore/SOF 7666 - Migration to New Platform, New Infrastructure Specs
timurbazhirov and others added 30 commits July 31, 2026 18:20
…sion

M5 is done and the agent is deployed and answering, so only the launch gate
remains. Records what was verified against the deployed service rather than
locally, including the rollback drill, and three findings worth not
rediscovering: a commit SHA cannot be passed to git clone --branch, so
pinning the index silently needed a different checkout; builds submit --tag
cannot pass the build argument that pinning requires; and a brand-new
project fails its first deploy on permission propagation rather than on any
policy.

The consequence for sequencing is now stated plainly. The service is live
and the widget appears only once it answers a health check, so merging the
widget to main is the launch, and it waits on the hardening gate rather
than on anything technical.

Marks D3 done - Vertex confirmed end to end from Cloud Run - and D8's
refusal bar as reached rather than pending.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three things the first real use made obvious.

The Mat3ra mark now sits beside the "Ask AI" label in the launcher and the
panel header. It is inlined and drawn in currentColor, so it works on the
purple launcher and the light header without shipping two assets or
depending on an image path that exists only on the documentation site.

Source URLs are now links. The model lists them as bare URLs and the
renderer only understood the bracketed Markdown form, so every citation
arrived as dead text - a source you cannot click is not much of a
citation.

Emphasised product terms - Materials Bank, Materials Designer - now link
to the page that defines them, which is what a reader wants to open. The
mapping comes from the service, which derives it from its index, so a link
can only ever name a page that exists; the model is never asked to produce
one. Terms stay styled as emphasis with a dotted underline rather than
turning the answer into a field of links.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sources and glossary terms opened in a new tab. A documentation link is
the continuation of the answer rather than a detour, and spawning tabs
behind the reader is a habit the documentation itself does not have.

The consequence is worth stating: the conversation is held in memory only,
so following a link ends it, and the back button returns to the
documentation page rather than to the exchange.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sources and glossary terms rendered in the surrounding text colour, so
nothing signalled they could be clicked. They now use the documentation's
own link colour where the theme defines one, and a conventional blue
wherever else the widget is embedded, since the service's try-it page and
the platform shell have no such variable.

This drops the earlier treatment of glossary terms as emphasis with a
dotted underline. Reading as prose is worth less than being visibly
clickable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Answers cite documentation pages and those links now open in place, so
every citation was ending the exchange that produced it: the reader
followed a source, the page reloaded, and the panel came back empty. The
conversation now outlives the page.

It is stored in the browser, restored on load, and the panel reopens if it
was open, so following a link reads as continuing rather than starting
over. Storage is bounded the same way the service bounds a request - the
last twenty messages, capped by total size - and expires after a week,
which is what makes "relive through multiple visits" tolerable rather than
indefinite.

A conversation that survives navigation also has to be endable, so the
header gains "New chat" and the footer says plainly that the exchange is
kept in the browser until then. Nothing new is sent anywhere: this is text
the page already displayed, on the reader's own machine.

Restoration waits for the glossary, so a restored answer gets the same
links a fresh one would. Storage being full or disabled is not an error -
the widget simply behaves as it did before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Following a source on a deploy preview jumped to production
docs.mat3ra.com, because the corpus stores canonical URLs and the widget
rendered them verbatim. That left the preview entirely, and took the
conversation with it - stored sessions are per-origin, so the panel came
back empty on the other side.

A documentation URL is now rewritten to the origin currently serving the
documentation, which is a no-op in production and makes previews and local
servers behave like it. Rewriting only happens where the widget mounts
itself onto a documentation build; the platform shell serves no
documentation, so its citations keep pointing at the real site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The widget ships to every documentation page, and until now nothing
guarded it. Each fix in it so far was verified by hand once and then left
unprotected, which is how the same bug comes back.

Seventeen Playwright tests run in under three seconds and need neither
cloud access nor a model call. The widget under test is the file the site
ships; only the agent service is faked, at the network boundary, because a
test that called a language model would be neither deterministic nor free
and the service has its own suite.

The fixture server answers every path with a page that mounts the widget,
which is what makes the navigation tests possible: answers cite canonical
production URLs that the widget rewrites onto the origin being read, so
following a citation has to land somewhere the widget mounts again. Both
mounting styles are covered, since they differ - the documentation site
lets the script mount itself only after a health check, while the platform
will mount it explicitly.

Coverage is deliberately weighted towards the properties that are easy to
break silently: markup in an answer is displayed rather than executed, a
non-https link never becomes clickable, unknown emphasis stays plain
rather than linking somewhere plausible, and the conversation survives a
followed citation without resurrecting a week-old one.

Both regressions these tests were written for were reintroduced on purpose
and confirmed to fail the run, so the suite is known to have teeth rather
than assumed to. The README says to do the same when adding to it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
M4 is finished and now has browser tests, so the plan records what the
first real use taught rather than only that it shipped: citations had to
become clickable and same-tab, emphasised terms link through a glossary
the service derives from its index, citations must stay on the build being
read, and the conversation has to outlive the page once links open in
place.

Adds the local Google Cloud SDK to the outstanding items. It is still the
June 2023 release, and Google rejects that client's token refresh, so a
fresh login lasts about an hour before every call fails. Re-authenticating
treats it hourly; upgrading fixes it. Nothing deployed depends on it,
since Cloud Run authenticates as its own service account, but no deploy
can be driven from a laptop in this state.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three stacked causes, not the one the plan guessed: the machine driving
builds is a different computer from the laptop whose logins kept not
helping; its 2023-era gcloud minted tokens Google began rejecting outright
on 2026-08-01; and the account session now requires interactive
reauthentication, which no non-interactive shell can satisfy.

Deploys run again - a current SDK from a local directory, fed tokens
minted from application-default credentials, which still refresh. CI with
federated identity bypasses the whole class of problem, which is one more
reason to merge the pipeline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reproduces the electronic-structure result of Liu et al., Nat. Commun. 5,
4966 (2014): the indirect gap of a MoS2 bilayer is set by the interlayer
distance, while the K-valley direct gap barely moves.

Covers the prerequisite structure notebook, the structures computed, where
the settings differ from the manuscript's and why, and the results.

Registered in both mkdocs.yml and mkdocs-guide.yml, as the sibling
simulation tutorial is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The out-of-plane cell is now 20 A as in the manuscript, so it is no longer
a divergence. The cutoff row is: 140 Ry is a norm-conserving wavefunction
cutoff and is not comparable with an ultrasoft one, which converges lower
and needs a higher charge-density cutoff instead.

Drop the walkthrough of the Mo-Mo to sulfur-plane conversion; the notebook
prints the resulting separation and cell height for checking.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The band-structure page did not read as the second half of the MoS2
manuscript reproduction:

- nav titled it "Twisted Bilayer MoS2 Band Structure" instead of the
  structure page's title plus "(Band Structure)", the form already used
  for the graphene pair
- in mkdocs-guide.yml it sat seven rows above its own structure page,
  directly under the graphene pair
- overview.md still advertised its Properties entry as "Coming Soon"

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows the _SIMULATION rename in api-examples. Two references in the
page - the GitHub source path and the jupyterlite_embed notebook_name.
The page slug is unchanged, so no published URL moves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cost-lever paragraph still named INTERFACE_VACUUM, which no longer
exists - the structure notebook takes TOTAL_CELL_HEIGHT and derives the
builder's vacuum from it. It also framed the cell as more generous than
the manuscript's, which stopped being true when it moved to 20 A.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sections 3 and 4 documented create_twisted_interface_with_commnesurate_
lattices.ipynb, while section 5 embeds
specific_examples/interface_bilayer_twisted_commensurate_lattices_
molybdenum_disulfide.ipynb. None of the variables the page told the
reader to set - TARGET_TWIST_ANGLE, INTERFACE_DISTANCE, INTERFACE_VACUUM,
MAX_SUPERCELL_MATRIX_INT, USE_CONVENTIONAL_CELL, STACKING_DIRECTION -
exist in the embedded notebook, so the prerequisite step of the band
structure tutorial could not be followed as written.

Rewritten against the embedded notebook: the INTERFACE_PARAMETERS list
replaces the one-angle-at-a-time walkthrough, TOTAL_CELL_HEIGHT replaces
INTERFACE_VACUUM, and the six angle snippets become uncommented entries.

Two things the old text got wrong about the physics, now called out:
d_mo_mo is the Mo-Mo separation Table S1 tabulates, not a gap - passing
6.5 A through as a gap separates the layers ~3 A too far; and the
interlayer distance belongs to the stacking registry, 6.1-6.2 A for
AA1/AB1 and AA2/AB2 against 6.8 A for AA3/AB3.

Atom counts, Mo-Mo separations, cell heights and the two registry
classifications in the new text were built and measured, not inferred.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SOF-8014: tutorial for twisted MoS2 bilayer band structure
Automated security fix generated by OrbisAI Security
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants