[arng4108]

🗣 Description

Add filters to report generation and shodan data collection scripts so that they only use Shodan findings with IPs that are contained within an organization's attested CIDRs.

💭 Motivation and context

We recently observed excessive shodan results that involved IP addresses that are not within an organization's attested CIDR blocks. This caused an issue with writing to excel (file too big) and the federal side also doesn't want to report non-CIDR ips to customers in general

🧪 Testing

Manually created sample data records to populate the CIDR, shodan_assets, and shodan_vulns tables locally. Then ran the updated report generation and shodan scripts to confirm the code is now filtering shodan results to only IPs within customer-owned CIDRs. Also updated unit tests to factor in filter changes

✅ Pre-approval checklist

• This PR has an informative and human-readable title.
• Changes are limited to a single goal - eschew scope creep!
• All future TODOs are captured in issues, which are referenced in code comments.
• All relevant type-of-change labels have been added.
• I have read the CONTRIBUTING document.
• These code changes follow cisagov code standards.
• All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
• Tests have been added and/or modified to cover the changes in this PR.
• AI was used for research and/or understanding purposes while creating this pull request.
• AI was used to generate code in this PR (either fully or in part).
  • I attest that I understand the AI-generated code and its functionality.
  • I am comfortable answering questions about the implementation of the AI-generated code.
• All new and existing tests pass.
• Bump major, minor, patch, pre-release, and/or build versions as appropriate via the bump_version script if this repository is versioned and the changes in this PR warrant a version bump.
• Create a pre-release (necessary if and only if the pre-release version was bumped).

✅ Pre-merge checklist

• Revert dependencies to default branches.
• Finalize version.

✅ Post-merge checklist

• Create a release (necessary if and only if the version was bumped).

[claires8]

Looks good!

[rapidray12]

Query_cidrs_by_org() returns an empty DataFrame on API/connection/JSON failure. With the new logic, a CIDR API outage becomes indistinguishable from "this org has no attested CIDRs". THis causes the scan to skip all IPs and remove all Shodan findings.

A check for API returns and a logged error need to be added

Also not sure about how old/inactive CIDRS are handled. We need to confirm that old CIDR (inactive) blocks are not reported as active

[arng4108]

I'll look into those query_cidrs_by_org() concerns.

As for old/inactive CIDRs, that's handled by the ASM Sync process. Every time the ASM Sync runs it adds in the latest CIDRs from the CyHy MongoDB database and updates the "last_seen" field. Any CIDR records who have a "last_seen" date > 15 days old gets marked as inactive and are not used in P&E scripts/reports.

[arng4108]

Updated query_cidrs_by_org() error handling and edge cases

[rapidray12]

Issues with handling API failures has been fixed.

[ameliav]

Spoke with Alex about a high finding from DHS chat which he will fix. Basically, he just needs to add a way to ensure that when null/none data is returned, it won't error out looking for the IP column.

[arng4108]

Updated code based on Amelia's feedback so that it explicity checks for the presence of an "ip" column in query_shodan()'s output

[ameliav]

Finding has been addressed so LGTM

[rapidray12]

Earlier status return issues have been fixed,

LGTM