You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 88d76ed
Browse filesBrowse the repository at this point in the historyBrowse files
logger.warn('Could not extract vendor domain for URL validation',{
66
+
vendorWebsite,
67
+
});
68
+
returnnull;
69
+
}
70
+
61
71
constprompt=`Complete cyber security research on the vendor "${vendorName}" with website ${vendorWebsite}.
62
72
73
+
CRITICAL: Only return URLs that belong to the domain "${vendorDomain}" or its subdomains (e.g., trust.${vendorDomain}, security.${vendorDomain}). Do NOT return URLs from any other domain. If you cannot find a page on ${vendorDomain}, return an empty string for that field rather than a URL from another website.
74
+
63
75
Extract the following information:
64
76
1. **Certifications**: Find any security certifications they have (SOC 2 Type I, SOC 2 Type II, ISO 27001 etc). For each certification found, determine:
65
77
- The type of certification
66
78
- Whether it's verified/current, expired, or not certified
67
79
- Any issue or expiry dates mentioned
68
-
- Link to the compliance/trust page or report if available
80
+
- Link to the compliance/trust page or report if available (must be on ${vendorDomain})
69
81
70
-
2. **Legal & Security Documents**: Find the direct URLs to:
82
+
2. **Legal & Security Documents**: Find the direct URLs on ${vendorDomain}to:
71
83
- Privacy Policy page (usually at /privacy, /privacy-policy, or linked in the footer)
72
84
- Terms of Service page (usually at /terms, /tos, /terms-of-service, or linked in the footer)
73
-
- Trust Center or Security page (typically could be at /trust, /security or trust.website.com or security.website.com)
85
+
- Trust Center or Security page (typically could be at /trust, /security or trust.${vendorDomain} or security.${vendorDomain})
74
86
75
87
3. **Recent News**: Find recent news articles (last 12 months) about the company, especially:
76
88
- Security incidents or data breaches
@@ -81,13 +93,26 @@ Extract the following information:
81
93
82
94
4. **Summary**: Provide an overall assessment of the vendor's security posture.
83
95
84
-
Focus on their official website (especially trust/security/compliance pages), press releases, and reputable news sources.`;
96
+
Focus on their official website ${vendorWebsite} (especially trust/security/compliance pages), press releases, and reputable news sources.`;
97
+
98
+
// Provide seed URLs covering common legal/security paths so the agent
99
+
// stays on the vendor's domain instead of wandering to unrelated sites.
100
+
constseedUrls=[
101
+
origin,
102
+
`${origin}/privacy`,
103
+
`${origin}/privacy-policy`,
104
+
`${origin}/terms`,
105
+
`${origin}/terms-of-service`,
106
+
`${origin}/security`,
107
+
`${origin}/trust`,
108
+
`${origin}/legal`,
109
+
`${origin}/compliance`,
110
+
];
85
111
86
-
// Using SDK (no maxCredits override, no explicit polling here)
87
-
// Important: avoid crawling huge sites with a wildcard (e.g. workspace.google.com).
88
112
constagentResponse=awaitfirecrawlClient.agent({
89
113
prompt,
90
-
urls: [origin],
114
+
urls: seedUrls,
115
+
strictConstrainToURLs: false,// allow following links from seed URLs, but seeds anchor it to the right domain
91
116
schema: {
92
117
type: 'object',
93
118
properties: {
@@ -173,7 +198,10 @@ Focus on their official website (especially trust/security/compliance pages), pr
logger.warn('Could not extract vendor domain for URL validation',{
56
+
website,
57
+
});
58
+
returnnull;
59
+
}
51
60
constapiKey=process.env.FIRECRAWL_API_KEY;
52
61
if(!apiKey){
53
62
logger.warn(
@@ -77,9 +86,11 @@ export async function firecrawlExtractVendorData(
77
86
78
87
Goal: return the MOST SPECIFIC, DIRECT URL for each document type below. Do not return general category pages.
79
88
80
-
You may crawl the site (including subdomains) and follow internal links. Trust portals are often linked in the header/footer under: "Trust", "Trust Center", "Security", "Compliance", "Legal", "Governance", "Privacy", "Data Processing", "DPA".
89
+
CRITICAL: Only return URLs that belong to the domain "${vendorDomain}" or its subdomains (e.g., trust.${vendorDomain}, security.${vendorDomain}). Do NOT return URLs from any other domain. If you cannot find a page on ${vendorDomain}, return an empty string for that field.
81
90
82
-
Return ONLY absolute https URLs. If you cannot find a dedicated page that matches the definition, return an empty string.
91
+
You may crawl the site (including subdomains of ${vendorDomain}) and follow internal links. Trust portals are often linked in the header/footer under: "Trust", "Trust Center", "Security", "Compliance", "Legal", "Governance", "Privacy", "Data Processing", "DPA".
92
+
93
+
Return ONLY absolute https URLs on ${vendorDomain}. If you cannot find a dedicated page that matches the definition, return an empty string.
83
94
84
95
DEFINITIONS (be strict):
85
96
1) trust_portal_url:
@@ -193,13 +204,46 @@ When multiple candidates exist, choose the most direct URL that best matches the
193
204
returnnull;
194
205
}
195
206
207
+
// Normalize URLs and filter out any that don't belong to the vendor's domain
0 commit comments