Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions src/content/docs/reference/blobs/page-metadata.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,26 @@ JSON-encoded HTTP Archive metadata about the page that was tested.
```
</details>

:::note
There is lots of other page-level metadata in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page.
Comment thread
tunetheweb marked this conversation as resolved.
Outdated
Comment thread
tunetheweb marked this conversation as resolved.
Outdated
Comment thread
tunetheweb marked this conversation as resolved.
Outdated

Always check if the information you need exists there before using the `response_body` column. Using the custom metrics from the `pages` table is likely less brittle than attempting to use regular expressions on HTML, includes the rendered DOM after JavaScript runs, is much cheaper to query, and so is considerably faster to get back results.
Comment thread
tunetheweb marked this conversation as resolved.
Outdated

For example, to get the `title` and `meta_description` you can use:
```sql
SELECT
page,
custom_metrics.wpt_bodies.title.rendered.primary.text AS title,
custom_metrics.wpt_bodies.meta_description.rendered.primary.text AS meta_description.
FROM
`httparchive.crawl.pages`
WHERE
...
```

The [source code](https://github.com/HTTPArchive/custom-metrics/) of the custom metrics can be useful to understand exactly how they are collected.
:::

## Schema

### `crawl_depth`
Expand Down
20 changes: 20 additions & 0 deletions src/content/docs/reference/tables/requests.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,26 @@ See the [Header](/reference/structs/header/) reference for more details.

Text-based response body

:::note
There is lots of extracted information in the [custom metrics](/reference/structs/custom-metrics/), which are custom pieces of JavaScript run during the crawl to extra information from the page.
Comment thread
tunetheweb marked this conversation as resolved.

Always check if the information you need exists there before using the `response_body` column. Using the custom metrics from the `pages` table is likely less brittle than attempting to use regular expressions on HTML, includes the rendered DOM after JavaScript runs, is much cheaper to query, and so is considerably faster to get back results.
Comment thread
tunetheweb marked this conversation as resolved.
Outdated

For example, to get the `title` and `meta_description` you can use:
```sql
SELECT
page,
custom_metrics.wpt_bodies.title.rendered.primary.text AS title,
custom_metrics.wpt_bodies.meta_description.rendered.primary.text AS meta_description.
FROM
`httparchive.crawl.pages`
WHERE
...
```

The source of the [custom metrics](https://github.com/HTTPArchive/custom-metrics/) can be useful to understand exactly what is collected.
:::

## Example queries

Here are some common operations you can perform with the `requests` table.
Expand Down