Google Cloud Storage Insights Datasets: Now with Activity Insights for Smarter Ops

Hey there, swamp dwellers! If you’ve been wrestling with a storage estate that’s grown faster than water hyacinths in a summer flood, Google Cloud just threw you a life preserver. Storage Insights datasets now come with activity insights—generally available—that let you peek under the lily pads and see exactly how your data is being accessed, moved, and modified. No more guessing if that cold data is really cold or if your multi-region bucket is just a pricey sunbathing spot for caimans.

What Are Storage Insights Datasets?

Think of them as your personal capybara scout, automatically building a queryable BigQuery index of your entire storage estate. You get daily metadata and activity updates (typically within four hours), so you can stop manually scraping logs and start asking smart questions. Customize the scope—org-wide, per folder, per project, or even specific buckets—and let the dataset refresh like a cool swamp breeze.

From Static Metadata to Live Intelligence

Before, you knew what data you had. Now you know how it’s being used. The new views capture:

• Object-level activity (writes, updates, deletes, errors)
• Bucket-level aggregate activity (total ops, error breakdown, most active prefixes)
• Bucket-level regional traffic (ingress/egress per region)
• Project-level aggregate activity

All this flows into BigQuery views, ready for analytics, Gemini queries, or snazzy Looker dashboards. It’s the difference between knowing your warehouse has boxes and knowing which ones are gathering dust.

Three Ways to Use Activity Insights Right Now

1. Right-Size Your Storage Estate

Got terabytes in Standard or Nearline that you think are cold? Prove it. Query buckets with minimal read/write activity over the last 30, 60, or 90 days, then apply lifecycle policies to move them to cheaper classes. Here’s a sample SQL to find the quietest buckets:

SELECT name, location, project, totalRequests
FROM `[project].[dataset].bucket_activity_view`
WHERE snapshotEndTime >= TIMESTAMP(DATE_SUB(DATE_TRUNC(CURRENT_DATE(), MONTH), INTERVAL 5 MONTH))
  AND snapshotEndTime < CURRENT_TIMESTAMP()
ORDER BY totalRequests ASC;

2. Architect for Global Performance

That multi-region bucket serving a global app? Check if 99% of traffic now comes from one region. Use the bucket_region_activity_view to see ingress/egress patterns and decide if a single-region bucket would save you a bundle. Example query:

SELECT
  requestLocation,
  bucketLocation,
  SUM(requestBytes) AS total_request_bytes,
  SUM(responseBytes) AS total_response_bytes
FROM `[project].[dataset].bucket_region_activity_view`
WHERE name = '[bucket name]'
GROUP BY requestLocation, bucketLocation;

Shipt, the same-day delivery platform, used this to move 1.3 PB from multi-region to regional storage, saving big while keeping performance smooth.

3. Demystify Operational Hotspots

Spike in 429 errors? Those retries can balloon your Class A costs. Now you can pinpoint exactly which objects or prefixes are causing the fuss. Query for 429s and see the culprits:

SELECT
  requestOperation,
  errorReason,
  objectName,
  bucketName,
  requestCompletionTimestamp,
  project
FROM `[project].[dataset].object_events_view`
WHERE responseStatus = 429
ORDER BY requestCompletionTimestamp DESC;

Getting Started

Ready to turn your storage swamp into a well-managed pond? Enable Storage Intelligence in the Google Cloud console, configure your dataset, and start querying. Or use the pre-built Looker Studio template for quick visualization. For more on optimizing your Cloud Storage footprint, check out our Google Cloud Review.

Original announcement published on Google Cloud.