How to find duplicate BLOBs in your Azure Storage Accounts

Mar 15 2021

Reduce your Azure Storage Costs

Azure Storage is like an all you can eat buffet, except the more you eat, the more you pay! This has provided organisations an almost limitless supply of storage, and as we all know, the more that’s available, the more we’ll use.

Azure Storage has changed the way many organisations operate both in terms of availability and service. No longer do organisations have to concern themselves with capacity planning or worry about the SAN running out of disk space, or even how to make that storage available in terms of disaster and across regions. Azure Storage has effectively resolved many of the issues traditional on-premises storage services faced.

However, because of this unlimited supply of storage in Azure, storage growth, cost and duplication has exploded. Managing and controlling the data across all the Storage Accounts has become increasingly difficult as users keep piling on more and more data.

OK, so how can we gain some control back, by finding and reducing the number of duplicate BLOBs and therefore reduce cost?

We have been asked that exact same question by many of our customers, and the short answer was, you couldn’t. There is no way in Azure to find out which BLOBs are duplicates, how many of them are duplicate BLOBs, which Azure Storage Account and container do they reside in, or how much data duplicate BLOBs in Azure Storage are consuming…until now.

Cloud Storage Manager to the rescue.

Cloud Storage Manager allows you to scan your entire Azure environment in order to review and manage your Azure Storage consumption.

Cloud Storage Manager has so many features in helping you control and manage your Azure Storage, but I won’t go into them here as the list is too long – however, click here to get more information and even download a free trial for yourself.

One of the many features of Cloud Storage Manager is the reporting option, which offers a number of reports you could run to get a better understanding of your Azure Storage and overall environment.

How many Duplicate Blobs are in your Azure Storage

However, the one we will be looking at today, is the “Duplicate BLOBs in Storage Account” report.

Once you’ve installed Cloud Storage Manager and allowed it to scan your environment, click on the
“Reports” tab.

Scroll down and find the “Duplicate BLOBs in Storage Account” report.

Duplicate Blob Report

Right click on the report and select to “Run Report”.

The result will be displayed showing you all the duplicate BLOBs in your Azure Storage Accounts, along with information such has which Subscription, Storage Account, Container, tier, filename, type, size, created date and modified date.

From here you can now open the report in Excel, and work out which duplicate blobs you need to keep, or perhaps even delete to reduce your Azure Storage and costs.

Duplicate Blob Analysis

You also have the option to export the report to a CSV file for further assessment.

If you want to get a better handle of your Azure Storage consumption and reduce your overall cost, give Cloud Storage Manager a try.

Download a FREE TRIAL of Cloud Storage Manager today.

What are the consuquences of having Duplicate Blobs?

Duplicate BLOBs in Azure Storage can be a result of a lack of proper management and control of data in the storage accounts. Azure Storage has become increasingly popular due to its unlimited supply of storage, which has led to an explosion of storage growth, cost, and duplication. This can be due to various reasons such as multiple copies of data being created, incorrect data management procedures, or even data being inadvertently duplicated.

The consequences of having duplicate BLOBs in Azure Storage can lead to increased storage costs, difficulty in managing and controlling data across all the storage accounts, and loss of visibility into the actual data usage. This can also result in data redundancy, which can negatively impact the performance and reliability of the storage environment. In addition, it can also lead to data versioning issues and make it difficult to determine which version of the data is the most recent and accurate.

Therefore, it is important to implement proper data management procedures and use tools such as Cloud Storage Manager to scan and manage your Azure Storage environment to help reduce the number of duplicate BLOBs, thus reducing costs and increasing overall control of the data.

by Mark