Skip to content
Knowledge base

Data deduplication in storage arrays and backup: what it really saves and when it hurts

A storage array quote shows two capacities, and it is easy to approve the budget for the larger one. Data deduplication saves exactly as much as your data repeats itself: a great deal on shares full of virtual machine images, almost nothing on encrypted archives and video, and vendor guarantees cover reducible data only.

A quote for a new storage array lists two capacities: raw and effective, several times larger. The board approves the budget for the second one, because that is the figure that answers how much data the company can store. A year later the array is filling up faster than planned, because a new system writes compressed recordings and the finance department’s archive is already encrypted on the server. Nobody got the arithmetic wrong. The effective capacity simply assumed data the company does not have.

Data deduplication saves exactly as much as your data repeats itself, and that single dependency explains many disappointments. It also has a cybersecurity dimension: NIST SP 800-209 (October 2020) warns that encrypting data at the application level, recommended for sensitive data, can make compression and deduplication drastically less effective. The choice of encryption layer is therefore also a capacity decision, and the two are best made together.

What is data deduplication and how does it differ from compression?

Data deduplication replaces multiple copies of the same data with references to one shared copy, while compression encodes data so that it takes up less space. In its security guidelines for storage infrastructure (SP 800-209, section 2.11.6), NIST lists both techniques as the two common approaches to data reduction and notes that they can be used together.

The mechanism is easiest to see in an example from the same NIST document: if an array holds 500 identical blocks, it stores just one and replaces the other 499 with references. According to NIST, deduplication can take place at the storage device level, at the transmission stage or at the file system level.

Microsoft’s documentation on deduplication in Windows Server (updated 18 February 2022) describes the difference from compression well. NTFS compression optimises each file individually at write time, whereas deduplication looks for savings across all the files on a volume. Files contain both internal repetition, which compression removes, and similarities to other files, which compression cannot see.

Deduplication is also more than spotting identical files. Its predecessor in Windows Server, Single Instance Store, only merged files that were completely identical, whereas deduplication also saves space on files that share many common chunks.

How does deduplication work and when does it process data?

Deduplication splits data into chunks, computes a fingerprint for each one and physically stores only the chunks whose fingerprint is not yet in the store. Products differ mainly in how large the chunks are and at what moment this processing happens.

Windows Server uses a post-processing model. According to Microsoft’s documentation, data is written to disk unoptimised, and an optimisation job, which runs every hour by default, splits files into variable-size chunks, places the unique chunks in the chunk store and replaces the file content with a reference. Changes to a file that has already been deduplicated are again written unoptimised and wait for the next run of the job.

The Dell PowerStore array deduplicates as data is written (inline). Dell’s paper “PowerStore: Data Efficiencies” (August 2026) explains that deduplication and compression take place as data moves from the write cache to the drives, and that fingerprints are computed for 4 KB blocks on first- and second-generation models and 8 KB blocks on the third generation. Under heavy load the system can defer deduplication while keeping compression on, and return to the skipped blocks later.

The model has practical consequences. With post-processing, the volume must hold new data at full size until the job runs, while with inline deduplication the computational cost sits in the write path.

How much does deduplication really save?

Deduplication saves as much as your data repeats itself, and the documentation from Microsoft, Veeam and Dell gives ranges or conditions rather than a single number.

Microsoft gives typical deduplication savings in Windows Server: 30–50% for user documents, 70–80% for software deployment shares, 80–95% for libraries of ISO images and virtual hard disks, and 50–60% for a general file share. Percentages and ratios describe the same thing, but they read differently. Microsoft itself pairs savings of up to 95% on highly duplicated datasets with a twentyfold reduction in storage use, and in the same way 50% corresponds to a 2:1 ratio and 80% to 5:1. When comparing quotes, convert every figure to the same form, because a gap of a dozen or so percentage points at high savings means a several-fold difference in occupied space.

Backup software documentation says so explicitly. The Veeam Backup & Replication user guide (updated 15 January 2026) states that a smaller file size is not guaranteed, because the effectiveness of compression and deduplication depends on factors such as the type of source data and the target file system.

You also need to separate what deduplication delivers from what compression delivers. In a sample output of the DDPEval tool in Microsoft’s documentation, a test folder shows 66% savings, but only 4% without compression. That split is worth asking about for every figure a vendor quotes.

How do you read a vendor’s guaranteed reduction ratio?

A data reduction guarantee is a promise about part of your data, bound by conditions, not a forecast of how much space every company will gain.

The terms of the Dell PowerStore/PowerMax Data Reduction Guarantee, effective from 14 May 2026, guarantee for reducible data a logical capacity equal to six times the usable physical capacity on third-generation PowerStore and five times on the second generation. The guarantee does not cover unreducible data, applies only once at least half of the physical space has been written, and requires all compression and deduplication functions to be enabled. Dell also states that it does not warrant that a specific data reduction ratio will be achieved, and the remedy is tuning or reconfiguring the system, or additional capacity of up to half of the physical capacity.

The second source of misunderstanding is the type of ratio. According to Dell’s paper on PowerStore efficiency, “overall efficiency” also includes the savings from thin provisioning and snapshots, whereas the data reduction ratio (DRR) compares only the data before deduplication and compression with the space it occupies afterwards. Since PowerStoreOS 4.0 the system shows Overall DRR, Reducible DRR and the amount of unreducible data separately, which lets you check how much of your data is reducible at all.

Other vendors’ data sheets need the same reading. The HPE Alletra Storage page in our catalogue gives model capacities as effective capacity, that is after compression and deduplication, so planning also requires the physical capacity and a realistic ratio for your own data.

Key takeaways:

  • Deduplication savings depend on the data, not on the device; vendors give ranges by data type.
  • A reduction guarantee covers reducible data only and applies once its conditions are met.
  • “Overall efficiency” includes thin provisioning and snapshots, while capacity planning needs the DRR.

Which data will deduplication not reduce?

Deduplication will not reduce data that was encrypted or compressed before it was written, nor audio, image and video files. In its guarantee terms, Dell lists host-compressed and host-encrypted data, together with audio, image, PDF and video files, as examples of unreducible data.

The reason is simple. Encryption and compression remove repeating patterns from data, and deduplication lives on repetition, so after those operations there is nothing left to store once instead of many times.

NIST describes this cost in recommendation EN-SS-R6 of SP 800-209: end-to-end encryption at the application level significantly increases security, but it can make compression and deduplication drastically less effective and makes management more complex. NIST describes built-in array or drive encryption as protection against device loss, misplacement or theft, which is not an effective control against attacks from a host already mapped to the storage or against privilege escalation.

For planning, the conclusion is concrete. Before the security team switches on application-level encryption for a large dataset, the storage administrator should recalculate capacity as if that dataset had stopped reducing.

How does deduplication work in a backup repository?

In a backup repository deduplication has particularly favourable conditions, because successive copies of the same systems differ only slightly. Microsoft lists backup applications as excellent candidates for deduplication precisely because of the significant duplication between successive copies.

Backup software often deduplicates before anything is written. According to the Veeam user guide, the source-side component obtains block digests from the previous restore point and does not transfer blocks that already exist there, while the target-side component stores only unique blocks in the backup file. How much space a whole cycle takes also depends on the backup scheme, covered in the article on how full, incremental and differential backups differ.

The result depends on settings that are easy to overlook. Veeam recommends 4 MB blocks for backup files larger than 16 TB and for HPE StoreOnce and Dell Data Domain deduplicating appliances, noting that the larger block gives the lowest deduplication ratio. The same guide recommends disabling compression when backups go to a device with its own hardware compression and deduplication, because data compressed beforehand stops being repetitive for such a device.

Backup encryption has a similar effect: with job encryption enabled and data blocks decompressed before storing in the repository, Veeam does not compress the virtual machine data, and the backup can end up larger than the original machine. There is no need to switch encryption off for this reason, but its place in the chain is worth planning together with repository capacity.

When does deduplication hurt performance?

Deduplication hurts performance when the workload reads data randomly from spinning disks, or when the server has no quiet period in which to process data. Microsoft recommends evaluating a workload before enabling deduplication and divides workloads into recommended ones (file servers, VDI, virtualised backup applications) and those that should be evaluated first: general-purpose Hyper-V hosts, SQL servers and line-of-business servers.

The reason lies in reads. Parts of the chunk store may sit anywhere on the volume, so sequential reads work well, while databases with random reads may see additional latency. Microsoft notes that this mainly affects spinning disks, and that flash, with equal access time to every location, is much less sensitive to it.

The second condition is resources. Post-processing needs idle windows, so workloads with an evening or weekend lull are good candidates, while those that run all day, every day may not be. On a hyper-converged host, running deduplication during working hours can starve virtual machines of resources.

The third cost appears when data leaves the device. According to Dell’s PowerStore paper, data sent to a host, replicated, migrated or backed up through Storage Direct must first be returned to full size, and with synchronous replication every write reaches the remote system in its original size. Replication links and backup windows must therefore be sized for data before reduction, not after it.

Does deduplication increase the risk of data loss?

Deduplication does not lose data by itself, but it means a single damaged chunk can affect many files at once. The same chunk in the store may be shared by many files, so its integrity matters to all of them.

Windows Server has dedicated safeguards for this. In Windows Server, the Integrity Scrubbing job detects corruption in the chunk store, reconstructs data from volume mirror or parity where possible, and keeps backup copies of popular chunks referenced more than 100 times. Microsoft also warns against manual changes to the chunk store, and in its interoperability notes (30 June 2025) advises against Robocopy, because certain Robocopy commands can corrupt the store.

The more important conclusion concerns backups. Copies stored in a single deduplicating repository share the same chunks, so they are not physically independent of one another. Protection against a device failure, an administrator’s mistake or an attack on the repository only comes from another backup copy on separate media or in another location, as explained in the guide to what backup is and the 3-2-1 strategy.

How do you check whether deduplication pays off before buying an array?

The most reliable way is to measure it on your own data, before effective capacity appears in the budget. For Windows servers, the DDPEval tool does this, and according to Microsoft’s documentation it evaluates potential savings on local drives, Cluster Shared Volumes, and mapped or unmapped network shares.

When buying a disk array with inline reduction, measurement requires the vendor’s involvement. Ask for a pilot on a copy of representative data and for a report split into reducible and unreducible data, because only that split shows whether the quoted ratio has a chance of materialising.

Base the capacity plan on physical capacity, and count application-encrypted data, video and archives of scanned PDFs at full size.

Where should you enable deduplication, and where should you skip it?

Enable deduplication where data naturally repeats itself, and skip it where data is already encrypted, compressed or read randomly from spinning disks. The table sets out the decisions for typical places in the infrastructure.

PlaceWhat deduplication gainsTypical pitfallWhat to measure before deciding
File server with team sharesrepeated documents and their versionsnew data takes up full space until the job runsDDPEval result on the share and optimisation job duration
VDI and virtual machine image librariesalmost identical virtual disksrunning jobs during working hourshost load during the processing window
All-flash array with inline reductionall writes, including snapshots and clonescomparing “overall efficiency” with DRRReducible DRR and amount of unreducible data after a pilot
Backup repositorysuccessive copies of the same systemsdouble compression and encryption ahead of a deduplicating deviceblock size, compression settings and restore time
Database on spinning disksdepends on the datalatency on random readsperformance test on a copy of the workload
Application-encrypted data, video, PDFpractically nothingcounting it towards effective capacitythe share of such data in the whole dataset

The larger the share of unreducible data, the further the average ratio in a quote drifts from the result in a specific company.

How can nFlo help you choose and optimise storage?

In its IT infrastructure pillar, nFlo provides Comprehensive Storage Services: from workload analysis (IOPS, throughput, latency, I/O patterns, data growth and retention requirements), through architecture design and deployment, to zero-downtime data migration. After deployment the service covers optimisation, including configuring deduplication and compression, auto-tiering, performance monitoring and regular capacity planning reviews.

In all-flash projects we work with, among others, Dell PowerStore, which deduplicates and compresses data inline, and HPE Alletra Storage. Analysing the workload before choosing the technology makes it possible to estimate how much of the data will actually reduce before that figure reaches the budget. Let’s talk about your environment: contact us.

What else is worth knowing about data deduplication?

Does deduplication slow down reads?

It can, if the workload reads data randomly from spinning disks, as a typical database does. The effect is much smaller on SSDs, but before enabling deduplication under a database it is worth running a performance test.

Is it worth deduplicating encrypted data?

Usually not. Encryption at the application or host level removes repetition from the data, and Dell classes host-encrypted and host-compressed data as unreducible, excluded from its reduction guarantee.

Does deduplication replace a second backup copy?

No. Deduplication removes repetition within a single device or repository, so the copies stored there share the same chunks. Protection against a failure or an attack only comes from a copy on separate media or in another location.

How do I check how much deduplication will save in my company?

Measure it on your own data. On Windows Server the DDPEval tool does this, and when buying an array it is worth asking the vendor for a pilot and for a reduction ratio reported separately for reducible and unreducible data.

How is deduplication different from compression?

Compression encodes data so that it takes up less space, within a file or block. Deduplication looks for the same chunks across many files and stores each of them once, and the two techniques can be combined.

Share:

Talk to an expert

Have questions about this topic? Get in touch with our specialist.

Sales Representative
Grzegorz Gnych

Grzegorz Gnych

Sales Representative

Response within 24 hours
Expert consultation
Individual approach

Providing your phone number will speed up contact.

Want to Reduce IT Risk and Costs?

Book a consultation - we respond within 24h

Response in 24h Custom quote No obligations