Your blog makes very interesting reading. I’m sure others will think so too I look forward to reading their comments….

Thanks for the information on Avamar. Yup, I also came to know about that some time back.

The approach followed by rsync, avamar and pure disk is to narrow down the delta change from the previous backup of the same data.

We are creating fingerprints which can be matched with any “similar” data from any source. The important step is to -

1. choose block boundaries which can be checked for hash against data from other sources.
2. compute this and check it against server in real time.

This makes sense for Laptop backups where duplicates are between machines. The pitch is for time not storage space.

Your claims about Avamar technology are not correct. Their technique is neither block-based, nor is their checksum computation “expensive”. They use a technique they call “sticky byte factoring”, which uses simple rolling checksums to generate variable sized chunks of data and to find differences. See their patent application for details: http://is.gd/C01 You can see that their technique is highly suited for remote office kind of situations. PureDisk used to do fixed-size chunking, but I had heard that they have also started moving towards an algorithm very much like Avamar’s.

Also, when you say “linear polynomial based hash”, I guess you mean Rabin fingerprinting. Rsync also uses similar fingerprinting to reduce latencies.

- Ankur.