Posts filed under 'Technology & Innovation'
One of the major goals for inSync 2.1 release (due this week) is improved performance. With this new release users should be able to experience almost 30% speed improvements specially while syncing smaller files.
While working on inSync 2.1, team Druvaa rediscovered some tips and tricks for performance improvement -
Code Profilers
They can give you very quick insights into bottlenecks. It’s better to start at profiler output than from a hypothesis. Start working out a hypothesis only after profiler points out a bad function. We used
gprof2dot, which plots a nice graph from prof or gprof output. An example is shown below -
The graph shows top down hierarchy of functions, the percentage of time each function consumes, the number of calls etc. The percentage of time consumed by a function puts the performance optimization exercise in the right perspective. You don’t want to optimize a function if it contributes just 1% to the whole processing time. The general idea is to concentrate on function that consumes substantial time and is not supposed to do it. Once a few functions like this are optimized, you can go for another round of profiling.
Network Utilization
It’s not sufficient to just reduce the network bandwidth usage. It’s equally important to completely utilize your share of the network bandwidth.
Especially for non-interactive applications, the throughput matters much more than the latency. In a system that uses a single threaded client to issue RPC calls, thethroughput is governed by the latency. If one RPC call takes a long time, the throughput is low even though there is no bottleneck, persay. Looking at it in a different way, the network is not being utilized when the server is processing the call. A multi-threaded client improves network utilization and also throughput. Sometimes the cause for poor network performance could be outside your code. For example, the TCP default window size shows poor performance with high latency-high bandwidth network. Increasing TCP window size improves performance for such networks and so does the use of multiple TCP connections.
Caching
Caching frequently used data reduces the database queries or disk reads. Database queries and disk reads may not consume the CPU cycles but they add to the latency in a big way.
Muti-threading can work around latency but it comes with its own overheads in terms of code complexity and resource consumption. Simple caching avoids frequent trips to database/disk. Databases and operating systems maintain their own cache but the overheads of connecting to a database or issuing a system call are avoided at best.
Beware of stale caches and serialization issues.
Delayed Writes
Synchronous writes are slow. Some writes, for example activity logs, can be delayed indefinitely. Other writes that need persistance gurantees can be synced in batches than individually.
This holds true for both databases and file systems. It’s cheaper to do multiple inserts in one sqlite transaction than to create one transaction for each insert. On the file system side, you are better
off writing a few MBytes to a file, followed by a fsync than multiple few KBytes of writes and a fsync for each write.
Batch requests
A batch of 10 queries sent to a database works faster than 10 queries issued one after the other. Encoding the 10 queries as a pl/sql function works even better. This is primarily due to the socket communication overheads, specifically the latency involved in it.
For inSync 2.1, we found that the lowest hanging fruits were with the database and file system interactions. We sure plucked all of them 
July 28th, 2008
A very nice article covered by Calley Nye (TechCrunch) | July 9, 2008
Druvaa, an enterprise data backup solutions provider, recently launched their new product, Druvaa inSync. InSync is an enterprise PC backup system, that boasts 10x faster data backup while reducing 90% of the bandwidth and storage utilization.
InSync is based around a new technology from Druvaa called SendUnique, that creates a fingerprint for each file to prevent duplication. When 80% of enterprise files and communication are common between users, de-duplication is an important consideration. SendUnique uses a Single Instance File System on the backup server, and compares the file fingerprints during the backup to the files already on the server to determine if it’s unique data. This works especially well for remote notebook computer users who have essentially the same data on multiple computers. There is also a timeline-based system to track changes to existing files in order to restore point-in-time data from any specific time.
Some other new solutions have attempted to fix these problems, but in very different ways. Microsoft Single Instance Service, and Data-Domain both attempt to filter out similar data through traditional single-instancing at backup server level. Newer solutions like Avamar (EMC) and PureDisk (Veritas, a division of Symantec), filter content at the backup-server level before sending data to a remote store. All of these solutions save bandwidth and storage, but the calculations weigh down the CPU and simple block checksum matching often fails to interpret common data.
Druvaa is a new startup based in India, and has received $250,000 in a seed round from the Indian Angel Network in January 2008. They recently were awarded the Indian Entrepreneurial Challenge 2008 Award, chosen out of 8 finalists and 140 applications.
July 23rd, 2008
Read an interesting article at McKinsey Quarterly, which discussed that the storage demand is increasing at a much higher rate than the falling storage pricing.
For backups, both the storage cost and the bandwidth availability are not able to catch up with increasing storage demand.
Take iPhone as example. The amount of storage in iPhone has increased from 4Gb to 16GB, but the media and bandwidth available for backup hasn’t changed much.
The problem is particularly challenging for remote and on-the-go backups. Its an interesting fact (i read somewhere) that almost 200 Million enterprise users are working remotely at any given time. And its a very good possibility that they wouldn’t have backed their data.
This is where Druvaa inSync comes in. The SendUnique technology, ensures that the duplicate data on enterprise devices is backed up just once, giving a clear 90% advantage for bandwidth and storage used for backup
Currently we ship the product for only notebooks, but soon plan to cover every device connected to enterprise network from PDA to Servers.
Any takers ?
July 19th, 2008
The Gartner Report (here) says storage data de-duplication and virtualization are two main technologies driving innovation in storage management software this year. This makes sense, considering the fact that corporate data is increasing at a whooping 60% annual rate. (Microsoft Report says here).
Server Backup
Data is very rarely common between production servers of different types. Its not difficult to imagine that Exchange email server may not have same content as Oracle database server. But data is largely duplicate within file-servers, exchange server and say a bunch of ERP servers (development and test). This duplication creates potential bottlenecks for bandwidth and storage used for backup.
Existing players have offered two solutions to this problem -
- Traditional single-instancing at backup server to filter out common content e.g Microsoft Single Instance Service (in Data center edition). This saves the just storage cost, depending upon at what level to filter commonalities - file / block / byte. A big player in this space is Data-Domain. These solutions don’t have a client component, they just save storage space.
- New innovative solutions like Avamar (now with EMC) and PureDisk (now with Veritas) which try filter content at backup server level before the data goes to the (remote) store. This makes these solutions much better suited for remote-office backups. They save bandwidth and storage.
But, there are two unsolved problems with both these approaches as well ( Which also, explains a poor response for these products in the market )-
- most of the times simple block checksum matching fails to figure out common data, as it may not fall on block boundaries . Eg. if you insert a simple byte in a file, the whole file changes and all the blocks shift. And the block checksum approach fails.
- Checksum calculation is very costly and makes backups CPU exhaustive.
- These approaches are targeting storage cost, not time/bandwidth which is more critical.
PC Backups
The problem is much more complex at PC level, as duplicated data is distributed among users and is as high as 90% in some cases. Emails / documents and similar file formats create large pool of duplicate data between users.
Also, since 50% of PC backup is mainly large email files, this is problem is particularly difficult to solve using simple file based de-duplication techniches used by servers.
Druvaa inSync v2.0 uses a on-wire (distributed) de-duplication technique which senses duplicate data before the backup starts and hences skips it from the backup. This is transparent to the user, all he notices is a 10 times boost in backup speed with over 90% reduction in bandwidth and storage usage.
How it works
This technology creates and maintains a Global “Single Instance” File System at backup server. Each time a user wants to backup a file, the insync clients prepares a file-fingerprint (using linear polynomial based hash) and compares it with the server. After the server sends a response, the backup happens only for the “unique” data within the file.
The (patent pending) advance file-fingerprinting makes it computationally very easy to filter common content like - same paragraphs in different documents, a same CCed email, media rich corporate presentations etc. This cuts down time for backup by 10 times and reduces bandwidth and storage utilization by 90%.
Other Interesting Features
Another good use of the Gobal Single Instance File System is - Continuous Data protection. The user after starting the restore can see how his files changes over time. Which gives him an option to restore point-in-time data from any point in the past. The marketing name for the feature is - “Eternity. Never lose a file. Ever.” A long name, but serves its meaning
Business Opportunities
The same technology/product can be stripped down to backup PDAs and scaled up to backup servers. A good use case would be to reduce time for backup of bunch of related remote servers.
June 15th, 2008
Druvaa announces the availability of Druvaa inSync 2.0 Beta. The idea behind v2.0 is fast and bandwidth/storage efficient backup. The much awaited release, brings four very interesting and unique features -
1. SendUnique - Enterprise wide on-wire data de-duplication. Almost 80% of PC data (emails/docs) within an enterprise is common between users. SendUnique technology fingerprints the user’s backup set to send only one copy of data (emails/docs) common between different users to the backup server. This speeds up backup by almost 10 times and cut bandwidth usage by 90%.
2. Eternity - Never Loose a file. Ever. Timeline based, from-the-past restore. Enables ultimate protection against data loss or virus attacks.
3. NetworkSense - Automatic network sensing and prioritization. Allocates a user defined percentage of bandwidth for backups.
4. TrueSecure - Client triggered secure backups. 256 byte network (SSL) and 256 bit (AES) storage encryption.
You can sign up here for beta evaluation and updates.
The following presentations describes the 2.0 feature set -
May 23rd, 2008
“We are paid to Backup not Restore … “, this was the company slogan at my last job.
And the team really followed it well, seriously. Look at the existing backup solutions, they have tons of options for backup and most of these options are derived for pre-historic tape-based backups. In fact most of the solutions confuse backup with archival. Take a look at following backups options -
- Backup Rule Engines
- Complicated Scheduling and archival times
- Naming a backup or scheme or snapshots
But, there are hardly any options for restore 
, besides they hardly even work 
I guess restores can be made much better, with options like -
- Self-serve, web based restore.
- Give and option to restore just a part of backup.
- Search a file in Restore.
- Restore based on Timeline. ( Choose a date and restore based on that date. )
- Support compression for faster restores (just like backups).
With Druvaa inSync we tried to address most these issues. Now with upcoming 2.0 we plan to add file-searching and time-line based restores.
April 20th, 2008
Backup is a necessary evil. At Druvaa, our goal is to get rid of the evil part of backup. The first step in that process is to find out the pain points of traditional backup.
- Backup schedules: Traditionally, a backup is a scheduled process that runs at fixed intervals. In case of a failure, the data updates since the last backup are lost. The recovery point objective (RPO) is weaker with traditional backup. Refer to Understanding RPO and RTO for a discussion on RPO.
- Backup slots: Traditional backup process is resource heavy. Also, the server appplication needs to be quisced to get a consistent backup image. This implies that the regular server activity cannot continue during backup. Hence, backup is schduled to run during a timeslot when the regular application activity is not present or is present at a lower scale. As the amount of data and the time to backup grows, it becomes harder to find time-slots for scheduling backup. The increase in the number of business hours also puts additional pressure on the backup slots.
- User interface: Traditional backup interface is complex due to the concepts of full/incremental backups and schedules. Due to the coplexity of the user interface, it becomes harder to let the end user control the backup process. Typically, the administrator configures the backup for enduser desktop/laptop. The configuration remains static and cannot easily adapt to dynamic data layout. Instead, the end user is asked to arrange his/her data to suit the backup configuration.
- Backup media: Traditional backup is performed on media like magnetic tapes or optical disks. Complete automation (using robotic media libraries) of the backup process is too costly. In absence of an automated process, an administrative attention is required to manage the backup media. Maintaining the backup media also requires administrative effort. The restore operation also requires administrative attention because the right backup media needs to be loaded.
- Special hardware: Tradional backup is performed using media like magnetic tapes that require special hardware like tape drives. Special hardware means additional procurenement and maintenance cost.
- Restore operation: With traditional backup, the end user cannot restore her files by herself. Typically, a service request is sent to the administrator, thus increasing the time taken for restore. The recovery time objective (RTO) is weaker with traditional backup. Refer to Understanding RPO and RTO for a discussion on RTO.
In the next post, I’ll discuss possible approaches to address the painpoints of traditional backup.
April 16th, 2008
