One of the major goals for inSync 2.1 release (due this week) is improved performance. With this new release users should be able to experience almost 30% speed improvements specially while syncing smaller files.
While working on inSync 2.1, team Druvaa rediscovered some tips and tricks for performance improvement -
Code Profilers
They can give you very quick insights into bottlenecks. It’s better to start at profiler output than from a hypothesis. Start working out a hypothesis only after profiler points out a bad function. We used
gprof2dot, which plots a nice graph from prof or gprof output. An example is shown below -
The graph shows top down hierarchy of functions, the percentage of time each function consumes, the number of calls etc. The percentage of time consumed by a function puts the performance optimization exercise in the right perspective. You don’t want to optimize a function if it contributes just 1% to the whole processing time. The general idea is to concentrate on function that consumes substantial time and is not supposed to do it. Once a few functions like this are optimized, you can go for another round of profiling.
Network Utilization
It’s not sufficient to just reduce the network bandwidth usage. It’s equally important to completely utilize your share of the network bandwidth.
Especially for non-interactive applications, the throughput matters much more than the latency. In a system that uses a single threaded client to issue RPC calls, thethroughput is governed by the latency. If one RPC call takes a long time, the throughput is low even though there is no bottleneck, persay. Looking at it in a different way, the network is not being utilized when the server is processing the call. A multi-threaded client improves network utilization and also throughput. Sometimes the cause for poor network performance could be outside your code. For example, the TCP default window size shows poor performance with high latency-high bandwidth network. Increasing TCP window size improves performance for such networks and so does the use of multiple TCP connections.
Caching
Caching frequently used data reduces the database queries or disk reads. Database queries and disk reads may not consume the CPU cycles but they add to the latency in a big way.
Muti-threading can work around latency but it comes with its own overheads in terms of code complexity and resource consumption. Simple caching avoids frequent trips to database/disk. Databases and operating systems maintain their own cache but the overheads of connecting to a database or issuing a system call are avoided at best.
Beware of stale caches and serialization issues.
Delayed Writes
Synchronous writes are slow. Some writes, for example activity logs, can be delayed indefinitely. Other writes that need persistance gurantees can be synced in batches than individually.
This holds true for both databases and file systems. It’s cheaper to do multiple inserts in one sqlite transaction than to create one transaction for each insert. On the file system side, you are better
off writing a few MBytes to a file, followed by a fsync than multiple few KBytes of writes and a fsync for each write.
Batch requests
A batch of 10 queries sent to a database works faster than 10 queries issued one after the other. Encoding the 10 queries as a pl/sql function works even better. This is primarily due to the socket communication overheads, specifically the latency involved in it.
For inSync 2.1, we found that the lowest hanging fruits were with the database and file system interactions. We sure plucked all of them 
July 28th, 2008
Backup is a necessary evil. At Druvaa, our goal is to get rid of the evil part of backup. The first step in that process is to find out the pain points of traditional backup.
- Backup schedules: Traditionally, a backup is a scheduled process that runs at fixed intervals. In case of a failure, the data updates since the last backup are lost. The recovery point objective (RPO) is weaker with traditional backup. Refer to Understanding RPO and RTO for a discussion on RPO.
- Backup slots: Traditional backup process is resource heavy. Also, the server appplication needs to be quisced to get a consistent backup image. This implies that the regular server activity cannot continue during backup. Hence, backup is schduled to run during a timeslot when the regular application activity is not present or is present at a lower scale. As the amount of data and the time to backup grows, it becomes harder to find time-slots for scheduling backup. The increase in the number of business hours also puts additional pressure on the backup slots.
- User interface: Traditional backup interface is complex due to the concepts of full/incremental backups and schedules. Due to the coplexity of the user interface, it becomes harder to let the end user control the backup process. Typically, the administrator configures the backup for enduser desktop/laptop. The configuration remains static and cannot easily adapt to dynamic data layout. Instead, the end user is asked to arrange his/her data to suit the backup configuration.
- Backup media: Traditional backup is performed on media like magnetic tapes or optical disks. Complete automation (using robotic media libraries) of the backup process is too costly. In absence of an automated process, an administrative attention is required to manage the backup media. Maintaining the backup media also requires administrative effort. The restore operation also requires administrative attention because the right backup media needs to be loaded.
- Special hardware: Tradional backup is performed using media like magnetic tapes that require special hardware like tape drives. Special hardware means additional procurenement and maintenance cost.
- Restore operation: With traditional backup, the end user cannot restore her files by herself. Typically, a service request is sent to the administrator, thus increasing the time taken for restore. The recovery time objective (RTO) is weaker with traditional backup. Refer to Understanding RPO and RTO for a discussion on RTO.
In the next post, I’ll discuss possible approaches to address the painpoints of traditional backup.
April 16th, 2008
Python is a powerful languauge for encoding configuration information for a software program. Especially the dictionary contruct and the ability to nest data structures allows to encode complex configuration parameters. Also storing the configuration as a text file allows for easier debugging and manual editing of the configuration. For example, the unix passwd file can be encoded as a dictionary with the user name as the key. Each entry in the dictionary would be another dictionary with name, uid, etc. as keys or it could be a tuple with fixed positions for name, uid, etc.
Offcourse since python is not designed to be used for encoding configuration, it does not directly provide routines to load and save configuration files written as python scripts. Saving the configuration file is fairly simple as the str method cleanly converts any python data structure to a string that can be directly written to a file. Note that for a string type configuration parameter, you need to explicitely add quotes while writing to the file. The python code to save a configuration file myconfig.cfg would look like
follows:
f = open(”/path/to/myconfig.cfg”, “w”)
f.write(”some_config_param = “)
f.write(str(some_config_param))
f.write(”\n”)
Loading a configuration file written as python data structures and then accessing the configuration information seemlessly is slightly more complex. Using the imp module is one possible way to load the configuration file. The imp module provides two functions, find_module to search for a module using the standard heuristics and load_module to load the file found by find_module and return a module object. The return values of find_module are to be passed to load_module as parameters. Once the module object is available, one can access the configurtion information through its attributes. There are couple of issues with using the imp module.
- The file needs to have a .py extension since findmodule searches files with only certain extentions and guesses the type of the file from the extension. This can be worked around by opening the config file instead of using find_module and passing the open file to load_module.
- The load_module method compiles the file as a .pyc file before importing it. That leaves a unrequired file behind. The compiled file is used by load_module if it is newer than the config file. In case of a race between two threads, one saving the config file and the other one loading it, the compiled file could get a timestamp same as the config file but with the old contents. Any further load_module calls load from the compiled file and hence, load the old config data.
The execfile function is a better way to load a configuration file. Again, the execfile method is not intended for loading python data structures. It’s primary use is to run an independent piece of python code. But the function allows us to specify the global and local dictionaries as parameters. Also, the effects of the executed code are reflected in the parameters. The python code to load a configuration file myconfig.cfg would look like follows:
configuration_globals = {}
configuration_locals = {}
execfile(”/path/to/myconfig.cfg”, configuration_globals, configuration_locals)
some_config_param = configuration_locals["some_config_param"]
Happy programming!
March 6th, 2008
