What You Should Know
With the abundance of Benchmarking Utilities available and the varying results they can report, it can be confusing to decide which tests to pay attention to and which to question. Sometimes, two blogs using the same tools present slightly differing results. In fact, there are many variables that can affect test results, including PC configuration and the options set by the user in the utility itself. Further complicating matters, the results may look cryptic to novices. In this paper, we will explore some of the most common benchmarking utilities, their configuration options, and how to interpret some common test results.
Types of Benchmarking Tools
There are 3 main types of Benchmarking Tools: Synthetic, Trace-Based, and Application-Based. This paper will explore some of the most common tools, CrystalDiskMark (CDM), Iometer, PCMark Vantage, and PCMark 7. In addition to delving into the details of each application, it is also helpful to understand the wider application categories to which the tools belong.
Synthetic Benchmarking Tools
Synthetic Benchmarking Tools mimic the type of stress a drive would experience during various types of workloads using patterns specifically designed to behave like real-world processes.
Test data may be either Non-Random (Repeating) or Random (Non-Repeating). Whether the data is Random or Non-Random may have an effect on SSD performance for some drives because of differences in controller technology, firmware design, NAND components, and DRAM cache size and performance. For example, repeating data is compressible, which means that an SSD that utilizes a compression algorithm will be able to shrink the amount of data it must actually write to the NAND flash. Samsung SSDs perform equally well with Repeating and Non-Repeating data because the Samsung MDX controller does not use compression. Thus, when there is a large discrepancy between competing drives, it is a good idea to check whether or not the benchmarking utility was sending Random or Non- Random data and whether or not the tested drives utilize compression.
Some Synthetic Benchmarking tools also allow the user to specify Sequential or Random data access patterns. Randomly accessed data, as its name suggests, is stored and accessed in a random order (non-neighboring memory locations), whereas sequentially accessed data is stored and accessed sequentially (in neighboring memory locations). File accessed randomly are typically small in size (less than 64KB), while files accessed sequentially are usually over 128KB.
Synthetic tests provide the most basic, fundamental information about drive performance, but they do not represent real-world application performance.
Crystal Disk Mark (CDM)
CrystalDiskMark, freely available via the web, is a very popular storage benchmarking program. Its simple interface allows the user to specify whether to test sequential or random data, the size of the test range, and the number of repeat tests. CDM, which Samsung uses when measuring the sequential performance of its SSDs, reports the maximum sequential and random read/write performance in Megabytes per second (MB/s). Random performance is reported for three separate workloads: 512KB data, 4KB data at a Queue Depth (QD) of 1, and 4KB data at a QD of 32. The most important of these are the latter two, as most consumer workloads will be similar to 4KB data at QD 1 and the SSD’s maximum random performance is represented by 4KB at QD 32.
Random performance is most often reported in Input/Output Operations per Second (IOPS), which can be calculated from CDM’s results by dividing the total Bytes per second by the data transfer size (4KB). For example, in the image to the left, if we divide the Random Read speed of 408.7MB/s by 4KB (4,000MB), we get 0.102175MB/s or 102,175 IOPS.
While the majority of client PC workloads will not exceed a QD of 1, some usage scenarios may generate a QD of 2-6 or even up to 10 (in limited applications). Data center applications, on the other hand, may generate massive numbers of Input/Output (I/O) requests, creating a QD of 32, 64, or even 128 in some cases (depending on the number of access requests per second). This is why it is useful to look at multiple performance metrics when evaluating the overall performance of a drive and why it is important to know which metrics are more appropriate for the intended usage application.
Iometer, also freely available via the web, is another popular synthetic benchmarking tool. It is commonly used to measure IOPS (random performance) and is the program Samsung uses to measure the random speeds of its SSDs. Its interface is significantly more complex than CDM, but it offers more control over the testing scenarios. Users can customize what types and sizes of data to use (or mix) as well as specify the test length, the number of workers (threads), or the queue depth, among other options.
When choosing variables for a Random performance benchmark, remember that IOPS performance will vary greatly depending on the parameters of the test data (e.g. the number of threads, queue depth, data block size, etc.) and system configuration (e.g. storage drivers, OS background processes, RAM, etc.). IOPS numbers are directly affected by the NAND components, storage controller, and NAND interface speeds, which may all influence random data seek time. For the vast majority of users, the most meaningful Iometer scores will be those of 4K random Read and Write performance at a Queue Depth of 1-32. The example screenshot above shows Random Write performance for the 840 PRO SSD.
Trace-Based Benchmarking Tools
Trace-based Benchmarking tools are essentially a recording of actual Input/ Output (IO) requests from a set of real-world processes (e.g. running Microsoft Word or Windows Media Center or moving image files from a USB disk to the SSD and subsequently deleting those files). Think of a cassette tape or a CD-R, which, once they contain a recording, can be played back on any number of compatible devices. A trace-based tool will record the time it takes to finish a complete trace (playback) on a particular system configuration, allowing a user to compare results among SSD vendors. Trace-based tools are more realistic than synthetic tools and are much faster to complete than an application-based benchmark.
PCMark Vantage and PCMark 7
PCMark Vantage, designed for Windows Vista, and PCMark 7, designed for Windows 7, consist of pre-recorded traces of real-world user scenarios (actual applications are not run, but emulated) designed to test PC performance at the system and component level. These tools include a variety of test scenarios that test all system components, but storage reviewers select only data relevant to SSDs to report in their reviews. Tasks could include: viewing and editing photos, video, music and other media, gaming, communications, productivity, and general storage.
Application-Based Benchmarking Tools
An Application-Based Benchmarking tool differs from both of the above types of tool in that actual applications (e.g. Microsoft Word or Adobe Photoshop) are installed and run while performance is recorded. Applications are chosen based on what system component the user wishes to stress. The problem with this approach, however, is that most applications are not very storage-centric. Since all SSDs are so fast in everyday tasks, these types of benchmarks are less useful because they will show minimal differences between SSDs.
Interpreting Benchmark Results
Among the many measurements you will come across, it is best to familiarize yourself with some of the most common:
- Sequential Read speeds are reported in megabytes per second (MB/s) and indicate how fast the SSD will be at completing tasks like accessing large multimedia files, transcoding, game level loading, some types of game play, watching and editing video. This is the speed at which the drive can read data from contiguous memory spaces.
- Sequential Write speeds, also reported in megabytes per second (MB/s) indicate how fast the SSD will be at tasks like application installation and document backup. This is the speed at which the drive can write data to contiguous memory spaces.
- Random Read speeds, reported in Input/Output Operations Per Second (IOPS) indicate how fast the SSD will be at completing tasks like antivirus scans, searching for email in Outlook, web browsing, application loading, PC booting, or working in Microsoft Word. This is the speed at which the drive can read data from non-contiguous memory spaces.
- Random Write speeds, also reported in IOPS, indicate how fast the drive will be able to complete tasks like downloading email, compressing files, or browsing the web. This is the speed at which the drive can write data to non-contiguous memory spaces.
Some other common terminology you will encounter includes:
- Idle Time, which is the period of time when an SSD is not handling requests to read or write data.
- Latency, which is the response time of the SSD, or the time between when you initiate a request to read or write data and when the SSD completes the request.
- Queue Depth/Concurrent IOs, which is the outstanding (pending) number of IO requests that must be handled by the drive. Multiple requests are placed in a queue of a specific length, where they will wait for their turn to be addressed by the SSD if another task is already underway. The most common queue depths to test are a Queue Depth of 1, which is typical of light consumer workloads, and a Queue Depth of 32, which is representative of a heavy workload as might be seen on a on a server (e.g. web server, database server, etc.). Where in that spectrum an SSD will perform best varies based on the drive’s firmware algorithm. Being optimized for high or low queue depths or vice versus is not necessarily a bad thing, but rather a design decision.
Your Results May Vary
Remember that each of the measurements discussed above will vary from system to system. For example, the number of IOPS possible on a given system will vary greatly depending on: the balance of read and write operations, the mix of sequential and random patterns, the number of worker threads, the queue depth, and the data block sizes – and this is without taking into account the effect of basic hardware configurations (microprocessor speed, DRAM size and speed, chipset type and drivers, BIOS settings, etc.).
When working with Synthetic Benchmarking tools, keep in mind that Random Write performance will differ based on the test range. For example, write tests with very small ranges may not be able to fully utilize the internal parallelism of the NAND chips, resulting in low performance result.
Another major factor is whether or not an SSD uses compression technology. SSDs that use compression will suffer significant performance drops (up to 90% of claimed maximum performance) when working with incompressible data (e.g. JPEG photos, digital audio or video files, zipped files, etc.). Because everyday home and office PC users have and use a considerable amount of this type of data in real-world usage scenarios, Samsung chooses not to use compression technology in any of its current SSDs in favor of delivering industry-leading read and write performance with all data types. This is also why Samsung uses incompressible (or Random) data when testing its SSDs.
The best approach is always to look at a variety of test results together, as one particular test in isolation can never show the whole picture of a drive’s performance profile.
Benchmarking tools are a great way to show off the raw performance capabilities of an SSD, but they do not, and cannot, truly represent a real world usage scenario. Samsung’s SSDs do not utilize compression, but many benchmarking tests rely on heavily compressible data. Therefore, in some mainstream tests, Samsung’s SSDs will appear to perform below a drive that uses compression. Today’s users work with a lot of multimedia files, however, which are largely incompressible (JPEGs, Movie files, MP3s). An 840 Series will deliver consistently excellent performance regardless of data type, rather than sacrificing performance with one type of file for the sake of another. This is an important distinction, and a point that may be glossed over if the data is not analyzed carefully.
Furthermore, peak speeds are not a good indication of everyday performance. Users are typically not installing applications or copying massive files on a regular basis. Many manufacturers like to brag about peak performance, and Samsung could have designed its SSD to have a similar performance profile. Instead, it chose to focus on real-world users as its baseline rather than topping a particular benchmark. That’s not to say the 840 Series won’t top the charts (it will) – it just wasn’t the design focus. The best picture of a Samsung SSD’s performance will come from a system- level, trace-based performance test like PCMark Vantage or PCMark 7 because these tests closely approximate actual consumer usage scenarios. If one must rely on benchmarking tests rather than hands-on testing, these tests are where to focus.
When making a purchase decision, benchmarking test results can be an important factor, but they should be just one of many. Reliability, the drive’s ability to sustain performance over extended use and over its entire lifespan, product quality, software solutions (like good data migration and SSD management tools), customer care, and technical support are also very important factors to consider. As the number one SSD supplier to the pre-installed SSD market (PC OEMs), Samsung has an unrivaled reputation in all of these criteria, making the 840 Series an ideal choice for almost any PC user looking to take their computing experience to a higher level.