printer friendly version

Selecting the right surveillance storage

Storage is a vital and expensive component of video surveillance. When installing a surveillance solution, smaller projects can make use of network video recorders (NVR) and other in-a-box storage solutions that make the process of setting up and using them fairly simple. Larger projects require dedicated servers in which storage subsystems are included with processors, memory and other technologies.

Today, however, storage is not just storage anymore. Vendors of storage systems have expanded and advanced their technologies to offer a wide variety of solutions, some even offering tiered solutions with different types of storage in each. Some companies provide modular solutions that slot seamlessly (almost) into server racks and include all you need to vastly expand your storage space, as well as additional memory and management systems to prevent issues such as bottlenecks and so forth.

In these days of cost cutting and doing more with less, making optimal use of your storage is vital, but this often means spending more on the equipment than one would like. If you just want hard drive space, there is a lower-cost solution to give you that, but if you want performance and reliability (which is more critical as analytics and artificial intelligence (AI) play a greater role in surveillance), you need to look for more than a bunch of disks. This is critical in the surveillance market as your storage will need to be able to handle continual streams of video being written, with intermittent requests to read existing data.

The fact is one can easily blow the budget on the latest and greatest storage solutions without even trying, which means your storage decisions need to be carefully considered if you want a reasonable cost of ownership and the guarantee that you can expand it as required, without issues.

Hi-Tech Security Solutions spoke to two companies focused on the storage market, Infinidat and Capsule Technologies. Infinidat was founded in 2011 and focuses on storage “performance, availability, and cost, at multi-petabyte scale”. Capsule is a South African company offering high-end storage and back-end solutions specifically for the surveillance market.

Tailored for surveillance

As noted above, surveillance projects put a specific strain on storage systems due to the continual flow of data to be written. Normal hard drives can handle a limited number of video streams, but because these drives aren’t designed for surveillance, the more streams one writes, the more chance there is for bottlenecks and losing frames.

Hayden Sadler, country manager for Infinidat in South Africa, notes that the company’s storage solutions are architected for performance. “The inherent characteristics of our solutions perfectly lend themselves to the demands of the CCTV and video surveillance environment,” he says. “Infinidat provides sufficient bandwidth and throughput to cater even to high-end, city-wide surveillance systems. In addition, our solutions offer the capacity required for large-scale storage.”

Explaining further, Sadler adds: “Write latency is significantly reduced by caching all writes in Dynamic Random-Access Memory (DRAM) which results in zero lost frames. By mitigating bandwidth and latency challenges, we ensure the quality of surveillance is not compromised. Aside from performance and scale, Infinidat solutions can also accommodate the applications around the surveillance environment and not just the footage. This consolidated approach helps to reduce cost and complexity.”

Franck Martinaux, chief technical officer at Capsule Technologies, takes the explanation further. “The I/O [input/output] workflow for video surveillance is composed of a high proportion of concurrent write operations and a smaller proportion of read operations used for playback of footage and indexing processes from the VMS (video management system). Moreover, analytics and AI applications require fast, random read operations.”

According to Martinaux, the storage unit in a surveillance setup must be able to address the following challenges:

• Offer low latency in writing with hundreds of simultaneous streams.

• Rapidly retrieve any footage within the retention period - this is basically random read operations.

• Proceed with maintenance operations such as rebuilding a drive without impacting the latency on write and read operations.

Capsule Technologies addresses these requirements by profiling each VMS it supports and adapting the RAID settings accordingly, based on the method used by the VMS software to write and read data (block size, number of processes writing, etc.). The company also uses a large cache mechanism to eliminate latency for high rates of concurrent write operations.

In addition, Capsule re-orders the write I/O to align the data size pushed to the storage with the ‘ideal’ full stripe size of the RAID set. It also regulates the priority of write/read operations by implementing a QoS (quality of service) mechanism that guarantees read operations will happen in an acceptable time.

A Flash in the pan?

The previous comments are based on traditional hard drives, but there is a significant move today to using Flash storage, or enterprise versions of the SD cards we find in our smartphones these days. Wikipedia describes Flash as an “electronic (solid-state) non-volatile computer storage medium that can be electrically erased and reprogrammed”. Flash has no mechanical components which makes people think it is more reliable than the clunky old hard drive – and it’s also faster.

But is Flash a good solution for surveillance operations? Martinaux states that from a performance standpoint there is a natural benefit to using Flash storage as it provides a very high IOPS (input/output operations per second) rate, especially in reading, which is beneficial for analytics and AI applications.

However, from a reliability standpoint, he says Flash storage needs to be demystified: “there is a belief that Flash drives are more reliable than traditional hard drives since there is no mechanical interaction. Experience has shown us that the crash of a Flash drive is harder to resolve than the crash of a hard drive.”

Expanding on the previous statement, Martinaux explains: “If a Flash drive gives up, it is not possible to see it from the system, it becomes a useless piece of plastic hooked to a cable; it is totally bricked. When a hard drive fails, there is still the possibility to recover some data, most of the time only a fraction of the surface of the disk is damaged, so there is hope to recover some data.

“We see too many Flash storage systems configured in RAID-0 (no redundancy), betting on the supposed higher reliability. This is a mistake. Our personal opinion is that Flash drives must be protected with at least a RAID 5 configuration, just like traditional hard drives, to prevent severe data loss – which could have major consequences.”

Adding to the caution, Sadler notes, “In order to be viable from a cost perspective, all Flash storage requires data reduction. However, this is not suitable for video surveillance.

“One cannot compress surveillance footage because frames will be lost and the quality and integrity compromised when using all Flash storage. The sheer volume of all Flash storage required for a surveillance system would also be cost prohibitive, and the economies just do not work. As a result, a more intelligent, software defined solution is required.”

No matter what solution is chosen, cost concerns have led to more affordable storage solutions being developed in conjunction with data prioritisation processes. Enterprise storage solutions often offer tiered solutions that place data that will be required immediately for the day-to-day operations of the business on high-performance storage for fast access, while data that is not required immediately, or often, can be stored on lower-cost and lower-performance systems. Of course, then there is also archived and backed-up data that can be stored off site or in the cloud. Can this tiered model be applied to surveillance data?

A tiered approach or not?

The simple answer is no, according to Sadler. “Video surveillance footage is either live or archived, therefore it cannot be split into primary and secondary storage. Surveillance footage also has a single archive repository, so this is not relevant.”

However, Martinaux says that data tiers are a good way to reduce cost, for example, using ‘Hot’ storage with a few days of retention and a larger ‘Cold’ storage option which is slower, but has a highly capacity for long-term archiving, either locally or in the cloud.

“Some VMS offer built-in functions to seamlessly archive to a secondary storage solution,” he notes. Alternatively, Capsule offers a solution based on Tiger Technologies, which is a software layer that is completely transparent to the VMS and the user, which is in charge of migrating and retrieving data between various tiers of storage.

“This is a solution we have tested and it works well, and with the rise of cloud solutions such as Azure or AWS in South Africa, we believe that it is a viable solution for many projects.”

Which brings us to the question of cloud storage. According to common expectations, using the cloud for storage is cheaper and more reliable, and allows customers to leave the storage issues to their provider while they simply pay for what they use – or that’s the ideal if you have no bandwidth restrictions. Numerous companies are already offering surveillance solutions ‘as-a-service’ that make use of cloud services, which implies that cloud can be a viable surveillance storage option.

Cloud storage: benefits and challenges

Sadler notes that various service providers offer Video Surveillance-as-a-Service (VSaaS) solutions that are essentially hosted cloud-based solutions. They provide centralised storage for customers’ footage which is transported from the cameras to the data centre via fibre.

“From a storage point of view, the requirement for these deployments is incredibly high bandwidth and scale. However, due to bandwidth and availability requirements, this is not something that can be delivered via the public cloud in any effective manner.”

Martinaux says there are various approaches to store surveillance data with cloud technology. “We see some VMS vendors offering their own cloud services, which may not work if their data centres are located outside Africa. The other risk with this approach is that the customer could be locked in with a certain VMS vendor for the lifecycle of the project.

“The approach that we [Capsule] are pushing as an open platform surveillance storage vendor is a transparent usage of the cloud. This is done by a software layer that combines local storage and cloud storage to present it to the application as a single drive. The data is pushed to the cloud periodically and can be recalled from the cloud and stored locally, transparent to the VMS software and the user. That can be implemented with any cloud providers or even on a private cloud solution.”

Avoiding downtime

No matter what model of storage or technology one uses, one of the most important components of your surveillance infrastructure is avoiding downtime. Your surveillance investment is wasted if, when you need footage, it turns out that it’s not there.

Downtime is never acceptable when it comes to video surveillance, says Sadler, because any downtime compromises the integrity of the system. The 24 x 7 x 365 nature of video surveillance requires constant availability. While the requirements on the system are not constant, and fewer writes take place when motion is not detected on cameras, storage needs to be up and running at all times in case it is required.

In addition to avoiding downtime, storage systems need to be scalable in terms of being able to expand as required. Sadler explains that Infinidat can deliver 1,3 MB IOPS, ‘seven nines’ (99,99999%) availability and scales to multiple petabytes of effective capacity in a single rack.

“Our solution is geared to add capacity on demand with zero downtime. This is built into its design. It offers the capability to scale up and scale out without disruption.”

“Unfortunately, we have noticed that, in the South African market, customers which include major corporates are completely bypassing the fundamental rules of a good redundancy to be able to offer a real 24 x 7 surveillance service,” states Martinaux. “That may sound surprising, but many of the sites we have been involved with do not have any storage or server redundancy, no network redundancy and no UPS.”

Capsule Technologies proposes the following for the storage aspect of a surveillance solution:

• Fully redundant hardware within a building block: mirrored SSD for the operating system, dual PSU, minimum of four NICs, and RAID 6 on the data storage.

• On the firmware level, Capsule offers additional free features: real-time replication of the VMS software on a separate unit with transparent failover and failback, snapshots of the VMS software instance allowing an immediate return to a previous stage (for example, immediate rollback after an OS or VMS update), and real-time replication on separate storage units of the recording dataflow.

Martinaux also adds that “scalability is what we do best at Capsule, either adding a simple building block enabling scalability in terms of recording bandwidth and storage capacity, or by using a CAPS-SCALER unit designed to scale up to 45 x 14 TB drives – an ideal configuration for a ‘pay as you grow’ model.

“No matter the storage option desired, no downtime is required if it is planned and designed properly.”

Not an off-the-shelf purchase

The solutions offered by Infinidat and Capsule are designed for specific users and situations and are therefore not ‘off-the-shelf’ purchases. To conclude the storage discussion, we asked both companies to tell us about their solutions, specifically in relation to the surveillance market.

Sadler notes that Infinidat does not offer a specific product for the surveillance market, however, “the capabilities of our storage systems are suited to this demanding environment both from a performance and capacity point of view.

“The InfiniBox represents a revolutionary storage solution with N+2 redundancy for all components and a grid architecture that enables superior performance and reliability. It consists of industry-standard components that are coupled with enterprise-proven software. The core storage software architecture features an innovative algorithm that is scalable and offers a dual-parity RAID schema. Additionally, the architecture delivers efficient in-memory data management and hotspot-free automatic data dispersion. These capabilities enable InfiniBox to deliver on performance, reliability and scale ideal for the demands of video surveillance.”

Capsule has three flagship systems; all of them are capable of 1200 Mbps of recording bandwidth and offer RAID 6 data protection. And because the surveillance software is hosted in virtual machines, Martinaux says there is no requirement for any extra servers, except if there is a need for AI, which requires a separate and specialised GPU server.

These systems have been successfully validated or certified for Bosch BVMS, Milestone, Genetec, Exacqvision, Digifort and Axxon Next. The systems are:

• CAPS-FIRE, a 12-bay system able to store up to 140 TB.

• CAPS-FATFIRE, a 24-bay system able to store 280 TB.

• CAPS-SCALER, a 45-bay system, able to store 588 TB.

For more information contact:

• Capsule Technologies, Franck Martinaux, [email protected], www.capsule-sa.co.za

• Infinidat, Sapna Capoor, [email protected], www.infinidat.com

Share this article:

Further reading: