Parallel Storage Clouds and the Video Data Explosion - Page 2
People do not realize how much surveillance video is being taken continuously. Almost every store (both inside and out), parking lots, highways, doctor's offices, schools, casinos (especially casinos), airports and fast food restaurants have cameras in place. Individuals with cell phone cameras upload video to YouTube, and neighbors that take surveillance video of their yard and home. (I have a crazy neighbor who has cameras to deter people from letting dogs into his yard, and he has actually taken people to court.) Other examples include dashcams in cars, videos inside restaurants as part of a reality show (Restaurant Stakeout or Mystery Diners) and on and on. It is important to realize that the amount of video footage that is taken is quite enormous even if the video is grainy.
Usually these videos get destroyed after a certain period of time because it is just taking up space. But increasingly, people are keeping the video for much longer so that it can be used as part of a data analysis.
One example is stores that analyze video to understand the shopping and buying habits of their customers. What do they see when they first walk in? Does it capture their attention? Do they have problems navigating through the store? What is their pattern in the store? Do they immediately get what they want/need and then pay for it? Or do they browse? How many times a minute do they blink? Are any of these habits a function of the time of the day? The day of the week? The month? Are they affected by weather? Are they affected by "other" events going on in the world? All of this is an effort to provide better store layouts, better signs and direction, and of course, to get you to buy more.
The algorithms to process images into information are under very serious development with the goal of understanding how things can be changed to improve sales. Consequently, stores are keeping this data much longer so they can get a history of the habits of their shoppers. In many cases this video data is too large to be kept on storage within the company, so they are increasingly resorting to the cloud. However, things are about to get worse, at least for people interested in keeping the data and keeping it in the cloud.
I think everyone understands the difference between a 720p television and a 1080p television. The 720p means that there are 720 horizontal scan lines of image display resolution (720 pixels of vertical resolution). 1080p means there are 1080 horizontal scan lines of image display resolution (1080 pixels vertically or with a 16:9 aspect ratio, 1920 x 1080 resolution). Currently, surveillance cameras generally use a much lower resolution so they don't have to store as much data. But stores, casinos, parking lots, etc. are upgrading so they can capture much higher resolution images which helps identify problems or issues. Insurance companies love this because they have more information about events such as shoplifters or accidents that they can use in court.
But wait! There's more! In some cases 4K videos (approximately 4,000 pixels vertically or with a 16:9 aspect ratio, 3840 x 2160 pixels) are being used for these videos. Unmanned Aerial Vehicles (UAVs) are already using very high resolution cameras, sometimes well above 4K. An example is DARPA's project for a 1.8 gigapixel camera. For a square image that is a resolution of about 42,426 x 42,426 or using a 16:9 ratio that is 48,373 x 37,210. According to the previous link, to record video for an entire city at 12 frames per second for one day, produces about 6 PB of data.
Let's assume that we're currently taking video at 12 frames per second and a 16:9 aspect ratio. The table below lists the total pixel count for each resolution.
|Description||Resolution||Total pixels||Size relative to VHS|
|VHS||480 x 320||153,600||1.0|
|720p||1280 x 720||921,600||6.0|
|1080p||1920 x 1080||2,073,600||13.5|
|4K||3840 x 2160||8,294,400||54|
|8K||7680 x 4320||33,177,600||216|
|DARPA 1.8 gigapixel||48,373 x 37,210||1,799,959,330||11,718|
Just going from VHS resolution to a 720p resolution increases our data storage requirements by a factor of 6. Going to a 4K resolution directly from VHS increases the data storage requirements by a factor of 54!
The trend is fairly easy to understand: many more cameras + much higher resolution + longer retention time = massive increase in data. And remember, this is only surveillance video. There are more areas where data volumes are increasing like this.