Object Storage

Different from the block storage provided by established SAN solutions or file storage provided by NAS systems, object storage is a new data storage system for fairly large and static objects like logs, media or archives.

As the amount of generated and used data increases it becomes clear that using the same file-system for all types of data, is not the most efficient way. Some files are created once, and move around the network using HTTP, FTP or something similar. They’re also usually large in size and use up most of the capacity while utilizing almost none of the features a modern file system provides.

Object storage systems on the other hand treat those files as they should be treated: objects. Storing and retrieving them over HTTP using a well-defined API leaves the precious filesystems to other applications while offering lots of different opportunities.

Media

object storage for media

One example for those types of files is the media files like image, audio or video. Even after compressing them using well known formats like JPEG or AVC/HEVC they’re still the largest objects we need to keep in our storage environments. And considering they’re almost always shared over the network, moving them to an object storage offers huge benefits. For example if they’re used within a web page, Wise Cluster O3 can be used as an accelerated HTTP server directly without an additional content server. Or you can configure object versioning to prevent accidental overwrites or deletes, without a need for a costly secondary backup system.

Archive

object storage for archive

Another example is the archives of documents all businesses need to keep. Text documents, PDF invoices, spreadsheet reports. There’s always a need to archive weekly, monthly or annually because of a regulation or because of the need to back things up. And those regulations sometimes dictate when the data should be deleted too. All those requirements can be met with easy-to-use lifecycle rules in Wise Cluster O3. And old files deep in the archive can still be found within seconds using metadata indexes.

Big Data

object storage for big data

Using object storage systems as the data source for Big Data tasks is almost as old as the big data concept itself. S3 connector was introduced shortly after Hadoop started gaining ground in early 2000s and it’s still arguably the most used storage system for Hadoop based tasks. 2nd generation tools like Spark and Flink also provide first-class support for accessing data in object storage systems. Various higher-level large-scale databases like druid and thanos also leverage object storage systems to benefit from the automatic reliability provided, instead of re-inventing the wheel themselves.