Some thoughts on Converged and Hyperconverged Infrastructure

First we had converged infrastructure and then hyperconverged infrastructure. If the trend continues, next up will be the ultimate hyperconverged infrastructure. As these systems continue to evolve and become more advanced, they are gaining in popularity. Very few people doubt that products from Nutanix or the newly released VMware EVO:RAIL will be huge hits. The real question is whether a hyperconverged infrastructure is right for your data center.

Before converged infrastructure, the IT world had limited choices when deploying x86-based architecture. Rack/tower units and blades were the only choices available. A common feature among these approaches is that they often used external storage to accommodate virtualization -- a common feature and a downside. Blades had an additional advantage in a shared networking, power and cooling infrastructure. Of course, the downside was that you had a higher density of servers in a single enclosure, which could increase your risk in the event of a failure. Rack servers had the advantage of separating the failure points at the additional cost of rack space and power, cooling and networking infrastructure. The middle ground between the two extremes did not exist until the introduction of converged and hyperconverged infrastructure.

Hyperconverged infrastructure in particular is looking to take the best points of the blades and rackmount servers and combine them into a better approach. This is a major step forward from most converged infrastructure solutions that marry traditional blade servers, external storage, and networking into a single consumable "block".

Compute: One of the drawbacks to blades was the high density of blades in the chassis. A single chassis failure could affect many hosts, bringing down hundreds or even thousands of virtual machines. A hyperconverged infrastructure often resembles four blade-style servers in a 2U form factor. This reduction of the possible outage footprint can be more appealing to the customer looking for higher availability. You are still gaining the benefit of data center consolidation while preserving a level of outage protection.

Networking: Connecting everything together presents a challenge in almost all environments. A rack server's ports are normally allocated for production traffic and storage. These ports come with a per-port cost, along with management overhead. In blade environments, virtual switching is often required, which adds an additional pair of switches to your environment but removes the trouble of having to cable the blades to it. A converged infrastructure does not include virtual switching and requires connections from each node to the existing switching infrastructure. This approach resembles the rack environment, just without the storage connections.

Storage: Traditional servers use internal server storage or larger external storage frames. The external storage frame enables the shared storage concept, and virtualization was quick to take advantage of the benefits shared storage enabled, including features like vMotion, load balancing and high availability. Hyperconverged infrastructure turns this on its head by using localized storage that is shared across the four hosts within the single frame or even further, across the entire cluster. This gives the advantage of shared storage without the need for the costly storage frames or the dedicated fiber infrastructure that often accompanies it. The local disk can be an enterprise-class spinning or solid-state drive, giving the converged infrastructure tremendous IOPS potential. As more converged infrastructure product vendors couple their hardware with software-defined storage abilities, the traditional storage frame designs are showing their age.

Compute, networking, storage: When you compare the benefits of the hyperconverged infrastructure over the traditional infrastructure, it's hard to not to see all of the positives (and very few negatives). Hyperconverged infrastructure has found that perfect midpoint between large blade enclosures and the single-server approach. With all of the benefits, where is the downside to going with the hyperconverged approach?

Design: When you look at your requirements and want to come to a decision about what hardware platform (hyperconverged or not converged) to go with, a key factor should come to mind: Do you plan to deploy or design in one or two deployment factors, or are you more likely to deploy in a two- to four-node method?

Nodes in a converged infrastructure are prepackaged, and knowing how you purchase is a big factor in knowing which is right for you. Converged infrastructure, by its nature, is not designed to support a single compute deployment where you would add additional compute nodes to an existing enclosure similar to a blade enclosure. The enclosure exists as a prepackaged unit that works with all of the compute nodes in the cluster.

As businesses trend toward the prepackaged approach, they also need to consider how they will approach working with a converged infrastructure.

Downside: With all the positives of a converged infrastructure, there are some downsides. The prepackaged nature of converged infrastructure is a higher investment in capital costs. This simple math would suggest it costs four times the price of a single server. However, this is a flawed assumption because a converged infrastructure also replaces some storage and networking needs. Traditional external storage frames with fiber cabling and switches are expensive.

The second downside can also be leveraged as an upside since today's IT departments are often silos of professionals responsible for defined infrastructure roles with little cross-training or responsibility. While virtualization has started to break down these IT staff silos, it is still a work in progress and moving very slowly for many organizations. Infrastructure ownership and the division of groups have deep roots in IT. Combining these silos in not simply about reducing the hardware pieces; it can also affect staffing levels. The integration of virtualization has caused some jobs to be eliminated and others expanded, but IT continues to survive and evolve, and the same will occur with a hyperconverged infrastructure. Sometimes the introduction of hyperconverged infrastructure can be the catalyst needed to break down these silo's since it often  comes with management software that handles servers, storage, and some networking tasks from the same GUI.

Hyperconverged infrastructure isn't for everyone right now, with its prepackaged requirements and price. However, its ease of use and integration of storage and networking means it's only going to grow. The breakdown of separate IT silos is not a stopping point, as it is something that the business world cannot ignore even if traditional IT would like to. Just like virtualization, it is not a question of whether to use Hyperconverged infrastructure. The real question is whether you choose to adopt it soon or find yourself in a constantly shrinking pool that is having trouble keeping up.

I think that the definition of company culture is wrong

Although cool, your company’s free organic food and Uber allowance are not “culture.” The fact that everyone, including the CEO, comes to work clad in jeans and a hoodie is not “culture” either.

When we talk about culture, too often we talk about the wildly luxurious perks, or we celebrate employees’ traits or behaviors that have little to do with their actual performance at work, as if those were the reasons why the tech sector has been so successful.

Something like, “Company X is cool because they give everyone a free puppy and they’re known to be rockstar engineers and they all kite surf,” is really only describing a company at a very surface level, if at all.

Unfortunately, celebrating, or at minimum acknowledging, that definition of culture is now the cost of doing business, especially when you’re hiring talented engineers, fresh out of school. I can imagine that when you’re looking for your first job, a doggy-day-care subsidy might seem more like a concrete, positive reason to work at a given company, versus transparency or continuous improvement.

Company culture should be about the business

Here’s how I would define company culture:

It’s the set of values, traits, and systems that are deliberate, obvious and inherent in a company, existing to make the company successful.

When you define (and celebrate!) company culture this way, it becomes obvious why culture should be important to an organization, why culture eats perks for breakfast. Culture is the way the work gets done, the decisions get made, the people get hired. If you’re not focused on culture as a holistic mechanism to build the company’s success, then you’re coming at it from a potentially bifurcated point of view.

Call it culture, beliefs, values, organizing principles, whatever. It can be real and right now, but also aspirational. But no matter what, it has to come back to driving company success.

Let’s take a value like distributed decision making: trusting and empowering the experts (not necessarily the execs) in a company to make decisions in their domain. A company might value distributed decision making because it increases the speed of business; employees can move faster and do more if they don’t have to wait on executive approval, and if the decisions are made by the people with the most information.

Sounds great, right? Is that kind of empowerment part of your company’s culture? Here’s how you can tell:

• When you hire someone, do you ask questions to see if they have decision making skills, if they’re able to make decisions without looking upwards?
• Do you get rewarded through formal rewards programs for having made good decisions?
• Are negotiation and evaluation skills critical to your career development?
• When you finish big projects, do you do a post-mortem that includes assessing whether or not you made the right decisions?

Basically, your company’s culture should be baked into the org structure, the people systems (hiring, firing, performance management), and especially into your business plan.

Let’s get to work on the real definition of company culture

And I know, this process of inculcating a true definition of culture sounds like a lot of work. No organization is perfect at living up to its values. But rather than spend hours and hours wordsmithing the perfect vision for your company’s culture, the real work is find and bridge the gap from where your culture is today and where you want it to be. Aligning the company culture with the success of your business takes it from a nice-to-have to an imperative - a way to achieve success, not just market it.

The Task of Democratizing Big Data

Companies that fail to take advantage of the opportunities presented by "big data" management and analytics technologies can expect to fall behind the competition and possibly go out of business altogether.

The world is just getting started with big data technologies like Hadoop and MapReduce, and several obstacles – such as a dearth of skills and old-fashioned thinking about data -- continue to stand in the way of their adoption.

But, companies that embrace the concept now are the ones who will lead the way in the not-too-distant future when entry barriers are not so high. Companies that exploit big data will gain the ability to make more informed decisions about the future and will ultimately bring in more money than those that do not.

The phrase "big data" is most often used to refer to the massive amounts of both structured and unstructured information being generated by machines, social media sites and mobile devices today. The phrase is also used to refer to the storage, management and analytical technologies used to draw valuable business insights from such information. Some of the more well-known big data management technologies include the Apache Hadoop Distributed File System, MapReduce, Hive, Pig and Mahout.

There is certainly no shortage of hype around big data management technologies, but actual adoption levels remain low for two main reasons. First, Hadoop and other big data technologies are extremely difficult to use and the right skill sets are in short order. Today, organizations often hire PhDs to handle the analytics side of the big data equation, and those well-educated individuals demand high salaries.

The skills used to manage, deploy and monitor Hadoop are not necessarily the same skills that an Oracle DBA might have. For instance, if you want to be a data scientist on the analytics side, you need to know how to write MapReduce jobs, which is not the same as writing SQL queries by any means.

The second major obstacle standing in the way of increased adoption centers on the notion that most companies currently lack the mindset required to get the most out of big data.

Most large companies today are accustomed to gaining business insights through a combination of data warehousing and business intelligence (BI) re¬porting technologies. But, the BI/data warehousing model is about using data to examine the past, whereas big data technologies are about using data to predict the future. To take advantage of big data requires a shift, a very basic shift in some organizations, to actually trusting data and actually going where the data leads you. Big data is about looking forward, making predictions and taking action.

As with all emerging technologies, big data management and analytics will eventually become more accessible to the masses -- or democratized -- over time. But some important things need to happen first.

For starters, new tools and technologies will be needed to reduce the complexity associated with working with big data technologies. Several companies -- like Talend, Hortonworks and Cloudera -- are working to reduce big data difficulties right now. But, more innovation is needed to make it easier for users to deploy, administer and secure Hadoop clusters and create integrations between processes and data sources.

Right now you need some pretty sophisticated skills around MapReduce and other languages, or SAS and others to be a top line data scientist. We need tools that can abstract away some of that expertise so that you don't need to have a PhD to really explore big data.

The task of democratizing big data will also require a great deal of user training and education on topics like big data infrastructure, deploying and managing Hadoop, integration and scheduling MapReduce jobs. We really need to tackle the problem from both ends. One is to make the tools and technologies easier to use. But we also have to invest in training and education resources to help DBAs and business analysts up their game and operate in the big data world.

Modernizing Your Backups

This week, I'd like to spend a little time talking about backup modernization, or as I prefer to call it, data protection modernization. The process we use for traditional backups hasn't really changed much in 20 or 30 years.  We do a full backup once per week, and take some kind of incremental backup of our data every day in between. These backups are aways copied to some other storage mechanism like tape, or now disk and a retention is attached to the backups that defines how long we need to keep that backup.  Those retentions are important, since they define things like how much dedicated backup disk we need, or how many tapes we need to have on hand, etc.  They also play an important roll later on when/if we decide to change the way we do backups.

But first, let's talk about the fact that traditional backup processes are really beginning to become more and more problematic.  Why?  There are actually a number of reasons. First, and perhaps the most obvious, are that data sets are becoming lager and larger every day.  This means that either the backups are taking longer and longer to complete, or,  more and more backup infrastructure needs to be put in place.  Dedicated 10GbE connections, backup to disk, more and more and faster and fast tape drives, all need to be put into place to keep up. Yet it's a losing battle. The data sets just keep getting bigger. For example having a NAS array today that holds a petabyte of data isn't terribly unusual like it was not all that long ago.  These bigger and bigger data sets are now benign to outstrip the ability of the storage system to send data to the backup system in a timely manner. Things like NDMP are just not able to keep up with these very large data sets. So data set size is certainly one of the more pressing reasons that people are beginning to look into modernizing their backups.

Another reason that people are bringing to look at modernizing their backup processes is that backup windows are getting smaller and smaller, and in some cases, closing completely. Back in the day, we had all night to run backups. Yes, of course we had to dodge in-between the batch jobs, that that was easy enough to do when  you had 12 or more hours to do that.  Those days are pretty much over. Today you are luck to get any time at all to backup the data, and as I said above, in some cases, you really don't have a window at all.

Finally, Recovery Time Objectives (RTO's) are getting shorter and shorter and the Recovery Point Objectives are getting smaller and smaller.  What this means for the backup administrator is that they must take more backups, and must be able to retire form those backups more quickly.

So, what to do?  The first step that many of my customers have taken s to start to include snapshots are part of the backup process.  This addressees the issue of RTOs and RPOs since you can take those snapshots quickly, and you can recover from them quickly.  You can also take multiple snapshots per day, so you have a much more fine grained ability to recover that data to a particular point in time. However, must people continue to do their regular backups as well, based on the premiss that snapshots aren't backups since they don't make a full copy of the data to another storage medium. However, for some customers it's beginning to become so problematic to do those tradition fills and incrementals, they are revisiting this position. Specifically, if they were to have a problem with their storage array, such that they lost data, and couldn't recover from a snapshot, isn't that the definition of a disaster in the data center?  if you accept that premise, then you can start to consider a combination of snapshots, and say data replication for disaster recover, as a fixable, complete backup solution and drop traditional backups entirely.

A move to nothing but snapshots and replication as your data protection mechanism solves a number of issues.  It address the ever growing backup infrastructure, for example, by leveraging space you already have on your storage array, and a DR plan (replication) you may very well already have in place. Admittedly, for some longer retentions it might mean you need a bit more disk space in your array, but because of the nature of snapshots it's probably the same or less space than you would need for disk based backups. If you are already backing up to an external backup to disk array like a Data Domain, you can repurpose your DD budget and add the space you need to your storage array to hold all of the snapshots you need/want.

Another method now bringing to become popular to modernize your backups is to leverage change block tracking.  This is a mechanism in which the backup application, the storage array, or the hypervisor keeps track of the specific blocks that have changed,  and the backup application only "backs up" these changed blocks.  This can reduce the amount of backup traffic from the storage array to the backup infrastructure significantly, thus addressing the issue of the ever growing backup data sets.  If you couple this with CDP (Continuous Data Protection) or near CDP functionality, it will also address the RPO issues, and since recovery from this kind of backup often means sending less data back to the storage array/application it can also address the RTO issues.

However, since you are probably already do some kind of backup, most like a traditional backup, the question becomes, how do I get from my current traditional backups to one of these more modern backup techniques?  While on the surface it may seem simple enough, there are a number of issues to consider. First,  you need to consider your existing backups. Those backups have a retention, and so you need to keep you existing backup software/mechanism in place, at least until the retentions on those existing backups have expired. One question that often crops up in this regard is what if I have backups with very long retentions, like 7 years?  Does this mean I need to keep my existing backup mechanism in place for 7 years?Well, that's certainly one way to handle the problem.  One way to mitigate the issue a little if you can, is to PTV your existing backup servers once you've switch all you backups to the new method.  You can then shut down those VM's, and only spin them up if you need to get back at that old data for some reason.  Another way to address the issue to to recognize that backups with long retentions are often not backups at all, they are actually archives, and they probably shouldn't have been backups in the first place. This is the perfect opportunity to start a dialog with your customers about the difference between backup and archive, and getting an archive mechanism in place to handle that data. The difference between archive and backup is a topic near and dear to my heart, but it's also beyond the scope of this posting. Just keep this in mind when you go to do your backup modernization planning.

The other issue to that you should consider when planning to modernize your backups is management.  Much of the utility of today's backup software such as CommVault, NetBackup, and TSM is around managing the backups.  Scheduling them, monitoring that they complete successfully,  and reporting on them both from a administrative perspective, but also up the management tree and to your customers so that everyone is assured that their data is protected.  Many people think that moving to a new more modern backup process means getting rid of these tried and true software programs. However, these may be an advantage to keeping them in place.  For example, that reportage mechanism that is so important to your business then also stays in place.  Considering that many snapshots, for example, are managed by software provided by the array manufacturer,  and often only manage the snapshots on once array at a time, you could end up in a situation where your backups are modernized, but your backup management has taken a step back in time. this is also true if you bring on several deterrent techniques to backup you data.  For example, I know of customers who use snapshots and replication for the databases, and then use something like Veeam to backup their virtual infrastructure.  This has the potential to create an even bigger management/administrative/reporting headache.

So, if you can leverage your current backup software   to manage your snapshots, and/or perform CDP like functions via change block tracking, then I believe that you've hit on the best of both worlds.  The good news is, that most of the backup software vendors are recognized this, and are moving aggressively to add these kinds of features into their products. Admittedly, some are further ahead in some areas than others, but it's not like you have to change overnight, so implementing the features as they appear in your backup software isn't necessarily a bad thing.

It takes courage to say "yes"

Today I want to talk about something a little different.  While my posts on here  have, in the past, all been technical, some of us are also in leadership roles.  So, I think that occasionally I might share some of my near 30 years of experience in that regard as well.

What I want to talk about in this post is, from a leadership pony of view, it really does take courage to say "yes", especially to a new idea.  “Definitely not,” is quicker, simpler, and easier than, saying, “Tell me more.” But, a quick “no” devalues and deflates teammates.

Some of the reasons that leaders are constantly saying "no" include:

1. They think that it makes them look weak when they say "yes" too often.
2. They prefer the "safety" of the state-quo.  This is another way of saying they are afraid of change, or at least it makes them uncomfortable.
3. They haven’t clearly articulated mission and vision. Off-the-wall suggestions indicate the people in the ranks don’t see the big picture.

There are some dangers to offhanded yeses however.  Offhanded yeses can dilute your resources,  divide energy, and distract focus.  So, what do good leaders do?  They explore "yes". I know that takes time, but I believe that the time spent is a good investment.

Here are 8 questions to yes:

1. What are you trying to accomplish?
2. How does this align with mission or vision?
3. Who does this idea impact? How?
4. How will this impact what we are currently doing?
5. What resources are required to pull this off?
6. How does this move us toward simplicity and clarity? But, remember new ideas often feel complex at first.
7. Is a test-run appropriate?
8. How will we determine success or failure?

Leaders who say yes end up doing what others want and that’s a good thing.  Remember too that courageous leaders are willing to risk being wrong sometimes in order to be right most of the time. They know that decisions move the organization forward. They know that a lack of a decision is in fact a decision; it’s a decision to do nothing and that’s a decision that is almost always wrong and at times catastrophic.

So, are you a leader that says "yes"?

Openstack Icehouse release, a first look

On April 17, the OpenStack Foundation announced the availability of the ninth release of OpenStack, codenamed Icehouse. The release boasts 350 new features, 2,902 bug fixes and contributions from over 1200 contributors.

Icehouse focuses on maturity and stability as can be seen by its attention to continuous integration (CI) systems, which featured the testing of 53 third party hardware and software systems on OpenStack Icehouse.

The hallmark of the Icehouse release consists of its support for rolling upgrades in OpenStack Compute Nova. With Icehouse's support for rolling upgrades, VMs no longer need to be shut down in order to install upgrades. Icehouse "enables deployers to upgrade controller infrastructure first, and subsequently upgrade individual compute nodes without requiring downtime of the entire cloud to complete." As a result, upgrades can be completed with decreased system downtime, thereby rendering OpenStack significantly more appealing to enterprise customers.  There are also some added functions for KVM, Hyper-V,  VMware, and XenServer which are too numerous to go into here. See the Openstack Icehouse release notes for more details.

Icehouse also features a "discoverability" enhancement to OpenStack Swift that allows admins to obtain data about which features are supported in a specific cluster by means of an API call. Swift now also supports system-level metadata on accounts and containers. System metadata provides a means to store internal custom metadata with associated Swift resources in a safe and secure fashion without actually having to plumb custom metadata through the core swift servers. The new gatekeeper middleware prevents this system metadata from leaking into the request or being set by a client.

On the networking front, OpenStack now contains new drivers and support for the IBM SDN-VE, Nuage, OneConvergence and OpenDaylight software defined networking protocols.  It also supports  new load balancing as a service drivers from Embane, NetScaler, and Radware as well as a new VPN driver that supports Cisco CSR.

Meanwhile, OpenStack Keystone identity management allows users to leverage federated authentication for "multiple identity providers" such that customers can now use the same authentication credentials for public and private OpenStack clouds. The assignments backend (the source of authorization data) has now been completely separated from the identity backend (the source of authentication data). This means that you can now back your deployment's identity data to LDAP, and your authorization data to SQL, for example.

The Openstack Dashboard (Horizon) has support for managing a number of new features.

Horizon Nova support now includes:

• Live Migration Support
• HyperV console support
• Disk config option support
• Improved support for managing host aggregates and availability zones.
• Support for easily setting flavor extra specs

Horizon Cinder support now includes:

• Role based access support for Cinder views
• v2 API support
• Extend volume support

Horizon Neutron support now includes:

• Router Rules Support -- displays router rules on routers when returned by neutron

Hoizon Swift support now includes:

• Support for creating public containers and providing links to those containers
• Support explicit creation of pseudo directories

Horizon Heat support now includes:

• Ability to update an existing stack
• Template validation
• Support for adding an environment files

Horizon Ceilometer support now includes:

• Administrators can now view daily usage reports per project across services.

In total, Icehouse constitutes an impressive release that focuses on improving existing functionality as opposed to deploying a slew of Beta-level functionalities. OpenStack's press release claims "the voice of the user" is reflected in Icehouse but the real defining feature of this release is a tighter integration of OpenStack's computing, storage, networking, identity and orchestration functionality.

Simplivity vs. Nutanix

At a high level both of these products provide the same service(s) for the user.  Certainly the two “leap-frog” each other in terms of features, but at this point in time, they are very close. Both of them are “Hyper-converged VMware appliances”, though Nutanix is able to support other hypervisors such as Hyper-V and KVM as well as VMware.  Simplivity will allow large customers to utilize their own hardware, however, the customer must buy the Simplivity software as well as the OmniCube Accelerator Card for each server since the card is what does all of the writes in the Simplivity architecture. 

From an architectural perspective both systems provide a “hyper-converged” solution made up of X86 servers with internal storage which are networked/clustered together. You grow the overall system by simply adding additional nodes to the cluster.  As of this writing, Nutanix offers more different kinds of options for those nodes, giving the user more flexibility on how the clusters is gown.  Both systems provide for multiple tiers of storage including SSD and HDDs and will automatically move hot data between tiers.   It should be noted that Nutanix offers an interesting feature that Simplivity does not.  Nutanix has the concept of “data locality”.  With data locality, when you v-motion a VM to a different node in the cluster, Nutanix will move the datastore(s) for that VM to the same node (assuming there is space).  This movement is done in the background, over time so as not to impact performance.

As of the latest version both systems provide deduplication of data natively built into the system. There is some discussion about which method of deduplication is “better”, however, in the end I believe that both will provide the user good deduplication results. Both systems also provide compression of data at the lower tiers. 

Again, in regards to backups, replication, DR, etc. both system provide very similar features. Both systems allow for replication of deduplicated/compressed data thus providing “WAN optimization”, both systems provide for snapshots, and both systems replicate data within the cluster for data durability. Simpilivity is able to provide one feature that Nutanix is not currently able to support, and that is replication to the “cloud”.  Specifically, Simplivity provides their software as a VM image running in AWS which can be federated to an Omnicube running in the users data center. 

In regards to management.  Both systems provide for a GUI management environment which allows the user to manage the entire footprint from a single pane of glass.  Again, how this is implemented is somewhat different. Nutanix provides a somewhat traditional management GUI based on HTML 5 that can be used to manage the Nutanix system. Simplivity takes a different approach. Simplivity utilizes a Vcenter plug-in to manage the Simplivity Omicube.  This ties Simplivity to VMware, and will make it more difficult to support other hypervisors.

In conclusion, I believe that the two products would provide effectively the same capabilities for most customers with the single exception of the AWS support that Simplivity  provides. This support would provide the ability for customers to create a Hybrid cloud infrastructure that span the customers private cloud and the AWS public cloud. 

Is 2014 the Year of Object Based Storage?

Object based storage has actually been around for a long time.  Some implementations started to appear as early as 1996, and there have been different vendors offering the technology ever since.  However, it has never experienced the “explosion” in usage that some were predicting that it would. 

It least until now.

IDC said the OBS market is still in its infancy but it offers a promising future for organizations trying to balance scale, complexity, and costs. The leaders include Quantum, Amplidata, Cleversafe, Data Direct Networks, EMC, and Scality, with other notables such as Caringo, Cloudian, Hitachi Data Systems, NetApp, Basho, Huawei, NEC, and Tarmin.

Last year OBS solutions were expected to account for nearly 37% of file-and-OBS (FOBS) market revenues, with the overall FOBS market projected to be worth $23 billion, and reach $38 billion in 2017, according to IDC. At a compound annual growth rate (CAGR) of 24.5% from 2012 to 2017, scale-out FOBS – delivered either as software, virtual storage appliances, hardware appliances, or self-built for delivering cloud-based offerings – is taking advantage of the evolution of storage to being software-based.

IDC predicts that scale-up solutions, including unitary file servers and scale-up appliances and gateways, will fall on hard times throughout the forecast period, experiencing sluggish growth through 2016 before beginning to decline in 2017.

IDC said emerging OBS technologies include: Compustorage (hyperconverged), Seagate Open Storage platform, and Intel’s efforts with OpenStack. The revenue of all of OBS vendors combined is relatively small right now (but expected to grow rapidly) with a total addressable market (TAM) expected to be in the billions.  Noted Ashish Nadkarni, Research Director, Storage Systems, IDC. “Vendors like EMC and NetApp have not ignored this market – if anything they have laid the groundwork for it.”

One of the challenges that IT continues to confront is the growth of unstructured data.  This growth creates challenges around data protection, as well as for users when they go to find their data.  Object based storage addresses both of these issues. Use of technologies like Erasure Codes allows OBS to store data in a way that is both highly durable, as well as geographically distributed.  This eliminates the need to create multiple full copies of the data in multiple locations, as you would have to do with traditional NAS arrays. So, rather than having to place storage systems that comprise 300% of your actual data size, you can utilize as little as 50%.

In addition, because many object storage systems are software solutions that can be run on nodes using low cost server hardware and high capacity disk drives, they can cost significantly less than proprietary NAS systems. Throw in better data protection and enhanced features that can enhance search performance and efficient data tiering and it’s easy to see why OBS is catching on.

So, what’s the downside?  There are a couple.  First, it’s performance.  OBS typically cannot match the performance of traditional NAS arrays. With object retrieval latency in the 30-50ms range, applications that require high performance are going to have a problem with OBS.  This is one of the reasons that AWS recommends that you put data on Elastic Block Storage if you need good performance, as opposed to using S3.  The other challenge is that applications today are often not written to access data on OBS. Therefore changes to applications must be made, or the OBS storage must be accessed through a NAS gateway.  Introducing a NAS gateway, however, eliminates the flat namespace, as well as the ability to attach meaningful metadata to your files/object.  This reduces the utility of OBS significantly.  However, the use of NAS gateways as an interim solution may simply be a necessity if OBS is to take over the NAS space.

Backing Up Openstack

Today,  I want to talk a little backup backups.  Specifically, how to backup your Openstack environment. But not only how to backup the contents of your Openstack environment, but how to backup Openstack itself. 

The thing to keep in mind here is that OpenStack is based around a modular architecture in which a number of different components can be combined together to offer cloud services on standardized hardware. These modules are freely available under the Apache license.

Backing up OpenStack

Backup solutions are typically developed with either operating systems or applications in mind. OpenStack is neither. OpenStack is merely a collection of components that can be combined to provide various types of cloud services. As such, OpenStack administrators must consider what needs to be backed up and how to perform the backup.

OpenStack backups should focus on backing up configuration files and databases for Openstack itself. The configuration files can be backed up at the file level since Openstack is just software running on a Linux machine.

The /etc/nova /var/lib/nova folders should be backed up on both the cloud controller and the compute nodes. However, you must exclude the /var/lib/nova/instances folder on any compute nodes. This folder contains live KVM instances. Restoring a backup that was made of a live KVM instance will typically result in an unbootable image.

One of the most important folders to include in your backup is /etc/swift/. This folder contains the ring files, ring builder files, and swift configuration files. If the contents of this folder are lost, the cluster data will become inaccessible. As such, it is a good idea to copy the contents of this folder to each storage node so that multiple backups exist within your storage cluster.

Some other folders that contain configuration data and should be included in your backups include:

/etc/keystone
/var/log/keystone
/etc/cinder
/var/log/cinder
/etc/glance
/var/log/glance
/var/lib/glance
/var/lib.glance/images

In addition to the folders listed above, there are also several databases that need to be backed up. Typically the databases will reside on the cloud controller, which doubles as a MySQL Server. This server hosts databases related to the Keystone, Cinder, Nova, and Glance components of OpenStack.

You can back up these databases by using the mysqldump command. The command requires you to specify the names of the databases that you want to back up as well as an output file. For example, if you wanted to back up the keystone database to a file named KeystoneBackup, you could do so with the following command:

# mysqldump –opt keystone > KeystoneBackup.sql

As a shortcut, you can substitute the all-databases parameter in place of the database name. For instance, if you wanted to back up all of the databases to a file named MyCloud, you could use the following command:

# mysqldump –opt –all-databases >MyCloud.sql

What's missing?

Backing up configuration files and databases will allow you to protect your OpenStack configuration, but there are some things that are not protected by this type of backup. This method does not protect individual objects within object storage. Similarly, block storage data is also left unprotected. According to the OpenStack documentation, these types of data are left for users to back up on their own.

You can use any compatible backup application. The OpenStack documentation basically says that it is up to the users to back up data residing on the  virtual machines that they create. As such, the backup application would have to be compatible with the virtual machines. It should be noted at this point that users of the public cloud have the same problem. Public cloud providers also leave it up to the user to backup their virtual machines and the data that this virtual machines use.

Of course, this raises the question of how you can better protect an OpenStack cloud. One thing to keep in mind is that like any cloud environment, OpenStack makes use of server virtualization. In fact, OpenStack is designed to work with a number of different hypervisors. You can see the full hypervisor support matrix at:https://wiki.openstack.org/wiki/HypervisorSupportMatrix

One way that you can better protect your OpenStack environment is to adopt a backup application that is specifically designed for the hypervisor that you are using. You will still need to protect the OpenStack configuration files and databases, but you can use the backup software to protect the individual virtual machines and their contents

Another thing that you can do is to adopt a backup application that is OpenStack aware. However, this is more easily said than done. As previously mentioned, OpenStack is a collection of modular components that can be used to construct a private cloud. As such, none of the major backup products come preconfigured to back up OpenStack clouds.

Backup vendor Druva recently made headlines when they announced that their inSync software now supports OpenStack based scale-out storage. The software is designed to access OpenStack storage using the SWIFT OpenStack storage access protocol. It will also have the ability to back up file and object storage, as well as mobile endpoints (laptops, smart phones, etc).

Similarly, Zmanda supports the OpenStack framework with its Amanda enterprise backup software. The software is designed to create backups from the remote server layer.

Both Druva and Zmanda back up specific OpenStack resources, as opposed to the entire OpenStack infrastructure. It should be possible to also use traditional backup apps like NetBackup for Linux to back up the required components. However, NetBackup is not OpenStack aware. It would, therefore, be the backup admin's responsibility to manually configure a backup job that includes all of the required config data and databases.

The key to adequately protecting your OpenStack environment is to determine what it is that needs to be protected and then build a backup solution to meet those needs. While there are commercial products that can back up certain OpenStack resources, those products may not offer the level of protection that you require. You may have to combine commercial backup products with script-based backup techniques.

PaaS Outlook for 2014

Yup, it's that time of year again where everyone makes their 2014 predictions. I guess I'm no exception...

In this blog posting I’d like to spend a little time talking about Cloud, and specifically, about PaaS.  But first, a little background material.

First, lets define the different “kinds” of Clouds there are:

Public Cloud – Gartner defines public cloud as a style of computing where scalable and elastic IT-enabled capabilities are provided as a service to external customers using Internet technologies—i.e., public cloud computing uses cloud computing technologies to support customers that are external to the provider’s organization. 

Private Cloud – Webopedia defines Private cloud as the phrase used to describe a cloud-computing platform that is implemented within the corporate firewall, under the control of the IT department. 

Hybrid Cloud – SearchCloudComputing.com defines a hybrid cloud as a cloud-computing environment in which an organization provides and manages some resources in-house and has others provided externally. 

There are some others, but they are all basically variations of the above.

Once you have a Cloud solution, the question is, what kind of Cloud is it? Here are the definitions that NIST provides.

 Infrastructure as a Service (Iaas) - The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, and deployed applications; and possibly limited control of select networking components (e.g., host firewalls). 

Platform as a Service (PaaS) - The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly configuration settings for the application-hosting environment. 

Software as a Service (SaaS) - The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure2. The applications are accessible from various client devices through either a thin client interface, such as a web browser (e.g., web-based email), or a program interface. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

I recently did some research on PaaS, but I found the research tough to do because it’s almost impossible to put all of the PaaS players in the same bucket, and common patterns are hard to find.

Unlike the IaaS players that provide IT resources as a service, PaaS providers are really solution development platforms.  Therefore, they are built around the types of problems they solve, not some industry-accepted approach.

At the heart of the problem is the fact that PaaS is today’s most ill-defined area of cloud computing.  The approaches, features, and definitions vary widely, with many PaaS providers offering a specific focus.  This may include support for specific programming languages such as Salesforce.com’s Heroku, support for Ruby, Node.js, Python, and Java, or perhaps tight integration with major databases, such as Oracle’s Cloud Platform.  Or, perhaps it’s the delivery model, with private PaaS offerings from Active State, App Fog, or Apprenda, for those of you who can’t yet trust the public PaaS offerings from Google or AWS. Then there is an entirely new set of PaaS providers such as Elastic Box that bring a completely different approach to the problem.

Overall it’s largely a function of the providers all of which are trying to be relevant in this emerging marketplace.  PaaS is the last frontier of cloud computing, and thus the least defined.  So, it’s still possible for vendors to manipulate the market by positioning their products to better define what PaaS is and its value, or, more likely, their campaigns will just confuse people.

In 2013, the PaaS market took on some new dimensions.  Private PaaS players saw strong growth as some enterprises looked to keep applications and data in-house.  Also, there is greater support for the emerging use of DevOps, better database integration, and better support for emerging multi-cloud deployments.  This builds upon, not replaces, the traditional uses of PaaS to automate application development, testing, and deployment processes.

Moreover, the PaaS market saw increased meshing with the IaaS space in 2013.  This includes strong showings from AWS Elastic Beanstalk, and other IaaS-focused players.  We also saw the arrival of some new PaaS players, including Oracle, and we got a clearer picture of how Saleforce.com’s and Pivotal’s PaaS offerings will likely exist in the emerging market.

Given all of these developments, there is a need to reevaluate the PaaS market and the PaaS players, in terms of how PaaS truly fits within an enterprise application development strategy.  Questions are emerging such as:  When will PaaS work for enterprise IT?  When will PaaS not work for enterprise IT?   What is the changing value of PaaS technology, now, and into 2014?

Some confusion in the attempts to answer these questions, and complexity has emerged.  This led to some pushback when it comes to PaaS within enterprise IT.  Many consider PaaS too complicated and too limiting for most development efforts, and for most developers.

For instance, most PaaS offerings place the developer into a sandbox, with only the features and functions that the PaaS provider furnishes to build and deploy applications.  While this makes development an easy and controlled process, many developers need to gain access to the resources and tools required to support specific features, such as remote and native APIs, as well as middleware and database services.  While the PaaS providers consider this abstraction from the underlying “metal” a path to productivity, many developers don’t agree.

PaaS does provide the ability to automate much of the development and deployment activities, as well as provide the developers with the ability to offer self- and auto-provisioning capabilities.  This means that application developers can focus on the applications, and not have to deal with the purchase of hardware, software, and development tools to support increasing demands on the applications or the need to scale.

Moreover, PaaS supports new and more innovative approaches to delivery, including DevOps and the move to “continuous delivery.”  Approaches such as continuous integration, automated testing, and continuous deployment allow software to be developed to a high standard and easily packaged and deployed.  This results in the ability to rapidly, reliably, and repeatedly push out enhancements and bug fixes to customers at low risk and with minimal manual overhead.

If there is a core pattern that is a part of most PaaS, it’s that  it’s solution-oriented.  PaaS providers are focused on being the factory for cloud applications, and they understand there are many paths to get to that goal.  As such, the offerings are very different from provider to provider, and thus the market is fragmented, complex, and confusing to those in enterprise IT.

I suspect this situation won’t improve much as we enter 2014.  However, PaaS continues to be a consideration for those moving to the cloud.  How and if it’s leveraged will be defined by the particular enterprise.