Storage Wisdom!

The good old world of storage…

Typically in organisations there are two distinct roles assigned in the storage department.

1. Storage Admin – the person who provisions and supports primary storage to application or server admins

2. Backup Admin – the person who administers the backup software, tape solution and/or VTL

So where is the role Backup Storage Admin then?
Firstly we need to describe what is changing to open up the field for this new role. Traditionally all backup task and operations have been dragged through some kind of backup application. This consists of a number of components:

1. The Backup server – tracks and catalogs all backups, manages schedules and retention etc
2. Media or Storage node – performs the data movement of the data to be protected
3. Backup client – gathers and informs the server what files need backing up.

All of these components make up the backup product. They need specialist skills and training to understand how to implement them. The application guy doesn’t know the details of how NetBackup, Networker or TSM work, he understands how his application need protecting.

So what needs to change to the way backups are done to allow for this new role?

In our last post we discussed the DD Boost for RMAN feature of the EMC Data Domain. This allows the application guy to use his own backup tools to speak directly to the back end Data Domain storage. This puts the application admin in control of scheduling, retention and management of backups.

Scale this up and think about how this might develop. Imagine you had a VMWare admin who wanted to manage his own backups to a central Backup repository, or a NAS administer that wanted to point his NAS device directly at the backup storage device. Repeat the same question for any number of apps and databases.

No need for a dedicated backup software?
What we have described is that each application uses its own custom data protection method to send backups to a central backup storage device. This means that no longer is the backup admin managing all the database and file modules of a backup product, but is provisioning storage to application teams to be used for backups.  In fact the provisioning of the backup storage could be automated too so that when a service is brought online the backup storage is automatically provisioned, I will avoid using the word cloud at this point

Who will know what is backed up though!?
Good question! There would have to be a way to catalog all these disparate types of backups into a common format that could then be used to report and review backup success. Easier said than done you may think but not an impossibility by any means.

So in summary then, what would the responsibilities of a Backup Storage Admin be:
- Manage pot of storage specifically as a backup target
- Provision backup storage based on requests from application or server teams
- Manage the backup catalog
- Report on capacity trending
- Chargeback storage usage to the application or server teams

So surely this is all a pipe dream, well keep your eye on some of the stuff at EMC World 2012 and what Stephen Manley has to say here

Previously on StorageWisdom!

Carrying on the theme from last post I can now talk about the new DD Boost software for Oracle RMAN.

What is DD Boost?
DD Boost is a software add-on to the Data Domain hardware that allows you to do a few funky things, namely:

- Distribute some of the steps in the dedupe process out to the media server or storage node.  This reduces the LAN traffic and weirdly the CPU load on the media server and the DD.

- Backup aware replication – Using the OST standard the DD can be sent commands from an OST compatible application to initiate the replication of backups.  This way the backup app is aware of all the copies of data even though the DD performed the replication process.

- Link Aggregation and Failover – The ports on the DD can be put into a DD Boost group meaning that the DD Boost software client on the host will balance the network traffic between them.  This improves the performance of backups.

So how does this apply to Oracle RMAN?
Up till now you have had a few options to back up Oracle to a DD:

1. Use the SBT_TAPE channel to write the data to a backup app client such as NetBackup or Networker

2. Use NFS or dNFS (this is NFS embedded in the Oracle code to improve performance) to the DD and get RMAN to write directly to the DD.

There was no way to use the SBT_TAPE method so that Oracle could write directly to the DD.  That is what DD Boost for Oracle RMAN provides.

The DD Boost software hijacks the SBT_TAPE channel (as all backup apps do) meaning when the RMAN script runs using that channel it will do the first steps of dedupe on the Oracle server, then send the optimized network traffic directly to the DD.

The other really cool feature is that the RMAN tool can keep in its catalog details of the DD replication copies.  This means that directly from the application you are aware of your level of protection.
So another way to skin the cat…

This is not going to be to everyone’s taste as it moves the control of the Oracle backups over to the DBA.  So it means that the backup admin can monitor what is happening but will have to go through the DBA and RMAN expert to change anything.

A good slide that covers the pro’s and con’s of the different methods is below, enjoy:

more info here – http://www.datadomain.com/pdf/h9687-so-emc-data-domain-rman.pdf

I don’t profess to be an Oracle DBA, but I do have experience of trying to protect Oracle databases.  I listened to a colleague today explain a really cool wheeze for minimizing bandwidth and backup window when you protect Oracle using a Data Domain.

The Problem.
Oracle RMAN allows you to do different types of backups:
1. Full
2. Incremental
3. Image

There is another type that is called an Incremental Merge, this combines the Image and the Incremental to create a new full backup.  This is created by applying the latest incremental backup on top of your previous image.
The problem is that in doing this you have overwritten the previous full backup.  In order to overcome this, typically what happens is that before the merge operation occurs a backup of the previous full is taken to tape directly or via a backup app such as NetBackup, TSM or Networker.  The diagram below shows the process:

The steps are:
1. Perform Oracle Incr Merge to update Image location.
2. Perform Oracle RMAN backup through a backup app of the image
3. The data is received from RMAN and the Oracle Backup Module by the Backup Server.  The server then allocates tape or disk resources to write the data.  If this is a 10TB database then the entire amount is transferred over the network irrespective of if the destination storage is deduped.
4. 24 hours later the next Incr Merge can occur (only if the backup app has completed its backup of the previous day’s version)

The major problems in this config are:

- Bandwidth usage on the network between the Oracle Server and Backup App
- Work done by the dedupe appliance having to ingest / analyze and dedupe the entire database size every time.  This means CPU and memory load taken on the appliance.
- Workload on the Backup App server having to transfer the I/O for the entire database

Data Domain and Fastcopy
So with that back ground the wheeze I learnt about today allows you to eliminate the Backup App step from our example and allow multiple versions of the Incr Merge backups to be instantly available.  This assumes that you have a Data Domain as the backup storage and have a network link between the DD and the Oracle server (either CIFS or NFS).  The feature of DD that allows this miracle to occur is Fastcopy.  This is a command that you can run on any DD to produce a “near-instant” read-writable copy of any file or folder.  This is done by using duplicating the pointers to the segments.  To give you an example of the speed, a 10TB folder can be copied in 3 seconds!
The way that Fastcopy allows you to have multiple Image (Incr Merge) states is show below:

The Steps are:

1.Incr Merge backup is placed in a folder on the Data Domain (presented to the server via NFS or CIFS)

2. Data Domain Fastcopy command is initiated on the “Monday” folder and it is copied to the “Tuesday” folder (however the image inside this folder is the backup from Monday)

3. Oracle then catalogs the new Tuesday folder and finds that it has a second copy of the Monday backup.

4. The next Incremental Merge backup is initiated and Oracle picks the most recent backup with the given tag to apply the merge to.  This will be the backup in the Tuesday folder.

What we now have is multiple image backups (generated using the Incr Merge technique) available to the Oracle DBA.

The great things about this method are:

- Only the incremental backups are ever transferred over the network to the DD
- Less DD workload to ingest and analyze the data as it is only receiving changed blocks
- Database backup load is taken away from the Backup App server for other workloads
- Oracle DBA has full visibility of all the backups available
- Recoverability is improved from all Image type backups (rather than a mix of Full, Incr, Incr Merge)

So there you go! does that sound good to you?  I would be interested to hear from Oracle DBAs that have an opinion, which is all of you I think
 

The big move

Some of you may know that I recently made a move to the EMC BRS Division. Leaving behind the years of experience of TSM and the IBM storage portfolio, I decided to see what was going on over the fence in the noisy neighbours garden.

I have never had so much information thrown at me in such a short period of time. I have got a staggering stack of notes on my iPad for all the new tech that I am getting my head around. Products such as:

• Data Domain
• Avamar
• NetWorker
• Atmos

here are a few examples of my frantic note taking!

So really this post is a bit of a place holder to say that I will be posting good techie information on these products when I have been appropriately indoctrinated!

Over and out…