IT Weblog

Google Wave

2009-12-02T19:29:00.001-08:00

If anybody would like a google wave invite, let me know and I will send you an invite.

Google does twice the storage for a quarter the price

2009-11-10T21:46:00.001-08:00

A couple of hours ago google changed their pricing structure for their storage plans that share storage between online Picasa photos and gmail.

http://googleblog.blogspot.com/2009/11/twice-storage-for-quarter-of-price.html

These prices are truly spectacular, and I think are going to put a lot of pressure on Mozy and Sugarsync. Google has storage as cheap as $5 for 20 gigabytes for a year, and you can get as much storage as 16 terabytes for $4096 a year. Of course I'm comparing apples with oranges: Google has yet to introduce a way to store files that don't integrate cleanly with their services. How do I store my mp3 files in the cloud for instance? You need something like Mozy or Sugarsync as of now. Another apples/oranges comparison would be Amazon S3 rates which are as follows:

$0.150 per GB – first 50 TB / month of storage used

And this doesn't even include data transfer costs. This would really be more exciting if Google allowed you to store any type of file and offered Sugarsync-type features. I'm sure the integration will come eventually. In the mean time, it's a pretty good way of storing your photos in the cloud. Note that I didn't say backup. I remember one virus a while back that zeroed all the jpg files on a friends machine. If those zeroed images would then be synced to the google cloud, your backups are shot too. For a proper backup service you need revision information, even for photos. Google does a good job of offering revision information for their docs, they need it for other media as well.

Separate Swap Files on ESX

2009-05-07T18:56:00.001-07:00

On Netapp's best practices documentation, they say if you will be using their snapmirror functionality for your virtual machines, that you may want to create a swap drive that houses temporary files such as the Windows pagefile. The thinking is that you don't want to waste valuable bandwidth replicating a pagefile that gets recreated at every boot.

I did some testing, and I'd be curious if any of you have different results. My theory is that the Windows pagefile actually doesn't get used very much if you have properly sized your memory for the VM. Why would it page something if it still has access to fast RAM? Therefore, the liklihood of bandwidth savings is very low even if you put the swap file on a different volume.

I ran a test with 3 VMs, that in aggregate used more than 10G of memory. The daily change rate was between 1.5 and 3GB before *AND* after separating the pagefiles on a different volume.

It's an administrative headache to set up every single VM with a separate drive. On another datastore there are over 40 VMs with a daily change rate of between 10-15G, and I doubt going through the process of separating the pagefile will decrease that much. I think I'm going to pass on this Netapp's best practice. Not worth it!

Netapp Aggregate Snapshots

2009-02-07T09:43:00.001-08:00

I'm sure you've heard that Netapp can take snapshots all day long with no performance penalty. That is not entirely true. The actual snapshot creation process is done super quick, but if you are not aware of what exactly occurs from snapshot creation to deletion on the filer, you may not be able to explain why your filer is performing the way it is.

Like I said, when you create a snapshot, there is no performance penalty. But most people operate their snapshots on a schedule, and when a new one is created, the very oldest one gets deleted. Next time you delete a snapshot, type this afterwards:

priv set advanced #get into advanced mode
wafl scan status
priv set #return to basic mode

You'll see that a container block reclamation process has begun on the volume that had a snapshot deleted. I believe this to be the process that checks all the blocks of the snapshot to make sure it is not in use in any other snapshots before unlocking the blocks and giving it back to the active file system. I'm pretty sure this runs as low priority, but I encourage you to make sure by typing

sysstat -x -s 2

during the snapshot deletion phase, and check that the Disk Util numbers are not abnormally high. If you are running only a single SATA shelf, and have hourly snapshots on multiple volumes, this process running simultaneously on multiple volumes can easily slow your performance to a crawl.

If you are a Netapp customer, you may never have heard of aggregate snapshots, but you should definitely be aware of them! If you only poke around in the GUI, you will never even see it, because it can only be configured via the CLI. Some commands:

snap sched -A #see what aggregate snapshots are scheduled
snap delta -A #how much space they are using

Now put all this together, and unbeknownst to you, your filer is running block reclamation on all your volumes and even your aggregate all day long! My filer came with multiple aggregate snapshots scheduled throughout the day. This was bad for performance. If you don't have a need for aggregate snapshots, and I imagine most of you don't, you can disable them and get some additional storage for your other volumes.

snap sched -A [aggr-name] 0 0 0 #no more schedule
snap reserve -A [aggr-name] 0 #remove snapshot reserve

Your aggregate comes with 5% reserved by default. Removing that reserve is a quick way of getting 5% more storage for free! Some may advise against this, but honestly....has anybody ever heard of doing an aggregate restore? Especially if you use a snapmirror configuration?

Netapp can do amazing things for you, but an improperly configured or sized system will leave you dissapointed, and that's true with any SAN manafacturer.

Netapp Performance Troubleshooting I

2009-02-06T20:37:00.000-08:00

Why is our filer performing poorly? Troubleshooting performance issues can be a hair-raising experience with any system, but Netapp makes it a bit easier. Let's look at one tool called statit. You can only invoke it from the command line from advanced mode:

priv set advanced #go into advanced mode
statit -b #begins collection period
statit -e #ends collection and displays results
priv set #return to basic mode

As soon as you end the collection period, you will be inundated with information. You may need to increase your putty buffers so it will be able to scroll through it all. If you look about halfway through, you'll see something roughly like the image below.

Some of these labels are self explanatory, but I'll run through some that were most interesting to me recently:

ut% - Average disk utilization percentage during collection period
usecs - Average latency per block in micro-seconds

Even if you didn't know what any of these mean, you can tell that 0d.26 is one sick disk. It's utilization percentage is almost twice as high as the others. This doesn't really look like a hot-spot problem because it's only one disk that is askew. If you look at the usecs, it's off by a factor of 4! Before you ask me about 16 and 17, please realize that those are parity disks and here is nothing wrong with them. Since any system is only as fast as it's weakest link, performance was horrible.

We replaced that one terabyte SATA disk and after it was rebuilt, another statit collection looked like this:

Problem resolved! Replacing the disk gave us noticable performance improvements.

You may feel like you are covered because you have Netapp autosupport enabled, but they can't catch everything. If you feel like you are having performance troubles and it may be disk related, this may be a good place to start.

Introduction

2009-02-06T20:05:00.000-08:00

I am a frequent reader of various web logs, with Scott Lowe and Brian Madden being my most visited recently. Scott Lowe especially has come to my aid numerous times with his excellent technical insight on Netapp storage and VMware issues. In my experience user group communities can be helpful, but nothing will give you as much in-depth technical information than a subject matter expert writing exclusively about his hands on experiences.

I work for a medium sized business as the IS Lead in the pacific northwestern United States. Just like any other IS person, I encounter technical issues day to day. Some technical issues are non-existent in the googlesphere, yet I'm certain that other people have run into them.

I expect this blog to begin with technical issues that have required considerable effort of mine to get resolved. Random musings to follow, perhaps.