Storage Efficiency

So, I've been sitting here thinking that with the current economic distress everyone is looking to save money. In the storage business, this means an almost myopic focus on something called "storage efficiency". Everyone wants to get the most "bang for the buck" that they can right now, and they really don't want to talk about much else, and that's really too bad.

I say it's too bad, because for those few who are bigger thinkers, people who are willing to go out on a limb and take a more strategic view of things, right now is a great time to make some changes that will, at the end of all this, leave their business with a stronger, better, more sustainable storage infrastructure. Or better yet, should those at the top of the IT pyramid actually have magically found some stones, they could create an entire IT organization that's better, stronger, and faster than it is now and one that even operates more efficiently than the one they have today.

Unfortunately, what I'm seeing is fear and the result of that is that people are pulling back. They are dragging out or postponing projects, turning the screws on their vendors to reduce costs, and some are laying off people or even going so far as to outsource. I won't even go into why I think that anyone who outsources today is both a fool and a traitor to this county, that's for another time/post.

To those few who have the courage to build instead of tear down. For those who recognize opportunity in the current economic climate, I say bravo. To the rest, I give the Bronx Cheer.

But back to the topic at hand. What I find interesting is that this myopic focus on "Storage Efficiency" on the part of both the consumers of storage and the resulting response from the vendors of storage. All of the big storage vendors have some kind of "Storage Efficiency" marketing strategy going. The blogosphere is full of arguments about how vendor A's storage is very inefficient, and the supporters of vendor A defending that vendor's storage efficiency. In the end, I don't think that any vendor's storage hardware in inherently more efficient, or less efficient, than any other vendors. It's all about how you lay out your applications on that array, how well you manage the space, and how you are able to properly tier the data. In other words, in the end, it's about people. In this case, Storage Architects and Storage Admins who do the grunt work of managing a company's storage infrastructure on a day to day basis. If they are good and are allowed to obtain the tools that they need, you get efficient storage utilization. Otherwise, you end up with very low utilization rates. My fear, however, is that with all of this focus on "Storage Efficiency" from a hardware perspective that those folks in the trenches won't be allowed to get what they need in order to truly make a company's storage more efficient than it is today. Management will fall prey to all that marketing hype and think that if they just switch from vendor A to vendor B that all of their problems will be solved. Oh, and to pay for that switch and since it's going to be soooo much easier to manager vendor B's storage, lets lay off a couple of those Storage Admins we aren't going to need anymore. Again, for those folks I have no sympathy, and they deserve the disaster that's waiting for them just around the corner.

In the end, I think that given the opportunity to do some storage virtualization in conjunction with server virtualization and network virtualization that storage could become very efficient. When you do all three together, you end up with a very efficient data center, as well as a very green data center. Yes, that's right, I said green data center. I fully realize that green sooooo 2008 and no one wants to talk about it anymore (back to that myopic focus on "Storage Efficiency"). But I think that if you look at the big picture, that the more efficient your storage/servers/networks are, the greener they are. That means reall dollar savings folks, so let's not stop talking about "green" just yet.

So, in my opinion, for those that are willing to invest in the future, I say build a "virtual datacenter". Some call it "Unified Computing", some call it "Cloud Computing", and some have other names for it. But as I see it, it's just creating an environment in which business users can run the applications they need in order to operate the business. I think that the "virtual datacenter" would allow for containerized applications. This means that the user's applications including the code and the data, would be in some kind of portable container that could be easily moved, expanded, shrunk, spun up or spun down, depending on the needs of the business. Add to this a way for business users to deploy their own applications into the environment and you completely change the relationship between IT and the business.

Yes, I know this concept isn't for the faint of heart, especially in today's economic climate. But in the end I truly believe what you would have is a much more efficient, flexible, responsive IT organization which has a much better relationship with the business. Heck you might even end up with IT being viewed by the business as something other than just a cost center which needs to be controlled! Yeah, I know, fat chance, but I can dream, can't I?

--joerg

Wide striping is a two edged sword

I have spent a lot of time lately talking with some of my coworkers, friends, etc. on the topic of wide striping. This topic keeps coming up since there are now a number of vendors selling storage arrays with SATA drives that claim to have "the same performance as fiber channel". Some of the Sales folks I work with keep asking how we are supposed to dissuade people from that idea, or if it's true. One of the prime offenders in this regard is IBM with their new XIV array. The XIV uses wide striping and SATA drives and they claim to have "enterprise performance" at a very low price point. But they aren't the only ones; you have Dell telling people the same thing about their EqualLogic line of storage as well, and there are other too. For an excellent article about the XIV and its performance claims, take a look at http://thestorageanarchist.typepad.com/weblog/2009/01/1037-xiv-does-hitachi-math-with-roman-numbers.html.

What I usually tell them is that the statement is true; you can get fiber channel performance by striping across a large number of SATA drives. The only problem is that you have to give up a lot of usable disk space in order to keep it that way. A quick example usually illustrates the point quite well. Let's say that for the sake of easy math the average application in your environment uses about 5TB of space (I'm sure some are a lot more, and some a lot less, but we are talking average here). Let's also say that you need about 2,000 IOPS per application in order to maintain the 20ms max response time you need in order to meet the SLAs you have with your customers. Finally, let's also assume that your SATA array has about 90TB of useable space using 180 750GB SATA drives and you can get about 20,000 IOPS in total from the array. So, let's do some basic math here. That means that you can run about 10 applications at 5 TB apiece which will take up about 50TB. So, your array will perform well, right up until you cross the ½ full barrier. After that, performance will slowly decline as you add more application/data to the array.

So, what does this mean? It means that the cost per GB of these arrays is really about twice what the vendors would have you believe. OK, but considering how much cheaper SATA drives are than 15K fiber channel drives, that's still OK, right? Sure, as long as you are willing to run your XIV at ½ capacity. In today's' economic climate, that's going to be tough to do. I can just imagine the conversation between your typical CIO and his Storage Manager.

Storage Manager – "I need to buy some more disk space."

CIO – "What are you talking about, you're only at 50% used in theses capacity reports you send me and we didn't budget for a storage expansion in the first year after purchase!"

Storage Manager – "Well, you know all that money we are saving by using SATA drives? Well, it means I can't fill up the array; I have to add space once I reach 50% or performance will suffer."

CIO – "So let performance suffer! We don't have budget for more disk this year. Why didn't you tell me this when you came to me with that 'great idea' of replacing our 'enterprise' arrays with a XIV?!?!"

Storage Manager – "Ahhh … ummmmm … gee, I didn't know, IBM didn't tell me! But we had some performance issues early on, and figured this out. Do you really want to tell the SAP folks that their response time is going to double over the next year?"

CIO – "WHAT! We can't let that happen, we have an SLA with the SAP folks and my bonus is tied to keeping our SLAs! How could you let something like this happen! Maybe I should use the money for your raise to pay for the disks!"

Storage Manager – "Um, well, actually, we need to buy an entire new XIV, the one we have is already full."

OK, enough fun, you get the idea … make sure you understand what wide striping really buys you and if you decide that the TCO and ROI make sense, make sure you communicate that up the management tree in the clearest possible terms. Look at the applications that you currently run, see how much space they require, but don't base the sizing of your EqualLogic (see, I'm not just bashing the XIV) just on your space requirements. Base them more on your IOPS requirements. With SATA drives chances are pretty good that if you size for IOPS, you'll have more than enough space.

--joerg

2009 Outlook

Like everyone else I'm looking at the business climate in 2009, and it makes me nervous. I listen to the news reports of more layoffs and cutbacks that come almost nightly, and wonder what that means to me and to the storage business. I have coworkers who suggest that storage is recession-proof. That no matter what the economy is doing, that data will continue to grow, and thus companies will have to continue to grow their storage infrastructure. I'm not sure that I buy it, but that just might be my nerves talking. Perhaps it's just that I tend to believe that the truth typically lies somewhere in the middle. So, I thought I'd take a minute and describe what I think is going to happen this year. No guarantees, I can't predict the future, but a little speculation is always fun.

Storage will continue to grow just not as fast
Yup, I do believe that the amounts of data that companies keep on hand will continue to grow. Just not at the same rate it has in the past. Depending on whom you want to believe, year to year growth for storage has been growing at 40-60% CAGAR or even more. I'm guessing that in 2009 we are not going to see that kind of growth. Due to the reduced sales volume that most companies will see due to the recession, there's got to be an attendant reduction in the amount of data that gets created. How much is the $64,000.00 question. I suspect that the growth rate might be cut in half, or even more. Add to this the fact that budgets are getting slashed and storage managers are going to be looking to expend the useful life of storage that they have on hand and it makes me think that this year the average growth rate for storage is going to sit somewhere between 5-10%. So, overall I believe that the volume of raw disk sales is going to drop dramatically. I'm probably not the only one looking at things that way, look at the major storage vendors, they are all cutting forecasts, laying off people, and generally cutting back.

It's not all doom and gloom
I think that in this situation, however, there is some opportunity. Storage providers that can help the storage managers at their clients address the issues of their budget reductions and to find ways to do more with less will get quite a bit of business. I also think that companies, like the one I work for, that can package best of breed hardware and software into solutions that are very cost effective will also do well. Vendor loyalty, however, is going to go out the window and companies that were once locked into a single vendor will look at other vendors if they perceive that other vendor as being more cost effective. Again, this means opportunity for vendors to get into companies that they had previously been locked out of. I predict that we are going to see some of the major storage users leave the "big four" (EMC, NepApp, Hitachi, IBM) and moving to storage from smaller players in an effort to reduce both CAPEX and OPEX costs.

The year of storage efficiency and virtualization
Finally, this year it will all be about efficiency and virtualization. I'm betting that CIOs will actually accelerate any server virtualization projects that they currently have in the works in order to get those reduced costs as quickly as they can get them. However, what they will find is that unless they are quite careful, their server virtualization project might result in increased spending on storage, backup/recovery, and DR that they hadn't planned for. This can be overcome to some extent by partnering with storage suppliers that understand the issues involved when dealing in a virtualized world. I also predict that sales of things like data deduplication, and thin provisioning are going to accelerate this year. Again, all of this is in an effort to "do more with less" on the part of storage consumers.
So, overall, I'm cautiously optimistic that for those that can show their customers how to "do more with less" this year will be just a challenge, but in the end the will survive. For those who continue to try and do business as usual, well, they mind for this year to be more difficult.

--joerg

IBM XIV Could Be Hazardous to Your Career

So, I haven't blogged in a while. I guess I should make all of the usual excuses about being busy (which is true), etc. But the fact of the matter is that I really haven't had a whole heck of a lot that I thought would be of interest, certainly there wasn't a lot that interested me!

But now, I have something that really get my juices flowing. The new IBM XIV. I don't know if you've heard about this wonderful new storage platform from the folks at IBM, but I'm starting to bump into a lot of flolks that are either looking seriously at one, or have one, or more, on the floor now. It's got some great pluses:

• It's dirt cheap. On top of that, I heard that IBM is willing to do whatever it takes on price to get you to buy one of these boxes, to the point that they are practically giving them away. And, as someone I know and love once said "what part of free, isn't free"?
  
• Fiber channel performance from a SATA box. I guess that's one of the ways that they are using to keep the price so low.
  
• Teir 1 performance and reliability at a significantly lower price point.
  

So, that's the deal, but like with everything in this world, there's no free lunch. Yes, that's right, I hate to break it to you folks, but you really can't get something for nothing. The question to ask yourself is, is the XIV really too good to be true? The answer is yes, it is.

But the title of this blog is pretty harsh, don't you think? Well, I think that once you understand that the real price you are paying for the "almost free' XIV could be your career, or at least your job, then you might start to understand where I'm coming from. How can that be? Well, I think that in most shops, if you are the person who brought in a storage array which eventually causes a multi-day outage in your most critical systems that your job is going to be in jeopardy. And that's what could happen to you if you buy into all of the above from IBM regarding the XIV.

What are you talking about Joerg?!? IBM says that the XIV is "self healing", and that it can rebuild the lost data on a failed drive in 30 minutes or less. So how can what your said be true? Well folks, here's the dirty little secret that IBM doesn't want you to know about the XIV. Due to its architecture if you ever lose two drives in the entire box (not a shelf, not a RAID group, the whole box all 180 drives) within 30 minutes of each other, you lose all of the data on the entire array. Yup, that's right, all your tier 1 applications are now down, and you will be reloading them from tape. This is a process that could take you quite some time, I'm betting days if not weeks to complete. That's right, SAP down for a week, Exchange down for 3 days, etc. Again, do you think that if you brought that box in after something like that your career at this company wouldn't be limited?

So, IBM will tell you that the likely hood of that happening is very small, almost infinitesimal. And they are right, but it's not zero, so you are the one taking on that risk. Here's another thing to keep in mind. Studies done at large data centers have show that disk drives don't fail in a completely random way. They actually fail in clusters, so the chances of a second drive failing within the 30 minute window after that first drive failed are actually a lot higher than IBM would like you to believe. But, hey, let's keep in mind that we play the risk game all the time with RAID protected arrays, right? But the big difference here is that the scope of the data loss is so much greater. If I have a failure in a 4+1 RAID-5 raid group, I'm going to lose some LUNs, and I'm going to have to reload that data from tape. However, it's not the entire array! So I've had a much smaller impact across my Tier 1 applications, and the recovery from that should be much quicker. With the XIV, all my Teir 1 applications are down, and they have to all be reloaded from tape.

Just so you don't think that I'm entirely negative about the XIV let me say that what I really object to here is the use of a XIV with Tier 1 applications or even Tier 2 applications. If you want to use one for Tier 3 applications (i.e. archive data) I think that makes a lot of sense. Having your archive down for a week or two won't have much in the way of a negative impact on your business, unlike having your Tier 1 or Tier 2 applications down. The once exception to that I can think of is VTL. I would never use a XIV as the disks behind a VTL. Ca you imagine what would happen if you lost all of the data in your VTL? Let's hope that you have second copies of the data!

Finally, one of the responses from IBM to all of this is "just replicate the XIV if your that worried". They right, but that doubles the cost of storage, right?

The Future of storage, or is the disk drive dead?

We are running out of places to put things.

Data continues to grow at a frightening rate. According to an IDC study there was about 281 Exabytes of data stored on disk in 2007 word wide. This data is growing at CAGR of about 70%. At this rate, in 3 years there will be about 1400 Exabytes of data sitting on disk.
Now, a lot of this data is sitting on people's desktops, laptops, ipods, phones, digital cameras, etc. right now. However, things like cloud storage will change all of that. Heck, we are seeing some of the change right now with things like social networking sites, photo sharing sites, etc. IDC says that for 85% of that data a corporate entity will be responsible for the protection and security of the data.
So, in the future, we are going to have to store a lot more data than we do today, a LOT more data. How are we going to do that? Just the physical aspect of getting exabytes of data on the floor is going to be a challenge. I don't even want to talk about protecting and managing that much data. But for now, I want to talk about the density of the hard disk drive since that's going to soon become the physical limit of what we can store on the floor of our data centers.

The bits are getting too small!

Enterprise disk drive capacity has obeyed Moore's Law and doubled every 18 months for quite a few years. However, this growth has appears to be slowing down over the last 5 years, and it is now taking approximately 29-30 months to double the capacity of an Enterprise disk Drive.
This shows that we are nearing the maximum areal density (max capacity) of current disk drive technology called the superparamagnetic limit. Areal density as it refers to disk drives is measured by the number of bits per inch (bpi) times the number of tracks per inch (tpi).
The areal density of disk storage devices has increased dramatically since IBM introduced the RAMAC in 1956. RAMAC had an areal density of two thousand bits per square inch, while current-day disks have reached 100 billion bits (100 gigabits per square inch). Perpendicular recording is expected to increase storage capacity even more over time, but we do appear to be approaching the limit.
As the magnetic bits get smaller, at some point they no longer hold their charge. Thermal fluctuations reduce the signal strength and render the bits unstable. However, this ultimate areal density keeps changing as researchers find new techniques for recording and sensing the bit. Years ago the limit was thought to be 20 gigabits per square inch. Today, the limit is several hundred gigabits per square inch, and more than a terabit is expected soon. But that's about all you can get out of the technology.

Denser is faster.

Increasing the density of hard disk drives has a side benefit. It makes the drives faster as well. This is really quite logical when you think about it. Since the closer things are together on the drive, the more data passes by a read/write head in the same period of time thus making the drive faster.

Shorter term solution.

So, if the disk drive is not going to be able to continue to provide us with the kinds of capacities we are going to need in the future, what will? Well, there are a number of things that are being looked at by a lot of folks who are a lot smarter than me! But in the short term, things like SSD look promising once we work out some of the kinks. Specifically, the write speed issue. Until we can get that up I'm not sure how much general acceptance SSD technology is going to get. Price, I am convinced, will take care of itself as the scales of economy kick in. Holographic storage, some people have been working on this for a very long time and it seems like such a promising technology, but it has yet to come to fruition. There is one company out there that's trying to ship a product, but they recently pushed off their release date until the end of this year. Still, if they can work out the kinks, it definitely has promise, especially for media applications. But what about beyond that? What technologies are the researchers looking at that sound really cool? I look at some of those next.

Sci-Fi data storage.

So, this is where it gets fun. Some of the technologies that researches are currently looking into really do sound like something out of a Sci-Fi movie. Here are some examples of the stuff I'm talking about:

Nanodots - A nanodot has north and south poles like a tiny bar magnet and switches back and forth (or between 0 and 1) in response to a strong magnetic field. Generally, the smaller the dot, the stronger the field required to induce the switch. Until now researchers have been unable to understand and control a wide variation in nanodot switching response. A NIST team significantly reduced the variation to less than 5 percent of the average switching field and also identified what is believed to be the key cause of variability. Nanodots, as small as 50 nanometers (nm) wide could be used to storage data.

Array's of magnetic snakes - According to a weekly digest from the American Physical Society (APS), physicists at Argonne National Laboratory (ANL) have found that under certain conditions, magnetic particles could form magnetic ‘snakes' able to control fluids. According to the researchers, this magnetic self-assembly phenomena may be used to make the next generation of magnetic recording media or transparent conductors based on self-assembled conducting networks of magnetic micro-particles.

Nanowires - Switchable fluorescent proteins, able to move reversibly between two optical states, have been known from some years. But now, German researchers have discovered the mechanism behind this optical switch in a protein found on the tentacles of a sea anemone. According to the researchers from the University of Pennsylvania, Drexel University and Harvard University, barium titanium oxide nanowires suspended in water could hold 12.8 million GB per square centimeter. If the memory density can be realized commercially, "a device the size of an iPod Nano could hold enough MP3 music to play for 300,000 years without repeating a song or enough DVD-quality video to play movies for 10,000 years without repetition," the University of Pennsylvania researchers said.

Is the disk drive dead?

So, does this mean that the disk drive is dead in the future? I don't think so. I believe that the disk drive we know and love will simply move from one tier of storage to another. We are already seeing some of this movement with the implementation is backup to disk. Technologies such as data deduplication will continue to accelerate this process, and the addition of new primary data storage technologies will simply end the process by pushing hard disk drives from on-line primary storage to what will be considered near-line storage in the future. Long live the disk drive!

VMWare and how it effects Storage

"VMWare Changes Everything"

That's a lovely marketing phrase, but when it comes to storage, it does, and it doesn't. What you really need to understand is how VMWare can effect your storage environment as well as the effects that storage has on your VMWare environment. Once you do, you'll realize that it's really just a slightly different take on what storage administrators have always battled. First some background.

Some Server Virtualization Facts

1. The trend of server virtualization is well under way and it's moving rapidly from test/dev environments into production environments. Some people are implementing in a very aggressive way. For example, I know one company who's basic philosophy is "it goes in a VM unless it absolutely can be proven it won't work, and even then we will try it there first."
2. While a lot of people think that server consolidation is the primary motivating factor in the WMVware trend, I have found that many companies are also driven by Disaster Recovery since replicating VMs is so much easier then building duplicate servers at a DR site.
3. 85% of all virtual environments are connected to a SAN, that's down from nearly 100% a short time ago. Why? Because NFS is making a lot of headway, and that makes a lot of sense since it's easier to address some of the VMWare storage challenges with NFS than it is with traditional fiber channel LUNs.
4. VMWare changes the way that servers talk to the storage. For example, they force the use of more advanced file systems like VMFS. VMFS is basically a clustered file system and that's needed in order to perform some of the more attractive/advanced things you want to do with VMWare like VMotion.

Storage Challenges in a VMWare Environment

1. Application performance is dependant on storage performance. This isn't news for most storage administrators. However, what's different is that since VMWare can combine a number of different workloads all talking through the same HBA(s), the result is that the workload as seen by the storage array turns into a highly random, usually small block I/O workload. These kinds of workloads are typically very sensitive to latency much more than they require a great deal of bandwidth. Therefore the storage design in a VMWare environment needs to be able to provide for this type of workload across multiple servers. Again, something that storage administrators have done in the past for Exchange servers, for example, but on a much larger scale.
2. End to end visibility from VM to physical disk is very difficult to obtain for storage admins with current SRM software tools. These tools were typically designed with the assumption that there was a one-to-one correspondence between a server and the application that ran on that server. Obviously this isn't the case with VMWare, so reporting for things like chargeback becomes a challenge. This also effects troubleshooting and change management as well since the clear lines of demarcation between server administration and storage administration are now blurred by things like VMFS, VMotion, etc.
3. Storage utilization can be significantly decreased. This is due to a couple of factors, the first of which is that VMWare requires more storage overhead to hold all of the memory, etc. so that it can perform things like VMotion. The second reason that VMWare uses more storage is that VMWare admins tend to want very large LUNs assigned to them to hold their VMFS file systems and to have a pool of storage that they can use to rapidly deploy a new VM. This means that there is a large pool of unused storage sitting around on the VMWare servers waiting to be allocated to a new VM. Finally, there is a ton of redundancy in the VMs. Think about how many copies of Windows are sitting around in all those VMs. This isn't new, but VMware sure shows it to be an issue.

Some Solutions to these Challenges

As I see it there are three technical solutions to the challenges posed above.

1. Advanced storage virtualization - Things like thin provisioning to help with the issue of empty storage pools on the VMWare servers. Block storage virtualization to provide the flexibility to move VMWare's underlying storage around to address issues of performance, storage array end of lease, etc. Data de-dupulication to reduce the redundancy inherent in the environment.
2. Cross domain management tools - Tools that have the ability to view storage all the way from the VM to the physical disk and to correlate issues between the VM, server, network, SAN, and storage array are beginning to come onto the market and will be a necessary part of any successful large VMWare rollout.
3. Virtual HBAs - These are beginning to make their way onto the market and will help existing tools to work in a VMWare environment.

Conclusion

Organizations need to come to the realization that with added complexity comes added management challenges and that cross domain teams that encompass VMWare Admins, Network Admins, and SAN/Storage Admins will be necessary in order for any large VMWare rollout to be successful. However, the promise of server virtualization to reduce hardware costs and make Disaster Recovery easier is just too attractive to ignore for many companies and the move to server virtualization over the last year shows that a lot of folks are being drawn in. Unfortunately, unless they understand some of the challenges I outlined above, they may be in for some tough times and learn these leassons the hard way.

--joerg

EMC World 2008

Well folks, I just got back from EMC World 2008 in Las Vegas. It was a fun trip, but man am I tired. There was a lot of walking at the concference as well as a lot of late nights having fun after the conference sessions were over each day.

I'll have some more detailed postings on what I think about some of the technology I saw at EMC World a little later. Right now I just wanted to talk a a bit about the general trends and feelings I got from the convention.

First and foremost, EMC has finally awakened to the fact that people want de-duplicating products, and they want them now. EMC has really been behind the eight ball when it comes to dedupe. I don't know if it was because of their close relationship with FalconStor in the past, or what, but they really didn't have much of a story to tel l when it came to dedupe, and start-ups like Data Domain where definatly eating EMC's lunch in that market. But the EMC giant has definitely awakened from it's slumber, and introduced some interesting new products.

Basically, the new products fall into two categories, first to a software addition to the existing DL400 line which provides deduplication. The second is a new line of deduplication engines that provide much the same capabilities as Data Domain does. The main differences are that EMC's appliances provide the users with a choice between in-line, post processing, or no deduplication at all. They also have a well designed VTL feature which is an area that Data Domain has been struggling in.

The other area that EMC was emphasizing was "green computing". A lot of this was nothing more than marketing hype and spin on existing products. However, they did mention a feature that they would provide soon that really was "green computing". The idea was to spin down drives that weren't currently in use. Now when EMC didn't introduce and specific products yet, they did suggest that we would see this technology first in the VTLs, but that it could make an appearance in the overall CLARiiON line in the not too distant future.

Overall, a lot of EMC marketing around "green", but some new technology and a good opportunity to talk with the folks at EMC about where they are going with some of the products. I got to spend a little sime talk ing with the folks who work on StorageScope about reporting, and support for AIX VIO in Control Center in general.

Finally, I took my wife along so she could have some fun as well, and I think she ended up having more fun than I did. Las Vegas is a great place for shopping, hanging out in the SPA, and generally having a good time. All of which she did while she was there. We also went to see Phantom of the Opera, which was great. Overall, a good trip for both of us. More details on a latter posting.

--joerg