Snyder’s First Law of Capacity Planning

There’s a phrase I hear all the time that’s been bothering me for a while now. I hear something like it about once a week on average when I ask about requirements for a system that needs to be provisioned. It usually goes something like the following:

“Well, I think we’ll need 20GB, so let’s go ahead and ’round up’ [sic] to 50GB…y’know, just to be safe. Storage is cheap, right? I mean, just yesterday I saw a 1TB drive for like $60!”

…and it drives me nuts.

I’d like to apologize ahead of time for using “you” below if you, personally, have never said this to me. It’s conversational in voice, not accusatory…

First of all, what you’ve done here isn’t “rounding up” in any way, shape, or form. What you’ve done is increase your requirement by 150%. Second: Yes, perhaps storage is cheap relative to other system components – like, say, RAM. However, throw in high-speed SCSI disk, multiple spindles for performance and redundancy, a support contract, support subsystems like SAN switches and cabling (also redundant), support staff costs, etc. and it’s significantly more expensive per-GB than that Hitachi SATA HDD you saw in the bargain bin at the Office Depot. Third, if you put two-and-a-half times more disk into every request than you really need, that means we have to buy two-and-a-half times more disk. Generally speaking, you know what that means as a function of cost? You guessed it! We’re now spending (approx.) two-and-a-half times more on storage than we need to! Taking all this into account, it gives the original statement the flavor of “Let’s spend more money than we need to because it’s cheap!”

Now, if this storage were just a one-time, fixed-cost item then it really wouldn’t be that bad…but it’s not. At this point, I’d like to coin a new Law – we’ll call it Snyder’s First Law of Capacity Planning. It goes something like this:

“A resource – once allocated – has a probability of being de-allocated of near zero.”

Applied to this problem, it becomes “Once storage has been allocated, you will very likely never get it back.” Not only will this storage be continuing to use power, cooling, etc. ad infinitum, but we will now need to take it into account in every storage subsystem outage, maintenance, and migration from now until the end of time.

On the other hand, the above is precisely the kind of thing I get paid to be mindful of so that the people who are coming to me asking for resources don’t have to. So, in the interests of continuing to make a living while remaining sane, how can I make this better?

Well, for one thing, it’s seemed a little bit silly to me for quite a while now that any incoming requester should be asking for resources this specifically. What I mean is: I’m the Systems Guy, shouldn’t I be taking incoming requirements and translating them into specific resources? So maybe a better approach to the problem would be asking questions of the form “How much data do you have and what are you going to do with it?” rather than “How much disk do you think you’ll need?” (Of course, this won’t work in all cases, but it seems like it could be a fairly effective approach most of the time.)

Another way to handle this when asked for more storage than is necessary might be to have numbers on-hand going into these meetings. This way, when presented with consumer storage costs of something like 6 cents/GB I could “counter” with, say, $14/GB (or whatever it actually costs – I’m just making up a number here). A little end user education might go a long way.

I’m sure there are other approaches that I haven’t thought of yet; I’ll be sure to update the post if I think of any. I’d be glad to hear any ideas you’ve got in the comments…


Five Years Ago

There are a few things I’ve learned since I graduated college. In fact, it’s probably fair to say that I’ve learned more since I graduated than I did in my entire four years. It might even be possible to say that I’ve learned more in the last 10 years than I did in the entirety of my time at school (16 or so years at this point), but that’s a little hard to quantify. I mean, how do you compare “I learned how to wipe my own ass” with “I learned that mortgages are expensive as balls”?

Okay, so I probably knew how to wipe my own ass long before I was in the first grade. My point is that it seems like every 5-10 years or so I look back at my 5- or 10-years-younger self and say something like, “Man, what an idiot I was!” I’d like to try and recreate that here:

5 (looking back on 0):

Good Lord, what a no-walking, no-talking, shitting-all-over-myself baby I was back then!

10 (looking back on 5):

Man, I’ve come a long way. I mean, I could hardly even read!

15 (looking back on 10):

Are you kidding me, with the playing tag and watching cartoons? If only I would’ve found out about girls sooner…

20 (looking back on 15):

If only I would’ve found out about girls sooner…

25 (looking back on 20):

If only I would’ve found out about girls later…

30 (looking back on 25):

If only I would’ve found out about compound interest sooner…

Alright, I probably didn’t do this justice – a one-liner per five years is pretty thin – but this is really meant to be food for thought. What would you tell yourself if you could go back in time 5 years?

 


Surprises

[Disclaimer: I’m a little scattered – typical “I need a vacation to recover from my vacation” state – so this post may be a little disjointed.]

I was thinking today about things I’ve been surprised by recently:

It Rains Inside in Mexico

Okay, so that’s a bit of an exaggeration – maybe not everywhere in Mexico. The resort where we stayed in Cancun had a really impressive open entryway and skylights running the length of the hotel. (Note to self: Put more picture in posts. People like pictures.) This seemed pretty reasonable, given that someone told us they only got something like 30 days of rain a year in Cancun. Well…we happened to get 3-4 of those days. As it turns out nothing in that hotel was particularly water-tight, resulting in towels lying about on the tile for the majority of our stay. But hey, it was still 80 degrees, so no complaints.

AI Class

Turns out, there are some bits of Artificial Intelligence that are fairly tricky. Particularly if one hasn’t taken a statistics course in 10 or so years. And particularly if one is out of the country sans PC while the online courses are taking place, giving one a single day to watch all the video lectures, take all the quizzes, and complete the homework. It begs the question: Why on earth couldn’t they have done the easy unit on depth-first search while one was on vacation?

Blizzcon

Not that Blizzcon took place – that happens every year – but what was announced at this year’s con. Specifically: The next expansion, Mists of Pandaria. Again, it’s no surprise that they’re releasing another expansion; I’ve come to expect that about once a year out of Bliz. I figured they’d probably tweak around with the talent trees and all of that stuff. Pretty much par for the course for an expansion that adds new levels. …but panda monks? Seriously? This one might be where I call it quits with WoW. I mean, I loved the movie…but…seriously?

Yes, that’s actually Donald Knuth

A buddy of mine sent me a link to a video of Randall Munroe speaking at Google in 2007. I hadn’t seen the video before and I love xkcd, so I figured I’d go ahead and check it out. Well, it just so happens that one of the Googlers had arranged for Donald Knuth – yes, the Donald Knuth – to be present at the talk. In fact, she went so far as to get hime to ask Munroe a question about a comic he had drawn about Knuth. It took me a minute to realize what was actually going on. I actually thought they were just kidding around – “ha ha, the guy who asked the question must be Donald Knuth.” Not kidding around…that is, in fact, him. In the flesh. Awesome.


The Bucket Problem – Part II

For the first part of this disussion, see The Bucket Problem – Part I.

So having multiple buckets fixes the problem beautifully…if  only problem you’re trying to solve is “Don’t Fuck Up Production”. (Incidentally, what I’ve described generally only mitigates this…but that’s a whole different discussion.) What if you have other goals? Like, say, not spending One Billion Dollars on replicating every server you build in every data center you manage in every country in which you have a presence? (Note: Seperate data centers actually is a form of the Bucket Problem, which should become apparent shortly).

Okay, so you start to delve into the problem a bit more thoroughly and say “Development will be set up thus and Production will be set up so“, and you feel totally satisfied that you’ve arranged it just perfectly – all of your resources are allocated in perfect harmony. …until they’re not. So you’ve got two buckets – dev and prod – and inevitably one of them fills up first. Now you’ve got to go to the people who write the checks for the equipment and say something like “We’re only using 70% of our capacity, but we’ve got to buy more.” Why? Because the resources in the particular bucket you need are used up, and you’ve set up your buckets such that Ne’er the Twain Shall Meet.

Think of this as the Shampoo/Conditioner problem, which is a form of the Bucket Problem stated thus: When you purchase two bottles of hair product – one of shampoo, one of conditioner – even if they are precisely the same size at purchase time, you will run out of one of the other (generally shampoo) significantly earlier than the other. The interesting thing about this particular problem is that it’s typically cyclical. Run out of shampoo? Buy just shampoo the next time you’re at the store. …then you run out of conditioner, since you only started out with a partial bottle to begin with. And so on.

Alternately, maybe you prefer to spend staff time playing the “Shell Game”, shuffling resources around between storage subsystems, networks, etc. to try to meet the requirements of this-or-that bucket. This could involve an arbitrary amount of difficulty, from just changing some configuration parameters to having to haul physical hardware to another data center. This problem is a little more like having multiple bank accounts, or perhaps a more-accessible way to state it is having one or more gift cards. I think everyone has had the following interaction with a cashier at least once: “Okay, so there’s $3.17 on this card, $7.49 on that one, put $20 of it on my Visa, and I’ll pay the rest in cash.” How fun is that?

There are a number of Bucket Problems and corrolaries I can come up with off the top of my head – multiple email addresses/address books (the particular email/contact information you’re looking for is never in the account where you’re looking), booze & mixers (you never run out of both at the same time), chips and salsa (classic problem!). I think you get the idea.

Note that most of the above examples involve exactly 2 buckets. As you might expect, this problem tends to get (exponentially?) worse as the number of buckets increases.

So, that’s a brief introduction to the Bucket Problem. Where have you seen this in your life?


The Bucket Problem – Part I

There is a problem that I like to think of as The Bucket Problem. I run into it perhaps most often at work, but the more I think about it I realize that this particular problem manifests itself in all different areas of life. It’s a pain in the ass, it takes up way more of my time than I feel it should, and I’d really like to be done with it once and for all.

So…what is it?

Well, The Bucket Problem can be stated (very) informally thus: Any time you split your resources into two (or more) buckets (“pools” or “piles”, if you prefer…I like “buckets”) you’ll wish you hadn’t. Even if you had very good reasons for doing so. In fact, even if you really had no choice other than to do so.

The (classic?) example that I’m thinking of: the splitting of resources between development environments and production environments – dev “buckets” and prod “buckets”. Now, if you’re not a technology person then this bears some amount of explanation. I’ll stick to Internet services, because presumably if you’re reading this you have some grasp of them – at the very least, you know they exist.

You have a service – say, WordPress – which you host at wordpress.com. End users of this service have all the usual expectations for this service – it has to be up all the time, it has to be easy to use, it has to have this-or-that feature, etc. As a guy on the systems side of the house, the first of those criteria is really the critical one. It can’t go down. Ever. (Or, if you prefer, it can’t go down unexpectedly.) That is what Production means. It is a “product” in the sense that you are making it available to customers (whether they’re directly paying for it or not), and where the Internet is concerned it’s simply unacceptable for that product to be unavailable.

…but in order to remain relevant and continue to be a great product (or to become a great product if it’s not to begin with) work probably has to be done on it over time. Code has to be written and modified as a part of the natural course of things. Now, you don’t want to do this work on the version of the service that’s running in production – I mean, what if you fuck it up? So where do you do it? Development, of course. It’s easy, right: We just split things into Development and Production buckets and put in place some kind of a promotion-to-production process/policy. Hell, maybe we even do QA! <gasp> We could even set up another bucket just for QA. Problem solved. Right?

[This discussion is continued at The Bucket Problem – Part II]

[I didn’t mean to digress this deeply into dev vs. prod, but I think it will end up being valuable in the end. Bonus points if you can see where I’m headed with this. (Oh…and I welcome comments, but no fair giving it away if you work with me!)]


This Space Reserved

Coffee: check.

Cookies: check.

Ommwriter: check.

Some days, inspiration just doesn’t come. I do have an idea for this space – one that’s already partially-written, in fact – but I’ve got to admit I’m not 100% comfortable posting it just yet. I think I’ve also already got “one foot on the beach”, as it were – vacation next week! – so I’m having a little trouble thinking about much of anything but wrapping things up at work and getting packed. (Incidentally, I doubt very much that I’ll be writing anything at all while I’m in Cancun.)
So…I reserve the right to post something in this space at a later date.

[I know, I know, it’s a cop-out. I’ll try and make it up, though. :-)]


Rap vs. Country

[Where were the posts on Saturday & Sunday? Well, I decided to take the weekend off. That wasn’t actually a part of the “rules” when I started out, but I think it’s generally a good policy and I’ll probably stick with it.]

It’s almost universally true that “country people” hate rap and “rap people” hate country. Sure, there are folks who take a middle ground or who generally listen to a little bit of everything, but by and large being at one end of the spectrum precludes one from liking both. Being a “middle of the road” kinda guy myself, I think this is an interesting phenomenon. Not because I think absolutes are absurd in most cases (they are), but because if you’re the type of person who actually listens to and thinks a bit about the lyrics, most of the time they’re singing/rapping/crooning/auto-tuning about the same damn thing.

Smoking & Drinking

What rapper worth his salt doesn’t pop a few bottles and blow a little kush in the urr? Examples of “chemical vice” abound in hip-hop. Well…this isn’t exactly unique to rap. The lyrics to an old Hank Williams classic (complete with the optional in-between shout-outs):

Why do ya drink? (To get drunk!)
Why do you roll smoke? (To get high!)

I think you get the picture here. Moving on…

Partying

This one is pretty easy, since just about any form of music has some kind of song associated with it, so we haven’t really revealed any great truths here: People like to have a good time.

Women

In both rap and country, they are objectified and exist almost exclusively as sex objects; this is true of almost any form of media. There are also female rappers/country stars who present a strong female image; also true in most/all media formats. However, there seems to be one universal exception: Momma. (Top 10 Country Songs for Mothers, 10 Hip-Hop Songs Mom Will Love).

Violence

Okay, so there’s a little divergence here in terms of scope. Most country songs are just going to talk about bar fights or scrappin’ with pappy, whereas most rap is more concerned with gang violence and shooting people. I’m not sure I can reconcile this one without over-simplifying to “They both have violence in them.” However, there is a reason I’ve put women and violence back-to-back – what could be the unifying factor on both fronts:

Infidelity

Major theme. Both rap and country feature a ton of it. Funny thing is, without hard data I’d say that there’s probably something like a 49/49 split in either genre between “giving” and “receiving” here – i.e., being the “cheater” and the “cheated-on” (with some small percentage going to the “cheated-with”).

So…maybe not as dissimilar as you’d thought?

[As this account has been completely anecdotal, I’d like to do some more research on this one and present some actual hard data. I reserve the right (since I wrote the damn thing in the first place) to expand upon this post in the future…]