Snyder’s First Law of Capacity Planning
Posted: October 27, 2011 Filed under: capacityplanning, sysadmin, systems Leave a commentThere’s a phrase I hear all the time that’s been bothering me for a while now. I hear something like it about once a week on average when I ask about requirements for a system that needs to be provisioned. It usually goes something like the following:
“Well, I think we’ll need 20GB, so let’s go ahead and ’round up’ [sic] to 50GB…y’know, just to be safe. Storage is cheap, right? I mean, just yesterday I saw a 1TB drive for like $60!”
…and it drives me nuts.
I’d like to apologize ahead of time for using “you” below if you, personally, have never said this to me. It’s conversational in voice, not accusatory…
First of all, what you’ve done here isn’t “rounding up” in any way, shape, or form. What you’ve done is increase your requirement by 150%. Second: Yes, perhaps storage is cheap relative to other system components – like, say, RAM. However, throw in high-speed SCSI disk, multiple spindles for performance and redundancy, a support contract, support subsystems like SAN switches and cabling (also redundant), support staff costs, etc. and it’s significantly more expensive per-GB than that Hitachi SATA HDD you saw in the bargain bin at the Office Depot. Third, if you put two-and-a-half times more disk into every request than you really need, that means we have to buy two-and-a-half times more disk. Generally speaking, you know what that means as a function of cost? You guessed it! We’re now spending (approx.) two-and-a-half times more on storage than we need to! Taking all this into account, it gives the original statement the flavor of “Let’s spend more money than we need to because it’s cheap!”
Now, if this storage were just a one-time, fixed-cost item then it really wouldn’t be that bad…but it’s not. At this point, I’d like to coin a new Law – we’ll call it Snyder’s First Law of Capacity Planning. It goes something like this:
“A resource – once allocated – has a probability of being de-allocated of near zero.”
Applied to this problem, it becomes “Once storage has been allocated, you will very likely never get it back.” Not only will this storage be continuing to use power, cooling, etc. ad infinitum, but we will now need to take it into account in every storage subsystem outage, maintenance, and migration from now until the end of time.
On the other hand, the above is precisely the kind of thing I get paid to be mindful of so that the people who are coming to me asking for resources don’t have to. So, in the interests of continuing to make a living while remaining sane, how can I make this better?
Well, for one thing, it’s seemed a little bit silly to me for quite a while now that any incoming requester should be asking for resources this specifically. What I mean is: I’m the Systems Guy, shouldn’t I be taking incoming requirements and translating them into specific resources? So maybe a better approach to the problem would be asking questions of the form “How much data do you have and what are you going to do with it?” rather than “How much disk do you think you’ll need?” (Of course, this won’t work in all cases, but it seems like it could be a fairly effective approach most of the time.)
Another way to handle this when asked for more storage than is necessary might be to have numbers on-hand going into these meetings. This way, when presented with consumer storage costs of something like 6 cents/GB I could “counter” with, say, $14/GB (or whatever it actually costs – I’m just making up a number here). A little end user education might go a long way.
I’m sure there are other approaches that I haven’t thought of yet; I’ll be sure to update the post if I think of any. I’d be glad to hear any ideas you’ve got in the comments…
Five Years Ago
Posted: October 25, 2011 Filed under: general, personal 1 CommentThere are a few things I’ve learned since I graduated college. In fact, it’s probably fair to say that I’ve learned more since I graduated than I did in my entire four years. It might even be possible to say that I’ve learned more in the last 10 years than I did in the entirety of my time at school (16 or so years at this point), but that’s a little hard to quantify. I mean, how do you compare “I learned how to wipe my own ass” with “I learned that mortgages are expensive as balls”?
Okay, so I probably knew how to wipe my own ass long before I was in the first grade. My point is that it seems like every 5-10 years or so I look back at my 5- or 10-years-younger self and say something like, “Man, what an idiot I was!” I’d like to try and recreate that here:
5 (looking back on 0):
Good Lord, what a no-walking, no-talking, shitting-all-over-myself baby I was back then!
10 (looking back on 5):
Man, I’ve come a long way. I mean, I could hardly even read!
15 (looking back on 10):
Are you kidding me, with the playing tag and watching cartoons? If only I would’ve found out about girls sooner…
20 (looking back on 15):
If only I would’ve found out about girls sooner…
25 (looking back on 20):
If only I would’ve found out about girls later…
30 (looking back on 25):
If only I would’ve found out about compound interest sooner…
Alright, I probably didn’t do this justice – a one-liner per five years is pretty thin – but this is really meant to be food for thought. What would you tell yourself if you could go back in time 5 years?
Surprises
Posted: October 24, 2011 Filed under: general, personal 1 Comment[Disclaimer: I’m a little scattered – typical “I need a vacation to recover from my vacation” state – so this post may be a little disjointed.]
I was thinking today about things I’ve been surprised by recently:
It Rains Inside in Mexico
Okay, so that’s a bit of an exaggeration – maybe not everywhere in Mexico. The resort where we stayed in Cancun had a really impressive open entryway and skylights running the length of the hotel. (Note to self: Put more picture in posts. People like pictures.) This seemed pretty reasonable, given that someone told us they only got something like 30 days of rain a year in Cancun. Well…we happened to get 3-4 of those days. As it turns out nothing in that hotel was particularly water-tight, resulting in towels lying about on the tile for the majority of our stay. But hey, it was still 80 degrees, so no complaints.
AI Class
Turns out, there are some bits of Artificial Intelligence that are fairly tricky. Particularly if one hasn’t taken a statistics course in 10 or so years. And particularly if one is out of the country sans PC while the online courses are taking place, giving one a single day to watch all the video lectures, take all the quizzes, and complete the homework. It begs the question: Why on earth couldn’t they have done the easy unit on depth-first search while one was on vacation?
Blizzcon
Not that Blizzcon took place – that happens every year – but what was announced at this year’s con. Specifically: The next expansion, Mists of Pandaria. Again, it’s no surprise that they’re releasing another expansion; I’ve come to expect that about once a year out of Bliz. I figured they’d probably tweak around with the talent trees and all of that stuff. Pretty much par for the course for an expansion that adds new levels. …but panda monks? Seriously? This one might be where I call it quits with WoW. I mean, I loved the movie…but…seriously?
Yes, that’s actually Donald Knuth
A buddy of mine sent me a link to a video of Randall Munroe speaking at Google in 2007. I hadn’t seen the video before and I love xkcd, so I figured I’d go ahead and check it out. Well, it just so happens that one of the Googlers had arranged for Donald Knuth – yes, the Donald Knuth – to be present at the talk. In fact, she went so far as to get hime to ask Munroe a question about a comic he had drawn about Knuth. It took me a minute to realize what was actually going on. I actually thought they were just kidding around – “ha ha, the guy who asked the question must be Donald Knuth.” Not kidding around…that is, in fact, him. In the flesh. Awesome.
The Bucket Problem – Part II
Posted: October 13, 2011 Filed under: general 1 CommentFor the first part of this disussion, see The Bucket Problem – Part I.
So having multiple buckets fixes the problem beautifully…if only problem you’re trying to solve is “Don’t Fuck Up Production”. (Incidentally, what I’ve described generally only mitigates this…but that’s a whole different discussion.) What if you have other goals? Like, say, not spending One Billion Dollars on replicating every server you build in every data center you manage in every country in which you have a presence? (Note: Seperate data centers actually is a form of the Bucket Problem, which should become apparent shortly).
Okay, so you start to delve into the problem a bit more thoroughly and say “Development will be set up thus and Production will be set up so“, and you feel totally satisfied that you’ve arranged it just perfectly – all of your resources are allocated in perfect harmony. …until they’re not. So you’ve got two buckets – dev and prod – and inevitably one of them fills up first. Now you’ve got to go to the people who write the checks for the equipment and say something like “We’re only using 70% of our capacity, but we’ve got to buy more.” Why? Because the resources in the particular bucket you need are used up, and you’ve set up your buckets such that Ne’er the Twain Shall Meet.
Think of this as the Shampoo/Conditioner problem, which is a form of the Bucket Problem stated thus: When you purchase two bottles of hair product – one of shampoo, one of conditioner – even if they are precisely the same size at purchase time, you will run out of one of the other (generally shampoo) significantly earlier than the other. The interesting thing about this particular problem is that it’s typically cyclical. Run out of shampoo? Buy just shampoo the next time you’re at the store. …then you run out of conditioner, since you only started out with a partial bottle to begin with. And so on.
Alternately, maybe you prefer to spend staff time playing the “Shell Game”, shuffling resources around between storage subsystems, networks, etc. to try to meet the requirements of this-or-that bucket. This could involve an arbitrary amount of difficulty, from just changing some configuration parameters to having to haul physical hardware to another data center. This problem is a little more like having multiple bank accounts, or perhaps a more-accessible way to state it is having one or more gift cards. I think everyone has had the following interaction with a cashier at least once: “Okay, so there’s $3.17 on this card, $7.49 on that one, put $20 of it on my Visa, and I’ll pay the rest in cash.” How fun is that?
There are a number of Bucket Problems and corrolaries I can come up with off the top of my head – multiple email addresses/address books (the particular email/contact information you’re looking for is never in the account where you’re looking), booze & mixers (you never run out of both at the same time), chips and salsa (classic problem!). I think you get the idea.
Note that most of the above examples involve exactly 2 buckets. As you might expect, this problem tends to get (exponentially?) worse as the number of buckets increases.
So, that’s a brief introduction to the Bucket Problem. Where have you seen this in your life?
The Bucket Problem – Part I
Posted: October 12, 2011 Filed under: general 1 CommentThere is a problem that I like to think of as The Bucket Problem. I run into it perhaps most often at work, but the more I think about it I realize that this particular problem manifests itself in all different areas of life. It’s a pain in the ass, it takes up way more of my time than I feel it should, and I’d really like to be done with it once and for all.
So…what is it?
Well, The Bucket Problem can be stated (very) informally thus: Any time you split your resources into two (or more) buckets (“pools” or “piles”, if you prefer…I like “buckets”) you’ll wish you hadn’t. Even if you had very good reasons for doing so. In fact, even if you really had no choice other than to do so.
The (classic?) example that I’m thinking of: the splitting of resources between development environments and production environments – dev “buckets” and prod “buckets”. Now, if you’re not a technology person then this bears some amount of explanation. I’ll stick to Internet services, because presumably if you’re reading this you have some grasp of them – at the very least, you know they exist.
You have a service – say, WordPress – which you host at wordpress.com. End users of this service have all the usual expectations for this service – it has to be up all the time, it has to be easy to use, it has to have this-or-that feature, etc. As a guy on the systems side of the house, the first of those criteria is really the critical one. It can’t go down. Ever. (Or, if you prefer, it can’t go down unexpectedly.) That is what Production means. It is a “product” in the sense that you are making it available to customers (whether they’re directly paying for it or not), and where the Internet is concerned it’s simply unacceptable for that product to be unavailable.
…but in order to remain relevant and continue to be a great product (or to become a great product if it’s not to begin with) work probably has to be done on it over time. Code has to be written and modified as a part of the natural course of things. Now, you don’t want to do this work on the version of the service that’s running in production – I mean, what if you fuck it up? So where do you do it? Development, of course. It’s easy, right: We just split things into Development and Production buckets and put in place some kind of a promotion-to-production process/policy. Hell, maybe we even do QA! <gasp> We could even set up another bucket just for QA. Problem solved. Right?
[This discussion is continued at The Bucket Problem – Part II]
[I didn’t mean to digress this deeply into dev vs. prod, but I think it will end up being valuable in the end. Bonus points if you can see where I’m headed with this. (Oh…and I welcome comments, but no fair giving it away if you work with me!)]
This Space Reserved
Posted: October 11, 2011 Filed under: general, personal 2 CommentsCoffee: check.
Cookies: check.
Ommwriter: check.
…
Some days, inspiration just doesn’t come. I do have an idea for this space – one that’s already partially-written, in fact – but I’ve got to admit I’m not 100% comfortable posting it just yet. I think I’ve also already got “one foot on the beach”, as it were – vacation next week! – so I’m having a little trouble thinking about much of anything but wrapping things up at work and getting packed. (Incidentally, I doubt very much that I’ll be writing anything at all while I’m in Cancun.)
So…I reserve the right to post something in this space at a later date.
[I know, I know, it’s a cop-out. I’ll try and make it up, though. :-)]
Rap vs. Country
Posted: October 10, 2011 Filed under: music 1 Comment[Where were the posts on Saturday & Sunday? Well, I decided to take the weekend off. That wasn’t actually a part of the “rules” when I started out, but I think it’s generally a good policy and I’ll probably stick with it.]
It’s almost universally true that “country people” hate rap and “rap people” hate country. Sure, there are folks who take a middle ground or who generally listen to a little bit of everything, but by and large being at one end of the spectrum precludes one from liking both. Being a “middle of the road” kinda guy myself, I think this is an interesting phenomenon. Not because I think absolutes are absurd in most cases (they are), but because if you’re the type of person who actually listens to and thinks a bit about the lyrics, most of the time they’re singing/rapping/crooning/auto-tuning about the same damn thing.
Smoking & Drinking
What rapper worth his salt doesn’t pop a few bottles and blow a little kush in the urr? Examples of “chemical vice” abound in hip-hop. Well…this isn’t exactly unique to rap. The lyrics to an old Hank Williams classic (complete with the optional in-between shout-outs):
Why do ya drink? (To get drunk!)
Why do you roll smoke? (To get high!)
I think you get the picture here. Moving on…
Partying
This one is pretty easy, since just about any form of music has some kind of song associated with it, so we haven’t really revealed any great truths here: People like to have a good time.
Women
In both rap and country, they are objectified and exist almost exclusively as sex objects; this is true of almost any form of media. There are also female rappers/country stars who present a strong female image; also true in most/all media formats. However, there seems to be one universal exception: Momma. (Top 10 Country Songs for Mothers, 10 Hip-Hop Songs Mom Will Love).
Violence
Okay, so there’s a little divergence here in terms of scope. Most country songs are just going to talk about bar fights or scrappin’ with pappy, whereas most rap is more concerned with gang violence and shooting people. I’m not sure I can reconcile this one without over-simplifying to “They both have violence in them.” However, there is a reason I’ve put women and violence back-to-back – what could be the unifying factor on both fronts:
Infidelity
Major theme. Both rap and country feature a ton of it. Funny thing is, without hard data I’d say that there’s probably something like a 49/49 split in either genre between “giving” and “receiving” here – i.e., being the “cheater” and the “cheated-on” (with some small percentage going to the “cheated-with”).
So…maybe not as dissimilar as you’d thought?
[As this account has been completely anecdotal, I’d like to do some more research on this one and present some actual hard data. I reserve the right (since I wrote the damn thing in the first place) to expand upon this post in the future…]
Morton
Posted: October 7, 2011 Filed under: morton 1 Comment“Your finger, sir.”
The Clerk stared disinterestedly off into the distance. Morton hadn’t been paying attention…that seemed to happen a lot these days.
“I’m sorry?”
“Umm…finger?”
She made a half-hearted gesture toward the device poking out at him from beside the credit card scanner at the front of the sales counter. Somewhat dazed, he apologized again.
“What is it you’re asking?”
She gave a sigh, part boredom part contempt, before corporate protocol kicked in.
“Sir, I need you to put your finger into the scanner before I can process this sale.”
“Ah…yes…of course…”
Morton tentatively placed his withered digit into the finger stirrup before him. A younger Morton would have thought he was past this. A younger Morton would have been wrong.
The Clerk’s eyebrows raised slightly as she scanned the screen before her.
“That will be $27.39, sir.”
“Excuse me?”
“$27.39.”
Already past her initial astonishment, The Clerk’s mind now wandered. By the clock, she’d be off in 2 hours or so; maybe Bobby would be off work by then, and…
“Excuse me, but I have a coupon…”
He’d caught her off-guard. Most of his kind didn’t put up this kind of fuss. Thank God for corporate protocol.
“I’m sorry sir, but we aren’t allowed to honor discounts for folks with your…signature.”
She spat out the last word with barely-concealed contempt.
“I don’t understand, young lady. I just wanted this Snickers bar…”
…but he did understand. All too well. Ever since the insurance lobbyists had convince the government that it would be in “everyone’s best interests” to collect personal health data on every individual with any kind of coverage, things had been headed down this path. …and the fact of the matter was, Morton happened to have a family history of diabetes. Add to that a high blood-glucose level – verified by a blood test at every CVS, Meijer, and Wal-mart he’d entered for the past several years, as a matter of course – and the Consumer Health and Wellness Act of 2011 hit him hard any time his sweet tooth came to call.
$1.39 for the candy, $26.00 for his “potential”.
[…nowhere near 500 words. I apologize, but I’m still fleshing this one out. We’ll be seeing Morton again…]
Minimalism
Posted: October 6, 2011 Filed under: general, minimalism Leave a commentMinimalism.
[Note: There is a certain appeal to just leaving the post at that, but the word counter is staring me in the face…]
It’s something I’ve always wanted to give a shot. Examples of people doing so abound on The Interwebs.
There’s Allmylifeforsale:
“…an online project that explored our relationship to the objects around us, their role in the concept of identity, as well as the emerging commercial systems of the Internet.”
So Dude sold all of his stuff – pretty crazy, eh? Well, check out Mark Boyle, who actually wrote a book about living on $0 a year. …and then you have Graham Hill, who talks about “Less stuff, more happiness.” Okay, so that last guy is kind of a smarmy designer prick who apparently has the money to throw at making a 420 sq. ft. place into a sexy Condo From the Future, but in general there seems to really be something to this “Less is More” concept.
So I’ve read all kinds of articles and watched talks and all of that, but I’m not really sure how I’d get started actually doing it.
For one thing, most of the guys who live this kind of lifestyle (and I’ve only ever seen guys talking about this kind of thing) are single. I’m pretty sure I could get rid of quite a bit of my stuff fairly easily; on the short list would be all the crap in the basement and at least 90% of the books I own. However, I’m positive that saying, “Babe, I’m gonna get rid of all my shit and all of your shit” wouldn’t go over so well.
I also don’t think any of these guys have pets. Having, say, a goldfish doesn’t really increase your “stuff footprint” much. You need a bowl, a place to put it, and mebbe a little canister of fish food. Having a dog, on the other hand, is fairly “stuff intensive”; giant bag of food, dishes, leashes, chew toys, little doggie clothes. (Yes, we occassionally dress our dog…and he likes it, so stop looking at me that way.)
Oh, and do any of these guys celebrate holidays? I mean, where on earth does Mr. Star Trek Apartment store his Christmas ornaments. Hmmm…actually, now that I think of it, if I were living alone I probably wouldn’t have any Christmas crap to store. Nothing against putting up a Christmas tree, I just doubt that I’d actually do it if it were just me.
Maybe some day we’ll move to a smaller place and that will be the catalyst for getting rid of all my crap. Maybe. Or maybe I’ll end up packing it all up and lugging it from place to place every time I move. Or maybe it’s time to go ahead and start throwing things out, giving things away, putting things up for sale on craigslist.
meh…I’ll start tomorrow…
Simplicitas 