When I first walked in here in 1994, not only did I have complete control over my first Notes server (it was 3.0c on OS/2), I had to reassemble the server before I could do anything with it. Then I had to grab that mountain of OS/2 floppies (there were over 30 of them) and install OS/2. Install the network card and driver. Install the Notes server and client. Create a certification hierarchy. Think out a method of administration that would be easy, flexible and still do what I wanted.
I did all those things.
Then and only then could I do anything useful with the machine. But I had total control, and total responsibility. If the thing died, everybody knew who to call, and I both knew what to do and had the power and access to do it. The server was under my desk, after all.
It is now 2008. I don't even have a shell account on my Solaris production servers. The people who do are generally not Domino people, and if they do know anything about it, it's pure admin... they know almost nothing about development.
As a result, when something gets screwed up, everyone still calls me, and I usually end up sitting here looking like a jackass because (a) I don't know why the server crashed; (b) I have no real access to restart it or debug it and (c) the people who do have no better solution than to send an NSD off to IBM and hope support decodes it sometime before the next Ice Age.
I cannot install updates to the system.
I cannot patch the OS.
I cannot stop or restart the service or the machine from the Solaris shell.
I can do all the usual Domino-y things we all can do, but if the server is too far gone to be responding to the Remote Console, I am screwed. And so are tens of thousands of users. I don't lack skill and experience... I lack access and permission. The people who have access and permission lack experience and skill. And, importantly, a sense that they need to communicate with me and my team.
A big watchword around here for years was "avoid the single point of failure." Yeah, bite me. There is no such thing, in a world where professionals take it as their duty to cross-teach themselves. I know how to administer a Unix machine... I've been doing it since back in the days when these guys were trying hard to pass their MSCEs so they could be allowed to run NT 3.51. I know Domino. But I am also locked in a structure where work has been segmented so heavily, probably as some sort of make-work effort so that the bloated IT department doesn't get the hell hacked out of their FTEs the next time the organization changes direction, that we're stuck far too often dealing with this scenario:
"Hi, this is Turtle, something's busted bad, I need this one thing done right now."
"Well, Joe Clueless, the only guy who knows how to do that, is in training all week."
"Hey, I know how to take care of this, can you get me a shell account and I can fix it in about five minutes?"
"No, that's the responsibility of the Office of Misguided Initiatives, so you'll have to ask them to do that for you."
"Yeah, but I can do it for myself, I just don't have a login I can use."
"Sorry, you'll have to wait until Joe Clueless gets back from training."
"Lady, right now 43,000 users are sitting looking at 404 errors, and it was caused by something Joe did and then didn't test before he went to training."
"I guess you could call the Help Desk..."
Translation: even though you could do this for yourself and get better results, and even though we suck at doing this thing, you have to let us do it, not because we're the best at it, but because it's been assigned as our job to do it.
Most of you in various governmental situation probably recognize this.
But is it like that where you work? You have responsibility for your Domino environment but don't have complete control over it? Do you have to delegate important tasks to people who don't give a shit? People who went to a two-day class and think they know more than your CLP and you years in the business? Or do you live in a Lotusphereian paradise, where everybody's smart, competent, conscientious, and willing to focus on the results rather than on the process?
And if you do, can I come work there? Please?
1. Kevin Pettitt03/29/2008 12:15:12 AM
Sounds very similar to my current day job environment, only it's even worse for me personally since as "the contractor" I can't administer anything in production. And don't get me started on "change control"!
I was thinking recently about why in general there seems to be a trend in IT toward "more bodies with less skill". Lots of reasons I suppose. Here's a partial list:
1a) IT managers are rewarded more for having larger headcounts than for producing better results
1b) same with larger budgets.
2) IT staffing firms are highly incentivized to place the MOST people on a job as opposed the BEST people (maybe when clients clue in to realize that one senior guru is a way better value than two (or six) junior clowns, this will change)
3) Keeping hourly labor rates low is a higher priority than productive value-add (outsourcing anyone?)
4) In what is perhaps a self-fulfilling prophecy, the notion of anyone being an "IT Generalist" seems to have lost ground as more and more narrowly trained "specialists" have been churned out by various technical colleges. This can almost be thought of as the "blue-collarization" of IT.
That last point is worth emphasizing because it touches on the core of what makes Notes "special". You see the "problem" with Notes is that it is "white-collar" technology. Obviously end users are typically white collar professionals, but more importantly the technical skills required to build and administer Notes applications are within reach of that same white collar talent pool. This presents the opportunity, too often unrealized, for people who already "know the business" to be the ones dreaming up and build the software tools to make it run better.
I suppose you could call it "Class Warfare in the IT Deparment"
2. Dave Harris03/27/2008 07:01:50 AM
I recognise the pain. I used to be a Domino developer in a part of the UK civil service, where there was that sort of segmentation, although our Notes Admins were on top of their game. Now, although I'm supposed to run our internal Notes network, I only have OS access to two of the six, which are here in Malaysia. The primary ones are in the UK and our security "manager" says I'm not allowed to touch them. He's also married to the MD so cannot be criticised.
3. Patrick M03/19/2008 05:14:16 PM
I truly feel for you, since I'm living in Lotus paradise according to your description:
Complete control over production servers on i5/OS and a few Win32 machines, wearing multiple hats as a developer, admin, support person and trainer. Then, it's a relatively small org with 300 users max.
Downside is that any patches, PTFs and server upgrades have to be done off business hours, because yanno, people can't live without Domino/Notes.
The other thing I'm tired off is getting support calls like "why does my mail not get through?", when it's more than obvious that they made a typo in the e-mail address. grrrr.....
Just saying that even paradise isn't what it used to be.
4. jimmy braco03/17/2008 08:49:17 PM
I understand the "logic" behind this kind of corporate separation...Heck, lord knows you do not want developers to have admin rights to your OS (or Dom servers for that matter) especially if their last name ends in "freeman" (LS '08 pun intended)...
The real problem is, why is there a problem in the first place?
any automatic OS update (should be illegal)
a service pack or new software installed on OS ? (should be a group effort)
The main problem with Org's that implement this depart-mental type hierarchy to a domino server is that they do not require the same hierarchy to implement each departments updates/changes.....and this INCLUDES our side of the house IMHO of course :)
5. David Jones03/17/2008 05:25:31 PM
I live almost in the "Lotusphereian paradise". I have full control over my servers - 4 being on Windows and the other 4 on i5 Series. I take care of the Windows boxes myself but I don't have a large working knowledge on the i5 boxes and we have two others here who have worked on the AS400s their entire career so I let them handle most of the upgrades on those.
6. Jim Casale03/17/2008 02:27:53 PM
I feel for you. Same thing here. Well almost the same thing. I have admin rights on server but they can install patches or mess with the SAN drives anytime they feel like it.