Windows PowerShell, Radioactive Sludge, and Documentation Failure
Summary: While trying to automate the creation and execution of a bunch of nearly identical command-line tasks, I attempted to learn Windows PowerShell. The code didn’t fail me, the massive quantity of unreadable and disorganized documentation did.
It’s a long story so go get a beer before continuing.
I’ve known about PowerShell for a few years now, back when it was known as the Monad shell and there were whispers of it being another great unix-killer (Windows will finally kill unix by having a shell more powerful than DOS 6.2!)
I’ve used Windows, DOS, and unix since the mid-80s and became very comfortable with unix around 1994. I try to avoid OS bigotry and I find that attitude counterproductive especially when I’m working in a heterogenous environment. As a sysadmin, that’s always. That doesn’t mean I don’t have preferences or opinions (if I cannot turn off and uninstall the GUI of your OS, your OS is absolutely not suitable as a server OS. Full stop. My headless rack-mounted remotely-managed server should not require a media player be installed. Full stop.) But I don’t always get my way and I have to play the hand I’m dealt. This leads to another conclusion: the applications I do use should run on any platform I’m likely to use or should have a platform-specific equivalent.
I’ve automated tasks in a lot of languages - DOS batch, VMS, shell (zsh, Bourne, bash, ksh), perl (win32 & unix), and ruby - so there are certain features that I come to expect from task automation languages. Aside from the basic test/branch/loop functions, various types of variables (boolean, integer, floating-point, string, scalar, list, associative array, object), and standard arithmetic and logical operators, there must be system functions for controlling process execution (exec, fork, wait, kill), system monitoring (ps, uptime, lsof, fuser, netstat, df), file and directory handling (find, locate, which, cd, mkdir, touch, rm, rmdir), I/O (pipes, cat, tee, xargs, read, write, overwrite, append, seek), and text-processing functions (awk, sed, grep, split, join, cut, fmt.) I don’t necessarily need all of these built into the core of the language or shell, but I use a lot of those functions on a regular basis.
When I was a full-time sysadmin I looked into PowerShell a little, primarily because Windows automation was lacking. My main turn-off at the time was that it was very verbose and seemed like a really thin shim over .NET. Any time invested learning it was not transferable to any other OS and required too much in-depth knowledge of a programming framework that I would never otherwise use or need. Most of the unix utilities are so small and focused that they illustrate a single task or role very well and tend to be portable across operating systems; if not portable, they are influential. This is the legacy of people like Donald Knuth and the wizards of Bell Labs - it was less about unix in particular, rather it was about good computer science in general (with the possible exception of csh.) So spending a lot of time and effort learning a verbose syntax and .NET so I could get access to the innards of WMI, Excel, and Word sank it for me. It was just easier to install cygwin or perl on the Windows boxes and write my management scripts in cross-platform Perl (hint: File::Spec and binmode are your friends.) If I’m going to spend my limited non-billable learning time on a new technology, it really should benefit me on as many platforms as possible or solve a particularly painful platform-specific problem. The unix toolset is very hard to beat because while it’s fragmented and takes a long time to learn, the tools are mature, interoperable, and each solve a specific problem very well. As the perl aphorism goes, “Easy things are simple and difficult things are possible.”
A quick warning: You do have to move out of your comfort zone to try new things once in a while. In this case, PowerShell has received some praise, none of my coworkers have anywhere near my unix background so finding new technology that works in our environment is a win, even if the knowledge gained is of limited utility elsewhere. There’s a reason I didn’t just punt to perl again.
From theory to practice - I left system administration a little over a year ago to return to the risk assessment and engineering analysis field. Kiss goodbye pagers, automated build and configuration management, monitoring and automated process recovery, backups and restores, and all the fiddly tasks like rotating log files, managing ssh keys and access control. I’m a ‘normal’ 9-to-5 user again with my spreadsheet, wordprocessor, mail client, web browser, MathCAD, and Fortran compiler.
Fortran compiler? FORTRAN?!
No. Fortran. ‘FORTRAN’ refers to FORTRAN77 and earlier, back when your keyboard may not have had a caps-lock key. ‘Fortran’ refers to Fortran 90 and later, which despite tracing it’s roots to the Eisenhower Administration (1956), is still the de facto language for numerical programming i.e. number crunching. It’s what computers were built for. Vannevar Bush’s analog computers computed tidal tables, ENIAC was funded by the Army to develop artillery ballistics tables, and proto-computers such as Bush’s rapid selector and Turing’s ‘bombe’ and ‘spider’ machines were used for cryptanalysis. Fortran is a dragster, a single-purpose language. It goes very fast in one direction but doesn’t handle cornering too well.
At times I have to run a lot of Fortran jobs to analyze the behavior of radioactive sludge leftover from five decades of plutonium production at the Hanford nuclear reservation. The faster I can make small changes to base input cases (text files), create directories and rename files, and run jobs, the faster I’ll converge on an accurate answer. The faster I deliver accurate answers, the faster some nice engineers in Washington state can move horrible radioactive glop out of wherever it is now into a nice shiny can which can be eventually sent to New Mexico for dewatering and permanent disposal (Harry Reid scotched Yucca Mountain so some of the metal-rich sludge is going to be parked at Hanford until a replacement repository comes on line hell freezes over.) The faster that gets done, the faster they can tear down the remaining rotting Cold War-era structures at Hanford and complete environmental remediation. In short, this nation fucked up the environment of the plateau north of Richland and it is our responsibility to clean it up the best we can. I take a weird sort of moral pride in my work. Rather than whining about the state of things, I’m literally helping clean it up. Yes, I’m getting paid but so are all those PIRG and Greenpeace canvassers. I have a tangible legacy of success, all without ramming a whaling ship or chaining myself to a tree.
Enough politics. Our goal is to help the Hanford staff design waste containers, determine how much waste they can store in them, how they can load the containers with waste so they stay cool and safe, how they can transport the waste safely, and how they can store it safely until it reaches its final destination, where and whenever the hell that is. The two big issues are managing the heat and hydrogen gas production. Not that anyone cares, but uranium metal has a bunch of really interesting properties. The one we’re most concerned with is its tendency to react with water to produce heat and hydrogen gas. For those of you who remember your high school chemistry:
U + 2 H2O -> UO2 + 2 H2 + heat
Oh, and the hotter the uranium gets, the faster it reacts with water and the more heat it gives off, making the uranium hotter, and … welcome to the (literally) explosive world of positive feedback and runaway chemical reactions! You don’t run the risk of a mushroom cloud, but you still don’t want a tank of radioactive sludge exploding just because the transport trailer got a flat and sat out in the sun for a week while the DOE fought with the state of Nevada over an air quality permit to change the fucking tire.
Ahem.
So we have to analyze a bunch of very similar cases - is moving 1 ton of sludge ok? What about 2 tons? The metal in the sludge tends to settle out before all the floaty bits leaving a layer of metal-rich sludge on the bottom of the container. What if we loaded half the sludge at a time so each metal layer was half as thick and one was midway up the container? What if someone threw a tarp over the container to cut down the sun by half? Once we move it into a building with a bunch of other containers, what if the building’s vent fans stop working? Some of the rooms we have in this building are bigger than others - does that make a difference? There’s a beautiful interplay between the designers proposing solutions, the risk people looking for ways things can go wrong, and the analysts answering all the what-ifs. Those what-ifs mean a lot of redundant work for me and my sysadmin background demands that redundancy be automated. This is where a good task automation language becomes vital.
So after many years (and for the Dear Reader, many paragraphs) I’m looking at Windows PowerShell again. It has some really nice features compared to DOS (and yes, that’s like saying mammals have some really nice features compared to phytoplankton) and the object model may prove useful, but the problem I’ve run into is that it’s almost impossible to get started with the language.
Yes, there’s a huge Microsoft website devoted to promoting Windows scripting. The problem is there’s nothing that I can find resembling a language reference manual for Windows PowerShell. For example, I want to know how variables work. Is PowerShell strongly-typed (char, string, int, long, float, double) or weakly-typed (variant)? Are common data structures available (scalar, list, associative array) and are they first-class elements? What are the intrinsic keywords? Is there a library or module system? If so, what is in the standard library? Are there third-party libraries I can use if I don’t find what I need? Is this a procedural, functional, or object-oriented language - or to what extent each? What are the standard operators and what is the operator precedence? What are the coding conventions? Formatting expectations - line-oriented or free-form? Is whitespace or case significant? Is there a character set limitation - is the language ’8-bit clean’? Does it support UTF-8, Unicode, etc.?
What I found was a huge, disorganized, incomprehensible mass of web pages and webinars and other media wrapped in a complex HTML frame that was absolutely useless for finding the answers I wanted. This is not a criticism of PowerShell - I can’t even get to that point - because there is no way for me to compare this language to the dozen or so I already know. There is, however, a portion of the site called “Sesame Script” for beginners.
Look kitten, I was four years old when Sesame Street came on the air. I am an order of magnitude older and I’d prefer to be treated as an adult, capisce? Thanks.
I do not need to learn how to change the text color to cyan. That is not helping.
I don’t want to diss on all the hard work put into Microsoft’s site, but it’s really aimed at the converted. I care not two shits about .NET. My more pressing problems are getting our developers to use a version control system better than a backup tape, some file folders, and a nuclear bureaucracy QA program. Maybe getting all of us using the same compiler. Personally, I want it to work when compiled with gfortran so I can develop under Eclipse because gfortran/Photran/Eclipse runs on all the platforms we’re likely to support (Visual Studio FAIL.)
The point is, I don’t come from a Microsoft shop and I don’t develop Windows applications. I’m automating tasks with the tools already provided by the OS so expecting me to already know all the .NET/C#/VBScript/whatever trivia like variable typing, operator precedence, &c. is a documentation failure. Making this information difficult to find or treating it as irrelevant or ‘unsexy’ is a documentation failure. Making it only accessible via dynamic web pages or video or interactive media is a documentation failure.
Look, I’m a yam-shaped middle-aged engineer pressed into service as an operator and sometime Fortran developer. My expertise is in crazy shit like Wigner energy release from irradiated graphite, fire analysis, radiological analysis, dose, and nuclide transport, probabilistic risk assessment, human reliability analysis, thermal hydraulics, industrial ventilation, relay logic, circuit analysis, and metal-water chemical reactions. I read books. I’m not so bad that I print out my email or think Twitter is something my aunt’s dead budgie no longer does. But I do expect clear boring printable documentation for languages I use. PowerShell is not my first language (that award goes to VIC-20 BASIC, ironically courtesy of Microsoft - a company I still respect for making tools for small computers when nobody else would) nor will it be my last. Suffice to say, I need a terse summary of the language and some basic examples, preferably something in PDF format that I can print and read while eating lunch, riding in a train or plane, or otherwise when I’m not next to a computer. Because when I’m next to a computer, I’m using it.
I know, I may be a special case here, but that’s how I work - high-bandwidth webinars and dynamic frame-in-a-frame-in-a-frame websites are utterly unreadable to me. I know, I can always drop $50 each for the books from Microsoft Press. Spending $50 out of pocket to wade through 400 pages of what I know will be useless to me at this point in the evaluation and learning process is a non-starter. Compare this to perldoc: the Perl distribution ships with a ton of high-quality documentation. ‘High-quality’ does not mean shiny expensive websites and webinars - it means plain text files which explain the language. See also Python, Ruby, Haskell, and a crap-ton of other languages.
I get the sense that PowerShell is being sold, not being used. Compare the ease of finding terse, well-structured documentation here compared to here. Hell, compare the ease in finding the installation package for each. Hint: There is no installer specifically and onlyfor Windows PowerShell.
To revisit my earlier comment about Perl. The Windows scripting website makes important things impossible and trivial things pervasive.
Here’s a concrete example: using something like bash or zsh or perl (overkill), I could automate some jobs with a one-liner like:
“foreach dn (CONT* SETT*) { cd $dn && run_cmd.sh }”
This goes into each directory matching CONT* or SETT* and runs the command run_cmd.sh provided the cd command succeeded. This is poor-man’s batch processing. This should be a no-brainer for PowerShell but given that I can’t find a basic summary of variable types, operators, etc. on the website, I’m SOL.
Yes, I found the 44-page PDF from Microsoft - Switzerland. That helped a lot, but it still didn’t answer my main questions or point me toward a better resource. get-Help churns out several screenfuls of information which is too disorganized and overwhelming to impart any meaningful information. “get-Help about_assignment_operator” starts getting closer with some explanation of types, but really, at this point I’m on the computer trying to use it, not sitting peacefully with a cup of coffee reading from a piece of paper and understanding the structure and intent of the language.
Going a bit further: ‘Get-ChildItem’ resembles ‘dir’ or ‘ls’ or ‘find’ which is useful. Sadly, it’s not clear if or how to specify multiple -Filter arguments; from above, I want a list of all directories matching both globs CONT* and SETT*. Do I need multiple Get-ChildItem statements? I can put the results of each into a variable, sure - but can I catenate the lists together so I have all the matching directory names in the same variable? Remember, I still don’t know what variable types are available or if/how PowerShell handles lists. I know unix has a quirk that if you cd into a directory as in the script above, at the end of the cd command, the working directory will revert to the current working directory - that’s why there’s a logical-AND between the cd command and the run_cmd.sh statement. No idea if PowerShell has this same bug/limitation/feature/insight and no idea how to find out other than by experimentation.
Again, I don’t expect Windows to be unix and I don’t expect PowerShell to be zsh or perl. It is what it is. The problem is that the documentation is organized like a FedEx cargo plane crashed into the Macy’s Thanksgiving Day Parade. The information I’m looking for is probably in there, but I’ll never find it under the burning inflatable Barney and mangled corpses of the Jonas Brothers.
To bastardize a Clint Eastwood quote from Heartbreak Ridge: “It’s a goddamn VAXClusterfuck.”
post/read comments