https://blog.wannemacher.us/categories/system-administration/ Recent content in System Administration on Forward or Reverse Hugo -- gohugo.io en-us Thu, 26 Mar 2015 13:05:54 +0000 https://blog.wannemacher.us/posts/p558/ Thu, 26 Mar 2015 13:05:54 +0000 https://blog.wannemacher.us/posts/p558/ For me, one of the easier ways to begin to learn something is to learn by example. Even if the example does not solve my exact problem, I can use it to figure out ways of extracting the information I need and patterns for implementing new functionality. As a response to a forum question regarding provisioning approvals, Kevin Morey pointed out a Github repository that I was unaware of (https://github.com/ramrexx/CloudFormsPOC). This repo is a great source of example code to get you started exploring more advanced usage of CloudForms. https://blog.wannemacher.us/posts/p540/ Fri, 21 Nov 2014 19:36:10 +0000 https://blog.wannemacher.us/posts/p540/ In our environment we want to use Foreman to build our OCP hardware. It was working well except that the installation would repeatedly PXE boot to reinstall because Foreman would not know that the PXE portion was complete. The standard notification code below was failing for some reason. 1 2 %post --interpreter=busybox --ignorefailure=true wget -O /dev/null <%= foreman_url %> After running the unattended install in a Fusion VM so I could have console access I reviewed the install log and a misleading message from wget about the URL being bad. https://blog.wannemacher.us/posts/p516/ Fri, 31 Jan 2014 01:59:01 +0000 https://blog.wannemacher.us/posts/p516/ I have spent a day or so trying to get the OpenZIS project up and running on a CentOS machine. A lack of updated instructions and an older code base made this process harder than I expected. While I am still investigating if OpenZIS will meet my needs, I wanted to publish what I did as a reference to others who might try to install it. The doc is in my forked version of OpenZIS. https://blog.wannemacher.us/posts/p482/ Thu, 09 Feb 2012 00:36:54 +0000 https://blog.wannemacher.us/posts/p482/ I often see/hear comments from people whose common sense approach to availability design includes eliminating a single point of failure. This is a great goal when required, but care must be take that the “common sense” approach is actually achieving what is required. I have found that using a fault tree approach to evaluating design decisions can be insightful. The last instance of this is a book, while otherwise great, gave the advice of separating a database from the application function onto separate servers because there was a single point of failure. https://blog.wannemacher.us/posts/p423/ Wed, 13 Apr 2011 21:13:00 +0000 https://blog.wannemacher.us/posts/p423/ Launch the command prompt via: Start | Programs | ADAM | ADAM Tools Command Prompt dsdiag.exe /s:localhost:389 repadmin.exe /viewlist * /homeserver:localhost:389 repadmin.exe /replsummary * /homeserver:localhost:389 repadmin.exe * /repsto /homeserver:localhost:389 repadmin.exe /showconn * /homeserver:localhost:389 https://blog.wannemacher.us/posts/p419/ Tue, 12 Apr 2011 16:57:03 +0000 https://blog.wannemacher.us/posts/p419/ I am testing the impact and behavior of memory limits and reservations along with balloon drivers and I needed a quick way to allocate memory in a user program. PowerShell to the rescue. This isn’t a good method for exact memory allocation, but you can consume MBs to GBs of memory pretty quickly. 1 2 3 4 5 # Allocate memory by creating a large string. Divide the length by 4 # (Unicode size) to get an approximation of the MB allocation. https://blog.wannemacher.us/posts/p383/ Thu, 28 Oct 2010 18:47:15 +0000 https://blog.wannemacher.us/posts/p383/ Here is a quick and dirty PowerShell script to parse the output of the rpowermt command used to manage PowerPath/VE on ESX (for EMC arrays). I wrote the code quickly to solve a particular problem, but if I get requests I could extend it to parse the rest of the output and do so in a little more robust fashion. Why do this? I am in the middle of migrations between frames and I needed an easy way to determine what needed moved when the storage admin said move all LUNs from CLARiiON XXXX to the new LUNs on YYYY – and make sure to move LUN 41 to LUN 88. https://blog.wannemacher.us/posts/p375/ Fri, 22 Oct 2010 18:43:58 +0000 https://blog.wannemacher.us/posts/p375/ One of the worst case scenarios when securing an ESXi host is disabling the DCUI, enabling lockdown mode, and then losing vCenter for some reason. If your vCenter database is corrupt then you have lost the ability to manage the host. The official answer at this point is to rebuild the host. While I hope you have an automated build process that would make this easy, there is at least one other option to recover your system. https://blog.wannemacher.us/posts/p365/ Mon, 09 Aug 2010 15:24:45 +0000 https://blog.wannemacher.us/posts/p365/ I have 5 difference vCenter servers installed over various timeframes and I am seeing different counts for the Non-Critical Host Updates across them. After some digging it appears that this is due to the Nexus 1000v updates. We have not deployed the Nexus to production yet so this was a little confusing for me. It appears that when Update Manager is installed it downloaded the patches for the Nexus 1000v, but they are not included in the updates that are downloaded because we have none running in the environment and Update Manager was not configured to download them. https://blog.wannemacher.us/posts/p356/ Fri, 06 Aug 2010 19:24:22 +0000 https://blog.wannemacher.us/posts/p356/ I am looking to do PXE booting of the ESXi installer as the first step in rolling out my new automated deployment solution. We already have an Altiris (HP RDP) PXE server in house for Windows deployments so I needed to figure out a way to make it work. Below is the first phase of this using standard pxelinux and not gPXE. Download the ESXi 4.1 installable release from VMware Download the current 3. https://blog.wannemacher.us/posts/p327/ Tue, 29 Jun 2010 17:52:11 +0000 https://blog.wannemacher.us/posts/p327/ As a basis for an upcoming post on splitting vmkernel traffic across over layer 3 boundaries I wanted to describe how vmkernel traffic is routed on an ESX host. There seems to be a lot of confusion in this area and hopefully this will help to clear it up. If you need a refresher on IP addresses, network masks, or subnets check out this Cisco article. Directly Connected Networks If a host is directly connected to a subnet it will use that interface to talk to devices in that subnet. https://blog.wannemacher.us/posts/p304/ Sun, 30 May 2010 11:00:47 +0000 https://blog.wannemacher.us/posts/p304/ Here is an script to make life a little easier when doing bulk updates to a password for an account as long as the account has a common password on the different hosts. The script has some documentation in it so make sure to use Get-Help to find more detail. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Get-Help . https://blog.wannemacher.us/posts/p295/ Fri, 30 Apr 2010 00:33:51 +0000 https://blog.wannemacher.us/posts/p295/ Update Manager 4.0 Update 1 was refusing to install either of these two patches. At first I was confused because I did not have Nexus deployed, but this was looking similar to the problem that Update 1 was supposed to fix. I downloaded the bundle from VMware’s web site and tried to install the patch manually and got this error: The following problems were encountered trying to resolve dependencies: cross_emulex-cim-provider_400.2.0.27.1-164009 provides 'emulex-cim-provider >= 400. https://blog.wannemacher.us/posts/p270/ Mon, 08 Feb 2010 21:32:51 +0000 https://blog.wannemacher.us/posts/p270/ Here is a short convenience script that will simulate a persistent login to a VMware host system when using the vCLI on Windows. Typically you have to specify a lot of parameters that include login information or a session file. With this method you just run the script and provide the hostname, username, and password for the connection. After running this script you can run commands like “vicfg-mpath.pl –list” without additional parameters. https://blog.wannemacher.us/posts/p242/ Mon, 30 Nov 2009 20:33:47 +0000 https://blog.wannemacher.us/posts/p242/ I am evaluating VMware Site Recovery Manager (SRM) and there are some things that I need to do in testing that I can not do in the base product. I am going to run some PowerShell to reconfigure the VMs and rename them. One of the first roadblocks is finding out which VMs are part of a particular recovery plan. Since the SRM API is limited I went into the database and dug around. https://blog.wannemacher.us/posts/p235/ Mon, 24 Aug 2009 20:55:05 +0000 https://blog.wannemacher.us/posts/p235/ There appears to be a discrepancy in the VMware documentation regarding the maximum number of vSwitches. The Configuration Maximums document states that the limit is 248. The configuration guide lists the maximum as 127 which is what it was in 3.5. I am not sure if I misunderstand what they mean by “Standard switches per host 248”. If you see the error of my ways let me know. Here is some code that I used to see how many vSwitches I could put on a host. https://blog.wannemacher.us/posts/p222/ Mon, 27 Jul 2009 13:00:12 +0000 https://blog.wannemacher.us/posts/p222/ If you are anywhere around Ohio and interested in Linux you may want to check out Ohio LinuxFest 2009. It is being held September 25th & 26th in Columbus Ohio. This will be my first year attending, but I have heard good things about the conference and am looking forward to talking with some other Open Source enthusiasts in the area. https://blog.wannemacher.us/posts/p169/ Thu, 09 Apr 2009 15:01:12 +0000 https://blog.wannemacher.us/posts/p169/ I have added some code to my Scripts/Programs page for securely storing credentials to disk to be used by scripts at a later time. I explored the VI Credential cmdlets and I was not 100% happy with them so I decided to implement my own version that uses the Microsoft DPAPI for encryption. Make sure to look at the README as it covers usage and current limitations. I wrote these for use in some automated VMware monitoring I am doing, but there is no reason they have to be used with the VI Toolkit. https://blog.wannemacher.us/posts/p153/ Thu, 26 Mar 2009 12:00:58 +0000 https://blog.wannemacher.us/posts/p153/ We have our service consoles set up to disallow root logins and use sudo (with password) for access. Every once in a while this causes us some pain. How do you update file /etc/filex or run some command across a cluster or a bunch hosts when root privileges are needed? There are multiple ways to approach this type of issue, but one I have not seen much of in the VMware blogosphere is clusterssh. https://blog.wannemacher.us/posts/p141/ Mon, 23 Mar 2009 12:00:00 +0000 https://blog.wannemacher.us/posts/p141/ Applications often cache data and that is good because that generally makes them perform better for users. However, if you make assumptions that what you see is the absolute truth you can run into trouble. 1 2 3 4 5 6 7 8 9 Connect-ViServer myVirtualCenter001 # Let's check the status of the server to make sure I can go home. (Get-vmhost myVmHost001.local | Get-View).OverallStatus green #Looks good. Well, maybe not. https://blog.wannemacher.us/posts/p135/ Fri, 20 Mar 2009 12:00:17 +0000 https://blog.wannemacher.us/posts/p135/ Now that we have covered how to make advanced settings changes we need to validate that the settings have been applied. Assuming that a script or command worked is never as good as checking it. Below is a script to do just that. The script lists all statuses so if you only want to see issues you can pipe the output into the Where-Object cmdlet like so: <vmObject(s)> | .\Validate-VmAdvancedSettings.ps1 | Where-Object {$_. https://blog.wannemacher.us/posts/p130/ Thu, 19 Mar 2009 15:58:14 +0000 https://blog.wannemacher.us/posts/p130/ If you use a method like that in the Powershell script I posted the changes can be made to a running VM. This is great because that dialog box is disabled in Virtual Center when the vm is up. The settings do not take effect until the VM is power cycled (not just restarted in the OS). This is not that big of a deal on clusters with VMotion licensed because a VMotion starts a new instance of the VM on the destination host. https://blog.wannemacher.us/posts/p79/ Thu, 12 Mar 2009 12:00:57 +0000 https://blog.wannemacher.us/posts/p79/ This is a good script to easily report on privileges and can easily be extended to audit against a known good list of expected roles/privileges. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 # Must already be connected to a viserver. This works on the default vi server. $si = Get-View ServiceInstance $am = Get-View ($si.Content.AuthorizationManager) $roles = $am.RoleList foreach ($role in $roles) { foreach ($privilege in $role. https://blog.wannemacher.us/posts/p99/ Tue, 10 Mar 2009 15:50:12 +0000 https://blog.wannemacher.us/posts/p99/ Here is a script that takes template objects and updates their VMware tools. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 # Usage: <template object(s)> | .\Upgrade-ToolsTemplate.ps1 # Example: Get-Template template1 | .\Upgrade-ToolsTemplate.ps1 # Get-Datacenter corporate | Get-Template | . https://blog.wannemacher.us/posts/p25/ Thu, 08 Jan 2009 16:09:20 +0000 https://blog.wannemacher.us/posts/p25/ I had an issue where I ran a vendor provided install script and my kernel sound drivers were deleted. I recovered them just fine, but it had me wondering about how I could be sure that no other files were missing or corrupted. After some searching online I found a reference to someone checking a package using dpkg’s md5sums. This and some poking around gave me what I needed. cd / sudo /usr/bin/md5sum -c /var/lib/dpkg/info/*. https://blog.wannemacher.us/posts/p22/ Fri, 05 Sep 2008 20:10:12 +0000 https://blog.wannemacher.us/posts/p22/ This looks like it might be an interesting development – a system admin podcast. The first one covers some substantial topics and I look forward to more. http://cuddletech.com/blog/pivot/entry.php?id=964 https://blog.wannemacher.us/posts/p17/ Wed, 27 Feb 2008 14:39:45 +0000 https://blog.wannemacher.us/posts/p17/ I am working on the configuration management problem and have been toying with getting bcfg2 working on ESX 3.5. My goal is to have an isolated install that does not require modifying the existing system files or upgrading any packages. My approach to doing this is to create a local Python 2.5 install and install the bcfg2 packages there. This is just a start, but I thought I would put it out there for anyone who wants to work on something similar. https://blog.wannemacher.us/posts/p16/ Fri, 15 Feb 2008 19:53:14 +0000 https://blog.wannemacher.us/posts/p16/ I am not very happy with the landscape of configuration management software out there for VMware ESX server. I am investigating using projects such as Bcfg2 to administer my servers. I am working with Bcfg2 because I do not have a lot of experience with these types of software and I am interested in learning more about Python. Of course these types of packages fail when you look at ESX 3i, but I am not convinced that I will be an early adopter of that platform. https://blog.wannemacher.us/posts/p14/ Mon, 04 Feb 2008 12:52:44 +0000 https://blog.wannemacher.us/posts/p14/ I have been working on methods to benchmark the Windows infrastructure team at work and I have not been finding much luck finding existing metrics. I was not too happy with the last Gartner review of our IT department, but perhaps that is all that is out there. I have developed some thoughts on the matter and once I am fairly certain they make sense I will post them here. https://blog.wannemacher.us/posts/p11/ Fri, 11 Jan 2008 14:02:02 +0000 https://blog.wannemacher.us/posts/p11/ A common problem in infrastructure groups is determining the value that we provide to the organization. One common way of thinking about this is in terms of asset utilization. Messages such as “take your servers from 5% utilization to 60%” are very common in virtualization value propositions, but I do not think that capacity utilization is the correct measurement of infrastructure value. That value is derived from being able to run an application at adequate performance levels. https://blog.wannemacher.us/posts/p9/ Wed, 02 Jan 2008 14:46:11 +0000 https://blog.wannemacher.us/posts/p9/ It amazes me the amount of incorrect information that is put on user forums. There are always mistakes and I have seen some posts where people realize and correct their errors. Primarily though, I see people ranting about “the correct way” when that way is not based in product knowledge, but rather on “it seemed to work”. This reminds me of the time I went to the pet store to buy some activated charcoal after my dog ate some wild mushrooms. https://blog.wannemacher.us/posts/p7/ Wed, 26 Dec 2007 21:19:47 +0000 https://blog.wannemacher.us/posts/p7/ I needed a way to allow our admins responsible for patching to include VMware Tools updates during a patch cycle. You can easily update via Virtual Center, but this requires multiple interfaces and it can be difficult to pick all of the same servers without making a mistake. I fired up Process Explorer and watched what VMware did with their existing tools update process. This is pretty cool since it does not require a network connection and your Tools install is up to date without needing to maintain a repository. https://blog.wannemacher.us/posts/p6/ Thu, 20 Dec 2007 15:37:08 +0000 https://blog.wannemacher.us/posts/p6/ It is interesting to read about more formal approaches to system administrator education. I have been reading “Master Education Programmes in Network and System Administration” from the LISA 07 proceedings. The authors’ approach is an interesting one, but I wonder what we do for all of the administrators already “in the wild”? A training vs. education mentality and the difficulty of adult education make introduction of a program like this very difficult.