Using Powershell to concatenate files and strip off the headers

We had a request from a client to merge some files for an import process that we have.  As part of one of our applications, we can import delimited (comma, tab, etc)  ASCII files.   The client was generating the files from 5 different processes.  Each exported file has a single line header row, and the files were from OS X.  That meant that the files were Unicode encoded and ended each line with only a carriage return.

Our import process can only handle a single ASCII file with DOS style line endings (carriage return + line feed).  It’s a legacy system and it can’t be touched.  What we can do to concatenate the multiple files into a single file and strip out the header rows for every file after the first one.  And change the encoding to ASCII and use the CR+LF combination that DOS loves so much.

I’ve been doing a lot of Powershell lately, so it made sense with me to continue with that Swiss Army knife of scripting languages.  I ended up with the following script:

[CmdletBinding()]
param(
  [parameter(Mandatory=$TRUE,Position=0)]
    [String] $Pattern, # input pattern to match, including path. Example: c:\export\*.txt
  [parameter(Mandatory=$TRUE,Position=1)]
    [String] $Destination # Destination file to write
)

begin {
    $FileCount = 0

    # Get all of the files that match the wildcard pattern

    Get-ChildItem -path $Pattern | ForEach-Object {
        Write-Verbose "$FileCount $_.FullName"

        if ($FileCount -eq 0) {
            # For the file, just write it out as ASCII
            Get-Content $_.FullName | out-file -FilePath $Destination -Encoding ascii
        }
        else {
            # For each additional file, skip the header line and append as ASCII to the destination
            Get-Content $_.FullName | select -skip 1 | out-file -FilePath $Destination -Append -Encoding ascii
        }

        $FileCount++
    }

    Write-Verbose "$FileCount file(s) processed"
}

The first five lines take care of the parameter binding. This script only needed two parameters. $Pattern is the path to the input files, including wildcards. $Destination is the full path to the file to concatenate the source files into. The optional “-verbose” parameter is supporting for displaying some text as the files are being processed. This is a standard Powershell parameter, that’s why it’s not included in the parameter list for this command.

    Get-ChildItem -path $Pattern | ForEach-Object {

We are using Get-ChildItem to get all of the files that match $Pattern and then call ForEach-Object to iterate through that list.

        if ($FileCount -eq 0) {
            # For the file, just write it out as ASCII
            Get-Content $_.FullName | out-file -FilePath $Destination -Encoding ascii
        }

For the first file, we just need to read it in and output it back out to the destination files. Powershell will handle the line endings and we will force it to ASCII. The $_ variable is a placeholder variable. When you are using a Foreach-Object loop, the $_ represents the current item of the loop. When you get use Get-ChildItem for a file or folder, .FileName is a property of the item or array of items being returned.

        else {
            # For each additional file, skip the header line and append as ASCII to the destination
            Get-Content $_.FullName | select -skip 1 | out-file -FilePath $Destination -Append -Encoding ascii
        }

For each additional file, we read the file in, select every line after the first line, and finally append it as ASCII to the destination file.

The “Write-Verbose” lines will only execute when the -verbose parameter is used. So if that script had been saved as c:\scripts\convert-files.psi, you would execute it like this:

c:\scripts\convert-files.ps1 -pattern c:\export\*.csv -destination c:\import\all.csv

c:\scripts\convert-files.ps1 -pattern c:\export\*.csv -destination c:\import\all.csv -verbose

That would concatenate all files that end with “.csv” in the c:\export folder and write a single file named c:\import\all.csv. If you add the “-verbose” parameter, you will see the names of the files as they are being processed.

Comcast did WHAT in Houston? You are now a public hotspot.

Yesterday, Comcast flicked the switch that turns the home routers for 50,000 of their Houston subscribers into public Wi-Fi hotspots. It’s been covered in a few newspaper articles, I came across it on this SeatlePI article and this followup by Dwight Silverman.  Dwight writes for the Houston Chronicle, but I saw the syndicated copy on the SeattePI site.

If you live in Houston and have a XFinity Router, then you may already be a public Wi-Fi hotspot.  The router serves up the public Wi-Fi access on a separate wireless network from your home network.  But it’s using your bandwidth and the possibly the same IP address.

If you do a search of Wi-Fi networks and you see one or more named “xfinitywifi”, then your or your neighbor is now a public hotspot.  This is an opt-out policy.  Unless you request not to be a public hotspot, Comcast is going to enable it on your router.  Comcast had sent out letters to their customers, but how many of them read those letters?

The idea sounds good on the surface.  If you are a Comcast customer, you will have Wi-Fi access whenever you are close to another Comcast customer.  You would be able to connect your tablet or laptop in more places.

One problem with this is that you are sharing the bandwidth available to that home subscriber.  Comcast is making more bandwidth available to the home, but they can’t create more Wi-Fi bandwidth.  You can only carry so much network traffic over Wi-Fi.  If you live in busy neighborhood or close to a shopping area, other Comcast customers connected to your router could use enough bandwidth to make it difficult for you to connect to your own router.  Comcast can alleviate that with Quality Of Service (QOS) rules in the router, but that wouldn’t eliminate the problem.

Then you have the security issues.  Unless Comcast has modified the routers to get two outside IP addresses, anyone who connects to the public hotspot from your router will provide the same IP address to the rest of the world that you provide  This is the external IP address that shows up when you are connected to a website, not the internal IP address that your device thinks it has.    That means if someone was using BitTorrent to download and share copyrighted material like music or movies, it would be your IP address that would be tracked.

When Comcast gets the subpoena from the MPAA asking for the identity of the person using your IP address because that IP was sharing copies of “Frozen”, will Comcast be able to determine that the offending activity was done over the public hotspot as opposed to your internal network?  If they have the same IP address, then you are going to be facing the legal battle to prove your innocence.

If the public Wi-fi can be traced separated from your use of the same router, then Comcast has just made it easy for people to run Bittorrent sharing sites or perform other nefarious activity anonymously.  Just connect a Hotspot named “xfinitywifi”.  It can from your router or your neighbor’s.  It doesn’t matter, it’s not traceable back to you.  Comcast can block ports used by Bittorrent clients, but it will be hard for them to block everything.

You can opt out after the fact.  SeattlePI’s Dwight Silverman posted the following instructions here.

  • Log into your Comcast account page at customer.comcast.com.
  •  Click on Users & Preferences.  
  • Look for a heading on the page for “Service Address.” Below your address, click the link that reads “Manage Xfinity WiFi.”  
  • Click the button for “Disable Xfinity Wifi Home Hotspot.”  
  • Click Save.

With the Comcast/Time Warner Cable merger currently in FCC review, this could affect more than Comcast customers,  If you are a Time Warner Cable customer, it’s possible that after the merger that public Wi-Fi program could be rolled out to TWC customers.  This is just one of those ideas that sounds good on people, but make me uneasy when they are actually implemented.

Been reading “iOS Development with Xamarin Cookbook”

iOS Development with Xamarin Cookbook
iOS Development with Xamarin Cookbook

A new Xamarin iOS book has just come out and if you’re programming for the iPhone on the Xamarin platform you will want it on your bookshelf.  It’s titled “iOS Development with Xamarin Cookbook” and is written by Xamarin MVP Dimitris Tavlikos.

This book will take you from the first steps in writing and compiling an iOS application with Xamarin to working with advanced topics like writing multi-touch gestures and working with iCloud.  The intended audience for this book are .NET developers new to iOS development and Objective-C developers who want to be able to use the Xamarin tools.

This is not a cross-platform guide.  While a few topics have corresponding features with Xamarin.Android, this is an iOS only book.  For example, chapter 12, “Multitasking”, covers the iOS specific APIs for running code in the background and how to respond to events raised by the background code.

The advantage of writing a guide for one platform is that Dimitris can go deep on specific features and APIs.  Each chapter has a set of “recipes” that explain how to do a task.  Each recipe is independent from the others.  You don’t have to build the first four recipes in a chapter in order to try out the code for the fifth.

Each recipe follows a pattern.

  1. Recipe Name: First Dimitris describes what the recipe will do
  2. Getting Ready:  What has to be done before the recipe code is written.
  3. How to do it:  A series of steps that describe the code to write
  4. How it works: An in-depth description of what that code does
  5. There’s more: Addition information about this feature
  6. See Also: A list of other recipes that are related to this one

For me, the book was worth it just for chapter 14, Deploying.  I don’t have to submit apps to App Store that often, it’s useful to have all of the steps required, all in one spot.

This one is for your grandparents (aka “I’m calling you from Windows and you a have virus”

Only you can save Grampy and Oma
Only you can save Grampy and Oma

OK people, this one is for your parents and your grandparents.  There is an annoying scam that has been going around where they get called from someone claiming to be from Windows and the weasels have detected a virus on their PC.  And then the weasels extort hundreds of dollars from the nice people who gave you money for every report card.  And you can protect your people. You are their only hope.

The weasels employ some cheap tricks to convince you that your machine has a virus and then get you to install a remote access app to let them hijack your PC.  With control over your PC, they can install pretty much anything they want.  Then you find out that you need to pay them lots of money to get control of your PC back.  Or they’ll lock it up so you can’t use it all.

This is what you need tell Grandma.  When the weasel calls from Windows, just hang up.  Don’t say anything once you hear them say they are from Windows or are from “Tech Support”, just hang up. They may call right back, or perhaps the next day.  Just hang up.

This scam has been going around for a few years and it’s classic social engineering.  They will walk you through launching the Windows event viewer and have you scroll through the list until you see an error message.  You’ll always see an error message, something is always complaining about something under the hood.  It’s more or less business as usual in Windows.

There's always an error, you can ignore most of them
There’s always an error, you can ignore most of them

But the weasel is going to tell you that it’s a virus and they will fix it for you.  he tell you that he needs to access your PC to verify what the problem is. He’ll have you run “a diagnostic tool”, that’s how the weasels install their malware.  Once their software is installed, they basically own the machine.  If you don’t pay up, that machine is basically under their control.

This scam has been going around for a few years.  Why?  Because it works.  Enough people are fooled to where this can make some money to people who are clearly deficient in the moral compass department.

This is on you to be the first line of defense for your family and friends.  If Mom and Dad (or Grandma and Pop-pop) have a PC and are not IT skilled, then make sure their important stuff is backed up somehow.

Here’s  a short list that easy to do:

  1. Have a copy of all of their email accounts and passwords.  If their machine gets compromised, you should be able to go online and change all of their passwords for them.
  2. Ditto for their Facebook and other social media accounts.
  3. Make sure that if they do any online banking, they use unique passwords for each account.  And have them write that stuff down.  They may not want you to have the keys to your future inheritance, but you need to be able to help them lock down the access to those account.
  4. Write down the product keys for anything that needs to be reinstalled.  That includes Windows.  There are free tools like Belarc Advisor or Magic Jelly Bean that will look this information for you.  If you need to reinstall Windows, this will savea  lot of time.
  5. The sky is the limit for backups, at the very least get portable hard drive and just backup their important files on a regular basis.  I would use two drives and swap them out.  Go on the assumption that the drives will fail after year.  It’s OK, Staples will have them on sale again.   In addition to protecting their files from someone locking up their PC, you get the backup protection in case the hard drive goes on the PC.

The first four steps, you just need to once and update when their accounts are changed.  The backup you just have to get them in the habit of doing.  Or you just do it for them.

Just remember the cardinal rule: hang up the phone.  If Grandpa doesn’t want to hang up in the case that it’s valid call, then let the call go to the answering machine.  If it’s a real call, they’ll leave a message and a call back number.  A weasel is not going to leave a number where you can call them back.

A great suggestion came in after I originally posted this.  When the weasels call, tell them that you work for Microsoft.  They’ll hang up and not call you again.  Different weasels will call, but not ones that called you before.

Photo from SalFalko’s photostream on Flickr, used under CC license.

Using the Windows Phone 8.1 Start Background to personalize your phone

A great way to personalize your Windows Phone 8.1 device is a custom start background.  You are no longer limited to using the 20 or so accent colors as the default tile color.  You can now select an image file to use as the background.

When you select an image, it becomes a fixed background.  When you scroll the start page, the location of the image stays fixed,   Most of the bundled apps will use a transparent background when a background image has been select, allowing the background image to show through.

This allows you to personalize your phone even more to your tastes.  You can use any image that is on your phone or accessible through OneDrive.  I have found that using darker, muted colors worked well, providing a good contrast with the foreground tile color.

You can select the image under “start+theme”, in the settings app. You should see something like this:

start+theme, under settings

When you click choose photo, you get the standard image picker and you can select an image that is on your phone or in your OneDrive account.

The image that I used here is from the DeviantART web site.  I did a search on “grunge” and selected a couple of images that I liked.  This image is named Hazard Grunge and is from the user named WheelOfFish.

After selecting the picture, you will then select which area of the picture to use as the background.  Since the image will mostly likely not match the aspect ratio of your phone, you select which part to use.

Clip the image

With the final result, you get a much cooler looking start page.  And it looks more like your phone than just another Windows Phone.

My Phone

Resolving the “The ‘CctSharedPackage’ package did not load correctly.” error message

"'a' stands for headache" by Eduardo Ortiz

After launching an Azure project in Visual Studio 2013, I was handed a “The ‘CctSharedPackage’ package did not load correctly.” error message.

The 'CctSharedPackage' package did not load correctly.

I was unfamiliar with that one, and a quick check of the forums showed that a few people had reported this happening after installing the latest Windows 8.1 update. An update that I had installed two nights ago.

If you check the Visual Studio activity log in %APPDATA%\Microsoft\VisualStudio\12.0\ActivityLog.xml, you’ll see something like this

<entry>
  <record>1638</record>
  <time>2014/04/14 02:36:02.772</time>
  <type>Information</type>
  <source>VisualStudio</source>
  <description>Begin package load [CctSharedPackage]  </description>
  <guid>{77A5A151-6A9B-4D08-BC38-340AB29566E2}</guid>
</entry>
<entry>
  <record>1639</record>
  <time>2014/04/14 02:36:02.773</time>
  <type>Error</type>
  <source>VisualStudio</source>
  <description>SetSite failed for package [CctSharedPackage]  </description>
  <guid>{77A5A151-6A9B-4D08-BC38-340AB29566E2}</guid>
  <hr>80070002</hr>
  <errorinfo>Could not load file or assembly 'Microsoft.VisualStudio.WindowsAzure.Diagnostics, Version=2.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.</errorinfo>
</entry>

Something in the Windows 8.1 April Update blew out at least one assembly that Visual Studio needed. This has an easy work around. Shut down any instances of Visual Studio that you may have running. Go to “Programs and Features” in the “Control Panel” and look for an item that starts with “Windows Azure Tools for Visual Studio 2013″.

Double-click that item to bring up the modify setup dialog. On that dialog select “Repair”.

Azure Tools Modify Setup dialog

That will churn for a minute or two. After that completes, you should be back in business. If you are using the Azure Tools with other versions of Visual Studio, you may may to repair them as well.

An alternative means of fixing this is by re-installing the missing assembly into the GAC as described on Stefan Camilleri’s blog.

<code>C:\Program Files (x86)\Windows Azure Tools\Visual Studio 11.0&gt;gacutil /i .\Microsoft.VisualStudio.WindowsAzure.Diagnostics.dll</code>

I chose the self repair option of the Azure Tools installer, because I had no idea if anything else was broken.  That’s the why Windows Installer technology has the repair option.  It will fix the stuff under the hood that you can’t easily see.

Once I got that straightened out, I checked to see if there was a later version. I was at v2.2 and v2.3 was released during //Build. So I grabbed that and installed. I probably could have just upgraded straight from 2.2 to 2.3 without repairing the v2.3, but I did this way with out any problems.

*The “a” is for headache image is from Eduardo Ortiz’s collection on Flickr

Best single source for dealing with iOS battery drain

If you or anyone you care about has been dealing excessive battery drain on the iPhone, a former Apple Genius Bar staffer has written the single best resource.  Scotty Loveless posted The Ultimate Guide to Solving iOS Battery Drain on his blog.  Bookmark it, print it, Instapaper it, just keep a copy of it somewhere.

His article was aimed towards iPhone users, a few of the tips are applicable to all smart phones.

Bundling jQuery 2.x and jQuery 1.x in ASP .Net MVC and staying NuGet fresh

Fork in the road
If you use Nuget to install jQuery into your application, you’ll need to take some extra steps to support multiple versions of jQuery.  By default, NuGet will uninstall previous versions of a package.  The way around this is to create another folder inside the scripts folder and put the older jQuery-1.. files in it. I got the idea for doing this from a post on StackOverflow

Right click on the Scripts folder and select Add->New Folder.  Give the folder a name that has some meaning to you, I used “compatibility”.  Select the jquery-1.* files and move them to the new folder.  You can copy them or move them, it doesn’t really matter.

Then install the latest version of jQuery using NuGet.  From the Tools menu, select Library Package Manager->Package Manager Console.  From the package manager window run the following:
Install-Package jQuery

You should see something like this:

Package Manager Console
Package Manager Console

It installed jQuery 2.1 and then it removed the jQuery 1.10 package.  It complained that it couldn’t find the intellisense file for the previous version. That’s because I had moved it to the new folder.

Now it’s time to update the bundles. When you create a new web project, ASP .Net will create some bundles for you. A bundle is a collection of scripts and stylesheets and does a neat little trick where at debug time you get the individual files as if you had specified each one in the web page. At runtime, the files are minified and consolidated into a single file. This compresses the payload and saves on loading time as the browser only needs to make a single request to get the files in the bundle. Any hoo, if you create a ASP .Net project in the current version of VS, you will get this bundle definition in BundleConfig.cs:

public class BundleConfig
{
    public static void RegisterBundles(BundleCollection bundles)
    {
        bundles.Add(new ScriptBundle("~/bundles/jquery").Include(
                    "~/Scripts/jquery-{version}.js"));
        // Other code removed...
    }
}

This call to bundles.Add creates a new bundle named “~/bundles/jquery” and uses the {version} regex placeholder, (\d+(?:.\d+){1,3}), to match the jQuery file without having to specify the version. You don’t have to worry about your code breaking due to an invalid reference to jQuery when NuGet updates jQuery.

What we do now is add a second bundle that will use the older version of jQuery. Since we copied those files to a new folder, NuGet is blissfully unaware of their continued existence and will not remove them. We just create a ScriptBundle with a new name and have it specify the folder that we put the old version of jQuery in.

public class BundleConfig
{
    public static void RegisterBundles(BundleCollection bundles)
    {
        bundles.Add(new ScriptBundle("~/bundles/jqueryold").Include(
            "~/Scripts/compatibility/jquery-{version}.js"));

        bundles.Add(new ScriptBundle("~/bundles/jquery").Include(
            "~/Scripts/jquery-{version}.js"));
        // Other code removed...
    }
}

The final step is to update the shared layout view that MVC puts the jQuery script reference in.

With a new project, you will edit the Views\Shared_Layout.cshtml file. The default jquery reference will look like something this:

    @Scripts.Render("~/bundles/jquery")
    @Scripts.Render("~/bundles/bootstrap")
    @RenderSection("scripts", required: false)
</body>
</html>

What we need to do is the replace the @Scripts.Render line with a conditional block of code that checks for older versions of Internet Explorer.

    <!--[if lt IE 9]>
    @Scripts.Render("~/bundles/jqueryold")
    <![endif]-->
    <!--[if gte IE 9]><!-->
    @Scripts.Render("~/bundles/jquery")
    <!--<![endif]-->
    @Scripts.Render("~/bundles/bootstrap")
    @RenderSection("scripts", required: false)
</body>
</html>

We use IE’s Conditional Comments to detect previous versions of IE. This a feature specific to IE versions 5 through 9. All other browsers will treat the Conditional Comments as regular HTML comments.

<!--[if lt IE 9]>

With this line, IE 5-9 (and 10 in compatibility view) will treat this as the start of a block of code. All other browsers will treat this as the start of a comment block. The block is closed when it hits “–>”.

    <!--[if gte IE 9]><!-->
    @Scripts.Render("~/bundles/jquery")
    <!--<![endif]-->

With this block, we are saying if IE is greater than or equal to version 9, to use this block. The “<!–>” on that line tells IE 10 and the other browsers to close out this comment block which makes that block of code available to the browser. This will allow you to stay current by letting NuGet updating jQuery, while still keeping some level of compatibility with older versions. This is one of the times where IE extending HTML in a proprietary way is useful.

The fork in the road image was modified from an creative commons licensed image posted by Eric Hamiter to his Flickr collection.

Sometimes 42 is the right answer

Prime Meridian

I was skimming the Codist blog when I read the following

One of my favorite data errors was when the latitude would be randomly in the 1000s even though the longitude was correct. MapKit would toss a fit so since I can’t know what the right value is, I just use 42 which puts the pin at the wrong location but at least it’s not crashing.

I just started laughing. Been there, done that, want it on a t-shirt.  When collecting positional data from other sources, sometimes you just get bad data in.  And in this case, 42 is the Ultimate Answer.

A while back I had written a data collection service that was collecting vehicle positions from onboard GPS device.  Every now and then, the locations became random.  This app was running in the continental US,  any latitude/longitude values outside that range for that area were just ignored.  In my case, any wrong location was unusable data.

The Prime Meridian image comes from Lindsay Holmwood’s Flickr feed.

A file versioning helper class in C# to make a backup copy of a file and keep the last N copies of that file

Cicero denouncing Cataline

While testing my configuration editor app, one of the QA guys asked if it made a backup copy of the file being edited.  My reply was “not yet”.  And I thought about it a bit and decided to implement the backup feature and to kick it up a notch.

In prehistoric times, VMS would automagically version your files when you saved them.  When you saved a file, the file system  would append a version number starting at 1.  You could open any version of a file by including the version number.  If you left out the version number, you got the latest version.  This goes back to the days when monitors supported both colors, green and not green.  To clear out older versions, you would periodically need to run the purge command.

Prehistoric times for web developers
Prehistoric times for web developers

I wanted something close to that in my code, except without having to manually purge the file system.  I wrote a simple static class that takes the name of the file and the number of versions to keep.

public static class FileHelper
{
    /// <summary>
    /// Make a numbered backup copy of the specified files.  Backup files have the name filename.exe.yymmdd##, where yymmdd is the date and ## is a zero justified sequence number starting at 1
    /// </summary>
    /// <param name="fileName">Name of the file to backup.</param>
    /// <param name="maxBackups">The maximum backups to keep.</param>
    public static void MakeBackup(string fileName, int maxBackups)
    {
        // Make sure that the file exists, you don't backup a new file
        if (File.Exists(fileName))
        {
            // First backup copy of the day starts at 1
            int newSequence = 1;

            // Get the list of previous backups of the file, skipping the current file
            var backupFiles = Directory.GetFiles(Path.GetDirectoryName(fileName), Path.GetFileName(fileName) + ".*")
                .ToList()
                .Where(d => !d.Equals(fileName))
                .OrderBy(d => d);

            // Get the name of the last backup performed
            var lastBackupFilename = backupFiles.LastOrDefault();

            // If we have at least one previous backup copy
            if (lastBackupFilename != null)
            {
                // Get the last sequence number back taking the last 2 characters and convert them to an int. And add 1 to that number
                if (Int32.TryParse(Path.GetExtension(lastBackupFilename).GetLast(2), out newSequence))
                    newSequence++;

                // If we have more backups than we need to keep
                if (backupFiles.Count() >= maxBackups)
                {
                    // Get a list of the oldest files to delele
                    var expiredFiles = backupFiles.Take(backupFiles.Count() - maxBackups + 1);

                    foreach (var expiredFile in expiredFiles)
                    {
                        File.Delete(expiredFile);
                    }
                }
            }

            // Create the file name for the newest back up file.
            var latestBackup = String.Format("{0}.{1:yyMMdd}{2:00}", fileName, DateTime.Now, newSequence);

            // Copy the current file to the new backup name and overwrite any existing copy
            File.Copy(fileName, latestBackup, true);
        }
    }
}

You would use this method like this:

FileHelper.MakeBackup("web.config", 5);

The first time you called it, you would get web.config.14031401. The next time would get web.config.14031402. When web.config.14031406 was created, web.config.14031401 would be deleted to keep the number of backups limited to 5.

A work in progress

%d bloggers like this: