The Windward Studio

Windward Blog Home

A Rant (and a Solution!) on Using Visual Studio Online for Continuous Integration

Posted on 05/05/2015

Please Share This


I’ve been setting up continuous integration with Visual Studio Online (VSO). The whole process has been a big pain in the butt, but I finally figured out what was going on.

I began my continuous integration adventure.

VisualStudioOnlineA while back my boss put me in charge of automated testing and I thought great, I’m into this stuff and we really need to improve the testing process for our software. Where should I start?

So first I looked into continuous integration with VSO, and of course, all the documentation I could find for VSO was about Team Foundation Server (TFS). So why didn’t they just name VSO something like “TFS Online” instead? Okay, fine, let’s move on.

I had to learn a bunch of terminology (build service/server, build controller, build agent, etc.). So I spent a few days trying to suss all this out until finally I was ready to get started doing some actual work.

Installation and setup started off just fine…

TFSLogoOnce I got to the point of knowing what I needed to install, the installation itself was fairly simple:

  1. Run TFS installer.
  2. Choose the build server.
  3. Choose all the defaults.
  4. Log into Visual Studio Online.

Ready to go. Not bad!

Next I created a build definition. It’s obviously all designed for .NET land so building for our Java Engine was a bit off the happy path. But it works using good ole msbuild.proj file. And as far as I can tell, you can run whatever you want at that point as long as it’s installed and configured on your build machine. Very cool.

My next step was to set up the unit tests. Again, it’s all designed for .NET land, so I needed some extensions for JUnit. No problem — Microsoft made some!

But Microsoft’s Java extensions don’t work.

Time to find another solution.

Let’s try open source!

JenkinsLogoOkay, fine, let’s see what Jenkins does. It has the same purpose as TFBuild — to automatically trigger builds and tests — and it was originally designed for Java. It’s open source, so you know that means it doesn’t have the greatest documentation, but it’s pretty easy to figure out. I got it installed in a fraction of the time it took me to figure out how to install TFBuild.

But after installing Jenkins, I still had to configure it and make sure it’s working. So I pointed it at my local Java Engine source and because it’s got Ant + Java support built in, compiling the Java Engine was a piece of cake.

A little while later I also got the JUnit tests integrated. Perfect! So then I wanted to get Jenkins pulling the source from Visual Studio Online.

Wham! I hit another wall.

Jenkins can’t pull the source from VSO.

I found out something new about Windows Live ID…

Windows Live IDAfter a lot of frustrating experimentation and research, I found that the problem is Windows Live ID authentication. With regular TFS using Windows authentication we’d be okay, but VSO is locked down behind Windows Live ID.

It turns out Visual Studio’s tf command line client, which Jenkins uses to interact with Team Foundation, does not work with Windows Live ID. You have to use the so-called Team Explorer Everywhere client for that.

So fine, I’ll use Team Explorer Everywhere. But of course, it doesn’t really work with Windows Live ID, at least not out of the box. You have to go into your Live ID profile in Visual Studio Online and enable this alternate authentication credential, which enables basic authentication. Okay, fine, so I set that up.

And what do you know – that didn’t work either.

Not only does it not work in Jenkins, I couldn’t even get it to work straight from the command line. And after some more digging, I found that “technically Windows is not a supported platform at all“.

I’m very frustrated with Visual Studio Online.

Finally, some relief.

In truth, I’m not so much frustrated with VSO as I am with Windows Live ID. But really it’s the whole system that pisses me off. None of this stuff works together, and I’m really tired of everyone’s BS. *sigh*

But finally, I found a solution. It turns out that alternate credentials seems to have been what was blocking the TFBuild Java extension I was trying to use.

I didn’t realize that at first, because the error message is completely unrelated, and apparently I’m the first person to ever have hit this problem because I couldn’t find it on the web. But anyway, I should eventually be able to integrate the JUnit tests properly.

On the bright side, I learned some stuff.

Here’s what I think is going on with all this.

  1. Visual Studio Online is not really meant to replace TFS. VSO is actually meant to get newer and smaller organizations hooked on Visual Studio and TFS, so it lacks some of the capabilities of TFS right now.
  2. Sorry, Windows users. TFBuild Java Extensions and Team Explorer Everywhere are designed for Linux and Mac users. This is by design because the developers want to get you hooked on Visual Studio and TFS.
  3. Microsoft hates Java. Java really runs everywhere, including Windows. And it’s very mature, and there’s a lot of tools like Jenkins that people have been using for a long time. The driving force and the nature of Microsoft is to get more CALs, and this is always how they do it – give a few free hits until the user is hooked.

Well anyways, as long as we’re using VSO, I don’t seem to have a choice but to use TFBuild. But at the same time, it’s looking a bit hopeful again, because I should eventually be able to get everything working nicely with it.

P.S. If you’re curious about the software I’m testing, then check out the Windward product overview.

Please Share This

Author: Tomas Ramirez

Tomas is a software developer turned QA manager. At Windward, he has developed and deployed solutions from end to end -- everything short of doing the sales calls.

Other posts by