Isshen, LLC: Category Subversion/SVK

Subversion 1.4.4 upgrade

Hosh — Sat, 25 Aug 2007 20:23:00 -0500

Adam had recently got a MacBookPro. We no longer have to mess with InstantRails on Windows. It also means he can use native Subversion and SVK. However, the OSX-native package uses Subversion 1.4. Our servers uses Subversion 1.3.x. To be fully compatible, that would mean compiling a custom copy of Subversion 1.3.

After reading about the feature additions for Subversion 1.4, I decided it would be much easier for me to upgrade the server and my dev laptop to 1.4. Gentoo masks 1.4.4, but most Subversion upgrades are fairly transparent. There are two main benefits from the upgrade I wanted. Subversion 1.4 claims to be better at storing and transmitting binary data. We version a lot of raw data, so this would work out well. Subversion 1.4 also has a svnsync command that works like SVK's svn mirroring. It lets you move or clone a repository. Maybe Linus Torvald's acerbic comments about Subversion has been getting noticed.

Upgrading to Subversion 1.4 on Gentoo is a piece of cake. You add:

echo "=dev-util/subversion-1.4.4-r3" >> /etc/portage/package.keywords

and then

emerge =dev-util/subversion-1.4.4-r3

You can run revdep-rebuild to make sure all the library references are working, but I found that this was not necessary, even with the SVK installed. (With the perl use-flag set on Subversion, all the Perl bindings gets automatically updated on the system.)

To take advantage of the more-efficient binary storage, all the repositories needed to be upgraded. I shut down Apache to prevent anyone from making commits, backed up my repository, and moved the repository to a different directory while I setup the new repository directories. I then wrote two quick-and-dirty shell scripts to do this:

#!/bin/sh mkdir tmp_dumps for repo in `ls repos-1.3.x`; do svnadmin dump repos-1.3.x/${repo} > tmp_dumps/${i} done

and

#!/bin/sh for repo in `ls tmp_dumps`; do svnadmin create --fs-type fsfs repos/${repo} svnadmin load repos/${repo} < tmp_dumps/${repo} done

Note the --fs-type fsfs flag. You will have to change that if you are using the BerkeleyDB storage engine. These two scripts essentially creates a dump of all the repositories found in repos directory, then reloads them. During the reload, Subversion 1.4's svnadmin create command creates a new repository. The newly reloaded repository uses the more efficient binary storage scheme.

Lastly, I upgraded all of my local working copies. Subversion does this transparently, but I have a number of large repositories I use often. A simple svn update on each of the working copies upgrades the working copy to 1.4. Some of them take ten minutes to upgrade -- I do this now so I don't have to wait, later on.

Ho-Sheng Hsiao
Isshen, LLC

How to do http + ssh with Subversion on Windows

Adam — Fri, 27 Jul 2007 20:31:00 -0500

Many public hotspots block svn traffic, even those accessed via http. This predicament can lead many (like me) that must develop in a Windows-based environment in an unfortunate place while on the road.
Accessing subversion shares, need not be a headache, there is a simple solution.

First you should install TortoiseSVN and verify it is working.

Then right click on the desktop (or an explorer window) highlight “TortoiseSVN” then click Settings.

Click the “Network” tab

Check the “Enable Proxy Server” box, and in the “Server Address:” field place “localhost” and in the “Port” field place an unused port greater than 1024 (remember this number!). Click “Apply” then “OK”

I recommend Putty Tray and their suite, including their improved version of Pagaent, for reasons that will soon become obvious for the next section.

Open Pageant, then open your keyfile in it.

Continuing on, open up Putty, fill in your server address (make sure SSH is filled in). Then go to Connection -> SSH -> Tunnels. Under “Source Port” place the same number you filled in earlier, and in destination, place the address of your server and port 80, ie: my.server.net:80, Click Add. Go back too the session tab, save and name your connection (don’t want too do this every time).

Click Open.

You should then be able too use TortoiseSVN normally, though remember you will always need the SSH connection running too use it, it will not work otherwise (unless you clear the proxy in the settings).

This is why Putty Tray is so great, you can minimize it too the system tray, and have one less window cluttering your developing environment, until you absolutely need too do something on that command line.

As a ‘best practice’ rule, I have learned too set the background colour on my tunnel window to something other (and obviously not) black, and not use it for much of anything besides tunnels.

Good Luck!

References:

How to do HTTP+SSH with Subversion on Linux

Branching and Merging with SVK

Hosh — Thu, 05 Jul 2007 04:15:00 -0500

Originally posted on a private mailing list.

Talking with a guy on #ledgersmb, I asked him about how well SVK and SVN worked together. I had previously used SVK for disconnected commits, but had not had the nerve to set up an SVK server.

It turns out, you don't need to set up a SVK server, per se, to take advantage of SVK's branching and merging.

To play with it, I set up a directory with a single file:

test/test.txt

And branched it with

svk cp test test-branch

If I modified test-branch/test.txt I can use svk push inside test-branch, and it will automerge back to test/test.txt

If I modified test/test.txt I can use svk pull inside test-branch and it will automerge modifications from test/

I had an svn client working with, and it was able to update the changes cleanly. I can use svn to do commits against each branches, and SVK handles that no problem. As far as the svn client is working, someone did branching/merging, no problem. This is made possible because SVK stores the merging information inside SVN's revision properties. That's why I don't need a special SVK server, just using SVK as a beefed up SVN client.

Now the next tricky thing was using svn cp test-branch test-svn-branch and playing with that. SVK still enters in merge tickets, but getting changes was no longer a simple svk update. I had to use svk pull at the parent directory to get it to update. For what I want to use this for, I will just have to remember to create branches using svk cp rather than svn cp.

Why bother with all of this stuff? The short time I worked for BankOne, I learned a neat method for releasing software. Changes flows from:

development -> testing -> qa -> production

When I first setup Subversion, I had attempted to do something like that. Commit changes to trunk, and have changes pushed upstream for testing (I skipped Q/A as that was done during testing), then to production. The problem is that Subversion's merging sucked. It sucked because it does not track merging changes.

Having played with this, I can now do something like:

(developer-1, developer-2, developer-3) -> trunk -> testing -> production

Pushing to testing and production would be a simple svk pull

Each developer has their own branches to isolate any potentially destructive changes. The developers do not necessarily need SVK (unless they are doing disconnected commits, such as if they are roaming around with a laptop like I do). This is a good thing, since TortoiseSVN is a known factor, and Windows may or may not play with SVK (written in Perl) very well.

The supervisor or PM would have commit priveledges to trunk, probably by using a different set of passwords, so he would do an svk push inside a developer branch once he makes sure it won't destroy anything. Lastly, all the changes, mistakes, whatever can get versioned, a habit I have similar to hitting "save" all the time. Nightly or continous builds would come off of trunk/testing.

Summary: SVK transparently makes merging as trivial as SVN branching without having to change anything server-side.

Disconnected versioning with SVK

Hosh — Sat, 20 Jan 2007 06:11:00 -0600

Originally published on a private mailing list.

We were talking about subversion and commits a couple days ago. Adam and I prefer doing fine-grained commits to the trunk, whether the commits work or not; UG and Travis prefer committing changes when the code builds / works. I briefly mentioned a tool called "svk", but had not really investigated this until now.

Very soon, I will need to be able to do development in an environment where I might not have internet -- 4 to 8 hours at a time. That's when I took a good look at SVK, and it is awesome.

My first concern with SVK is how much of a migration is needed from Subversion. For what I need it for, the answer is: nothing. SVK lives mostly as Perl libraries, and operates on the client (local) side. I would be able to use SVK for doing local development, and Adam can continue doing updates using the same Subversion tools he had been using. His software (TortoiseSVN) wouldn't even know I was using SVK instead of Subversion.

The concept works like this:

svk depot --init svk mirror http://svn.mydomain.com/tools/trunk //tools/trunk svk sync //tools/trunk

That creates the mirror of the Subversion repository. The depot --init command initializes a repository defaulting to my home directory on my laptop. No messing with server or system configurations.

I branch my local copy

svk cp -p -m "Branching for local copy." //tools/trunk/calendar //devel/calendar

svk co //devel/calendar ~/work/calendar

And that's it. No .svn/ directories in my work area.

I'm the only developer, but if/when we get others working, there will need to be updates from the trunk to the branch. That's done by:

svk sync //tools/trunk svk smerge -C //tools/trunk //devel/calendar svk smerge -l //tools/trunk //devel/calendar svk update ~/work/calendar

Merging changes back is:

svk sync //tools/trunk svk smerge -l //devel/calendar //tools/trunk

Since changes are made into the mirror, SVK will automagically committhose changes back to the Subversion server. smerge is the Star Merging algorithim introduced in by GNU arch. It will keep track of what you merged from branch to branch. Since Subversion doesn't keep track of this, doing something like this in Subversion requires a lot of digging and manually specifying revision ranges. Not fun.

There are some downsides to SVK. The main one is that it is written as a series of Perl libraries, and may require CPAN dependencies. I have no idea how well it would run on a Windows desktop.

The second is that I can't use the handy-dandy script/generate controller mycontroller --svn

... the way around it though takes advantage of how SVK stores local branches. Since mine's is at ~/.svk/local , I can do:

svn co file:///home/hhh/.svk/local/devel/calendar

and use my regular svn commands.

For further documentation on SVK, check out:

http://www.bieberlabs.com/wordpress/svk-tutorials/
http://svk.bestpractical.com/view/InstallingSVK

How to do http + ssh with Subversion on Linux

Hosh — Sat, 16 Dec 2006 08:30:00 -0600

Originally posted on a private mailing list.

One of the consequences of having a mobile development station is being able to go to places such as Panera's or the local coffee shop to do development work. These days, it is trivially easy for people to get something similar to Kismet and do packet sniffing ... so I tend to proxy my internet connection through ssh. Fortunately, most of the GUI programs does this -- Firefox, Thunderbird, Gaim, Xchat2.

Unfortunately, Subversion doesn't.

We have it set up so that Subversion runs through Apache in order to take advantage of authenticating (but not authorizing) against a MySQL database. In this way, svn+ssh is not an option. However, the svn command line does not understand SOCKS4 or SOCKS5. This is where a transparent socks proxy comes in. I'm using something called proxychains, though tsocks is also available on the internet. You set it up by writing into ~/.proxychains/proxychains.conf with:

DynamicChain
tcp_read_time_out 15000
tcp_connect_time_out 10000

[ProxyList]
socks5 127.0.0.1 1080

and then you have two methods for accessing it. For svn commands, you can do:

proxychain svn ls http://svn.mydomain.com/project/trunk/

but I prefer just setting the environment variable,

export LD_PRELOAD=/usr/lib/libproxychains.so

Update 2007-07-22: I now put in my .bashrc file alias svn="proxychains svn" and alias svk="proxychains svk"

The library intercepts TCP connect calls and reroutes them through a socks proxy. Transparent, effect, ... and kinda scary, considering how easy it is to replace DLL calls like that. In any case, although I did this mainly because I have a laptop and Panera's uses an http proxy, I think I'll need to do this for my workstation as well. Then require a proxy to access the Subversion server.

References:

How to do HTTP+SSH with Subversion on Windows