Dear Lazywebs: filesystems

[call_waiting]

I have a problem. I'm sure I've written about or ranted about this particular problem before, but I can't find any reference to it, so here goes.

My problem can't be that uncommon. It is the conjunction of these things:

• I have more than one computing device (including work computer, and VMs), which run different operating systems: Mac OS, Linux

Comments

[andrewducker]

So what you want is it to start off with a folder full of things that look like files, but are actually triggers to go and get the file.

The first time you access one it goes and gets the file in the background, and thereafter works with that file exactly like Dropbox does.

When you run short of space it should take the files that haven't been accessed in the longest time and revert them back to being triggers.

Presumably it should also allow you to say "Bring down a local copy of this file/folder" (so that you can make sure you have a local copy before going offline), and also "Always keep a local copy of this file/folder".

Seems simple enough. Now that Dropbox have an API you could probably write it yourself :->

[call_waiting]

So what you want is it to start off with a folder full of things that look like files, but are actually triggers to go and get the file.

Pretty much. A 'trigger' would need some sort of support in the underlying filesystem; instead, using a symlink will redirect to the original on the server. Running a 'sync' operation will spot that the file has been accessed recently and decide to duplicate it locally or not.

'Presumably it should also allow you to say "Bring down a local copy of this file/folder" (so that you can make sure you have a local copy before going offline), and also "Always keep a local copy of this file/folder".'

Yep - I'd imagine a simple version-control-like command line interface to implement options like 'sync' / 'push' / 'pull' / ''cache' / 'dropcache' or similar.

'Seems simple enough. Now that Dropbox have an API you could probably write it yourself :->'Well, not without some underlying filesystem support: using symlinks needs an actual real file to link to so that inital accesses are successful, but that fits

[andrewducker]

Aaah, gotcha now - although your method would mean that your base file system has to be local when you start. If the files were all held in Dropbox originally then that wouldn't work - because you can't symlink to Dropbox, and the files aren't actually instantly available.

I'd say that you could write your current version pretty easily - but it has the major limitation that if you're away from home then you can't access your files at all, even if you have blazing internet access. Maybe if you had a VPN to your home network.

On a larger scale, I think that writing your own filesystem so you could fetch files as needed, no matter where you are, would be a good thing :->

[call_waiting]

Yes - I don't imagine the base filesystem being anything other than a 'real' filesystem (at least, not for the purposes of starting working on something ;) but that's more likely to be an sshfs filesystem that mounts my Dreamhost file space as a local filesystem through the magic of FUSE, without Dreamhost anywhere in the loop.

A filesystem similar to TsumuFS but back-ending onto sftp / S3 / DropBox would be a great longer-term project, but there's lots of troubles once you start putting new code in between your applications and your underlying filesystem, the effects of bugs can be catastrophic and lead to serious data loss so you have a much, much higher requirement for reliability and testing before anyone will trust it.

And, of course, if it could do it on a level smaller than a whole file, it would also be great for VM disk images and movies, heh... ;)

[andrewducker]

Have you checked out Git Annex?

It's not what you want, but it's interesting.

http://git-annex.branchable.com/

[call_waiting]

Heh, yes - I mentioned git-annex in the same breath as Dropbox :) And in fact I'm pondering replacing my usage of Unison with git-annex

[call_waiting]

Actually... git-annex does more than I thought it did... and it might just be possible to make it do everything I need with trivial amounts of extra scripting... Hmm... awesome!

[call_waiting]

Hmm, actually, in the end, not quite. Drat. Very close though. A-coding I shall go...

[anonymous]

I believe I have the exact same problem and have been shopping around as well.

bittorrent sync looks like the best solution so far, though I haven't started using it myself. I just started consolidating my files in order to take stock of how I should change my cloud service (currently wuala) subscription pattern.

I also run a NAS, but unfortunately it cannot run BT sync itself. If you use an RPi or synology server, it appears that you can.

[momentsmusicaux]

I think that different systems for different types of stuff is actually the way to go.

Git for source code is a good example. I wouldn't want my git repos to magically sync across machines; it would be terribly confusing, as I'm often working on different things on different machines. I suppose if they were synced it would be like always being on the one machine that magically appears either in my laptop or my desktop... but then I'd need the databases to sync too, and they're outside of the home folder, and in fact, my laptop has PHP 5.4 and my desktop 5.3 and urgh....

[call_waiting]

It's basically like having a single filesystem: so for that use-case, instead of having two repos differentiated by what machine they're physically located on, you'd have two repos of different name / location within the filesystem, available on either machine, and differentiated by what you're actually working on at the time. Much more sensible in any case to my mind ;)

The trouble with having an application-level solution is that you need a solution for *every* application. Picasa and iPhoto don't solve the problem, so I'd need to find another application for that. There doesn't seem to be a solution for movies either. Documents... would work with Dropbox etc but only because they're small enough to fit mostly... A sync-level or filesystem-level solution solves it once for all applications.

[anonymous]

Third Try,

You may desire to take a look at Git-Annex-Assistant.

Based on Git, yet still intended for general use, with a consumer-oriented graphic interface.
It is open and free, and available on Linux, Max OSX, and Android.
A work in progress. Windows is being improved over the next several months (as of July 2013)
With satisfied users, contributing money to the creator / maintainer, Joey Hess

It can store and keep track of files in multiple locations, on multiple machines you own, all at the same time, and in multiple on-line cloud storage providers.

Look up via Google or other search method, since Live journal considers anonymous web links as spam comments.

Windows is ALPHA quality as of July 2013, Android BETA
See the download page for current status of platforms (via search if necessary)

As of July 2013, also, there is an effort to sustain and support another 6 to 12 months more of programmer effort for improvement. (search for Joey Hess, campaign, 2013, Git-annex-assistant)