Introduction

Imagine you recorded a large video file that you want to share with a friend. Let’s say it’s 4GB in size. How would you get it to them? You could upload it to Google Drive or Dropbox. But then you’d have to upload the entire file, and your friend would have to download the whole thing before they could watch it.

If you’re hip you could use a service such as FilePizza or Instant.io, which use the the WebRTC protocol to send the file directly from your browser to your friend’s. This is better, but the file is still taking space on your friend’s computer. Maybe they want to watch it on their phone and don’t have 4GB to spare.

If you’re more technical, you could upload the file to Amazon S3 or a private web server or Plex server. This would save your friend the trouble of downloading it (S3 supports HTTP range requests, which would allow them to stream it), but it doesn’t save you the trouble of uploading it or managing the server.

fibridge was designed for a very similar type of situation. fibridge (FIle BRIDGE) creates a bridge from your computer, to a server, to your friend’s computer. It allows your friend to use any app capable of playing videos over HTTP (browser, VLC, etc) to stream the video directly from your computer, without ever having to download the entire file. Seeking and pausing work as expected.

How does it work?

Note: This section is somewhat technical and more targeted towards developers. Feel free to skip it.

The core of fibridge is the proxy server, which is a web server speaking HTTP on one side, and a WebSocket server speaking the fibridge protocol on the other. In order to host a file, a JavaScript library running in the browser connects to the WebSocket server. The server gives that browser tab an ID, which the browser uses to generate URLs for the files. Any requests coming over HTTP for that ID are forwarded to that browser through the proxy.

The fibridge protocol is pretty simple. It could easily be implemented outside the browser, for example if you wanted to make a Node script to host a whole folder of images without having to select them each individually.

All of the heavy lifting is done by omnistreams, which we plan to more fully introduce in a future post.

Note: This concept is very similar to the browserver, project, which appears to be no longer developed. The key difference is that fibridge implements HTTP range requests, which enables the files to be streamed.

Why did the iobio team create this?

Although I think the video streaming example is pretty cool, why did we actually create this?

iobio is a suite of open source tools for real-time genomic data analysis in the browser. As such we do a lot of web streaming. Genomic data files can easily be in the 2GB-200GB range, or even bigger (single files in the terabyte range are not unheard of). Load up this link: https://bam.iobio.io (sorry, not mobile friendly) and click “launch with demo data”.

The source data for what you see in that app is a 187GB BAM file that lives on our Amazon S3 bucket. Streaming is the reason we’re able to visualize that file in real time. We sample the file, and figure out exactly which chunks we need to grab, then use HTTP range requests to get them.

But what if that file happened to be on the user’s hard drive? That’s where fibridge comes in. If a user selects a local BAM file, it is routed through fibridge, over HTTP, and back to the user’s browser. Why all the roundabout? Why does the data have to leave the browser in the first place? Because the software for processing BAM files is specialized and doesn’t run in web browsers, which necessitates handling the data on our servers.

Using fibridge allows us to use the exact same code for handling local files as we do for any other file on the web.

Demo

WARNING: Do NOT visit the website shown in the video. We don’t own the domain anymore, and the last time I checked it was NSFW.

If you want to run fibridge, you can host it on your own server. Linux executables can be downloaded from the release page.

Code

The source code is available on GitHub.