Downloading vidd.me: The fun of sequential file naming

Vidd.me launched recently, as a simple and effective way to share videos online. You upload your video file or gif, it converts it into mp4, and you get a link to the nice html 5 page it's displayed on. It seems to work rather nicely and the team are adding new features all the time without adding new limits.

Since it loads the raw mp4 file into the browser, getting the file is easy. Open up the source to any page and you have something like this:

And with our handy friend wget, you can download the full video from the commandline without any special tools at all.

So far it's business as usual. You could have used anything to download it, (including your browser. Things get a little more interesting when you look at the source of their 'new' or 'top videos' page:
 

There's three things to note here:

• Each preview on the page has '-clip' added to the filename, to distinguish it from the main mp4 file
• Every file seems to be stored on the same server using the same directory structure, d1wst0behutosd.cloudfront.net/videos/
• The files appear to be named sequentially, with gaps for deleted or private files.

This presents a very simple way to find content. At the time I'm writing this they've got around 4200 videos, so let's pull down all the previews on their site in just a few lines:

#!/bin/bash
for i in {1..4200}
        do
                wget "https://d1wst0behutosd.cloudfront.net/videos/"$i"-clip.mp4"
        done
exit

A little while later.

If you feel like downloading everything on the site - full length - you just have to modify the script above ever-so-slightly, like so:

#!/bin/bash
for i in {1..4200}
        do
                wget "https://d1wst0behutosd.cloudfront.net/videos/"$i".mp4"
        done
exit

To take it further, It'd be pretty simple to set up a script that updates your mirror of the site: A cron job that grabs the 'latest videos' page, parses it for the highest video number, and then goes from the last downloaded video until that point (or just count up from the last known video until you hit too many failures in a row). As a side-effect of the one-way sync, you'd have a copy of any videos that were subsequently removed.

I have to point out that the last paragraph could violate the Terms of service, which has a few conditions against scraping and DDOSing you might fall foul of. While they don't seem to have yet, it's also likely they'll throttle heavy users

Instead of that, I'm more interested in picking and choosing based on what looks interesting from those previews I grabbed. Let's create a quick script that lets me specify an arbitrary number of videos (based on their number) and download them:

#!/bin/bash
if [ "$#" -eq 0 ]; then
    echo "no input arguments detected"
else
    args=("$@")
    for arg in "${args[@]}"; do
        wget -nv "https://d1wst0behutosd.cloudfront.net/videos/"$arg".mp4"
    done
fi
exit

It takes the video numbers as command-line arguments, so now I can do this:

 No need for browser extensions or custom software, you can now just grab any video you like with what you have installed.

As long as the privacy controls in place are solid and they put some throttling in place to stop people ripping the whole site every hour, there's nothing wrong with how they've set things up. The site is designed to be as open and accessible as possible, and right now they're doing that from the ground up.

ExifPeeler: A web-based exif removal tool

I bought a cheap-and-cheerful VPS a year or two ago from the gentlemen at BuyVM.net. I've used it on and off for the many useful things that you can do with a VPS, but hadn't really tested it's limits. But the server is single core with an astounding 128mb of ram and 10gb hard drive, what can you really do with that?

Enter ExifPeeler. It does one thing and does it competently: You upload a batch of pictures, it strips out the EXIF data from them and gives them back to you nice and clean. You can then do anything you like with the pics without worrying about accidentally giving away your location or any other personal data.

Here's what it looks like.

And here's what it looks like after you upload files. Simple, right?

Features:

• Batch upload. You can upload up to 100 files or 50mb at once, and download the results individually or as one big zip file. 
• Unique URLs: Each pic has a unique and distinctive URL. Only you or anybody you link to your images will be able to see them.
• Secure Timed Destruction: After an hour, your files are deleted. There are no options to keep them. Because I have such limited hard server space, I can't afford to leave your holiday snaps lying around for weeks. Let's call it a security feature.
• Cross-Browser Support: Works great on firefox, chrome, and safari. IE9 and below support single-file upload only. I haven't tested with IE10, so let me know how it goes.
• Mobile Device Support: It handles batch uploading from ios, android, and blackberry like a pro.   I haven't tested on win phone 7 or 8, so let me know if you try it. 
• Duplicate Renaming: If you upload 5 pics called 'image.jpg' at once, they'll all get appropriately renamed.
• Relatively Graceful Failure: if any individual pictures you upload fail - say, you accidentally upload a word document at the same time - it will still clean all the legitimate pictures and list any failures separately at the end.

Technology:

The server I'm running it on pretty much dictated the rest of the choices. One PHP script that does all the admin (uploading, removing anything that isn't photos, creating thumbnails, etc), while it farms out all the hard work to exiftool. It double-checks to see that the exif data is definitely gone, and will refuse to even display the image if it's not successfully removed. Every 5 minutes a cron job runs and removes any files uploaded more than an hour ago. 

The key consideration is traffic. I've done some stress-testing, and it should be able to handle multiple file uploads per second. Under heavy load you might get a 5-second delay when uploading large batches (50+ files), but we'll have to see how it goes in real-world conditions.

Why an exif remover?

I really just wanted to see how easily it could be done. While my solution isn't particularly elegant, it works and is damn simple to implement. It consists of one php script, one third-party program, and one cron job. 

I also couldn't find another site that does the same thing. There are sites that will let you remove exif data from single files, there are sites that have exif removal as a bonus feature hidden away in menus, and there are of course hundreds of local exif-removal tools. I couldn't spot a web-based exif removal tool that lets you upload more than one file at once. This fills that niche.

Don't a lot of websites remove EXIF data anyway?

Yes, they do. Sites like blogger, facebook, and flickr will prevent other users from seeing some or all exif data from your pics. The data is still present, and the companies can still use it for what they like. They just don't provide it to viewers.

Most smaller sites don't seem to remove EXIF data. If you're uploading pics to a forum or emailing them to somebody, it's definitely good to remove the EXIF first.

If I shouldn't send files around without removing the EXIF, Doesn't that mean I shouldn't upload them to ExifPeeler?

Yes. If you're worried about it, strip out the data before you send it anywhere. The aforementioned exiftool is good and there are lots of other options. Exifpeeler is great for casual use but if you have information you genuinely need to keep private it's best it never goes on the internet at all. 

Where's the script so I can run it in my own project?

I'll probably put it up here soon, but in the meantime you can email me if you want more info. The script is nothing special, I just want to clean it up a bit and see how it works in the real world before I post it. 

How many times was EXIF mentioned on this page?

18.

If you have the time try it out, and let me know if you manage to break it. If you do break it, please send a screenshot along with a brief description of what you were doing.

Dead Man's Switch on Linux, Part 1: Basic bash

I've always liked the idea of a dead man's switch. It's a partial fix for the 'what happens if I get hit by a bus' problem: how would you give your important account details to your family? How can you get a last minute message to your loved ones? Most importantly, who will delete your porn?

Traditionally, they work with the assistance of a third party. They've been used to launch nukes. Wikileaks used it to cause shenanigans. And you can sign up for some web-based solutions that will send pre-recorded emails for you when you show signs of being dead.

Of course, you have no control over that. the whole site might go down without you realising, rendering your efforts useless. So here's the same basic principle, implemented on your own software in bash script.

If you just want the script, skip to the end. It does things like this:

The Simplest Script Of All

We need three basic components to make this work:

1. The main script file. When run, it checks to see if the timer has expired. If it has, it performs pre-set actions (like sending email)
2. A method to 'reset' the timer. A one-line script file will work fine here.
3. A way to trigger the first script to run. We'll use cron.

We also need a way to check whether we've hit the time limit. To keep things simple, I'm going to use the 'modified' time on the main script file. When it runs, it'll compare when it was last modified against the current time to act as the timer. To reset, you just need to run 'touch myscriptname.sh' to change the modified time. Easy, right? 

So here's the main script, set to do the most important task of all: Delete Your Porn.

#! /bin/bash
# Simple Dead Man's Script

timelimit=30 #Number of days to wait without update

let timelimit=$timelimit*60*60*24 #turn that into seconds
lastaccessed=$(stat -c %Y $BASH_SOURCE)
timenow=$(date +%s)
let timeleft=($timenow-$lastaccessed)
if [ $timeleft -gt $timelimit ]
then 
    rm -rf $HOME/myporndirectory
fi

Here's the 'reset' script, saved somewhere on your desktop:

#! /bin/bash

touch pathto/themainscript.sh

And here's the line you have to add to cron (crontab -e to get there):
0 0 * * * pathto/themainscript.sh

Simple, right? make the scripts executable, and if you ever go 30 days without running the reset script it'll delete all your porn. Careful with your holidays.
It can do anything that can be done from the command line. with minimal trouble it can ftp files around, modify/upload entire websites, start torrents, or just update twitter with what you had for lunch the other day. I want to use it to send an email.



The Slightly Less Simple Script


Obviously I took a simple concept and made it both unnecessarily complex and annoyingly basic. It now has a setup option that should work most of the time, and a 'test' option that emails yourself and makes sure it works.

Boring Notes:

• The setup option assumes a generic install - that you have crontab access, that you have a home directory, etc. It will fail horribly if run as root.
• The setup script creates a .dmswitch/ folder in your home dir. Inside will be the check script, the message file, and an attachment directory. The message file is plain text and as long as you need.
• It also adds a reset_switch.sh file to your desktop. each time you run this, it resets the counter.
• The setup is optional. If you manually move the files where you like and change the variables, it'll work just fine.
• I'm using sendemail to do the email sending since it plays nicely with gmail. You can use any other method you like. If you use the script unmodified you'll need to install sendemail.
• Any files in the attachment directory will be attached to the email. No spaces in filenames though, because I am lazy.

Quick Install:

On debian/ubuntu, it takes four lines and some variable editing.

• sudo apt-get install sendemail libio-socket-ssl-perl
• gedit  dmswitch.sh (or nano, vim, emacs, etc.)
• paste in the script, and edit the email account details to your own. Save.
• chmod +x dmswitch.sh
• ./dmswitch.sh setup

Test it by running .dmswitch/check_switch.sh test . If the email settings are correct, you should have a test email arrive shortly.

Removal:

• rm -rf .dmswitch/
• crontab -e, remove the entry pointing to the script.

Finally, The Script itself.

#!/bin/bash
# Basic Dead Man's Switch v1.0
# Options:
# 1) dmswitch setup
#    sets up the script in your home dir with some default settings.
#    best to set it up manually or automatically rather than a mix of the two.
# 2) dmswitch test
#    sends a test email to the itself.
# 3) dmswitch reset
#    'checks in' with the dead man switch and resets the counter to zero
#    Just does the same thing as touching the scriptfile.
# 4) dmswitch
#    default. if the time has expired, IT WILL SEND AN EMAIL.


#SETUP VARIABLES
#some running variables are based on the setup vars
setupdir=$HOME"/.dmswitch"
setupattachmentdir="attachments"
setupmessage="message"
checkscript="check_switch.sh"
resetscript=$HOME"/Desktop/reset_switch.sh"
croncommand="0 0 * * * "$setupdir"/"$checkscript #cron line for how often it checks expiry. Default is daily.


#RUNNING VARIABLES
#make sure you change the email settings. 
dmdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
timelimit=30 #Number of days to wait without update
expired=0
emailto="someemail@gmail.com" #The target address
emailfrom="myemail@gmail.com" #The account you send with
emailusername="myemail@gmail.com" #The username for your email account
emailpass="myemailpassword" #password
emailserver="smtp.gmail.com"
emailport="587"
emailsubject="Automated email from Dead Man's Switch."
message=$dmdir"/"$setupmessage # the text file you want your message stored in
attachmentdir=$dmdir"/"$setupattachmentdir # put any attachments you want to include here


#checks if the time has run out. Does the maths in unix time.
function checkifexpired() {
    let timelimit=$timelimit*60*60*24 
    lastaccessed=$(stat -c %Y $BASH_SOURCE)
    timenow=$(date +%s)
    let timeleft=($timenow-$lastaccessed)
    if [ $timeleft -gt $timelimit ]
    then expired=1
    fi
}

#sends an email. replace sendemail with whatever program you prefer.
function sendemail() {
    attachmentlist=$(ls $attachmentdir)
    cd $attachmentdir
    sendEmail -f $emailfrom -t $emailto -u $emailsubject -s $emailserver":"$emailport -xu $emailusername -xp $emailpass -a $attachmentlist -o message-file=$message
}

#sets up a directory to run from and creates the necessary files.
function setupdm() {
    mkdir -p $setupdir
    cp $BASH_SOURCE $setupdir"/"$checkscript 
 rm $BASH_SOURCE 
    cd $setupdir
    chmod +x $checkscript
    mkdir $setupattachmentdir
    touch $setupmessage
    echo "If you can read this, I'm dead or arrested or something" >>$setupmessage

    #append cron job to existing cron file
    (crontab -l; echo "$croncommand" ) | crontab
    
    #setup reset script on desktop
 touch $resetscript
    echo "#! /bin/bash" >>$resetscript
    echo "checkfile="$setupdir"/"$checkscript >>$resetscript
    echo "touch $""checkfile" >>$resetscript
    chmod +x $resetscript
    echo "setup complete"
    
}


#main script starts here

checkifexpired

if [ "$1" == "setup" ]
then 
    setupdm
elif [ "$1" == "test" ]
then
    #send test email to yourself
    emailto=$emailfrom
    emailsubject="TEST: "$emailsubject
    sendemail
 
elif [ "$1" == "reset" ]
then
 touch $BASH_SOURCE
elif [ $expired -eq 1 ]
then
    #send the email, and disable the script from running
    sendemail
    chmod -x $BASH_SOURCE
else exit
fi

Go wild.
Part two will cover the fact that not everybody leaves their desktop on for months at a time and suggest several overly-elaborate ways to trigger the reset while running it on a remote machine.

Vodafone Australia Leaking Private Picture messages?

Over the weekend, one of our users had a pretty strange event happen. The basic chain of events went like this:

• The user receives an (expected) MMS message to their work and personal phones at the same time. Both these numbers are on vodafone.
• The user immediately starts receiving dozens of other MMS messages from numbers they don't recognise.
• After 10 minutes, the user turns off both phones.When they're turned back on, the messages have stopped.

While they'd been wiped off the personal phone, I got to have a look at the work phone this morning. The MMSes had all arrived at her email address which makes things a bit easier to analyse.

From this we can see:

• These were, beyond a doubt, not intended for her. The only thread even linking all the recipients (and senders) is that they're australian.
•  The messages all came through vodafones servers.
• The messages all have a send time approximately that of the receive. That doesn't necessarily mean that this was essentially a live capture of their MMS traffic, but it seems likely.
• A quick look at some messages shows a high incidence of people from WA. That could mean it was a WA-only issue, or it could be due to the time difference between us and the Eastern States.
• These are real MMSes. They are not spam, they were sent by real people who did not expect them to be made public.

 Here's a good example of the kind of thing that was leaked.
.

A student card. To go with his phone number, they gave us the high school, full name, date of birth, and some photo ID of a minor. Believe me when I say that this was far from the most personal piece of information there.

Most worryingly, Somebody has mentioned to me since then that a friend of theirs had the same thing happen yesterday - hundreds of PXTs being misdirected to his phone. He thought it was some kind of spam and changed his number, which is a shame.

I'd love to find somebody else who had this happen and still has some messages. With some more data we can work out a few more details:

• The time period. It was about 10 minutes for this case, but that may have just been the tail end - it could have been going for weeks in the right conditions.
• The trigger. I'm guessing it was 'receiving or sending a PXT message', but again I need more data.
•  Whether the same messages were sent to everybody. Everybody getting the same stream of messages is a much smaller problem than everybody receiving separate streams

With all of that, we can work out the scale of what happened here, and hopefully notify some of the customers who unexpectedly had their personal messages made public.

I should also mention:  I've contacted vodafone about this (via the authorised partner we deal with), but I haven't heard back yet. I'm very interested to hear their response.

Blogger removes GPS EXIF data from uploaded pictures

While writing my last post, I found out something interesting: Google Blogger strips out the GPS EXIF tags when you post it to your blog while leaving the rest intact.

Here's the exif data of the pic I posted yesterday - all of it identical to when it was taken.

 And here's the same pic uploaded to blogger and then downloaded again. Everything is the same except the GPS data, which is mysteriously missing. (The thumbnail is upside down, but I'm guessing that's because I used different programs to rotate them).

And just to make the situation more complex, I can go and have a look at the uploaded photos in picasa - and it shows the gps data as being intact!

So google doesn't modify the original at all, but will quietly rip out GPS data from any copies that are displayed publicly. This is a pretty useful safety feature. With it, I can just upload any picture without worrying that I'm giving away where I live or work.

It's confusing that I can't find this feature noted anywhere. It's the kind of subtle but important feature that shows a product has been carefully thought out and sets it apart from the rest, but neither the help documents nor my frantic googling found a single reference.

If you want a job done right

It's great when a phone company wants your business. Whether or not you're considering changing over, they offer nice perks. The latest company gave me an iPad and a playbook for a few days, to 'test out their network'. (Summary: the iPad is built better and more polished, but I love some of the playbook UI choices. After swiping from outside the screen to switch apps, double-tapping the iPad's home button felt clunky and wrong. But I digress.)

Before The Salesman handed over the iPad, he made a point of spending a few minutes clearing user data - it had just come from another company  Okay, he's not doing a secure wipe, but a quick wipe in each app is enough. At least they care about confidentiality.

This was the first sign they hadn't been thorough.

The best way to test out hardware is to buy miniskirts. Try it sometime. 

I've hidden one entry to protect The Salesman (who we might end up dealing with), but the search history is interesting regardless - mens fashion and porn. Later on I poked around to see if there was more.

I didn't investigate too much, but there's a good mix: email, mens clothing, ASX announcements, straight porn, gay porn, email again, and finally cruise ships. From this we can deduce he's an open minded young man who takes care of his looks, has a solid financial basis, and keeps track of his investments.

And since The Salesman didn't know how to clear all the pics, the man with the interesting browsing habits works with these guys:

It looks like he works in a car sales yard, but you can't tell much else. Most of the other key apps are clear of data, so the privacy violation should stop here.

Except the apps he installed require his account to upgrade.

This is getting awkward. The email is his full name including initials and there's only one guy in this city with that name. So that leads straight to his facebook account, which leads to his 500+ friends, his girlfriend, his family, and so on. I've never met this guy, but from an iPad he used for one day we can link his porn habits all the way back to his family and workplace, even after someone tried to wipe it.. Imagine what the phone you use every day has?

The moral of this story: Your personal data is important. Don't trust others to handle it for you if you can handle it yourself.
And Wipe your test devices when you're done with them. 




UPDATE:
Somebody pointed out that I hadn't checked to see if the photos had the GPS location they were taken stored. And hey, what do you know?

So in addition to everything else, we've been provided with the very building he works in. Awesome.

A brief analysis of Yahoo captchas

Captchas, initially a huge annoyance, are generally recognized as a necessary evil now. They stop bots from abusing your services, and there's a lot of interesting variants to use. The biggest is google's recaptcha, which is so popular even microsoft uses it occasionally. Today my attention is on Yahoo's implementation. You'll know them, they look like this:

In a nutshell: I had to type a few of these lately, and the character distribution didn't look quite right. I grabbed a hundred captchas, laboriously typed them out, and broke it down by character.


What you can't see here: Yahoo works with the traditional 'random combinations of letters and numbers' form of captcha. They use at least three different fonts, which are then physically skewed in a variety of ways. There's no additional visible interference between you and the letters, and the average length is 7.2 characters.

What you can see: Yahoo captchas use a relatively small subset of alphanumeric characters. A, B ,F ,G ,H ,J ,L ,M, T and V appear only in uppercase while c, d, e, n, p, r, s, t, y, q, y and z appear only in lowercase. Out of the numbers we have only 2, 3, 4, 5, 6, 7, and 8. This leaves 8 alphanumeric characters completely unrepresented - i, k, o, q, x, 1, 9 and 0.

Most of these seem to be omitted due to possible confusion. O, o and 0 are easily mistaken and so all are avoided, and the same goes for l/1 and K/X. Additionally, some two-character combinations which look similar to existing characters are omitted.

In this example the letter d is very easily mistaken for either 'cl' or 'ol' due to the font. However c, l, and o never appear in the captchas, presumably for this reason.. The letter p suffers similarly, while B and 8 manage to escape despite being sometimes difficult to distinguish.

I'm not entirely sure of the strategy here. They're purposefully obfuscating the word by overlapping the characters, but at the same time dramatically reducing the number of characters that could be present. By cutting down the total alphanumeric characters from 62 to 28 they're making it easier for OCR to render their technique ineffective.

RIM: Now checking your email from the wrong country

I like the way my blackberry handles multiple email accounts. RIM servers effortlessly stream my gmail traffic directly to me wherever I am, handling two-way sync, calendars and contacts with ease.

Today google forced me to re-authenticate in gmail a couple of times. A quick look at the recent activity list revealed why:

216.9.249.99 turns out to be bda-216-9-249-99.bis3.ap.blackberry.com and is the first time a foreign IP has been handling my mail for an extended period.

They have Australian servers that work perfectly well, and I cant find any notices about local downtime. I'm not entirely happy with this, we'll have to see if it stays.

Edit: A week later, It's still all Canada. Either google has reclassified all RIM IP space as physically in canada, or RIM has made a fairly drastic change to how they access your email account.

Boxify.me: A brief lesson in what privacy isn't

Boxify.me is getting a bit of press lately as a new site that allows sharing multiple files with others in 'boxes'. No sign up required; you just click on the 'start sharing' button, upload files, and share the URL around with others.

Here's their current front page.

Before I go further, I should mention this kind of sharing is inherently insecure - they make no promises about keeping your data safe and there's no password protection. Anything you put up there can be accessed by whoever has the private URL, and that's how it's designed. The only thing they specify on that page there is that your box has a 'private URL'.

That should be easy, right? All they have to do is set up robots.txt so that nobody can spider their site, after all.

Unfortunately, I'm wrong.

And as a result of that:

Now, I know what you're thinking -"That's not too big a deal, google can't find anything that's not already linked to". That's where the second mistake comes in, which is only obvious because of the first.

When you click on the link to uploads.boxify.me, you get this lovely page.

Yup. That a public xml file with hard links for the thousand most recently uploaded files. I'm not sure why the file exists, but the fact it's publicly accessible is just terrible. In 15 minutes you could knock up a shell script that regularly checks for and downloads every single file uploaded to the site.

So the outcome here: Boxify.me may turn out fine one day, but so far Loren Burton's  claims of a private URL aren't holding up.. If you don't want your files immediately available to the general internet, don't use boxify.


Interesting Sidenode: The example box they link to on the front page can also be edited by the masses. Consequently, it has naughty material for the discerning visitor.

Gmail forwarding security reminders

Since Gmail was first released in 2004, it's led the field in webmail security. They were the first big player to pioneer systems like two-factor authentication and geographic checking, and they've still got a few nice settings that nobody else has implemented yet - like demanding immediate verification if user activity seems suspicious rather than notifying you after the event.

But with everything else, they've always had a bit of a hole in their account security: Forwarding. Hidden away in the gmail options is a setting that will silently forward a copy of all your received email to an email address. It's incredibly useful (I've been using it since 2005), but it's a setting few people pay attention to. All an attacker would need is two minutes of access to your account - say when you wandered off to get a cup of coffee or answer the phone - and they'd have access to all your incoming emails for the indefinite future.

As of today, google seems to have addressed the issue: If you have with forwarding enabled, when you sign in there's a notification bar at the top of the screen letting you know just what those settings are and asking you to double check that they're correct. It's a fairly understated but you also can't dismiss it manually.

If you click on 'why is this notice here?' you get an explanation:
You’re seeing a notice to help you confirm that the forwarding setting that’s active on your account is accurate. If your account has this feature enabled, you should see this notice ... For about a week, this notice will appear for a few minutes each time you sign in to your account. Displaying the notification in this way helps ensure that you have a chance to see the notice, rather than someone who might try to gain unauthorized access to your account and use this setting improperly.

It's a great start, hopefully after the initial week is up it'll turn into one of their occasional reminders rather than being scrapped completely.