WTA20 – Photoshopped or not?

Date: October 15th, 2011 12:00 p.m. – 2:00 p.m. EST

Attendees: Justin Rohrman, Eugenia Yakhnin, CJ Weyandt, Ageev Ilya, Alan A. Jorgensen, Abigail Buell, Heba Hosny, Melissa Bingham

Facilitator: Albert Gareev

“Photoshopped or not?” is a title of a simple program we evaluated from two perspectives. See mission assignments and reports below. Chat transcripts: GroupA, GroupB, SessionChat. The notable piece is comparison of test strategies.

Mission One

The editor of “True Shots” magazine requested a professional evaluation of digital photo analysis program www.pskiller.com capable to recognize if an image was manipulated in some way.

You have 45 minutes to come up with your testing strategy, implement it, and prepare a report.

Mission Report

Assumptions:

1 – Since we don’t know the user base size, we can only evaluate performance in so far as it relates to the analysis of single images of varying sizes.

2 – We do not know what software will need to be identified so we will go with large commercial examples such as Photoshop, GIMP, …

Strategy:

Determine whether pskiller can consistently and correctly identify whether an image was modified and also the tool that was used to modify it.

Results:

Pskiller does not provide a reliable response as to whether an uploaded image has been modified. This tool consistently reports false positives (responses indicating that an image has been modified when in fact, it has not) for .JPG images, a primary image format. This failure occurs in uploaded images as well as images that are provided through a URL.

Mission Two

The owner of “Retouch-R-Us” web resource is concerned about www.pskiller.com program claimed to be capable to recognize if an image was manipulated in some way. He requested you provide some evidences where PSKiller gives false positives or false negatives.

You have 45 minutes to come up with your testing strategy, implement it, and prepare a report.

Mission Report

Strategy was exploratory. We tested multiple tools and color manipulation, and also a bit of size boundary testing. We received false reads on jpgs that were not manipulated.  We also got a pass on a photo switched to black and white. We do not feel that PSKiller is sound in its analysis of photos.