About a month and a half ago I had a good friend visiting us for almost a month. Being also a photograph enthusiast, and one with a nervous finger, he took just over 4 thousand photos with his Nikon D5000. Of course we needed to download the files from his camera and free his SD card often, and we did that on my home computer, which is where I usually do the same with the photos taken with my Nikon D40. At the time he did not have his laptop with him and since I would want a copy of his photos as well, transferring them to my computer seemed only logical, I have a good system, backups in place, the whole shebang.
So it became common practice during that month, we would download the files to my computer, deleting the originals from his camera. To do that, instead of using the operating system to simply copy the files, I was using the software that came with my Nikon D40, that is, Nikon Picture Project 1.7 and Nikon Transfer, because of its renaming and auto-rotating option. I had never updated the software to Nikon ViewNX and the newer Nikon Transfer because the older one had always worked quite well. Little did I know that the older software was ruining some of my friend’s photos.
Because my friend took so many photos we did not go through a lot of them immediately, we were just downloading them, and only after he had left we found out that all his photos that were supposed to be in portrait orientation were corrupted. The preview was there, the thumbnail was there, but Photoshop Lightroom wouldn’t open them and despite having the latest Nikon RAW Codec installed, none of the viewing software we used, like Picasa Photo Viewer or Windows Photo Gallery would be able to show the files correctly.
After trying a couple of things like converting the files to DNG with Adobe DNG Converter and trying to open the files with other software, I had decided to contact Nikon and ask for help, while also asking for help on Photo.Net. I described the problem in detail and to Nikon I actually sent them one of the problematic photos, and the first answer I got from Nikon was that they were able to open the photo with their latest version of ViewNX and that I should download that. I did, and yes, I could view the file, but I could not export the file to anything, therefore I could not process the file or open it with any other software. Fiddling with ViewNX a bit I found out that what I was looking at was the embedded JPEG file inside the NEF file, not the actual RAW data, and selecting the option to view the actual RAW data in ViewNX rendered me a blank photo.
I replied that to Nikon and continued with my own investigations. In the meantime I got a lot of replies from Photo.Net but none that really solved the problem, although it was a lot of good advice that will help me in the future. We had a hunch that the problem was not with the image itself, but with its metadata, based on the fact that it is pretty much just metadata that is changes with the auto-rotate option, which was the option that most probably ruined the files since only portrait-oriented files were corrupt. So I started to play with the file’s metadata using a great tool: ExifTool by Phil Harvey.
To start I removed all metadata from the file, which didn’t work since RAW files do need some basic metadata to be opened, unlike JPEG files. That gave me another idea, so I asked my friend to take a portrait-oriented photo of a white wall (a gray card would have been better) with his Nikon D5000 and send me the RAW file. With that file in hands, I extracted all the metadata from it and replaced the metadata of one of the corrupt files with that. Voilà, I was able to open the corrupt file with Lightroom, but I still had some problems: the embedded JPEG of the corrupt file was substituted as well, meaning the thumbnail of that file was a white wall, and all other useful metadata, like aperture, shutter speed, ISO, date and time of exposure, etc. was also substituted with that of the white wall photo.
I informed these results to Nikon hoping that this would prove to them that the problem was a metadata problem only and that they would be able to narrow down on that and provide me with information on which metadata had the problem and how to correct it. I had also determined that it wasn’t one of the standard metadata (by substituting only that part and trying to open the file, which failed), but one of Nikon’s custom, encrypted, binary information. My hopes were up for a while but Nikon quickly replied just affirming again that Picture Project would have caused the issue with that RAW file, that I should use ViewNX to do that and that to access the original metadata from the corrupt file I should download them from the SD card again using ViewNX. Finally they said, and I quote, “unfortunately if you have deleted the images from your card the data is lost”, so they could not do anything else for the matter.
I was quite disappointed with Nikon for that, but I proceeded with my quest, this time writing a script that would do the following:
- Extract the embedded JPEG from the corrupt RAW files
- Save the original non-Nikon-specific metadata from the corrupt RAW files to XMP files
- Overwrite all metadata of the corrupt RAW files with metadata from the good, “white wall” file
- Restore the original non-Nikon-specific metadata from the XMP files to the once-corrupt RAW files
- Restore the embedded JPEG to the once-corrupt RAW files
I am pleased to say that this solution worked quite well and both me and my friend are quite happy with it. Something was lost still, like the “as shot” white balance, but nothing that can’t be adjusted with Lightroom afterwards.
Download the script here. Use it at your own risk, and please be very careful when using it, it will modify your RAW files, so please back them up first. To run this script you need to have a folder that contains only your corrupt NEF files, a good NEF file with a photo of a white wall or a gray card called GOODFILE.NEF, this script, and the ExifTool executable needs to be on your path.
Thank you for everybody at Photo.Net who gave this problem a shot!