[Note: This message is from Marc Cull, Vice President CafePress and recently appointed GM of ImageKind.]
As you’ve noticed, Imagekind.com has been having a bit of a rough ride over the last few weeks. With regret that I haven’t been communicating clearly throughout the site issues, I’d like to now take the opportunity to share with you what happened, what we’ve been doing about it, and where things stand now.
So what happened?
In April, a critical file server failed irrecoverably. The failed server was the only repository for all of your artwork thumbnails (about 40 million of them), as well as some graphics and css files used on the website and artist galleries. Note: the failed file server was not the repository for your print-ready original files, just the thumbnails generated from them for display on the web.
What we’ve done about it
As soon as the file server failed, we began trying to revive it–with limited and sporadic success. We also immediately started regenerating all the thumbnails from the print-ready originals. We authored custom software and marshaled hundreds of high-capacity servers to grind through seven years of image processing in a matter of days.
Our initial thumbnail re-generating was powerful but imperfect, and although it restored the lion’s share of thumbnails correctly, you may have noticed some funky-looking or dark thumbnails as well. As we discover those thumbnail generation errors, we are fixing and re-running the thumbnails regenerator, and we will continue to do so until we eliminate all the remaining quirks. Please continue to contact email@example.com with screenshots if your thumbnails look wrong.
Rest assured, in the course of regenerating the thumbnails, we also upgraded the storage system we keep them on to premium grade, high-availability, multiple-redundancy servers, so this kind of failure absolutely will not happen again.
Similarly, we moved all the graphics and css files needed for properly formatted artist galleries to a new storage back end.
Making progress on both took longer than any of us wanted, but the team dropped everything, working nights and weekends until it was done. I regret not communicating more clearly what was going on at the time. We were just single-mindedly devoting all possible resources to fixing it.
Where we are now?
We believe we are nearly out of the woods in terms of the acute problems caused by the file server failure. As you help us discover remaining examples of not-quite-right thumbnails, we are able to spot-fix and regenerate. Uploading new art seems to be consistently working well now, from the desktop as well as from Flickr. The Bulk Uploader is still not working, and we expect to fix it within days. Most aspects of artist galleries seem to be working again as well, though there are certain graphics (for frames, etc) that are still missing and we will be following through on replacing as fast as we can. If your gallery is not formatting on the screen well, please contact firstname.lastname@example.org with a link and screenshot so we can address it as soon as possible.
There is one major remaining issue we need to work through carefully: As we regenerated the thumbnails, we discovered gaps in the archives of original, print-ready art. For some amount of artwork throughout the last seven years, thumbnails had been created although the original never made it to long-term storage. Those presence of the thumbnails on the site masked the fact that, in fact, we couldn’t print some pieces without asking the artist to re-upload the original. However, in losing the thumbnails and having to regenerate them from the print-ready source art, we also exposed the gaps in source art storage. That’s why there are still lingering broken thumbnails on the site. Those represent thumbnails we simply will not be able to regenerate until the original art is re-uploaded to the system.
We’re taking final measure of the missing original art pieces, and rapidly putting together individual communication to those affected, as well as an easy way to identify and re-upload the missing art. Stay tuned.
Where we go from here
I can’t tell you how much I appreciate the artists’ community behind ImageKind, and the hours and energy you’ve all invested. Where we go from here is up. I’m enthusiastic as hell about Imagekind and am excited about helping it soar. For those of you who may read into the behind-the-scenes corporate organization, I want to put your minds at ease if possible: Moving the operations and leadership of Imagekind to Corporate headquarters where it can benefit from more shared resources, and where its growth and daily operations can be the singular focus of the executive with the longest tenure at Cafepress–these all are signs that we’re very much behind Imagekind, and very committed to reinvesting in its growth.
I’ll wrap up by re-iterating that I understand and share your distress, both about the choppy service and the poor communication about it. Needless to say, we don’t take your investment for granted, and it’s our solemn intention to work our hardest every day to make ImageKind bigger and better for you.