One Small Step for VHS Tapes (Hayou Tsurugi v2)

When Orphan released Hayou Tsurugi last year, I noted that "the tape had copy protection, which caused problems during the ripping process and may have contributed to audio sync issues in the encode." In another small step for improved analog ripping, that issue has been overcome, and Orphan is releasing a new, improved, and properly synced version of the show.

Since the Domesday Duplicator became available for ripping laserdiscs, and uncompressed capture was shown to be the best way to rip VHS tapes, progress on improving analog media capture has been slow. The Duplicator software (ld-decode) has continued to improve. A VHS analog, called VHS-decode, is making progress, but it's not ready for prime time. AI-based image cleanup has not panned out; AI can't cope (yet) with blended frames and other imperfections in source material.

However, one option for improving VHS tape capture has been available for quite a while - the time-base corrector (TBC). A TBC eliminates the timing stuttering and playback artifacts introduced by the mechanical instability of the VHS playback hardware or by copy protection. It's not essential with many tapes, but it's absolutely necessary to accurately capture tapes protected by Macrovision's analog copy protection (ACP). Although not common in anime, Hayou Tsurugi used Macrovision ACP. As a result, Orphan's original capture had both video errors (washout) and audio sync problems.

Recently, our media guru in Japan was able to acquire a working TBC at a reasonable price. When he tried it on the Hayou Tsurugi tapes, he got a capture with proper audio sync and without the awful washout that had afflicted certain scenes. Old version:

New version:

While the audio sync issues were fairly consistent (around five frames), they were not constant, particularly in the second episode. This is fixed in the new release.

I won't repeat my description of the show from the previous blog entry. It's still a routine sword-and-sorcery fest. For the original release, Iri translated, ninjacloud timed, I edited and typeset (very little), and Topper3000, Uchuu, VigorousJammer, and ImAWasteOfHair QCed. For this release, I tweaked the timing, and Nemesis did a release check. If you downloaded the first version, you should replace it with this one. You can get the show from the usual torrent site or from IRC bot Orphan|Arutha in channels #nibl or #news in irc.rizon.net.

The TBC won't set off a flood of new versions, because unlike previous improvements - the Domesday Duplicator and uncompressed tape capture - it applies to only a small amount of source material. Bakumatsu no Spasibo was also Macrovision protected, but it has other issues, so a new release isn't immanent. VHS-decode will be the next big milestone in the pursuit of analog perfection.

Aspect Ratios

In this blog entry, I am venturing into deep water without a life jacket, because I'm not an encoder, and I don't play one on TV. To keep things simple, I'm only going to discuss US standards (NTSC). European standards (PAL and Secam) only make matters worse.

The aspect ratio is the ratio between the number of pixels horizontally and the number of pixels vertically. This can be expressed as the actual number of pixels (e.g., 1280 x 720), as a ratio of whole numbers (e.g, 16:9), or as a ratio against one (e.g., 1.77:1). The last one is only approximate, because the decimal digits repeat indefinitely.

Up until the 1950s, aspect ratios were fixed at 4:3 (1.33:1), the so-called Academy Ratio, because everything was done on 16mm or 35mm film stock, and the frame size was 1.33:1. In the early 1950s, the movie industry created a raft of wide-screen formats - Cinemascope, VistaVision, Panavision, etc - at a variety of aspect ratios, including 2.40:1, 1.85:1, and 1.66:1. Some of these were done by anamorphically stretching the 35mm film stock, using special lenses; others used wider film stock, such as 65mm or 70mm. TV production retained 1.33:1 until this century. Wide-screen content such as movies had to be presented by one of two techniques - letterboxing, which showed the wide-screen image on a subset of the TV screen, with black bars and top and bottom; or pan-and-scan, which moved the viewpoint around to show the "essential" part of wide-screen image on the full TV screen.

The Japanese analog home-media formats - VHS tape and laserdiscs - were invented when all TVs were 1.33:1, and they basically copied the NTSC standard. Thus, it was fairly common for VHS tapes of wide-screen movies to be offered in two versions, letterboxed and pan-and-scan. (Laserdiscs were more typically letterboxed, because TV screens were bigger by then.) For fansubbers, then, there is (or should be) no issue about the aspect ratio of VHS and laserdisc-sourced material - it's 4:3 (1.33:1) prior to any cropping or upscaling.

Starting in 1984, Dr. Kerns Powers of the SMPTE Working Group on High-Definition Electronic Production proposed 16:9 (1.77:1) as a suitable wide-screen compromise between the US de-facto standard (1.85:1), the European standard (1.66:1), and the older Academy ratio (1.33:1). Driven by the EU, the proposal gained traction and became the standard for HDTV broadcasts, at 1280 x 720 or 1920 x 1080. And in the mid-1990s, the proposal was sufficiently far advanced that it was supported by the spanking new DVD standard.

DVDs can be considered an intermediate format, looking backward to the still-predominant 4:3 standard and forward to the forthcoming 16:9 standard. The actual DVD resolution was a compromise: 720 x 480 (1.5:1). Metadata on the DVD would tell a player whether to "stretch" the resolution anamorphically to 16:9 (1.77:1) or "squeeze" it to 4:3 (1.33:1). When TVs evolved to a 16:9 aspect ratio, the players had to handle four cases:

1. 16:9 display on a 16:9 TV.
2. 16:9 display on a 4:3 TV.
3. 4:3 display on a 16:9 TV.
4. 4:3 display on a 4:3 TV.

Blu-ray did away with the anamorphic mess. Blu-rays are always 16:9, and the content is letterboxed, pillarboxed, or cropped to fit. Modern web streams are the same.

However, DVDs had more than enough rope for fansubbers to hang themselves and six of their best friends too. A common mistake was to take the 720 x 480 (3:2) ratio as correct. The caused wide-screen material to look squashed and Academy ratio material to look stretched. This can be fixed pretty easily in players or by creating a Matroshka mux with the proper aspect ratio set.

Another mistake was to assume that the wide-screen ratio was 1.85:1. This was based on an unconscious assumption about the universal acceptance of US standards, as well as a set of arguments about pixel shape, but in fact, it's not what the standard says. A different justification was that 1.85:1 movies on DVD played on a wide-screen TV without letterboxing. In fact, there is letterboxing, but it falls in the overscan area and can't be seen. If the TV can turn off overscan, the letterboxing becomes visible, and a bit more of the frame appears at the edges. Encoding at 1.85:1 made anime looked subtly stretched. It wasn't as bad as the previous case - it's only 5% - but it's annoying if you're familiar with a DTV or HDTV broadcast of the same material.

Fansub encoders found a clever way out of this mess: they encoded at 720 x 480 and then set the display aspect ratio as part of Matroshka mux. That way, it was Not Their Problem. However, it created headaches for other members of the team, particularly the stylist and typesetter, who had to account for the stretch or squeeze in their work. Anamorphic encodes have been the bane of my existence as a typesetter.

At this point, you've probably gone "tl;dr" and are wondering what this has to do with Orphan, which works mostly from analog media anyway. Analog media are 4:3, so no headaches, right? Wrong. The DVD format has created problems in the handling of analog media, because analog captures were and are often distributed in DVD ISO format. That has led to mistakes about the origins of the material ("it's a DVD") and how to process it.

• aarinfantasy's groundbreaking release of Zetsuai 1989 used an ISO image of a laserdisc capture and assumed the dimensions were correct. After cropping the pillarboxing, the encode was 700 x 480 - too wide - which actually made the ultra-narrow character designs more reasonable.
• Toy-RC's groundbreaking release of Call Me Tonight also used an ISO image of a laserdisc capture. The encode used vertical expansion to correct the ratio (720 x 540) and is thus technically a 12.5% upscale.

More serious issues have arisen with letterboxed widescreen releases on laserdisc. The obvious way to deal with letterboxed laserdiscs is to encode them at 640 x 480, letterboxing and all. That's what Erik of Piyo Piyo Productions does routinely. I like the approach, because the letterboxing provides a nice place to put the subtitles without infringing on the image. Another simple approach is to crop the image exactly as shown, to 640 x whatever. That too is faithful to the original source.

However, sometimes encoders want to create a normal 480p (vertical) resolution encode, which involves upscaling. That poses a problem: what should the horizontal resolution be? Again, there's a simple answer: measure the original image to get the aspect ratio and upscale accordingly. However, sometimes the groups plump for the US widescreen standard (1.85:1), on the theory that the laserdisc must have been scaled wrong. However, that's incorrect. Japan did not use the US widescreen ratio of 1.85:1 but rather the "VistaVision" (aka European "flat") ratio of 5:3 (1.66:1).  It later showed up in Japan's first HDTV standard (Hi-Vision).

The result is that upscaled versions of widescreen Japanese movies sourced from laserdiscs can have incorrect aspect ratios. Orphan's first version of Hashire, Melos! was 1.85:1, because the encoder thought that's what widescreen movies had to be. As a result, it's too wide: more than 10% stretched horizontally. This can be seen by comparing the first version, done with "anamorphic stretch":

 

versus the second, which was done at standard laserdisc resolution with letterboxing intact:

It's subtle, but the faces in the first version are too wide.

A similar problem occurs in Toy-RC's B.B. Fish. It is encoded (non-anamorphically) at 1.77:1 (16:9), as though it were a DVD. Looking at an untouched frame of the laserdisc output, with letterboxing intact, shows that it's actually 1.66:1 (15:9). The laserdisc jacket even says "Vista version." Orphan's first attempt at encoding the widescreen release of Hashire! Ookami Shiroi was also too wide, as a comparison with the 1.33:1 (4:3) US dub clearly showed. The widescreen version is actually 1.66:1 (15:9).

By now, your eyes have probably glazed over (mine have). The bottom line is twofold:

• A DVD ISO does not mean a DVD source. The DVD ISO format is just a container. The encode has to be aware of the actual source media.
• A Japanese media property mastered in the 70s and 80s will conform to the Japanese standards of the period, not to modern standards.

Being cognizant of these two observations will save a world of problems in encoding analog media.

Technology Continues to March On

In previous posts, I've written about advances in technology that have enabled Orphan, and other fansub teams, to get more out of analog sources. These included:

• The Domesday Duplicator, which taps into a laserdisc player's laser and records the raw RF data, bypassing the player's aging analog electronics. The RF information is then processed in software.
• An uncompressed VHS capture setup, which eliminates compression artifacts from VHS capture cards and allows software to deal with VHS issues.

These improvements have produced noticeably better raws from analog sources and, as a result, a raft of re-issues of old projects.

Technology does not stand still, though, and improvements happen almost continuously, particularly in software processing. For example, ld-decode, the software for the Domesday Duplicator, is now on its sixth revision and has improved in every aspect. A continuous "diet" of rips from flawed laserdiscs has made the software much more robust against bit rot and other physical problems. The latest development allows direct decoding to YUV colorspace instead of RGB, skipping the RGB -> YUV conversion that was previously required. The software is also much faster. However, processing of CX-compressed analog audio still requires improvement.

Another fascinating development is the application of AI to anime. For decades, AI was a pipedream, and even today, it requires massive computing power. However, today's gaming computers are monsters: 8 to 32 cores, 16 to 64 threads, all the memory you can eat, and ultra-fast NVMe-based mass storage. Open source AI software is available. One of Orphan's encoders has been training an AI on hand-drawn anime from the 80s and 90s, and he is seeing substantial image quality improvements. On a commercial scale, AI is being used to produce better upscales of standard-definitions sources to HD.

Of course, there are severe limits; it's still early days. The underlying sources have to be free of mastering issues such as blended frames and interlacing issues. The technology can do cleanup when none is required or desired, so manual inspection of results remains a necessity. But the promise is there. Someday, it may be possible to undo the problems in even badly damaged sources.

One development remains tantalizingly out of reach: a "Domesday Duplicator" for VHS tapes. This would tap into the output from the helical scan playback head and then do all the decoding in software. It's a very complicated problem in both hardware and software, but it would get around some of the limitations in uncompressed capture, such as timing issues that require a Time Base Corrector. Hardware that would allow the tape to be "oversampled" as it is read might help even more.

Do all these improvements presage yet another round of Orphan re-releases? Thankfully, no. The initial jump in video quality from years-old Internet raw to Domesday Duplicator capture or uncompressed VHS capture was massive; just look at Meisou ou Border (VHS) or Al Caral no Isan (laserdisc) if you need convincing. Further improvements are more incremental, and they'll be applied to new releases, not old ones. Unless, of course, AI can someday undo all the frame blending in Amon Saga or Every Day Is Sunday... just kidding.

Choosing Sources

Orphan does many of its show from analog sources - laserdisc by preference, VHS if need be. This reflects the team's interest in the "anime that time forgot" rather than nostalgia for old technology. But occasionally, as with the recent Amon Saga, we'll use an analog source (in this case, laserdisc) instead of an available digital source (in this case, DVD). This raises a valid question: why? Aren't modern digital sources always better?

For modern anime, which are created on computers, the answer is always yes. For back catalog shows, particularly those originating on film stock, the answer is mostly yes. Digital sources have error correction; analog sources do not. Digital sources can be replayed and will always yield the same bit stream; analog sources will not. However, there are cases where an analog source is a valid alternative to a digital source, or even a preferable alternative.

Case 1 - the digital source has been altered in some irremediable way. A notorious example of this is the Maze TV series. The DVDs ares censored; the laserdiscs are not. Further, the series masters are apparently lost, so the laserdiscs are the only possible source for the original version. Another example is Legend of the Galactic Heroes (the 110 episode OVA version). Scenes in episodes 06, 07, 15, 18, 20, 21, 23, 25, 26, 27, 28, 31, 34, 35, 37, 38, 39, 40, 42, 43, 44, 46, 47, 48, 50, 52, 56 and 92 were redrawn for the digital releases. Is the new animation better or worse? I don't know (reviewers have mixed opinions), but it's different.

Case 2 - the digital source has been carelessly mastered. This is more controversial, because the appearance of a source is in the eye of the beholder. In the early days of DVDs, mastering was often terrible. For example, the Yawara! DVDs looked like VHS transfers, and the Laughing Salesman DVDs were awful too. Erik of Piyo Piyo Productions ripped Amon Saga despite the availability of a DVD (something he almost never does) because prevailing opinion said the DVD was terrible. Members of the Orphan team prefer the laserdiscs of Project A-ko to the DVDs and, apparently, so does Discotek.

Case 3 - the digital source is unaffordable. Yes, economics play a role, especially in a volunteer hobby like fansubbing. Some DVDs were produced in small quantities and have become collector's items. For example, Heart Cocktail was released on six laserdiscs; the first four were released on DVD. When it's for sale, which is rarely, the second-hand DVD sells for hundreds of dollars. So if Orphan ever does sub Heart Cocktail, it will be from laserdiscs. Cost considerations may also dictate the use of a VHS source rather than a laserdisc source. The Aoki Honoo laserdisc, and the first volume of Kasei Yakyoku, when they come up for sale, sell for exceedingly high prices, so Orphan used VHS tapes for those shows.

 

I maintain a list of "problematic sources" used in prior Orphan releases. The team has standing searches on Japanese second-hand sites to find better source material, but most of the searches have been fruitless, even over several years. The best hope may be that the properties show up on a streaming site, either remastered or with a good transfer, but that's unpredictable. The ownership rights for old shows can be very murky, and the masters may no longer exist. Still, you never know.

Imperfect Raws

Streaming services in Japan are republishing classic anime series and movies in record quantities, often in high definition, sometimes remastered. That's good news for fans of old anime, because many of these shows have been unavailable or available only on imperfect sources like VHS tapes, laserdiscs, or hastily mastered DVDs. For example, the streaming source for the original Laughing Salesman looks vastly better than the DVD box set.

However, the streaming raws can also have problems. Sometimes, the high-definition stream is just an upscale of an old laserdisc or DVD, with no attempt at remastering. They result looks grainy and crude, with thick, jagged lines and blotched textures. Sometimes, the streaming service skimps on bandwidth. This introduces blocking and other artifacts. And sometimes, the streaming raw has encoding mistakes - incorrect frame rates (29.97 fps instead of 23.976 fps) or unnecessary black borders.

Orphan has a pile of high-definition raws for shows it has already released, but many of them have issues that introduce problems in the subtitling process:

• Incorrect frame rate. The high-definition Sangokushi movie streams are at 29.97 fps. This can't be correct: the originals were movies and must have been created at 23.976 fps. The incorrect frame rate not only bloats the encode, it also wrecks all the motion-tracked typesetting. Because the Sangokushi movies required months of typesetting, this is a big deal, at least for me as the typesettter. So far, no one has been willing to re-encode them back to the proper frame rate.
• Lack of cropping. The high-definition raws for the Nine OVAs are at the correct frame rate but are not cropped. The first OVA has been released as though it were a wide-screen movie, with horizontal black bars to achieve the movie aspect ratio of 1.85:1 instead of the HD standard of 1.77:1. The second and third OVAs are square but have been padded with vertical black bars to widen the 1440 x 1080 resolution to 1920 x 1080. In both cases, this wrecks the typesetting from the laserdiscs, which are 4:3.
• Ugly upscaling. The original high-definition releases of all the 60's Toei animation films appear to be simple upscales of DVD sources. They look terrible at 1080p and need to be downscaled to 720p to be tolerable, as was done for Wan Wan Chuushingura. (Toei is now remastering some of its best known titles, like Horus and Hakujaden; we'll see if they keep going.)

So I have a question to my readers (all three of you): would you like to see high-definition releases of our past shows, even if the raws have black bars or look terrible? Or should Orphan hold out for properly encoded/cropped raws? Of course, if we had a few more encoders, this would not be an issue...

Translators, Encoders Wanted! (aka Status)

I've mentioned before that Orphan's three-year run of frequent releases has come to an end, due to lack of resources, but I wanted to show how dire the situation actually is:

• Waiting on encoding (VHS lossless captures): Kentauros no Densetsu, Mahou Tsukai Tai vs Shamanic Princess (v2), Mother Saigo no Shoujo Eve (v2), Oedo ga Nemurenai (v2), Ohoshi-sama no Rail, Okama Hakusho, Shin Dousei Jidai- Hawaiian Breeze, Tottoi: Secret of the Seals, Yamazaki Long Distance Call (v2)
• Waiting on encoding (LD lossless captures): Ichigatsu wa Christmas, Super Real Mahjong
• Waiting on encoding (DVD ISOs): Amatsuki (v2), Gokuu no Daibouken, Haruka Naru Toki no Naka de 2 (v2), Mars OVA, One Pound Gospel, Hakujaden, Sanada 10
• Waiting on encoding (net streams): Sangokushi movies HD, Saiyuuki HD
• Waiting on translation checking: Akai Akiba Blue Sonnet, Bakumatsu no Spasibo, Genji Part 1 (OVA), Karuizawa Syndrome, Mother Saigo no Shoujo Eve, Nayuta
• Waiting on translation: Bocchan, DAYS OAD (2017), Magma Taishi, Mellow, Raiyantsuurii no Uta, Sanada 10 10-12, Sugata Sanshiro, Tengai Makyou, Tezuka Osamu Kyoto Animation Theater, Ziggy Soreyuke! R&R Band

The situation is so bad that the team has stopped buying additional video media. We've also stopped reprocessing previously purchased media through the lossless capture systems. And I'm not even counting the backlog of laserdiscs that Piyo Piyo Productions has already encoded and released.

In view of the backlog, it's fair to ask what projects are actually progressing. The answer is, not a lot:

• Hashire Melos v2. A new laserdisc encode. At release check.
• AWOL Compression Remix. Yes, at long last, scripts for the remix have been extracted from the exceedingly rare English VHS subs of the original series. They could use a thorough translation check, but beggars can't be choosers. Episode 1-2 in RC, episodes 3-4 in QC.
• Chameleon ep3. In editing.
• Tales from the Old Testament. Episodes 1-6 released; episodes 7+ tbd.

Beyond that, the team is limited to shows with existing scripts and existing encodes, for example, BD versions of the Tezuka Osamu Animerama movies.

So if you're an encoder who relishes a challenge (almost all of Orphan's sources present challenges) or a translator who likes to work on shows that differ from today's cookie-cutter anime, please give me a shout. I'm available on IRC almost all the time.

Appendix: A Sampler of Encoding Challenges

• VHS lossless captures. Soft deinterlacing, and then all the usual VHS problems: cropping, blended frames, halos, rainbows, etc. Every capture presents unique issues.
• LD lossless captures. Soft deinterlacing, and then all the usual LD problems: cropping, halos, rainbows, chroma blending, etc. Every capture presents unique issues.
• Haruka Naru Toki no Naka de 2 (DVD ISO). Progressive, but very strange color balance.
• Sanada 10 (DVD ISO). Mostly progressive, but interlaced fades and 30 fps credits. A formula for tackling the episodes is available.
• Sangokushi HD movies. Wrong frame rate (29.97 fps instead of 23.976 fps).
• Saiyuuki HD movie. Clearly an upscale. Needs to be downscaled to an appropriate size and cleaned up, a la Wan Wan Chuushingura.

[Updated 28-Aug-2019]

No Safe Harbor

In the bad old days, like the mid-00s, getting subtitled anime to play properly was an exercise in frustration. The new Matroska (MKV) format was barely supported. There were many players, each with its own idiosyncrasies. Every codec was standalone and had to be installed separately. Codecs often stole each other's identities or registry entries, producing chaos. It was every subber for him/herself, and God against all.

Then a group of fansubbers got fed up and decided to set a de facto standard. They created the mischievously-named Combined Community Codec Pack (CCCP, the Cyrillic acronym for the USSR) to dictate how fansubs should be decoded. They assembled a player, a set of filters, and a subtitle renderer into a coherent package, and it basically all worked. Peace and stability returned, to Windows at least. But in 2016, the CCCP project disbanded.

Personally, I barely noticed. True, the last CCCP didn't handle PGS subs correctly; I didn't care much. PGS subs are for (lazy) BD rips, not fansubs. However, Commie recently released its batch of Chihayafuru S1, and it didn't play. When I complained about this, I was peremptorily told that CCCP was "really old" (3 1/2 years!), and I should upgrade to the latest and greatest version of MPC-HC or MPC-BE or mplayer or some other playback gadget. So I did. Chihayafuru played correctly. PGS subs played correctly. Victory!

...until I started my release check on the second Sangokushi movie. To my horror, signs that I had set in Aegisub and that had played correctly before no longer looked right. Correct in VLC and CCCP; incorrect in MPC-HC and -BE. After some digging, I traced it to a bug (feature?) in the internal subtitle renderer in MPC-HC. Sangokushi is encoded anamorphically. The correct display aspect ratio is set during Matroska muxing. The subtitle renderer is ignoring the Matroska aspect ratio and using the encode aspect ratio. CCCP overrode the default MPC-HC renderer setting and forced use of an external package called vsfilter. Viewers using later versions of MPC-HC/BE must do that as well.

So, if you have abandoned CCCP and want to use MPC-HC/BE, then you  must first install xy-vsfilter separately (look it up on Google). Then, in MPC-HC/BE, override the default internal subtitle renderer and choose xyfilter instead. The option to do this is in different places, so you'll need to poke around. Once you've changed the subtitle renderer, Orphan's anamorphically encoded videos will play properly. Chaos tamed, at least temporarily, and at least on Windows. Everyone else will have to use VLC unless/until the internal renderer starts to respect the Matroska aspect ratio.

In the longer term, Orphan will stop making anamorphic encodes. They're only an issue for DVDs anyway, not VHS tapes, laserdiscs, or Blu-rays. They're an incredible PITA for typesetting, as well as a fruitful source of playback errors. In the meantime, here's a list of affected releases:

• Alice in Dreamland
• Bronze: Zetsuai Since 1989
• Cosprayers
• Code Breakers OVAs
• DAYS OVAs
• Hidamari no Ki
• Okane ga nai!
• Sangokushi movie 1
• Seikima II: Humane Society
• Stop!! Hibari-kun!
• Ultra Nyan 2
• Wild 7
• Yume Tsukai

It is, unfortunately, very unlikely that we'll ever re-encode these shows. We apologize for the inconvenience.

The Relentless Pursuit of (Analog) Perfection

As readers of my blog (or my comments on BakaBT) know, I have been highly skeptical of the seemingly endless "one-upping" of digital media encodes. Each new release claims to have better this or less that, when all I see is ever increasing bloat. Thus, it gives me a queasy feeling to find that Orphan may be doing something similar with its analog encodes, with v2s and maybe v3s and v4s.

Some of this is understandable. Orphan started off using Internet sources for almost everything. As the team got access to original media, it was natural to encode the new media and release a new version (for example, Kage, Kakyuusei 1995, A Girl, Tsuki ga Noboru made ni). Sometimes, a title was released in higher definition on streaming media or even Blu-ray (for example, Dallos, I am Son Goku, Techno Police 21C, the Animerama titles). However, technology improvements in capturing and encoding analog sources provide endless opportunities for releasing better raws - the encoding equivalent of Editor's Remorse.

For Orphan's laserdisc projects, the evolution has been from good lossy capture to fully lossless capture of the RF output - an enormous leap forward.

• Good lossy capture was done with a high-quality, near professional setup.
• Encoding has been improved by using color calibration discs to set color levels.
• The Domesday Duplicator allows for capture of the RF output - as close to an archival level transcription as possible.

The first results from the Duplicator are pretty impressive, although the processing software is still rough around the edges. There are now Duplicator setups in both the US and Japan.

For Orphan's VHS projects, the evolution has been from bad lossy capture to better lossy capture to lossless capture:

• Bad lossy capture was done with an S-VHS recorder into an el-cheapo USB capture device. It frequently lost sync and could not cope with copy protection.
• Better lossy capture was done with a D-VHS recorder that produced a digital output on Firewire. However, the built-in compression seemed to play havoc with scene transitions, which appeared to smear over multiple frames.
• Fully lossless capture is done with an ancient All in Wonder card on an equally ancient XP system. It throws the entire burden of cleaning up the capture on software, which has a much wider array of filters and fixers available than the hardware would.

The first trials of lossless capture are out for encoding now. If it produces significant results, we could be redoing almost all of our VHS releases.

So where does the possibility (or threat) of endless versions come from? The Duplicator hardware appears to be the definitive answer for capturing laserdiscs and is quite stable. However, the Duplicator software has endless possibilities for improvement. Some of the roadmap is already known - decoding digital audio, decoding analog audio with noise reduction, implementing a 3D-comb filter, implementing IVTC. After that, who knows? Oil companies have reprocessed their seismic shot data from the 1950s almost every decade, as computers and algorithms have improved, and they keep finding new features of interest. I think (or fear) this may be this case with the Duplicator laserdisc captures.

For VHS tapes, the promise (or threat) is something like the Duplicator for VHS tapes - that is, a device that would capture the output of the helical scan read head directly. Because tape reading is a contact sport, the quality of the capture would still depend on the quality of the deck (and the tape, of course). However, this technique would bypass the analog electronics in the VHS deck and allow for advanced signal processing techniques to be applied. This has already been demonstrated with computer tapes, where oversampling using modern read heads and application of advanced signal processing techniques has allowed data to be recovered despite print through, tape stretch, and other physical blemishes.

For me and the other members of the Orphan team, the dilemma posed by all these opportunities is where to draw the line and make a new release. The laserdisc Duplicator software is almost there - all it really needs is reasonable audio decoding. Old captures that were done with the wrong frame rates (like Tomoe ga Yuko) or uncalibrated color compensation are prime candidates for new captures and encodes. The VHS situation is more fluid. We still don't know whether lossless capture really improves the results enough to matter. It's certainly better than the Internet VHS raws we've used (like Meisou-ou Border).

So I must beg our faithful followers - all ten of you - for forbearance as Orphan revisits its sins of the past and tries to make amends. The digital side of the equation is straightforward. The team now has DVD ISOs for Amatsuki (the DmzJ raws are at the wrong frame rate) and Harukanaru Toki no Naka de 2 (the aF raws are corrupt and R2E to boot), and at some point we'll make new encodes and re-release. The analog side, though... the sky's the limit. Unfortunately.

Media: The State of Play

Orphan gets its original media from both digital and analog sources. Digital media - DVDs, Blu-rays, streaming - are straightforward but rare for our sorts of shows. Most of our sources are analog - VHS tapes or LaserDiscs. Both present knotty problems. This blog entry provides a snapshot of the "state of play" in Orphan's media processing.

LaserDiscs

Until quite recently, all our LaserDiscs came were ripped by private collectors. The most prolific and helpful has been Erik of Piyo Piyo Productions. Erik has an extensive collection, an excellent capture capability, and deep knowledge of the encoding issues in LaserDiscs. ics- has also contributed some captures, which M74 has encoded. And finally, an anonymous friend of a friend has provided a few rare discs.

All the LaserDiscs collectors are at the mercy of the analog electronics in their players and in their capture cards. This can introduce some pretty ugly problems. LaserDisc players have been out of production for twenty years, and their electronics are, well, old. Most capture cards introduce some form of compression, which complicates software filtering of the many artifacts in the sources. Lossless capture has required insanely expensive (professional) equipment  or insanely arcane (Windows XP era) hardware and software... until now.

The Domesday86 project, which I've written about before, seeks to bypass the main sources of introduced error in LaserDisc rips by capturing the RF output of the laser directly and post-processing the results entirely in software. This sounds so promising that Orphan has created a Domesday Duplicator setup in Japan - LaserDisc player, special hardware, and software. It's been a long, frustrating, and expensive saga. Japanese anime LaserDiscs had many attributes that the Domesday project team had never seen. It's taken months to get the decoding software into usable shape. With the release of v4 of ld-decode, the project is just about there and produces usable lossless captures. ld-decode still can't handle the digital audio tracks, but those can be captured directly.

The first LaserDisc to be processed through Orphan's Domesday Duplicator will be Boyfriend. The Duplicator lossless decode still needs to be encoded down to reasonable size, but at least that's a tractable problem. Further, as ld-decode improves, the RF captures can be processed again, if the improvement in quality is sufficient.

Props to Intrepid for championing the project, assembling the hardware, and doing all the hard work of capturing and decoding numerous test cases both for Orphan and for the ld-decode team. The team provided financial support for the equipment.

VHS Tapes

Tapes have been an even more frustrating story than LaserDiscs. The initial attempt at capture in Japan involved an inexpensive USB capture device and a standard S-VHS deck. The results were hit or miss, mostly miss, with dropped frames and video/audio sync problems galore. The cause of the problems varied, but one might be the presence of copy protection on some tapes. LaserDiscs couldn't be duplicated easily and had no copy protection. VHS tapes could be duped (with significant quality loss, of course), and copy protection was sometimes employed.

The next attempt was to buy a much better deck, a D-VHS deck with a built-in Time Base Correction (TBC) and digital (Firewire) output. This produced better results, but the digital output had MPEG2 compression that couldn't be bypassed. The compression tended to blend frames and muck up deinterlacing. The TBC handles only part of the dropped frame and lost sync problems, so the results were still hit or miss.

The current plan is to do lossless capture of the standard S-video output and try to clean up the results in software. This requires an ancient, XP-era AGP PC and specific versions of old ATI All in Wonder cards. The first card Intrepid bought was damaged in shipment and didn't work. The second one lost its heat sink within five minutes. Fortunately, the card seems to work once the heat sink was reattached. An external TBC might still be needed.

Ultimately, we'd like to see something like a Domesday-for-tapes: a way to capture the output of the helical scan read head directly, bypassing everything else in the VHS deck. However, that's a complex problem, both in hardware and in software, and it's not clear that VHS decks have a maintenance access point for capturing the RF data.

Intrepid is once again doing all the heavy lifting on this project, with financial support from the team and its anonymous benefactor.

Tl;dr

Orphan's Domesday Duplicator for LaserDiscs is up and running, and the first encodes will show up in projects this year. VHS tapes remain a work-in-progress, and the light at the end of the tunnel is still fairly distant.

Raws from the Abyss

The team members in Orphan tend to collect rarities on analog media, even if there's no real intent of ever subbing them. So here's another raw release, the 1990 movie Tenjou Hen: Utsu no Miko. Not much is known about this movie, but it appears to be a digest or compilation of some or all of the 13-part OVA series, Tenjou Hen: Utsu no Miko. The timing is a little odd; the movie was released before the first OVA. Furthermore, it includes the previews for OVAs 6 through 13. So it may be a "teaser" for the OVA series or a complete summary. Adventurous fansubbers are welcome to try matching up scenes in the movie with Ukyuu's subs of the OVAs.

Tenjou Hen: Utsu no Miko was ripped from a VHS tape, as a test case for Orphan's New and Improved™ VHS transcription capability in Japan. Our media guy had been having trouble with video/audio synchronization between his S-VHS deck and his capture card (something to do with "TBC"). As an alternative, he bought a used D-VHS (digital VHS) deck to see if it worked better. D-VHS was the dying gasp of VHS technology; it supported tapes with digital encoding. Unfortunately, D-VHS came out after DVDs were established, and the format never took off. But the deck did have one redeeming feature: digital outputs. It put out MPEG-2 over Firewire and PCM audio over an optical cable. This removed the capture card completely and produced a better image, usually, then the S-video output. It also allowed for high-resolution audio. Would you like FLAC with your VHS rip, sir? I wouldn't, but it's now a possibility.

This new setup isn't a panacea. VHS tapes are still analog, and most of them are quite old. Fancy electronics can't overcome the intrinsic limits of the medium. The tapes are interlaced, and the output compression makes software-based correction problematic. Vertical pans, in particular, look pretty bad. But unless Tenjou Hen: Utsu no Miko is given a digital makeover, this raw is about as good as it's going to get.

Our media guy ripped the video, and M74 encoded it. The new setup was funded by a generous donation from a benefactor who wishes to remain anonymous. The entire team is grateful for his continuing support and is trying very hard not to blow their entire Christmas gift allowance on old tapes.

Domesday

The Orphan team has been expending a lot (too much) of its free cash buying analog media (VHS tapes and laserdiscs) in Japan and getting them transcribed to digital form. We really feel we're in a race against time. Magnetic tapes and optical disks have a finite lifespan. The phenomenon of laserdisc "bit rot" is well documented. Worse yet, Erik (of Piyo Piyo Productions) has had discs come apart during playback. The clock truly is ticking.

Up until now, the best way of transcribing a laserdisc has been to play it back on a high-quality player, capture the video with little or no compression, and do all the filtering, color correction, deinterlacing, etc. in software. The old ATI All-in-Wonder cards were ideal for this and are still highly prized by specialists. However, most of those cards only run under XP, and it's becoming harder to find and maintain an XP system, not to mention exceedingly dangerous to plug one into a network. USB capture devices usually have fairly crappy electronics. And no matter how the signal is captured, it has already been processed and distorted by the ancient (and often crappy) electronics in the laserdisc player itself.

Fortunately, Orphan and Piyo Piyo have not been the only groups pondering how to capture and preserve laserdiscs. A far more august body, the BBC, has been running a project to recreate the experience of its Domesday laserdiscs from the 1980s. An offshoot of that, the Domesday Duplicator project, is taking a novel approach to the problem of electronics distortion in the capture chain by removing most of the consumer electronics from the path. Instead, the laserdisc player's laser signal is captured as early as possible, while it is still RF. Direct RF sampling allows all information on the laserdisc to be duplicated (unlike conventional RGB sampling of the video output). The sampled RF is stored in a computer and then post-processed by highly sophisticated modern signal processing software. This results in images that are much clearer than anything the laserdisc player can produce on its own. (There are some examples on the Domesday duplicator site.)

[Side note: this is very similar to how the computer hobbyist community salvaged old 7-track magnetic tapes. Old magtape drives are difficult to find and harder to get working. The tapes themselves suffer from flaking of the magnetic media and also from "print through" (distortion of magnetization through exposure to the bits on the surrounding layers). To deal with this, the hobbyists mounted a more modern 36-track tape head on a board, with tape hubs controlled by precision stepper motors. The tape was stepped across the head in precise increments and oversampled both horizontally and vertically. The captured data was post-processed by modern software that applied not only local signal processing techniques but spatial compensation for data that would have been in adjacent layers, in order to account for print-through effects.]

After much agonizing (and searching of threadbare wallets), Orphan has decided to try creating a Domesday Duplicator capture capability in Japan. Up until now, we've had to ship laserdiscs to the US or elsewhere for transcription: a hazardous and expensive process. With an in-country capture capability, we can purchase used laserdiscs in Japan and have them shipped inside the country at much lower cost. And getting better lossless rips is an added bonus. We've already bought the laserdisc player, a reconditioned top-end unit. The next step is to buy the Domesday duplicator electronics. The signal processing software is, fortunately, Open Source and free.

As you can imagine, all this has put a serious financial drain on the team. Orphan has no fund-raising mechanisms: no ads on this website, no referrals to Amazon, nothing.There's no donation button anywhere. An anonymous benefactor helped buy the Hidamari no Ki DVD set, but that was a one-off. There's simply no cash to buy the duplicator itself. So if our fans (all six of you) are feeling generous and would like to help with this project, please let me know.

Bloat

This blog has mostly been descriptive, providing introductions to Orphan's releases. The thought and opinion pieces that peppered the early years of the blog have by and large disappeared. It's time to tick some people off again, so here's a column on encode bloat.

Orphan's offerings have been growing in size lately. Techno Police 21C (BD 1080p) was 6GB in size. Kindaichi Shounen no Jikenbo movie 1 (web 720p) was also 6GB in size. The Usagi Drop live action movie (BD 720p) was almost 8GB in size. This is a far cry from the ~300MB for Orphan's typical 30 minute 480p DVD or laserdisc encode.

However, the size of our encodes pales in comparison to what I see elsewhere. On BakaBT, the first Urusei Yatsura movie (BD 1080p) comes in at 10.5GB. The first two Tenchi Muyo movies (BD 1080p) come in at 14.5GB and 15.7GB. The Night Is Short, Walk on Girl (BD 1080p) comes in at 12.4GB, and Redline (BD 1080p) came in at 16.2GB. And on and on.

Back in 2011, when Hi10P encoding was just coming in, and encoders were again claiming it would allow for superior quality at reduced file sizes, I predicted that any reductions in file size would be short-lived, and file sizes would continue to rise inexorably. So it has proved. Partly this is driven by the demise of 720p/AAC as an intermediate encoding point. These days, encoders want to do everything at 1080p with FLAC audio, even for TV broadcasts, whether justified or not (it usually isn't). But mostly this is driven by the use of higher bit rates in the name of preserving detail. Every grain must be reproduced, or some other encoder will release an allegedly better (and certainly bigger) encode.

I was baffled by this seven years ago, and I'm baffled by it now. To me, film grain is usually a defect of bad or aging film stock. I'm pretty sure that the original animators didn't have film grain in their cels and drawings, and they certainly don't in today's digital files (with rare exceptions, like the paper effect in this year's Angolmois). So when an encoder tells me that the bit rate must be higher to preserve the grain, I feel they're throwing bits at preserving the grime of time. We restore paintings (and films) for a reason - to remove the grime and let the original shine through.

I recognize that others - including several of the encoders in Orphan - disagree, and as a result, some of Orphan's encodes are rather large. But I have a hard and fast rule - no single Orphan file can be bigger than a DVD9 (7.8GB). That's why there's no 1080p version of the Usagi Drop live action movie. The 720p version is 7.8GB; a 1080p would be around 15GB - not happening.

My rule is as baffling to others as the size of present encodes is to me. Optical media are dead, they say. Just store those mammoth files on hard drives! That's fine for a small number of individual files, but with an anime collection exceeding 16TB (my nick is Collectr for a reason), a file server of requisite capacity is neither cheap nor reliable. Optical discs will last out my lifetime, and they don't require RAID6, system management, software updates, protection from viruses, or power.

I don't expect this diatribe to hold back the rising tide of big encodes. What I would like to see is a return of quality 720p/AAC encodes. I don't watch anime on a big screen TV; I watch it on my computer monitor and listen to it through my computer speakers. 720p is, for me, the sweet spot for comfortable viewing. AAC is just as good as FLAC for the tiny speakers nestled behind my monitor (or the reference monitors in my stereo system, for that matter). A few encoders (Atsui, DmonHiro) still work at 720p. Almost everyone else has moved up. More is better, right? Well, not always.

Of course, the next wave will probably be 4K encodes with 5.1 or 7.1 FLAC audio, at even bigger file sizes. So I'll be down at the shore, railing against the rising tide, for a loooong time.

Encoding Wars - The Return of the Revenge of the Sequel

Over the past decade, the video compression technology used in fansubs has changed several times - from DivX to XviD, from XviD to H.264 - as has the preferred container format - from AVI to OGM to MKV and MP4.  Each of these transitions has been accompanied by bitter complaints and a fair amount of inconvenience.  The transition to H.264 was very painful for viewers with slow computers.  When MKV first appeared, the comment section of BoxTorrents (now BakaBT) was filled with jeremiads against the new format.  Even the DivX/XviD transition has left scars - some encodes from that period will not play correctly, because they were done with beta versions of the codecs and contain encoding bugs.

Time has fixed most of these issues.  Today's multi-core PC's laugh off H.264.  The Combined Community Codec Project (CCCP) has provided a standardized playback kit for Windows, as well as a benchmark for testing the correctness of encodes.  Backward compatibility is quite good, barring only the occasional ancient, buggy encode.

What time hasn't fixed is the problem of encode bloat.  Most encoding advances have been introduced with the claim that they would reduce the size of the video.  When H.264 was introduced, that was true initially: H.264 episodes were smaller than their XviD counterparts.  But fairly quickly, they became equal size, and then larger: the codec improvements were used to preserve more detail rather than reduce file size.  File sizes grew, and then grew some more.  The transition to HD resolutions exacerbated the problem.

This is circuitous introduction (tl;dr) to the latest improvement in encoding - 10-bit H.264, sometimes called Hi10P.  I don't confess to understand the technical details, but the claim is that Hi10P reduces video encode size by 30%, at level quality.  Experiments with Hi10P began in the spring.  This summer, CCCP added formal support, and now the floodgates are open.

Compared to previous transitions, this one is pretty painless.  CCCP seems to work correctly "out of the box."  Many groups are moving to Hi10P in a sensible way: shows that were started in 8-bit technology are being finished that way.  And as with previous transitions, the improved compression technology is being used to reduce file size.

Would anyone care to take a bet on how long that will last?  My prediction is that by this time next year, Hi10P encodes will be as big as their 8-bit counterparts are today, and after that, the seemingly inexorable rise in file sizes will continue.  As a (near) senior citizen with so-so eyesight, this leaves me baffled.  I don't need to see every imperfection in the original film stock or cels.  Personally, I think this is a form of competition among encoders: mine is bigger than yours, so to speak.  The offers section of BakaBT is filled with new encodes claiming to produce the next minute improvement in visual quality (often at the expense of subtitle legibility, timing accuracy, or typesetting fluidity).  A few encoders buck this trend: for example, Atsui produces a fine balance between quality and size.  But for almost everyone else, it seems that bigger is better.

One final note on the stampede to Hi10P.  An encoding colleague says that at the moment, there are mutually canceling bugs in the encoding and decoding software that produce slightly "tinted" encodes and then correct it in playback (something about color-space translation).  As a result, when the software is corrected, these early Hi10P encodes will look slightly tinted.  I won't notice it of course, but I wonder if we'll see a rash of "Hi10Pv2" encodes at some point?