Now, I'm _really_ not an emulator expert, but as I recall both snes9x and zsnes will read rom dumps from zip files. So it seems to be possible to do this without too much cooperation.
If archivers started including useful metadata in addition to the rom dumps, I'm sure the emulators would start to use it as well.
I kind of think that part of the problem is that there needs to be a bit of a network effect here for these kinds of .zip packaged formats to take off:
1) An emulator of sufficient popularity has to fully support it.
2) The format has to provide some kind of benefit to the user such that they're willing to convert/duplicate their entire collection.
3) Somebody has to make and distribute collections of packaged roms that include enough extra stuff to make them worthwhile.
To be honest, you might be in a good position w/r to #1 since your SNES emulator is now more or less the default in the community (at least for big multi emulation systems like retroarch).
For #2, I think the benefit really derives from what's in the package: manuals, savestates, cheats, soundtracks, playthroughs (that you can jump to in the middle of), patches, variants, meta-data (publication info, credits, etc.) and so on could all be really interesting to lots of people, but really would rely on the emulator to take advantage of in some way.
For #3, There's probably already enough information freely available to make a utility that would take somebody's existing ROM collection and yank down all the rest of the stuff from various websites all at once to build up an initial package of stuff.
I kind of almost just think you should just go ahead and support something like this and have it slowly percolate through the community. If it achieve some kind of network effect it'll slowly replace the other formats in a de-facto way.
It's not just "the ROM in it", it's "the ROMs in it". Single-rom games don't benefit from containers. The whole point of treating all games as containers is to allow multi-rom games to be free of metadata. 30% of NES games, 1% of SNES games, and most arcade games are multi-rom. In order for the emulator to tell contents apart without metadata, you have to have a naming scheme. So, for example, Galaxian.fc.zip would contain "program.rom" and "character.rom". Currently, those pieces are blobbed together with a header in front. Thus you can't open a .NES file in your file browser and see if it has a character rom, what size it is, what sha256 it has. This causes massive tedium with NES dump verification since you can't assume anything at face value. All parts and values are obscured and need extracted.
The other reason metadata is evil is that it's a royal PITA to maintain. It's far easier to update your emulator than thousands of individual files. Do not listen to anyone who tells you that we need to drag licensed games into the metadata mud for the sake of unlicensed games. They are pulling your chain, an unlicensed db for nes is entirely feasible, or even the mapper number applied to the extension ala "Cheetahmen II.224.fc.zip".
MAME did this right with a database and NEVER writing inside the container, but only because their hand was forced into it. So many arcade games have 3 or more roms, so they were never tempted to take shortcuts and blob shit together to avoid zip containers. Then they screwed up their own good fortune with some truly bizarre stuff. Google non-merged vs merged and behold threads and threads of confusion. Now weigh that confusion against the supposed benefit.
I very much agree. My vision was a ZIP file that when selected in an emulator, could be extracted for instant access, and any save RAM, cheat codes, save states, etc could be stored alongside the game in said folder. It would be bundled with an extensible markup description of the board, and optional pack-in content like box art and user manuals. For games stored on flash memory that could write to themselves (BS Memory, Neo Geo Pocket), this would also prevent the archived, pristene copy from being damaged. The folder would then act like a physical cartridge, and all your game saves would move along with it.
It ... was not a popular idea, to put it mildly. I further shot myself in the foot by trying to force the idea on people. You will never win people over with an approach like that.
Let's avoid mentioning names for obvious reasons, but unfortunately those most capable of adopting and promoting such a format through their database effort are unwilling to use ZIP archives in this way. They will only allow for an uncompressed single-file-per-game approach.
As mentioned in my article, this turns into quite the problem. When you take the 8KB program ROM for Galaxian, and append a character ROM for graphics onto it, but your header can only tell you the size of the program ROM in 16KB increments, it means there's no way of specifying where the program ROM ends, and the character ROM begins. Any emulator that supports Galaxian does it via game-specific workarounds like checksum identification.
how about MAME, which has a zip-based format containing raw ROM dumps, and the "CHD" format for disk/disc based media? obviously all of the metadata is still contained within MAME itself.
The conclusion that they downloaded the rom from the internet because it has an iNes header is kind of ridiculous to start with; if you build an NES emulator, you're going to need to have some sort of header to indicate what mapper (if any) was used on the cartridge and what stuff was present. Given that the community has already built a standard for that, why not use it?
If you go ahead and use the community header format; of course your rom image will be indistinguishable from a pirated rom -- the header and data should be the same. It's not clear if they downloaded the rom from the internet, read it off a cartridge in their archive (or obtained from employee's collections or the used market), or from a possible archive of all roms they've ever had made.
Given the long life and mass distribution of almost all their titles, there's not a danger of loss of the published material. Certainly, unpublished roms and the occasional title that was not widely produced are in danger of loss if they aren't archived. The source materials are also subject to loss if Nintendo doesn't archive them.
There's some indication that Nintendo may have an archival program after all; Star Fox 2 was recently released on the SNES Classic from a ready for release, but never released or leaked rom; different than the leaked prototype roms. They also add in-memory patches to some roms that run in virtual console and nes/snes classics; I haven't seen anything indicating if they do that based on disassembly of the roms or based on the original sources.
I, for one, really like the idea of a single .zip file per game. Then you can include the unmodified ROM, a separate file with configuration data, any necessary firmware, box art, and any other information an emulator might want. And it saves a little space to!
Also, if two different emulators want two different configuration formats, you could just include both! (As long as the file names aren't the same.)
SNES ROMs have a checksum right in the header, but there were still bad dumps circulating for years. Presumably they escaped before the header was properly reverse engineered.
I'm not exactly sure why that checksum exists, the console doesn't check it, and in manufacturing you could just read and compare. But it's certainly convenient for verifying a good dump.
> Locking it down to specific "best we know of as of this moment" prevents people from easily using materials we know to be incorrect
One big downside I would see is that it somewhat inhibits the creation/growth of an arcade-machine romhacking scene. (Unless there's some explicit provision in MAME for loading arbitrary patches on top of the known-good game versions.)
> which in turn also prevents people from sending invalid bug reports
Something to consider: some other emulators do this not by burning hashes into the code, but rather by downloading at runtime (tracking via RSS or the like) an independently-maintained known-good-dumps-hashes DB.
This approach enables a separation of responsibilities between "building good emulator software" vs "archiving and conserving specific games" — with one of the touted advantages usually being that that the database of good dump hashes can then be used for things other than just powering one specific emulator. It can be shared by many emulators, and other ecosystem tools; and it can even be used by anthropologists to study the "family trees" of game releases.
Also, when emulators besides MAME that use such techniques, detect that a game has a known-bad hash, they don't prevent you from playing it. They just tell you really loudly that you're about to play a bad dump (implicitly, disclaiming all liability for your doing so.) Like a modern web browser's "insecure" warning when retrieving a page over a non-HTTPS connection.
The emulation community also has "ROMsets" — collections of game ROM images, where the ROM images for a given game title are all grouped together into an archive. So you'd have one archive for e.g. "every release, dump, and ROMhack of Super Mario Bros 1."
These ROM-set archives — especially when using more modern compression algorithms, like LZMA/7zip — end up about 1.1x the size of a single one of the contained game ROM images, despite sometimes containing literally hundreds of variant images.
1. Make a header similar to Game Boy. The first four bytes are jump past the header (which might also be used to define the length of the header, in case of being extended in future), and then there is some sort of logo or code to identify the presence of the header, and then the data of the header.
2. Put metadata in a separate file, to avoid using up memory in the program file.
3. Put all of the files together in a Hamster archive file. Lump names starting with a underscore can be used for the special use, e.g. "_default.rom" for the program code and "_metadata" for the metadata. In this case an extra step is required to run the program if the emulator does not support it, but I wrote a program in uxn which is capable of doing this, therefore it can be used with any implementation of uxn.
So, any of these should be able to work with any emulator, even existing implementations which do not implement metadata.
I remember when Nesticle and some of the other viable Nintendo Entertainment System emulators first started showing up. It took a while but eventually everybody just ended up settling on the iNES format by Marat Fayzullin. This was a very useful thing.
I've always found it extremely frustrating that later systems had so many competing formats for the roms as you inevitably end up with lots of duplicate games just so different emulators can handle them. It gets even worse for optical media with all the .cue file stuff and distributions that provide the audio tracks outside of the iso in half a dozen different formats. It's almost impossible to assemble a good collection for those kinds of systems.
I honestly think a .zip based distribution makes a lot of sense since most emulators of cartridge based systems can open zipped archives and run the first compatible file they find in them. But all the rest of the information for the game could be "built in" to the zip.
If you look at the Gamebase projects, there's plenty of hobbyists who are willing to pour thousands of hours into assembling games, metadata, manual scans and other such things into those kinds of projects...but they lack the ability to unify all that ephemera into a single archive per game.
If such a thing were to become standard, I'm certain people/groups outside of the "established" database groups would begin to publish collections.
I for one would definitely welcome a format with built-in cheats, patches and so on and would love to download a collection of say, Japanese RPGs that come with scanned manuals, translated manuals and can run the game in the original and patched versions....or to try out all the weird and wacky Mario/Sonic patched game variants. I just can't be bothered to go download each patch myself to try it out, but if they were already present in the .zip :) I think even support soundtracks would be awesome and could see people firing up collections of games just to listen to the music, without having to keep track of a separate soundtrack archive.
I think this would be an even bigger benefit for modern optical media. The .bin/.cue stuff is honestly terrible. More than half the time I've ended up with a .bin or some kind of .something that I need to convert for some reason, or mount and extract stuff and then handbuild .cue files which don't work half the time. I don't even know if the problem is what I've downloaded, the .cue I made or the emulation compatibility. A standard distribution "format" that just solved all that would be incredibly useful.
This hits on something I've wondered about for a long time. Since carts did not just include data, but also mappers and coprocessors, how do you write an emulator that can accurately play an arbitrary ROM dump?
Does your emulator have to have knowledge of every single game, and what mappers and/or "helper" chips each had, and emulate just the ones appropriate for a given game?
I mean, you'd want to use a file format that would allow you to cross-test your emulator's behavior on the same ROM files against existing emulators' behavior on those ROM files. So you'd format your ROMs (which you dumped yourself) the way that existing emulators would expect them to be formatted.
I agree it's not optimal. It's not like every game changes on every MAME release, but some indeed get re-dumped from time to time. The usual example is encrypted audio data or color palette ROMs. In an earlier version, lacking the ability to decrypt them they would be emulated with samples or code respectively, then once it's possible to dump them they get integrated into the romset for better accuracy.
If archivers started including useful metadata in addition to the rom dumps, I'm sure the emulators would start to use it as well.
Or maybe the archivers are the problem?
reply