Members 
Search 
Help 
Register 
Login 
Home Show: Today's Messages :: Show Polls :: Message Navigator
|
   |
| Does Mednafen support *any* compressed formats for disc-based systems? [message #6493] | Wed, 14 July 2021 15:41  |
|||
| ||||
|
Formats include but are not limited to: • CHD v5 (.chd) This is very popular amongst emulators (and their users) these days (as of the last year it's gained massive support). The only emulator that has no interest in supporting this lossless, well-documented and supported, compressed disc format is PPSSPP and that's because they use CSO which brings me to CSO. This format is best for BIN / CUEs. • CSO This is much less popular than CHD as it's only used by PCSX2 and PPSSPP - but it's just as efficient as CHD, tho documentation is scarce (the developer of the format tho, is very responsive). This format is best for ISOs. I'm very invested in the future of Mednafen and I believe the massive space saving aspects of these formats are so beneficial to users that they would choose not to use Mednafen in the future simply due to not supporting this (as it reduces disk usage up to literally 95% in some cases, but it's average is around 50% compression). But maybe you guys have plans to support compressed file formats in the future? | ||||
| ||||
| Re: Does Mednafen support *any* compressed formats for disc-based systems? [message #6494 is a reply to message #6493 ] | Thu, 15 July 2021 17:46  |
|||
| ||||
|
I would like to support a new compression format that would be used on top of an uncompressed ZIP file, allowing for existing CUE/* and CloneCD dumps to be used, but haven't really had the energy to design, document, and implement it. There are some questions regarding CHD that would need to be answered favorably before it could be supported in Mednafen: Is it formally documented outside of MAME source code? Is it used by any people or groups actually doing the effort of dumping and preserving old CDs? Is there anything(like test suites) preventing the official tools from screwing up the compressed data or metadata, like IIRC they have in the past regarding pregaps? Does it store subchannel data in a separate contiguous chunk, or is it interleaved with the main data, making more accurate drive seek emulation infeasible without reading the entire disc image into memory first? Considering the frustration MAME developers have expressed regarding forks of old versions of MAME and their outdated and incomplete sets of ROM images, do they really want other emulators to support CHDv5, when there's no guarantee that the other emulator developers will want to or be able to support a newer version that MAME developers come up with in the future? | ||||
| ||||
| Re: Does Mednafen support *any* compressed formats for disc-based systems? [message #6495 is a reply to message #6493 ] | Thu, 15 July 2021 18:54 |
||||||||||
| |||||||||||
At Github there is a stand-alone version of MAME CHD for the formats v1-5 that is actively developed. As for user documentation for people that are imaging their game collections, that can be found many places and it has quickly become the standard (example of such a guide: https://retropie.org.uk/docs/CHD-files/) This is the link to the library: https://github.com/rtissera/libchdr As you said, this was original developed by MAME, so they were the first to utilize the spec, but I could give you plenty of emulators examples, here's a few: https://github.com/mamedev/mame/tree/master/src/lib/util (Look for CHD Files at top) https://github.com/stenzek/duckstation/blob/master/src/common/cd_image_chd.cpp https://github.com/hayate891/nulldcr/blob/master/plugins/ImgReader/chd.cpp
The format is 100% lossless which is why its adoption rate has shot up so high since version 5 came out a few years back. It was designed with BIN / CUEs in mind and preserves their every single track. Quick to compress (when dumping their disc), quick to decompress (when they want to re-burn) for those groups and users as well. It's 1:1 and results in a 50% on average file size. The only people who don't use it that are part of those efforts you mentioned, are simply behind the times and don't know about it.
CHD v1 - 4 had their issues, you're remembering correctly. Since v5 came out a few years ago there has been no such issue and it's likely why the emudev scene (with the exception of PPSSPP) has almost universally adopted it. Tbh, I've never seen this fast adoption of a file format. PCSX2 already supported a perfectly fine lossless format CSO. I asked them directly why they went out of their way to support CHD and they said it was because technically it was the only format that preserved original PS1 games which they plan to support one day aside from it giving them no real benefit today since CSO is just as suitable.
Subchannel data is stored externally actually in this format ".sbi". Though, I'll be honest, the .sbi is just a portion of subchannel data and the position they're located. The few copy protections that use illegal subchannel data simply have it in certain areas, so only that relevant data is stored. It makes some sense to do it that way since subdata is so hard to dump without errors. As for the stream ability of the disc for input / output / seeking - well, it was designed with that in mind. In its most basic form a CHD is a series of compressed "hunks" of data. Hunks are relatively small (512KB max supposedly) and are compressed individually to allow reasonably random access. For CDs, extra metadata is stored to handle the stuff that would be in the TOC and to keep track of whether the track was ripped in RAW format or not.
As of version 5, it seems like that's when they felt they reached their gold standard and I haven't heard a single qualm since that version was released. So, in a nutshell, CHDv5: - LZMA, ZLIB and FLAC compression in chunks of MAX 512kb - Automatic choose of compression based on sector type - Designed to be seekable - Stores checksum inside image - Supports compressed CD Audio data (FLAC) - Supports SBI files for subchannel data - Supports multi-track bins as long as their matching cue isn't corrupt (unfortunately most are but there are groups working on that to recover this data being lost to time and bad dumps) - Decompressed image is 1:1 of source data The library everyone is using that I linked above seems easy to implement and detached enough from MAME to be useful to all emulator developers which is why I think PCSX2 went out of their way to support it even though they're "years away from supporting PS1". But these are just my personal thoughts, feelings, experiences. I think you'd be making the right call. Since so many emulators support it, and there's only one emulator against it, I think we finally have a winner for CD-ROMs / GD-ROMS (Dreamcast Specific). Now DVD-ROMs (like PS2), that's a slightly different story - but I don't know if any of your emulation cores need DVD-ROM support. If you have any more questions, feel free to ask. Like I mentioned, I'm fairly invested in mednafen's future so I'll do whatever I can to help. Oh, and FWIW, one of the lead PCSX dev's said to me the other day "I've never once noticed a difference in load times between raw formats and CHD in our emulator." Now, they use powerful hardware so maybe that's why they never noticed, but it seemed worth mentioning. [Updated on: Thu, 15 July 2021 22:07] | |||||||||||
| |||||||||||
| Re: Does Mednafen support *any* compressed formats for disc-based systems? [message #6514 is a reply to message #6493 ] | Thu, 26 August 2021 09:52 |
|||
| ||||
|
hello. I would like to inform that PCSX2 already supports PS1 and PS2 compressed CHD images. It Would be great if Mednafen one day also add support to this format. PS: Sorry about my broken english. | ||||
| ||||
| Re: Does Mednafen support *any* compressed formats for disc-based systems? [message #6515 is a reply to message #6493 ] | Fri, 27 August 2021 12:22 |
|||
| ||||
|
hi, i believe not even the mame devs consider chdman and the resulting .chd files as an ideal solution for preserving optical discs. at least i remember there was some talking about this years ago when one dev mentioned, everything will have to be converted again once an ideal solution is found. and by ideal i think they mean a format which stores all data+subchannel data and all this preferable in some raw formmat. and i am 100% sure none of the preserving groups will ever switch to .chd. but i am just a nobdy without the need to store 30000+ console disc images just because 
| ||||
| ||||
| Re: Does Mednafen support *any* compressed formats for disc-based systems? [message #6518 is a reply to message #6493 ] | Sat, 28 August 2021 16:45 |
|||
| ||||
| That was a problem. It’s not anymore as of CHD Version 5 and has been widely adopted since. | ||||
| ||||
| Re: Does Mednafen support *any* compressed formats for disc-based systems? [message #6645 is a reply to message #6493 ] | Fri, 18 February 2022 12:32 |
|||
| ||||
|
I strongly support any effort to adopt CHD format in Mednafen. By 2022 it has already become the defacto standard for BIN/CUE file storage. For open-source examples of it's successful implementation, see MAME, Duckstation, PCSX2 and Redream. | ||||
| ||||
| |
| Previous Topic: | Mednafen build failing because of recent changes to MinGW_w64 |
| Next Topic: | Hex edit PS1 CD region auto detection |
]