New Family BASIC Disassembly Available

[micahcowan]

(This announcement was also posted to nesdev.org; I also plan to announce on r/Famicom) (Feel free to add this to the Famibe mega-thread!)

Hello! I'm announcing today a new disassembly project for Family BASIC v3.0 (for now - disassembly work on v2 is planned to begin soon!).

Although it's still a WIP and quite far from complete, it is quite a bit further along than existing disassemblies I've been able to find, and I've decided to publicize it early because the project has already revealed important information that I have not found anywhere else, and that should be available to people immediately. This includes:

• Complete details on the representation of BASIC programs and bg screens on cassette saves (diagrams included!)
  
• Complete details on the representation of BASIC programs in cartridge RAM
  
• The locations in memory of all of the handler routines for each BASIC command (most of these routines have not been analyzed further).
  
• A few interesting Family BASIC bugs to be avoided: failure to catch overflow on some multiplications (4097 * 4097, for instance), inappropriate overflow errors for (-32768 - 0) (v3 only), and the REM keyword is fundamentally broken for many Japanese words (fortunately, apostrophe works in its place).
  

Numerous other things not particularly worth mentioning, as well; have a scroll through the disassembly as it currently exists, and feel free to stop anywhere you start seeing more comments, or locations labeled with English-looking names.

A lot has been done, much more remains to be done. But enough work is accomplished, that the remaining work will likely be significantly easier.

This project was begun with specific discovery goals in mind--particularly, program representation on cassette and in memory. (I've a project in mind that needs this information--I won't talk about it here, bc who knows if I'll even get around to starting it!) Although the annotations are far from complete, at this point I have learned the specific things I had set out to learn, for Family BASIC v3 (specifically, details of how a BASIC program or background screen is represented, both in memory and on cassette tape.)

Now that those tasks are accomplished, I'll likely shift efforts to reaching roughly the same stage in disassembling Family BASIC v2, because I want to know the same things for it. Family BASIC v2 saves are incompatible with v3 saves, even when restricting yourself to v2 keywords, so I need to discover how they differ. But work will most likely continue (albeit more slowly) on completing an annotated disassembly for both Family BASICs (possibly absent the silly "playbox" stuff at the start of v2; I really don't care if I never annotate that stuff).

[UglyJoe]

Nice work, best of luck to you!

What you are probably seeing with V3 tape dumps not working with V2 is that V3 introduced short-hand values for the numbers 0-9, so that they get tokenized as a single byte rather than three bytes.  You'll see what I mean if you go looking

I've made some Family BASIC tools over the years (lexer/parser, tape reader / converter, etc.).  Mostly I'm working on preserving old magazine type-ins now.  Let me know if you want to talk Family BASIC sometime

[P]

Nice work indeed! It's fantastic to finally see the cassette tape format documented!

[UglyJoe]

Someone pointed out to me on the Gaming Alexandria discord that the tape format is the same as the Sharp MZ (and probably some other Japanese microcomputers).  Definitely seems to track, and backed up everything that I had started to work out empirically by going cross-eyed staring at waveforms in Audacity

I found this to be a good reference (now on archive.org:
https://web.archive.org/web/20240606013759/https://original.sharpmz.org/mz-700/tapeproc.htm

[micahcowan]

Someone pointed out to me on the Gaming Alexandria discord that the tape format is the same as the Sharp MZ (and probably some other Japanese microcomputers).  Definitely seems to track, and backed up everything that I had started to work out empirically by going cross-eyed staring at waveforms in Audacity

I found this to be a good reference (now on archive.org:
https://web.archive.org/web/20240606013759/https://original.sharpmz.org/mz-700/tapeproc.htm

Doesn't look related to the Famibe tape format, apart from possibly that the same representation might be used to represent ones and zeroes, and for individual bytes ("might"... that link is a little sparse on details at that level). The higher-level format appears to be utterly unconnected. Small wonder, the one used by Famibe is... pretty shit, TBH.

[micahcowan]

Nice work, best of luck to you!

Thank you very much!

What you are probably seeing with V3 tape dumps not working with V2 is that V3 introduced short-hand values for the numbers 0-9, so that they get tokenized as a single byte rather than three bytes.

Yes! I noticed that as well. Didn't realize that it's v3-specific! I'll add a note.

I don't actually remember what trouble I had intermingling saves across version numbers, just remember running into trouble the last time I was messing around with Family BASIC, like 7 years ago. But my dim memory was it either failed to load completely, or the tokens were all jumbled around (maybe one was inserted mid-list? I could especially see that happening with a command that takes line-num args, as Family BASIC requires those to occur before all other commands in the keyword list (their numeric args are tokenized differently, so they have to be recognized).

I've made some Family BASIC tools over the years (lexer/parser, tape reader / converter, etc.).  Mostly I'm working on preserving old magazine type-ins now.  Let me know if you want to talk Family BASIC sometime

Do you have a link to any of these things? I did a cursory check of the top post on the mega-thread, but didn't see it, didn't see a (different) website mentioned in your profile, and don't see a downloads area to this one.

I've some tools planned myself (probably in-browser stuff mostly), and some of the main ones are aimed at making preservation of type-ins a lot more practical/versatile.

[UglyJoe]

Do you have a link to any of these things? I did a cursory check of the top post on the mega-thread, but didn't see it, didn't see a (different) website mentioned in your profile, and don't see a downloads area to this one.

I host them here: https://hvc7.dev/

My lexer/parser mini-IDE thing isn't currently on there as I need to rewrite the docs for it, but I could share it if interested.

[micahcowan]

I found this to be a good reference (now on archive.org:
https://web.archive.org/web/20240606013759/https://original.sharpmz.org/mz-700/tapeproc.htm

Doesn't look related to the Famibe tape format, apart from[...]

On closer inspection, there are many similarities, though they do differ when you get to the details. I'd misread a couple things at first glance, and the largest/most obvious of the dissimilarities threw me. (It's also possible I'm using a slightly different document than what you linked: archive.org was down for maintenance, I used this one instead. I think the archive one didn't have nearly this detail about bit representation, but could be mistaken since I can't verify that earlier impression for now.) I've also read some of your JS code for your Family Basic Tape Tool.

They both issue a header block followed by a file block, and use LTM for the header, STM for the file. They both use the same pattern for data bytes. They both begin the blocks with a "GAP", followed by a "TAPE MARK", followed by "LONG pulse", and then the data in bytes, a checksum, and a final long pulse.

The header format is an extremely close match, and both are 128 bytes in length. The core differences are that many of the fields in MZ go unused, and Famibe uses $00 to terminate the filename where MZ uses $0D. Famibe repurposes byte 22, in the header for BG saves, to hold the vertical scroll register's value.

Differences (based on my knowledge, which is currently exclusive to Famibe v3 of course):

• Bits are very similarly represented, but not the same. The frequencies/wavelengths differ, which makes sense, bc so do their CPU clockspeeds (Edit: actually, the freqs are at least comparable). MZ starts with the high period and finishes with the low, while the reverse is true for Famibe (for many readers, depending on the implementation, that might not matter at all - certainly doesn't for your tape conversion tool!).
• Biggest difference: payloads are not sent twice on Famibe. Everything after the second "L" in each of those block diagrams, does not exist on Famibe tape saves. (It was seeing this MZ's lengthy string of format sections that misled me to conclude it had little in common - it has more than twice as many structures involved as Famibe has!)
• There are no SGAPs on Famicom, only LGAPs for both, and they are (I believe) 20,000 19,744 cycles instead of 22,000. I can't imagine this mattering.

But I'm guessing you're already familiar with these discrepancies, from writing your WAV conversion tool.

...I've read your doc on emulator formats. I'm confused about you saying the VirtuaNES format "happens to be usable as a WAV" file because they have a consistent "one-byte-per-frame" format. To my understanding, WAV files have to be RIFF format, so a file that is [interpretable as] raw 8-bit PCM data couldn't be interpreted as a WAV file, because it would be missing the RIFF id and header structures. Are there some very forgiving, probably older audio-file programs that would interpret things this way by default?

[UglyJoe]

...I've read your doc on emulator formats. I'm confused about you saying the VirtuaNES format "happens to be usable as a WAV" file because they have a consistent "one-byte-per-frame" format. To my understanding, WAV files have to be RIFF format, so a file that is [interpretable as] raw 8-bit PCM data couldn't be interpreted as a WAV file, because it would be missing the RIFF id and header structures. Are there some very forgiving, probably older audio-file programs that would interpret things this way by default?

It is effectively raw/headerless WAV data.  You would have to tell your audio decoder the format of the data (basically filling in the info that the RIFF header would have provided).  You can import raw audio data into Audacity, for example.  I think 32000Hz, unsigned 8-bit PCM, is what you would use for VirtuaNES and Nestopia tape files.

I would just use my FBTT nowadays, but it was useful info for me getting started to more easily go to/from emulator/hardware.

[P]

Family BASIC was initially made by Hudson as part of the Sharp My Computer Televi C1 set which includes an early version of the Famibe keyboard. Hudson had previously developed BASIC for Sharp's computers so I guess Sharp picked them for the job to make BASIC for Famicom as well.
So it's no coincidence the tape protocols are similar, Hudson likely designed both.

IIRC all first-party games that support the data recorder (Excitebike, Mach Rider and Wrecking Crew) uses the same Hudson-designed protocol too.

Castle Excellent uses Kansas City Standard (or more specifically the CUTS variant) since it's a port the MSX game (MSX uses a variant of CUTS), but they inverted some of the bits, maybe to make the save data incompatible with the MSX version.