New Family BASIC Disassembly Available

[micahcowan]

(This announcement was also posted to nesdev.org; I also plan to announce on r/Famicom) (Feel free to add this to the Famibe mega-thread!)

Hello! I'm announcing today a new disassembly project for Family BASIC v3.0 (for now - disassembly work on v2 is planned to begin soon!).

Although it's still a WIP and quite far from complete, it is quite a bit further along than existing disassemblies I've been able to find, and I've decided to publicize it early because the project has already revealed important information that I have not found anywhere else, and that should be available to people immediately. This includes:

• Complete details on the representation of BASIC programs and bg screens on cassette saves (diagrams included!)
  
• Complete details on the representation of BASIC programs in cartridge RAM
  
• The locations in memory of all of the handler routines for each BASIC command (most of these routines have not been analyzed further).
  
• A few interesting Family BASIC bugs to be avoided: failure to catch overflow on some multiplications (4097 * 4097, for instance), inappropriate overflow errors for (-32768 - 0) (v3 only), and the REM keyword is fundamentally broken for many Japanese words (fortunately, apostrophe works in its place).
  

Numerous other things not particularly worth mentioning, as well; have a scroll through the disassembly as it currently exists, and feel free to stop anywhere you start seeing more comments, or locations labeled with English-looking names.

A lot has been done, much more remains to be done. But enough work is accomplished, that the remaining work will likely be significantly easier.

This project was begun with specific discovery goals in mind--particularly, program representation on cassette and in memory. (I've a project in mind that needs this information--I won't talk about it here, bc who knows if I'll even get around to starting it!) Although the annotations are far from complete, at this point I have learned the specific things I had set out to learn, for Family BASIC v3 (specifically, details of how a BASIC program or background screen is represented, both in memory and on cassette tape.)

Now that those tasks are accomplished, I'll likely shift efforts to reaching roughly the same stage in disassembling Family BASIC v2, because I want to know the same things for it. Family BASIC v2 saves are incompatible with v3 saves, even when restricting yourself to v2 keywords, so I need to discover how they differ. But work will most likely continue (albeit more slowly) on completing an annotated disassembly for both Family BASICs (possibly absent the silly "playbox" stuff at the start of v2; I really don't care if I never annotate that stuff).

[UglyJoe]

Nice work, best of luck to you!

What you are probably seeing with V3 tape dumps not working with V2 is that V3 introduced short-hand values for the numbers 0-9, so that they get tokenized as a single byte rather than three bytes.  You'll see what I mean if you go looking

I've made some Family BASIC tools over the years (lexer/parser, tape reader / converter, etc.).  Mostly I'm working on preserving old magazine type-ins now.  Let me know if you want to talk Family BASIC sometime

[P]

Nice work indeed! It's fantastic to finally see the cassette tape format documented!

[UglyJoe]

Someone pointed out to me on the Gaming Alexandria discord that the tape format is the same as the Sharp MZ (and probably some other Japanese microcomputers).  Definitely seems to track, and backed up everything that I had started to work out empirically by going cross-eyed staring at waveforms in Audacity

I found this to be a good reference (now on archive.org:
https://web.archive.org/web/20240606013759/https://original.sharpmz.org/mz-700/tapeproc.htm

[micahcowan]

Someone pointed out to me on the Gaming Alexandria discord that the tape format is the same as the Sharp MZ (and probably some other Japanese microcomputers).  Definitely seems to track, and backed up everything that I had started to work out empirically by going cross-eyed staring at waveforms in Audacity

I found this to be a good reference (now on archive.org:
https://web.archive.org/web/20240606013759/https://original.sharpmz.org/mz-700/tapeproc.htm

Doesn't look related to the Famibe tape format, apart from possibly that the same representation might be used to represent ones and zeroes, and for individual bytes ("might"... that link is a little sparse on details at that level). The higher-level format appears to be utterly unconnected. Small wonder, the one used by Famibe is... pretty shit, TBH.

[micahcowan]

Nice work, best of luck to you!

Thank you very much!

What you are probably seeing with V3 tape dumps not working with V2 is that V3 introduced short-hand values for the numbers 0-9, so that they get tokenized as a single byte rather than three bytes.

Yes! I noticed that as well. Didn't realize that it's v3-specific! I'll add a note.

I don't actually remember what trouble I had intermingling saves across version numbers, just remember running into trouble the last time I was messing around with Family BASIC, like 7 years ago. But my dim memory was it either failed to load completely, or the tokens were all jumbled around (maybe one was inserted mid-list? I could especially see that happening with a command that takes line-num args, as Family BASIC requires those to occur before all other commands in the keyword list (their numeric args are tokenized differently, so they have to be recognized).

I've made some Family BASIC tools over the years (lexer/parser, tape reader / converter, etc.).  Mostly I'm working on preserving old magazine type-ins now.  Let me know if you want to talk Family BASIC sometime

Do you have a link to any of these things? I did a cursory check of the top post on the mega-thread, but didn't see it, didn't see a (different) website mentioned in your profile, and don't see a downloads area to this one.

I've some tools planned myself (probably in-browser stuff mostly), and some of the main ones are aimed at making preservation of type-ins a lot more practical/versatile.

[UglyJoe]

Do you have a link to any of these things? I did a cursory check of the top post on the mega-thread, but didn't see it, didn't see a (different) website mentioned in your profile, and don't see a downloads area to this one.

I host them here: https://hvc7.dev/

My lexer/parser mini-IDE thing isn't currently on there as I need to rewrite the docs for it, but I could share it if interested.

[micahcowan]

I found this to be a good reference (now on archive.org:
https://web.archive.org/web/20240606013759/https://original.sharpmz.org/mz-700/tapeproc.htm

Doesn't look related to the Famibe tape format, apart from[...]

On closer inspection, there are many similarities, though they do differ when you get to the details. I'd misread a couple things at first glance, and the largest/most obvious of the dissimilarities threw me. (It's also possible I'm using a slightly different document than what you linked: archive.org was down for maintenance, I used this one instead. I think the archive one didn't have nearly this detail about bit representation, but could be mistaken since I can't verify that earlier impression for now.) I've also read some of your JS code for your Family Basic Tape Tool.

They both issue a header block followed by a file block, and use LTM for the header, STM for the file. They both use the same pattern for data bytes. They both begin the blocks with a "GAP", followed by a "TAPE MARK", followed by "LONG pulse", and then the data in bytes, a checksum, and a final long pulse.

The header format is an extremely close match, and both are 128 bytes in length. The core differences are that many of the fields in MZ go unused, and Famibe uses $00 to terminate the filename where MZ uses $0D. Famibe repurposes byte 22, in the header for BG saves, to hold the vertical scroll register's value.

Differences (based on my knowledge, which is currently exclusive to Famibe v3 of course):

• Bits are very similarly represented, but not the same. The frequencies/wavelengths differ, which makes sense, bc so do their CPU clockspeeds (Edit: actually, the freqs are at least comparable). MZ starts with the high period and finishes with the low, while the reverse is true for Famibe (for many readers, depending on the implementation, that might not matter at all - certainly doesn't for your tape conversion tool!).
• Biggest difference: payloads are not sent twice on Famibe. Everything after the second "L" in each of those block diagrams, does not exist on Famibe tape saves. (It was seeing this MZ's lengthy string of format sections that misled me to conclude it had little in common - it has more than twice as many structures involved as Famibe has!)
• There are no SGAPs on Famicom, only LGAPs for both, and they are (I believe) 20,000 19,744 cycles instead of 22,000. I can't imagine this mattering.

But I'm guessing you're already familiar with these discrepancies, from writing your WAV conversion tool.

...I've read your doc on emulator formats. I'm confused about you saying the VirtuaNES format "happens to be usable as a WAV" file because they have a consistent "one-byte-per-frame" format. To my understanding, WAV files have to be RIFF format, so a file that is [interpretable as] raw 8-bit PCM data couldn't be interpreted as a WAV file, because it would be missing the RIFF id and header structures. Are there some very forgiving, probably older audio-file programs that would interpret things this way by default?

[UglyJoe]

...I've read your doc on emulator formats. I'm confused about you saying the VirtuaNES format "happens to be usable as a WAV" file because they have a consistent "one-byte-per-frame" format. To my understanding, WAV files have to be RIFF format, so a file that is [interpretable as] raw 8-bit PCM data couldn't be interpreted as a WAV file, because it would be missing the RIFF id and header structures. Are there some very forgiving, probably older audio-file programs that would interpret things this way by default?

It is effectively raw/headerless WAV data.  You would have to tell your audio decoder the format of the data (basically filling in the info that the RIFF header would have provided).  You can import raw audio data into Audacity, for example.  I think 32000Hz, unsigned 8-bit PCM, is what you would use for VirtuaNES and Nestopia tape files.

I would just use my FBTT nowadays, but it was useful info for me getting started to more easily go to/from emulator/hardware.

[P]

Family BASIC was initially made by Hudson as part of the Sharp My Computer Televi C1 set which includes an early version of the Famibe keyboard. Hudson had previously developed BASIC for Sharp's computers so I guess Sharp picked them for the job to make BASIC for Famicom as well.
So it's no coincidence the tape protocols are similar, Hudson likely designed both.

IIRC all first-party games that support the data recorder (Excitebike, Mach Rider and Wrecking Crew) uses the same Hudson-designed protocol too.

Castle Excellent uses Kansas City Standard (or more specifically the CUTS variant) since it's a port the MSX game (MSX uses a variant of CUTS), but they inverted some of the bits, maybe to make the save data incompatible with the MSX version.

[micahcowan]

The Family BASIC v3 disassembly is now effectively complete. All autogenerated labels have been replaced with bespoke mnemonic labels, virtually all the zero-page vars have mnemonic names, and the code is dense with rather thorough comments.

I'm planning to do a walk-through from the top, to ensure that no local variable tables have leaked out to obscure global names where they shouldn't (here I'm referring to global-purpose zero-page vars, where they're temporarily used within some subroutines for very different purposes from their global meaning, for which I used temporary, scoped names), replace more magic numbers with mnemonics, and add additional comments if they seem necessary. But the current version is extremely close to what I expect the final state to be.

I'd also like to add write-ups for details on how DEF MOVE animations are managed by the NMI handler, and on how keyboard input is processed. It's unclear at this time how motivated I'll be to actually finish those, though.

It is also doubtful at this point in time whether I will put much effort into a disassembly for Family BASIC v2.0 or v2.1. Documents like Enri's guide do an excellent job at roadmapping it already, and translating my v3 work onto v2 will likely require the aid of additional tools I'm not terribly motivated to write. I may leave it as a project for some other person to attempt... though as I discover important differences in behavior, I may annotate them in a (very) partial disassembly file for v2.

I'm hoping to write an "attract mode" BASIC program, or possibly an interactive demo, showcasing FB's unique features. Something I could load onto real hardware running FB, to pique the interest of passersby at a convention or club meeting. Likely a thin BASIC shell around mostly machine code.

I'd also like to create a demo program showcasing a bespoke wrapper around the NMI handler, including a simple BGM player, and some additional automatic sprite handling. Family BASIC provides the opportunity for a user to replace the default NMI handler (responsible for sprite animation, playing music, and virtually all screen operations, such as printing text). Normally, you wouldn't want to replace it, because it handles all kinds of vital tasks, and FB would freeze up without it. But, you could wrap it, or replace with a mildly-modified copy that alters some behaviors. You could, for instance, add some timing-critical logic around animated sprites, to improve responsiveness; and I suspect you can add a BGM player. I'd like to demo this capability.

There are also some projects whose existence would make me happy, but which I'm personally not very motivated to work on:

• A cheat sheet of useful PEEK/POKE locations (derived mostly from the zero-page vars, which fortunately seem to be relatively stable between FB v2 and v3)
• A terse guide to the many interesting and useful entry points and useful routines
• A more thorough guide to using assembly and machine code within Family BASIC. I'd probably have lots of contributions for such a thing, but don't necessarily want to write an exhaustive guide, myself. Maybe a series of YouTube videos, idk.

I have several other project ideas in mind, but describing them well would take too much space, and I'm still working on concept details.