Well, log05.txt ended with some great excitement. I double-checked and all of the open TODOs are now closed. So I think I'll dip into design-notes.txt and pick the next thing to do. I just remembered one thing, I need to _remove_ a feature: the return stack doesn't need to be a stack at all because my "inline all the things!" language can't have nested word calls anyway: [ ] Replace return stack with single addr So, that's not super rewarding, but I do enjoy deleting uneeded code. Oh, I know which feature I'm doing after that! Time to reward myself for staying on track with something fun and visual: [ ] Pretty-print meta info about word! [ ] Loop through dictionary, list all names [ ] Loop through dictionary, pretty-print all Next night: De-evolving the return mechanism for immediate word calling was easy, so that one's done. Now for the fun ones. I'm more of a strings programmer than a numbers programmer. So the ultra-primitive state of my string printing is a bit of a bummer. Before I start storing a billion little pieces of strings in the DATA segment, I'd like to consider adding some convenience words for string handling. It would be nice to have, at the very least, string literals in the language. [ ] Add string literals. [ ] Re-define 'meow' using a string literal. I like the idea of just writing "anonymous" strings to be printed into the dictionary space where all the words are. And I think my choice to null-terminate my strings will pay off here (I hope). Adding immediate mode strings that are just references to the input buffer turned out to be super easy: ; IMMEDIATE version of " scans forward until it finds end ; quote '"' character in input_buffer and replaces it with ; the null terminator. Leaves start addr of string on the ; stack. Use it right away! DEFWORD quote mov ebp, [input_buffer_pos] inc ebp ; skip initial space push ebp ; we leave this start addr on the stack .look_for_endquote: inc ebp cmp byte [ebp], '"' ; endquote? jne .look_for_endquote ; nope, loop mov byte [ebp], 0 ; replace endcquote with null inc ebp ; move past the new null terminator mov [input_buffer_pos], ebp ; save position ENDWORD quote, '"', (IMMEDIATE) And now I can do my first legit Hello World: db ' " Hello world!" print newline exit ' Which works just fine: $ mr Hello world! But since it just saves a reference to the input buffer, real world usage won't really be safe. Unless the input buffer is limitless, I hae no idea if the string address will still be valid by the time I try to use it. For that reason, I'm gonna have to copy any strings from the input buffer to somewhere. I could either have a special-purpose buffer just for storing strings, or I could write to the stack, or I could write to the compile area. The other thing that's really messing with my mind is trying to think ahead (probably way too much) towards how I might handle this stuff in a stand-alone executable program produced by Meow5...which, now that I've written it out, is DEFINITELY thinking ahead too far ahead. Next night: Moving on, I've also decided that I should extract the part of 'get_token' that eats any initial space characters (or other whitespace) out into its own word. [ ] New word: 'eat_spaces' That will allow me to use it to "peek ahead" if I want to in the outer interpreter and possibly switch into a "string mode" (which is something I'm contemplating). But all these paragraphs are me getting way ahead of myself. Back to the assembly! Okay, done. I had just one mistake, but GDB was a clumsy way to debug it. So I added some more print debugging, leading to this extremely verbose output once it worked: $ mr Running ":" Inlining "meow" Inlining "meow" Inlining "meow" Inlining "meow" Inlining "meow" Running ";" Running "meow5" Meow. Meow. Meow. Meow. Meow. Running "newline" Running "exit" I'll comment those out for now, but I'm betting I'll be using them again soon. $ mr Meow. Meow. Meow. Meow. Meow. There we are, good as new. Next night: So while it's true that I could save strings (and other data) in a variety of clever places, my understanding is that modern CPUs do much better with separate instruction and data memory. So I'm gonna say for now that there will be three types of memory in Meow5: 1. The stack for all register-sized parameters 2. The "compile area" where all inlined words go 3. The "data area" where all variables and other data (such as "anonymous" strings) will go. In fact, I'm gonna name #2 and #3 exactly like that: section .bss ... compile_area: resb 1024 data_area: resb 1024 here: resb 4 free: resb 4 Where 'here' points to the next free spot in the compile_area (the 'here' name comes from Forth). And 'free' points to the next free spot in the data_area. And I'm gonna go against the Forth grain and add a special handler for quote syntax. I'll go ahead and peek at the next character of input. If it's a quote, I'll handle the rest as a string. Otherwise, keep processing tokens as usual. The word is called 'quote' instead of '"' and I'm going to call it explicitly in my outer interpreter. The point of this is to allow "normal looking" strings like this: "Hello world" Rather than requring a token delimeter after the '"' word as in traditional Forth: " Hello world" Between that and copying the string from the input buffer to a new variable space, the change in my immediate mode hello world is just the missing space, but it's a world of difference: db ' "Hello World!" print newline exit ' Does it work? $ mr Hello World! Compile mode is exactly the same (I'll put the string in the data_area at compile time), but instead of pushing the address of the string to the stack right at that momment, I need to inline (or "compile") the machine code to push the address *when the word being compiled runs*! To do that, I need to actually "assemble" the i386 opcode to push the 32-bit address onto the stack. So that'll be the "PUSH imm32" instruction in Intel documentation parlance. Handy reference: https://www.felixcloutier.com/x86/push 6A PUSH imm8 66 PUSH imm16 68 PUSH imm32 And I'm gonna test that out with NASM and GDB: push byte 0x99 push word 0x8888 push dword 0x77777777 disassembles as: 0x0804942d <+0>: 6a 99 push $0xffffff99 0x0804942f <+2>: 66 68 88 88 pushw $0x8888 0x08049433 <+6>: 68 77 77 77 77 push $0x77777777 Bingo! So I'm going to want opcode 0x68 followed by the address value. mov edx, [here] mov byte [edx], 0x68 ; i386 opcode for PUSH imm32 mov dword [edx + 1], ebx ; address of string add edx, 5 ; update here mov [here], edx ; save it Well, here goes nothing... db ': meow "Meow." print ; meow newline exit ' There's no way that's gonna work... $ mr Running ":" Inlining "print" Running ";" Running "meow" Meow.Running "newline" Running "exit" What?! It worked! As you can see, I had also turned my debugging statements back on 'cause I was expecting trouble. They help assure me that this is, in fact compiling a word called 'meow' that prints a string stored in memory at compile time. I'll turn the debugging off again. And while I'm at it, I'll remove the old assembly test 'meow' word and define it like this in order to create the 'meow5' word. input_buffer: db ': meow "Meow." print ; ' db ': meow5 meow meow meow meow meow ; ' db 'meow5 ' db 'newline ' db 'exit',0 ./build.sh: line 33: 2650 Segmentation fault ./$F Aw man. Okay, were are we crashing? (gdb) r Starting program: /home/dave/meow5/meow5 Running ":" Inlining "print" Running ";" Running ":" Inlining "meow" Inlining "meow" Inlining "meow" Program received signal SIGSEGV, Segmentation fault. find.test_word () at meow5.asm:165 165 and eax, [edx + T_FLAGS] ; see if mode bit is set... Hmmm. Weird that it dies while trying to find the fourth 'meow' to inline. I bumped up the compile area memory to 4kb and it wasn't that. So I guess I'll be stepping through this. Three nights later (I think): I did step through it quite a bit with GDB, but this thing is getting to the point where it feels like there's a pretty big mismatch between GDB's strengths (stepping through C) and this crazy machine code concatenation I'm doing. I've always prefered "print debugging" anyway. So I've made what I think is a neat little DEBUG print macro. It takes a string and an expression to print as a 32-bit hex number. The expression is anything that would be valid as the source for a MOV to a register: mov eax, . Examples: DEBUG "Value in eax: ", eax DEBUG "My memory: ", [mem_label] DEBUG "32 bits of glory: ", 0xDEADBEEF Since the segfault is happening after a fourth iteration of inline, I feel almost certain that this is a memory clobbering problem. But all my data areas seem more than big enough, so there must be a bug. I've peppered 'inline' and 'find' (where the actual crash takes place) with DEBUG statements. Here's a sampling: Start [here]: 0804a280 Start [last]: 08049ad8 find [last]: 08049ad8 find edx: 08049ad8 find [edx]: 08049a3b find [edx+T_FLAGS]: 00000003 ... Running ":" ... Inlining "print" ... ... Running ";" semicolon end of machine code [here]: 0804a2a5 inline to [here]: 0804a2a5 inline len: 00000007 inline from: 0804961a inline done, [here]: 0804a2ac semicolon tail [here]: 0804a2ac semicolon linking to [last]: 08049ad8 semicolon done with [last]: 0804a2ac [here]: 0804a264 find [last]: 0804a2ac ... Running ":" ... Inlining "meow" inline to [here]: 0804a264 inline len: 00000025 inline from: 0804a280 inline done, [here]: 0804a289 ... Inlining "meow" ... inline done, [here]: 0804a2ae Inlining "meow" ... inline done, [here]: 0804a2d3 find [last]: 0804a2ac find edx: 0804a2ac find [edx]: 595a5a80 find [edx+T_FLAGS]: 0004b859 find [last]: 0804a2ac find edx: 595a5a80 Even viewing exactly what I want to see, all of these addresses are still enough to make me go cross-eyed. So immediately after compiling a new word, I should have this: (word's machine code) tail: link: 0x0804____ <-- [last] points here (offsets and flags) end of tail <-- [here] points here The [last] address should point to the tail of the last compiled word and [here] should point to the next available free space in the compile_area. Time to examine the output. When Meow5 beings, [last] is pointing to the last word created in assembly and [here] is pointing to the very beginning of the compile_area: Start [here]: 0804a280 Start [last]: 08049ad8 After a run-time word is compiled (such as 'meow'), [here] should always be a little larger than [last]. Running ";" semicolon end of machine code [here]: 0804a2a5 inline to [here]: 0804a2a5 inline len: 00000007 inline from: 0804961a inline done, [here]: 0804a2ac semicolon tail [here]: 0804a2ac semicolon linking to [last]: 08049ad8 semicolon done with [last]: 0804a2ac [here]: 0804a264 Which is indeed the case - [here] is a tail's worth of bytes after [last]. So far so good. Then we crash while finding and inlining the 'meow' machine code into a new 'meow5' word. Here's the first: Inlining "meow" inline to [here]: 0804a264 inline len: 00000025 inline from: 0804a280 inline done, [here]: 0804a289 To double-check, I put in even more DEBUG statements in 'inline': Inlining "meow" word tail: 0804b30c len: 00000025 code offset: 0000002c source: 0804b2e0 dest [here]: 0804b30e dest edi: 0804b30e end edi: 0804b333 end [here]: 0804b333 No, that all seems fine. It doesn't look like 'inline' is at fault here. But _something_ is making the linked list incorrect in the tail: find [last]: 0804b30c find edx: 0804b30c find [edx]: 595a5a80 <--- not a valid address Sure, semicolon could have a bug...but that should be causing the problem immediately, not between inlining 'meow' the third and fourth times. Okay, acutally, I think GDB can help me here. I need to know when this value in memory is getting clobbered. Here's the syntax for watching a specific address. Have to cast it - "int" is 32 bits for 32-bit elf and '*' tells GDB that our value is a pointer. It's all very 'C'. (gdb) watch *(int)0x0804b30c Hardware watchpoint 1: *(int)0x0804b30c And let's see what happens: ... semicolon linking to [last]: 08049dc8 Hardware watchpoint 1: *(int)0x0804b30c Old value = 0 New value = 134520264 @124.continue () at meow5.asm:454 454 mov [last], eax ; and store this tail as new 'last' (gdb) x/x *(int)0x0804b30c 0x8049dc8 : 0x08049d2b Okay, that's a good address. So semicolon is doing the right thing so far. Let's continue... ... Inlining "meow" word tail: 0804b30c len: 00000025 code offset: 0000002c source: 0804b2e0 dest [here]: 0804b2e9 dest edi: 0804b2e9 Hardware watchpoint 1: *(int)0x0804b30c Old value = 134520264 New value = 134520192 @36.continue () at meow5.asm:247 247 rep movsb ; copy [esi]...[esi+ecx] into [edi] Bingo! Well, then it *is* inline then. Yeah, clearly it is. Ah, I see, but that's the first 'meow' inline. Which kinda explains why I missed it. So it's gotta be with a [here] that wasn't updated correctly at some point. Wait, has it been staring me in the face this whole time? semicolon done with [last]: 0804b30c [here]: 0804b2c4 Ah geez. Yeah, [here] should certainly be *after* [last]: [last]: 0804b30c <-- 30c (after) [here]: 0804b2c4 <-- 2c4 (before) Dangit! Okay, so some more DEBUGs: tail eax: 0804b35c tail eax: 0804b360 tail eax: 0804b364 tail eax: 0804b368 tail eax: 0804b30c <-- yup! lost some ground here :-( tail eax: 0804b310 Got it! So it was my descision to go against Chuck Moore's advice to always have words consume their parameters from the stack so you don't have to remember which words do and which words don't: %macro STRLEN_CODE 0 mov eax, [esp] ; get string addr (without popping!) ... Sure enough, I forgot to pop to throw away this one unique case where I really do just need the string length: ; Call strlen again so we know how much string name we ; wrote to the tail: push name_buffer STRLEN_CODE pop ebx ; get string len pushed by STRLEN_CODE pop eax ; get saved 'here' position That second pop was getting the name_buffer address I'd pushed before STRLEN_CODE. And now the novice is enlightened. I'll fix that behavior right now and always heed that particular bit of advice from here on out! Okay, then it wouldn't find 'meow' after the *second* try: Finding...0804b2fc meow find [last]: 0804b369 find edx: 0804b369 find [edx]: 0804a062 flags okay: 00000001 finding: 776f656d finding: 00776f65 finding: 0000776f finding: 00000077 finding: 00000000 It turns out I had one more problem with my strlen: add eax, ebx ; advance 'here' by that amt inc eax ; plus one for the null Had to add that last inc because strlen doesn't count the null terminator as a character. So why did it find 'meow' the first time? Because I hadn't yet written anything to the compile area, and the "blank" memory acted as a terminator, but once I started to inline a copy of 'meow' right after 'meow's tail as the definition of 'meow5', that null was no longer there! Now I'm gonna remove about two dozen DEBUG statements... And will this work? input_buffer: db ': meow "Meow." print ; ' db ': meow5 meow meow meow meow meow ; ' db 'meow5 ' db 'newline ' db 'exit',0 Crossing fingers: $ mr Meow.Meow.Meow.Meow.Meow. At last! Guess the pretty-printing the dictionary got super delayed, but this was vital stuff. I'll put those todos in a new log in just a bit. But leaving this much simpler todo for tomorrow night: [ ] Factor out a PRINTSTR macro from DEBUG, then use it *from* DEBUG and also anywhere else I'm currently hard-coding strings in the data section and printing them in the interpreter. Go ahead and push/pop the 4 registers in that one too. Performance is totally not a concern with these convenience macros in the interpreter. Well, that was even easier than I expected. Now to test (I'm using PRINTSTR in DEBUG and stand-alone): PRINTSTR "Hello world!" NEWLINE_CODE DEBUG "[here] starting at 0x", [here] Run: $ mr Hello world! [here] starting at 0x0804a114 Meow.Meow.Meow.Meow.Meow. And replaced all the strings in the data section with my PRINSTR macro - which makes those parts at least 30% shorter and MUCH easier to read. Here's where I'm at with the TODOs in this log: [x] Replace return stack with single addr [ ] Pretty-print meta info about word! [ ] Loop through dictionary, list all names [ ] Loop through dictionary, pretty-print all [x] Add string literals. [x] Re-define 'meow' using a string literal. [x] New word: 'eat_spaces' [x] Factor out a PRINTSTR macro from DEBUG [x] Use it in DEBUG [x] Replace data strings + CALLWORD print So I'll start the next log where I started this one: With the fun dictionary pretty-printer words. This log's progress has been better than I'd hoped for and I think now I'm in a good position for the fun stuff!