Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
root of all eval
Dec 28, 2002

I was just looking into storage for CD based consoles the last few days.

Initially I was thinking a simple USB HDD would be fine but I do have a low power SoC not doing much.

Ends up the MiSTer USB hubs (and the nano?) use USB 2.x ports. Benchmarks from Porkchop show that Ethernet gives 2-3x faster access times than USB or SD.

Some people seem to have issues with USB mounting consistently based on my googling.

I don't quite wanna shell out for a Synology just yet so I'm going to use the odroid as a Samba server with the USB3 HDD I have and put it and the MiSTer on Ethernet.

I asked my wife for a 2 bay HDD hub and a 12+ TB internal drive for Christmas.

Wrt all or nothing mounting, mount priority is based on core. It's common afaik to house some cores on the SD card and some on eternal/mounted drives. I think the SD /games/core/ folder just needs to be empty for the MiSTer to fall back? I haven't tried it yet.
Core priority docs

Edit: if you want to be really frugal just run the Samba mount from your desktop. You'd just need to make sure it's on and shared before you boot up the MiSTer. So many options!

root of all eval fucked around with this message at 02:38 on Nov 21, 2023

Adbot
ADBOT LOVES YOU

root of all eval
Dec 28, 2002

Unrelated... I just discovered Internet Archive has a CLI tool that makes downloading collections of sundry data you may want to put on a NAS really convenient. Lovely

root of all eval
Dec 28, 2002

I was digging in on the initial mednafen MR for groovymister and it's actually pretty easy to grok!

I'm part way through a typescript client to broadcast canvas data or generic image blobs to the core over UDP. If/when I get it working it could act as a proxy but I really wanna port it to Godot in GDScript directly.

This is my first time really working with low level stream buffers and it's pretty wild and fun so far!

root of all eval
Dec 28, 2002

This is terrific news, I was googling for any indication about the cheat engine last week

root of all eval
Dec 28, 2002

My hacking about on a GroovyMister client has paid off! I now have RGB patterns broadcasting to the core. My switchres modeline encoding is off, but I'm sure I can figure that out.

Best news is that it appears simpler than the mednafen client. Looks like the emulator clients need to worry much more about blit timing response for frame steps.

I'll be able to write a 15khz Godot client for it without the lz4 compression which was really the goal.

Dang browsers and cors make it impossible to just use the in browser canvas to send directly, I think I'd need to proxy a web socket to a local UDP relay server which I may try after the Godot client.

I'll open source the V1 and the payload contracts

root of all eval fucked around with this message at 15:48 on Feb 21, 2024

root of all eval
Dec 28, 2002

I had to reach out on discord to get it figured out, but as of about an hour ago I can broadcast moving patterns from the command line!

The API is relatively simple all told, especially for non interlaced 15khz size streams. I'll document the min reqs for the byte ordering on the buffers.

I think I saw someone else in the discord already working on a broader casting client too.

I'm going to start on a Godot client tonight,

Here's a gist of the working test, not cleaned up yet, runnable with Deno
https://gist.github.com/BossRighteous/acb0301afed7ead6c976f102a463692a

root of all eval fucked around with this message at 01:21 on Feb 24, 2024

root of all eval
Dec 28, 2002

The Godot client is technically working but I'm having a tough time optimizing the threads and render cycles to actually cap the viewport, generate a buffer, and send the UDP packets all in one vsync cycle.

Another big annoyance is that the groovy client expects [B,G,R] ordering per pixel instead of RGB. Since the client is already post-gpu I want to try a channel swap shader to keep those cycles off the CPU and get the ordering I want

root of all eval
Dec 28, 2002

Yeah it's real slick! I was talking to the guy on discord. He's nice and offered to respond to DMs with technique questions. Big recommend to anyone on windows looking to experiment or cast other sources.

I guess one thing about a specific Godot client is all the low level APIs are already cross platform out of the gate. Plus I just like the engine :blush:

root of all eval fucked around with this message at 20:58 on Feb 25, 2024

root of all eval
Dec 28, 2002

Tonight I'm gonna drop my test script into the Linux shell of the MiSTer and see if I can get a low res broadcast to the groovy core.

If that works well enough I might ask about adding an input bitmask to the blit ack message so the core can notify the client of general input events. A double could handle 64 input bools in as many bits which is as good as it gets for size/speed.

Honestly that may be a worthwhile feature on the core for other reasons

root of all eval
Dec 28, 2002

With MiSTerCast specifically is designed to be post-gpu. The Windows compositor is the source to make the capture agnostic to rendering platform at the expense of buffer repeats and delays.

Anyone can write an image broadcasting client, as the API is pretty simple. Then it's an issue of bespoke handling for apps that may not use GPU at all.

The compression is lz4 and is automatic above the usual 15khz vertical resolutions.

Anyway Shane is feverishly taking feedback on discord which is super cool.

Also psakhis confirmed input mapping from MiSTer to client is on the roadmap. Exciting times

root of all eval
Dec 28, 2002

You might give this thread a look too if you're seeing what look like compression issues on output. It may not be MiSTerCast at all
https://misterfpga.org/viewtopic.php?t=7565

root of all eval
Dec 28, 2002

I'm just giggling to myself right now realizing I don't even have a 15khz CRT. I think I'm testing this as an excuse to find one.

root of all eval
Dec 28, 2002

When the input handlers are added I imagine there will be a few launcher or media player apps for sure.

I tried running my Deno script on the MiSTer last night but couldn't get a supported build, womp.

Also the Godot client is painfully stuck. No matter how much I try to optimize or multithread the UDP packets will get chunked up and flicker. Super frustrating. I'll post the repo around and see if anyone has any ideas

root of all eval
Dec 28, 2002

I'm putting a pin in it. Oh well, it was a fun first rodeo with byte streams so I'm chalking it up to a win
https://github.com/BossRighteous/groovy-mister-godot

If anyone godots give it a shot

root of all eval
Dec 28, 2002

Coffee Jones posted:

Wizzo’s scripts are neat but I’d like a scraper for downloading box art assets and a display like Emulation Station, where all the mister does is serve a static webpage and the assets themselves.

https://youtu.be/2hpF3laclaM?feature=shared

The cpu core of the mister is pretty anemic, so, offloading rendering onto a phone or desktop via rendering instead of dealing with the standard mister UI.

This was a cool idea I've been hacking on and I'll be damned, it seems possible. I've got python3 on the MiSTer itself updating the core with some random solid color boxes at 240p@60hz. The frame speeds should be fine with really basic compositing.

The input is a problem though, the Linux /dev/input streams don't receive events while a core is loaded. Taking psakhis at their word that there will be client level input support, I'm gonna keep going.

Another big gotcha is the speed and need for C bindings to do a lot of raw graphics work or xml parsing from file system. I'm toying with a non mister util that can parse gamelist.xmls and generate discrete binary DAT files that have what a browser script would want in easier to consume byte format. I figure storage space for uncompressed screenshots and little ASCII data description data is less of an issue if the thing can work on the device without external servers

root of all eval
Dec 28, 2002

I suppose a weekly ish update isn't spamming.

The GUI client work is coming along pretty well all told! Blitting of rectangles, BGR8 streams and even monospace bitmap font rendering is working at full 60 fps with idle frame queues.

I'm now very confident in the ability to make a responsive GUI at least, so I'm worrying about sourcing details now.

MiSTer python build does not support many pip c bindings, which kills PIL pillow image processing. I tried some manual wheel stuff but no dice. Not a surprise but it galvanized the need for pre-processing. MiSTer CPU and Python is not cut out for image decoding in real time,c especially jpg

TheGamesDB is a main scraper for emulation station and is open source and has DB dumps.
Instead of trying to parse gamelist.xml files I'm going to go in 2 support directions:
Scraper > off-mister API > DAT
DAT dumps hosted on archive.org

In both cases I'll be making an info dat and an image dat based on simplified game file name matching. This should allow constant time lookups from core filesystem scans.

Initially I'll need to clone and dockerize the GamesDB API set for my initial full DAT dump but don't want it to be an explicit requirement.

I guess in my ideal state without a custom server or API keys you should be able to download a temp DAT library, scrape your local collection for matches to move, and delete the temp folder remaining. Or if you have the space and are worried about SD writes you could just keep the whole collection

As with most things I gotta build the tooling for myself first. I REALLY want this to be pure MiSTer and network based. The need for image processing just works conveniently in the scraper paradigm.

root of all eval
Dec 28, 2002

fwiw at the price point I don't see a ton of new adoption in the works anyway. It takes a particular retro enthusiast to even care about hardware vs software emulation. I feel late to the game getting mine last year.

Also my perception of core development has been that a second ram slot has been useful for access speeds at times but hasn't been make or break in main adoption. Maybe some arcade cores I'm unaware of would make solid use but there seems to be either a lack of need or a lack of desire to fragment the base. I don't see that changing but I dunno

root of all eval
Dec 28, 2002

Feeling incredibly stoked. Just got Joystick support implemented into my GUI test build.

That was the last unknown on whether the project was even going to be possible.

I've already proven out image/text loading and rendering, directory parsing, and tracked down wizzos launcher GML implementation.

From here it will just take time!

It's taken all my will power to step away the last few days, I can tell I'll just get more obsessed from here though.

I've only talked about it here but got some GitHub stars already which warms my heart

root of all eval
Dec 28, 2002

Poor python on the ARM chip just wasn't cutting it. Frame rates dropping below 24fps was happening with super basic bitmap compositions

Instead of abandoning the project for a GUI I took the last 2 weeks to learn Go.

Currently porting some of the key modules and got the blitting over UDP working.

Go is a good base language for the project for a few reasons so I'm glad I changed. It will allow better native support for jpg and PNG decoding as well as lz4 compression of the UDP packets since the groovy core has 'hardware' decompression on the fly. This will allow 31khz resolutions and hopefully better layouts.

I wanna build a test that shows fps and a combination of text and image blitting benchmarks but even if it's hella slow the goroutines should allow async blitting to a temp buffer while the loop maintains 60hz to the core. As long as the animation isnt heavy the preparation of bitmaps to composite and the sliding of their masks shouldn't be a big deal.

It doesn't explicitly need any animation anyway beyond basic swaps and updates for listing text, meta text, and image galleries.

root of all eval
Dec 28, 2002

The onboard Linux OS has on MiSTer feels like a pretty big feature and ecosystem beyond basic io. Am I just not thinking about it right?

root of all eval
Dec 28, 2002

Cross posting from emulation thread

I'm working on an emulation station like GUI browser for MiSTer and wanted to solicit feedback. Figured this is a good thread.

What scraper data sources and interfaces do you use when curating Meta data?

I'm considering coupling myself pretty tightly to the gamelist.xml model and parallel media folder for each core.

I'm torn between a custom scraper engine against screenscraper API or just consuming the XML and punting the scraping to other tools.

Also let me know if you went about culling larger sets of their beta/hacks/etc for scraping, or whether you went full completionist.

I'm at a crossroads where I didn't know whether to check the XML and use it to sweeten directory listings, or lean in and make it the requirement for the UI since it's meant to be a media based GUI anyway. The main core menus already provide directory browsing based on filenames and dirs

I think the main use case will be browsing local sets with media, but I won't be surprised to find out there are folks with 2500 SNES roms in their gamelist.xml

root of all eval
Dec 28, 2002

Tbh a big part of me wants to scrape it all myself, and save them all out as longlowercasetrimmedalphanumeric.json/jpg/png and put them up on archive.org so as to avoid the need for scraping and deduping and be able to do quick filename based lookups.

The problem scraping tries to solve is the 20 revisions of the same "game" with different names that a regex can handle 95% of the time anyway. But there is probably a big subset of users with existing scrapes and collections I should maybe just accept.

I'll see what the API looks like when I get approved

Adbot
ADBOT LOVES YOU

root of all eval
Dec 28, 2002

w00tazn posted:

Retroarch/LibRetro/Lakka has their own database and images that they curate and you can just download a zip pack that has all the thumbnails.

This was a blind spot to me and very very helpful. I wanted an open source standards based data set even if it is basically married to no-intro and redump sets. Very cool, thank you

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply