I will second the recommendation to go with an Everdrive. At this point when they are almost universally compatible and game prices are climbing every year, they make a lot of sense. The SNES and NES versions can easily pay for themselves with one hypothetical game purchase.
Otherwise a MiSTer is another great option to get your fix of pre-1999 systems in a very accurate way. It is an all-in-one solution covering dozens of systems, including SNES, Genesis, NES, Turbografx, and dozens more. Check out the MiSTer OT here which has helped me a lot.
Speaking to original carts and discs though, I think the PS1 is a great entry point. The console itself is very cheap, and loose discs can be found at fairly low cost in most cases. Some games are shooting up in price but the library is so large you can still find great titles at lower prices.