No such thing as an inexpensive dirt bike. Well, one that you ride anyway and want to do so for an extended period of time.
That said, a two stroke is the least expensive in the fun category. If you're smaller in stature, a 125 might be the ticket. I hear they are fun, but I am just too fat and don't like to rev it so much. That said, high revs often cause more frequent rebuilds. Talking rebuilds and/or maintenance you MUST do for a long ride, stay away from strokers. Well, MX strokers, YZF, CRF, RMZ, etc. XR, DR, KLR, TTR, etc. is your trail type 4 stroke which are less perfoemance based, take a lot more abuse, and last a lot longer. The latter might be your ticket vs. something form the motocross arena.
Good luck!