Once you are out into the street, let's take care of that little side quest involving the ring (don't worry, we'll get it back in a minute).
Somewhere to your left should be a Wood Elf-looking fellow named Fargoth (he's a level 2 commoner with nothing worth stealing). Talk to him about his "ring" and return it to him (and watch his disposition soar). This will also raise the disposition of the local merchant, so you will get better prices when you unload all of the junk you swiped from the Excise office. Don't panic. You're going to steal the ring back in a little while. Now, on to bigger and better things.
Go across the street and up onto the boardwalk by Arille's Tradehouse. The front door is on the other side of the building.
Arille's initial disposition depends on whether you returned Fargoth's ring or not. If you did not return the ring, it will be about 50. If you did return it should be about 85 or so. Here is a good place to unload your junk and pick up some worthwhile things. If you have any Moon Sugar or Skooma on you, drop it. Arille won't do business with you if you have any in your possession. Sell what you can (which should be just about everything) and buy what you need. Bump the prices up (if you're selling) or down (if you're buying) to work on the Mercantile skill. If you're in need of even more cash (and who isn't?), go out and collect potion ingredients and sell them to Arille.
Everyone in town will tell you that Arille provides rooms, but he doesn't. There is a locked bedroom upstairs and Arille has the key, but it's up to you to figure out how to get it off of him. Of course the bed belongs to Arille and it's illegal to sleep in someone else's bed, but it might be worth a try (save first).
On the subject of potion ingredients, if you would like to save yourself a trip back this way later, you can pick up four ingredients in this area that you will need for your first quest for the Mages Guild in Balmora. You'll need one each of Bungler's Bane, Hypha Facia (they are the fungus-looking things growing on the sides of some of the trees hereabouts), Violet Coprinus and Luminous Russula (the two mushrooms).
Upstairs you will find Raflod the Braggart (offers training in Block [44%], Long Blade [49%] and Medium Armor [54%]), Elone (offers training in Athletics [52%], Medium Armor [52%] and Long Blade [62%], but only to Blades members), Albecius Colonius (mostly useless info), Tandram Andalen (mostly useless info), and Hrisskar Flat-Foot (a small quest).
Hrisskar has a small problem. Seems the dice just aren't going his way lately and he's a little strapped for cash. He has been shaking down the locals, but it still isn't enough. He's convinced that Fargoth has been holding out on him and he wants you to get the money. If you agree, he'll tell you to wait until dark and then climb up the lighthouse and watch Fargoth to see where he hides his loot. Sounds like a plan.
So out you go, over to the lighthouse (snag the gold pieces and silver goblet in the broken tree stump by the door; there is also an enchanted axe in the broken stump behind the lighthouse)), enter, go up the stairs to the top (read the book at the top of the stairs for a skill increase) and wait. You can eat the bread if you're terribly hungry. It doesn't really matter when you do this as long as it's after dark. Fargoth won't go to his hidey hole until you're up on the lighthouse, anyway. So, you stand there (and stand there, and stand there), waiting for Fargoth to get around to checking his stash. Eventually he will and you'll see that he's got his stuff hidden in a hollow stump in the little bog. Once you've got the location, go clean it out (you get the ring back, plus 300 golds and a lockpick). Now it's decision time. If you go back to Hrisskar, he'll take your 300 gold, give you back 100 gold and tell you that you can keep whatever else was in the hiding place. His disposition will go up to decent levels, but he doesn't really have anything that you need. On the other hand, you can keep the 300 golds and be on your merry way, leaving Hrisskar flapping in the breeze.
If you ask around town (latest rumors), you'll hear that the local tax collector, Processus Vitellius, is missing and no one is terribly upset about it. You'll find his body in the Bitter Coast area just outside of Seyda Neen (head west from town). He's got the tax book and 200 golds. When you head back into town to report the murder, no one seems to care, but you are told that you can report it to Socucius Ergalla at the Census and Excise Office if you want to. If you report it and decide to keep the gold, you're done with thanks from Socucius Ergalla. If you choose to return the gold, he will hire you to investigate and promise 500 gold for taking care of the matter.
Back out to town and start asking around about the murder to find that Processus had a thing going with Thavere Vedrano over at the lighthouse. Talk to her and she'll tell you that she saw Processus having an argument with Foryn Gilnith. So go talk to Gilnith. He's pretty up front about the fact that he killed Vitellius. You can let him go or kill him. If you kill him, you'll recover Vitellius' ring (which you can give to Thavere and get a couple of healing potions for a reward) and can freely loot the shack. When you report back to Ergalla, you'll get 500 golds.
There are no other major quests available in Seyda Neen, but there is one very minor one if you're feeling like a terribly nice person and have a hundred golds to spare.
Find Vodunius Nuccius wandering out in the street. He tells you to mention his name to the caravaneer at the silt strider platform. The caravaneer tells you that Nuccius isn't terribly happy in Morrowind. Go back to talk to Nuccius and he'll ask you to buy a cursed ring from him (100 golds) so that he can get back to the Empire. You can recoup most of the money by selling the ring to Arille. With Nuccius out of the way, you should be able to freely loot his house (not that there is much in there, but it should be enough to make back the rest of your 100 golds).
Out to the west of Seyda Neen, in the Bitter Coast region near where you found Processus Vitellius' body, you'll hear a loud scream. If you look up quickly, you'll see a mage-looking fellow fall out of the sky and hit the ground (hard) next to a book. When you go search the body, you'll find some extravagant clothes (which you can sell to Arille), 3 scrolls of Icaran Flight, a Sparksword and a few golds (and the book, of course).
If you read the book, you'll discover that the mage was scared of flying, so he developed a super-jump spell. It really is a SUPER jump spell (7000 points to your Acrobatics skill for <ready?> SEVEN seconds). That's about enough time to get you to the top of your arc (you can't see the ground, so you'll have to steer by using your map). The scrolls are a real hoot. I've managed to jump from Balmora (the top of one of the guard towers works well) to within sight of Pelagiad and back again, and no random critters to mess with, either.
However, our "afraid to fly" mage didn't make any provisions for landing. If you have a Levitate or Slowfall spell that you can cast before you go "splat!", you're in good shape. If you've got potions to fortify your health so that you can take the damage from the fall, you're OK, too. Failing in either of those, you might want to right-click really fast and load one of the other "Icaran Flight" scrolls and cast it just before you hit the ground. Otherwise you're going to be a big greasy spot on the landscape. If you can survive the landings, all three scrolls are worth about half of an Acrobatics skill increase, maybe a full one if your skill was low to start with.
If you ask around town, you'll hear about a smuggler's cave (Addamasartus, though no one will tell you the name) right across from the silt strider platform. It's behind a hill, so it's not very easy to see from the road. There are three baddies inside (all low level, so it's a good place to try out your combat skills), three slaves that you can free (which gets you on the road to the Twin Lamps quests in Vivec), and some goodies that you can sell to Arille. None of the loot is anything to really write home about, but the place was designed for a starting character. There is even some loot underwater, so break out the Water Breathing spell/potion and go exploring. If you decide to free the slaves, make sure that you get the key (there is one laying around and one of the baddies has it in his loot). After freeing each slave, be sure to follow the "go free" conversation thread to increment the FreedSlaves counter by one (the game tracks how many slaves you have freed).
For all of you budding thieves out there, here's a little tidbit. Every time that you steal something and get caught in Seyda Neen, you lose your loot. But it doesn't just disappear into a black hole somewhere. It goes into the evidence chest in the front room of the Census and Excise office (it actually goes into the evidence chest of the closest Imperial Garrison -- in this case it's the Census and Excise Office). Whenever it is convenient, you can simply sneak in and steal it back (save the game first -- if you were caught the first time, you'll likely be caught the second time). The chest has a level 50 lock on it, so either have a very good Security skill or steal the key from the shelf behind Sellus Gravius.
Census and Excise Office (right across the street from where you entered the game): the door is a level 45 lock. Behind it you will find a warehouse of sorts. Andraria Vandacia runs it and can train you in Sneak (45%), Light Armor (50%) and Speechcraft (55%). She might also have a few spells to sell you. There are boxes and crates galore, all owned by the Imperial Guard. There are a few useful items, but you need a very good Sneak skill to grab it.
Draren Theralas' House: He's home and has a few goodies socked away in chests, crates or cupboards in his house. There area a few leveled items, but at the start of the game your level is so low that it's probably not worth your time to generate the ill will by snooping.
Eldafire's House: Behind a locked door (level 25), is an empty house with a few goodies stashed away (golds, a silver longsword, a book that raises your restoration skill, etc.).
Erene Llenim's House: an empty house with mostly junk.
Fargoth's House: Hrisskar already shook him down and checked his house. Nothing of value behind a level 25 locked door.
Fine-Mouth's Shack: He's home and he's got a crate with a small chance of something decent showing up, but is it worth the hassle?
Foryn Gilnith's Shack: He's home and he doesn't have much. You can clean the place out after you kill him as part of a side quest.
Indrele Rathryon's Shack: Empty and mostly junk.
Lighthouse: Lots and lots of crates, boxes and goodies. All under the watchful eye of Thavere Vedrano. There is an Iron Shardaxe (1-32/1-24/1-3 + 0-7 Frost Damage) in a tree stump behind the lighthouse in a little bog and it's not under anyone's watchful eye. You'll have to jump up on the stump to be able to reach it, though.
Terurise Girgayne's House: other than a silver dagger, mostly useless and she's home to boot.
Vodunius Nuccius' House: the door is locked (level 30). Inside is mostly junk.