|
Vives #1 problem
Posted: 02 Mar 2004 10:58 AM |
Please don't take this as anything but a constructive critique. This is far and away the best NwN server I have ever played on (and that's saying a lot as Samwise would say). heh.
The content here, the DM's, the server policies, the reward vs. risk and the players; these are all top-notch here. I couldn't be more pleased to be here and have been thoroughly enjoying myself.
The bad: LAG. I know latency happens. I was there in the first months of UO when we invented the term 'slideshow' lag. Still, what I see here is some of the worse server-side lag I have ever experienced. How do I know it is server-side? Because I can play on many other servers at the exact same side and it is glass smooth. Because I can log into my own local server and it is glass-smooth.
I have noticed a HUGE seemingly CPU-utilization whenever someone logs on. I noticed 'slide-show' lag when I first logged on. I have noted onclient enter script errors several times. I imagine that you (Vives) are doing some major DB calls whenever someone logs in. I suspect this has something to do with it.
Yes, some of this may be due to internet backbone issues between many of the players and the server. I know nothing can be done about that.
I put this to you: Vives would be the KING of all NwN server if not for one flaw, LAG. It is your only shortcoming. If you can polish out this one flaw in your diamond you will have a server as close to perfection as any out there.
Thanks for listening. Again. Do not think I do not appreciate all the hard work done so far. In fact it is because I enjoy playing on this server so greatly that I long for this one issue to be resolved.
:) |
|
|
  |
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 11:06 AM |
Seems that vives is stable untill the number of people logged in reacehs a certain point and then it just dies very quickly.
Also not everyone causes huge lag spikes when they log in, Its only people that carry a redicuously large ammount of items around with them (stuffed inside magic bags so they dont weigh alot). I wont say any names but there are a few people that scare me when they come on. |
mannaka no
ana ni sounyuu
awari kana |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 11:15 AM |
I think everyone is more or less agreed the lag is starting to become a real pain up to the point where it makes playing nearly impossible.
Even regular Roleplay non action, walking related, becomes a hard effort if your dialogue hangs for minutes. I guess now would be a good time to start evaluating what we have, what we really need, what causes the lag and modify accordingly. Essentials should have top priority, custom content should go to a lower plan again and only be added if it is 200 % certain it is not adding a shitload more of lag.
Especially if we are going to promote guilds it might be an interesting idea to exchange personal rooms for guild stuff so it is open to a wider public. As to equally share of the available resources till there is more to share.
I'm not sure what causes the lag though, is it also craft related? |
Why can't I PM myself?
Don't iron out the Irony. |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 11:21 AM |
I just read Fenarisk's suggestion on the Wish-List forums and he makes a good statement and asks a valid question: Fen says it is the CNR system that causes most of the server-side lag, he then asks if we should consider removing it?
Now, I for one don't much care for crafting but I will admit I have played around with CNR here in Vives and it is well done. I can imagine there are some folks who have put a HUGE amount of time and effort into their CNR skills. I can also imagine they would be somewhat hesistant to loose all that time and effort.
I for one would rather have a smoother server than the CNR system. Yet, I can see how some Master Armorsmiths might see things differently. Perhaps we could remove the CNR system and yet compensate folks who have very high CNR skill levels? |
|
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 11:51 AM |
I agree on both counts. Vives is a fabulous world, and the lag is the #1 problem here. If dropping CNR will get rid of the lag, then I am all for it, but as you say, we would need to find an equitable solution for the folks who have invested time in trade skills.
All I can say is that Vives would be a much better experience without the lag. |
|
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 11:59 AM |
| I'd say temporarly remove cnr and see if it fixes the lag. If it reduces the lag enought to warrant its removal the i think it should be removed untill it can be worked out to a point that it does not cause much lag. If the effect of removing it is neglagable then there is no reason to remove it. I dont think anyone can guess what effect it will have untill its done, computers are just like that. |
mannaka no
ana ni sounyuu
awari kana |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 12:39 PM |
Things that can help lag: (please take these as what they are, suggestions from the peanut gallery. I am not on the Vives team, and cannot possibly know what their behind-the-scenes is like.)
1. Reduce object count -- There are a lot of ways to approach this. Reduce static placeable count in non-essential areas. Reduce/eliminate crafting and crafting related objects (maybe we could have 1000 recipes, instead of 2000?). Write a "cleanup" script to find and delete all Remains and dropped items when the last PC leaves an area. Initiate a limited "item exchange" policy where PC's are able to trade X number of low- to mid-level magic items to a DM in exchange for 1 high-level magic item w/ a custom description (eg. Fri'el's Robes of Utter Badness or Kayel's Armor of Leprechaun Disguise). Set X to a very high number.
2. Reduce external calls/load -- This website probably does very little to load the server, honestly. But you are reaching a point where ever picocycle of CPU time is a premium. There are a number of very reasonable hosting solutions out there that could easily handle this website for ~$5 per month, ~$60 per year. Archive half of the Dirtnap League then remove their records from the server. Set the Dirtnap League to only update during Auto-Resets rather than in (more or less) real-time. Remove the website-side "Currently In Game" information, as it is: a) external load, b) redundant in light of /pcinfo, and c) horribly OOC as it doesn't seem to be possible to exclude one's self and location from the website list the way you can w/ /pcinfo.
3. Reduce NPC/monster count -- Spawn points do create spikes in CPU usage, it is true. But every server-birth monster or static NPC requires a constant RAM signature while the .mod is running. The more, the slower. Also, consider replacing the voice spiders with regularly spaced invisible placeables w/ an OnHeard event. No "follow" commands means less pathfinding work for the server (since the server must control all those NPC's).
4. Script Optimization -- I am, at best, a script monkey. There are people in the NWN community who are, to put it plainly, script geniuses. Small, subtle changes in script architecture can make massive differences in CPU load at runtime. Try posting on the Bioware Scripters forum looking for people willing to volunteer some time to review your scripts for efficiency. Be prepared to wade through a ton of idiot posts after you do, in hopes of finding one or two good seeds. (/me is terribly disenchanted with the Bioware forums)
5. Spend obscene amounts of money -- I hate to even mention this one, as it's the worst possible alternative for a group of people engaged in a free hobby. But Business Class DSL w/ a high upload, or Dual-Xeon 2.0Ghz (or better) server w/ RAID and a Gig of PC3200 DDR RAM can make a huge difference in performance. And I know the team has decided against donations in the past, but your playerbase WILL support you if you give them the chance. Not out of hopes of favoritism (weed those folks out, anyhow...) but because they want to play on Vives! I've bought my DM's rulebooks before, not to mention copious amounts of pizza and beer.
Look at it this way... I've spent somewhere in the neighborhood of $200 buying NWN and the expansions for myself and my wife so that we could play Vives. That's true in a very real way. I've played about half the first OC, about 1/4 of the second OC, and basically never even touched the third one, and my wife never even made it through half of chapter 1 of the first OC. NWN is an online game client for us, that's what it is. The longer I can continue to play online w/ it, the more value it has. I have (for other servers) and would be happy to (for Vives) kick in $50 or $100, as I can spare it, to keep my favorite game worlds alive. That's part of what will make NWN last after Bioware stops supporting it (and they WILL stop supporting it).
Bug catching and optimization, QA really, are slow, dull, boring, long-headed tasks. Unfortunately they're also vital. Try setting up a test machine that just runs your .mod w/ no network connection and start stress-testing. Fire off scripts one at a time and chart CPU usage. Spawn increasing numbers of monsters and see what number makes the server crash. Start a character of each class and perform every action on their radial dial. Cast every spell in the game, one-by-one. Stuff like that.
If, as you say, server load seems to be fine up to a "magic number" of PC's (and for the record, the fact that you've got your player limit capped at 30 and are still experiencing this speaks volumes, as I've played on servers capped at 64 that stayed at 20+ players 24/7 w/o anywhere near this kind of lag) really honestly suggests to me the issue is those voice spiders. It seems to me that lag started to be a real serious issue w/in the past week really, which would be right around the time that voice spiders started getting assigned automatically on OnClientEnter. It's the only thing, right now, that all PC's have in common. Can't be attributed to inventory count, or spells-in-use, or it would be more readily identifiable, IMO.
-- Figleaf |
|
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 12:48 PM |
| For what it is worth, the voice spiders -don't- follow. That's why you have to invoke the spider widget again, to call it to your side, when you want to use it. |
Dieties preserve us from the evil chaotics named Lag, Crash, and Server Reset.
PCs: Loli Dankirk (Ftr 18/Brd 8.5 @ 1777 hrs) left Vives for Origins; Gnora Gnombody (Sor 5 @ 54 hrs) and Amanda Stark (Wiz 3 @ ~10 hrs) left Vives for Feline Providence. |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 01:03 PM |
Errrm?
I know that sometimes they don't update right away when you transition, but I can pretty much guarantee that, to the best of their ability, they do follow you.
If you play for an extended length of time w/ the TAB key taped down, you'll sometimes be able to see a nameless, green "Uninjured" following you around.
Also, for those of us who are "keyboard drivers" the voice spider is a constant threat. Nothing like trying to run away from a monster only to get stuck behind your four partymates voice spiders because you were at the back of the group. Oh hello, Mister Lich. You won't mind if I just fall down dead right here, will you?
And have you ever transitioned to a new area, only to immediately get "bumped" back through the transition to the area you just left without hitting a single key or mouse button at all? You spawn in, then the spider spawns in at pretty much the exact same place, which (thanks to some HIGHLY EXPLOITABLE clipping behavior in the NWN engine) pushes you backwards through the transition trigger.
Again, I'm not on the team, so maybe I'm wrong. But my play experience, and that of my wife's, makes me fairly certain the spiders are doing their best to follow.
-- Figleaf |
|
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 01:10 PM |
| Just a friendly bump and a nod in support of slink and Figleaf =p |
Humbly Submitted,
Sinjin Kane |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 01:23 PM |
According to Aria, who programmed the new spider, it does not follow. That may have changed since she told me that, but the last time I tried to use it without first summoning it to me it was not possible.
Obviously only the people who actually have the scripts can say for certain what's happening. |
Dieties preserve us from the evil chaotics named Lag, Crash, and Server Reset.
PCs: Loli Dankirk (Ftr 18/Brd 8.5 @ 1777 hrs) left Vives for Origins; Gnora Gnombody (Sor 5 @ 54 hrs) and Amanda Stark (Wiz 3 @ ~10 hrs) left Vives for Feline Providence. |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 01:57 PM |
I had some people testing some new areas that went IG in the last update, and can confirm a nameless, invincible spider followed em around here and there .
~Fenarisk |
|
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 02:06 PM |
The listener does follow, ala AssignCommand(object, ActionForceFollowObject(oPC, 10.0);
I dont know that replacing the following listener with static listeners would be a solution. I dont know what the hearing range of the static listeners would be, but even if we used one or two per area, thats 500-1000 added placeables, with a listening pattern assigned.
The comment you made about monsters is a very valid one figleaf. One thing I would like to look at is replacing all the placed monsters with encounters. This would, theorhetically, greatly reduce the startup lag.
Cleaning up wasted area's wouldn't hurt. We do have a tonne of them. Removing unused or wasted area's is something the builders should discuss.
I believe the majority of server side lag, aside from the first few minutes after a server reset, is related to bandwidth. Dealing with this is entirely up to what Quietus is willing or able to do.
Opening up donations from players is not something that we are not open to doing, as it puts alot more responsibility onto whomever is the recipient of the funds. Both responsibilty to those that have donated of themselves, as well as managing how the funds are put towards server use, also, there is having to deal with any tax issues that may arrise from it. Additionally, having to rely on a monthly source of funds to run the server, puts a heavy burden on the person in charge if the funds are short for one month, or slowly dry up in the long run. All of these are ontop of the fact that the thought of accepting donations is uncomforatable to us. |
~Alosynth
|
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 04:37 PM |
| Well, I checked this past time in, and there is nothing following Loli around. Do you suppose there is a difference between newly distributed spider widgets and the pre-existing spider widgets? The spider -did- used to follow Loli, before Aria worked on it. |
Dieties preserve us from the evil chaotics named Lag, Crash, and Server Reset.
PCs: Loli Dankirk (Ftr 18/Brd 8.5 @ 1777 hrs) left Vives for Origins; Gnora Gnombody (Sor 5 @ 54 hrs) and Amanda Stark (Wiz 3 @ ~10 hrs) left Vives for Feline Providence. |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 04:43 PM |
And whoever posted it before... I agree, there seems to be something wonky in OnUsed for the .mod.
For example, every time Kriayna and Kayel are partied, if Kriayna uses the actual PCInfo tool, she is set to autofollow Kayel immediately before the convo. menu comes up. Every time.
-- Figleaf |
|
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 05:29 PM |
Yeah, that was me.
If the spiders could be made to be "not-following", that would be great. I never liked them following anyway, because they re-opened doors behind the PC. We can summon them to us with the spider widget when we need them. That means only 30, and not moving. Would that not be acceptable? |
Dieties preserve us from the evil chaotics named Lag, Crash, and Server Reset.
PCs: Loli Dankirk (Ftr 18/Brd 8.5 @ 1777 hrs) left Vives for Origins; Gnora Gnombody (Sor 5 @ 54 hrs) and Amanda Stark (Wiz 3 @ ~10 hrs) left Vives for Feline Providence. |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 05:36 PM |
| Along with the HOTU upgrade, there were skyboxes added and the fog was reduced. |
Dieties preserve us from the evil chaotics named Lag, Crash, and Server Reset.
PCs: Loli Dankirk (Ftr 18/Brd 8.5 @ 1777 hrs) left Vives for Origins; Gnora Gnombody (Sor 5 @ 54 hrs) and Amanda Stark (Wiz 3 @ ~10 hrs) left Vives for Feline Providence. |
|
|
|
|
Re: Vives #1 problem
Posted: 02 Mar 2004 05:44 PM |
| I also got problems like using the players stone, and get the PC emote wand open, the folow bug I got it too... |
Why search around when the answer is within you...?
|
|
|
|
|
Problems are cool, but Solutions are better.
Posted: 02 Mar 2004 07:40 PM |
Here's some more relevant discussion on the topic.
It seems to me that lag started to be a real serious issue w/in the past week really, which would be right around the time that voice spiders started getting assigned automatically on OnClientEnter. It's the only thing, right now, that all PC's have in common. Can't be attributed to inventory count, or spells-in-use, or it would be more readily identifiable, IMO.
To enable the voice functions, you need listeners that hear and match universal patterns. Previously these were wired into every single of the 2000 creature in Vives; every time anything was spoken within the earshot of a NPC, part of the parsing script was fired. With the voice-spiders, the listening patterns are relieved from the NPCs and localized onto the voice-spiders instead, one for each PC. The voice spiders were done with the intentions to trim performance.
(There is a section of highly inefficient listening code, but the DMs think the function quintessential to their operation. It was on my list of trimming downs.)
According to Q, the server was running at 40% CPU usage. I think this number is for a 10-15 ppl in-game, so I'm really not sure whether it's responsible for the constant level of lag that players seemingly experience. Perhaps some of the network gurus can help do traceroutes to the server (or better, teach others how to do it), so the Vives team know for positive it's not from bandwidth limitations? For again, the 20-30 on the website, 15-25 in-game, Q's personal use, and the FTP are all sitting on one home cable.
Edit: Response to slink's bash-an-aria removed. |
Aria
So talented, so troubled. |
|
|
|
|
Network Statistics
Posted: 02 Mar 2004 08:00 PM |
Here's some numbers I've got from "pinging" the server thru http://www.tracert.com/cgi-bin/ping.pl...
----vives.dyndns.org PING Statistics----
5 packets transmitted, 5 packets received, 0% packet loss
round-trip (ms) min/avg/max = 97/365/773
--- vives.dyndns.org ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 416.9/499.4/621.5 ms
--- vives.dyndns.org ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 330.686/419.234/632.328/112.084 ms
--- vives.dyndns.org ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 251.136/391.817/581.222 ms
Then I did it a second time...
----vives.dyndns.org PING Statistics----
5 packets transmitted, 5 packets received, 0% packet loss
round-trip (ms) min/avg/max = 140/411/702
--- vives.dyndns.org ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 195.9/548.6/880.7 ms
--- vives.dyndns.org ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 105.916/486.618/981.627/349.891 ms
--- vives.dyndns.org ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 28.513/419.379/878.229 ms
I don't know anything about computers/networking, so if someone can help interpret these (possibly meaningless) results it'd be great!
These numbers are obtained at 8pm EST, when there was 19 players in-game, and 15 on the wesite.
|
Aria
So talented, so troubled. |
|
|
|
|
Re: Network Statistics
Posted: 02 Mar 2004 08:50 PM |
| I wonder if there could a server gate between towns.. im saying that certain towns would be on one server, and other towns or areas could be on another server. I dont know if this can be done or not.. I have heard that another server did it or was going to do it, but I dont know for sure. just putting out ideas. |
|
|
|
|
|
Re: Network Statistics
Posted: 02 Mar 2004 09:25 PM |
.mod splitting can be done, but the headache it causes is something that should not be taken lightly.
Doing a traceroute from a server in Alameda, CA (closest I could get to home) to the game server, in the middle of Legolas' archery lesson with a HUGE number of people on the server returned this:
dsl081-053-001.sfo1.dsl.speakeasy.net 0% 20 20 10.82 12.08 13.79
110.ge-0-0-0.cr1.sfo1.speakeasy.net 0% 20 20 8.84 12.06 49.26
border4.g3-1.speakeasy-48.sje.pnap.net 0% 20 20 9.86 24.65 176.59
core2.pc0-0-bbnet1.sje.pnap.net 0% 20 20 10.17 30.23 125.84
ge-0-1-0.106.ar1.SJC2.gblx.net 0% 20 20 9.66 11.37 13.73
pos6-0-2488M.cr2.SFO1.gblx.net 0% 20 20 11.62 13.07 14.50
so5-0-0-2488M.ar4.SFO1.gblx.net 0% 20 20 11.98 13.58 20.93
208.51.243.250 0% 20 20 16.82 32.66 53.76
tbr1-p013302.sffca.ip.att.net 0% 20 20 17.00 33.47 54.18
tbr1-cl1.cgcil.ip.att.net 0% 20 20 59.62 74.11 93.72
tbr2-cl2.cgcil.ip.att.net 0% 20 20 57.79 73.88 94.08
tbr2-cl5.cb1ma.ip.att.net 0% 20 20 80.53 94.93 116.37
gbr1-p100.cb1ma.ip.att.net 0% 20 20 82.45 95.16 115.93
gar2-p360.cb1ma.ip.att.net 0% 20 20 82.28 100.70 195.59
12.125.33.34 0% 20 20 80.71 95.36 115.25
bar02-p7-0.ndhmhe1.ma.attbb.net 0% 20 20 80.05 96.32 118.06
c-24-218-191-206.ne.client2.attbi.com 0% 20 20 79.92 96.21 113.18
c-24-218-191-150.ne.client2.attbi.com 0% 20 20 79.19 95.23 114.78
c-24-218-191-234.ne.client2.attbi.com 0% 20 20 81.67 96.63 119.17
bar01-d5-0.mefdhe1.ma.attbb.net 0% 20 20 78.76 95.39 114.92
ubr01-p1-0.mefdhe1.ma.attbb.net 0% 20 20 80.60 99.10 117.00
vives.dyndns.org 10% 18 20 95.89 124.01 201.16
There are a lot of jumps here, most of them in the att.net and att.com range. Which makes me wonder if the company doing your DNS resolution has you set up on some sort of ugly internal server farm.
However, the ping range for every one of these jumps is VERY reasonable, and doing a straight ping from my home machin to the server returned not a SINGLE response above 110ms.
At this point, I'm going to say that bandwidth is not your issue. You aren't getting, say, Google's ping of ~60ms, but you are getting a DURNED good ping for an NWN server.
What concerns me here is that you got 10% packet loss on the actual game server itself. That just isn't O.K., and would explain some of the really bad "ping-ponging" going on ingame.
This may be an overly simplistic answer, but have you tried a new network card for the server? Packet loss on a ping or traceroute shouldn't really have anything to do with the function of the server itself, as these are such monumentally low-level operations that they are generally handled by the network card itself.
QA is such a scattershot thing... Let's review:
CPU load isn't going above 40% for the most part.
Pings are in the 90-110 range.
DNS resolution is requiring every packet jump through a ridiculous number of hoops to get to it's location.
You are experiencing packet loss at the server address.
I know there's another thread going asking people to report lag experiences. If people could include in their lag report:
Time/Date of play.
# of players ingame.
"Symptoms" of lag. (eg. "I typed something and it took 45 seconds to say." "I kept bouncing around all over the screen w/o moving." "I moved around for 5 minutes, but the map didn't update, then all the sudden I moonwalked back to where I started from."
It might help narrow down the problem. Suspects right now include:
1. Your hosting/DNS package.
2. Server network card.
3. Modem or firewall.
-- Figleaf
PS: I could be wrong, but I suspect that objects listening would tax the server less than objects following. I also will do some experimenting in the toolset to see if I can't find a way to make the "listener" an item in the PC's inventory instead of a placeable or monster. HotU provided some new scripting hooks that I've been dying to misuse and misunderstand. *laughs* |
|
|
|
|
|
Re: Network Statistics
Posted: 02 Mar 2004 10:17 PM |
I would just like to state how DUMB I feel for being a part of this mod on the admin staff and understand exactly 0% of what Figleaf says on this thread .
~Fenarisk |
|
|
|
|
|
Re: Network Statistics
Posted: 02 Mar 2004 10:26 PM |
Basically what he is saying is that its sending all the signals all over the place. Basically its like throwing a ping pong ball at the door so it hits the lamp, then hits the tv, then hits the radio, goes off the washing machine and finally lands in the trash can rather than simply throwing at the trash can in the first place.
Packet loss is bad, basically what happens is you and the server exchange information in short bursts called packets (Includes such stuff as where you moved, if you took any dammage, and stuiff like that). 10% loss means that for everyone 10 packets you or the server sends 1 is just lost, corrputed, or flat out disappears. If a packet containing where you just moved to is lost and never makes it to the server you will be jumped back to the last know location (rubberbanding).
I dont know how to fix anything i was just trying to explain some vocabulary here. |
mannaka no
ana ni sounyuu
awari kana |
|
|
|
|
Re: Network Statistics
Posted: 02 Mar 2004 10:31 PM |
| dang Fenarisk.. you must be the only one here thats not a nerd.. lol |
|
|
|
|