Help with Quake Bot

Discuss programming topics for any language, any source base. If it is programming related but doesn't fit in one of the below categories, it goes here.
Tremor
Posts: 13
Joined: Wed Aug 10, 2011 5:38 am

Help with Quake Bot

Post by Tremor »

Hello everyone.

I'm in the process of building a 'qsmack'-like server admin bot for Quake 1 (netquake). I'm creating this bot in python, mainly for the network byte-level programming experience. I've studied the Quake Specs 3.4 and the Unnoficial Quake Network Protocol in great detail. I've also been using wireshark to capture packets sent by a quake client and received by a quake server.

Section 8.6.3 of the Quake specs (above) show in some detail how the initial quake connection works.

Code: Select all

Server sends a big message that contains:
  - the server banner "VERSION 1.01 SERVER (21456 CRC)"
  - a server info message (code 0x0B) with map name, and precached models and sounds
  - an indication that client should move to PRESPAWN state.
Client sends a No Operation (code 0x01)
Client sends a console command (code 0x04) "prespawn", indicating prespawn is finished.
Server sends a big message that contains order to spawn static sounds and static entities
  - For all the static entities described in the .BSP file, 
     One "Spawn Static entity" order, and possibly a Static Sound order.
Client sends a player information message:
  - console order "name PLAYER\n" (where PLAYER is the player name)
  - console order "color 0 0\n"   (shirt and pants colors)
  - console order "spawn " to indicate the client is ready to play.
Server sends a big message:
  - For each possible player in the game (including those not connected):
     a set of orders UPDATE NAME, UPDATE FRAGS, UPDATE COLORS
  - For each of the 64 light style:
     a SET LIGHT STYLE message.
  - an UPDATE STATE message for total and found monsters and secrets
  - a SET ANGLE order, to orient the player's view
  - a CLIENT DATA order, fix the status bar display
  - an order to move to Start 3d Rendering state.
  - a last update NAMe, FRAGS, and COLORS, for the client.
Client sends a console message "begin".
According to the documentation, the client should be completely connected at this point.

After I send the 'begin' packet, the connection is accepted (in full) by the server and I start receiving the expected server packets. The server displays the connection being accepted as well. However, at around the 30 second mark, the server drops the connection with a 'no reply from playername'. My bot sends an acknowledgement packet for every reliable (0x01 and 0x09) packet. It seems to be sending this information correctly. Unreliable packets (0x10 are not acknowledged. Apparently the server is not getting a packet that it is expecting - I know that there is a net_messagetimeout on the server, so I've tried sending game keepalive packets at various intervals to no avail.

When I examine the wireshark captures, the real quake client starts sending a flood of unreliable (0x10) packets with the client movement orders (0x03) regardless of whether movement is actually made or not.

My question is this:
What kind of packets is the server requiring of the client and how often should they be sent? Do I need to duplicate these unreliable client movement orders (even if I do not want the bot to actually move)? That is to say, send the server client movement orders with the exact same coordinates over and over for the server to keep my connection alive? If that's not the case, what am I missing?

Secondary: Can someone point me to some documentation on the character mappings in quake so that I can better parse the server messages. At this point, my bot can parse the message 'header' to determine what kind of packet is being received, but it cannot read the entirety of the message itself. For example, I can distinguish between all the main packet types but not read the rest of the message.

Let me know what you guys think. Thanks!
LordHavoc
Posts: 322
Joined: Fri Nov 05, 2004 3:12 am
Location: western Oregon, USA
Contact:

Re: Help with Quake Bot

Post by LordHavoc »

You need to respond to each svc_signon message with the appropriate reliable command (prespawn, spawn, begin), or the server drops you for having a half-open connection (which has been used as a network flood exploit in the past - hence the "brief" 30 seconds - note that upgraded server protocols like darkplaces DP7 do not suffer from this exploitable connection process).

The keepalive messages should work fine (clc_nop), but be aware that the server tells you which port to talk to, the original connection port won't accept clc_nop and other packets (unless hosted by darkplaces servers running quake protocol where it uses a single port for quake servers, or fteqw running quake protocol which will do the same thing).
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Help with Quake Bot

Post by Spike »

the protocol is built on spam.
you need to spam unreliables or you'll time out. unreliables contain some timing information which is used to determine pings etc.
either 60 or 72 per second is a good number. more than that is excessive and/or cheaty.
servers that support player prediction will require a continuous stream of movement messages or the player is likely to freeze in place despite having a velocity.
reliables require that you wait for a response before you send the next. you cannot trivially spam these, and must be prepared to resend the same message multiple times.
vanilla waited for a server message before sending an unreliable. this is unsafe with certain dodgy NAT routers. its generally safe nowadays, but your movement then becomes doubly-dependant upon server->client stalls.

each packet contains multiple svcs, both reliables and unreliables. once you've delt with reliables fragmentation, you can then just read the list of packed svcs from the message. each begins directly after the previous. there's no padding after the final byte (ethernet padding will not normally exist beyond the ip layer, so there shouldn't be any padding in your code).

the engine source is open. read protocol.h for the svc list. IIRC the simple ones contain a comment saying what data there is in the svc. for more complex things, go read cl_parse.c

you're probably getting dropped because you're not doing sequences with reliables correctly. this means the server will reject the message and then think you're not sending any (because you didn't send any VALID ones). again, reliables require that you wait for an ack before you can send the next, or to timeout and resend the previous one. pay attention to your sequences!
Tremor
Posts: 13
Joined: Wed Aug 10, 2011 5:38 am

Re: Help with Quake Bot

Post by Tremor »

LordHavoc,

I believe that I am doing the 'signon' process correctly and successfully. The first reply after the initial connect request gives me the new port to talk with the server, and I am certainly using that. After having done this sequence, the server sends me player informations etc. Before having sent the prespawn, name/color, spawn, and begin messages, it would certainly do just as you said - drop the half connected state.

Spike:

As far as the initial connection sequences go, I am waiting on a reply before sending the next. However, after the connection sequence, I don't currently have any reliables to send. Of course, I will have in the future when I build some functionality into it. What I am taking from what you and LordHavoc said is to simply send unreliables at the rate you specified and the server should not drop the connect? I was previously sending unreliable keepalives but LH seems to think that the noop will work? My goal (at this point) is to simply maintain a game connection and have the bot sit in observer mode indefinitely.

Also, I will most certainly check out protocol.h and clparse
frag.machine
Posts: 2126
Joined: Sat Nov 25, 2006 1:49 pm

Re: Help with Quake Bot

Post by frag.machine »

I'd suggest you to use the source code as your primary reference, since the QSpecs are result of reverse engineering efforts back in 1996/1997.
Also I remember there were protocol changes between versions 1.01 and 1.06 (or was 1.08 ? Can't remember fro sure), even though the protocol ID was the same.
I know FrikaC made a cgi-bin version of the quakec interpreter once and wrote part of his website in QuakeC :) (LordHavoc)
Tremor
Posts: 13
Joined: Wed Aug 10, 2011 5:38 am

Re: Help with Quake Bot

Post by Tremor »

Ok, so as it turns out, the server dislikes my noop spam:

The data field is like such:

Code: Select all

00:10:00:09:00:00:00:0d:00
Server reports "unknown cmd char"
Last edited by Tremor on Fri Mar 07, 2014 12:10 am, edited 1 time in total.
Tremor
Posts: 13
Joined: Wed Aug 10, 2011 5:38 am

Re: Help with Quake Bot

Post by Tremor »

frag.machine wrote:I'd suggest you to use the source code as your primary reference, since the QSpecs are result of reverse engineering efforts back in 1996/1997.
Also I remember there were protocol changes between versions 1.01 and 1.06 (or was 1.08 ? Can't remember fro sure), even though the protocol ID was the same.
This is a good point. I noticed that there were some things in the quake specs that were either out of order or outright wrong.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Help with Quake Bot

Post by Spike »

there is no such thing as 'observer mode', unless its some 3rd-party mod-specific thing - but that has nothing what so ever to do with the protocol.
by sequences, I mean the sequence numbers in the packets. if you repeat the sequence number, the packet WILL be dropped.

if you're testing with fte as a server, use sv_port 26000 (to default it for nq instead of qw), sv_listen_nq 1 (so you don't have to implement svc_stuffcmd properly yet), showpackets 1 (so you can see the packets the server receives without needing to resort to wireshark), developer 1 (so you can see stale/dupe sequences), showdrop 1 (so you can see dropped sequences). if developer or showdrop result in any network-related prints then it means that the server is dropping packets for some reason, on a lan this is always a bad sign. I expect DP has some equivelent settings, but I can't help with those.

note that a timeout error only occurs if no valid messages arrived - it really doesn't matter what's in the message, nops, moves, reliable, unreliable, whatever. It just has to be valid and complete. individual fragments don't count, but *any* unreliable does, and any reassembled reliable does.
timing out thus proves that your packets are getting dropped by the server's network layer (ie: before the clc parsing layer), or they're simply not arriving at all (wrong address or dodgy NAT or firewall or something out of quake's control).
also, remember that the sequence value (bytes 4-7) is big endian, just as the flags+length value is.

vanilla nq has some annoying quirks about exactly when particular messages are allowed. send things at the wrong times and you can easily crash clients. be glad that you're writing a client and not a server. :P

also, clc_nop is 1. 0 is clc_bad, and should never be sent.
Tremor
Posts: 13
Joined: Wed Aug 10, 2011 5:38 am

Re: Help with Quake Bot

Post by Tremor »

Spike,

Thanks for your insight on this. Just to be specific, the bot is connecting to a proquake 3.90 server running crmod 6.5d. I've tried connecting it to a different proquake server running the CAX mod (by r00k). "observer" mode is really just a mode in these mods where you fly around invisible style so you can watch or eyecam or whatever. In fact, when CRMOD is in 'match mode', players join as observers by default. You must type 'ready' to actually spawn for the first time.

The FTE server sounds very sophisticated. There are no such commands that I know of in proquake to show any network related data. I may play around with it.
Spike wrote:note that a timeout error only occurs if no valid messages arrived
This is valuable; I had not previously noted that.
Spike wrote: remember that the sequence value (bytes 4-7) is big endian, just as the flags+length value is.
Absolutely. That's one of those things I found that was incorrect when looking at those quake specs (some little some big). Here's an example of how I am doing this in python with the result being a 4 byte integer:

Code: Select all

txnum = 12
struct.pack('>l', txnum)
'\x00\x00\x00\x0c'
Each time a reliable packet is built/sent, I increment txnum (as long as the server replies accordingly). When you say 'flags+length' what do you consider the flags to be? I thought the flags = length. At least that's the way wireshark shows it to be, 2 byte flag that represents the length of the entire data load.
Spike wrote:also, clc_nop is 1. 0 is clc_bad, and should never be sent.
This messed me up too, because the quake specs mention that 0x00 is noop and represented an error. It lists the 0x01 as a keepalive client message rather than a 'noop'.

So here is what a good keepalive packet should look like, I believe this because it matches the first unreliable packet that a standard quake client sends immediately following the initial server response (containing the new port for the conversation):

Code: Select all

\x00 - game packet
\x10 - unreliable
\x00\x09 - Length of entire packet
\x00\x00\x00\x00 - sequence number that is incremented
\x01 - keepalive
This equates into a data load that looks like:
00:10:00:09:00:00:00:00:01
Unless I'm insane, this packet should be correct. Yet, upon spamming the server with it - I've tried every 10ms, 50ms, and so on... - I always get a 'impulse 39 no response from player' from the server. I'm pretty certain that all my replies to the server are correct - simply a 0x02 response with the server sequence..and I'm not getting duplicate server messages. My signon sequence is seemingly accurate and matches what a standard quake client (using qrack/proquake) gives the server. I do get messages from the server stating what 'mode' its currently in and so forth. By the way, and in order to minimize confusion: Players who are connected to the server can see that the bot has logged on..he just doesn't stay logged on for long =)

Thanks for your time on this and helping me understand it. I feel like it's gotta be something simple or stupid. I'm ready to start parsing the server messages and giving this bot a bit more functionality, but alas I'm stalled for the moment.
LordHavoc
Posts: 322
Joined: Fri Nov 05, 2004 3:12 am
Location: western Oregon, USA
Contact:

Re: Help with Quake Bot

Post by LordHavoc »

You might want to start sequence at 1 but I don't recall off hand.

Note that reliable and unreliable sequence numbers are independent, and unreliable sequence numbers increment regardless of any reply from the server (they're not asking for a reply, they are unreliable by nature), whereas reliable sequence number advances when you get back an ACK packet (8 bytes, specific bits are set in the header only, it has no body).

The header you're seeing is two big-endian int32 numbers, one of which is the size of the packet and the type flags, the second is the sequence of the packet.

So if you send a 1 byte message (like clc_nop), be sure you send 9 as the size of the packet (header + content length).

The maximum size of a reliable message in stock quake is 8192 bytes, which is sent as 1024 byte fragments (that is, 1032 bytes with the header), the ack is sent back when the final fragment is received. I don't recall the details too well at the moment as it's been years since I poked at that code, but you may find netconn.c in darkplaces to be useful because I completely reimplemented the quake network stack there in one file (whereas quake had net_dgrm and a host of other pieces that worked in layers).

In general numbers in the quake protocol are little endian, except the header ints (2x int32) are big endian.

I believe 1024 bytes of content is the longest single packet that stock Quake can send or receive, darkplaces allows up to 65536 to be received, but won't send more than 1400 in one packet (to leave plenty of room for network tunneling headers like VPN stuff), of course that cap is dropped to 1024 in quake protocol for compatibility.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Help with Quake Bot

Post by Spike »

#define NETFLAG_LENGTH_MASK 0x0000ffff
#define NETFLAG_DATA 0x00010000
#define NETFLAG_ACK 0x00020000
#define NETFLAG_NAK 0x00040000
#define NETFLAG_EOM 0x00080000
#define NETFLAG_UNRELIABLE 0x00100000
#define NETFLAG_CTL 0x80000000
this is the first four bytes, in big-endian order (ie: as you see them written in hex).
as lh says, the length is the total length of the udp packet, including this 8-byte header... how he manages to pack 65536 bytes of payload in there I have no idea... surely the max is 65527.
the next four bytes are the sequence number.
the clc data follows that, in little-endian order.
don't bother with naks, and ctl messages are for connectionless things.
if you receive a reliable fragment, you will _NEED_ to ack it. failure to ack a reliable means the server will just keep resending it.
reliables will consume multiple sequences. if you have a 2500 reliable due for you, the first two packets will both have length values of 1032 with 1024 bytes of payload data. the flags for the first two parts will be the same, with just NETFLAG_DATA set. the sequences will be first and first+1, the third packet will have a payload of 328 (total length 336), with NETFLAG_DATA|NETFLAG_EOM, and sequence first+2. Every part of the message needs to be acked separately as they come. You can only safely parse the reliable once you receive the final EOM, because otherwise you'll find yourself wanting to read data that isn't available yet, so buffer+ack reliables until you receive one with EOM, and then ack that too then parse the whole lot as a single lump.
unreliables are never fragmented and don't use EOM at all.

you really do need to read the docs (ie: the engine source). one of:
net_dgram.c (vanilla, Datagram_GetMessage for reading, Datagram_SendUnreliableMessage for sending unreliables, Datagram_SendMessage, SendMessageNext, ReSendMessage Datagram_CanSendMessage, for sending+resending reliables.)
net_chan.c (fte, look for NQPROT ifdefs, the parsing code is 150ish lines net_message contains the entire inbound buffer and contains the result too but with a changed offset, and the transmit functionality (100ish lines) is meant to be spammed once a frame with the new unreliable buffer passed via arguments while the active reliable is in internally stored in chan->reliable_* and the waiting/next reliable is in chan->message.* which is bumped to active when the active reliable becomes clear due to being acked)
netconn.c (dp)

clients generally don't need to care about the exact limits. python probably hides all the memory limits for you anyway. you will need to support defragmenting the server's message.
Tremor
Posts: 13
Joined: Wed Aug 10, 2011 5:38 am

Re: Help with Quake Bot

Post by Tremor »

LordHavoc and Spike:

Thank you so much for taking your time on this. I appreciate all the input and extra info! I will study these last couple of posts in detail. I appreciate being pointed in the right direction on this. I'll update this thread as I progress - I may have other questions or whatever. Gonna need a day or so to check through the engine source.

I do have one last question for now.

I am indeed responding to each and every reliable. No doubt about it. However, I'm not currently buffering the fragments. So, as I understand it, I simply need to check that 1. the packet is reliable and 2. the length is 1032 - as soon as that happens, start buffering until an EOM appears? Are the quake specs correct when they say that an EOM message will always be a 0x09 reliable type?
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Help with Quake Bot

Post by Spike »

DATA|EOM means its the final reliable in the block.
DATA means that its a section that is NOT the final one.
ACK means that its a reliable ack
UNRELIABLE means that its an unreliable message.
any other combination is invalid for an active stream (ignoring length and CCREQ/CCREP messages).

don't bother checking the segment length for validating reliable fragments, by convention they'll be 1024/1032 (depending on how you see it), but they might be smaller for MTU reasons, or larger for efficiency reasons if the engine is modified. The protocol works for anything between 1 and 65535, and remember that its a single segment so the maximum size can be somewhat unbounded... 20mb reliables is fun... yeah... you might want to provide a sanity limit for the max size somewhere (ideally more than 8192), but don't depend on the segment always being 1024.
you might want to ensure that a non-final segment is always at least 256 bytes though, as anything smaller is pointless and can result in wasted resources (read: probably a dos attack), just be sure to allow other sizes too.
Tremor
Posts: 13
Joined: Wed Aug 10, 2011 5:38 am

Re: Help with Quake Bot

Post by Tremor »

10-4, although I don't see the MTU as being problematic sine that's a layer 2 issue that should be auto-resolved/rebuilt. I suppose though, it wouldn't hurt to account for that.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Help with Quake Bot

Post by Spike »

ipv4 says an mtu must be at least 576. anything bigger is allowed to fragment. and if its allowed to fragment, its allowed to be dropped because of dodgy routers that can't handle it or hosts that think its a DoS attack or hosts that are merely memory limited.
be generous in what you accept, and strict in what you generate, that's the old moto. Which naturally means you don't test the things you're generous with accepting, opening your program up to all sorts of remote exploits, but hey...


ideally a server/client would use a size equal to totalsize/fragmentsrequired. then each fragment has the same sort of duration to it, which can make a difference on slow (56k) links with random reliables blocking the unreliables interspersed within it. keeping the sizes more consistant is just a somewhat sane idea, where possible.
but yeah, implementations ignore all that and just use 1024 for all but the EOM, in a really lame way.
Post Reply