Network Packet Structures

  • I'm making a placeholder here to discuss the network packet structures and what's in them.

    Each packet is a structure of the following Source:

    using System;
    using FFXIVAPP.Common.Core.Network.Interfaces;
    namespace FFXIVAPP.Common.Core.Network
        public class NetworkPacket : INetworkPacket
            public uint Key { get; set; }
            public byte[] Buffer { get; set; }
            public int CurrentPosition { get; set; }
            public int MessageSize { get; set; }
            public DateTime PacketDate { get; set; }

    The Key is the packet type while buffer contains the data being sent.

    I'll have more info as it comes along, feel free to add posts with information and I will update this one with help for everyone.

  • The network reading code seems to be dropping some packets. I'm going to look into it.

  • Any update on this? I've been writing a plugin that parses some NetworkPacketEvents, and it kind of works. Sometimes a lot of network data goes through at once, and sometimes it stops working entirely, using the latest official build.

    I've tried building from git (both master and sharpdx branches) however they both seem to throw overflow exceptions on TCPHeader.cs line 71, when headerLength is larger than nReceived, and no NetworkPacketEvents are generated in this case.

    Thanks for the great network support for plugins so far! I'd hate to figure all this out on my own.

  • Not yet no.

    I've been working 16 hour days for half a year now so I try when I get time but usually too exhausted most of the time.

    All the magic happens in here however:

    If there is an issue around it not fixing a packet it would be there. FFXIV packets come in chunks and parts of a chunk can be in one packet and the other part in another.

  • I just got interested in this project tonight and looked into some of this - I don't know very much about TCP & how networking in games work, and I thought this would be a neat way to learn.

    NetworkWorker.cs looks a bit confused. It looks like it used to be multi-threaded but now isn't.

    For example, there are locks everywhere, but everything that's locking should (as far as I can tell) be executed on the UI thread.

    Also, this dispatch in the OnReceive call is likely a cause of random failures:

    DispatcherHelper.Invoke(() => ParseData(asyncState, asyncState.Buffer, nReceived));

    From what I can tell, that's going to dispatch the parsing of the buffer, but just after this, it's going to trash the buffer with a new BeginReceive / EndReceive call. I actually though that would be the cause of the random failures above, but alas, a naive change does not fix it:

    byte[] bufferCopy = new byte[0x20000];
    asyncState.Buffer.CopyTo(bufferCopy, 0);
    DispatcherHelper.Invoke(() => ParseData(asyncState, bufferCopy, nReceived));

    In the latest code from master, I actually don't get any networking events to fire. I may try and look more into this later this week.

  • I'm not the most experienced, but I decoded many of the game's frames just after launch. I have some information. Whether you all find it of any utility, I leave to your own determination.

    Even with the best network capture code in the world, it's still going to miss some. You are watching the buffer the same as the game client is. If the game client sees > processes > acknowledges the buffer before you even see it, you won't see those packets. You can increase your process priority towards realtime to attempt to get the jump on the client, but experience has told me even that is not fool proof.

    The game uses a rolling frame system that also makes it fraught to rely on TCP.

    If TCP resends a packet, the one that comes back won't Necessarily match the old one. Sometimes it can even include new frames, because the resend causes a reblock of any lost data. The server isn't aware of TCP's transmission windows, only the contents of any frames present where that cleave happened. They seem to have co-opted the TCP retransmission system a bit to shovel in more data if it's opportunistically possible. You can especially see this if there is a new tick frame available in a retransmission, since it will have the new tick time!

    The best method I found is to simply treat all data as a time-delimited list of frames. Rubberbanding in movement is probably the most logical way I have to explain the process. The server sends position 1, then 2, then resends 1 (because of TCP retransmission due to them not receiving the ACK), then sends 3. If you treated each frame as a single data point and process in order, then when the secondary 1 comes in, you'll rubberband, then snap forward at 3. The better way to handle it is to realize you've already processed 1 and 2, discard the secondary 1, and then process 3 when it's ready. I'm pretty sure the game handles this as part of it's network stack, before the engine renders anything. It's not perfect (observe stuck animations) even for the client, but it's the cleanest method I found. This is made especially difficult as the game doesn't seem to utilize any concept of a serial number for frames, so how it tells them apart.. I don't know!

    Some notes I made:

    • Some frames are zlib encoded. These contain more frames. Fun Times.
    • Some frames contain more frames. Fun Times. Chat messages a good example if two come in at once. There will be a super-header with sub-headers for the actual frames. Usually you get just the actual frames, but sometimes the super-header is present, I don't know what differentiates them. Another example is paginated results. You need to process the super-header and gather it's information, but then process each frame individually. They may or may not be sequential, depending on actions around the focus.
    • What is labeled the key seems too small of a type. There are multiple pieces of information that form the frame type, namely the first 2 bytes, and a byte deeper in (the one I believe flagged as the key). The first two bytes seem to be the sub type, and the one further in is the main type. I say it's this way because I need all 3 bytes to differentiate packets properly.

    Right now all my code is Python, and it's ugly as hell (I parse with unpack() ). I'm working on getting it cleaned up into something useful and once I do I will pass it along. There are still some frames that come through that for all intents and purposes seem to be random noise (encrypted?), but I hope to sort those out with a renewed push.

    P.S. A packet can contain the "middle" of a frame. P1 (Packet 1), P2, P3 can all be one single frame. P1 contains the frame size, then you need to attempt to append P2 and P3 to it. TCP retransmission is a bugger here, as it can immediately throw a wrench into that process.

  • @Zacharot - thanks for that information, it's really valuable. In the end, resorting to WireShark to try and parse the frames that were incoming / outgoing.

    My end goal was to be able to dump market data to excel & do some analysis on it for what's useful to craft while leveling. I got pretty excited - I was able to decode several important pieces of outgoing frames, such as "This is a request for items" and "This is a request for the next page of items". I even started writing a small C# lib to parse it, then I hit a wall. The incoming frames have a huge amount of entropy. Are those frames the zlib encoded ones or are they blowfish encrypted? I thought I read somewhere that they were blowfish encrypted, but I could be wrong. I've never used OllyDbg / IDA, so the ability to dump the memory after it's been unzipped/unencrypted is a really, really huge wall for me. This technique is completely outside of my realm of experience.

  • So some new fixes have been pushed up to the master branch; it should have no packet loss. Will be testing and providing two ways to parse network. Raw sockets or using WinPCap.

    With raw sockets you have to add ffxivapp as an allowed tcp/udp incoming application. WinPCap requires it's own allowances.

  • After latest update. Nothing work anymore. I have change the setting in Performance & Logging. The FFXIAAP status is still offline.

    ** Windows 10 Pro x64 with Kaspersky internet security.

  • @Mr.-Cook If the status says offline that means it's not finding the game running at all. Has nothing to do with the internet though as far as I'm aware.

    Does anything else load up? Plugins/etc.?

  • Yes, evertyhing load up, but event and radar not working (didn't test the parse) . I'm running the game on Dx9 . It is seem it doesn;t detect my game.

    I rolled back to the previous version. (Luckily i backup the folder) and it is running fine. But I not dare to update the plugin yet.

  • @agoln There is no encryption anywhere on the incoming. I have a 100% parse rate to get the inner messages.

    My terminology sometimes isn't the clearest, but I'm uncertain of exactly what these things are called in an information system like this.

    Packet (frame header + chat messages | zlib(blocks)))


    Since 3.0 a lot seems to have been rearranged. There is more consistency in what is encoded and what isn't. Amusingly only the tells seem to not be zlib compressed, though I can't figure out why.

    Byte offsets are 0 indexed. Feel free to convert my notes if you are simply counting as I so often do.

    On the frame, headers seem to be 40 bytes. The only bytes I actively make use of are 0:1, 24:27, 30:31, 33 and 40:41. 0:2 and 40:41 are my "type" bytes. 24:27 are the frame length. Types are 0:1==5252 and 0:1==0000 && 40:41 == 789C || 1800
    I haven't found useful data in anything else so I just filter and use these.

    Once you have a complete frame:
    A frame can contain multiple blocks - they usually do. Number of expected blocks is 30:31. If the blocks are compressed, that's stored in 33.
    If the blocks are compressed, strip the first 40 bytes and the rest is a zlib encoded set of blocks.
    If it's not compressed, strip the first 40 bytes and the rest is a set of blocks.

    From here it's simple. Blocks are stacked, the length bytes are just the first 4 bytes of each block.

    In rare cases, I've found that a block can span frames, I don't know why this happens, but it's a very rare thing. If you have a short block, hang onto it and check the next frame's first block to see if the two go together.

    There are other exceptions, such as I've found frames inside of frames. I have a check for this in my code. I've found zlib'd frames inside of zlib'd frames instead of blocks. As noted above I've found truncated blocks. I've found blocks that contain more blocks!

    Once you have the string of blocks, which are likely just structs, you are off to the races. I recommend writing something to compare like structs to look for changes, it was the fastest way I found to decode them.

    Nearly every block appears to have a 32 byte header, I haven't figured out those nuances between them all yet, or what they represent, such as why some patterns are repeated over and over.

    Some blocks/structs inform ones to follow. An example is the history block for the market board.

    Byte 14 and Byte 15 I call b1 and b2 - these are my keys to block type, history count is b1==04 && b2 == 01
    I'll use 32+N to signify that the header is 32 bytes, but 32+28 would be the 61st byte, not the 29th. My code is set up to strip the header, but I want to clarify the exact position and don't feel like converting all the bytes and making a mistake in translation.

    byte 32+28:31 = item_id
    byte 32+38:39 = number of items for sale rows (not history)

    You need these, because the items data block doesn't specify number of expected rows.

    history data block is b1==09 && b2==01
    32+0:3 = item_id
    every 52 bytes is a history row so 32+:51, 32+52:103, etc
    i=integer (4 bytes)
    b=byte (1 byte)
    s=string (N bytes)
    row structure is IIIIBBs(34)
    item_id, price, date(unix), quantity, is_hq, has_materia, buyer_name

    Item rows: b1 == 5 && b2 == 1
    Header I believe is 28 bytes, there may be useful info in there, but I just iterate the data itself.

    Row is 112 bytes

    u = unknown
    auction_id,u,u,u,u,retainer id,user1,u,user_id,user2,u,crafted by,user3,u,price,fee,quant,item_id,post_date(unix),is_item,u,u,u,u,u,u,materia1,materia2,materia3,materia4,u,u,u,u,u,u,u,u,retainer name, is_hq, has_materia, city,u

    I may have messed up those offsets, but I believe you have enough info either way.

    That should speed you on your way.

Log in to reply

Looks like your connection to Network Packet Structures was lost, please wait while we try to reconnect.